2024-09-12T02:27:12.487 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-12T02:27:12.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T02:27:12.607 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7901630' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 086c737fa75c22d521a3932c59b31c5e1d1c9ef4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 481 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 57/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 086c737fa75c22d521a3932c59b31c5e1d1c9ef4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM+8LNgQ/MXkSz+klrPENBX67jmc4KevN7m/9UOtnNMDCx9YhTZfSz8OKbOw10A/NmSkdSaT257nnc3uKUz/J5M= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFAMM/uD/wsQ5JfUnpy9eJjI/6tl0IKQ+GYJCbiSAwglgYtnw2v+Yd6eE5EQxwB1hrF7GYYV9Amae5A03jthcrw= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFmgP6gepFQ42GZK6ygS/An6V31qUHOGex6LaUJ000F70TjboddvrTo87hFYKVTUu5LdSald+1iImuHHwkbt6Dc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_21:24:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-12T02:27:12.607 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa; will attempt to use it 2024-09-12T02:27:12.608 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa/tasks 2024-09-12T02:27:12.608 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-12T02:27:12.610 INFO:teuthology.task.internal:Checking packages... 2024-09-12T02:27:12.631 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-12T02:27:12.631 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-12T02:27:12.631 INFO:teuthology.packaging:ref: None 2024-09-12T02:27:12.632 INFO:teuthology.packaging:tag: None 2024-09-12T02:27:12.632 INFO:teuthology.packaging:branch: squid 2024-09-12T02:27:12.632 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:27:12.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-12T02:27:12.835 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-314.g086c737f 2024-09-12T02:27:12.837 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-12T02:27:12.844 INFO:teuthology.task.internal:no buildpackages task found 2024-09-12T02:27:12.844 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-12T02:27:12.917 INFO:teuthology.task.internal:Saving configuration 2024-09-12T02:27:12.937 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-12T02:27:12.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-12T02:27:12.968 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 02:22:32.729386', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM+8LNgQ/MXkSz+klrPENBX67jmc4KevN7m/9UOtnNMDCx9YhTZfSz8OKbOw10A/NmSkdSaT257nnc3uKUz/J5M='} 2024-09-12T02:27:12.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 02:22:32.730827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFAMM/uD/wsQ5JfUnpy9eJjI/6tl0IKQ+GYJCbiSAwglgYtnw2v+Yd6eE5EQxwB1hrF7GYYV9Amae5A03jthcrw='} 2024-09-12T02:27:13.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 02:22:32.732218', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFmgP6gepFQ42GZK6ygS/An6V31qUHOGex6LaUJ000F70TjboddvrTo87hFYKVTUu5LdSald+1iImuHHwkbt6Dc='} 2024-09-12T02:27:13.012 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-12T02:27:13.018 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-12T02:27:13.018 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-12T02:27:13.018 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-12T02:27:13.018 INFO:teuthology.run_tasks:Running task console_log... 2024-09-12T02:27:13.118 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f449564b490>, signals=[15]) 2024-09-12T02:27:13.118 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-12T02:27:13.125 INFO:teuthology.task.internal:Opening connections... 2024-09-12T02:27:13.125 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T02:27:13.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:27:13.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-09-12T02:27:13.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:27:13.276 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-09-12T02:27:13.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:27:13.354 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-12T02:27:13.361 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-12T02:27:13.380 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-12T02:27:13.380 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-09-12T02:27:13.437 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-09-12T02:27:13.438 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T02:27:13.438 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-09-12T02:27:13.438 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T02:27:13.438 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T02:27:13.438 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T02:27:13.438 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-12T02:27:13.460 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-09-12T02:27:13.478 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-09-12T02:27:13.479 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-09-12T02:27:13.536 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-09-12T02:27:13.536 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-09-12T02:27:13.536 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-09-12T02:27:13.536 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-09-12T02:27:13.536 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T02:27:13.537 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T02:27:13.537 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-09-12T02:27:13.560 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-09-12T02:27:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-09-12T02:27:13.579 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-09-12T02:27:13.637 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-09-12T02:27:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T02:27:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-09-12T02:27:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T02:27:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T02:27:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T02:27:13.638 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-09-12T02:27:13.660 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-12T02:27:13.675 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-12T02:27:13.702 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-12T02:27:13.702 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-12T02:27:13.705 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-09-12T02:27:13.708 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-09-12T02:27:13.725 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-12T02:27:13.731 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-12T02:27:13.731 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-12T02:27:13.762 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-09-12T02:27:13.765 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-09-12T02:27:13.793 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-12T02:27:13.869 INFO:teuthology.run_tasks:Running task kernel... 2024-09-12T02:27:13.882 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-12T02:27:13.883 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-12T02:27:13.883 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-12T02:27:13.883 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T02:27:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:27:13.900 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-12T02:27:13.958 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-503.el9.x86_64 2024-09-12T02:27:13.958 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-503.el9.x86_64 2024-09-12T02:27:13.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-09-12T02:27:18.031 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:02 2024-09-12T02:27:19.857 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-12T02:27:24.375 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-12T02:27:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-12T02:27:26.746 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-12T02:27:32.377 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-12T02:27:34.467 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-12T02:27:34.467 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-12T02:27:34.566 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T02:27:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T02:27:34.575 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T02:27:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-12T02:27:34.577 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:27:34.577 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 79 M 2024-09-12T02:27:34.577 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 126 M 2024-09-12T02:27:34.577 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T02:27:34.820 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 199 kB/s | 23 kB 00:00 2024-09-12T02:27:41.612 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-12T02:27:47.319 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.3 MB/s | 29 MB 00:12 2024-09-12T02:27:50.311 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-12T02:27:50.311 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T02:27:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Total 5.0 MB/s | 79 MB 00:15 2024-09-12T02:27:51.017 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T02:27:51.402 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T02:27:51.402 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T02:27:52.411 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T02:27:52.412 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T02:27:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T02:27:55.688 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-12T02:27:55.770 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T02:27:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T02:27:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T02:28:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T02:28:03.866 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:09.010 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-12T02:28:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T02:28:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:28:49.068 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T02:28:49.435 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T02:28:49.526 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T02:28:49.527 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-12T02:28:49.527 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T02:28:49.527 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T02:28:49.527 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-12T02:28:49.527 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T02:28:49.527 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T02:28:49.585 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T02:28:49.586 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T02:28:49.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 82.el9 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-12T02:28:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8274670 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:28:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-12T02:28:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-12T02:28:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-12T02:28:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-09-12T02:28:49.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:28:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-09-12T02:28:49.620 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-12T02:28:49.620 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-505.el9.x86_64 2024-09-12T02:28:49.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-12T02:28:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-12T02:28:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-12T02:28:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-12T02:28:51.816 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-12T02:28:51.831 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.XXJ7STnNq3 2024-09-12T02:28:51.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XXJ7STnNq3 2024-09-12T02:28:51.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.XXJ7STnNq3 2024-09-12T02:28:52.032 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.XXJ7STnNq3 is 6KB 2024-09-12T02:28:52.081 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.XXJ7STnNq3 2024-09-12T02:28:52.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-09-12T02:28:52.157 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-12T02:28:52.158 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-12T02:28:52.158 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-12T02:28:52.158 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-12T02:28:52.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-12T02:28:52.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-12T02:29:22.312 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-12T02:29:22.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:29:22.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:29:40.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-12T02:29:49.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:29:49.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:29:52.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-12T02:30:04.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:30:04.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:30:07.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-12T02:30:22.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:30:22.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:30:26.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-12T02:30:44.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:30:44.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:31:02.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-12T02:31:23.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:31:23.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:31:23.819 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-12T02:31:24.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:31:24.165 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-12T02:31:24.166 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-12T02:31:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-505.el9.x86_64 2024-09-12T02:31:24.181 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-12T02:31:24.181 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T02:31:24.181 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-12T02:31:25.182 DEBUG:teuthology.parallel:result is None 2024-09-12T02:31:25.182 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-12T02:31:25.189 INFO:teuthology.task.internal:Creating test directory... 2024-09-12T02:31:25.189 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T02:31:25.192 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T02:31:25.195 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T02:31:25.214 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-12T02:31:25.219 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-12T02:31:25.268 INFO:teuthology.task.internal:Creating archive directory... 2024-09-12T02:31:25.269 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T02:31:25.271 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T02:31:25.274 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T02:31:25.370 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-12T02:31:25.376 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-12T02:31:25.376 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T02:31:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:31:25.393 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T02:31:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:31:25.411 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T02:31:25.413 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T02:31:25.436 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T02:31:25.471 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.493 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.498 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.514 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.528 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.534 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T02:31:25.536 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-12T02:31:25.543 INFO:teuthology.task.internal:Configuring sudo... 2024-09-12T02:31:25.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T02:31:25.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T02:31:25.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T02:31:25.575 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-12T02:31:25.584 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-12T02:31:25.584 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T02:31:25.616 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T02:31:25.619 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T02:31:25.637 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:25.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:25.769 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:25.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:25.900 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:31:25.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T02:31:25.969 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:26.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:26.068 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:26.152 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:26.198 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:31:26.198 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T02:31:26.267 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:26.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T02:31:26.371 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:26.456 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T02:31:26.499 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:31:26.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T02:31:26.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-12T02:31:26.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-09-12T02:31:26.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-09-12T02:31:26.607 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T02:31:26.620 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T02:31:26.648 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T02:31:27.091 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-12T02:31:27.107 INFO:teuthology.task.internal:Starting timer... 2024-09-12T02:31:27.107 INFO:teuthology.run_tasks:Running task pcp... 2024-09-12T02:31:27.150 INFO:teuthology.run_tasks:Running task selinux... 2024-09-12T02:31:27.196 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-12T02:31:27.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-09-12T02:31:27.253 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-09-12T02:31:27.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-09-12T02:31:27.325 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-09-12T02:31:27.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-09-12T02:31:27.396 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-09-12T02:31:27.399 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-12T02:31:27.399 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-09-12T02:31:27.422 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-09-12T02:31:27.422 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-09-12T02:31:27.447 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-09-12T02:31:27.447 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-09-12T02:31:27.481 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-09-12T02:31:27.482 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-09-12T02:31:27.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T02:31:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:31:27.513 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T02:31:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:31:27.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T02:31:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:31:27.572 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-12T02:31:27.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-09-12T02:31:27.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-09-12T02:31:27.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-09-12T02:31:27.674 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-12T02:31:27.687 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-12T02:31:27.698 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-12T02:31:27.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi033.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-12T02:35:39.571 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-09-12T02:35:39.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:35:39.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:35:39.656 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-12T02:35:39.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T02:35:39.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-12T02:35:39.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:35:39.816 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-09-12T02:35:39.894 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-09-12T02:35:39.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-12T02:35:39.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T02:35:39.974 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-12T02:35:40.054 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-12T02:35:40.055 INFO:teuthology.run_tasks:Running task clock... 2024-09-12T02:35:40.069 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-12T02:35:40.070 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T02:35:40.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T02:35:40.073 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T02:35:40.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T02:35:40.076 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T02:35:40.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T02:35:40.106 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T02:35:40.108 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T02:35:40.122 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T02:35:40.124 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T02:35:40.125 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T02:35:40.139 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T02:35:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-09-12T02:35:40.151 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-09-12T02:35:40.162 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-09-12T02:35:40.163 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-09-12T02:35:40.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-09-12T02:35:40.174 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-09-12T02:35:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T02:35:40.178 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T02:35:40.188 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T02:35:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T02:35:40.193 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T02:35:40.202 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T02:35:40.251 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-09-12T02:35:40.252 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-12T02:35:40.254 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T02:35:40.254 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-09-12T02:35:40.254 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.255 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.255 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.255 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.255 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T02:35:40.255 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-12T02:35:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.256 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.260 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T02:35:40.260 INFO:teuthology.run_tasks:Running task install... 2024-09-12T02:35:40.269 DEBUG:teuthology.task.install:project ceph 2024-09-12T02:35:40.269 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-12T02:35:40.269 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-12T02:35:40.269 INFO:teuthology.task.install:Using flavor: default 2024-09-12T02:35:40.275 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-12T02:35:40.275 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-12T02:35:40.275 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-12T02:35:40.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:35:40.279 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-12T02:35:40.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:35:40.280 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-12T02:35:40.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:35:40.417 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-12T02:35:40.418 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-12T02:35:40.436 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-12T02:35:40.436 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-12T02:35:40.441 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-12T02:35:40.442 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-12T02:35:40.534 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T02:35:40.534 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:35:40.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T02:35:40.544 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T02:35:40.544 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:35:40.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T02:35:40.554 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T02:35:40.554 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:35:40.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T02:35:40.561 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-12T02:35:40.561 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T02:35:40.570 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-12T02:35:40.571 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T02:35:40.579 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-12T02:35:40.579 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T02:35:40.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T02:35:40.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T02:35:40.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T02:35:40.705 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T02:35:40.712 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T02:35:40.719 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T02:35:40.771 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-09-12T02:35:40.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-09-12T02:35:40.776 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-09-12T02:35:40.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-09-12T02:35:40.783 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-09-12T02:35:40.785 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-09-12T02:35:41.002 INFO:teuthology.orchestra.run.smithi184.stdout:45 files removed 2024-09-12T02:35:41.016 INFO:teuthology.orchestra.run.smithi033.stdout:45 files removed 2024-09-12T02:35:41.024 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-09-12T02:35:41.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-12T02:35:41.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-12T02:35:41.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-12T02:35:41.581 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 303 kB/s | 84 kB 00:00 2024-09-12T02:35:41.603 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 301 kB/s | 84 kB 00:00 2024-09-12T02:35:41.724 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 224 kB/s | 84 kB 00:00 2024-09-12T02:35:41.865 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-12T02:35:41.895 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-12T02:35:42.016 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-12T02:35:42.132 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-12T02:35:42.263 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-09-12T02:35:42.292 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-12T02:35:42.749 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-12T02:35:42.804 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-12T02:35:44.900 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.2 MB 00:02 2024-09-12T02:35:44.950 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-12T02:35:45.774 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-12T02:35:48.253 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 20 MB 00:02 2024-09-12T02:35:49.850 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.5 MB 00:01 2024-09-12T02:35:51.276 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-12T02:35:51.373 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-12T02:35:52.322 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-12T02:35:52.821 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-12T02:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-12T02:35:53.761 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-12T02:35:54.347 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-12T02:35:55.451 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-12T02:35:57.686 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-12T02:35:59.434 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-12T02:35:59.988 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:35:59.989 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:36:00.000 INFO:teuthology.orchestra.run.smithi184.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-12T02:36:00.000 INFO:teuthology.orchestra.run.smithi184.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-12T02:36:00.001 INFO:teuthology.orchestra.run.smithi184.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-12T02:36:00.002 INFO:teuthology.orchestra.run.smithi184.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-12T02:36:00.003 INFO:teuthology.orchestra.run.smithi184.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-12T02:36:00.006 INFO:teuthology.orchestra.run.smithi184.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-12T02:36:00.007 INFO:teuthology.orchestra.run.smithi184.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-12T02:36:00.068 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-12T02:36:00.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-12T02:36:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 228 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-shell x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 27 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-top noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 18 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-12T02:36:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-12T02:36:00.095 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-12T02:36:00.096 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T02:36:00.097 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T02:36:00.098 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T02:36:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-12T02:36:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T02:36:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T02:36:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T02:36:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T02:36:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout:Install 148 Packages 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 8 Packages 2024-09-12T02:36:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:00.106 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 226 M 2024-09-12T02:36:00.106 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-12T02:36:00.770 INFO:teuthology.orchestra.run.smithi184.stdout:(1/156): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-09-12T02:36:00.962 INFO:teuthology.orchestra.run.smithi184.stdout:(2/156): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86 4.6 MB/s | 897 kB 00:00 2024-09-12T02:36:01.054 INFO:teuthology.orchestra.run.smithi184.stdout:(3/156): ceph-immutable-object-cache-19.1.1-298 1.5 MB/s | 145 kB 00:00 2024-09-12T02:36:01.138 INFO:teuthology.orchestra.run.smithi184.stdout:(4/156): ceph-base-19.1.1-298.gb53b6d3e.el9.x86 11 MB/s | 5.4 MB 00:00 2024-09-12T02:36:01.222 INFO:teuthology.orchestra.run.smithi184.stdout:(5/156): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_ 18 MB/s | 1.5 MB 00:00 2024-09-12T02:36:01.223 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-12T02:36:01.280 INFO:teuthology.orchestra.run.smithi184.stdout:(6/156): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_ 11 MB/s | 2.4 MB 00:00 2024-09-12T02:36:01.489 INFO:teuthology.orchestra.run.smithi184.stdout:(7/156): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_ 18 MB/s | 4.7 MB 00:00 2024-09-12T02:36:01.831 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:36:01.831 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:36:01.846 INFO:teuthology.orchestra.run.smithi033.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-12T02:36:01.847 INFO:teuthology.orchestra.run.smithi033.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-12T02:36:01.848 INFO:teuthology.orchestra.run.smithi033.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-12T02:36:01.849 INFO:teuthology.orchestra.run.smithi033.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-12T02:36:01.851 INFO:teuthology.orchestra.run.smithi033.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-12T02:36:01.854 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-12T02:36:01.855 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-12T02:36:01.856 INFO:teuthology.orchestra.run.smithi184.stdout:(8/156): ceph-common-19.1.1-298.gb53b6d3e.el9.x 17 MB/s | 21 MB 00:01 2024-09-12T02:36:01.898 INFO:teuthology.orchestra.run.smithi184.stdout:(9/156): ceph-selinux-19.1.1-298.gb53b6d3e.el9. 608 kB/s | 25 kB 00:00 2024-09-12T02:36:01.922 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-09-12T02:36:01.943 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-12T02:36:01.943 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-09-12T02:36:01.943 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-12T02:36:01.943 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-09-12T02:36:01.943 INFO:teuthology.orchestra.run.smithi033.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-12T02:36:01.944 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 228 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 27 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 18 k 2024-09-12T02:36:01.945 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-12T02:36:01.946 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-12T02:36:01.947 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-12T02:36:01.948 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-12T02:36:01.949 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T02:36:01.950 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T02:36:01.951 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T02:36:01.952 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T02:36:01.953 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T02:36:01.954 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T02:36:01.955 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T02:36:01.956 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T02:36:01.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T02:36:01.958 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T02:36:01.959 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T02:36:01.959 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T02:36:01.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T02:36:01.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T02:36:01.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T02:36:01.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T02:36:01.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T02:36:01.961 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:Install 148 Packages 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 8 Packages 2024-09-12T02:36:01.962 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:01.963 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 226 M 2024-09-12T02:36:01.963 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-09-12T02:36:02.073 INFO:teuthology.orchestra.run.smithi184.stdout:(10/156): ceph-radosgw-19.1.1-298.gb53b6d3e.el9 18 MB/s | 11 MB 00:00 2024-09-12T02:36:02.123 INFO:teuthology.orchestra.run.smithi184.stdout:(11/156): cephfs-mirror-19.1.1-298.gb53b6d3e.el 4.5 MB/s | 228 kB 00:00 2024-09-12T02:36:02.324 INFO:teuthology.orchestra.run.smithi184.stdout:(12/156): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86 16 MB/s | 17 MB 00:01 2024-09-12T02:36:02.341 INFO:teuthology.orchestra.run.smithi184.stdout:(13/156): cephfs-shell-19.1.1-298.gb53b6d3e.el9 124 kB/s | 27 kB 00:00 2024-09-12T02:36:02.366 INFO:teuthology.orchestra.run.smithi184.stdout:(14/156): libcephfs-devel-19.1.1-298.gb53b6d3e. 765 kB/s | 32 kB 00:00 2024-09-12T02:36:02.399 INFO:teuthology.orchestra.run.smithi184.stdout:(15/156): libcephfs2-19.1.1-298.gb53b6d3e.el9.x 13 MB/s | 752 kB 00:00 2024-09-12T02:36:02.425 INFO:teuthology.orchestra.run.smithi184.stdout:(16/156): libcephsqlite-19.1.1-298.gb53b6d3e.el 2.7 MB/s | 163 kB 00:00 2024-09-12T02:36:02.450 INFO:teuthology.orchestra.run.smithi184.stdout:(17/156): librados-devel-19.1.1-298.gb53b6d3e.e 2.5 MB/s | 127 kB 00:00 2024-09-12T02:36:02.483 INFO:teuthology.orchestra.run.smithi184.stdout:(18/156): libradosstriper1-19.1.1-298.gb53b6d3e 8.5 MB/s | 506 kB 00:00 2024-09-12T02:36:02.542 INFO:teuthology.orchestra.run.smithi184.stdout:(19/156): python3-ceph-argparse-19.1.1-298.gb53 772 kB/s | 45 kB 00:00 2024-09-12T02:36:02.659 INFO:teuthology.orchestra.run.smithi184.stdout:(20/156): librgw2-19.1.1-298.gb53b6d3e.el9.x86_ 26 MB/s | 5.3 MB 00:00 2024-09-12T02:36:02.684 INFO:teuthology.orchestra.run.smithi184.stdout:(21/156): python3-ceph-common-19.1.1-298.gb53b6 959 kB/s | 136 kB 00:00 2024-09-12T02:36:02.693 INFO:teuthology.orchestra.run.smithi033.stdout:(1/156): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-09-12T02:36:02.710 INFO:teuthology.orchestra.run.smithi184.stdout:(22/156): python3-cephfs-19.1.1-298.gb53b6d3e.e 3.1 MB/s | 162 kB 00:00 2024-09-12T02:36:02.735 INFO:teuthology.orchestra.run.smithi184.stdout:(23/156): python3-rados-19.1.1-298.gb53b6d3e.el 6.2 MB/s | 321 kB 00:00 2024-09-12T02:36:02.760 INFO:teuthology.orchestra.run.smithi184.stdout:(24/156): python3-rbd-19.1.1-298.gb53b6d3e.el9. 5.9 MB/s | 301 kB 00:00 2024-09-12T02:36:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:(25/156): python3-rgw-19.1.1-298.gb53b6d3e.el9. 1.9 MB/s | 100 kB 00:00 2024-09-12T02:36:02.811 INFO:teuthology.orchestra.run.smithi184.stdout:(26/156): rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-12T02:36:03.053 INFO:teuthology.orchestra.run.smithi184.stdout:(27/156): rbd-mirror-19.1.1-298.gb53b6d3e.el9.x 12 MB/s | 3.1 MB 00:00 2024-09-12T02:36:03.086 INFO:teuthology.orchestra.run.smithi184.stdout:(28/156): ceph-grafana-dashboards-19.1.1-298.gb 961 kB/s | 32 kB 00:00 2024-09-12T02:36:03.129 INFO:teuthology.orchestra.run.smithi033.stdout:(2/156): ceph-base-19.1.1-298.gb53b6d3e.el9.x86 8.1 MB/s | 5.4 MB 00:00 2024-09-12T02:36:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:(29/156): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e 2.9 MB/s | 146 kB 00:00 2024-09-12T02:36:03.169 INFO:teuthology.orchestra.run.smithi033.stdout:(3/156): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86 1.8 MB/s | 897 kB 00:00 2024-09-12T02:36:03.194 INFO:teuthology.orchestra.run.smithi033.stdout:(4/156): ceph-immutable-object-cache-19.1.1-298 2.2 MB/s | 145 kB 00:00 2024-09-12T02:36:03.271 INFO:teuthology.orchestra.run.smithi184.stdout:(30/156): rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_ 372 kB/s | 171 kB 00:00 2024-09-12T02:36:03.361 INFO:teuthology.orchestra.run.smithi033.stdout:(5/156): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_ 9.1 MB/s | 1.5 MB 00:00 2024-09-12T02:36:03.437 INFO:teuthology.orchestra.run.smithi184.stdout:(31/156): ceph-mgr-dashboard-19.1.1-298.gb53b6d 12 MB/s | 3.6 MB 00:00 2024-09-12T02:36:03.563 INFO:teuthology.orchestra.run.smithi184.stdout:(32/156): ceph-mgr-modules-core-19.1.1-298.gb53 1.9 MB/s | 249 kB 00:00 2024-09-12T02:36:03.655 INFO:teuthology.orchestra.run.smithi184.stdout:(33/156): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el 539 kB/s | 49 kB 00:00 2024-09-12T02:36:03.722 INFO:teuthology.orchestra.run.smithi184.stdout:(34/156): ceph-prometheus-alerts-19.1.1-298.gb5 247 kB/s | 16 kB 00:00 2024-09-12T02:36:03.728 INFO:teuthology.orchestra.run.smithi033.stdout:(6/156): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-09-12T02:36:03.743 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:36:03.744 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-12T02:36:03.756 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-12T02:36:03.756 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-12T02:36:03.757 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-12T02:36:03.758 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-12T02:36:03.760 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-12T02:36:03.762 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-12T02:36:03.763 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-12T02:36:03.814 INFO:teuthology.orchestra.run.smithi184.stdout:(35/156): ceph-mgr-diskprediction-local-19.1.1- 14 MB/s | 7.4 MB 00:00 2024-09-12T02:36:03.829 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T02:36:03.845 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-12T02:36:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 228 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 27 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 18 k 2024-09-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-12T02:36:03.849 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-12T02:36:03.850 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T02:36:03.851 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T02:36:03.852 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T02:36:03.853 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-12T02:36:03.854 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T02:36:03.855 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T02:36:03.856 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T02:36:03.857 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T02:36:03.858 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T02:36:03.859 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-12T02:36:03.860 INFO:teuthology.orchestra.run.smithi016.stdout:Install 148 Packages 2024-09-12T02:36:03.860 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 8 Packages 2024-09-12T02:36:03.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:03.860 INFO:teuthology.orchestra.run.smithi184.stdout:(36/156): ceph-volume-19.1.1-298.gb53b6d3e.el9. 2.2 MB/s | 285 kB 00:00 2024-09-12T02:36:03.861 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 226 M 2024-09-12T02:36:03.861 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T02:36:03.878 INFO:teuthology.orchestra.run.smithi033.stdout:(7/156): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_ 3.4 MB/s | 2.4 MB 00:00 2024-09-12T02:36:03.898 INFO:teuthology.orchestra.run.smithi184.stdout:(37/156): cephadm-19.1.1-298.gb53b6d3e.el9.noar 8.9 MB/s | 765 kB 00:00 2024-09-12T02:36:03.923 INFO:teuthology.orchestra.run.smithi184.stdout:(38/156): cephfs-top-19.1.1-298.gb53b6d3e.el9.n 243 kB/s | 18 kB 00:00 2024-09-12T02:36:04.040 INFO:teuthology.orchestra.run.smithi184.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 562 kB/s | 80 kB 00:00 2024-09-12T02:36:04.132 INFO:teuthology.orchestra.run.smithi184.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-12T02:36:04.191 INFO:teuthology.orchestra.run.smithi184.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-12T02:36:04.224 INFO:teuthology.orchestra.run.smithi184.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-12T02:36:04.292 INFO:teuthology.orchestra.run.smithi016.stdout:(1/156): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.r 42 kB/s | 6.5 kB 00:00 2024-09-12T02:36:04.308 INFO:teuthology.orchestra.run.smithi184.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-12T02:36:04.353 INFO:teuthology.orchestra.run.smithi033.stdout:(8/156): ceph-common-19.1.1-298.gb53b6d3e.el9.x 11 MB/s | 21 MB 00:01 2024-09-12T02:36:04.375 INFO:teuthology.orchestra.run.smithi184.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-09-12T02:36:04.403 INFO:teuthology.orchestra.run.smithi033.stdout:(9/156): ceph-selinux-19.1.1-298.gb53b6d3e.el9. 505 kB/s | 25 kB 00:00 2024-09-12T02:36:04.409 INFO:teuthology.orchestra.run.smithi184.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-09-12T02:36:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-12T02:36:04.501 INFO:teuthology.orchestra.run.smithi184.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-12T02:36:04.618 INFO:teuthology.orchestra.run.smithi184.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 4.2 MB/s | 689 kB 00:00 2024-09-12T02:36:04.644 INFO:teuthology.orchestra.run.smithi184.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-09-12T02:36:04.668 INFO:teuthology.orchestra.run.smithi184.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-09-12T02:36:04.694 INFO:teuthology.orchestra.run.smithi184.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 435 kB/s | 22 kB 00:00 2024-09-12T02:36:04.719 INFO:teuthology.orchestra.run.smithi184.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-09-12T02:36:04.734 INFO:teuthology.orchestra.run.smithi016.stdout:(2/156): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86 2.0 MB/s | 897 kB 00:00 2024-09-12T02:36:04.736 INFO:teuthology.orchestra.run.smithi184.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 792 kB/s | 33 kB 00:00 2024-09-12T02:36:04.745 INFO:teuthology.orchestra.run.smithi033.stdout:(10/156): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86 17 MB/s | 17 MB 00:01 2024-09-12T02:36:04.761 INFO:teuthology.orchestra.run.smithi184.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 754 kB/s | 18 kB 00:00 2024-09-12T02:36:04.786 INFO:teuthology.orchestra.run.smithi184.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 489 kB/s | 11 kB 00:00 2024-09-12T02:36:04.809 INFO:teuthology.orchestra.run.smithi016.stdout:(3/156): ceph-base-19.1.1-298.gb53b6d3e.el9.x86 8.1 MB/s | 5.4 MB 00:00 2024-09-12T02:36:04.812 INFO:teuthology.orchestra.run.smithi033.stdout:(11/156): cephfs-mirror-19.1.1-298.gb53b6d3e.el 3.4 MB/s | 228 kB 00:00 2024-09-12T02:36:04.835 INFO:teuthology.orchestra.run.smithi016.stdout:(4/156): ceph-immutable-object-cache-19.1.1-298 1.4 MB/s | 145 kB 00:00 2024-09-12T02:36:04.837 INFO:teuthology.orchestra.run.smithi184.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 25 MB/s | 3.0 MB 00:00 2024-09-12T02:36:04.862 INFO:teuthology.orchestra.run.smithi184.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-09-12T02:36:04.880 INFO:teuthology.orchestra.run.smithi033.stdout:(12/156): cephfs-shell-19.1.1-298.gb53b6d3e.el9 405 kB/s | 27 kB 00:00 2024-09-12T02:36:04.888 INFO:teuthology.orchestra.run.smithi184.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-09-12T02:36:04.913 INFO:teuthology.orchestra.run.smithi184.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-09-12T02:36:04.939 INFO:teuthology.orchestra.run.smithi184.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 12 MB/s | 581 kB 00:00 2024-09-12T02:36:04.963 INFO:teuthology.orchestra.run.smithi033.stdout:(13/156): libcephfs-devel-19.1.1-298.gb53b6d3e. 381 kB/s | 32 kB 00:00 2024-09-12T02:36:04.964 INFO:teuthology.orchestra.run.smithi184.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 498 kB/s | 24 kB 00:00 2024-09-12T02:36:04.989 INFO:teuthology.orchestra.run.smithi184.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-09-12T02:36:05.014 INFO:teuthology.orchestra.run.smithi184.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-12T02:36:05.055 INFO:teuthology.orchestra.run.smithi033.stdout:(14/156): libcephfs2-19.1.1-298.gb53b6d3e.el9.x 8.0 MB/s | 752 kB 00:00 2024-09-12T02:36:05.056 INFO:teuthology.orchestra.run.smithi184.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-09-12T02:36:05.081 INFO:teuthology.orchestra.run.smithi184.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 579 kB/s | 38 kB 00:00 2024-09-12T02:36:05.105 INFO:teuthology.orchestra.run.smithi033.stdout:(15/156): libcephsqlite-19.1.1-298.gb53b6d3e.el 3.2 MB/s | 163 kB 00:00 2024-09-12T02:36:05.110 INFO:teuthology.orchestra.run.smithi016.stdout:(5/156): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_ 7.9 MB/s | 2.4 MB 00:00 2024-09-12T02:36:05.155 INFO:teuthology.orchestra.run.smithi033.stdout:(16/156): librados-devel-19.1.1-298.gb53b6d3e.e 2.5 MB/s | 127 kB 00:00 2024-09-12T02:36:05.247 INFO:teuthology.orchestra.run.smithi033.stdout:(17/156): libradosstriper1-19.1.1-298.gb53b6d3e 5.4 MB/s | 506 kB 00:00 2024-09-12T02:36:05.302 INFO:teuthology.orchestra.run.smithi016.stdout:(6/156): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-09-12T02:36:05.427 INFO:teuthology.orchestra.run.smithi016.stdout:(7/156): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_ 15 MB/s | 4.7 MB 00:00 2024-09-12T02:36:05.590 INFO:teuthology.orchestra.run.smithi184.stdout:(66/156): ceph-test-19.1.1-298.gb53b6d3e.el9.x8 13 MB/s | 49 MB 00:03 2024-09-12T02:36:05.665 INFO:teuthology.orchestra.run.smithi184.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 8.1 MB/s | 4.9 MB 00:00 2024-09-12T02:36:05.722 INFO:teuthology.orchestra.run.smithi033.stdout:(18/156): librgw2-19.1.1-298.gb53b6d3e.el9.x86_ 11 MB/s | 5.3 MB 00:00 2024-09-12T02:36:05.732 INFO:teuthology.orchestra.run.smithi184.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-09-12T02:36:05.774 INFO:teuthology.orchestra.run.smithi184.stdout:(69/156): postgresql-13.16-1.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-09-12T02:36:05.798 INFO:teuthology.orchestra.run.smithi033.stdout:(19/156): python3-ceph-argparse-19.1.1-298.gb53 603 kB/s | 45 kB 00:00 2024-09-12T02:36:05.808 INFO:teuthology.orchestra.run.smithi184.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 6.4 MB/s | 910 kB 00:00 2024-09-12T02:36:05.833 INFO:teuthology.orchestra.run.smithi184.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 1.4 MB/s | 137 kB 00:00 2024-09-12T02:36:05.873 INFO:teuthology.orchestra.run.smithi033.stdout:(20/156): python3-ceph-common-19.1.1-298.gb53b6 1.8 MB/s | 136 kB 00:00 2024-09-12T02:36:05.875 INFO:teuthology.orchestra.run.smithi184.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-09-12T02:36:05.900 INFO:teuthology.orchestra.run.smithi184.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-12T02:36:05.956 INFO:teuthology.orchestra.run.smithi033.stdout:(21/156): python3-cephfs-19.1.1-298.gb53b6d3e.e 1.9 MB/s | 162 kB 00:00 2024-09-12T02:36:05.984 INFO:teuthology.orchestra.run.smithi184.stdout:(74/156): postgresql-server-13.16-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-09-12T02:36:06.009 INFO:teuthology.orchestra.run.smithi184.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 576 kB/s | 61 kB 00:00 2024-09-12T02:36:06.023 INFO:teuthology.orchestra.run.smithi033.stdout:(22/156): python3-rados-19.1.1-298.gb53b6d3e.el 4.7 MB/s | 321 kB 00:00 2024-09-12T02:36:06.043 INFO:teuthology.orchestra.run.smithi184.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 4.3 MB/s | 245 kB 00:00 2024-09-12T02:36:06.082 INFO:teuthology.orchestra.run.smithi033.stdout:(23/156): python3-rbd-19.1.1-298.gb53b6d3e.el9. 5.1 MB/s | 301 kB 00:00 2024-09-12T02:36:06.086 INFO:teuthology.orchestra.run.smithi016.stdout:(8/156): ceph-radosgw-19.1.1-298.gb53b6d3e.el9. 16 MB/s | 11 MB 00:00 2024-09-12T02:36:06.118 INFO:teuthology.orchestra.run.smithi184.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-09-12T02:36:06.132 INFO:teuthology.orchestra.run.smithi033.stdout:(24/156): python3-rgw-19.1.1-298.gb53b6d3e.el9. 1.9 MB/s | 100 kB 00:00 2024-09-12T02:36:06.143 INFO:teuthology.orchestra.run.smithi184.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-09-12T02:36:06.168 INFO:teuthology.orchestra.run.smithi184.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 208 kB/s | 26 kB 00:00 2024-09-12T02:36:06.174 INFO:teuthology.orchestra.run.smithi033.stdout:(25/156): rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86 2.0 MB/s | 85 kB 00:00 2024-09-12T02:36:06.194 INFO:teuthology.orchestra.run.smithi184.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-12T02:36:06.219 INFO:teuthology.orchestra.run.smithi184.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-12T02:36:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-09-12T02:36:06.286 INFO:teuthology.orchestra.run.smithi184.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-09-12T02:36:06.311 INFO:teuthology.orchestra.run.smithi016.stdout:(9/156): ceph-common-19.1.1-298.gb53b6d3e.el9.x 9.8 MB/s | 21 MB 00:02 2024-09-12T02:36:06.311 INFO:teuthology.orchestra.run.smithi184.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-09-12T02:36:06.336 INFO:teuthology.orchestra.run.smithi016.stdout:(10/156): ceph-selinux-19.1.1-298.gb53b6d3e.el9 101 kB/s | 25 kB 00:00 2024-09-12T02:36:06.383 INFO:teuthology.orchestra.run.smithi033.stdout:(26/156): ceph-radosgw-19.1.1-298.gb53b6d3e.el9 4.2 MB/s | 11 MB 00:02 2024-09-12T02:36:06.386 INFO:teuthology.orchestra.run.smithi184.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 5.1 MB 00:00 2024-09-12T02:36:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:(11/156): cephfs-mirror-19.1.1-298.gb53b6d3e.el 3.8 MB/s | 228 kB 00:00 2024-09-12T02:36:06.412 INFO:teuthology.orchestra.run.smithi184.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 629 kB/s | 77 kB 00:00 2024-09-12T02:36:06.437 INFO:teuthology.orchestra.run.smithi184.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 343 kB/s | 42 kB 00:00 2024-09-12T02:36:06.450 INFO:teuthology.orchestra.run.smithi033.stdout:(27/156): rbd-mirror-19.1.1-298.gb53b6d3e.el9.x 11 MB/s | 3.1 MB 00:00 2024-09-12T02:36:06.462 INFO:teuthology.orchestra.run.smithi184.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-12T02:36:06.475 INFO:teuthology.orchestra.run.smithi033.stdout:(28/156): rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_ 1.8 MB/s | 171 kB 00:00 2024-09-12T02:36:06.488 INFO:teuthology.orchestra.run.smithi184.stdout:(89/156): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-09-12T02:36:06.495 INFO:teuthology.orchestra.run.smithi016.stdout:(12/156): cephfs-shell-19.1.1-298.gb53b6d3e.el9 270 kB/s | 27 kB 00:00 2024-09-12T02:36:06.509 INFO:teuthology.orchestra.run.smithi033.stdout:(29/156): ceph-grafana-dashboards-19.1.1-298.gb 542 kB/s | 32 kB 00:00 2024-09-12T02:36:06.513 INFO:teuthology.orchestra.run.smithi184.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-09-12T02:36:06.526 INFO:teuthology.orchestra.run.smithi033.stdout:(30/156): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e 2.8 MB/s | 146 kB 00:00 2024-09-12T02:36:06.529 INFO:teuthology.orchestra.run.smithi184.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 773 kB/s | 51 kB 00:00 2024-09-12T02:36:06.546 INFO:teuthology.orchestra.run.smithi184.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-09-12T02:36:06.554 INFO:teuthology.orchestra.run.smithi016.stdout:(13/156): libcephfs-devel-19.1.1-298.gb53b6d3e. 544 kB/s | 32 kB 00:00 2024-09-12T02:36:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-12T02:36:06.589 INFO:teuthology.orchestra.run.smithi184.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-12T02:36:06.622 INFO:teuthology.orchestra.run.smithi184.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-12T02:36:06.639 INFO:teuthology.orchestra.run.smithi184.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 735 kB/s | 36 kB 00:00 2024-09-12T02:36:06.665 INFO:teuthology.orchestra.run.smithi184.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2024-09-12T02:36:06.665 INFO:teuthology.orchestra.run.smithi016.stdout:(14/156): libcephfs2-19.1.1-298.gb53b6d3e.el9.x 6.8 MB/s | 752 kB 00:00 2024-09-12T02:36:06.706 INFO:teuthology.orchestra.run.smithi184.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 4.6 MB/s | 304 kB 00:00 2024-09-12T02:36:06.723 INFO:teuthology.orchestra.run.smithi184.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-09-12T02:36:06.746 INFO:teuthology.orchestra.run.smithi016.stdout:(15/156): libcephsqlite-19.1.1-298.gb53b6d3e.el 1.9 MB/s | 163 kB 00:00 2024-09-12T02:36:06.757 INFO:teuthology.orchestra.run.smithi184.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-12T02:36:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-12T02:36:06.808 INFO:teuthology.orchestra.run.smithi184.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 941 kB/s | 45 kB 00:00 2024-09-12T02:36:06.869 INFO:teuthology.orchestra.run.smithi033.stdout:(31/156): ceph-mgr-dashboard-19.1.1-298.gb53b6d 9.9 MB/s | 3.6 MB 00:00 2024-09-12T02:36:06.918 INFO:teuthology.orchestra.run.smithi033.stdout:(32/156): ceph-mgr-modules-core-19.1.1-298.gb53 4.9 MB/s | 249 kB 00:00 2024-09-12T02:36:06.958 INFO:teuthology.orchestra.run.smithi184.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-12T02:36:06.969 INFO:teuthology.orchestra.run.smithi033.stdout:(33/156): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el 987 kB/s | 49 kB 00:00 2024-09-12T02:36:06.980 INFO:teuthology.orchestra.run.smithi016.stdout:(16/156): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86 10 MB/s | 17 MB 00:01 2024-09-12T02:36:07.005 INFO:teuthology.orchestra.run.smithi016.stdout:(17/156): librados-devel-19.1.1-298.gb53b6d3e.e 489 kB/s | 127 kB 00:00 2024-09-12T02:36:07.017 INFO:teuthology.orchestra.run.smithi184.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-12T02:36:07.019 INFO:teuthology.orchestra.run.smithi033.stdout:(34/156): ceph-prometheus-alerts-19.1.1-298.gb5 330 kB/s | 16 kB 00:00 2024-09-12T02:36:07.039 INFO:teuthology.orchestra.run.smithi016.stdout:(18/156): libradosstriper1-19.1.1-298.gb53b6d3e 8.5 MB/s | 506 kB 00:00 2024-09-12T02:36:07.042 INFO:teuthology.orchestra.run.smithi184.stdout:(105/156): lua-devel-5.4.4-4.el9.x86_64.rpm 87 kB/s | 22 kB 00:00 2024-09-12T02:36:07.084 INFO:teuthology.orchestra.run.smithi184.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-09-12T02:36:07.102 INFO:teuthology.orchestra.run.smithi033.stdout:(35/156): ceph-volume-19.1.1-298.gb53b6d3e.el9. 3.3 MB/s | 285 kB 00:00 2024-09-12T02:36:07.109 INFO:teuthology.orchestra.run.smithi184.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-09-12T02:36:07.114 INFO:teuthology.orchestra.run.smithi016.stdout:(19/156): python3-ceph-argparse-19.1.1-298.gb53 603 kB/s | 45 kB 00:00 2024-09-12T02:36:07.134 INFO:teuthology.orchestra.run.smithi184.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-12T02:36:07.159 INFO:teuthology.orchestra.run.smithi184.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-12T02:36:07.193 INFO:teuthology.orchestra.run.smithi184.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-12T02:36:07.206 INFO:teuthology.orchestra.run.smithi016.stdout:(20/156): python3-ceph-common-19.1.1-298.gb53b6 1.4 MB/s | 136 kB 00:00 2024-09-12T02:36:07.237 INFO:teuthology.orchestra.run.smithi184.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-12T02:36:07.237 INFO:teuthology.orchestra.run.smithi033.stdout:(36/156): cephadm-19.1.1-298.gb53b6d3e.el9.noar 5.6 MB/s | 765 kB 00:00 2024-09-12T02:36:07.277 INFO:teuthology.orchestra.run.smithi184.stdout:(112/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-09-12T02:36:07.281 INFO:teuthology.orchestra.run.smithi016.stdout:(21/156): python3-cephfs-19.1.1-298.gb53b6d3e.e 2.1 MB/s | 162 kB 00:00 2024-09-12T02:36:07.327 INFO:teuthology.orchestra.run.smithi184.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-12T02:36:07.328 INFO:teuthology.orchestra.run.smithi033.stdout:(37/156): cephfs-top-19.1.1-298.gb53b6d3e.el9.n 199 kB/s | 18 kB 00:00 2024-09-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi184.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-12T02:36:07.365 INFO:teuthology.orchestra.run.smithi016.stdout:(22/156): librgw2-19.1.1-298.gb53b6d3e.el9.x86_ 15 MB/s | 5.3 MB 00:00 2024-09-12T02:36:07.390 INFO:teuthology.orchestra.run.smithi016.stdout:(23/156): python3-rados-19.1.1-298.gb53b6d3e.el 2.9 MB/s | 321 kB 00:00 2024-09-12T02:36:07.416 INFO:teuthology.orchestra.run.smithi016.stdout:(24/156): python3-rbd-19.1.1-298.gb53b6d3e.el9. 5.9 MB/s | 301 kB 00:00 2024-09-12T02:36:07.419 INFO:teuthology.orchestra.run.smithi184.stdout:(115/156): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-12T02:36:07.436 INFO:teuthology.orchestra.run.smithi033.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 738 kB/s | 80 kB 00:00 2024-09-12T02:36:07.445 INFO:teuthology.orchestra.run.smithi184.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 349 kB/s | 32 kB 00:00 2024-09-12T02:36:07.449 INFO:teuthology.orchestra.run.smithi016.stdout:(25/156): python3-rgw-19.1.1-298.gb53b6d3e.el9. 1.7 MB/s | 100 kB 00:00 2024-09-12T02:36:07.470 INFO:teuthology.orchestra.run.smithi184.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-12T02:36:07.474 INFO:teuthology.orchestra.run.smithi016.stdout:(26/156): rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86 1.4 MB/s | 85 kB 00:00 2024-09-12T02:36:07.504 INFO:teuthology.orchestra.run.smithi184.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-12T02:36:07.529 INFO:teuthology.orchestra.run.smithi184.stdout:(119/156): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-09-12T02:36:07.545 INFO:teuthology.orchestra.run.smithi033.stdout:(39/156): libgfortran-11.5.0-2.el9.x86_64.rpm 7.2 MB/s | 801 kB 00:00 2024-09-12T02:36:07.550 INFO:teuthology.orchestra.run.smithi016.stdout:(27/156): rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_ 2.2 MB/s | 171 kB 00:00 2024-09-12T02:36:07.562 INFO:teuthology.orchestra.run.smithi184.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-12T02:36:07.587 INFO:teuthology.orchestra.run.smithi033.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-12T02:36:07.588 INFO:teuthology.orchestra.run.smithi184.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-09-12T02:36:07.621 INFO:teuthology.orchestra.run.smithi184.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 2.3 MB/s | 220 kB 00:00 2024-09-12T02:36:07.629 INFO:teuthology.orchestra.run.smithi033.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-09-12T02:36:07.642 INFO:teuthology.orchestra.run.smithi016.stdout:(28/156): rbd-mirror-19.1.1-298.gb53b6d3e.el9.x 16 MB/s | 3.1 MB 00:00 2024-09-12T02:36:07.655 INFO:teuthology.orchestra.run.smithi184.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-09-12T02:36:07.667 INFO:teuthology.orchestra.run.smithi016.stdout:(29/156): ceph-grafana-dashboards-19.1.1-298.gb 272 kB/s | 32 kB 00:00 2024-09-12T02:36:07.680 INFO:teuthology.orchestra.run.smithi184.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-09-12T02:36:07.693 INFO:teuthology.orchestra.run.smithi016.stdout:(30/156): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e 2.9 MB/s | 146 kB 00:00 2024-09-12T02:36:07.696 INFO:teuthology.orchestra.run.smithi033.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-09-12T02:36:07.706 INFO:teuthology.orchestra.run.smithi184.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-12T02:36:07.731 INFO:teuthology.orchestra.run.smithi184.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-09-12T02:36:07.738 INFO:teuthology.orchestra.run.smithi033.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-12T02:36:07.765 INFO:teuthology.orchestra.run.smithi184.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-09-12T02:36:07.780 INFO:teuthology.orchestra.run.smithi033.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-12T02:36:07.789 INFO:teuthology.orchestra.run.smithi184.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-09-12T02:36:07.814 INFO:teuthology.orchestra.run.smithi033.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 3.7 MB/s | 125 kB 00:00 2024-09-12T02:36:07.815 INFO:teuthology.orchestra.run.smithi184.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-12T02:36:07.849 INFO:teuthology.orchestra.run.smithi184.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-12T02:36:07.856 INFO:teuthology.orchestra.run.smithi033.stdout:(46/156): python3-urllib3-1.26.5-6.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-09-12T02:36:07.874 INFO:teuthology.orchestra.run.smithi184.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-12T02:36:07.916 INFO:teuthology.orchestra.run.smithi184.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-09-12T02:36:07.941 INFO:teuthology.orchestra.run.smithi184.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-09-12T02:36:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:(31/156): ceph-mgr-dashboard-19.1.1-298.gb53b6d 12 MB/s | 3.6 MB 00:00 2024-09-12T02:36:07.975 INFO:teuthology.orchestra.run.smithi184.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-09-12T02:36:08.000 INFO:teuthology.orchestra.run.smithi184.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-12T02:36:08.025 INFO:teuthology.orchestra.run.smithi184.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-12T02:36:08.043 INFO:teuthology.orchestra.run.smithi016.stdout:(32/156): ceph-mgr-modules-core-19.1.1-298.gb53 2.9 MB/s | 249 kB 00:00 2024-09-12T02:36:08.051 INFO:teuthology.orchestra.run.smithi184.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-09-12T02:36:08.076 INFO:teuthology.orchestra.run.smithi184.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-12T02:36:08.093 INFO:teuthology.orchestra.run.smithi184.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 2.1 MB/s | 36 kB 00:00 2024-09-12T02:36:08.118 INFO:teuthology.orchestra.run.smithi184.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-09-12T02:36:08.124 INFO:teuthology.orchestra.run.smithi033.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-09-12T02:36:08.144 INFO:teuthology.orchestra.run.smithi016.stdout:(33/156): ceph-mgr-diskprediction-local-19.1.1- 16 MB/s | 7.4 MB 00:00 2024-09-12T02:36:08.144 INFO:teuthology.orchestra.run.smithi184.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-12T02:36:08.169 INFO:teuthology.orchestra.run.smithi016.stdout:(34/156): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el 394 kB/s | 49 kB 00:00 2024-09-12T02:36:08.169 INFO:teuthology.orchestra.run.smithi184.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-12T02:36:08.194 INFO:teuthology.orchestra.run.smithi016.stdout:(35/156): ceph-prometheus-alerts-19.1.1-298.gb5 327 kB/s | 16 kB 00:00 2024-09-12T02:36:08.195 INFO:teuthology.orchestra.run.smithi184.stdout:(143/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-12T02:36:08.228 INFO:teuthology.orchestra.run.smithi016.stdout:(36/156): ceph-volume-19.1.1-298.gb53b6d3e.el9. 4.7 MB/s | 285 kB 00:00 2024-09-12T02:36:08.231 INFO:teuthology.orchestra.run.smithi033.stdout:(48/156): ceph-mgr-diskprediction-local-19.1.1- 4.3 MB/s | 7.4 MB 00:01 2024-09-12T02:36:08.244 INFO:teuthology.orchestra.run.smithi184.stdout:(144/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-09-12T02:36:08.270 INFO:teuthology.orchestra.run.smithi184.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-09-12T02:36:08.270 INFO:teuthology.orchestra.run.smithi016.stdout:(37/156): cephadm-19.1.1-298.gb53b6d3e.el9.noar 9.9 MB/s | 765 kB 00:00 2024-09-12T02:36:08.296 INFO:teuthology.orchestra.run.smithi184.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-12T02:36:08.297 INFO:teuthology.orchestra.run.smithi016.stdout:(38/156): cephfs-top-19.1.1-298.gb53b6d3e.el9.n 272 kB/s | 18 kB 00:00 2024-09-12T02:36:08.320 INFO:teuthology.orchestra.run.smithi184.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-09-12T02:36:08.371 INFO:teuthology.orchestra.run.smithi184.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-12T02:36:08.396 INFO:teuthology.orchestra.run.smithi016.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 637 kB/s | 80 kB 00:00 2024-09-12T02:36:08.454 INFO:teuthology.orchestra.run.smithi016.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-12T02:36:08.479 INFO:teuthology.orchestra.run.smithi016.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-12T02:36:08.505 INFO:teuthology.orchestra.run.smithi016.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-09-12T02:36:08.547 INFO:teuthology.orchestra.run.smithi016.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-12T02:36:08.554 INFO:teuthology.orchestra.run.smithi184.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-09-12T02:36:08.613 INFO:teuthology.orchestra.run.smithi184.stdout:(150/156): librados2-19.1.1-298.gb53b6d3e.el9.x 11 MB/s | 3.4 MB 00:00 2024-09-12T02:36:08.614 INFO:teuthology.orchestra.run.smithi016.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-09-12T02:36:08.663 INFO:teuthology.orchestra.run.smithi184.stdout:(151/156): NetworkManager-libnm-1.49.90-1.el9.x 17 MB/s | 1.8 MB 00:00 2024-09-12T02:36:08.665 INFO:teuthology.orchestra.run.smithi033.stdout:(49/156): ceph-test-19.1.1-298.gb53b6d3e.el9.x8 11 MB/s | 49 MB 00:04 2024-09-12T02:36:08.672 INFO:teuthology.orchestra.run.smithi016.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-12T02:36:08.698 INFO:teuthology.orchestra.run.smithi033.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 1.6 MB/s | 943 kB 00:00 2024-09-12T02:36:08.714 INFO:teuthology.orchestra.run.smithi184.stdout:(152/156): librbd1-19.1.1-298.gb53b6d3e.el9.x86 8.0 MB/s | 3.2 MB 00:00 2024-09-12T02:36:08.739 INFO:teuthology.orchestra.run.smithi184.stdout:(153/156): NetworkManager-team-1.49.90-1.el9.x8 325 kB/s | 41 kB 00:00 2024-09-12T02:36:08.764 INFO:teuthology.orchestra.run.smithi184.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 2.4 MB/s | 249 kB 00:00 2024-09-12T02:36:08.764 INFO:teuthology.orchestra.run.smithi016.stdout:(46/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-12T02:36:08.782 INFO:teuthology.orchestra.run.smithi033.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 186 kB/s | 22 kB 00:00 2024-09-12T02:36:08.807 INFO:teuthology.orchestra.run.smithi033.stdout:(52/156): boost-program-options-1.75.0-8.el9.x8 186 kB/s | 107 kB 00:00 2024-09-12T02:36:08.815 INFO:teuthology.orchestra.run.smithi016.stdout:(47/156): python3-cryptography-36.0.1-4.el9.x86 3.7 MB/s | 1.2 MB 00:00 2024-09-12T02:36:08.841 INFO:teuthology.orchestra.run.smithi033.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-09-12T02:36:08.866 INFO:teuthology.orchestra.run.smithi033.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-09-12T02:36:08.892 INFO:teuthology.orchestra.run.smithi033.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-09-12T02:36:08.906 INFO:teuthology.orchestra.run.smithi184.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 7.1 MB/s | 1.4 MB 00:00 2024-09-12T02:36:08.908 INFO:teuthology.orchestra.run.smithi033.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 265 kB/s | 11 kB 00:00 2024-09-12T02:36:08.942 INFO:teuthology.orchestra.run.smithi033.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-12T02:36:08.959 INFO:teuthology.orchestra.run.smithi033.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 898 kB/s | 45 kB 00:00 2024-09-12T02:36:08.972 INFO:teuthology.orchestra.run.smithi184.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-09-12T02:36:08.979 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-12T02:36:08.979 INFO:teuthology.orchestra.run.smithi184.stdout:Total 25 MB/s | 226 MB 00:08 2024-09-12T02:36:09.051 INFO:teuthology.orchestra.run.smithi033.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-12T02:36:09.110 INFO:teuthology.orchestra.run.smithi033.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 3.8 MB/s | 581 kB 00:00 2024-09-12T02:36:09.126 INFO:teuthology.orchestra.run.smithi033.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 326 kB/s | 24 kB 00:00 2024-09-12T02:36:09.152 INFO:teuthology.orchestra.run.smithi033.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-12T02:36:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-12T02:36:09.236 INFO:teuthology.orchestra.run.smithi033.stdout:(64/156): openblas-0.3.26-2.el9.x86_64.rpm 772 kB/s | 38 kB 00:00 2024-09-12T02:36:09.249 INFO:teuthology.orchestra.run.smithi016.stdout:(48/156): ceph-test-19.1.1-298.gb53b6d3e.el9.x8 17 MB/s | 49 MB 00:02 2024-09-12T02:36:09.261 INFO:teuthology.orchestra.run.smithi033.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-12T02:36:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-09-12T02:36:09.374 INFO:teuthology.orchestra.run.smithi016.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 855 kB/s | 107 kB 00:00 2024-09-12T02:36:09.408 INFO:teuthology.orchestra.run.smithi016.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 371 kB/s | 22 kB 00:00 2024-09-12T02:36:09.449 INFO:teuthology.orchestra.run.smithi016.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-09-12T02:36:09.475 INFO:teuthology.orchestra.run.smithi016.stdout:(53/156): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 943 kB 00:00 2024-09-12T02:36:09.509 INFO:teuthology.orchestra.run.smithi016.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-09-12T02:36:09.534 INFO:teuthology.orchestra.run.smithi016.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 310 kB/s | 18 kB 00:00 2024-09-12T02:36:09.559 INFO:teuthology.orchestra.run.smithi016.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-12T02:36:09.585 INFO:teuthology.orchestra.run.smithi016.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-12T02:36:09.610 INFO:teuthology.orchestra.run.smithi016.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-12T02:36:09.644 INFO:teuthology.orchestra.run.smithi016.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-09-12T02:36:09.686 INFO:teuthology.orchestra.run.smithi016.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 7.6 MB/s | 581 kB 00:00 2024-09-12T02:36:09.702 INFO:teuthology.orchestra.run.smithi016.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 420 kB/s | 24 kB 00:00 2024-09-12T02:36:09.719 INFO:teuthology.orchestra.run.smithi016.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-12T02:36:09.753 INFO:teuthology.orchestra.run.smithi016.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-12T02:36:09.779 INFO:teuthology.orchestra.run.smithi016.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-12T02:36:09.803 INFO:teuthology.orchestra.run.smithi016.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 766 kB/s | 38 kB 00:00 2024-09-12T02:36:09.844 INFO:teuthology.orchestra.run.smithi033.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.9 MB/s | 3.0 MB 00:01 2024-09-12T02:36:09.871 INFO:teuthology.orchestra.run.smithi016.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.1 MB/s | 3.0 MB 00:00 2024-09-12T02:36:09.971 INFO:teuthology.orchestra.run.smithi016.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 26 MB/s | 4.9 MB 00:00 2024-09-12T02:36:10.362 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-12T02:36:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-09-12T02:36:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:(69/156): postgresql-13.16-1.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-09-12T02:36:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-12T02:36:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 5.1 MB/s | 910 kB 00:00 2024-09-12T02:36:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-12T02:36:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-12T02:36:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-12T02:36:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-09-12T02:36:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-12T02:36:10.370 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-12T02:36:10.370 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-12T02:36:10.382 INFO:teuthology.orchestra.run.smithi016.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-12T02:36:10.408 INFO:teuthology.orchestra.run.smithi016.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-09-12T02:36:10.433 INFO:teuthology.orchestra.run.smithi016.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-09-12T02:36:10.467 INFO:teuthology.orchestra.run.smithi016.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-12T02:36:10.492 INFO:teuthology.orchestra.run.smithi016.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 591 kB/s | 35 kB 00:00 2024-09-12T02:36:10.576 INFO:teuthology.orchestra.run.smithi016.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-12T02:36:10.601 INFO:teuthology.orchestra.run.smithi016.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-09-12T02:36:10.627 INFO:teuthology.orchestra.run.smithi016.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-12T02:36:10.645 INFO:teuthology.orchestra.run.smithi033.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-09-12T02:36:10.652 INFO:teuthology.orchestra.run.smithi016.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-12T02:36:10.677 INFO:teuthology.orchestra.run.smithi016.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-09-12T02:36:10.702 INFO:teuthology.orchestra.run.smithi016.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-09-12T02:36:10.736 INFO:teuthology.orchestra.run.smithi016.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-09-12T02:36:10.761 INFO:teuthology.orchestra.run.smithi016.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-09-12T02:36:10.786 INFO:teuthology.orchestra.run.smithi016.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-12T02:36:10.812 INFO:teuthology.orchestra.run.smithi016.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-12T02:36:10.887 INFO:teuthology.orchestra.run.smithi016.stdout:(92/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-12T02:36:10.904 INFO:teuthology.orchestra.run.smithi016.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-09-12T02:36:10.921 INFO:teuthology.orchestra.run.smithi016.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-09-12T02:36:10.946 INFO:teuthology.orchestra.run.smithi016.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-09-12T02:36:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 856 kB/s | 36 kB 00:00 2024-09-12T02:36:10.980 INFO:teuthology.orchestra.run.smithi016.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-12T02:36:11.014 INFO:teuthology.orchestra.run.smithi016.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-12T02:36:11.031 INFO:teuthology.orchestra.run.smithi016.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-12T02:36:11.056 INFO:teuthology.orchestra.run.smithi016.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 2.2 MB/s | 96 kB 00:00 2024-09-12T02:36:11.081 INFO:teuthology.orchestra.run.smithi016.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-12T02:36:11.107 INFO:teuthology.orchestra.run.smithi016.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 889 kB/s | 45 kB 00:00 2024-09-12T02:36:11.120 INFO:teuthology.orchestra.run.smithi033.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 1.9 MB/s | 910 kB 00:00 2024-09-12T02:36:11.178 INFO:teuthology.orchestra.run.smithi033.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2024-09-12T02:36:11.229 INFO:teuthology.orchestra.run.smithi033.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.2 MB/s | 137 kB 00:00 2024-09-12T02:36:11.332 INFO:teuthology.orchestra.run.smithi016.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-09-12T02:36:11.391 INFO:teuthology.orchestra.run.smithi016.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-12T02:36:11.424 INFO:teuthology.orchestra.run.smithi016.stdout:(105/156): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-09-12T02:36:11.449 INFO:teuthology.orchestra.run.smithi016.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 782 kB/s | 19 kB 00:00 2024-09-12T02:36:11.504 INFO:teuthology.orchestra.run.smithi033.stdout:(71/156): openblas-openmp-0.3.26-2.el9.x86_64.r 2.2 MB/s | 4.9 MB 00:02 2024-09-12T02:36:11.588 INFO:teuthology.orchestra.run.smithi033.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-09-12T02:36:11.617 INFO:teuthology.orchestra.run.smithi016.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2024-09-12T02:36:11.642 INFO:teuthology.orchestra.run.smithi016.stdout:(108/156): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2024-09-12T02:36:11.667 INFO:teuthology.orchestra.run.smithi016.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 516 kB/s | 26 kB 00:00 2024-09-12T02:36:11.701 INFO:teuthology.orchestra.run.smithi016.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-12T02:36:11.735 INFO:teuthology.orchestra.run.smithi016.stdout:(111/156): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-09-12T02:36:11.754 INFO:teuthology.orchestra.run.smithi033.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.9 MB/s | 1.0 MB 00:00 2024-09-12T02:36:11.760 INFO:teuthology.orchestra.run.smithi016.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-09-12T02:36:11.802 INFO:teuthology.orchestra.run.smithi016.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-12T02:36:11.813 INFO:teuthology.orchestra.run.smithi033.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-09-12T02:36:11.819 INFO:teuthology.orchestra.run.smithi016.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-12T02:36:11.861 INFO:teuthology.orchestra.run.smithi016.stdout:(115/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-12T02:36:11.878 INFO:teuthology.orchestra.run.smithi016.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-09-12T02:36:11.894 INFO:teuthology.orchestra.run.smithi016.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 420 kB/s | 14 kB 00:00 2024-09-12T02:36:11.920 INFO:teuthology.orchestra.run.smithi016.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-12T02:36:11.945 INFO:teuthology.orchestra.run.smithi016.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-12T02:36:11.963 INFO:teuthology.orchestra.run.smithi033.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-12T02:36:11.971 INFO:teuthology.orchestra.run.smithi016.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-09-12T02:36:11.996 INFO:teuthology.orchestra.run.smithi016.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-12T02:36:12.021 INFO:teuthology.orchestra.run.smithi016.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-12T02:36:12.046 INFO:teuthology.orchestra.run.smithi016.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-09-12T02:36:12.080 INFO:teuthology.orchestra.run.smithi016.stdout:(124/156): protobuf-compiler-3.14.0-13.el9.x86_ 887 kB/s | 863 kB 00:00 2024-09-12T02:36:12.113 INFO:teuthology.orchestra.run.smithi033.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-09-12T02:36:12.114 INFO:teuthology.orchestra.run.smithi016.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-09-12T02:36:12.147 INFO:teuthology.orchestra.run.smithi016.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 619 kB/s | 19 kB 00:00 2024-09-12T02:36:12.163 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-12T02:36:12.163 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-12T02:36:12.163 INFO:teuthology.orchestra.run.smithi033.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-09-12T02:36:12.173 INFO:teuthology.orchestra.run.smithi016.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-09-12T02:36:12.198 INFO:teuthology.orchestra.run.smithi016.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-09-12T02:36:12.214 INFO:teuthology.orchestra.run.smithi033.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-12T02:36:12.223 INFO:teuthology.orchestra.run.smithi016.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-12T02:36:12.249 INFO:teuthology.orchestra.run.smithi016.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-12T02:36:12.274 INFO:teuthology.orchestra.run.smithi016.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-12T02:36:12.324 INFO:teuthology.orchestra.run.smithi016.stdout:(132/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.8 MB/s | 2.0 MB 00:00 2024-09-12T02:36:12.331 INFO:teuthology.orchestra.run.smithi033.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-12T02:36:12.349 INFO:teuthology.orchestra.run.smithi016.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-12T02:36:12.367 INFO:teuthology.orchestra.run.smithi016.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-09-12T02:36:12.381 INFO:teuthology.orchestra.run.smithi033.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-09-12T02:36:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-09-12T02:36:12.417 INFO:teuthology.orchestra.run.smithi016.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-09-12T02:36:12.434 INFO:teuthology.orchestra.run.smithi016.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-09-12T02:36:12.451 INFO:teuthology.orchestra.run.smithi016.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-12T02:36:12.480 INFO:teuthology.orchestra.run.smithi016.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 3.0 MB/s | 188 kB 00:00 2024-09-12T02:36:12.505 INFO:teuthology.orchestra.run.smithi016.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 671 kB/s | 36 kB 00:00 2024-09-12T02:36:12.539 INFO:teuthology.orchestra.run.smithi016.stdout:(141/156): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-09-12T02:36:12.564 INFO:teuthology.orchestra.run.smithi016.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 887 kB/s | 74 kB 00:00 2024-09-12T02:36:12.589 INFO:teuthology.orchestra.run.smithi016.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-12T02:36:12.615 INFO:teuthology.orchestra.run.smithi016.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 806 kB/s | 20 kB 00:00 2024-09-12T02:36:12.640 INFO:teuthology.orchestra.run.smithi016.stdout:(145/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-12T02:36:12.665 INFO:teuthology.orchestra.run.smithi016.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-12T02:36:12.699 INFO:teuthology.orchestra.run.smithi016.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-12T02:36:12.749 INFO:teuthology.orchestra.run.smithi016.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-12T02:36:12.966 INFO:teuthology.orchestra.run.smithi016.stdout:(149/156): librbd1-19.1.1-298.gb53b6d3e.el9.x86 12 MB/s | 3.2 MB 00:00 2024-09-12T02:36:13.025 INFO:teuthology.orchestra.run.smithi016.stdout:(150/156): librados2-19.1.1-298.gb53b6d3e.el9.x 9.5 MB/s | 3.4 MB 00:00 2024-09-12T02:36:13.109 INFO:teuthology.orchestra.run.smithi016.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 490 kB/s | 41 kB 00:00 2024-09-12T02:36:13.217 INFO:teuthology.orchestra.run.smithi016.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 2.2 MB/s | 249 kB 00:00 2024-09-12T02:36:13.417 INFO:teuthology.orchestra.run.smithi016.stdout:(153/156): NetworkManager-1.49.90-1.el9.x86_64. 3.5 MB/s | 2.3 MB 00:00 2024-09-12T02:36:13.451 INFO:teuthology.orchestra.run.smithi016.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 5.8 MB/s | 1.4 MB 00:00 2024-09-12T02:36:13.509 INFO:teuthology.orchestra.run.smithi016.stdout:(155/156): NetworkManager-libnm-1.49.90-1.el9.x 3.3 MB/s | 1.8 MB 00:00 2024-09-12T02:36:14.098 INFO:teuthology.orchestra.run.smithi033.stdout:(81/156): postgresql-server-13.16-1.el9.x86_64. 2.0 MB/s | 5.8 MB 00:02 2024-09-12T02:36:14.126 INFO:teuthology.orchestra.run.smithi016.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.4 MB/s | 2.4 MB 00:00 2024-09-12T02:36:14.133 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T02:36:14.133 INFO:teuthology.orchestra.run.smithi016.stdout:Total 22 MB/s | 226 MB 00:10 2024-09-12T02:36:14.239 INFO:teuthology.orchestra.run.smithi033.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-09-12T02:36:14.381 INFO:teuthology.orchestra.run.smithi033.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-09-12T02:36:14.465 INFO:teuthology.orchestra.run.smithi033.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 927 kB/s | 77 kB 00:00 2024-09-12T02:36:14.515 INFO:teuthology.orchestra.run.smithi033.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 847 kB/s | 42 kB 00:00 2024-09-12T02:36:14.649 INFO:teuthology.orchestra.run.smithi033.stdout:(86/156): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-09-12T02:36:14.758 INFO:teuthology.orchestra.run.smithi033.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 1.1 MB/s | 269 kB 00:00 2024-09-12T02:36:14.825 INFO:teuthology.orchestra.run.smithi033.stdout:(88/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.1 MB/s | 5.1 MB 00:02 2024-09-12T02:36:14.850 INFO:teuthology.orchestra.run.smithi033.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 791 kB/s | 159 kB 00:00 2024-09-12T02:36:14.875 INFO:teuthology.orchestra.run.smithi033.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-12T02:36:14.900 INFO:teuthology.orchestra.run.smithi033.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-12T02:36:14.951 INFO:teuthology.orchestra.run.smithi033.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-09-12T02:36:14.984 INFO:teuthology.orchestra.run.smithi033.stdout:(93/156): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 279 kB 00:00 2024-09-12T02:36:15.010 INFO:teuthology.orchestra.run.smithi033.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 755 kB/s | 44 kB 00:00 2024-09-12T02:36:15.060 INFO:teuthology.orchestra.run.smithi033.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 720 kB/s | 36 kB 00:00 2024-09-12T02:36:15.114 INFO:teuthology.orchestra.run.smithi033.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-12T02:36:15.148 INFO:teuthology.orchestra.run.smithi033.stdout:(97/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-12T02:36:15.195 INFO:teuthology.orchestra.run.smithi033.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-12T02:36:15.278 INFO:teuthology.orchestra.run.smithi033.stdout:(99/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 1.1 MB/s | 96 kB 00:00 2024-09-12T02:36:15.303 INFO:teuthology.orchestra.run.smithi033.stdout:(100/156): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-12T02:36:15.345 INFO:teuthology.orchestra.run.smithi033.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-12T02:36:15.362 INFO:teuthology.orchestra.run.smithi033.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 760 kB/s | 64 kB 00:00 2024-09-12T02:36:15.387 INFO:teuthology.orchestra.run.smithi033.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 532 kB/s | 22 kB 00:00 2024-09-12T02:36:15.405 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T02:36:15.437 INFO:teuthology.orchestra.run.smithi033.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-12T02:36:15.462 INFO:teuthology.orchestra.run.smithi033.stdout:(105/156): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-09-12T02:36:15.488 INFO:teuthology.orchestra.run.smithi033.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 772 kB/s | 19 kB 00:00 2024-09-12T02:36:15.574 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T02:36:15.574 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T02:36:15.596 INFO:teuthology.orchestra.run.smithi033.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-12T02:36:15.622 INFO:teuthology.orchestra.run.smithi033.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-12T02:36:15.647 INFO:teuthology.orchestra.run.smithi033.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-12T02:36:15.672 INFO:teuthology.orchestra.run.smithi033.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-12T02:36:15.697 INFO:teuthology.orchestra.run.smithi033.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-09-12T02:36:15.740 INFO:teuthology.orchestra.run.smithi033.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-12T02:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-12T02:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-12T02:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-12T02:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 568 kB/s | 14 kB 00:00 2024-09-12T02:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 9.9 MB/s | 172 kB 00:00 2024-09-12T02:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-09-12T02:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 12 MB/s | 303 kB 00:00 2024-09-12T02:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout:(120/156): python3-google-auth-2.34.0-1.el9.noa 8.6 MB/s | 220 kB 00:00 2024-09-12T02:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout:(121/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-09-12T02:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-12T02:36:16.060 INFO:teuthology.orchestra.run.smithi033.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-12T02:36:16.085 INFO:teuthology.orchestra.run.smithi033.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 714 kB/s | 18 kB 00:00 2024-09-12T02:36:16.110 INFO:teuthology.orchestra.run.smithi033.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-09-12T02:36:16.160 INFO:teuthology.orchestra.run.smithi033.stdout:(126/156): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-09-12T02:36:16.185 INFO:teuthology.orchestra.run.smithi033.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-09-12T02:36:16.211 INFO:teuthology.orchestra.run.smithi033.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-12T02:36:16.237 INFO:teuthology.orchestra.run.smithi033.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-12T02:36:16.262 INFO:teuthology.orchestra.run.smithi033.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-12T02:36:16.287 INFO:teuthology.orchestra.run.smithi033.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-12T02:36:16.321 INFO:teuthology.orchestra.run.smithi033.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-12T02:36:16.346 INFO:teuthology.orchestra.run.smithi033.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 662 kB/s | 16 kB 00:00 2024-09-12T02:36:16.371 INFO:teuthology.orchestra.run.smithi033.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-12T02:36:16.464 INFO:teuthology.orchestra.run.smithi033.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-09-12T02:36:16.497 INFO:teuthology.orchestra.run.smithi033.stdout:(136/156): python3-kubernetes-26.1.0-2.el9.noar 3.3 MB/s | 1.0 MB 00:00 2024-09-12T02:36:16.523 INFO:teuthology.orchestra.run.smithi033.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 529 kB/s | 31 kB 00:00 2024-09-12T02:36:16.556 INFO:teuthology.orchestra.run.smithi033.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-12T02:36:16.581 INFO:teuthology.orchestra.run.smithi033.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-09-12T02:36:16.606 INFO:teuthology.orchestra.run.smithi033.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-09-12T02:36:16.632 INFO:teuthology.orchestra.run.smithi033.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-12T02:36:16.657 INFO:teuthology.orchestra.run.smithi033.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-12T02:36:16.685 INFO:teuthology.orchestra.run.smithi033.stdout:(143/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-12T02:36:16.699 INFO:teuthology.orchestra.run.smithi033.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-09-12T02:36:16.733 INFO:teuthology.orchestra.run.smithi033.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-09-12T02:36:16.767 INFO:teuthology.orchestra.run.smithi033.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-12T02:36:16.817 INFO:teuthology.orchestra.run.smithi033.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-12T02:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout:(148/156): librbd1-19.1.1-298.gb53b6d3e.el9.x86 12 MB/s | 3.2 MB 00:00 2024-09-12T02:36:17.334 INFO:teuthology.orchestra.run.smithi033.stdout:(149/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-12T02:36:17.384 INFO:teuthology.orchestra.run.smithi033.stdout:(150/156): librados2-19.1.1-298.gb53b6d3e.el9.x 5.5 MB/s | 3.4 MB 00:00 2024-09-12T02:36:17.426 INFO:teuthology.orchestra.run.smithi033.stdout:(151/156): NetworkManager-1.49.90-1.el9.x86_64. 6.6 MB/s | 2.3 MB 00:00 2024-09-12T02:36:17.468 INFO:teuthology.orchestra.run.smithi033.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 5.9 MB/s | 249 kB 00:00 2024-09-12T02:36:17.502 INFO:teuthology.orchestra.run.smithi033.stdout:(153/156): NetworkManager-team-1.49.90-1.el9.x8 348 kB/s | 41 kB 00:00 2024-09-12T02:36:17.535 INFO:teuthology.orchestra.run.smithi033.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 20 MB/s | 1.4 MB 00:00 2024-09-12T02:36:17.602 INFO:teuthology.orchestra.run.smithi033.stdout:(155/156): NetworkManager-libnm-1.49.90-1.el9.x 6.7 MB/s | 1.8 MB 00:00 2024-09-12T02:36:17.669 INFO:teuthology.orchestra.run.smithi033.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-09-12T02:36:17.674 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-09-12T02:36:17.674 INFO:teuthology.orchestra.run.smithi033.stdout:Total 14 MB/s | 226 MB 00:15 2024-09-12T02:36:17.900 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T02:36:17.900 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T02:36:18.391 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-12T02:36:18.476 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-12T02:36:18.515 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-12T02:36:18.576 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-12T02:36:18.610 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-12T02:36:18.635 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-12T02:36:18.678 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-12T02:36:18.890 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-09-12T02:36:19.000 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-12T02:36:19.056 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-09-12T02:36:19.056 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-09-12T02:36:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:19.488 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:19.580 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:19.609 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:36:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-12T02:36:19.822 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-12T02:36:19.850 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:19.954 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:19.976 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:20.022 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:20.065 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:20.263 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:20.542 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-12T02:36:20.636 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:20.735 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:20.950 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-09-12T02:36:20.950 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-09-12T02:36:20.966 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:21.082 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-12T02:36:21.179 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-12T02:36:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-12T02:36:21.242 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-12T02:36:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-12T02:36:21.371 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-12T02:36:21.402 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-12T02:36:21.455 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-12T02:36:21.487 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-12T02:36:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-12T02:36:21.584 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-12T02:36:21.637 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-12T02:36:21.668 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-12T02:36:21.703 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-12T02:36:21.781 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-12T02:36:21.822 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 34/164 2024-09-12T02:36:21.865 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 35/164 2024-09-12T02:36:21.894 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/164 2024-09-12T02:36:21.962 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-12T02:36:22.008 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-12T02:36:22.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-12T02:36:22.222 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-12T02:36:22.285 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-12T02:36:22.815 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-12T02:36:22.889 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-12T02:36:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-12T02:36:22.978 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-12T02:36:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 46/164 2024-09-12T02:36:23.067 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-12T02:36:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-12T02:36:23.294 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-12T02:36:23.349 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-12T02:36:23.423 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-12T02:36:23.454 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-12T02:36:23.498 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-12T02:36:23.553 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-12T02:36:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-12T02:36:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-12T02:36:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-12T02:36:23.810 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-12T02:36:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-12T02:36:23.901 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-12T02:36:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-12T02:36:24.054 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-12T02:36:24.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-12T02:36:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-09-12T02:36:24.141 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-12T02:36:24.214 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-12T02:36:24.344 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-12T02:36:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-12T02:36:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-12T02:36:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T02:36:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-12T02:36:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-12T02:36:24.524 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-12T02:36:24.564 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-12T02:36:24.564 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-12T02:36:24.597 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-12T02:36:24.631 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-12T02:36:24.672 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-12T02:36:24.707 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-12T02:36:24.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-12T02:36:24.793 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-12T02:36:24.896 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-12T02:36:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-12T02:36:24.935 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-12T02:36:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-12T02:36:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-12T02:36:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-12T02:36:25.045 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-12T02:36:25.103 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-12T02:36:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-12T02:36:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:25.634 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:25.668 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:36:25.820 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-12T02:36:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-12T02:36:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-12T02:36:26.004 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-12T02:36:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-12T02:36:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-12T02:36:26.070 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-12T02:36:26.097 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-12T02:36:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-12T02:36:26.220 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-12T02:36:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:26.341 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:26.361 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-12T02:36:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-12T02:36:26.535 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-12T02:36:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-12T02:36:27.003 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:27.075 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:27.179 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-12T02:36:27.289 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-12T02:36:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:27.435 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-12T02:36:27.508 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-09-12T02:36:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-12T02:36:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-12T02:36:27.615 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-12T02:36:27.647 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-12T02:36:27.686 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-12T02:36:27.690 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-12T02:36:27.742 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-12T02:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-12T02:36:27.782 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-12T02:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-12T02:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-12T02:36:27.825 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-12T02:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-12T02:36:27.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-12T02:36:27.885 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-12T02:36:27.957 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-12T02:36:28.017 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-12T02:36:28.047 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-12T02:36:28.091 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-12T02:36:28.169 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-12T02:36:28.198 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-12T02:36:28.210 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 34/164 2024-09-12T02:36:28.254 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 35/164 2024-09-12T02:36:28.282 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/164 2024-09-12T02:36:28.325 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:28.355 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-12T02:36:28.405 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-12T02:36:28.449 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:36:28.475 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:28.508 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-12T02:36:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-12T02:36:28.568 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:36:28.596 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:36:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-12T02:36:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-12T02:36:28.653 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-12T02:36:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-12T02:36:28.709 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-12T02:36:28.765 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-12T02:36:28.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-12T02:36:28.854 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:28.961 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:36:28.973 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-12T02:36:28.989 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:29.028 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:29.051 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 14/164 2024-09-12T02:36:29.074 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:29.095 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-12T02:36:29.119 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:29.157 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 102/164 2024-09-12T02:36:29.193 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-12T02:36:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-12T02:36:29.292 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-12T02:36:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-12T02:36:29.427 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-12T02:36:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-12T02:36:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 46/164 2024-09-12T02:36:29.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-12T02:36:29.554 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-12T02:36:29.646 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:29.670 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-12T02:36:29.695 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-12T02:36:29.705 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:29.739 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:29.750 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-12T02:36:29.777 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:29.806 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-12T02:36:29.807 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 106/164 2024-09-12T02:36:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-12T02:36:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-12T02:36:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-12T02:36:29.965 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-12T02:36:30.073 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-12T02:36:30.162 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-12T02:36:30.169 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-12T02:36:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-12T02:36:30.206 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-12T02:36:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-12T02:36:30.249 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-12T02:36:30.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-12T02:36:30.325 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-12T02:36:30.376 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-12T02:36:30.415 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-12T02:36:30.435 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-12T02:36:30.468 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-12T02:36:30.474 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-12T02:36:30.509 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-12T02:36:30.516 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-12T02:36:30.536 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-12T02:36:30.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-12T02:36:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-12T02:36:30.660 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-12T02:36:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-12T02:36:30.689 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-12T02:36:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-12T02:36:30.724 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-12T02:36:30.735 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-09-12T02:36:30.759 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-12T02:36:30.803 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-12T02:36:30.835 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 34/164 2024-09-12T02:36:30.848 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-12T02:36:30.879 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 35/164 2024-09-12T02:36:30.916 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/164 2024-09-12T02:36:30.983 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-12T02:36:30.990 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-12T02:36:31.018 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-12T02:36:31.040 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-12T02:36:31.055 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-12T02:36:31.093 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-12T02:36:31.117 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-12T02:36:31.136 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-12T02:36:31.176 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-12T02:36:31.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-12T02:36:31.257 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-12T02:36:31.264 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-12T02:36:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-12T02:36:31.318 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-12T02:36:31.340 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-12T02:36:31.374 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-12T02:36:31.464 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-12T02:36:31.523 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-12T02:36:31.561 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-12T02:36:31.669 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-12T02:36:31.730 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-12T02:36:31.879 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-12T02:36:31.965 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-12T02:36:31.999 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-12T02:36:32.042 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-12T02:36:32.080 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 46/164 2024-09-12T02:36:32.122 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-12T02:36:32.294 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-12T02:36:32.375 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-12T02:36:32.431 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-12T02:36:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-12T02:36:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-12T02:36:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-12T02:36:32.546 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-12T02:36:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-12T02:36:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-12T02:36:32.627 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-12T02:36:32.652 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:32.662 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-12T02:36:32.683 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-12T02:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:32.705 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-12T02:36:32.736 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-12T02:36:32.756 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-12T02:36:32.781 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-12T02:36:32.873 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-12T02:36:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-12T02:36:32.971 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-12T02:36:33.107 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-12T02:36:33.125 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-12T02:36:33.149 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-12T02:36:33.155 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-12T02:36:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:33.236 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-12T02:36:33.261 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-12T02:36:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:33.290 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-12T02:36:33.330 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-12T02:36:33.355 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-09-12T02:36:33.372 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:33.373 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T02:36:33.373 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T02:36:33.373 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:33.380 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-12T02:36:33.400 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:33.459 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-12T02:36:33.596 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-12T02:36:33.648 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-12T02:36:33.676 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-12T02:36:33.710 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-12T02:36:33.750 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-12T02:36:33.790 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-12T02:36:33.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-12T02:36:33.870 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-12T02:36:33.904 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-12T02:36:33.945 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-12T02:36:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-12T02:36:33.980 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-12T02:36:34.071 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-12T02:36:34.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-12T02:36:34.119 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-12T02:36:34.149 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-12T02:36:34.265 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-12T02:36:34.327 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-12T02:36:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-12T02:36:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-12T02:36:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-12T02:36:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-12T02:36:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-12T02:36:35.494 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-12T02:36:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-12T02:36:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-12T02:36:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-12T02:36:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-12T02:36:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-12T02:36:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-12T02:36:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-12T02:36:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-12T02:36:36.009 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-12T02:36:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-12T02:36:36.059 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 102/164 2024-09-12T02:36:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-12T02:36:36.701 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-12T02:36:36.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-12T02:36:36.779 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-12T02:36:36.851 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:36.889 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 106/164 2024-09-12T02:36:38.124 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-12T02:36:38.202 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-12T02:36:38.229 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-12T02:36:38.263 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-12T02:36:38.590 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-12T02:36:38.633 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:38.708 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 101/164 2024-09-12T02:36:38.747 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 102/164 2024-09-12T02:36:38.783 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-12T02:36:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-12T02:36:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 105/164 2024-09-12T02:36:39.430 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 106/164 2024-09-12T02:36:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:40.471 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T02:36:40.471 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T02:36:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:41.844 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:41.876 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:42.450 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 107/164 2024-09-12T02:36:42.488 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:42.580 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 108/164 2024-09-12T02:36:42.580 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T02:36:42.581 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T02:36:42.581 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:42.614 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-09-12T02:36:45.066 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:45.236 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:45.253 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:45.253 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:45.253 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T02:36:45.253 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:46.714 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:46.742 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:46.964 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:46.988 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:47.423 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:47.447 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:47.561 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-12T02:36:47.627 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-12T02:36:47.682 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-12T02:36:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-12T02:36:47.924 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-12T02:36:48.010 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-12T02:36:48.106 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-12T02:36:48.246 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-12T02:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-12T02:36:48.339 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-12T02:36:48.367 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-12T02:36:48.759 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:48.790 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:48.821 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 126/164 2024-09-12T02:36:48.884 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 127/164 2024-09-12T02:36:48.922 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:36:50.963 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:36:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:36:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:36:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 131/164 2024-09-12T02:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:36:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:51.368 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:36:51.380 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:36:51.485 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 134/164 2024-09-12T02:36:51.533 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 135/164 2024-09-12T02:36:52.457 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-12T02:36:52.487 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:36:52.511 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:52.554 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:36:52.578 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:36:52.578 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:52.579 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T02:36:52.579 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:52.630 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:36:52.654 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:52.926 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-09-12T02:36:52.926 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-09-12T02:36:52.926 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-09-12T02:36:52.926 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-09-12T02:36:52.927 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-09-12T02:36:52.927 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-09-12T02:36:52.927 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-09-12T02:36:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:36:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:36:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:53.132 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:53.132 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:53.132 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T02:36:53.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:54.705 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:54.705 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T02:36:54.705 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:54.705 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:54.789 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 109/164 2024-09-12T02:36:54.789 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-09-12T02:36:54.789 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-09-12T02:36:54.789 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-09-12T02:36:54.790 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-09-12T02:36:54.790 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-09-12T02:36:54.790 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-09-12T02:36:54.790 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-09-12T02:36:54.790 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:54.967 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:54.969 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:54.970 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T02:36:54.970 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:54.970 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:54.982 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 110/164 2024-09-12T02:36:54.983 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:54.983 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T02:36:54.983 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:55.428 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:55.454 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:55.454 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:55.454 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T02:36:55.454 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:55.455 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:55.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:55.559 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-12T02:36:55.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-12T02:36:55.682 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-12T02:36:55.884 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-12T02:36:55.930 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-12T02:36:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-12T02:36:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-12T02:36:56.252 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-12T02:36:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-12T02:36:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-12T02:36:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-12T02:36:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 111/164 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T02:36:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:56.770 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:56.793 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 112/164 2024-09-12T02:36:56.793 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:56.794 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T02:36:56.794 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:56.794 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T02:36:56.794 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 126/164 2024-09-12T02:36:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 127/164 2024-09-12T02:36:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:57.255 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 113/164 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T02:36:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:36:57.402 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-12T02:36:57.461 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-12T02:36:57.524 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-12T02:36:57.526 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 141/164 2024-09-12T02:36:57.567 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 142/164 2024-09-12T02:36:57.616 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 143/164 2024-09-12T02:36:57.648 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-12T02:36:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-12T02:36:57.719 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-12T02:36:57.773 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-12T02:36:57.854 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-12T02:36:57.858 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 146/164 2024-09-12T02:36:57.906 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-12T02:36:57.936 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 148/164 2024-09-12T02:36:57.943 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-12T02:36:57.966 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-12T02:36:58.012 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-12T02:36:58.085 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-12T02:36:58.148 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-12T02:36:58.179 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-12T02:36:58.206 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-12T02:36:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:36:58.618 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:36:58.650 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:58.659 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:36:58.680 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 125/164 2024-09-12T02:36:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:36:58.712 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 126/164 2024-09-12T02:36:58.778 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 127/164 2024-09-12T02:36:58.803 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:59.289 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:36:59.314 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:36:59.425 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:36:59.513 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 152/164 2024-09-12T02:36:59.592 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 131/164 2024-09-12T02:36:59.605 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-12T02:36:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:36:59.642 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:36:59.642 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:36:59.642 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T02:36:59.643 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:36:59.643 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:36:59.643 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:36:59.649 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-12T02:36:59.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:36:59.704 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:36:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-12T02:36:59.758 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-12T02:36:59.784 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-12T02:36:59.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 134/164 2024-09-12T02:36:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-12T02:36:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:36:59.866 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 135/164 2024-09-12T02:36:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 128/164 2024-09-12T02:36:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:36:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:36:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:37:00.003 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:37:00.028 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-12T02:37:00.034 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:00.034 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:00.213 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:00.214 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:00.306 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:00.369 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:00.849 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-12T02:37:00.886 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:37:00.911 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:37:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T02:37:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:37:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:37:00.912 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:00.953 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:37:00.964 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 129/164 2024-09-12T02:37:00.978 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:37:00.978 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:00.978 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T02:37:00.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:37:01.029 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:37:01.054 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:37:01.054 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:01.055 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-12T02:37:01.055 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:37:01.055 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:37:01.055 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:01.111 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 130/164 2024-09-12T02:37:01.274 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 131/164 2024-09-12T02:37:01.307 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:37:01.330 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 132/164 2024-09-12T02:37:01.331 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:01.331 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T02:37:01.331 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:37:01.331 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T02:37:01.331 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:37:01.383 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:37:01.394 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 133/164 2024-09-12T02:37:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:37:01.397 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:01.397 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T02:37:01.397 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:37:01.397 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:37:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:01.496 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 134/164 2024-09-12T02:37:01.549 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 135/164 2024-09-12T02:37:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-12T02:37:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 137/164 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T02:37:02.584 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:02.627 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:37:02.649 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 138/164 2024-09-12T02:37:02.649 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:02.649 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T02:37:02.650 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:02.694 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 139/164 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-12T02:37:02.718 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:03.037 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:37:03.059 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 140/164 2024-09-12T02:37:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T02:37:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T02:37:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:37:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T02:37:03.060 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:06.105 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:06.105 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/164 2024-09-12T02:37:06.105 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/164 2024-09-12T02:37:06.105 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/164 2024-09-12T02:37:06.105 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 5/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:37:06.106 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:37:06.107 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/164 2024-09-12T02:37:06.107 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/164 2024-09-12T02:37:06.107 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:37:06.107 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 14/164 2024-09-12T02:37:06.107 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 15/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 16/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 17/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 18/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 19/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 20/164 2024-09-12T02:37:06.108 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 21/164 2024-09-12T02:37:06.109 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 22/164 2024-09-12T02:37:06.109 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/164 2024-09-12T02:37:06.109 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 24/164 2024-09-12T02:37:06.109 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 25/164 2024-09-12T02:37:06.109 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 28/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 30/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 31/164 2024-09-12T02:37:06.110 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 32/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 33/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 34/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 35/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 36/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 37/164 2024-09-12T02:37:06.111 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 38/164 2024-09-12T02:37:06.112 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 39/164 2024-09-12T02:37:06.112 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-12T02:37:06.112 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-12T02:37:06.112 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-12T02:37:06.112 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-12T02:37:06.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-12T02:37:06.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-12T02:37:06.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-12T02:37:06.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-12T02:37:06.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-12T02:37:06.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-12T02:37:06.115 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-12T02:37:06.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-12T02:37:06.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-12T02:37:06.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-12T02:37:06.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-12T02:37:06.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-12T02:37:06.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-12T02:37:06.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-12T02:37:06.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-12T02:37:06.120 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-12T02:37:06.120 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-12T02:37:06.120 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-12T02:37:06.120 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-12T02:37:06.120 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-12T02:37:06.121 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-12T02:37:06.122 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-12T02:37:06.123 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-12T02:37:06.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-12T02:37:06.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 149/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 151/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-12T02:37:06.126 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-12T02:37:06.127 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-12T02:37:06.127 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-12T02:37:06.127 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-12T02:37:06.127 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-12T02:37:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 141/164 2024-09-12T02:37:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 142/164 2024-09-12T02:37:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 143/164 2024-09-12T02:37:06.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-12T02:37:06.392 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-12T02:37:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 146/164 2024-09-12T02:37:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-12T02:37:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 148/164 2024-09-12T02:37:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-12T02:37:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-12T02:37:07.427 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:07.469 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:07.490 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:07.899 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 141/164 2024-09-12T02:37:07.949 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 142/164 2024-09-12T02:37:07.990 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 143/164 2024-09-12T02:37:08.021 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-12T02:37:08.049 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-12T02:37:08.246 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 146/164 2024-09-12T02:37:08.353 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-12T02:37:08.383 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 148/164 2024-09-12T02:37:08.411 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 152/164 2024-09-12T02:37:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-12T02:37:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-12T02:37:08.499 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-12T02:37:08.548 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-12T02:37:08.635 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-12T02:37:08.669 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-12T02:37:08.694 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-12T02:37:08.720 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-12T02:37:08.720 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:37:08.813 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:37:08.813 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:37:08.890 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:08.890 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:37:08.890 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: automake-1.16.2-8.el9.noarch 2024-09-12T02:37:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: bison-3.7.4-5.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: flex-2.6.4-9.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T02:37:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-12T02:37:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-13.16-1.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-12T02:37:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T02:37:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:08.899 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T02:37:08.900 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T02:37:08.901 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T02:37:08.902 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:37:08.903 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-12T02:37:08.905 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:37:08.938 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-12T02:37:08.945 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:08.945 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:09.074 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:09.107 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:09.126 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-12T02:37:09.141 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:09.141 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:09.225 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:09.293 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:09.328 DEBUG:teuthology.parallel:result is None 2024-09-12T02:37:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 152/164 2024-09-12T02:37:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-12T02:37:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-12T02:37:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-12T02:37:10.264 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-12T02:37:10.290 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-12T02:37:10.316 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-12T02:37:10.316 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:37:10.400 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-12T02:37:10.400 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:37:10.484 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-12T02:37:10.509 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-12T02:37:10.514 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:10.515 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:10.719 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-12T02:37:10.719 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:10.812 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-12T02:37:10.879 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 5/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:37:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 14/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 15/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 16/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 17/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 18/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 19/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 20/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 21/164 2024-09-12T02:37:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 22/164 2024-09-12T02:37:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/164 2024-09-12T02:37:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 24/164 2024-09-12T02:37:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 25/164 2024-09-12T02:37:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/164 2024-09-12T02:37:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/164 2024-09-12T02:37:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 28/164 2024-09-12T02:37:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 30/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 31/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 32/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 33/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 34/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 35/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 36/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 37/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 38/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 39/164 2024-09-12T02:37:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-12T02:37:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-12T02:37:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-12T02:37:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-12T02:37:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-12T02:37:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-12T02:37:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-12T02:37:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-12T02:37:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 149/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 151/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-12T02:37:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-12T02:37:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-12T02:37:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-12T02:37:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-12T02:37:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-12T02:37:16.551 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:16.551 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/164 2024-09-12T02:37:16.551 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/164 2024-09-12T02:37:16.551 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/164 2024-09-12T02:37:16.551 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d 5/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 14/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_6 15/164 2024-09-12T02:37:16.552 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 16/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 17/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 18/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_ 19/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 20/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9 21/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x 22/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 24/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 25/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 28/164 2024-09-12T02:37:16.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.e 30/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noar 31/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.no 32/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b 33/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9 34/164 2024-09-12T02:37:16.565 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 35/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el 36/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 37/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 38/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 39/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-12T02:37:16.566 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-12T02:37:16.567 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-12T02:37:16.568 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-12T02:37:16.569 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-12T02:37:16.570 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-12T02:37:16.571 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-12T02:37:16.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-12T02:37:16.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 149/164 2024-09-12T02:37:16.574 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 151/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-12T02:37:16.575 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-12T02:37:18.218 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:18.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:18.218 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-09-12T02:37:18.218 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:18.218 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T02:37:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T02:37:18.222 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T02:37:18.223 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.16-1.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T02:37:18.224 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T02:37:18.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T02:37:18.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T02:37:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T02:37:18.229 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-12T02:37:18.230 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T02:37:18.231 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-12T02:37:18.231 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:37:18.231 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T02:37:18.770 DEBUG:teuthology.parallel:result is None 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.135 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: automake-1.16.2-8.el9.noarch 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: bison-3.7.4-5.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.136 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.138 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-12T02:37:19.140 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-12T02:37:19.140 INFO:teuthology.orchestra.run.smithi033.stdout: flex-2.6.4-9.el9.x86_64 2024-09-12T02:37:19.140 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T02:37:19.140 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T02:37:19.141 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T02:37:19.142 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-13.16-1.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T02:37:19.143 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T02:37:19.144 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T02:37:19.145 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T02:37:19.146 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.147 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T02:37:19.148 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T02:37:19.149 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T02:37:19.150 INFO:teuthology.orchestra.run.smithi033.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-12T02:37:19.150 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-12T02:37:19.150 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T02:37:19.151 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-12T02:37:19.151 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:37:19.151 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-09-12T02:37:19.578 DEBUG:teuthology.parallel:result is None 2024-09-12T02:37:19.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:19.727 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T02:37:19.755 INFO:teuthology.orchestra.run.smithi016.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:19.755 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:19.755 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-12T02:37:19.757 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:19.902 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T02:37:19.925 INFO:teuthology.orchestra.run.smithi033.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:19.926 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:19.926 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-12T02:37:19.928 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:20.076 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T02:37:20.102 INFO:teuthology.orchestra.run.smithi184.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:20.103 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-12T02:37:20.103 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-12T02:37:20.106 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-12T02:37:20.106 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:37:20.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T02:37:20.140 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:37:20.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T02:37:20.173 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:37:20.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T02:37:20.205 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-12T02:37:20.205 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:37:20.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T02:37:20.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T02:37:20.306 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:37:20.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T02:37:20.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T02:37:20.405 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:37:20.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T02:37:20.435 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T02:37:20.505 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-12T02:37:20.506 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:37:20.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T02:37:20.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T02:37:20.607 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:37:20.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T02:37:20.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T02:37:20.704 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:37:20.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T02:37:20.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T02:37:20.802 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-12T02:37:20.802 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:37:20.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T02:37:20.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T02:37:20.904 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:37:20.904 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T02:37:20.933 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T02:37:21.001 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:37:21.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T02:37:21.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T02:37:21.098 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-12T02:37:21.195 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-12T02:37:21.195 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:21.196 INFO:tasks.cephadm:Cluster fsid is ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:37:21.196 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-12T02:37:21.196 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.33', 'mon.c': '172.21.15.184'} 2024-09-12T02:37:21.197 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-09-12T02:37:21.197 INFO:tasks.cephadm:First mgr is x 2024-09-12T02:37:21.197 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-12T02:37:21.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-09-12T02:37:21.228 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-09-12T02:37:21.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-09-12T02:37:21.283 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-12T02:37:21.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:21.424 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56038/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-298-gb53b6d3e', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-298.gb53b6d3e'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-09 17:24:58.599585', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-12T02:37:21.565 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:37:21.566 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm 2024-09-12T02:37:21.566 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm 2024-09-12T02:37:21.567 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:21.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 784498 Sep 12 02:37 /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:21.962 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.349 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 784498 Sep 12 02:37 /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.350 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.737 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 784498 Sep 12 02:37 /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.737 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.757 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.776 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-12T02:37:22.809 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc on all hosts... 2024-09-12T02:37:22.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc pull 2024-09-12T02:37:22.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc pull 2024-09-12T02:37:22.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc pull 2024-09-12T02:37:24.394 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc... 2024-09-12T02:37:24.401 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc... 2024-09-12T02:37:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc... 2024-09-12T02:37:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)", 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594", 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66" 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T02:37:52.754 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)", 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594", 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66" 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-12T02:37:56.471 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)", 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594", 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66" 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-09-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-12T02:37:57.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-09-12T02:37:58.023 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-09-12T02:37:58.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-09-12T02:37:58.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-09-12T02:37:58.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-09-12T02:37:58.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-09-12T02:37:58.165 INFO:tasks.cephadm:Writing seed config... 2024-09-12T02:37:58.166 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-12T02:37:58.166 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-12T02:37:58.167 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-12T02:37:58.168 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-12T02:37:58.169 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-12T02:37:58.170 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-09-12T02:37:58.171 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:37:58.171 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-12T02:37:58.189 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ef86fd2a-70af-11ef-bcea-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-12T02:37:58.189 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service 2024-09-12T02:37:58.232 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x.service 2024-09-12T02:37:58.275 INFO:tasks.cephadm:Bootstrapping... 2024-09-12T02:37:58.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc -v bootstrap --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-12T02:37:58.488 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T02:37:58.489 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', '-v', 'bootstrap', '--fsid', 'ef86fd2a-70af-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-09-12T02:37:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-12T02:37:58.489 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-09-12T02:37:58.513 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-12T02:37:58.513 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-09-12T02:37:58.513 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-09-12T02:37:58.521 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-12T02:37:58.521 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-12T02:37:58.528 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-12T02:37:58.528 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:37:58.534 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-12T02:37:58.541 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-12T02:37:58.541 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-12T02:37:58.541 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-09-12T02:37:58.565 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-12T02:37:58.565 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-12T02:37:58.565 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-09-12T02:37:58.565 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-09-12T02:37:58.572 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-12T02:37:58.572 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-12T02:37:58.579 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-12T02:37:58.579 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:37:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140514697053184 on /run/cephadm/ef86fd2a-70af-11ef-bcea-c7b262605968.lock 2024-09-12T02:37:58.593 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140514697053184 acquired on /run/cephadm/ef86fd2a-70af-11ef-bcea-c7b262605968.lock 2024-09-12T02:37:58.594 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-09-12T02:37:58.594 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-09-12T02:37:58.594 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-12T02:37:58.597 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-09-12T02:37:58.597 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-09-12T02:37:58.600 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-12T02:37:58.600 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-12T02:37:58.600 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-09-12T02:37:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-12T02:37:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-12T02:37:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-12T02:37:58.603 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-12T02:37:58.604 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-12T02:37:58.604 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc... 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc... 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:60180723200353b4e13b39fd162aecaf1bc097c859c7c6537324229e97c78567 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-12T02:38:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 2024-09-12T02:38:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-12T02:38:02.581 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable) 2024-09-12T02:38:02.581 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable) 2024-09-12T02:38:02.581 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-09-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-09-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-09-12T02:38:04.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQALVOJmz2f6MBAAp9zK86CJKY+A4sl8DViFIQ== 2024-09-12T02:38:05.407 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQAMVOJmPBtiLRAAcEgLHoXp2EVX2SQaSO/e8Q== 2024-09-12T02:38:06.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQANVOJmwMShLBAAsCu+hCsEdSqGeUvZrAVF1g== 2024-09-12T02:38:06.305 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-09-12T02:38:10.330 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:38:10.331 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-09-12T02:38:11.236 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-09-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-12T02:38:11.594 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-12T02:38:11.783 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target. 2024-09-12T02:38:11.784 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target. 2024-09-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a 2024-09-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target.wants/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@.service. 2024-09-12T02:38:12.652 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-12T02:38:12.652 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-12T02:38:12.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:12 smithi016 systemd[1]: Started Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:38:12.659 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-12T02:38:12.659 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:38:12.659 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-12T02:38:12.659 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-12T02:38:12.660 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-09-12T02:38:12.660 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-09-12T02:38:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-09-12T02:38:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-12T02:38:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.539271s) 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.086+0000 7f1af4bfe640 1 Processor -- start 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.086+0000 7f1af4bfe640 1 -- start start 2024-09-12T02:38:13.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1af4bfe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1af4bfe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1af007be60 con 0x7f1af0106010 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1aee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1aee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45506/0 (socket says 172.21.15.16:45506) 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1aee59c640 1 -- 172.21.15.16:0/916232398 learned_addr learned my addr 172.21.15.16:0/916232398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.087+0000 7f1aee59c640 1 -- 172.21.15.16:0/916232398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af0106950 con 0x7f1af0106010 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.089+0000 7f1aee59c640 1 --2- 172.21.15.16:0/916232398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009820 tx=0x7f1ae002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc20153acc370f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1aed59a640 1 -- 172.21.15.16:0/916232398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ae0037070 con 0x7f1af0106010 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1aed59a640 1 -- 172.21.15.16:0/916232398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae0032440 con 0x7f1af0106010 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1aed59a640 1 -- 172.21.15.16:0/916232398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ae0032740 con 0x7f1af0106010 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 -- 172.21.15.16:0/916232398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 msgr2=0x7f1af0106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 --2- 172.21.15.16:0/916232398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009820 tx=0x7f1ae002a270 comp rx=0 tx=0).stop 2024-09-12T02:38:13.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 -- 172.21.15.16:0/916232398 shutdown_connections 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 --2- 172.21.15.16:0/916232398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af0106410 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 -- 172.21.15.16:0/916232398 >> 172.21.15.16:0/916232398 conn(0x7f1af0101e60 msgr2=0x7f1af0104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 -- 172.21.15.16:0/916232398 shutdown_connections 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.090+0000 7f1af4bfe640 1 -- 172.21.15.16:0/916232398 wait complete. 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.091+0000 7f1af4bfe640 1 Processor -- start 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.091+0000 7f1af4bfe640 1 -- start start 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.091+0000 7f1af4bfe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.091+0000 7f1af4bfe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0106ef0 con 0x7f1af0106010 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1aee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1aee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45510/0 (socket says 172.21.15.16:45510) 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1aee59c640 1 -- 172.21.15.16:0/3260505768 learned_addr learned my addr 172.21.15.16:0/3260505768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1aee59c640 1 -- 172.21.15.16:0/3260505768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af019d6f0 con 0x7f1af0106010 2024-09-12T02:38:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1aee59c640 1 --2- 172.21.15.16:0/3260505768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00090b0 tx=0x7f1ae00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ae0041070 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae003c7a0 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ae0037040 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af019d980 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af01a0670 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.092+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ae0044070 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.093+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1ae003d3f0 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.093+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af0106410 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.096+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1ae0046450 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.157+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f1af01a0bc0 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1adf7fe640 1 -- 172.21.15.16:0/3260505768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f1ae0046630 con 0x7f1af0106010 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 msgr2=0x7f1af019d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:13.715 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1af4bfe640 1 --2- 172.21.15.16:0/3260505768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00090b0 tx=0x7f1ae00046c0 comp rx=0 tx=0).stop 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 shutdown_connections 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1af4bfe640 1 --2- 172.21.15.16:0/3260505768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0106010 0x7f1af019d1b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.158+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 >> 172.21.15.16:0/3260505768 conn(0x7f1af0101e60 msgr2=0x7f1af0193790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.159+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 shutdown_connections 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:13.159+0000 7f1af4bfe640 1 -- 172.21.15.16:0/3260505768 wait complete. 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-09-12T02:38:13.716 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: monmap epoch 1 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: last_changed 2024-09-12T02:38:06.636712+0000 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: min_mon_release 19 (squid) 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: election_strategy: 1 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: fsmap 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: osdmap e1: 0 total, 0 up, 0 in 2024-09-12T02:38:13.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: mgrmap e1: no daemons active 2024-09-12T02:38:13.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:13 smithi016 ceph-mon[30561]: from='client.? 172.21.15.16:0/3260505768' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-12T02:38:14.759 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-12T02:38:14.760 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f825df640 1 Processor -- start 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f825df640 1 -- start start 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f825df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f825df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c1070f0 con 0x7f1f7c106720 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45512/0 (socket says 172.21.15.16:45512) 2024-09-12T02:38:14.761 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.102+0000 7f1f7bfff640 1 -- 172.21.15.16:0/4247955278 learned_addr learned my addr 172.21.15.16:0/4247955278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.103+0000 7f1f7bfff640 1 -- 172.21.15.16:0/4247955278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c107920 con 0x7f1f7c106720 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.103+0000 7f1f7bfff640 1 --2- 172.21.15.16:0/4247955278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009820 tx=0x7f1f6c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f46937d47f8264 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.103+0000 7f1f7affd640 1 -- 172.21.15.16:0/4247955278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c037070 con 0x7f1f7c106720 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f7affd640 1 -- 172.21.15.16:0/4247955278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6c032440 con 0x7f1f7c106720 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f7affd640 1 -- 172.21.15.16:0/4247955278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c032740 con 0x7f1f7c106720 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 -- 172.21.15.16:0/4247955278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 msgr2=0x7f1f7c106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 --2- 172.21.15.16:0/4247955278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009820 tx=0x7f1f6c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 -- 172.21.15.16:0/4247955278 shutdown_connections 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 --2- 172.21.15.16:0/4247955278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c106b20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 -- 172.21.15.16:0/4247955278 >> 172.21.15.16:0/4247955278 conn(0x7f1f7c101f10 msgr2=0x7f1f7c104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 -- 172.21.15.16:0/4247955278 shutdown_connections 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.104+0000 7f1f825df640 1 -- 172.21.15.16:0/4247955278 wait complete. 2024-09-12T02:38:14.762 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f825df640 1 Processor -- start 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f825df640 1 -- start start 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f825df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f825df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c107ec0 con 0x7f1f7c106720 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45518/0 (socket says 172.21.15.16:45518) 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f7bfff640 1 -- 172.21.15.16:0/363358773 learned_addr learned my addr 172.21.15.16:0/363358773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.105+0000 7f1f7bfff640 1 -- 172.21.15.16:0/363358773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c18f820 con 0x7f1f7c106720 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f7bfff640 1 --2- 172.21.15.16:0/363358773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0090b0 tx=0x7f1f6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c041070 con 0x7f1f7c106720 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6c03c7a0 con 0x7f1f7c106720 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c037040 con 0x7f1f7c106720 2024-09-12T02:38:14.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c18da10 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c18df50 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1f6c044070 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.106+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1f6c03d3f0 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.107+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c108510 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.109+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1f6c046450 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.169+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1f7c18e250 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.173+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f1f6c046630 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.174+0000 7f1f797fa640 1 -- 172.21.15.16:0/363358773 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1f6c0584b0 con 0x7f1f7c106720 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.174+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 msgr2=0x7f1f7c18f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.174+0000 7f1f825df640 1 --2- 172.21.15.16:0/363358773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0090b0 tx=0x7f1f6c0046c0 comp rx=0 tx=0).stop 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.175+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 shutdown_connections 2024-09-12T02:38:14.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.175+0000 7f1f825df640 1 --2- 172.21.15.16:0/363358773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c106720 0x7f1f7c18f2e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:14.765 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.175+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 >> 172.21.15.16:0/363358773 conn(0x7f1f7c101f10 msgr2=0x7f1f7c07b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:14.765 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.175+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 shutdown_connections 2024-09-12T02:38:14.765 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:14.175+0000 7f1f825df640 1 -- 172.21.15.16:0/363358773 wait complete. 2024-09-12T02:38:14.765 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-09-12T02:38:14.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:14 smithi016 ceph-mon[30561]: from='client.? 172.21.15.16:0/363358773' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-12T02:38:14.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:14 smithi016 ceph-mon[30561]: from='client.? 172.21.15.16:0/363358773' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-12T02:38:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:15 smithi016 ceph-mon[30561]: from='client.? 172.21.15.16:0/423814414' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.158+0000 7fcefc02d640 1 Processor -- start 2024-09-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcefc02d640 1 -- start start 2024-09-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcefc02d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcefc02d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4108200 con 0x7fcef4107830 2024-09-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcef9dc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcef9dc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45534/0 (socket says 172.21.15.16:45534) 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcef9dc9640 1 -- 172.21.15.16:0/1171779119 learned_addr learned my addr 172.21.15.16:0/1171779119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.159+0000 7fcef9dc9640 1 -- 172.21.15.16:0/1171779119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef41089e0 con 0x7fcef4107830 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.160+0000 7fcef9dc9640 1 --2- 172.21.15.16:0/1171779119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009820 tx=0x7fcef002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c607f8e1568737 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.160+0000 7fcef8dc7640 1 -- 172.21.15.16:0/1171779119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef0037070 con 0x7fcef4107830 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.160+0000 7fcef8dc7640 1 -- 172.21.15.16:0/1171779119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcef0032900 con 0x7fcef4107830 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.160+0000 7fcef8dc7640 1 -- 172.21.15.16:0/1171779119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef0030850 con 0x7fcef4107830 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 -- 172.21.15.16:0/1171779119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 msgr2=0x7fcef4107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 --2- 172.21.15.16:0/1171779119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009820 tx=0x7fcef002a270 comp rx=0 tx=0).stop 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 -- 172.21.15.16:0/1171779119 shutdown_connections 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 --2- 172.21.15.16:0/1171779119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef4107c30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 -- 172.21.15.16:0/1171779119 >> 172.21.15.16:0/1171779119 conn(0x7fcef4103040 msgr2=0x7fcef4105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 -- 172.21.15.16:0/1171779119 shutdown_connections 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.161+0000 7fcefc02d640 1 -- 172.21.15.16:0/1171779119 wait complete. 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcefc02d640 1 Processor -- start 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcefc02d640 1 -- start start 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcefc02d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcefc02d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcef410bd90 con 0x7fcef4107830 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcef9dc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcef9dc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45546/0 (socket says 172.21.15.16:45546) 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcef9dc9640 1 -- 172.21.15.16:0/423814414 learned_addr learned my addr 172.21.15.16:0/423814414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcef9dc9640 1 -- 172.21.15.16:0/423814414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef419bd40 con 0x7fcef4107830 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.162+0000 7fcef9dc9640 1 --2- 172.21.15.16:0/423814414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcef00097f0 tx=0x7fcef0030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef0037040 con 0x7fcef4107830 2024-09-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcef0040070 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef419bfd0 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef003a390 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef419ecc0 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcef003a600 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.163+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcef00459a0 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.164+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef4107cb0 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.166+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fcef0045460 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.226+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fcef419ee90 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.226+0000 7fceeaffd640 1 -- 172.21.15.16:0/423814414 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fcef003b430 con 0x7fcef4107830 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 msgr2=0x7fcef419b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 --2- 172.21.15.16:0/423814414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcef00097f0 tx=0x7fcef0030d80 comp rx=0 tx=0).stop 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 shutdown_connections 2024-09-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 --2- 172.21.15.16:0/423814414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef4107830 0x7fcef419b800 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 >> 172.21.15.16:0/423814414 conn(0x7fcef4103040 msgr2=0x7fcef4103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 shutdown_connections 2024-09-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:15.227+0000 7fcefc02d640 1 -- 172.21.15.16:0/423814414 wait complete. 2024-09-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-09-12T02:38:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 systemd[1]: Stopping Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:38:16.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a[30557]: 2024-09-12T02:38:16.527+0000 7f21bfa9e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T02:38:16.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a[30557]: 2024-09-12T02:38:16.527+0000 7f21bfa9e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-12T02:38:16.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 podman[30801]: 2024-09-12 02:38:16.579222245 +0000 UTC m=+0.148573067 container died 7c975591e240876c78721606c9e1a9f7c6ab091a875f7faff8dd342d65dbc5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-12T02:38:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 podman[30801]: 2024-09-12 02:38:16.952188736 +0000 UTC m=+0.521539532 container cleanup 7c975591e240876c78721606c9e1a9f7c6ab091a875f7faff8dd342d65dbc5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-12T02:38:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:16 smithi016 bash[30801]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a 2024-09-12T02:38:17.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 podman[30813]: 2024-09-12 02:38:17.194763613 +0000 UTC m=+0.618402963 container remove 7c975591e240876c78721606c9e1a9f7c6ab091a875f7faff8dd342d65dbc5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T02:38:17.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-12T02:38:17.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 systemd[1]: Stopped Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:38:17.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 systemd[1]: Starting Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:38:17.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 podman[30906]: 2024-09-12 02:38:17.453468512 +0000 UTC m=+0.016484476 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:38:17.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 podman[30906]: 2024-09-12 02:38:17.576658756 +0000 UTC m=+0.139674763 container create 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-12T02:38:17.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 podman[30906]: 2024-09-12 02:38:17.789575453 +0000 UTC m=+0.352591402 container init 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-12T02:38:17.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 podman[30906]: 2024-09-12 02:38:17.792747208 +0000 UTC m=+0.355763157 container start 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-12T02:38:17.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: set uid:gid to 167:167 (ceph:ceph) 2024-09-12T02:38:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-12T02:38:18.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable), process ceph-mon, pid 2 2024-09-12T02:38:18.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: pidfile_write: ignore empty --pid-file 2024-09-12T02:38:18.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: load: jerasure load: lrc 2024-09-12T02:38:18.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: RocksDB version: 7.9.2 2024-09-12T02:38:18.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Git sha 0 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Compile date 2024-09-09 16:23:28 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: DB SUMMARY 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: DB Session ID: 8HXYAYZWPESXYKHI45VF 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: CURRENT file: CURRENT 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: IDENTITY file: IDENTITY 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106764 ; 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.error_if_exists: 0 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.create_if_missing: 0 2024-09-12T02:38:18.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.paranoid_checks: 1 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.env: 0x559accfb6940 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.fs: PosixFileSystem 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.info_log: 0x559acefcf6e0 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_file_opening_threads: 16 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.statistics: (nil) 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.use_fsync: 0 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_log_file_size: 0 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-12T02:38:18.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.keep_log_file_num: 1000 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.recycle_log_file_num: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_fallocate: 1 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_mmap_reads: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_mmap_writes: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.use_direct_reads: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.create_missing_column_families: 0 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.db_log_dir: 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.wal_dir: 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-12T02:38:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.advise_random_on_open: 1 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.db_write_buffer_size: 0 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.write_buffer_manager: 0x559acefdedc0 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-12T02:38:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.rate_limiter: (nil) 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.wal_recovery_mode: 2 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enable_thread_tracking: 0 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enable_pipelined_write: 0 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.unordered_write: 0 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.row_cache: None 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.wal_filter: None 2024-09-12T02:38:18.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_ingest_behind: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.two_write_queues: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.manual_wal_flush: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.wal_compression: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.atomic_flush: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.log_readahead_size: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.best_efforts_recovery: 0 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-12T02:38:18.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.allow_data_in_errors: 0 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.db_host_id: __hostname__ 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_background_jobs: 2 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_background_compactions: -1 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_subcompactions: 1 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_total_wal_size: 0 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-12T02:38:18.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_open_files: -1 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bytes_per_sync: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_readahead_size: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_background_flushes: -1 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Compression algorithms supported: 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kZSTD supported: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kXpressCompression supported: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kBZip2Compression supported: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kLZ4Compression supported: 1 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kZlibCompression supported: 1 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kLZ4HCCompression supported: 1 2024-09-12T02:38:18.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: kSnappyCompression supported: 1 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.merge_operator: 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_filter: None 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_filter_factory: None 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.sst_partitioner_factory: None 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559acefcf320) 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-12T02:38:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x559acefc31f0 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-09-12T02:38:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-09-12T02:38:18.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.write_buffer_size: 33554432 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_write_buffer_number: 2 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression: NoCompression 2024-09-12T02:38:18.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression: Disabled 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.prefix_extractor: nullptr 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.num_levels: 7 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-12T02:38:18.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.level: 32767 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.strategy: 0 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.enabled: false 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-12T02:38:18.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.target_file_size_base: 67108864 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-12T02:38:18.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.arena_block_size: 1048576 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.disable_auto_compactions: 0 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-12T02:38:18.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.inplace_update_support: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.bloom_locality: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.max_successive_merges: 0 2024-09-12T02:38:18.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.paranoid_file_checks: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.force_consistency_checks: 1 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.report_bg_io_stats: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.ttl: 2592000 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enable_blob_files: false 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.min_blob_size: 0 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_file_size: 268435456 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-12T02:38:18.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.blob_file_starting_level: 0 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 483d2b5c-97fc-493d-9a92-0cc17494f619 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108697820529, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108697821547, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 334, "table_properties": {"data_size": 99583, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14671, "raw_average_key_size": 50, "raw_value_size": 91638, "raw_average_value_size": 312, "num_data_blocks": 12, "num_entries": 293, "num_filter_entries": 293, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726108697, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "483d2b5c-97fc-493d-9a92-0cc17494f619", "db_session_id": "8HXYAYZWPESXYKHI45VF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108697821647, "job": 1, "event": "recovery_finished"} 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-12T02:38:18.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559aceff0e00 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: DB pointer 0x559acf0ec000 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-12T02:38:18.122 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 101.17 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 101.17 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-12T02:38:18.123 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 29.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 29.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-12T02:38:18.124 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x559acefc31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???) e1 preinit fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).mds e1 new map 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).mds e1 print_map 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: btime 2024-09-12T02:38:12:619852+0000 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-12T02:38:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 bash[30906]: 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 systemd[1]: Started Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: monmap epoch 1 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: last_changed 2024-09-12T02:38:06.636712+0000 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: min_mon_release 19 (squid) 2024-09-12T02:38:18.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: election_strategy: 1 2024-09-12T02:38:18.127 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:38:18.127 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: fsmap 2024-09-12T02:38:18.127 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: osdmap e1: 0 total, 0 up, 0 in 2024-09-12T02:38:18.127 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:17 smithi016 ceph-mon[30925]: mgrmap e1: no daemons active 2024-09-12T02:38:18.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.284+0000 7f755e312640 1 Processor -- start 2024-09-12T02:38:18.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.284+0000 7f755e312640 1 -- start start 2024-09-12T02:38:18.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.284+0000 7f755e312640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:18.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.284+0000 7f755e312640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f755807c7d0 con 0x7f755807dde0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.284+0000 7f7557fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.285+0000 7f7557fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47420/0 (socket says 172.21.15.16:47420) 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.285+0000 7f7557fff640 1 -- 172.21.15.16:0/3928763572 learned_addr learned my addr 172.21.15.16:0/3928763572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.285+0000 7f7557fff640 1 -- 172.21.15.16:0/3928763572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755807c950 con 0x7f755807dde0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.286+0000 7f7557fff640 1 --2- 172.21.15.16:0/3928763572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7548009820 tx=0x7f754802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695e9f3e60d8b7e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f7556ffd640 1 -- 172.21.15.16:0/3928763572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7548037070 con 0x7f755807dde0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f7556ffd640 1 -- 172.21.15.16:0/3928763572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7548032900 con 0x7f755807dde0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f7556ffd640 1 -- 172.21.15.16:0/3928763572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7548030850 con 0x7f755807dde0 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f755e312640 1 -- 172.21.15.16:0/3928763572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 msgr2=0x7f755807c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f755e312640 1 --2- 172.21.15.16:0/3928763572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7548009820 tx=0x7f754802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f755e312640 1 -- 172.21.15.16:0/3928763572 shutdown_connections 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f755e312640 1 --2- 172.21.15.16:0/3928763572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755807c290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.287+0000 7f755e312640 1 -- 172.21.15.16:0/3928763572 >> 172.21.15.16:0/3928763572 conn(0x7f7558103040 msgr2=0x7f7558105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.288+0000 7f755e312640 1 -- 172.21.15.16:0/3928763572 shutdown_connections 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.288+0000 7f755e312640 1 -- 172.21.15.16:0/3928763572 wait complete. 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.288+0000 7f755e312640 1 Processor -- start 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.288+0000 7f755e312640 1 -- start start 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f755e312640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f755e312640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f755810de60 con 0x7f755807dde0 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f7557fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:18.913 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f7557fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47434/0 (socket says 172.21.15.16:47434) 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f7557fff640 1 -- 172.21.15.16:0/3950438717 learned_addr learned my addr 172.21.15.16:0/3950438717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f7557fff640 1 -- 172.21.15.16:0/3950438717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755819df10 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f7557fff640 1 --2- 172.21.15.16:0/3950438717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7548032ce0 tx=0x7f7548032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7548042070 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755819e1a0 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75480308c0 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.289+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7548037040 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.290+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75581a0e90 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.290+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7548045070 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.290+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7548047bc0 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.290+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f755807c310 con 0x7f755807dde0 2024-09-12T02:38:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.293+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f754803c830 con 0x7f755807dde0 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.352+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f75581a1060 con 0x7f755807dde0 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.354+0000 7f75557fa640 1 -- 172.21.15.16:0/3950438717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f754803ca10 con 0x7f755807dde0 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 msgr2=0x7f755819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 --2- 172.21.15.16:0/3950438717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7548032ce0 tx=0x7f7548032d10 comp rx=0 tx=0).stop 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 shutdown_connections 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 --2- 172.21.15.16:0/3950438717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f755807dde0 0x7f755819d9d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 >> 172.21.15.16:0/3950438717 conn(0x7f7558103040 msgr2=0x7f7558103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 shutdown_connections 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:18.355+0000 7f755e312640 1 -- 172.21.15.16:0/3950438717 wait complete. 2024-09-12T02:38:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-12T02:38:18.937 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-12T02:38:18.938 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-09-12T02:38:18.938 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-12T02:38:18.938 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-12T02:38:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x 2024-09-12T02:38:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x.service: Unit ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x.service not loaded. 2024-09-12T02:38:19.352 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968.target.wants/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@.service. 2024-09-12T02:38:19.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:19 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3950438717' entity='client.admin' 2024-09-12T02:38:19.586 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 systemd[1]: Starting Ceph mgr.x for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:38:19.884 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 podman[31138]: 2024-09-12 02:38:19.488255115 +0000 UTC m=+0.016160510 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-12T02:38:19.884 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 podman[31138]: 2024-09-12 02:38:19.606616698 +0000 UTC m=+0.134522124 container create 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T02:38:19.884 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 podman[31138]: 2024-09-12 02:38:19.80888259 +0000 UTC m=+0.336787984 container init 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-12T02:38:19.884 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 podman[31138]: 2024-09-12 02:38:19.811607743 +0000 UTC m=+0.339513128 container start 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-12T02:38:19.884 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 bash[31138]: 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 2024-09-12T02:38:19.898 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-12T02:38:19.898 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-12T02:38:19.906 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-12T02:38:19.906 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:38:19.906 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-12T02:38:19.907 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-12T02:38:19.915 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-12T02:38:19.916 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-12T02:38:19.923 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-12T02:38:19.924 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:38:19.924 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-12T02:38:19.924 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-12T02:38:19.924 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-09-12T02:38:19.924 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-09-12T02:38:20.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 systemd[1]: Started Ceph mgr.x for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:38:20.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:19 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:19.938+0000 7f2395dd3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-12T02:38:20.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.000+0000 7f2395dd3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:20.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.160+0000 7f2395dd3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-12T02:38:20.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.230+0000 7f2395dd3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-12T02:38:20.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.389+0000 7f2395dd3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:20.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2636718483' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-12T02:38:20.740 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.540+0000 7f2395dd3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-12T02:38:21.024 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.873+0000 7f2395dd3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-12T02:38:21.024 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:20.937+0000 7f2395dd3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-12T02:38:21.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:38:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-12T02:38:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-12T02:38:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-12T02:38:21.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-12T02:38:21.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:21.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-12T02:38:21.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-12T02:38:21.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-12T02:38:21.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-12T02:38:21.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-12T02:38:21.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-12T02:38:12:619852+0000", 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-12T02:38:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-12T02:38:12.620486+0000", 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.360+0000 7ff2830ec640 1 Processor -- start 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.360+0000 7ff2830ec640 1 -- start start 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.360+0000 7ff2830ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.361+0000 7ff2830ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c107080 con 0x7ff27c1066b0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.361+0000 7ff280e88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.361+0000 7ff280e88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47450/0 (socket says 172.21.15.16:47450) 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.361+0000 7ff280e88640 1 -- 172.21.15.16:0/1031267930 learned_addr learned my addr 172.21.15.16:0/1031267930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.361+0000 7ff280e88640 1 -- 172.21.15.16:0/1031267930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c1078b0 con 0x7ff27c1066b0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.362+0000 7ff280e88640 1 --2- 172.21.15.16:0/1031267930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff278009820 tx=0x7ff27802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd0f33ffee37628 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.362+0000 7ff2737fe640 1 -- 172.21.15.16:0/1031267930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278037070 con 0x7ff27c1066b0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.362+0000 7ff2737fe640 1 -- 172.21.15.16:0/1031267930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff278032900 con 0x7ff27c1066b0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.362+0000 7ff2737fe640 1 -- 172.21.15.16:0/1031267930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278030850 con 0x7ff27c1066b0 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 -- 172.21.15.16:0/1031267930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 msgr2=0x7ff27c106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 --2- 172.21.15.16:0/1031267930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff278009820 tx=0x7ff27802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 -- 172.21.15.16:0/1031267930 shutdown_connections 2024-09-12T02:38:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 --2- 172.21.15.16:0/1031267930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c106ab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 -- 172.21.15.16:0/1031267930 >> 172.21.15.16:0/1031267930 conn(0x7ff27c101e60 msgr2=0x7ff27c104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 -- 172.21.15.16:0/1031267930 shutdown_connections 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.363+0000 7ff2830ec640 1 -- 172.21.15.16:0/1031267930 wait complete. 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff2830ec640 1 Processor -- start 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff2830ec640 1 -- start start 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff2830ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff2830ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c10ac60 con 0x7ff27c1066b0 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff280e88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff280e88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47460/0 (socket says 172.21.15.16:47460) 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff280e88640 1 -- 172.21.15.16:0/2636718483 learned_addr learned my addr 172.21.15.16:0/2636718483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.364+0000 7ff280e88640 1 -- 172.21.15.16:0/2636718483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c191e20 con 0x7ff27c1066b0 2024-09-12T02:38:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff280e88640 1 --2- 172.21.15.16:0/2636718483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff278009940 tx=0x7ff278032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278042070 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2780308c0 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27c190010 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278037040 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff278045070 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27c190550 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.365+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff2780463b0 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.366+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27c10ade0 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.368+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff27804d030 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.433+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff27c1951f0 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.434+0000 7ff271ffb640 1 -- 172.21.15.16:0/2636718483 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7ff2780327b0 con 0x7ff27c1066b0 2024-09-12T02:38:21.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 msgr2=0x7ff27c1918e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 --2- 172.21.15.16:0/2636718483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff278009940 tx=0x7ff278032ce0 comp rx=0 tx=0).stop 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 shutdown_connections 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 --2- 172.21.15.16:0/2636718483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27c1066b0 0x7ff27c1918e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 >> 172.21.15.16:0/2636718483 conn(0x7ff27c101e60 msgr2=0x7ff27c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 shutdown_connections 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:20.435+0000 7ff2830ec640 1 -- 172.21.15.16:0/2636718483 wait complete. 2024-09-12T02:38:21.048 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-09-12T02:38:21.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:21 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:21.136+0000 7f2395dd3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-12T02:38:22.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:22.060+0000 7f2395dd3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-12T02:38:22.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:22.131+0000 7f2395dd3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-12T02:38:23.155 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:22.884+0000 7f2395dd3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-12T02:38:23.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.004+0000 7f2395dd3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-12T02:38:23.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.155+0000 7f2395dd3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:23.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.219+0000 7f2395dd3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-12T02:38:23.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.379+0000 7f2395dd3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-12T02:38:23.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/653095862' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-12T02:38:23.758 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.453+0000 7f2395dd3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-12T02:38:23.758 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.517+0000 7f2395dd3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-12T02:38:23.758 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.688+0000 7f2395dd3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-12T02:38:23.758 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.757+0000 7f2395dd3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-12T02:38:24.038 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.827+0000 7f2395dd3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-12T02:38:24.039 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.914+0000 7f2395dd3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:24.039 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:23.977+0000 7f2395dd3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-12T02:38:24.039 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:24.038+0000 7f2395dd3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-12T02:38:24.168 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:24.168 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:24.168 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:38:24.168 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-12T02:38:24.168 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:24.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-12T02:38:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-12T02:38:24.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-12T02:38:24.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-12T02:38:12:619852+0000", 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-12T02:38:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-12T02:38:12.620486+0000", 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f9e6d2640 1 Processor -- start 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f9e6d2640 1 -- start start 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f9e6d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f9e6d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f98107080 con 0x7f0f981066b0 2024-09-12T02:38:24.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47472/0 (socket says 172.21.15.16:47472) 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f97fff640 1 -- 172.21.15.16:0/2015069063 learned_addr learned my addr 172.21.15.16:0/2015069063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.439+0000 7f0f97fff640 1 -- 172.21.15.16:0/2015069063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f981078b0 con 0x7f0f981066b0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.440+0000 7f0f97fff640 1 --2- 172.21.15.16:0/2015069063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009820 tx=0x7f0f8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e442d185ac8b9d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.440+0000 7f0f977fe640 1 -- 172.21.15.16:0/2015069063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88037070 con 0x7f0f981066b0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.440+0000 7f0f977fe640 1 -- 172.21.15.16:0/2015069063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f88032900 con 0x7f0f981066b0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.440+0000 7f0f977fe640 1 -- 172.21.15.16:0/2015069063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88030850 con 0x7f0f981066b0 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/2015069063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 msgr2=0x7f0f98106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 --2- 172.21.15.16:0/2015069063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009820 tx=0x7f0f8802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/2015069063 shutdown_connections 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 --2- 172.21.15.16:0/2015069063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f98106ab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/2015069063 >> 172.21.15.16:0/2015069063 conn(0x7f0f98101e60 msgr2=0x7f0f98104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/2015069063 shutdown_connections 2024-09-12T02:38:24.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.441+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/2015069063 wait complete. 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f9e6d2640 1 Processor -- start 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f9e6d2640 1 -- start start 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f9e6d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f9e6d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9810ac60 con 0x7f0f981066b0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47478/0 (socket says 172.21.15.16:47478) 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.442+0000 7f0f97fff640 1 -- 172.21.15.16:0/653095862 learned_addr learned my addr 172.21.15.16:0/653095862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f97fff640 1 -- 172.21.15.16:0/653095862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9819e550 con 0x7f0f981066b0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f97fff640 1 --2- 172.21.15.16:0/653095862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0f880097f0 tx=0x7f0f88030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88037040 con 0x7f0f981066b0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f88040070 con 0x7f0f981066b0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f9819e720 con 0x7f0f981066b0 2024-09-12T02:38:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f8803a390 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f9819eac0 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0f8803a5e0 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.443+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0f88045950 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.444+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f9810ade0 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.446+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0f88045460 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.509+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0f9819ede0 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.509+0000 7f0f95ffb640 1 -- 172.21.15.16:0/653095862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f0f8803b430 con 0x7f0f981066b0 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.510+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 msgr2=0x7f0f9807ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.510+0000 7f0f9e6d2640 1 --2- 172.21.15.16:0/653095862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0f880097f0 tx=0x7f0f88030d80 comp rx=0 tx=0).stop 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.510+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 shutdown_connections 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.510+0000 7f0f9e6d2640 1 --2- 172.21.15.16:0/653095862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f981066b0 0x7f0f9807ba00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.510+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 >> 172.21.15.16:0/653095862 conn(0x7f0f98101e60 msgr2=0x7f0f981029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.511+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 shutdown_connections 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:23.511+0000 7f0f9e6d2640 1 -- 172.21.15.16:0/653095862 wait complete. 2024-09-12T02:38:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-09-12T02:38:25.157 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:24.866+0000 7f2395dd3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: Activating manager daemon x 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: mgrmap e2: x(active, starting, since 0.00317978s) 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: Manager daemon x is now available 2024-09-12T02:38:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-12T02:38:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' 2024-09-12T02:38:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-12T02:38:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' 2024-09-12T02:38:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:24 smithi016 ceph-mon[30925]: from='mgr.14100 172.21.15.16:0/179377729' entity='mgr.x' 2024-09-12T02:38:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:26 smithi016 ceph-mon[30925]: mgrmap e3: x(active, since 1.0075s) 2024-09-12T02:38:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:26 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3313610021' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-12T02:38:27.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:27.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-12T02:38:27.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-12T02:38:27.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-12T02:38:27.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-12T02:38:12:619852+0000", 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-12T02:38:27.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-12T02:38:12.620486+0000", 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-12T02:38:27.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc78f2640 1 Processor -- start 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc78f2640 1 -- start start 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc78f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc78f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc007c8a0 con 0x7f6dc007de20 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc568e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc568e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54646/0 (socket says 172.21.15.16:54646) 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.601+0000 7f6dc568e640 1 -- 172.21.15.16:0/2935811138 learned_addr learned my addr 172.21.15.16:0/2935811138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.602+0000 7f6dc568e640 1 -- 172.21.15.16:0/2935811138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc007ca20 con 0x7f6dc007de20 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.602+0000 7f6dc568e640 1 --2- 172.21.15.16:0/2935811138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009820 tx=0x7f6dbc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b504b3b63b551d20 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6db7fff640 1 -- 172.21.15.16:0/2935811138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dbc037070 con 0x7f6dc007de20 2024-09-12T02:38:27.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6db7fff640 1 -- 172.21.15.16:0/2935811138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6dbc032900 con 0x7f6dc007de20 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6db7fff640 1 -- 172.21.15.16:0/2935811138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dbc030850 con 0x7f6dc007de20 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 -- 172.21.15.16:0/2935811138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 msgr2=0x7f6dc007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/2935811138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009820 tx=0x7f6dbc02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 -- 172.21.15.16:0/2935811138 shutdown_connections 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/2935811138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc007c2d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 -- 172.21.15.16:0/2935811138 >> 172.21.15.16:0/2935811138 conn(0x7f6dc0103040 msgr2=0x7f6dc0105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 -- 172.21.15.16:0/2935811138 shutdown_connections 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.603+0000 7f6dc78f2640 1 -- 172.21.15.16:0/2935811138 wait complete. 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.604+0000 7f6dc78f2640 1 Processor -- start 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.604+0000 7f6dc78f2640 1 -- start start 2024-09-12T02:38:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.604+0000 7f6dc78f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.604+0000 7f6dc78f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc010de20 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.604+0000 7f6dc568e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc568e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54648/0 (socket says 172.21.15.16:54648) 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc568e640 1 -- 172.21.15.16:0/3313610021 learned_addr learned my addr 172.21.15.16:0/3313610021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc568e640 1 -- 172.21.15.16:0/3313610021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc019df20 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc568e640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc032ce0 tx=0x7f6dbc032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dbc042070 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6dbc0308c0 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc019e1b0 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dbc037040 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.605+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc01a0ea0 con 0x7f6dc007de20 2024-09-12T02:38:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.606+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) ==== 46170+0+0 (secure 0 0 0) 0x7f6dbc045070 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.606+0000 7f6db67fc640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 0x7f6dac03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.606+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc007c350 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.606+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6dbc06bdf0 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.606+0000 7f6dc4e8d640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 0x7f6dac03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.607+0000 7f6dc4e8d640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 0x7f6dac03b2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6db00098c0 tx=0x7f6db0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.612+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6dbc04b170 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.856+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6dc01a14f0 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.856+0000 7f6db67fc640 1 -- 172.21.15.16:0/3313610021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7f6dbc03cd50 con 0x7f6dc007de20 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 msgr2=0x7f6dac03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 0x7f6dac03b2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6db00098c0 tx=0x7f6db0006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 msgr2=0x7f6dc019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc032ce0 tx=0x7f6dbc032d10 comp rx=0 tx=0).stop 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 shutdown_connections 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f6dac038e10 0x7f6dac03b2d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 --2- 172.21.15.16:0/3313610021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc007de20 0x7f6dc019d9e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 >> 172.21.15.16:0/3313610021 conn(0x7f6dc0103040 msgr2=0x7f6dc0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 shutdown_connections 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:26.860+0000 7f6dc78f2640 1 -- 172.21.15.16:0/3313610021 wait complete. 2024-09-12T02:38:27.501 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-09-12T02:38:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:27 smithi016 ceph-mon[30925]: mgrmap e4: x(active, since 2s) 2024-09-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f294e11a640 1 Processor -- start 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f294e11a640 1 -- start start 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f294e11a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f294e11a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2948108200 con 0x7f2948107830 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f29477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f29477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54650/0 (socket says 172.21.15.16:54650) 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f29477fe640 1 -- 172.21.15.16:0/1900296557 learned_addr learned my addr 172.21.15.16:0/1900296557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.844+0000 7f29477fe640 1 -- 172.21.15.16:0/1900296557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29481089e0 con 0x7f2948107830 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.845+0000 7f29477fe640 1 --2- 172.21.15.16:0/1900296557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2938009820 tx=0x7f293802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ab16b0bc13852e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.845+0000 7f29467fc640 1 -- 172.21.15.16:0/1900296557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938037070 con 0x7f2948107830 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.845+0000 7f29467fc640 1 -- 172.21.15.16:0/1900296557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2938032900 con 0x7f2948107830 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.845+0000 7f29467fc640 1 -- 172.21.15.16:0/1900296557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938030850 con 0x7f2948107830 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 -- 172.21.15.16:0/1900296557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 msgr2=0x7f2948107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 --2- 172.21.15.16:0/1900296557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2938009820 tx=0x7f293802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 -- 172.21.15.16:0/1900296557 shutdown_connections 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 --2- 172.21.15.16:0/1900296557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f2948107c30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 -- 172.21.15.16:0/1900296557 >> 172.21.15.16:0/1900296557 conn(0x7f2948103040 msgr2=0x7f2948105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 -- 172.21.15.16:0/1900296557 shutdown_connections 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 -- 172.21.15.16:0/1900296557 wait complete. 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.846+0000 7f294e11a640 1 Processor -- start 2024-09-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f294e11a640 1 -- start start 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f294e11a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f294e11a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f294810bd90 con 0x7f2948107830 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f29477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f29477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54662/0 (socket says 172.21.15.16:54662) 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f29477fe640 1 -- 172.21.15.16:0/4018397794 learned_addr learned my addr 172.21.15.16:0/4018397794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f29477fe640 1 -- 172.21.15.16:0/4018397794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294819bd40 con 0x7f2948107830 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f29477fe640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2938009940 tx=0x7f2938032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.847+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938042070 con 0x7f2948107830 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.848+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f29380308c0 con 0x7f2948107830 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.848+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294819bfd0 con 0x7f2948107830 2024-09-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.848+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2938037040 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.848+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294819ecc0 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.848+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f2938045070 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.849+0000 7f2944ff9640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 0x7f293003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.849+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f293806bf70 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.849+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294810bf10 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.849+0000 7f2946ffd640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 0x7f293003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.849+0000 7f2946ffd640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 0x7f293003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f293c0098c0 tx=0x7f293c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:27.854+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2938045360 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.029+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2948107cb0 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.030+0000 7f2944ff9640 1 -- 172.21.15.16:0/4018397794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7f293806b990 con 0x7f2948107830 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 msgr2=0x7f293003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 0x7f293003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f293c0098c0 tx=0x7f293c006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 msgr2=0x7f294819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2938009940 tx=0x7f2938032ce0 comp rx=0 tx=0).stop 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 shutdown_connections 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f2930038f70 0x7f293003b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 --2- 172.21.15.16:0/4018397794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948107830 0x7f294819b800 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 >> 172.21.15.16:0/4018397794 conn(0x7f2948103040 msgr2=0x7f2948103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 shutdown_connections 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:28.032+0000 7f294e11a640 1 -- 172.21.15.16:0/4018397794 wait complete. 2024-09-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-09-12T02:38:29.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:28 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4018397794' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-12T02:38:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:29 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3873878510' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-12T02:38:30.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ignoring --setuser ceph since I am not root 2024-09-12T02:38:30.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ignoring --setgroup ceph since I am not root 2024-09-12T02:38:30.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.025+0000 7efe2770e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-12T02:38:30.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.084+0000 7efe2770e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f485a806640 1 Processor -- start 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f485a806640 1 -- start start 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f485a806640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f485a806640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f485407c8a0 con 0x7f485407de70 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f4853fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f4853fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54674/0 (socket says 172.21.15.16:54674) 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f4853fff640 1 -- 172.21.15.16:0/1181207246 learned_addr learned my addr 172.21.15.16:0/1181207246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:30.490 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.152+0000 7f4853fff640 1 -- 172.21.15.16:0/1181207246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485407ca20 con 0x7f485407de70 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.153+0000 7f4853fff640 1 --2- 172.21.15.16:0/1181207246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4844009820 tx=0x7f484402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6ca2044c51e3ef server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.153+0000 7f4852ffd640 1 -- 172.21.15.16:0/1181207246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4844037070 con 0x7f485407de70 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.153+0000 7f4852ffd640 1 -- 172.21.15.16:0/1181207246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4844032900 con 0x7f485407de70 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.153+0000 7f4852ffd640 1 -- 172.21.15.16:0/1181207246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4844030850 con 0x7f485407de70 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 -- 172.21.15.16:0/1181207246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 msgr2=0x7f485407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 --2- 172.21.15.16:0/1181207246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4844009820 tx=0x7f484402a270 comp rx=0 tx=0).stop 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 -- 172.21.15.16:0/1181207246 shutdown_connections 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 --2- 172.21.15.16:0/1181207246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485407c2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 -- 172.21.15.16:0/1181207246 >> 172.21.15.16:0/1181207246 conn(0x7f4854102ff0 msgr2=0x7f4854105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:30.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 -- 172.21.15.16:0/1181207246 shutdown_connections 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 -- 172.21.15.16:0/1181207246 wait complete. 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.154+0000 7f485a806640 1 Processor -- start 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f485a806640 1 -- start start 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f485a806640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f485a806640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f485410de50 con 0x7f485407de70 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f4853fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f4853fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54690/0 (socket says 172.21.15.16:54690) 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f4853fff640 1 -- 172.21.15.16:0/3873878510 learned_addr learned my addr 172.21.15.16:0/3873878510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f4853fff640 1 -- 172.21.15.16:0/3873878510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485419dfd0 con 0x7f485407de70 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.155+0000 7f4853fff640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f48440097f0 tx=0x7f4844030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4844042070 con 0x7f485407de70 2024-09-12T02:38:30.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485419e260 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4844032ca0 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48541a0f50 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4844037040 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.156+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f4844045070 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.157+0000 7f48517fa640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 0x7f483c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.157+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f485407c350 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.157+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f484406bed0 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.157+0000 7f48537fe640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 0x7f483c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.157+0000 7f48537fe640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 0x7f483c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48480098c0 tx=0x7f4848006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.162+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4844046e00 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.385+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f48541a15d0 con 0x7f485407de70 2024-09-12T02:38:30.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.877+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f484406e070 con 0x7f485407de70 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.879+0000 7f48517fa640 1 -- 172.21.15.16:0/3873878510 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f4844071020 con 0x7f485407de70 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 msgr2=0x7f483c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 0x7f483c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48480098c0 tx=0x7f4848006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 msgr2=0x7f485419da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f48440097f0 tx=0x7f4844030ec0 comp rx=0 tx=0).stop 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 shutdown_connections 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f483c038f20 0x7f483c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 --2- 172.21.15.16:0/3873878510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485407de70 0x7f485419da90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.880+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 >> 172.21.15.16:0/3873878510 conn(0x7f4854102ff0 msgr2=0x7f4854103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.881+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 shutdown_connections 2024-09-12T02:38:30.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:29.881+0000 7f485a806640 1 -- 172.21.15.16:0/3873878510 wait complete. 2024-09-12T02:38:30.608 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.238+0000 7efe2770e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-12T02:38:30.609 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.307+0000 7efe2770e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-12T02:38:30.609 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.461+0000 7efe2770e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:30.877 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.613+0000 7efe2770e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-12T02:38:31.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3873878510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-12T02:38:31.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-mon[30925]: mgrmap e5: x(active, since 5s) 2024-09-12T02:38:31.131 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:30.949+0000 7efe2770e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-12T02:38:31.131 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:31.015+0000 7efe2770e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-12T02:38:31.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:31.217+0000 7efe2770e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.883+0000 7f82a7b52640 1 Processor -- start 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a7b52640 1 -- start start 2024-09-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a7b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a7b52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0107080 con 0x7f82a01066b0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a58ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a58ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54710/0 (socket says 172.21.15.16:54710) 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.884+0000 7f82a58ee640 1 -- 172.21.15.16:0/3705710861 learned_addr learned my addr 172.21.15.16:0/3705710861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.885+0000 7f82a58ee640 1 -- 172.21.15.16:0/3705710861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a01078b0 con 0x7f82a01066b0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.885+0000 7f82a58ee640 1 --2- 172.21.15.16:0/3705710861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f829c009820 tx=0x7f829c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377d8c8cc93783fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a48ec640 1 -- 172.21.15.16:0/3705710861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f829c037070 con 0x7f82a01066b0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a48ec640 1 -- 172.21.15.16:0/3705710861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f829c032900 con 0x7f82a01066b0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a48ec640 1 -- 172.21.15.16:0/3705710861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f829c030850 con 0x7f82a01066b0 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a7b52640 1 -- 172.21.15.16:0/3705710861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 msgr2=0x7f82a0106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a7b52640 1 --2- 172.21.15.16:0/3705710861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f829c009820 tx=0x7f829c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a7b52640 1 -- 172.21.15.16:0/3705710861 shutdown_connections 2024-09-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a7b52640 1 --2- 172.21.15.16:0/3705710861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a0106ab0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.886+0000 7f82a7b52640 1 -- 172.21.15.16:0/3705710861 >> 172.21.15.16:0/3705710861 conn(0x7f82a0101e60 msgr2=0x7f82a0104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 -- 172.21.15.16:0/3705710861 shutdown_connections 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 -- 172.21.15.16:0/3705710861 wait complete. 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 Processor -- start 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 -- start start 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a7b52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82a010ac60 con 0x7f82a01066b0 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.887+0000 7f82a58ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a58ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54724/0 (socket says 172.21.15.16:54724) 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a58ee640 1 -- 172.21.15.16:0/1762489275 learned_addr learned my addr 172.21.15.16:0/1762489275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a58ee640 1 -- 172.21.15.16:0/1762489275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a019bd90 con 0x7f82a01066b0 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a58ee640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097f0 tx=0x7f829c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f829c037040 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f829c040070 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a019c020 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f829c03a390 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.888+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a019ed10 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.889+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f829c03a5e0 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.890+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a010ade0 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.890+0000 7f8296ffd640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 0x7f828c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.890+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f829c06c0f0 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.890+0000 7f82a50ed640 1 -- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 msgr2=0x7f828c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.890+0000 7f82a50ed640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 0x7f828c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:30.895+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f829c03a9d0 con 0x7f82a01066b0 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.091+0000 7f82a50ed640 1 -- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 msgr2=0x7f828c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.091+0000 7f82a50ed640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 0x7f828c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.117+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f82a002d9d0 con 0x7f82a01066b0 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.118+0000 7f8296ffd640 1 -- 172.21.15.16:0/1762489275 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f829c06be50 con 0x7f82a01066b0 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 msgr2=0x7f828c03b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 0x7f828c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 msgr2=0x7f82a019b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097f0 tx=0x7f829c030d80 comp rx=0 tx=0).stop 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 shutdown_connections 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f828c038fc0 0x7f828c03b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 --2- 172.21.15.16:0/1762489275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a01066b0 0x7f82a019b850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 >> 172.21.15.16:0/1762489275 conn(0x7f82a0101e60 msgr2=0x7f82a0102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 shutdown_connections 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:31.120+0000 7f82a7b52640 1 -- 172.21.15.16:0/1762489275 wait complete. 2024-09-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-09-12T02:38:32.133 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:31 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1762489275' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-12T02:38:32.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:32.171+0000 7efe2770e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-12T02:38:32.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:32.242+0000 7efe2770e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-12T02:38:33.108 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:32.988+0000 7efe2770e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-12T02:38:33.108 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.108+0000 7efe2770e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-12T02:38:33.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.256+0000 7efe2770e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:33.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.320+0000 7efe2770e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-12T02:38:33.774 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.476+0000 7efe2770e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-12T02:38:33.774 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.547+0000 7efe2770e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-12T02:38:33.774 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.609+0000 7efe2770e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-12T02:38:33.774 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.774+0000 7efe2770e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-12T02:38:34.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.841+0000 7efe2770e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-12T02:38:34.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.910+0000 7efe2770e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-12T02:38:34.059 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:33.996+0000 7efe2770e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:34.059 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:34.058+0000 7efe2770e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-12T02:38:34.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:34.119+0000 7efe2770e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-12T02:38:35.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:34.943+0000 7efe2770e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: Active manager daemon x restarted 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: Activating manager daemon x 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: osdmap e2: 0 total, 0 up, 0 in 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: mgrmap e6: x(active, starting, since 0.00324724s) 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-12T02:38:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: Manager daemon x is now available 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-12T02:38:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:34 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-12T02:38:36.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:35 smithi016 ceph-mon[30925]: Found migration_current of "None". Setting to last migration. 2024-09-12T02:38:36.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:35 smithi016 ceph-mon[30925]: mgrmap e7: x(active, since 1.0052s) 2024-09-12T02:38:36.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.132+0000 7f9ff4cea640 1 Processor -- start 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.132+0000 7f9ff4cea640 1 -- start start 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.132+0000 7f9ff4cea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.132+0000 7f9ff4cea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff007bdc0 con 0x7f9ff01082c0 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.133+0000 7f9fee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.133+0000 7f9fee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54728/0 (socket says 172.21.15.16:54728) 2024-09-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.133+0000 7f9fee59c640 1 -- 172.21.15.16:0/973401465 learned_addr learned my addr 172.21.15.16:0/973401465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.133+0000 7f9fee59c640 1 -- 172.21.15.16:0/973401465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff010abf0 con 0x7f9ff01082c0 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.134+0000 7f9fee59c640 1 --2- 172.21.15.16:0/973401465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009820 tx=0x7f9fe002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d428d74c3a18b8d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.134+0000 7f9fedd9b640 1 -- 172.21.15.16:0/973401465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe0037070 con 0x7f9ff01082c0 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.134+0000 7f9fedd9b640 1 -- 172.21.15.16:0/973401465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9fe0032900 con 0x7f9ff01082c0 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.134+0000 7f9fedd9b640 1 -- 172.21.15.16:0/973401465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe0030850 con 0x7f9ff01082c0 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 -- 172.21.15.16:0/973401465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 msgr2=0x7f9ff010a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/973401465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009820 tx=0x7f9fe002a270 comp rx=0 tx=0).stop 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 -- 172.21.15.16:0/973401465 shutdown_connections 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/973401465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff010a6b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 -- 172.21.15.16:0/973401465 >> 172.21.15.16:0/973401465 conn(0x7f9ff0101e80 msgr2=0x7f9ff01042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 -- 172.21.15.16:0/973401465 shutdown_connections 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.135+0000 7f9ff4cea640 1 -- 172.21.15.16:0/973401465 wait complete. 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9ff4cea640 1 Processor -- start 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9ff4cea640 1 -- start start 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9ff4cea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9ff4cea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff010bbd0 con 0x7f9ff01082c0 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9fee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9fee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54730/0 (socket says 172.21.15.16:54730) 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.136+0000 7f9fee59c640 1 -- 172.21.15.16:0/2164594485 learned_addr learned my addr 172.21.15.16:0/2164594485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9fee59c640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff019bd30 con 0x7f9ff01082c0 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9fee59c640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00097f0 tx=0x7f9fe0030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe0042070 con 0x7f9ff01082c0 2024-09-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9fe0032ca0 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff019bfc0 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe0037040 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.137+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff019ecb0 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f9fe0045070 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe77fe640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fdc03bb90 con 0x7f9fdc038fc0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe7fff640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 msgr2=0x7f9fdc03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9fe006bf30 con 0x7f9ff01082c0 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.138+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.339+0000 7f9fe7fff640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 msgr2=0x7f9fdc03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.339+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.740+0000 7f9fe7fff640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 msgr2=0x7f9fdc03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:32.740+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:33.541+0000 7f9fe7fff640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 msgr2=0x7f9fdc03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2594473615 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:33.541+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:34.946+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f9fe006a9f0 con 0x7f9ff01082c0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:34.947+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 msgr2=0x7f9fdc03b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:34.947+0000 7f9fe77fe640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.947+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f9fe003c720 con 0x7f9ff01082c0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.948+0000 7f9fe77fe640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 0x7f9fdc03ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.948+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fe003d940 con 0x7f9fdc03c8f0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.948+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 0x7f9fdc03ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.949+0000 7f9fe7fff640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 0x7f9fdc03ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8003980 tx=0x7f9fd80092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.949+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9fe003d940 con 0x7f9fdc03c8f0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.956+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9ff0108720 con 0x7f9fdc03c8f0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9fe77fe640 1 -- 172.21.15.16:0/2164594485 <== mgr.14118 v2:172.21.15.16:6800/744312028 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9ff0108720 con 0x7f9fdc03c8f0 2024-09-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 msgr2=0x7f9fdc03ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 0x7f9fdc03ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8003980 tx=0x7f9fd80092b0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 msgr2=0x7f9ff019b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00097f0 tx=0x7f9fe0030ec0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 shutdown_connections 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9fdc03c8f0 0x7f9fdc03ece0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:6800/2594473615,v1:172.21.15.16:6801/2594473615] conn(0x7f9fdc038fc0 0x7f9fdc03b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 --2- 172.21.15.16:0/2164594485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff01082c0 0x7f9ff019b7f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 >> 172.21.15.16:0/2164594485 conn(0x7f9ff0101e80 msgr2=0x7f9ff0102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 shutdown_connections 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:35.957+0000 7f9ff4cea640 1 -- 172.21.15.16:0/2164594485 wait complete. 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-09-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-09-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.954+0000 7f1b88ef2640 1 Processor -- start 2024-09-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.954+0000 7f1b88ef2640 1 -- start start 2024-09-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b88ef2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b88ef2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84108650 con 0x7f1b84107c80 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39198/0 (socket says 172.21.15.16:39198) 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b8259c640 1 -- 172.21.15.16:0/538429369 learned_addr learned my addr 172.21.15.16:0/538429369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.955+0000 7f1b8259c640 1 -- 172.21.15.16:0/538429369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841068e0 con 0x7f1b84107c80 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b8259c640 1 --2- 172.21.15.16:0/538429369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009820 tx=0x7f1b7402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380392a705149607 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b81d9b640 1 -- 172.21.15.16:0/538429369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b74037070 con 0x7f1b84107c80 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b81d9b640 1 -- 172.21.15.16:0/538429369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b74032900 con 0x7f1b84107c80 2024-09-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b81d9b640 1 -- 172.21.15.16:0/538429369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b74030850 con 0x7f1b84107c80 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b88ef2640 1 -- 172.21.15.16:0/538429369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 msgr2=0x7f1b84108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/538429369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009820 tx=0x7f1b7402a270 comp rx=0 tx=0).stop 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b88ef2640 1 -- 172.21.15.16:0/538429369 shutdown_connections 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.956+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/538429369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b84108080 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.957+0000 7f1b88ef2640 1 -- 172.21.15.16:0/538429369 >> 172.21.15.16:0/538429369 conn(0x7f1b84101620 msgr2=0x7f1b84103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.957+0000 7f1b88ef2640 1 -- 172.21.15.16:0/538429369 shutdown_connections 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.957+0000 7f1b88ef2640 1 -- 172.21.15.16:0/538429369 wait complete. 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.957+0000 7f1b88ef2640 1 Processor -- start 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b88ef2640 1 -- start start 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b88ef2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b88ef2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410aca0 con 0x7f1b84107c80 2024-09-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39212/0 (socket says 172.21.15.16:39212) 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b8259c640 1 -- 172.21.15.16:0/1717710506 learned_addr learned my addr 172.21.15.16:0/1717710506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b8259c640 1 -- 172.21.15.16:0/1717710506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8419bd90 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b8259c640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009940 tx=0x7f1b740326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.958+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b74042070 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.959+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8419c020 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.959+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b74032be0 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.959+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b74037040 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.959+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8419ed10 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.960+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b84108100 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.960+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f1b74045070 con 0x7f1b84107c80 2024-09-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.960+0000 7f1b7bfff640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 0x7f1b7003b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.960+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1b7406bd50 con 0x7f1b84107c80 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.960+0000 7f1b79d9b640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 0x7f1b7003b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.961+0000 7f1b79d9b640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 0x7f1b7003b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0098c0 tx=0x7f1b6c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:36.965+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b740371e0 con 0x7f1b84107c80 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.155+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f1b841028d0 con 0x7f1b70038e60 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.160+0000 7f1b7bfff640 1 -- 172.21.15.16:0/1717710506 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1b841028d0 con 0x7f1b70038e60 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 msgr2=0x7f1b7003b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 0x7f1b7003b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0098c0 tx=0x7f1b6c006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 msgr2=0x7f1b8419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009940 tx=0x7f1b740326e0 comp rx=0 tx=0).stop 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 shutdown_connections 2024-09-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f1b70038e60 0x7f1b7003b320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 --2- 172.21.15.16:0/1717710506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b84107c80 0x7f1b8419b850 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.162+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 >> 172.21.15.16:0/1717710506 conn(0x7f1b84101620 msgr2=0x7f1b841021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.163+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 shutdown_connections 2024-09-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:37.163+0000 7f1b88ef2640 1 -- 172.21.15.16:0/1717710506 wait complete. 2024-09-12T02:38:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:37.911+0000 7efd8ea28640 -1 log_channel(cephadm) log [ERR] : [12/Sep/2024:02:38:37] ENGINE Error in HTTPServer.serve 2024-09-12T02:38:37.997 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: Traceback (most recent call last): 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._connections.run(self.expiration_interval) 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._run(expiration_interval) 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: new_conn = self._from_server_socket(self.server.socket) 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: s = self.context.wrap_socket( 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-12T02:38:37.998 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: return self.sslsocket_class._create( 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self.do_handshake() 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._sslobj.do_handshake() 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-12T02:38:37.999 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:38.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:38.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: [12/Sep/2024:02:38:37] ENGINE Bus STARTING 2024-09-12T02:38:38.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: mgrmap e8: x(active, since 2s) 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: [12/Sep/2024:02:38:37] ENGINE Serving on http://172.21.15.16:8765 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: [12/Sep/2024:02:38:37] ENGINE Serving on https://172.21.15.16:7150 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: [12/Sep/2024:02:38:37] ENGINE Bus STARTED 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:38 smithi016 ceph-mon[30925]: [12/Sep/2024:02:38:37] ENGINE Error in HTTPServer.serve 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-12T02:38:38.901 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-12T02:38:38.902 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-12T02:38:38.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f03823b3640 1 Processor -- start 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f03823b3640 1 -- start start 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f03823b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f03823b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f037c07c880 con 0x7f037c07de50 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f037bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f037bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39220/0 (socket says 172.21.15.16:39220) 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.076+0000 7f037bfff640 1 -- 172.21.15.16:0/2933125789 learned_addr learned my addr 172.21.15.16:0/2933125789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.077+0000 7f037bfff640 1 -- 172.21.15.16:0/2933125789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c07ca00 con 0x7f037c07de50 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.077+0000 7f037bfff640 1 --2- 172.21.15.16:0/2933125789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f036c009820 tx=0x7f036c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c537fc5a28545acd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.077+0000 7f037affd640 1 -- 172.21.15.16:0/2933125789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f036c037070 con 0x7f037c07de50 2024-09-12T02:38:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.077+0000 7f037affd640 1 -- 172.21.15.16:0/2933125789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f036c032900 con 0x7f037c07de50 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f037affd640 1 -- 172.21.15.16:0/2933125789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f036c030850 con 0x7f037c07de50 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 -- 172.21.15.16:0/2933125789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 msgr2=0x7f037c07c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 --2- 172.21.15.16:0/2933125789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f036c009820 tx=0x7f036c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 -- 172.21.15.16:0/2933125789 shutdown_connections 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 --2- 172.21.15.16:0/2933125789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c07c2b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 -- 172.21.15.16:0/2933125789 >> 172.21.15.16:0/2933125789 conn(0x7f037c103000 msgr2=0x7f037c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 -- 172.21.15.16:0/2933125789 shutdown_connections 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.078+0000 7f03823b3640 1 -- 172.21.15.16:0/2933125789 wait complete. 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f03823b3640 1 Processor -- start 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f03823b3640 1 -- start start 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f03823b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f03823b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f037c10de60 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f037bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f037bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39222/0 (socket says 172.21.15.16:39222) 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f037bfff640 1 -- 172.21.15.16:0/600979373 learned_addr learned my addr 172.21.15.16:0/600979373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.079+0000 7f037bfff640 1 -- 172.21.15.16:0/600979373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c19df30 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f037bfff640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f036c0097f0 tx=0x7f036c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f036c037040 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f036c040070 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037c19e1c0 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f036c03a390 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.080+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037c1a0eb0 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f036c03a5e0 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037c07c330 con 0x7f037c07de50 2024-09-12T02:38:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f03797fa640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 0x7f036403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f036c06c5a0 con 0x7f037c07de50 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f037b7fe640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 0x7f036403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.081+0000 7f037b7fe640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 0x7f036403b480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03700098c0 tx=0x7f0370006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.086+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f036c043760 con 0x7f037c07de50 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.263+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f037c104240 con 0x7f0364038fc0 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.264+0000 7f03797fa640 1 -- 172.21.15.16:0/600979373 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f037c104240 con 0x7f0364038fc0 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 msgr2=0x7f036403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 0x7f036403b480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03700098c0 tx=0x7f0370006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 msgr2=0x7f037c19d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f036c0097f0 tx=0x7f036c030d80 comp rx=0 tx=0).stop 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 shutdown_connections 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f0364038fc0 0x7f036403b480 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 --2- 172.21.15.16:0/600979373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f037c07de50 0x7f037c19d9f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:38.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.266+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 >> 172.21.15.16:0/600979373 conn(0x7f037c103000 msgr2=0x7f037c103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:38.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.267+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 shutdown_connections 2024-09-12T02:38:38.912 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:38.267+0000 7f03823b3640 1 -- 172.21.15.16:0/600979373 wait complete. 2024-09-12T02:38:38.912 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-09-12T02:38:39.788 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: Generating public/private rsa key pair. 2024-09-12T02:38:39.788 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: Your identification has been saved in /tmp/tmpggiiypkk/key 2024-09-12T02:38:39.788 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: Your public key has been saved in /tmp/tmpggiiypkk/key.pub 2024-09-12T02:38:39.788 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: The key fingerprint is: 2024-09-12T02:38:39.788 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: SHA256:ARWEX2ZPi25mYEo2EIbWs9qxnhhJIZfl9jlHfZE9Cw0 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: The key's randomart image is: 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: +---[RSA 3072]----+ 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | =+..++. E= | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: |. *o+ .. .+ +.+ | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | + .o+ .o+.+.o o | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | ..o.+o+...o . | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | . + =++So | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | + o .o = | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | + . + | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | . o | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: | | 2024-09-12T02:38:39.789 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: +----[SHA256]-----+ 2024-09-12T02:38:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-mon[30925]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:39 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.328+0000 7f29ffa5d640 1 Processor -- start 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29ffa5d640 1 -- start start 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29ffa5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29ffa5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29f81081a0 con 0x7f29f81077d0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29fd7f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29fd7f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39236/0 (socket says 172.21.15.16:39236) 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29fd7f9640 1 -- 172.21.15.16:0/3177967227 learned_addr learned my addr 172.21.15.16:0/3177967227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.329+0000 7f29fd7f9640 1 -- 172.21.15.16:0/3177967227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f81089d0 con 0x7f29f81077d0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.330+0000 7f29fd7f9640 1 --2- 172.21.15.16:0/3177967227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009820 tx=0x7f29f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c4279c58faf930 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.330+0000 7f29effff640 1 -- 172.21.15.16:0/3177967227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4037070 con 0x7f29f81077d0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.330+0000 7f29effff640 1 -- 172.21.15.16:0/3177967227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f29f4032900 con 0x7f29f81077d0 2024-09-12T02:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.330+0000 7f29effff640 1 -- 172.21.15.16:0/3177967227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4030850 con 0x7f29f81077d0 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3177967227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 msgr2=0x7f29f8107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3177967227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009820 tx=0x7f29f402a270 comp rx=0 tx=0).stop 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3177967227 shutdown_connections 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3177967227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f8107bd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3177967227 >> 172.21.15.16:0/3177967227 conn(0x7f29f8102f40 msgr2=0x7f29f81053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3177967227 shutdown_connections 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.331+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3177967227 wait complete. 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29ffa5d640 1 Processor -- start 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29ffa5d640 1 -- start start 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29ffa5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29ffa5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29f810bd80 con 0x7f29f81077d0 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29fd7f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29fd7f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39244/0 (socket says 172.21.15.16:39244) 2024-09-12T02:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29fd7f9640 1 -- 172.21.15.16:0/3416488174 learned_addr learned my addr 172.21.15.16:0/3416488174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.332+0000 7f29fd7f9640 1 -- 172.21.15.16:0/3416488174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f819dfb0 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29fd7f9640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009940 tx=0x7f29f40326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4042070 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f29f4032be0 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f819e240 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4037040 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.333+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f81a0f30 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.334+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f29f4045070 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.334+0000 7f29ee7fc640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 0x7f29e403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.334+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f29f406c060 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.334+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f810bf00 con 0x7f29f81077d0 2024-09-12T02:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.334+0000 7f29fcff8640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 0x7f29e403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.335+0000 7f29fcff8640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 0x7f29e403b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29e80098c0 tx=0x7f29e8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.339+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f29f4046e00 con 0x7f29f81077d0 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.516+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f29f8104210 con 0x7f29e4038fc0 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.666+0000 7f29ee7fc640 1 -- 172.21.15.16:0/3416488174 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f29f8104210 con 0x7f29e4038fc0 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.668+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 msgr2=0x7f29e403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.668+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 0x7f29e403b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f29e80098c0 tx=0x7f29e8006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.668+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 msgr2=0x7f29f819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.668+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009940 tx=0x7f29f40326e0 comp rx=0 tx=0).stop 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 shutdown_connections 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f29e4038fc0 0x7f29e403b480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 --2- 172.21.15.16:0/3416488174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f81077d0 0x7f29f819da70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 >> 172.21.15.16:0/3416488174 conn(0x7f29f8102f40 msgr2=0x7f29f8103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 shutdown_connections 2024-09-12T02:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:39.669+0000 7f29ffa5d640 1 -- 172.21.15.16:0/3416488174 wait complete. 2024-09-12T02:38:40.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:40 smithi016 ceph-mon[30925]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:40.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:40 smithi016 ceph-mon[30925]: Generating ssh key... 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba47347640 1 Processor -- start 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba47347640 1 -- start start 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba47347640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba47347640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba40108200 con 0x7fba40107830 2024-09-12T02:38:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba450e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba450e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39256/0 (socket says 172.21.15.16:39256) 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba450e3640 1 -- 172.21.15.16:0/1453180417 learned_addr learned my addr 172.21.15.16:0/1453180417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.649+0000 7fba450e3640 1 -- 172.21.15.16:0/1453180417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba401089e0 con 0x7fba40107830 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.650+0000 7fba450e3640 1 --2- 172.21.15.16:0/1453180417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009820 tx=0x7fba3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b575e487c8086ea3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.650+0000 7fba37fff640 1 -- 172.21.15.16:0/1453180417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba3c037070 con 0x7fba40107830 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.650+0000 7fba37fff640 1 -- 172.21.15.16:0/1453180417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fba3c032900 con 0x7fba40107830 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.650+0000 7fba37fff640 1 -- 172.21.15.16:0/1453180417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba3c030850 con 0x7fba40107830 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 -- 172.21.15.16:0/1453180417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 msgr2=0x7fba40107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 --2- 172.21.15.16:0/1453180417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009820 tx=0x7fba3c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 -- 172.21.15.16:0/1453180417 shutdown_connections 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 --2- 172.21.15.16:0/1453180417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba40107c30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 -- 172.21.15.16:0/1453180417 >> 172.21.15.16:0/1453180417 conn(0x7fba40103040 msgr2=0x7fba40105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 -- 172.21.15.16:0/1453180417 shutdown_connections 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.651+0000 7fba47347640 1 -- 172.21.15.16:0/1453180417 wait complete. 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba47347640 1 Processor -- start 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba47347640 1 -- start start 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba47347640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba47347640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba4010bd90 con 0x7fba40107830 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba450e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba450e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39266/0 (socket says 172.21.15.16:39266) 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba450e3640 1 -- 172.21.15.16:0/345786946 learned_addr learned my addr 172.21.15.16:0/345786946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba450e3640 1 -- 172.21.15.16:0/345786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4019bd40 con 0x7fba40107830 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.652+0000 7fba450e3640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009940 tx=0x7fba3c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.653+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba3c042070 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.653+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fba3c0308c0 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.653+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4019bfd0 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.653+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba3c037040 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.653+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4019ecc0 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fba3c045070 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4010bf10 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba367fc640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 0x7fba2c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fba3c06c030 con 0x7fba40107830 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba448e2640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 0x7fba2c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.654+0000 7fba448e2640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 0x7fba2c03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba300098c0 tx=0x7fba30006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.659+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba3c03cd90 con 0x7fba40107830 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.836+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fba401042d0 con 0x7fba2c038f70 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.837+0000 7fba367fc640 1 -- 172.21.15.16:0/345786946 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fba401042d0 con 0x7fba2c038f70 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.839+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 msgr2=0x7fba2c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.839+0000 7fba47347640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 0x7fba2c03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba300098c0 tx=0x7fba30006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 msgr2=0x7fba4019b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009940 tx=0x7fba3c032ce0 comp rx=0 tx=0).stop 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 shutdown_connections 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fba2c038f70 0x7fba2c03b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 --2- 172.21.15.16:0/345786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba40107830 0x7fba4019b800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 >> 172.21.15.16:0/345786946 conn(0x7fba40103040 msgr2=0x7fba40103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 shutdown_connections 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:40.840+0000 7fba47347640 1 -- 172.21.15.16:0/345786946 wait complete. 2024-09-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-09-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-09-12T02:38:41.897 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:41 smithi016 ceph-mon[30925]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:42.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:42 smithi016 ceph-mon[30925]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:43 smithi016 ceph-mon[30925]: Deploying cephadm binary to smithi016 2024-09-12T02:38:46.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.705+0000 7f32a8181640 1 Processor -- start 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.705+0000 7f32a8181640 1 -- start start 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.705+0000 7f32a8181640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.705+0000 7f32a8181640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32a007c860 con 0x7f32a007de30 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.706+0000 7f32a5f1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.706+0000 7f32a5f1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39274/0 (socket says 172.21.15.16:39274) 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.706+0000 7f32a5f1d640 1 -- 172.21.15.16:0/898888313 learned_addr learned my addr 172.21.15.16:0/898888313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.706+0000 7f32a5f1d640 1 -- 172.21.15.16:0/898888313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a007c9e0 con 0x7f32a007de30 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.706+0000 7f32a5f1d640 1 --2- 172.21.15.16:0/898888313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3298009820 tx=0x7f329802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc17d8fb12f03954 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a4f1b640 1 -- 172.21.15.16:0/898888313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3298037070 con 0x7f32a007de30 2024-09-12T02:38:46.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a4f1b640 1 -- 172.21.15.16:0/898888313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3298032900 con 0x7f32a007de30 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a4f1b640 1 -- 172.21.15.16:0/898888313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3298030850 con 0x7f32a007de30 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 -- 172.21.15.16:0/898888313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 msgr2=0x7f32a007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 --2- 172.21.15.16:0/898888313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3298009820 tx=0x7f329802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 -- 172.21.15.16:0/898888313 shutdown_connections 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 --2- 172.21.15.16:0/898888313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a007c290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 -- 172.21.15.16:0/898888313 >> 172.21.15.16:0/898888313 conn(0x7f32a0102ff0 msgr2=0x7f32a0105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 -- 172.21.15.16:0/898888313 shutdown_connections 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.707+0000 7f32a8181640 1 -- 172.21.15.16:0/898888313 wait complete. 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a8181640 1 Processor -- start 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a8181640 1 -- start start 2024-09-12T02:38:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a8181640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a8181640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32a010de30 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a5f1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a5f1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39286/0 (socket says 172.21.15.16:39286) 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.708+0000 7f32a5f1d640 1 -- 172.21.15.16:0/715372318 learned_addr learned my addr 172.21.15.16:0/715372318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f32a5f1d640 1 -- 172.21.15.16:0/715372318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a019df80 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f32a5f1d640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3298031000 tx=0x7f32980326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3298042070 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3298032be0 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a019e210 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3298037040 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.709+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a01a0f00 con 0x7f32a007de30 2024-09-12T02:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.710+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3298045070 con 0x7f32a007de30 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.710+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a007c310 con 0x7f32a007de30 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.710+0000 7f329effd640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 0x7f328803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.710+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f329806c0f0 con 0x7f32a007de30 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.710+0000 7f32a571c640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 0x7f328803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.711+0000 7f32a571c640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 0x7f328803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32940098c0 tx=0x7f3294006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.715+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3298047440 con 0x7f32a007de30 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:41.897+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) -- 0x7f32a0104330 con 0x7f3288038fc0 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.851+0000 7f329effd640 1 -- 172.21.15.16:0/715372318 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f32a0104330 con 0x7f3288038fc0 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 msgr2=0x7f328803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 0x7f328803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32940098c0 tx=0x7f3294006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:46.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 msgr2=0x7f32a019da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3298031000 tx=0x7f32980326e0 comp rx=0 tx=0).stop 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 shutdown_connections 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f3288038fc0 0x7f328803b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 --2- 172.21.15.16:0/715372318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32a007de30 0x7f32a019da40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 >> 172.21.15.16:0/715372318 conn(0x7f32a0102ff0 msgr2=0x7f32a0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 shutdown_connections 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:45.854+0000 7f32a8181640 1 -- 172.21.15.16:0/715372318 wait complete. 2024-09-12T02:38:46.558 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-09-12T02:38:47.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:46 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:46 smithi016 ceph-mon[30925]: Added host smithi016 2024-09-12T02:38:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:46 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2cacc51640 1 Processor -- start 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2cacc51640 1 -- start start 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2cacc51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2cacc51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca807c8c0 con 0x7f2ca807de40 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2ca659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2ca659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35734/0 (socket says 172.21.15.16:35734) 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.959+0000 7f2ca659c640 1 -- 172.21.15.16:0/706177542 learned_addr learned my addr 172.21.15.16:0/706177542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.960+0000 7f2ca659c640 1 -- 172.21.15.16:0/706177542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca807ca40 con 0x7f2ca807de40 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.960+0000 7f2ca659c640 1 --2- 172.21.15.16:0/706177542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009820 tx=0x7f2c9802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de821528e278bbc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2ca559a640 1 -- 172.21.15.16:0/706177542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c98037070 con 0x7f2ca807de40 2024-09-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2ca559a640 1 -- 172.21.15.16:0/706177542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c98032900 con 0x7f2ca807de40 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2ca559a640 1 -- 172.21.15.16:0/706177542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c98030850 con 0x7f2ca807de40 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 -- 172.21.15.16:0/706177542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 msgr2=0x7f2ca807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 --2- 172.21.15.16:0/706177542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009820 tx=0x7f2c9802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 -- 172.21.15.16:0/706177542 shutdown_connections 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 --2- 172.21.15.16:0/706177542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca807c2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 -- 172.21.15.16:0/706177542 >> 172.21.15.16:0/706177542 conn(0x7f2ca8103040 msgr2=0x7f2ca8105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 -- 172.21.15.16:0/706177542 shutdown_connections 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.961+0000 7f2cacc51640 1 -- 172.21.15.16:0/706177542 wait complete. 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2cacc51640 1 Processor -- start 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2cacc51640 1 -- start start 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2cacc51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2cacc51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca810de00 con 0x7f2ca807de40 2024-09-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2ca659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2ca659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35740/0 (socket says 172.21.15.16:35740) 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.962+0000 7f2ca659c640 1 -- 172.21.15.16:0/1371374397 learned_addr learned my addr 172.21.15.16:0/1371374397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2ca659c640 1 -- 172.21.15.16:0/1371374397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca819def0 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2ca659c640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c98032ce0 tx=0x7f2c98032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c98042070 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca819e180 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c980308c0 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c98037040 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.963+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca81a0e70 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.964+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2c98045070 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.964+0000 7f2c977fe640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 0x7f2c9003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.964+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2c9806c040 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.964+0000 7f2ca5d9b640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 0x7f2c9003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.964+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca807c370 con 0x7f2ca807de40 2024-09-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.965+0000 7f2ca5d9b640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 0x7f2c9003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0098c0 tx=0x7f2c9c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:46.969+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2c98045520 con 0x7f2ca807de40 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.174+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2ca81042b0 con 0x7f2c90038f70 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.179+0000 7f2c977fe640 1 -- 172.21.15.16:0/1371374397 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2ca81042b0 con 0x7f2c90038f70 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 msgr2=0x7f2c9003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 0x7f2c9003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0098c0 tx=0x7f2c9c006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 msgr2=0x7f2ca819d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c98032ce0 tx=0x7f2c98032d10 comp rx=0 tx=0).stop 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 shutdown_connections 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2c90038f70 0x7f2c9003b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 --2- 172.21.15.16:0/1371374397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ca807de40 0x7f2ca819d9b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 >> 172.21.15.16:0/1371374397 conn(0x7f2ca8103040 msgr2=0x7f2ca8103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.181+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 shutdown_connections 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:47.182+0000 7f2cacc51640 1 -- 172.21.15.16:0/1371374397 wait complete. 2024-09-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-09-12T02:38:48.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:48 smithi016 ceph-mon[30925]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:48.250 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:48 smithi016 ceph-mon[30925]: Saving service mon spec with placement count:5 2024-09-12T02:38:48.251 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:48 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:49.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:49 smithi016 ceph-mon[30925]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:38:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:49 smithi016 ceph-mon[30925]: Saving service mgr spec with placement count:2 2024-09-12T02:38:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:49 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:49 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:50.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.414+0000 7f50994c5640 1 Processor -- start 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.414+0000 7f50994c5640 1 -- start start 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f50994c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f50994c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f509407c7d0 con 0x7f509407dde0 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f5092ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f5092ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35754/0 (socket says 172.21.15.16:35754) 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f5092ffd640 1 -- 172.21.15.16:0/2741095658 learned_addr learned my addr 172.21.15.16:0/2741095658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.415+0000 7f5092ffd640 1 -- 172.21.15.16:0/2741095658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509407c950 con 0x7f509407dde0 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.416+0000 7f5092ffd640 1 --2- 172.21.15.16:0/2741095658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f508c009820 tx=0x7f508c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcaa18a3940037a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.416+0000 7f5091ffb640 1 -- 172.21.15.16:0/2741095658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f508c037070 con 0x7f509407dde0 2024-09-12T02:38:50.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.416+0000 7f5091ffb640 1 -- 172.21.15.16:0/2741095658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f508c032900 con 0x7f509407dde0 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.416+0000 7f5091ffb640 1 -- 172.21.15.16:0/2741095658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f508c030850 con 0x7f509407dde0 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 -- 172.21.15.16:0/2741095658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 msgr2=0x7f509407c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 --2- 172.21.15.16:0/2741095658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f508c009820 tx=0x7f508c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 -- 172.21.15.16:0/2741095658 shutdown_connections 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 --2- 172.21.15.16:0/2741095658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509407c290 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 -- 172.21.15.16:0/2741095658 >> 172.21.15.16:0/2741095658 conn(0x7f5094103040 msgr2=0x7f5094105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 -- 172.21.15.16:0/2741095658 shutdown_connections 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.417+0000 7f50994c5640 1 -- 172.21.15.16:0/2741095658 wait complete. 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f50994c5640 1 Processor -- start 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f50994c5640 1 -- start start 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f50994c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f50994c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f509410de60 con 0x7f509407dde0 2024-09-12T02:38:50.112 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f5092ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f5092ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35766/0 (socket says 172.21.15.16:35766) 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f5092ffd640 1 -- 172.21.15.16:0/3357978614 learned_addr learned my addr 172.21.15.16:0/3357978614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.418+0000 7f5092ffd640 1 -- 172.21.15.16:0/3357978614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509419dee0 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f5092ffd640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f508c0097f0 tx=0x7f508c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f508c037040 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509419e170 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f508c040070 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f508c03a390 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.419+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50941a0e60 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.420+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f508c03a630 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.420+0000 7f508bfff640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 0x7f508003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.420+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f508c06c510 con 0x7f509407dde0 2024-09-12T02:38:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.420+0000 7f50927fc640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 0x7f508003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.420+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509407c310 con 0x7f509407dde0 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.421+0000 7f50927fc640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 0x7f508003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f507c0098c0 tx=0x7f507c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.425+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f508c04a200 con 0x7f509407dde0 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.612+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f50941042b0 con 0x7f5080038f70 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.616+0000 7f508bfff640 1 -- 172.21.15.16:0/3357978614 <== mgr.14118 v2:172.21.15.16:6800/744312028 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f50941042b0 con 0x7f5080038f70 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 msgr2=0x7f508003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 0x7f508003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f507c0098c0 tx=0x7f507c006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 msgr2=0x7f509419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f508c0097f0 tx=0x7f508c030d80 comp rx=0 tx=0).stop 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 shutdown_connections 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f5080038f70 0x7f508003b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 --2- 172.21.15.16:0/3357978614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509407dde0 0x7f509419d9a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:50.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 >> 172.21.15.16:0/3357978614 conn(0x7f5094103040 msgr2=0x7f5094103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:50.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 shutdown_connections 2024-09-12T02:38:50.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:48.619+0000 7f50994c5640 1 -- 172.21.15.16:0/3357978614 wait complete. 2024-09-12T02:38:51.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:51 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:51.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:51 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4068575691' entity='client.admin' 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43ced0640 1 Processor -- start 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43ced0640 1 -- start start 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43ced0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43ced0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4381070f0 con 0x7ff438106720 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35770/0 (socket says 172.21.15.16:35770) 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.514+0000 7ff43659c640 1 -- 172.21.15.16:0/4265501444 learned_addr learned my addr 172.21.15.16:0/4265501444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.515+0000 7ff43659c640 1 -- 172.21.15.16:0/4265501444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff438107920 con 0x7ff438106720 2024-09-12T02:38:52.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.515+0000 7ff43659c640 1 --2- 172.21.15.16:0/4265501444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff428009820 tx=0x7ff42802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0155f6db61a56c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.515+0000 7ff43559a640 1 -- 172.21.15.16:0/4265501444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428037070 con 0x7ff438106720 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.515+0000 7ff43559a640 1 -- 172.21.15.16:0/4265501444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff428032920 con 0x7ff438106720 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43559a640 1 -- 172.21.15.16:0/4265501444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428030810 con 0x7ff438106720 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 -- 172.21.15.16:0/4265501444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 msgr2=0x7ff438106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4265501444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff428009820 tx=0x7ff42802a270 comp rx=0 tx=0).stop 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 -- 172.21.15.16:0/4265501444 shutdown_connections 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4265501444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff438106b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 -- 172.21.15.16:0/4265501444 >> 172.21.15.16:0/4265501444 conn(0x7ff438101f10 msgr2=0x7ff438104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 -- 172.21.15.16:0/4265501444 shutdown_connections 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.516+0000 7ff43ced0640 1 -- 172.21.15.16:0/4265501444 wait complete. 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43ced0640 1 Processor -- start 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43ced0640 1 -- start start 2024-09-12T02:38:52.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43ced0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43ced0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff43810acd0 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35778/0 (socket says 172.21.15.16:35778) 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.517+0000 7ff43659c640 1 -- 172.21.15.16:0/4068575691 learned_addr learned my addr 172.21.15.16:0/4068575691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff43659c640 1 -- 172.21.15.16:0/4068575691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43819bdd0 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff43659c640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4280097f0 tx=0x7ff4280327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428037040 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff428030cc0 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43819c060 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff42803a640 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.518+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43819ed50 con 0x7ff438106720 2024-09-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ff42803a940 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff4277fe640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 0x7ff42003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff42806c510 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff43810ae50 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff435d9b640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 0x7ff42003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.519+0000 7ff435d9b640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 0x7ff42003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098c0 tx=0x7ff42c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.524+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff42803ba40 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.706+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7ff43819f290 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.709+0000 7ff4277fe640 1 -- 172.21.15.16:0/4068575691 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7ff42802fd60 con 0x7ff438106720 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 msgr2=0x7ff42003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 0x7ff42003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098c0 tx=0x7ff42c006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 msgr2=0x7ff43819b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4280097f0 tx=0x7ff4280327b0 comp rx=0 tx=0).stop 2024-09-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 shutdown_connections 2024-09-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7ff420038f70 0x7ff42003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 --2- 172.21.15.16:0/4068575691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff438106720 0x7ff43819b890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 >> 172.21.15.16:0/4068575691 conn(0x7ff438101f10 msgr2=0x7ff438102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 shutdown_connections 2024-09-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:50.711+0000 7ff43ced0640 1 -- 172.21.15.16:0/4068575691 wait complete. 2024-09-12T02:38:52.635 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:52 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:53.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95b5350640 1 Processor -- start 2024-09-12T02:38:53.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95b5350640 1 -- start start 2024-09-12T02:38:53.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95b5350640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:53.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95b5350640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95b007be00 con 0x7f95b01082f0 2024-09-12T02:38:53.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35782/0 (socket says 172.21.15.16:35782) 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95aeffd640 1 -- 172.21.15.16:0/4134655964 learned_addr learned my addr 172.21.15.16:0/4134655964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.448+0000 7f95aeffd640 1 -- 172.21.15.16:0/4134655964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b010ac20 con 0x7f95b01082f0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.449+0000 7f95aeffd640 1 --2- 172.21.15.16:0/4134655964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009820 tx=0x7f95a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb9432270e8bf9f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.449+0000 7f95adffb640 1 -- 172.21.15.16:0/4134655964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95a0037070 con 0x7f95b01082f0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.449+0000 7f95adffb640 1 -- 172.21.15.16:0/4134655964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a0032920 con 0x7f95b01082f0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.449+0000 7f95adffb640 1 -- 172.21.15.16:0/4134655964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95a0030810 con 0x7f95b01082f0 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 -- 172.21.15.16:0/4134655964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 msgr2=0x7f95b010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 --2- 172.21.15.16:0/4134655964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009820 tx=0x7f95a002a270 comp rx=0 tx=0).stop 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 -- 172.21.15.16:0/4134655964 shutdown_connections 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 --2- 172.21.15.16:0/4134655964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b010a6e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 -- 172.21.15.16:0/4134655964 >> 172.21.15.16:0/4134655964 conn(0x7f95b0101f10 msgr2=0x7f95b0104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:53.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 -- 172.21.15.16:0/4134655964 shutdown_connections 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.450+0000 7f95b5350640 1 -- 172.21.15.16:0/4134655964 wait complete. 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95b5350640 1 Processor -- start 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95b5350640 1 -- start start 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95b5350640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95b5350640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95b010bca0 con 0x7f95b01082f0 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35784/0 (socket says 172.21.15.16:35784) 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95aeffd640 1 -- 172.21.15.16:0/1715824680 learned_addr learned my addr 172.21.15.16:0/1715824680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.451+0000 7f95aeffd640 1 -- 172.21.15.16:0/1715824680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b019bd70 con 0x7f95b01082f0 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f95aeffd640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f95a0030380 tx=0x7f95a0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95a0037040 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a0040070 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b019c000 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95a003a7b0 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.452+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b019ecf0 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.453+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f95a003b420 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.453+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b0108750 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.453+0000 7f9597fff640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 0x7f959803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.453+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f95a006c600 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.453+0000 7f95ae7fc640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 0x7f959803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.454+0000 7f95ae7fc640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 0x7f959803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95a40098c0 tx=0x7f95a4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.458+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f95a0043360 con 0x7f95b01082f0 2024-09-12T02:38:53.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.633+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f95b019f350 con 0x7f95b01082f0 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.635+0000 7f9597fff640 1 -- 172.21.15.16:0/1715824680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f95a0040220 con 0x7f95b01082f0 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 msgr2=0x7f959803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 0x7f959803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95a40098c0 tx=0x7f95a4006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 msgr2=0x7f95b019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f95a0030380 tx=0x7f95a0032670 comp rx=0 tx=0).stop 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 shutdown_connections 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f9598038f70 0x7f959803b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 --2- 172.21.15.16:0/1715824680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b01082f0 0x7f95b019b830 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 >> 172.21.15.16:0/1715824680 conn(0x7f95b0101f10 msgr2=0x7f95b0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 shutdown_connections 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:52.638+0000 7f95b5350640 1 -- 172.21.15.16:0/1715824680 wait complete. 2024-09-12T02:38:53.297 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-09-12T02:38:53.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1715824680' entity='client.admin' 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:54 smithi016 ceph-mon[30925]: from='mgr.14118 172.21.15.16:0/335675442' entity='mgr.x' 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3152423576' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ignoring --setuser ceph since I am not root 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ignoring --setgroup ceph since I am not root 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:54.979+0000 7f90b6aa7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-12T02:38:55.137 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.039+0000 7f90b6aa7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:55.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.194+0000 7f90b6aa7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-12T02:38:55.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.264+0000 7f90b6aa7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-12T02:38:55.790 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.421+0000 7f90b6aa7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:55.790 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.568+0000 7f90b6aa7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.706+0000 7f2922727640 1 Processor -- start 2024-09-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.706+0000 7f2922727640 1 -- start start 2024-09-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.706+0000 7f2922727640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.706+0000 7f2922727640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f291c07c7d0 con 0x7f291c07dde0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.706+0000 7f291bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.707+0000 7f291bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35800/0 (socket says 172.21.15.16:35800) 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.707+0000 7f291bfff640 1 -- 172.21.15.16:0/768688552 learned_addr learned my addr 172.21.15.16:0/768688552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.707+0000 7f291bfff640 1 -- 172.21.15.16:0/768688552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c07c950 con 0x7f291c07dde0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.707+0000 7f291bfff640 1 --2- 172.21.15.16:0/768688552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f290c009820 tx=0x7f290c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19ff88df471ea8f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f291b7fe640 1 -- 172.21.15.16:0/768688552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f290c037070 con 0x7f291c07dde0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f291b7fe640 1 -- 172.21.15.16:0/768688552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c032920 con 0x7f291c07dde0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f291b7fe640 1 -- 172.21.15.16:0/768688552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f290c030810 con 0x7f291c07dde0 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 -- 172.21.15.16:0/768688552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 msgr2=0x7f291c07c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 --2- 172.21.15.16:0/768688552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f290c009820 tx=0x7f290c02a270 comp rx=0 tx=0).stop 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 -- 172.21.15.16:0/768688552 shutdown_connections 2024-09-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 --2- 172.21.15.16:0/768688552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c07c290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 -- 172.21.15.16:0/768688552 >> 172.21.15.16:0/768688552 conn(0x7f291c103040 msgr2=0x7f291c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 -- 172.21.15.16:0/768688552 shutdown_connections 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.708+0000 7f2922727640 1 -- 172.21.15.16:0/768688552 wait complete. 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f2922727640 1 Processor -- start 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f2922727640 1 -- start start 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f2922727640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f2922727640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f291c10de60 con 0x7f291c07dde0 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f291bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f291bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35802/0 (socket says 172.21.15.16:35802) 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.709+0000 7f291bfff640 1 -- 172.21.15.16:0/3152423576 learned_addr learned my addr 172.21.15.16:0/3152423576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f291bfff640 1 -- 172.21.15.16:0/3152423576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c19df00 con 0x7f291c07dde0 2024-09-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f291bfff640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f290c0097f0 tx=0x7f290c0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f290c037040 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291c19e190 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c030cc0 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f290c03a6b0 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.710+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f291c1a0e80 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.711+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f290c03b3f0 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.711+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f291c07c310 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.711+0000 7f2919ffb640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 0x7f290803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.711+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f290c06c4e0 con 0x7f291c07dde0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.711+0000 7f2913fff640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 0x7f290803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.712+0000 7f2913fff640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 0x7f290803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29040098c0 tx=0x7f2904006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.716+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f290c0371e0 con 0x7f291c07dde0 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:53.962+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f291c1a1050 con 0x7f291c07dde0 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.828+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f290c03b6e0 con 0x7f291c07dde0 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.829+0000 7f2919ffb640 1 -- 172.21.15.16:0/3152423576 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f290c03baf0 con 0x7f291c07dde0 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 msgr2=0x7f290803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 0x7f290803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29040098c0 tx=0x7f2904006db0 comp rx=0 tx=0).stop 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 msgr2=0x7f291c19d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f290c0097f0 tx=0x7f290c0327b0 comp rx=0 tx=0).stop 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 shutdown_connections 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f2908038f70 0x7f290803b430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 --2- 172.21.15.16:0/3152423576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291c07dde0 0x7f291c19d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 >> 172.21.15.16:0/3152423576 conn(0x7f291c103040 msgr2=0x7f291c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 shutdown_connections 2024-09-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:54.831+0000 7f2922727640 1 -- 172.21.15.16:0/3152423576 wait complete. 2024-09-12T02:38:56.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3152423576' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-12T02:38:56.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-mon[30925]: mgrmap e9: x(active, since 19s) 2024-09-12T02:38:56.050 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.899+0000 7f90b6aa7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-12T02:38:56.051 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:55.964+0000 7f90b6aa7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-12T02:38:56.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:56.160+0000 7f90b6aa7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-12T02:38:57.072 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:56 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2380509798' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-12T02:38:57.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:38:57.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd855e9640 1 Processor -- start 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd855e9640 1 -- start start 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd855e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd855e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd801081e0 con 0x7fcd80107810 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60164/0 (socket says 172.21.15.16:60164) 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd7effd640 1 -- 172.21.15.16:0/3485135602 learned_addr learned my addr 172.21.15.16:0/3485135602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:57.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.231+0000 7fcd7effd640 1 -- 172.21.15.16:0/3485135602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd801089c0 con 0x7fcd80107810 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.232+0000 7fcd7effd640 1 --2- 172.21.15.16:0/3485135602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009820 tx=0x7fcd7002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e826b400e993be server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.232+0000 7fcd7e7fc640 1 -- 172.21.15.16:0/3485135602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70037070 con 0x7fcd80107810 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.232+0000 7fcd7e7fc640 1 -- 172.21.15.16:0/3485135602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd70032920 con 0x7fcd80107810 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.232+0000 7fcd7e7fc640 1 -- 172.21.15.16:0/3485135602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70030810 con 0x7fcd80107810 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 -- 172.21.15.16:0/3485135602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 msgr2=0x7fcd80107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 --2- 172.21.15.16:0/3485135602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009820 tx=0x7fcd7002a270 comp rx=0 tx=0).stop 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 -- 172.21.15.16:0/3485135602 shutdown_connections 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 --2- 172.21.15.16:0/3485135602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd80107c10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 -- 172.21.15.16:0/3485135602 >> 172.21.15.16:0/3485135602 conn(0x7fcd80102f60 msgr2=0x7fcd801053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 -- 172.21.15.16:0/3485135602 shutdown_connections 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.233+0000 7fcd855e9640 1 -- 172.21.15.16:0/3485135602 wait complete. 2024-09-12T02:38:57.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd855e9640 1 Processor -- start 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd855e9640 1 -- start start 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd855e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd855e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8010bd70 con 0x7fcd80107810 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60168/0 (socket says 172.21.15.16:60168) 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd7effd640 1 -- 172.21.15.16:0/2380509798 learned_addr learned my addr 172.21.15.16:0/2380509798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.234+0000 7fcd7effd640 1 -- 172.21.15.16:0/2380509798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8019bd70 con 0x7fcd80107810 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd7effd640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd70030560 tx=0x7fcd70032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70037040 con 0x7fcd80107810 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd70040070 con 0x7fcd80107810 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8019c000 con 0x7fcd80107810 2024-09-12T02:38:57.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd7003a740 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.235+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd8019ecf0 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fcd7003b420 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd8010bef0 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd7cff9640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 0x7fcd6c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcd7006c5f0 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd77fff640 1 -- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 msgr2=0x7fcd6c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.236+0000 7fcd77fff640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 0x7fcd6c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.241+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcd7004a200 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.437+0000 7fcd77fff640 1 -- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 msgr2=0x7fcd6c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.437+0000 7fcd77fff640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 0x7fcd6c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.467+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fcd8019eec0 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.467+0000 7fcd7cff9640 1 -- 172.21.15.16:0/2380509798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fcd70031e80 con 0x7fcd80107810 2024-09-12T02:38:57.083 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 msgr2=0x7fcd6c03b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 0x7fcd6c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 msgr2=0x7fcd8019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd70030560 tx=0x7fcd70032670 comp rx=0 tx=0).stop 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 shutdown_connections 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7fcd6c038fc0 0x7fcd6c03b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 --2- 172.21.15.16:0/2380509798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd80107810 0x7fcd8019b830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 >> 172.21.15.16:0/2380509798 conn(0x7fcd80102f60 msgr2=0x7fcd80103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 shutdown_connections 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:56.470+0000 7fcd855e9640 1 -- 172.21.15.16:0/2380509798 wait complete. 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-12T02:38:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-09-12T02:38:57.323 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:57.102+0000 7f90b6aa7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-12T02:38:57.323 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:57.173+0000 7f90b6aa7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-12T02:38:58.051 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:57.929+0000 7f90b6aa7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-12T02:38:58.051 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.049+0000 7f90b6aa7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-12T02:38:58.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.199+0000 7f90b6aa7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:38:58.407 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.262+0000 7f90b6aa7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-12T02:38:58.720 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.419+0000 7f90b6aa7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-12T02:38:58.720 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.490+0000 7f90b6aa7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-12T02:38:58.720 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.553+0000 7f90b6aa7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-12T02:38:58.720 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.718+0000 7f90b6aa7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-12T02:38:59.005 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.787+0000 7f90b6aa7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-12T02:38:59.006 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.856+0000 7f90b6aa7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-12T02:38:59.006 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:58.942+0000 7f90b6aa7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-12T02:38:59.006 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:59.004+0000 7f90b6aa7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-12T02:38:59.336 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:59.066+0000 7f90b6aa7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-12T02:39:00.175 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:38:59.912+0000 7f90b6aa7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-12T02:39:00.175 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: Active manager daemon x restarted 2024-09-12T02:39:00.175 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: Activating manager daemon x 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: osdmap e3: 0 total, 0 up, 0 in 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: mgrmap e10: x(active, starting, since 0.00361089s) 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: Manager daemon x is now available 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:00.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:38:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-12T02:39:01.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-12T02:39:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:00 smithi016 ceph-mon[30925]: mgrmap e11: x(active, since 1.00617s) 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f80a0536640 1 Processor -- start 2024-09-12T02:39:01.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f80a0536640 1 -- start start 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f80a0536640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f80a0536640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8098107090 con 0x7f80981066c0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f809e2d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.479+0000 7f809e2d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60184/0 (socket says 172.21.15.16:60184) 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.480+0000 7f809e2d2640 1 -- 172.21.15.16:0/1024485927 learned_addr learned my addr 172.21.15.16:0/1024485927 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.480+0000 7f809e2d2640 1 -- 172.21.15.16:0/1024485927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80981078c0 con 0x7f80981066c0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.480+0000 7f809e2d2640 1 --2- 172.21.15.16:0/1024485927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f809400bac0 tx=0x7f809402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e1c2324fa5c76c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f809d2d0640 1 -- 172.21.15.16:0/1024485927 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80940326d0 con 0x7f80981066c0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f809d2d0640 1 -- 172.21.15.16:0/1024485927 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8094038070 con 0x7f80981066c0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f809d2d0640 1 -- 172.21.15.16:0/1024485927 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8094033d00 con 0x7f80981066c0 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 -- 172.21.15.16:0/1024485927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 msgr2=0x7f8098106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 --2- 172.21.15.16:0/1024485927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f809400bac0 tx=0x7f809402c920 comp rx=0 tx=0).stop 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 -- 172.21.15.16:0/1024485927 shutdown_connections 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 --2- 172.21.15.16:0/1024485927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f8098106ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 -- 172.21.15.16:0/1024485927 >> 172.21.15.16:0/1024485927 conn(0x7f8098101e90 msgr2=0x7f80981042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 -- 172.21.15.16:0/1024485927 shutdown_connections 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.481+0000 7f80a0536640 1 -- 172.21.15.16:0/1024485927 wait complete. 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f80a0536640 1 Processor -- start 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f80a0536640 1 -- start start 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f80a0536640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f80a0536640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f809807b100 con 0x7f80981066c0 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f809e2d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f809e2d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60192/0 (socket says 172.21.15.16:60192) 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.482+0000 7f809e2d2640 1 -- 172.21.15.16:0/1680214587 learned_addr learned my addr 172.21.15.16:0/1680214587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f809e2d2640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8098199be0 con 0x7f80981066c0 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f809e2d2640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f809400b2c0 tx=0x7f80940327d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8094042070 con 0x7f80981066c0 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8094032c80 con 0x7f80981066c0 2024-09-12T02:39:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8098199e70 con 0x7f80981066c0 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f809403c610 con 0x7f80981066c0 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.483+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809819cb60 con 0x7f80981066c0 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f809403c840 con 0x7f80981066c0 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f808f7fe640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f808403bbf0 con 0x7f8084039020 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8094030050 con 0x7f80981066c0 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f809dad1640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 msgr2=0x7f808403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.484+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.685+0000 7f809dad1640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 msgr2=0x7f808403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:57.685+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:58.086+0000 7f809dad1640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 msgr2=0x7f808403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:58.086+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-12T02:39:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:58.887+0000 7f809dad1640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 msgr2=0x7f808403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/744312028 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:58.887+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:59.915+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f809406dc50 con 0x7f80981066c0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:59.916+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 msgr2=0x7f808403b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:38:59.916+0000 7f808f7fe640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.917+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f809404f3f0 con 0x7f80981066c0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.918+0000 7f808f7fe640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 0x7f808403eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.918+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f808403bbf0 con 0x7f808403cae0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.918+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 0x7f808403eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.919+0000 7f809dad1640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 0x7f808403eed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8088003980 tx=0x7f80880092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.920+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f808403bbf0 con 0x7f808403cae0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.927+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8098106ac0 con 0x7f808403cae0 2024-09-12T02:39:01.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.927+0000 7f808f7fe640 1 -- 172.21.15.16:0/1680214587 <== mgr.14150 v2:172.21.15.16:6800/2564464416 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f8098106ac0 con 0x7f808403cae0 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 msgr2=0x7f808403eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 0x7f808403eed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8088003980 tx=0x7f80880092b0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 msgr2=0x7f80981996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f809400b2c0 tx=0x7f80940327d0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 shutdown_connections 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/744312028,v1:172.21.15.16:6801/744312028] conn(0x7f8084039020 0x7f808403b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f808403cae0 0x7f808403eed0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 --2- 172.21.15.16:0/1680214587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80981066c0 0x7f80981996a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 >> 172.21.15.16:0/1680214587 conn(0x7f8098101e90 msgr2=0x7f809818fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 shutdown_connections 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:00.928+0000 7f80a0536640 1 -- 172.21.15.16:0/1680214587 wait complete. 2024-09-12T02:39:01.505 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-09-12T02:39:01.506 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-09-12T02:39:02.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:39:02.189+0000 7f901c570640 -1 log_channel(cephadm) log [ERR] : [12/Sep/2024:02:39:02] ENGINE Error in HTTPServer.serve 2024-09-12T02:39:02.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: Traceback (most recent call last): 2024-09-12T02:39:02.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-12T02:39:02.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._connections.run(self.expiration_interval) 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._run(expiration_interval) 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: new_conn = self._from_server_socket(self.server.socket) 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: s = self.context.wrap_socket( 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: return self.sslsocket_class._create( 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self.do_handshake() 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-12T02:39:02.365 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: self._sslobj.do_handshake() 2024-09-12T02:39:02.366 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-12T02:39:02.366 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:39:02.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-12T02:39:02.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-12T02:39:02.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:02.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:02.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:02.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:02.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f826d73b640 1 Processor -- start 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f826d73b640 1 -- start start 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f826d73b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f826d73b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f826807be20 con 0x7f82681082b0 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60278/0 (socket says 172.21.15.16:60278) 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.105+0000 7f8266ffd640 1 -- 172.21.15.16:0/2830992254 learned_addr learned my addr 172.21.15.16:0/2830992254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.106+0000 7f8266ffd640 1 -- 172.21.15.16:0/2830992254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826810abe0 con 0x7f82681082b0 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.107+0000 7f8266ffd640 1 --2- 172.21.15.16:0/2830992254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8258009820 tx=0x7f825802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcc9d6f85b958d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.107+0000 7f82667fc640 1 -- 172.21.15.16:0/2830992254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8258037070 con 0x7f82681082b0 2024-09-12T02:39:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.107+0000 7f82667fc640 1 -- 172.21.15.16:0/2830992254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258032920 con 0x7f82681082b0 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.107+0000 7f82667fc640 1 -- 172.21.15.16:0/2830992254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8258030810 con 0x7f82681082b0 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 -- 172.21.15.16:0/2830992254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 msgr2=0x7f826810a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 --2- 172.21.15.16:0/2830992254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8258009820 tx=0x7f825802a270 comp rx=0 tx=0).stop 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 -- 172.21.15.16:0/2830992254 shutdown_connections 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 --2- 172.21.15.16:0/2830992254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826810a6a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 -- 172.21.15.16:0/2830992254 >> 172.21.15.16:0/2830992254 conn(0x7f8268101f30 msgr2=0x7f8268104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 -- 172.21.15.16:0/2830992254 shutdown_connections 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.108+0000 7f826d73b640 1 -- 172.21.15.16:0/2830992254 wait complete. 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f826d73b640 1 Processor -- start 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f826d73b640 1 -- start start 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f826d73b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f826d73b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f826810bc60 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60284/0 (socket says 172.21.15.16:60284) 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.109+0000 7f8266ffd640 1 -- 172.21.15.16:0/3659532682 learned_addr learned my addr 172.21.15.16:0/3659532682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f8266ffd640 1 -- 172.21.15.16:0/3659532682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826819bd90 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f8266ffd640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8258030420 tx=0x7f8258032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8258037040 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826819c020 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258040070 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826819ed10 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.110+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f825803a6e0 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.111+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f825803a990 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.111+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8268108710 con 0x7f82681082b0 2024-09-12T02:39:02.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.111+0000 7f8264ff9640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 0x7f825403f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.111+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f825806c500 con 0x7f82681082b0 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.112+0000 7f825e7fc640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 0x7f825403f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.112+0000 7f825e7fc640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 0x7f825403f7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82500098c0 tx=0x7f8250006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.116+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8258041270 con 0x7f82681082b0 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.299+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f8268103250 con 0x7f825403d310 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.350+0000 7f8264ff9640 1 -- 172.21.15.16:0/3659532682 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f8268103250 con 0x7f825403d310 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.352+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 msgr2=0x7f825403f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 0x7f825403f7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82500098c0 tx=0x7f8250006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 msgr2=0x7f826819b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8258030420 tx=0x7f8258032670 comp rx=0 tx=0).stop 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 shutdown_connections 2024-09-12T02:39:02.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f825403d310 0x7f825403f7d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 --2- 172.21.15.16:0/3659532682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82681082b0 0x7f826819b850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 >> 172.21.15.16:0/3659532682 conn(0x7f8268101f30 msgr2=0x7f8268102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 shutdown_connections 2024-09-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:02.353+0000 7f826d73b640 1 -- 172.21.15.16:0/3659532682 wait complete. 2024-09-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: [12/Sep/2024:02:39:01] ENGINE Bus STARTING 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: [12/Sep/2024:02:39:02] ENGINE Serving on http://172.21.15.16:8765 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: [12/Sep/2024:02:39:02] ENGINE Serving on https://172.21.15.16:7150 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: [12/Sep/2024:02:39:02] ENGINE Bus STARTED 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: [12/Sep/2024:02:39:02] ENGINE Error in HTTPServer.serve 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-12T02:39:03.846 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:03.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:03 smithi016 ceph-mon[30925]: mgrmap e12: x(active, since 2s) 2024-09-12T02:39:04.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$tQrJh/mRVdebaznmXBD0/uad8yrSDmICGddM2h.zG6ESmbmbakRyq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726108743, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-12T02:39:04.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d78164640 1 Processor -- start 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d78164640 1 -- start start 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d78164640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d78164640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7007be20 con 0x7f1d701082d0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d75f00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d75f00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60300/0 (socket says 172.21.15.16:60300) 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.393+0000 7f1d75f00640 1 -- 172.21.15.16:0/461441690 learned_addr learned my addr 172.21.15.16:0/461441690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.394+0000 7f1d75f00640 1 -- 172.21.15.16:0/461441690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7010ac00 con 0x7f1d701082d0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.394+0000 7f1d75f00640 1 --2- 172.21.15.16:0/461441690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009820 tx=0x7f1d6c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3b81da06a6c8b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.394+0000 7f1d74efe640 1 -- 172.21.15.16:0/461441690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c037070 con 0x7f1d701082d0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.394+0000 7f1d74efe640 1 -- 172.21.15.16:0/461441690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d6c032920 con 0x7f1d701082d0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.394+0000 7f1d74efe640 1 -- 172.21.15.16:0/461441690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c030810 con 0x7f1d701082d0 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 -- 172.21.15.16:0/461441690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 msgr2=0x7f1d7010a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 --2- 172.21.15.16:0/461441690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009820 tx=0x7f1d6c02a270 comp rx=0 tx=0).stop 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 -- 172.21.15.16:0/461441690 shutdown_connections 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 --2- 172.21.15.16:0/461441690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d7010a6c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 -- 172.21.15.16:0/461441690 >> 172.21.15.16:0/461441690 conn(0x7f1d70101f10 msgr2=0x7f1d70104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 -- 172.21.15.16:0/461441690 shutdown_connections 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.395+0000 7f1d78164640 1 -- 172.21.15.16:0/461441690 wait complete. 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d78164640 1 Processor -- start 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d78164640 1 -- start start 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d78164640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d78164640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7010bc80 con 0x7f1d701082d0 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d75f00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d75f00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60316/0 (socket says 172.21.15.16:60316) 2024-09-12T02:39:04.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d75f00640 1 -- 172.21.15.16:0/2343044934 learned_addr learned my addr 172.21.15.16:0/2343044934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.396+0000 7f1d75f00640 1 -- 172.21.15.16:0/2343044934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d70108090 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d75f00640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0097f0 tx=0x7f1d6c0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c037040 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d6c030cc0 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d70106280 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c03a6b0 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.397+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d701067c0 con 0x7f1d701082d0 2024-09-12T02:39:04.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.398+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f1d6c03a850 con 0x7f1d701082d0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.398+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d70108730 con 0x7f1d701082d0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.398+0000 7f1d66ffd640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 0x7f1d5c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.398+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d6c06c710 con 0x7f1d701082d0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.398+0000 7f1d756ff640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 0x7f1d5c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.399+0000 7f1d756ff640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 0x7f1d5c03b4e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d600098c0 tx=0x7f1d60006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.403+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1d6c044920 con 0x7f1d701082d0 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.595+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f1d70106a40 con 0x7f1d5c039020 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.844+0000 7f1d66ffd640 1 -- 172.21.15.16:0/2343044934 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f1d70106a40 con 0x7f1d5c039020 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 msgr2=0x7f1d5c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 0x7f1d5c03b4e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d600098c0 tx=0x7f1d60006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 msgr2=0x7f1d70107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:04.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0097f0 tx=0x7f1d6c0327b0 comp rx=0 tx=0).stop 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 shutdown_connections 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1d5c039020 0x7f1d5c03b4e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 --2- 172.21.15.16:0/2343044934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d701082d0 0x7f1d70107b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.846+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 >> 172.21.15.16:0/2343044934 conn(0x7f1d70101f10 msgr2=0x7f1d70102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.847+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 shutdown_connections 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:03.847+0000 7f1d78164640 1 -- 172.21.15.16:0/2343044934 wait complete. 2024-09-12T02:39:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-09-12T02:39:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:04 smithi016 ceph-mon[30925]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:05 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1951408950' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.092+0000 7f0d5fcf6640 1 Processor -- start 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.092+0000 7f0d5fcf6640 1 -- start start 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.092+0000 7f0d5fcf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.092+0000 7f0d5fcf6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d58107080 con 0x7f0d581066b0 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.093+0000 7f0d5da92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.093+0000 7f0d5da92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60332/0 (socket says 172.21.15.16:60332) 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.093+0000 7f0d5da92640 1 -- 172.21.15.16:0/1477535281 learned_addr learned my addr 172.21.15.16:0/1477535281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.093+0000 7f0d5da92640 1 -- 172.21.15.16:0/1477535281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d581078b0 con 0x7f0d581066b0 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5da92640 1 --2- 172.21.15.16:0/1477535281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009820 tx=0x7f0d5402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe184002dbc7e0a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5ca90640 1 -- 172.21.15.16:0/1477535281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d54037070 con 0x7f0d581066b0 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5ca90640 1 -- 172.21.15.16:0/1477535281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54032920 con 0x7f0d581066b0 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5ca90640 1 -- 172.21.15.16:0/1477535281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d54030810 con 0x7f0d581066b0 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1477535281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 msgr2=0x7f0d58106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1477535281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009820 tx=0x7f0d5402a270 comp rx=0 tx=0).stop 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1477535281 shutdown_connections 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1477535281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d58106ab0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.094+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1477535281 >> 172.21.15.16:0/1477535281 conn(0x7f0d58101e60 msgr2=0x7f0d58104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1477535281 shutdown_connections 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1477535281 wait complete. 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 Processor -- start 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 -- start start 2024-09-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5fcf6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5810ac60 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5da92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5da92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60342/0 (socket says 172.21.15.16:60342) 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.095+0000 7f0d5da92640 1 -- 172.21.15.16:0/1951408950 learned_addr learned my addr 172.21.15.16:0/1951408950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d5da92640 1 -- 172.21.15.16:0/1951408950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5819df80 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d5da92640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d540097f0 tx=0x7f0d540327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d54037040 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5819e210 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54030cc0 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d5403a640 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.096+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d581a0f00 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.097+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f0d5403a940 con 0x7f0d581066b0 2024-09-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.097+0000 7f0d4affd640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 0x7f0d4403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.097+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0d5406c6e0 con 0x7f0d581066b0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.097+0000 7f0d5d291640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 0x7f0d4403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.097+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d5810ade0 con 0x7f0d581066b0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.098+0000 7f0d5d291640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 0x7f0d4403b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0098c0 tx=0x7f0d4c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.103+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0d5403ba40 con 0x7f0d581066b0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.283+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f0d5802d8d0 con 0x7f0d581066b0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.284+0000 7f0d4affd640 1 -- 172.21.15.16:0/1951408950 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f0d54044920 con 0x7f0d581066b0 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 msgr2=0x7f0d4403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 0x7f0d4403b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0098c0 tx=0x7f0d4c006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 msgr2=0x7f0d5819da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d540097f0 tx=0x7f0d540327b0 comp rx=0 tx=0).stop 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 shutdown_connections 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d44038fc0 0x7f0d4403b480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 --2- 172.21.15.16:0/1951408950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d581066b0 0x7f0d5819da40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 >> 172.21.15.16:0/1951408950 conn(0x7f0d58101e60 msgr2=0x7f0d58102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 shutdown_connections 2024-09-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:05.286+0000 7f0d5fcf6640 1 -- 172.21.15.16:0/1951408950 wait complete. 2024-09-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: Password: 9etirvznmj 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config directory 2024-09-12T02:39:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.693+0000 7f08d87df640 1 Processor -- start 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.693+0000 7f08d87df640 1 -- start start 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d87df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d87df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08d007bde0 con 0x7f08d0106090 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d657b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d657b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49184/0 (socket says 172.21.15.16:49184) 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d657b640 1 -- 172.21.15.16:0/2636169428 learned_addr learned my addr 172.21.15.16:0/2636169428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.694+0000 7f08d657b640 1 -- 172.21.15.16:0/2636169428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d01069d0 con 0x7f08d0106090 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.695+0000 7f08d657b640 1 --2- 172.21.15.16:0/2636169428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009820 tx=0x7f08cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3397ad8a87a6c1c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.695+0000 7f08d5579640 1 -- 172.21.15.16:0/2636169428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc037070 con 0x7f08d0106090 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.695+0000 7f08d5579640 1 -- 172.21.15.16:0/2636169428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08cc032920 con 0x7f08d0106090 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.695+0000 7f08d5579640 1 -- 172.21.15.16:0/2636169428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc030810 con 0x7f08d0106090 2024-09-12T02:39:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 -- 172.21.15.16:0/2636169428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 msgr2=0x7f08d0106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 --2- 172.21.15.16:0/2636169428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009820 tx=0x7f08cc02a270 comp rx=0 tx=0).stop 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 -- 172.21.15.16:0/2636169428 shutdown_connections 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 --2- 172.21.15.16:0/2636169428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d0106490 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 -- 172.21.15.16:0/2636169428 >> 172.21.15.16:0/2636169428 conn(0x7f08d0101e60 msgr2=0x7f08d0104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 -- 172.21.15.16:0/2636169428 shutdown_connections 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.696+0000 7f08d87df640 1 -- 172.21.15.16:0/2636169428 wait complete. 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d87df640 1 Processor -- start 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d87df640 1 -- start start 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d87df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d87df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08d0107b10 con 0x7f08d0106090 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d657b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d657b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49194/0 (socket says 172.21.15.16:49194) 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d657b640 1 -- 172.21.15.16:0/963325142 learned_addr learned my addr 172.21.15.16:0/963325142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.697+0000 7f08d657b640 1 -- 172.21.15.16:0/963325142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d019dee0 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08d657b640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f08cc030380 tx=0x7f08cc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc037040 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08cc040070 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d019e170 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc03a7b0 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.698+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01a0e60 con 0x7f08d0106090 2024-09-12T02:39:07.677 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.699+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f08cc03a9e0 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.699+0000 7f08c77fe640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 0x7f08bc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.699+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f08cc06c850 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.699+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d0106490 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.699+0000 7f08d5d7a640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 0x7f08bc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.700+0000 7f08d5d7a640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 0x7f08bc03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08c00098c0 tx=0x7f08c0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.704+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f08cc03adc0 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.846+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f08cc045980 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.973+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f08d002d860 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.975+0000 7f08c77fe640 1 -- 172.21.15.16:0/963325142 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f08cc06f130 con 0x7f08d0106090 2024-09-12T02:39:07.678 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 msgr2=0x7f08bc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 0x7f08bc03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08c00098c0 tx=0x7f08c0006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 msgr2=0x7f08d019d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f08cc030380 tx=0x7f08cc032670 comp rx=0 tx=0).stop 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 shutdown_connections 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f08bc039020 0x7f08bc03b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 --2- 172.21.15.16:0/963325142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d0106090 0x7f08d019d9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 >> 172.21.15.16:0/963325142 conn(0x7f08d0101e60 msgr2=0x7f08d0194640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 shutdown_connections 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-09-12T02:39:06.978+0000 7f08d87df640 1 -- 172.21.15.16:0/963325142 wait complete. 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-12T02:39:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:07.681 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-09-12T02:39:07.721 INFO:tasks.cephadm:Fetching config... 2024-09-12T02:39:07.721 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:07.721 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-12T02:39:07.739 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-12T02:39:07.740 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:07.740 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-12T02:39:07.797 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-12T02:39:07.798 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:07.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-12T02:39:07.865 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-12T02:39:07.865 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:07.865 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-12T02:39:07.922 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-12T02:39:07.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-12T02:39:07.986 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:07 smithi016 ceph-mon[30925]: mgrmap e13: x(active, since 6s) 2024-09-12T02:39:07.986 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:07 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/963325142' entity='client.admin' 2024-09-12T02:39:08.009 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:08.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-12T02:39:08.066 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:08.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-12T02:39:08.120 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCQwdebrBqkJ2aXzQvAW3tBT3GBQic/Se2XNo8s3y7v/ag3MmeYmfgHbKlCSUmrzuvAEEJ0u05+ux8FyapkTB4HLCb2AA9VnmzQYUpDZpNsZMlNwwYALcmzCB+pwvT0IKEWsSZq1ungkq8K4DUaZn4UoJSJ+WSmFLRq/NrOtNMk5W+hxM4IMgqOz2wRAaOsfHNujhuOuMEZ3s/ILs8h+X2s0UQ393va9cboP5tAH5NupLkB5Tat9QLUqePZETAUsGp8yt7jUgCjo+fH2kdtwfhJe8M1xtXfUZGzsNmMFxoK4THoC9+RHAltmbZjql5zNJvqOpdzWMzYoOPSC6KTiVF5Bz1HQIKyNA7EIBg189Xb/p3IQA35tp86/Ysd9n1MWoms7GzLqN6SjW4hKst8gubXMJLPZw63E0FiEQrxOo7gXmnNMQMXKM0b3soJy7SOIITwlQgotQ2MSUfs9LheqrBHnLmOue2BwScF53kkUoCgVr5Mgp0q59byCWSJt7DKxVU= ceph-ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:08.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-12T02:39:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 -- 172.21.15.16:0/1111076464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 msgr2=0x7feedc1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 --2- 172.21.15.16:0/1111076464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc1044c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feecc009880 tx=0x7feecc02a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 -- 172.21.15.16:0/1111076464 shutdown_connections 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 --2- 172.21.15.16:0/1111076464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc1044c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 -- 172.21.15.16:0/1111076464 >> 172.21.15.16:0/1111076464 conn(0x7feedc0fbcd0 msgr2=0x7feedc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 -- 172.21.15.16:0/1111076464 shutdown_connections 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 -- 172.21.15.16:0/1111076464 wait complete. 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.898+0000 7feee182b640 1 Processor -- start 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feee182b640 1 -- start start 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feee182b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feee182b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feedc107670 con 0x7feedc1020d0 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49222/0 (socket says 172.21.15.16:49222) 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feedaffd640 1 -- 172.21.15.16:0/617300466 learned_addr learned my addr 172.21.15.16:0/617300466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feedaffd640 1 -- 172.21.15.16:0/617300466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc101e90 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feedaffd640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feecc004f60 tx=0x7feecc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.899+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feecc03f070 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.900+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feecc003ad0 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.900+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc100080 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.900+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feecc035030 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.900+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc1005c0 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.900+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7feecc042070 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.901+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feedc107df0 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.901+0000 7feee0829640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 0x7feec803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.901+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7feecc06cee0 con 0x7feedc1020d0 2024-09-12T02:39:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.901+0000 7feeda7fc640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 0x7feec803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.901+0000 7feeda7fc640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 0x7feec803b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feec400ad30 tx=0x7feec40093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:08.906+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feecc039350 con 0x7feedc1020d0 2024-09-12T02:39:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.084+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7feedc190350 con 0x7feedc1020d0 2024-09-12T02:39:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.087+0000 7feee0829640 1 -- 172.21.15.16:0/617300466 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7feecc036cd0 con 0x7feedc1020d0 2024-09-12T02:39:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 msgr2=0x7feec803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 0x7feec803b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feec400ad30 tx=0x7feec40093f0 comp rx=0 tx=0).stop 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 msgr2=0x7feedc101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feecc004f60 tx=0x7feecc0048f0 comp rx=0 tx=0).stop 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 shutdown_connections 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feec8039060 0x7feec803b520 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 --2- 172.21.15.16:0/617300466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc1020d0 0x7feedc101950 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 >> 172.21.15.16:0/617300466 conn(0x7feedc0fbcd0 msgr2=0x7feedc0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 shutdown_connections 2024-09-12T02:39:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:09.089+0000 7feee182b640 1 -- 172.21.15.16:0/617300466 wait complete. 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/617300466' entity='client.admin' 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:10.192 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-12T02:39:10.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-12T02:39:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 -- 172.21.15.16:0/2558293438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 msgr2=0x7f53f0100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 --2- 172.21.15.16:0/2558293438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f0100940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009880 tx=0x7f53ec02a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 -- 172.21.15.16:0/2558293438 shutdown_connections 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 --2- 172.21.15.16:0/2558293438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f0100940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 -- 172.21.15.16:0/2558293438 >> 172.21.15.16:0/2558293438 conn(0x7f53f00fbcd0 msgr2=0x7f53f00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 -- 172.21.15.16:0/2558293438 shutdown_connections 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.105+0000 7f53f7c73640 1 -- 172.21.15.16:0/2558293438 wait complete. 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f7c73640 1 Processor -- start 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f7c73640 1 -- start start 2024-09-12T02:39:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f7c73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f7c73640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01065f0 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f5a0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f5a0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49250/0 (socket says 172.21.15.16:49250) 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f5a0f640 1 -- 172.21.15.16:0/478058415 learned_addr learned my addr 172.21.15.16:0/478058415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.106+0000 7f53f5a0f640 1 -- 172.21.15.16:0/478058415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f0199af0 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53f5a0f640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53ec004f60 tx=0x7f53ec0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53ec03f070 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec004d40 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f0199cc0 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53ec035030 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.107+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f019a060 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.108+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f53ec042070 con 0x7f53f0100540 2024-09-12T02:39:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.108+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f0106d70 con 0x7f53f0100540 2024-09-12T02:39:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.108+0000 7f53e6ffd640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 0x7f53dc03b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.108+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f53ec06cc80 con 0x7f53f0100540 2024-09-12T02:39:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.108+0000 7f53f520e640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 0x7f53dc03b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.109+0000 7f53f520e640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 0x7f53dc03b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f53e00098c0 tx=0x7f53e0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.115+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f53ec073080 con 0x7f53f0100540 2024-09-12T02:39:11.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.297+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f53f002d930 con 0x7f53dc0390c0 2024-09-12T02:39:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.300+0000 7f53e6ffd640 1 -- 172.21.15.16:0/478058415 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f53f002d930 con 0x7f53dc0390c0 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 msgr2=0x7f53dc03b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 0x7f53dc03b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f53e00098c0 tx=0x7f53e0006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 msgr2=0x7f53f006a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53ec004f60 tx=0x7f53ec0048f0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 shutdown_connections 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53dc0390c0 0x7f53dc03b580 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 --2- 172.21.15.16:0/478058415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53f0100540 0x7f53f006a130 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.302+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 >> 172.21.15.16:0/478058415 conn(0x7f53f00fbcd0 msgr2=0x7f53f00fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.303+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 shutdown_connections 2024-09-12T02:39:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:11.303+0000 7f53f7c73640 1 -- 172.21.15.16:0/478058415 wait complete. 2024-09-12T02:39:12.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:12.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:12.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:12.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: Updating smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:12.557 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:12.633 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi033 2024-09-12T02:39:12.633 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:39:12.633 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.conf 2024-09-12T02:39:12.652 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:39:12.653 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:12.710 INFO:tasks.cephadm:Adding host smithi033 to orchestrator... 2024-09-12T02:39:12.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch host add smithi033 2024-09-12T02:39:12.962 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:13.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:13 smithi016 ceph-mon[30925]: Updating smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.406+0000 7f4c07313640 1 -- 172.21.15.16:0/3241718944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 msgr2=0x7f4c00106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.406+0000 7f4c07313640 1 --2- 172.21.15.16:0/3241718944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00106940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009880 tx=0x7f4bfc02a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.406+0000 7f4c07313640 1 -- 172.21.15.16:0/3241718944 shutdown_connections 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.406+0000 7f4c07313640 1 --2- 172.21.15.16:0/3241718944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00106940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.406+0000 7f4c07313640 1 -- 172.21.15.16:0/3241718944 >> 172.21.15.16:0/3241718944 conn(0x7f4c000fc1b0 msgr2=0x7f4c000fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 -- 172.21.15.16:0/3241718944 shutdown_connections 2024-09-12T02:39:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 -- 172.21.15.16:0/3241718944 wait complete. 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 Processor -- start 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 -- start start 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.407+0000 7f4c07313640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0010ab90 con 0x7f4c00106560 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c050af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c050af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49264/0 (socket says 172.21.15.16:49264) 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c050af640 1 -- 172.21.15.16:0/2270271872 learned_addr learned my addr 172.21.15.16:0/2270271872 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c050af640 1 -- 172.21.15.16:0/2270271872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c00197640 con 0x7f4c00106560 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c050af640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009230 tx=0x7f4bfc003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bfc03f070 con 0x7f4c00106560 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c001978d0 con 0x7f4c00106560 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc03bc80 con 0x7f4c00106560 2024-09-12T02:39:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bfc035030 con 0x7f4c00106560 2024-09-12T02:39:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.408+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0019b9d0 con 0x7f4c00106560 2024-09-12T02:39:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.409+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4bfc042070 con 0x7f4c00106560 2024-09-12T02:39:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.409+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c00101bd0 con 0x7f4c00106560 2024-09-12T02:39:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.410+0000 7f4bf67fc640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 0x7f4bec03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.410+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4bfc06ccd0 con 0x7f4c00106560 2024-09-12T02:39:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.410+0000 7f4c048ae640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 0x7f4bec03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.410+0000 7f4c048ae640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 0x7f4bec03b520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00098c0 tx=0x7f4bf0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:13.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.414+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4bfc04c350 con 0x7f4c00106560 2024-09-12T02:39:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:13.594+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}) -- 0x7f4c00198460 con 0x7f4bec039060 2024-09-12T02:39:14.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:14 smithi016 ceph-mon[30925]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:15.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:15 smithi016 ceph-mon[30925]: Deploying cephadm binary to smithi033 2024-09-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.480+0000 7f4bf67fc640 1 -- 172.21.15.16:0/2270271872 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4c00198460 con 0x7f4bec039060 2024-09-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi033' with addr '172.21.15.33' 2024-09-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 msgr2=0x7f4bec03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 0x7f4bec03b520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00098c0 tx=0x7f4bf0006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 msgr2=0x7f4c00197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009230 tx=0x7f4bfc003d40 comp rx=0 tx=0).stop 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 shutdown_connections 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4bec039060 0x7f4bec03b520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 --2- 172.21.15.16:0/2270271872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c00106560 0x7f4c00197100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 >> 172.21.15.16:0/2270271872 conn(0x7f4c000fc1b0 msgr2=0x7f4c00069ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 shutdown_connections 2024-09-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:17.483+0000 7f4c07313640 1 -- 172.21.15.16:0/2270271872 wait complete. 2024-09-12T02:39:18.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:18.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:18.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:18 smithi016 ceph-mon[30925]: Added host smithi033 2024-09-12T02:39:18.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:18.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:18.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.799+0000 7f329372d640 1 -- 172.21.15.16:0/1952777940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 msgr2=0x7f328c102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.799+0000 7f329372d640 1 --2- 172.21.15.16:0/1952777940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c102b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3288009810 tx=0x7f328802a540 comp rx=0 tx=0).stop 2024-09-12T02:39:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.799+0000 7f329372d640 1 -- 172.21.15.16:0/1952777940 shutdown_connections 2024-09-12T02:39:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.799+0000 7f329372d640 1 --2- 172.21.15.16:0/1952777940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c102b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.799+0000 7f329372d640 1 -- 172.21.15.16:0/1952777940 >> 172.21.15.16:0/1952777940 conn(0x7f328c077310 msgr2=0x7f328c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 -- 172.21.15.16:0/1952777940 shutdown_connections 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 -- 172.21.15.16:0/1952777940 wait complete. 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 Processor -- start 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 -- start start 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f329372d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f328c10bc80 con 0x7f328c102730 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.800+0000 7f32914c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32914c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44862/0 (socket says 172.21.15.16:44862) 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32914c9640 1 -- 172.21.15.16:0/2041352341 learned_addr learned my addr 172.21.15.16:0/2041352341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32914c9640 1 -- 172.21.15.16:0/2041352341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c199900 con 0x7f328c102730 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32914c9640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32880045e0 tx=0x7f3288004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3288030e70 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32880318e0 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32880049c0 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328c199b90 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.801+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328c199ff0 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.802+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3288039040 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.802+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f328c069f90 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.802+0000 7f32827fc640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 0x7f327803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.802+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f328806d2c0 con 0x7f328c102730 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.802+0000 7f3290cc8640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 0x7f327803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.803+0000 7f3290cc8640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 0x7f327803b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f327c0098c0 tx=0x7f327c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.807+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3288040030 con 0x7f328c102730 2024-09-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.987+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f328c19a780 con 0x7f3278039060 2024-09-12T02:39:18.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.988+0000 7f32827fc640 1 -- 172.21.15.16:0/2041352341 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f328c19a780 con 0x7f3278039060 2024-09-12T02:39:18.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:18.988 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}] 2024-09-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 msgr2=0x7f327803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 0x7f327803b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f327c0098c0 tx=0x7f327c006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 msgr2=0x7f328c1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32880045e0 tx=0x7f3288004610 comp rx=0 tx=0).stop 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 shutdown_connections 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3278039060 0x7f327803b520 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 --2- 172.21.15.16:0/2041352341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328c102730 0x7f328c1993c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 >> 172.21.15.16:0/2041352341 conn(0x7f328c077310 msgr2=0x7f328c1078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 shutdown_connections 2024-09-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:18.990+0000 7f329372d640 1 -- 172.21.15.16:0/2041352341 wait complete. 2024-09-12T02:39:19.599 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-09-12T02:39:19.600 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:39:19.600 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-09-12T02:39:19.616 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:39:19.617 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:19.671 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-09-12T02:39:19.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch host add smithi184 2024-09-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 -- 172.21.15.16:0/1898593579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 msgr2=0x7f95a4106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/1898593579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4106940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9594009880 tx=0x7f959402a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 -- 172.21.15.16:0/1898593579 shutdown_connections 2024-09-12T02:39:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/1898593579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4106940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 -- 172.21.15.16:0/1898593579 >> 172.21.15.16:0/1898593579 conn(0x7f95a40fc1b0 msgr2=0x7f95a40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 -- 172.21.15.16:0/1898593579 shutdown_connections 2024-09-12T02:39:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.371+0000 7f95a9c9f640 1 -- 172.21.15.16:0/1898593579 wait complete. 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a9c9f640 1 Processor -- start 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a9c9f640 1 -- start start 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a9c9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a9c9f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95a410ab90 con 0x7f95a4106560 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44874/0 (socket says 172.21.15.16:44874) 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a37fe640 1 -- 172.21.15.16:0/3004848805 learned_addr learned my addr 172.21.15.16:0/3004848805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.372+0000 7f95a37fe640 1 -- 172.21.15.16:0/3004848805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a4198490 con 0x7f95a4106560 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a37fe640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9594009230 tx=0x7f9594003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f959403f070 con 0x7f95a4106560 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f959403bc80 con 0x7f95a4106560 2024-09-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a4198720 con 0x7f95a4106560 2024-09-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9594035030 con 0x7f95a4106560 2024-09-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.373+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a4198b80 con 0x7f95a4106560 2024-09-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.374+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9594042070 con 0x7f95a4106560 2024-09-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.374+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a4101bd0 con 0x7f95a4106560 2024-09-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.374+0000 7f95a0ff9640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 0x7f958c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.374+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f959406ccd0 con 0x7f95a4106560 2024-09-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.374+0000 7f95a2ffd640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 0x7f958c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.375+0000 7f95a2ffd640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 0x7f958c03b520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95980098c0 tx=0x7f9598006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.379+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f959404c350 con 0x7f95a4106560 2024-09-12T02:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:20.558+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}) -- 0x7f95a41069c0 con 0x7f958c039060 2024-09-12T02:39:21.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:20 smithi016 ceph-mon[30925]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:39:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: Deploying cephadm binary to smithi184 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:23 smithi016 ceph-mon[30925]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:23 smithi016 ceph-mon[30925]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:23 smithi016 ceph-mon[30925]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:24.478 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi184' with addr '172.21.15.184' 2024-09-12T02:39:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.477+0000 7f95a0ff9640 1 -- 172.21.15.16:0/3004848805 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f95a41069c0 con 0x7f958c039060 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.479+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 msgr2=0x7f958c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.479+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 0x7f958c03b520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95980098c0 tx=0x7f9598006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.479+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 msgr2=0x7f95a4197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.479+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9594009230 tx=0x7f9594003d40 comp rx=0 tx=0).stop 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 shutdown_connections 2024-09-12T02:39:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f958c039060 0x7f958c03b520 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 --2- 172.21.15.16:0/3004848805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a4106560 0x7f95a4197f50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 >> 172.21.15.16:0/3004848805 conn(0x7f95a40fc1b0 msgr2=0x7f95a40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 shutdown_connections 2024-09-12T02:39:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:24.480+0000 7f95a9c9f640 1 -- 172.21.15.16:0/3004848805 wait complete. 2024-09-12T02:39:25.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-12T02:39:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:25.768 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:25.768 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:25.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: Added host smithi184 2024-09-12T02:39:25.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:25.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:25.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 -- 172.21.15.16:0/3717160766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 msgr2=0x7fe49c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/3717160766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c107a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe490009880 tx=0x7fe49002a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 -- 172.21.15.16:0/3717160766 shutdown_connections 2024-09-12T02:39:25.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/3717160766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c107a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 -- 172.21.15.16:0/3717160766 >> 172.21.15.16:0/3717160766 conn(0x7fe49c0fd290 msgr2=0x7fe49c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 -- 172.21.15.16:0/3717160766 shutdown_connections 2024-09-12T02:39:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.776+0000 7fe4a30f3640 1 -- 172.21.15.16:0/3717160766 wait complete. 2024-09-12T02:39:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a30f3640 1 Processor -- start 2024-09-12T02:39:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a30f3640 1 -- start start 2024-09-12T02:39:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a30f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a30f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe49c10bc90 con 0x7fe49c107660 2024-09-12T02:39:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a0e8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a0e8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59314/0 (socket says 172.21.15.16:59314) 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a0e8f640 1 -- 172.21.15.16:0/4198178626 learned_addr learned my addr 172.21.15.16:0/4198178626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a0e8f640 1 -- 172.21.15.16:0/4198178626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49c197630 con 0x7fe49c107660 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.777+0000 7fe4a0e8f640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe490009230 tx=0x7fe490003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.778+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe49003f070 con 0x7fe49c107660 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.778+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe49003bc80 con 0x7fe49c107660 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.778+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49c1978c0 con 0x7fe49c107660 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.778+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe490035030 con 0x7fe49c107660 2024-09-12T02:39:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.778+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe49c19b9c0 con 0x7fe49c107660 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe490042070 con 0x7fe49c107660 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe499ffb640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 0x7fe48c03b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe49c102cd0 con 0x7fe49c107660 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe49006cf30 con 0x7fe49c107660 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe49bfff640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 0x7fe48c03b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.779+0000 7fe49bfff640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 0x7fe48c03b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe4880098c0 tx=0x7fe488006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:25.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.784+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe4900314c0 con 0x7fe49c107660 2024-09-12T02:39:25.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.978+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe49c107b20 con 0x7fe48c0390c0 2024-09-12T02:39:25.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.979+0000 7fe499ffb640 1 -- 172.21.15.16:0/4198178626 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fe49c107b20 con 0x7fe48c0390c0 2024-09-12T02:39:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:25.980 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 msgr2=0x7fe48c03b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 0x7fe48c03b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe4880098c0 tx=0x7fe488006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 msgr2=0x7fe49c1970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe490009230 tx=0x7fe490003d40 comp rx=0 tx=0).stop 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 shutdown_connections 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe48c0390c0 0x7fe48c03b580 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 --2- 172.21.15.16:0/4198178626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe49c107660 0x7fe49c1970f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 >> 172.21.15.16:0/4198178626 conn(0x7fe49c0fd290 msgr2=0x7fe49c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 shutdown_connections 2024-09-12T02:39:25.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:25.981+0000 7fe4a30f3640 1 -- 172.21.15.16:0/4198178626 wait complete. 2024-09-12T02:39:26.570 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-12T02:39:26.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-12T02:39:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:26 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 -- 172.21.15.16:0/2614531161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 msgr2=0x7f4e74107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2614531161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e74107a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009880 tx=0x7f4e6402a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 -- 172.21.15.16:0/2614531161 shutdown_connections 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2614531161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e74107a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 -- 172.21.15.16:0/2614531161 >> 172.21.15.16:0/2614531161 conn(0x7f4e740fd290 msgr2=0x7f4e740ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 -- 172.21.15.16:0/2614531161 shutdown_connections 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.249+0000 7f4e7a993640 1 -- 172.21.15.16:0/2614531161 wait complete. 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e7a993640 1 Processor -- start 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e7a993640 1 -- start start 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e7a993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e7a993640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410bc90 con 0x7f4e74107660 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59330/0 (socket says 172.21.15.16:59330) 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e73fff640 1 -- 172.21.15.16:0/2210927461 learned_addr learned my addr 172.21.15.16:0/2210927461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.250+0000 7f4e73fff640 1 -- 172.21.15.16:0/2210927461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e740763e0 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e73fff640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009230 tx=0x7f4e64003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e6403f070 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e74074a30 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e6403bc80 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e64035030 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.251+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e74074ef0 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4e64042070 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e74102cd0 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e717fa640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 0x7f4e5c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e6406ced0 con 0x7f4e74107660 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e737fe640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 0x7f4e5c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.252+0000 7f4e737fe640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 0x7f4e5c03b520 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098c0 tx=0x7f4e68006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.257+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4e6404c350 con 0x7f4e74107660 2024-09-12T02:39:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.438+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4e74075a20 con 0x7f4e74107660 2024-09-12T02:39:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.860+0000 7f4e717fa640 1 -- 172.21.15.16:0/2210927461 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f4e64031e20 con 0x7f4e74107660 2024-09-12T02:39:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-12T02:39:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 msgr2=0x7f4e5c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 0x7f4e5c03b520 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098c0 tx=0x7f4e68006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 msgr2=0x7f4e740744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009230 tx=0x7f4e64003d40 comp rx=0 tx=0).stop 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 shutdown_connections 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e5c039060 0x7f4e5c03b520 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 --2- 172.21.15.16:0/2210927461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e74107660 0x7f4e740744f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 >> 172.21.15.16:0/2210927461 conn(0x7f4e740fd290 msgr2=0x7f4e740febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 shutdown_connections 2024-09-12T02:39:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:27.863+0000 7f4e7a993640 1 -- 172.21.15.16:0/2210927461 wait complete. 2024-09-12T02:39:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:27 smithi016 ceph-mon[30925]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:27 smithi016 ceph-mon[30925]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:39:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-12T02:39:28.448 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-09-12T02:39:28.448 INFO:tasks.cephadm:Adding mon.b on smithi033 2024-09-12T02:39:28.449 INFO:tasks.cephadm:Adding mon.c on smithi184 2024-09-12T02:39:28.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c' 2024-09-12T02:39:28.691 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-12T02:39:28.735 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:39:29.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:29.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:29.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.193+0000 7f348bf5c640 1 -- 172.21.15.184:0/1627273044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 msgr2=0x7f34841019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.193+0000 7f348bf5c640 1 --2- 172.21.15.184:0/1627273044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f34841019e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3480009880 tx=0x7f348002a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.193+0000 7f348bf5c640 1 -- 172.21.15.184:0/1627273044 shutdown_connections 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.193+0000 7f348bf5c640 1 --2- 172.21.15.184:0/1627273044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f34841019e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.193+0000 7f348bf5c640 1 -- 172.21.15.184:0/1627273044 >> 172.21.15.184:0/1627273044 conn(0x7f34840fcd70 msgr2=0x7f34840ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.194+0000 7f348bf5c640 1 -- 172.21.15.184:0/1627273044 shutdown_connections 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.194+0000 7f348bf5c640 1 -- 172.21.15.184:0/1627273044 wait complete. 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.194+0000 7f348bf5c640 1 Processor -- start 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.194+0000 7f348bf5c640 1 -- start start 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f348bf5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f348bf5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34841076f0 con 0x7f34841015e0 2024-09-12T02:39:29.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f3489cf8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f3489cf8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.184:60356/0 (socket says 172.21.15.184:60356) 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f3489cf8640 1 -- 172.21.15.184:0/857391303 learned_addr learned my addr 172.21.15.184:0/857391303 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f3489cf8640 1 -- 172.21.15.184:0/857391303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34841972a0 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.195+0000 7f3489cf8640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3480004f60 tx=0x7f34800048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.196+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f348003f070 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.196+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3484197530 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.196+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3480004cc0 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.196+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3480035030 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.196+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348419a220 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.197+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3480042070 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.197+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3484107e70 con 0x7f34841015e0 2024-09-12T02:39:29.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.197+0000 7f347affd640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 0x7f347003b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:29.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.197+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f348006ce90 con 0x7f34841015e0 2024-09-12T02:39:29.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.197+0000 7f34894f7640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 0x7f347003b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:29.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.198+0000 7f34894f7640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 0x7f347003b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34740098c0 tx=0x7f3474006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:29.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.202+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f34800433b0 con 0x7f34841015e0 2024-09-12T02:39:29.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.397+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}) -- 0x7f348402d830 con 0x7f3470039060 2024-09-12T02:39:29.404 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.404+0000 7f347affd640 1 -- 172.21.15.184:0/857391303 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f348402d830 con 0x7f3470039060 2024-09-12T02:39:29.404 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 msgr2=0x7f347003b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 0x7f347003b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34740098c0 tx=0x7f3474006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 msgr2=0x7f3484196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3480004f60 tx=0x7f34800048f0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 shutdown_connections 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3470039060 0x7f347003b520 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 --2- 172.21.15.184:0/857391303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34841015e0 0x7f3484196d60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 >> 172.21.15.184:0/857391303 conn(0x7f34840fcd70 msgr2=0x7f34840fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 shutdown_connections 2024-09-12T02:39:29.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:29.406+0000 7f348bf5c640 1 -- 172.21.15.184:0/857391303 wait complete. 2024-09-12T02:39:30.005 DEBUG:teuthology.orchestra.run.smithi033:mon.b> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.b.service 2024-09-12T02:39:30.008 DEBUG:teuthology.orchestra.run.smithi184:mon.c> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.c.service 2024-09-12T02:39:30.011 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-12T02:39:30.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-12T02:39:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:30.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:30.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:30.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:30.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:30.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:30 smithi016 ceph-mon[30925]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:30 smithi016 ceph-mon[30925]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:30 smithi016 ceph-mon[30925]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:30 smithi016 ceph-mon[30925]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c;count:3 2024-09-12T02:39:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:30 smithi016 ceph-mon[30925]: Deploying daemon mon.c on smithi184 2024-09-12T02:39:31.212 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:31 smithi016 ceph-mon[30925]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:32.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- 172.21.15.184:0/3882397957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 msgr2=0x7f4c70106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:32.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/3882397957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70106940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009880 tx=0x7f4c6402a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:32.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- 172.21.15.184:0/3882397957 shutdown_connections 2024-09-12T02:39:32.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/3882397957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70106940 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:32.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- 172.21.15.184:0/3882397957 >> 172.21.15.184:0/3882397957 conn(0x7f4c700fc1b0 msgr2=0x7f4c700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- 172.21.15.184:0/3882397957 shutdown_connections 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- 172.21.15.184:0/3882397957 wait complete. 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 Processor -- start 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.162+0000 7f4c76fe9640 1 -- start start 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c76fe9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:32.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c76fe9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7010ab90 con 0x7f4c70106560 2024-09-12T02:39:32.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c74d85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:32.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c74d85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.184:60384/0 (socket says 172.21.15.184:60384) 2024-09-12T02:39:32.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c74d85640 1 -- 172.21.15.184:0/356272846 learned_addr learned my addr 172.21.15.184:0/356272846 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:32.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.163+0000 7f4c74d85640 1 -- 172.21.15.184:0/356272846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c70198490 con 0x7f4c70106560 2024-09-12T02:39:32.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c74d85640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009230 tx=0x7f4c64003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c6403f070 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c70198720 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6403bd40 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64035030 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.164+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c70198b80 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.165+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4c64042070 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.165+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c70101bd0 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.165+0000 7f4c6dffb640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 0x7f4c6003b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.165+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4c6406cd00 con 0x7f4c70106560 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.165+0000 7f4c6ffff640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 0x7f4c6003b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:32.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.166+0000 7f4c6ffff640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 0x7f4c6003b520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098c0 tx=0x7f4c5c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:32.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.170+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4c64036df0 con 0x7f4c70106560 2024-09-12T02:39:32.405 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.404+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4c701069c0 con 0x7f4c70106560 2024-09-12T02:39:32.406 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.405+0000 7f4c6dffb640 1 -- 172.21.15.184:0/356272846 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f4c6403b860 con 0x7f4c70106560 2024-09-12T02:39:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:39:32.406 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","modified":"2024-09-12T02:38:06.636712Z","created":"2024-09-12T02:38:06.636712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-12T02:39:32.406 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 msgr2=0x7f4c6003b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 0x7f4c6003b520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098c0 tx=0x7f4c5c006db0 comp rx=0 tx=0).stop 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 msgr2=0x7f4c70197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009230 tx=0x7f4c64003d80 comp rx=0 tx=0).stop 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 shutdown_connections 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4c60039060 0x7f4c6003b520 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 --2- 172.21.15.184:0/356272846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c70106560 0x7f4c70197f50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:32.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 >> 172.21.15.184:0/356272846 conn(0x7f4c700fc1b0 msgr2=0x7f4c700fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:32.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 shutdown_connections 2024-09-12T02:39:32.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:32.407+0000 7f4c76fe9640 1 -- 172.21.15.184:0/356272846 wait complete. 2024-09-12T02:39:33.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/356272846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 podman[35302]: 2024-09-12 02:39:34.008649267 +0000 UTC m=+0.125687185 container create d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-12T02:39:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:33 smithi016 ceph-mon[30925]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.437 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-12T02:39:34.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 podman[35302]: 2024-09-12 02:39:34.228022811 +0000 UTC m=+0.345060728 container init d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 podman[35302]: 2024-09-12 02:39:34.231176774 +0000 UTC m=+0.348214692 container start d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: set uid:gid to 167:167 (ceph:ceph) 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable), process ceph-mon, pid 2 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pidfile_write: ignore empty --pid-file 2024-09-12T02:39:34.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: load: jerasure load: lrc 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: RocksDB version: 7.9.2 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Git sha 0 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Compile date 2024-09-09 16:23:28 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: DB SUMMARY 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: DB Session ID: UR3JPCCJ7QBQPRWT30DM 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: CURRENT file: CURRENT 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: IDENTITY file: IDENTITY 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.error_if_exists: 0 2024-09-12T02:39:34.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.create_if_missing: 0 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.paranoid_checks: 1 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.env: 0x559e08467940 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.fs: PosixFileSystem 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.info_log: 0x559e0a6316e0 2024-09-12T02:39:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_file_opening_threads: 16 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.statistics: (nil) 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.use_fsync: 0 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_log_file_size: 0 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.keep_log_file_num: 1000 2024-09-12T02:39:34.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.recycle_log_file_num: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_fallocate: 1 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_mmap_reads: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_mmap_writes: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.use_direct_reads: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.create_missing_column_families: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.db_log_dir: 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.wal_dir: 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-12T02:39:34.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.advise_random_on_open: 1 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.db_write_buffer_size: 0 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.write_buffer_manager: 0x559e0a640dc0 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.rate_limiter: (nil) 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.wal_recovery_mode: 2 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enable_thread_tracking: 0 2024-09-12T02:39:34.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enable_pipelined_write: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.unordered_write: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.row_cache: None 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.wal_filter: None 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_ingest_behind: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.two_write_queues: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.manual_wal_flush: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.wal_compression: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.atomic_flush: 0 2024-09-12T02:39:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.log_readahead_size: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.best_efforts_recovery: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.allow_data_in_errors: 0 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.db_host_id: __hostname__ 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_background_jobs: 2 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_background_compactions: -1 2024-09-12T02:39:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_subcompactions: 1 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_total_wal_size: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_open_files: -1 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bytes_per_sync: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_readahead_size: 0 2024-09-12T02:39:34.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_background_flushes: -1 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Compression algorithms supported: 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kZSTD supported: 0 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kXpressCompression supported: 0 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kBZip2Compression supported: 0 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kLZ4Compression supported: 1 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kZlibCompression supported: 1 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kLZ4HCCompression supported: 1 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: kSnappyCompression supported: 1 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-12T02:39:34.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.merge_operator: 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_filter: None 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_filter_factory: None 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.sst_partitioner_factory: None 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559e0a631320) 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-12T02:39:34.473 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_type: 0 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_index_type: 0 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_shortening: 1 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: checksum: 4 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: no_block_cache: 0 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache: 0x559e0a6251f0 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_options: 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: capacity : 536870912 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_shard_bits : 4 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: strict_capacity_limit : 0 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-12T02:39:34.474 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_compressed: (nil) 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: persistent_cache: (nil) 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size: 4096 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size_deviation: 10 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_restart_interval: 16 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_block_restart_interval: 1 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: metadata_block_size: 4096 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: partition_filters: 0 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: use_delta_encoding: 1 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: filter_policy: bloomfilter 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: whole_key_filtering: 1 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: verify_compression: 0 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: format_version: 5 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_index_compression: 1 2024-09-12T02:39:34.475 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_align: 0 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout: prepopulate_block_cache: 0 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.write_buffer_size: 33554432 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_write_buffer_number: 2 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression: NoCompression 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression: Disabled 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.prefix_extractor: nullptr 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.num_levels: 7 2024-09-12T02:39:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-12T02:39:34.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.level: 32767 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.strategy: 0 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.enabled: false 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.target_file_size_base: 67108864 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-12T02:39:34.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-12T02:39:34.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.arena_block_size: 1048576 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.disable_auto_compactions: 0 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-12T02:39:34.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.inplace_update_support: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.bloom_locality: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.max_successive_merges: 0 2024-09-12T02:39:34.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.paranoid_file_checks: 0 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.force_consistency_checks: 1 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.report_bg_io_stats: 0 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.ttl: 2592000 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-12T02:39:34.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enable_blob_files: false 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.min_blob_size: 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_file_size: 268435456 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.blob_file_starting_level: 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-12T02:39:34.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e687778b-3c7e-4887-9400-c49479f3338c 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108774258506, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108774258919, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726108774, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e687778b-3c7e-4887-9400-c49479f3338c", "db_session_id": "UR3JPCCJ7QBQPRWT30DM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726108774259010, "job": 1, "event": "recovery_finished"} 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559e0a652e00 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: DB pointer 0x559e0a74e000 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-12T02:39:34.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** DB Stats ** 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-12T02:39:34.485 INFO:journalctl@ceph.mon.c.smithi184.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-12T02:39:34.486 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache BinnedLRUCache@0x559e0a6251f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: using public_addr v2:172.21.15.184:0/0 -> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: starting mon.c rank -1 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(???) e0 preinit fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:34.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).mds e1 new map 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: e1 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: btime 2024-09-12T02:38:12:619852+0000 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: legacy client fscid: -1 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout: No filesystems configured 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-12T02:39:34.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Added host smithi184 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:34.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c;count:3 2024-09-12T02:39:34.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: Deploying daemon mon.c on smithi184 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/356272846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 ceph-mon[35321]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 bash[35302]: d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa 2024-09-12T02:39:34.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:34 smithi184 systemd[1]: Started Ceph mon.c for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:39:34.688 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1290084831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 msgr2=0x7fb5bc007bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1290084831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5bc007bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb5d01016d0 tx=0x7fb5c002c490 comp rx=0 tx=0).stop 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1290084831 shutdown_connections 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1290084831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5bc007bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1290084831 >> 172.21.15.184:0/1290084831 conn(0x7fb5d00fc250 msgr2=0x7fb5d00fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1290084831 shutdown_connections 2024-09-12T02:39:39.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.481+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1290084831 wait complete. 2024-09-12T02:39:39.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.483+0000 7fb5d4a28640 1 Processor -- start 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.483+0000 7fb5d4a28640 1 -- start start 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5d4a28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5d01994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5d4a28640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5d4a28640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01075d0 con 0x7fb5bc005740 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5d4a28640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d0107450 con 0x7fb5d0199a10 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:46236/0 (socket says 172.21.15.184:46236) 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 -- 172.21.15.184:0/1355222530 learned_addr learned my addr 172.21.15.184:0/1355222530 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5ce59c640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5d01994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 -- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 msgr2=0x7fb5d019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 -- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 msgr2=0x7fb5d019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-12T02:39:39.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-12T02:39:39.487 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5c5d9b640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:39:39.487 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5ce59c640 1 -- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 msgr2=0x7fb5d019dda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:39:39.487 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5ce59c640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.487 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.484+0000 7fb5ce59c640 1 -- 172.21.15.184:0/1355222530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d019a030 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5ce59c640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5d01994d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5c002ca80 tx=0x7fb5c00339a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb5c00329e0 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c0005520 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d019e2e0 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb5c0004bc0 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.485+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d019e870 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.486+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb5c0043020 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.486+0000 7fb5c7fff640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 0x7fb5b003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.486+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d0101cd0 con 0x7fb5bc005740 2024-09-12T02:39:39.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.486+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb5c0041020 con 0x7fb5bc005740 2024-09-12T02:39:39.489 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.487+0000 7fb5c5d9b640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 0x7fb5b003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:39.489 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.487+0000 7fb5c5d9b640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 0x7fb5b003b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009100 tx=0x7fb5b8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:39.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.492+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb5c004a020 con 0x7fb5bc005740 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: mon.a calling monitor election 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: mon.c calling monitor election 2024-09-12T02:39:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: monmap epoch 2 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: last_changed 2024-09-12T02:39:34.284418+0000 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: min_mon_release 19 (squid) 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: election_strategy: 1 2024-09-12T02:39:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: fsmap 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: mgrmap e13: x(active, since 39s) 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:39.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:39.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.725+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb5d01931a0 con 0x7fb5bc005740 2024-09-12T02:39:39.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:39.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: mon.a calling monitor election 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: mon.c calling monitor election 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: monmap epoch 2 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:39.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: last_changed 2024-09-12T02:39:34.284418+0000 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: min_mon_release 19 (squid) 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: election_strategy: 1 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: fsmap 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: mgrmap e13: x(active, since 39s) 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:39.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:39.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.726+0000 7fb5c7fff640 1 -- 172.21.15.184:0/1355222530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fb5c003a940 con 0x7fb5bc005740 2024-09-12T02:39:39.730 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:39:39.730 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","modified":"2024-09-12T02:39:34.284418Z","created":"2024-09-12T02:38:06.636712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 msgr2=0x7fb5b003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 0x7fb5b003b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009100 tx=0x7fb5b8008040 comp rx=0 tx=0).stop 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 msgr2=0x7fb5d01994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5d01994d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5c002ca80 tx=0x7fb5c00339a0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 shutdown_connections 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d0199a10 0x7fb5d019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5b0038e50 0x7fb5b003b310 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 --2- 172.21.15.184:0/1355222530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5bc005740 0x7fb5d01994d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 >> 172.21.15.184:0/1355222530 conn(0x7fb5d00fc250 msgr2=0x7fb5d00fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:39.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 shutdown_connections 2024-09-12T02:39:39.732 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:39.729+0000 7fb5d4a28640 1 -- 172.21.15.184:0/1355222530 wait complete. 2024-09-12T02:39:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:40 smithi016 ceph-mon[30925]: Deploying daemon mon.b on smithi033 2024-09-12T02:39:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:40 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/1355222530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:40 smithi184 ceph-mon[35321]: Deploying daemon mon.b on smithi033 2024-09-12T02:39:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:40 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/1355222530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:41.388 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-12T02:39:41.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-12T02:39:41.637 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:39:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:41 smithi016 ceph-mon[30925]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:41.658 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:39:41 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:39:41.285+0000 7f90557b0640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-12T02:39:41.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:41 smithi184 ceph-mon[35321]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:42.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 -- 172.21.15.184:0/3033533837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c107660 msgr2=0x7f822c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:42.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 --2- 172.21.15.184:0/3033533837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c107660 0x7f822c107a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f821c009880 tx=0x7f821c02a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 -- 172.21.15.184:0/3033533837 shutdown_connections 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 --2- 172.21.15.184:0/3033533837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c107660 0x7f822c107a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 -- 172.21.15.184:0/3033533837 >> 172.21.15.184:0/3033533837 conn(0x7f822c0fd320 msgr2=0x7f822c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 -- 172.21.15.184:0/3033533837 shutdown_connections 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.085+0000 7f8231b30640 1 -- 172.21.15.184:0/3033533837 wait complete. 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 Processor -- start 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 -- start start 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f822c10bd20 con 0x7f822c074a30 2024-09-12T02:39:42.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.086+0000 7f8231b30640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f822c10bba0 con 0x7f822c107660 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.184:49146/0 (socket says 172.21.15.184:49146) 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 -- 172.21.15.184:0/491036189 learned_addr learned my addr 172.21.15.184:0/491036189 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822b7fe640 1 -- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 msgr2=0x7f822c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822b7fe640 1 -- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 msgr2=0x7f822c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822b7fe640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822b7fe640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:39:42.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 -- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 msgr2=0x7f822c0744f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 -- 172.21.15.184:0/491036189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c0763e0 con 0x7f822c074a30 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f822affd640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f822000b4d0 tx=0x7f822000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.087+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8220010670 con 0x7f822c074a30 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.088+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8220010cf0 con 0x7f822c074a30 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.088+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f822c1a3650 con 0x7f822c074a30 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.088+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f822000fe60 con 0x7f822c074a30 2024-09-12T02:39:42.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.088+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f822c1a3be0 con 0x7f822c074a30 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.088+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8220017070 con 0x7f822c074a30 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.089+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f822c102da0 con 0x7f822c074a30 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.089+0000 7f8228ff9640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 0x7f821403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.089+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f822004bd00 con 0x7f822c074a30 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.089+0000 7f822b7fe640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 0x7f821403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:42.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.090+0000 7f822b7fe640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 0x7f821403b2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821c0314e0 tx=0x7f821c0058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:42.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.094+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f822001e030 con 0x7f822c074a30 2024-09-12T02:39:42.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.318+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f822c1a3ef0 con 0x7f822c074a30 2024-09-12T02:39:42.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.319+0000 7f8228ff9640 1 -- 172.21.15.184:0/491036189 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f822004b4f0 con 0x7f822c074a30 2024-09-12T02:39:42.321 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:39:42.321 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","modified":"2024-09-12T02:39:34.284418Z","created":"2024-09-12T02:38:06.636712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-12T02:39:42.321 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 msgr2=0x7f821403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 0x7f821403b2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821c0314e0 tx=0x7f821c0058b0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 msgr2=0x7f822c074e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f822000b4d0 tx=0x7f822000b990 comp rx=0 tx=0).stop 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 shutdown_connections 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8214038e00 0x7f821403b2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c074a30 0x7f822c074e90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 --2- 172.21.15.184:0/491036189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f822c107660 0x7f822c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:42.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 >> 172.21.15.184:0/491036189 conn(0x7f822c0fd320 msgr2=0x7f822c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:42.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 shutdown_connections 2024-09-12T02:39:42.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:42.321+0000 7f8231b30640 1 -- 172.21.15.184:0/491036189 wait complete. 2024-09-12T02:39:42.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:42 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/491036189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:42.665 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:42 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/491036189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:43.088 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-12T02:39:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-12T02:39:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-12T02:39:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Added host smithi184 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2210927461' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:43.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c;count:3 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Deploying daemon mon.c on smithi184 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/356272846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.a calling monitor election 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.c calling monitor election 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: monmap epoch 2 2024-09-12T02:39:43.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: last_changed 2024-09-12T02:39:34.284418+0000 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: min_mon_release 19 (squid) 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: election_strategy: 1 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: fsmap 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:43.503 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mgrmap e13: x(active, since 39s) 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: Deploying daemon mon.b on smithi033 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/1355222530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/491036189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 ceph-mon[35092]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-12T02:39:43.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 bash[35073]: 067da0701270c27a30f054da5347cac89d0702d3309ac28c9d79de72beb2524f 2024-09-12T02:39:43.505 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:43 smithi033 systemd[1]: Started Ceph mon.b for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:39:43.918 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-12T02:39:43.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-12T02:39:44.166 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.217+0000 7f2f844ef640 1 -- 172.21.15.184:0/4271291617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 msgr2=0x7f2f68007bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.217+0000 7f2f844ef640 1 --2- 172.21.15.184:0/4271291617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f68007bd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c069dc0 tx=0x7f2f6c02c3f0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.218+0000 7f2f844ef640 1 -- 172.21.15.184:0/4271291617 shutdown_connections 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.218+0000 7f2f844ef640 1 --2- 172.21.15.184:0/4271291617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f68007bd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.218+0000 7f2f844ef640 1 -- 172.21.15.184:0/4271291617 >> 172.21.15.184:0/4271291617 conn(0x7f2f7c0fb9a0 msgr2=0x7f2f7c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.218+0000 7f2f844ef640 1 -- 172.21.15.184:0/4271291617 shutdown_connections 2024-09-12T02:39:48.220 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.218+0000 7f2f844ef640 1 -- 172.21.15.184:0/4271291617 wait complete. 2024-09-12T02:39:48.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 Processor -- start 2024-09-12T02:39:48.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 -- start start 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f7c1951d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c1075b0 con 0x7f2f68005740 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c107430 con 0x7f2f7c195710 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f844ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c107730 con 0x7f2f7c199fe0 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f82a8c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f82a8c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:46240/0 (socket says 172.21.15.184:46240) 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f82a8c640 1 -- 172.21.15.184:0/778557973 learned_addr learned my addr 172.21.15.184:0/778557973 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:48.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.219+0000 7f2f8228b640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f7c1951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f82a8c640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 msgr2=0x7f2f7c19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f82a8c640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 msgr2=0x7f2f7c19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f82a8c640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f81a8a640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f82a8c640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f81a8a640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 msgr2=0x7f2f7c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f81a8a640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 msgr2=0x7f2f7c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f81a8a640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f81a8a640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 msgr2=0x7f2f7c19a490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:39:48.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 msgr2=0x7f2f7c199aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 -- 172.21.15.184:0/778557973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c1a6160 con 0x7f2f68005740 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f8228b640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f7c1951d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c036040 tx=0x7f2f6c034c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.220+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0054e0 con 0x7f2f68005740 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.221+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c003ad0 con 0x7f2f68005740 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.221+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c012a30 con 0x7f2f68005740 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.221+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c1a6330 con 0x7f2f68005740 2024-09-12T02:39:48.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.221+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c1a6890 con 0x7f2f68005740 2024-09-12T02:39:48.226 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.222+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2f6c012bd0 con 0x7f2f68005740 2024-09-12T02:39:48.226 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.222+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c1086a0 con 0x7f2f68005740 2024-09-12T02:39:48.226 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.222+0000 7f2f737fe640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 0x7f2f6003b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:48.227 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.222+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2f6c06e370 con 0x7f2f68005740 2024-09-12T02:39:48.227 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.222+0000 7f2f81a8a640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 0x7f2f6003b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:48.227 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.223+0000 7f2f81a8a640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 0x7f2f6003b9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:48.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.228+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2f6c044300 con 0x7f2f68005740 2024-09-12T02:39:48.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:48.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: mon.a calling monitor election 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: mon.c calling monitor election 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: mon.b calling monitor election 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: monmap epoch 3 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: last_changed 2024-09-12T02:39:43.093117+0000 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: min_mon_release 19 (squid) 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: election_strategy: 1 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: fsmap 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: mgrmap e13: x(active, since 48s) 2024-09-12T02:39:48.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T02:39:48.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.470+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2f7c18e740 con 0x7f2f68005740 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: mon.a calling monitor election 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: mon.c calling monitor election 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: mon.b calling monitor election 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: monmap epoch 3 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: last_changed 2024-09-12T02:39:43.093117+0000 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:48.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: min_mon_release 19 (squid) 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: election_strategy: 1 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: fsmap 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: mgrmap e13: x(active, since 48s) 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.470+0000 7f2f737fe640 1 -- 172.21.15.184:0/778557973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f2f6c003c80 con 0x7f2f68005740 2024-09-12T02:39:48.475 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:39:48.475 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":3,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","modified":"2024-09-12T02:39:43.093117Z","created":"2024-09-12T02:38:06.636712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-12T02:39:48.476 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 3 2024-09-12T02:39:48.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 msgr2=0x7f2f6003b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:48.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 0x7f2f6003b9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78008040 comp rx=0 tx=0).stop 2024-09-12T02:39:48.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 msgr2=0x7f2f7c1951d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f7c1951d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c036040 tx=0x7f2f6c034c50 comp rx=0 tx=0).stop 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 shutdown_connections 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f7c199fe0 0x7f2f7c19a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f7c195710 0x7f2f7c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f600394f0 0x7f2f6003b9b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 --2- 172.21.15.184:0/778557973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f68005740 0x7f2f7c1951d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 >> 172.21.15.184:0/778557973 conn(0x7f2f7c0fb9a0 msgr2=0x7f2f7c105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 shutdown_connections 2024-09-12T02:39:48.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:48.473+0000 7f2f844ef640 1 -- 172.21.15.184:0/778557973 wait complete. 2024-09-12T02:39:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:39:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: mon.a calling monitor election 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: mon.c calling monitor election 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: mon.b calling monitor election 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-12T02:39:48.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: monmap epoch 3 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: last_changed 2024-09-12T02:39:43.093117+0000 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: created 2024-09-12T02:38:06.636712+0000 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: min_mon_release 19 (squid) 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: election_strategy: 1 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: fsmap 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: mgrmap e13: x(active, since 48s) 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:48.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:49.045 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-12T02:39:49.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-12T02:39:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:49 smithi184 ceph-mon[35321]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:49 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/778557973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:49.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:49 smithi016 ceph-mon[30925]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:49.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:49 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/778557973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:49 smithi033 ceph-mon[35092]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:49 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/778557973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-12T02:39:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.728+0000 7f11ca117640 1 -- 172.21.15.16:0/859146856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4107660 msgr2=0x7f11c4107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.728+0000 7f11ca117640 1 --2- 172.21.15.16:0/859146856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4107660 0x7f11c4107a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009880 tx=0x7f11b402a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 -- 172.21.15.16:0/859146856 shutdown_connections 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 --2- 172.21.15.16:0/859146856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4107660 0x7f11c4107a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 -- 172.21.15.16:0/859146856 >> 172.21.15.16:0/859146856 conn(0x7f11c40fd290 msgr2=0x7f11c40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 -- 172.21.15.16:0/859146856 shutdown_connections 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 -- 172.21.15.16:0/859146856 wait complete. 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 Processor -- start 2024-09-12T02:39:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.729+0000 7f11ca117640 1 -- start start 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 0x7f11c4199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 0x7f11c419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410bc90 con 0x7f11c4199a60 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410bb10 con 0x7f11c419e330 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11ca117640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410be10 con 0x7f11c4107660 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48490/0 (socket says 172.21.15.16:48490) 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 -- 172.21.15.16:0/1399069634 learned_addr learned my addr 172.21.15.16:0/1399069634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11c37fe640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 0x7f11c4199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11c3fff640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 0x7f11c419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 -- 172.21.15.16:0/1399069634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 msgr2=0x7f11c4199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 0x7f11c4199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 -- 172.21.15.16:0/1399069634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 msgr2=0x7f11c419e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 0x7f11c419e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 -- 172.21.15.16:0/1399069634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c41a60a0 con 0x7f11c4199a60 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11c3fff640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 0x7f11c419e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11c37fe640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 0x7f11c4199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11baffd640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00b520 tx=0x7f11ac00b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.730+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac004230 con 0x7f11c4199a60 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.731+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ac0043d0 con 0x7f11c4199a60 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.731+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c41a62d0 con 0x7f11c4199a60 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.731+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac003d60 con 0x7f11c4199a60 2024-09-12T02:39:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.731+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c41a6860 con 0x7f11c4199a60 2024-09-12T02:39:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.731+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f11ac00e070 con 0x7f11c4199a60 2024-09-12T02:39:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.732+0000 7f11c17fa640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 0x7f11b003b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.732+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f11ac010b50 con 0x7f11c4199a60 2024-09-12T02:39:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.732+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c4107ac0 con 0x7f11c4199a60 2024-09-12T02:39:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.732+0000 7f11c37fe640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 0x7f11b003b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.732+0000 7f11c37fe640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 0x7f11b003b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11b402ab40 tx=0x7f11b402aad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.737+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f11ac014020 con 0x7f11c4199a60 2024-09-12T02:39:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.918+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f11c41a6b50 con 0x7f11c4199a60 2024-09-12T02:39:49.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.918+0000 7f11c17fa640 1 -- 172.21.15.16:0/1399069634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f11ac01ccd0 con 0x7f11c4199a60 2024-09-12T02:39:49.920 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:49.920 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-12T02:39:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:39:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.920+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 msgr2=0x7f11b003b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.920+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 0x7f11b003b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11b402ab40 tx=0x7f11b402aad0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.920+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 msgr2=0x7f11c419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.920+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00b520 tx=0x7f11ac00b9e0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 shutdown_connections 2024-09-12T02:39:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11c419e330 0x7f11c419e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f11b00394f0 0x7f11b003b9b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c4199a60 0x7f11c419ddf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 --2- 172.21.15.16:0/1399069634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11c4107660 0x7f11c4199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 >> 172.21.15.16:0/1399069634 conn(0x7f11c40fd290 msgr2=0x7f11c4074420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 shutdown_connections 2024-09-12T02:39:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:49.921+0000 7f11ca117640 1 -- 172.21.15.16:0/1399069634 wait complete. 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1399069634' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1399069634' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-12T02:39:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: Updating smithi033:/etc/ceph/ceph.conf 2024-09-12T02:39:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-12T02:39:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1399069634' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:50.880 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-12T02:39:50.880 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:50.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-09-12T02:39:50.908 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:50.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:50.974 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:39:50.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-09-12T02:39:51.001 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:39:51.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:51.066 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:39:51.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-09-12T02:39:51.094 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:39:51.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-12T02:39:51.160 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-09-12T02:39:51.160 INFO:tasks.cephadm:Adding mgr.y on smithi033 2024-09-12T02:39:51.160 INFO:tasks.cephadm:Adding mgr.z on smithi184 2024-09-12T02:39:51.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch apply mgr '3;smithi016=x;smithi033=y;smithi184=z' 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: Updating smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: Reconfiguring mon.a (unknown last config time)... 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:51.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:51.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:51 smithi016 ceph-mon[30925]: Reconfiguring daemon mon.a on smithi016 2024-09-12T02:39:51.434 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:39:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: Updating smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: Reconfiguring mon.a (unknown last config time)... 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:51 smithi184 ceph-mon[35321]: Reconfiguring daemon mon.a on smithi016 2024-09-12T02:39:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: Updating smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: Updating smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: Updating smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:51.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: Reconfiguring mon.a (unknown last config time)... 2024-09-12T02:39:51.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:51.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:51.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:51.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:51 smithi033 ceph-mon[35092]: Reconfiguring daemon mon.a on smithi016 2024-09-12T02:39:51.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 -- 172.21.15.184:0/2426791917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c1005f0 msgr2=0x7fe21c1009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:51.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 --2- 172.21.15.184:0/2426791917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c1005f0 0x7fe21c1009f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe210009880 tx=0x7fe21002a5d0 comp rx=0 tx=0).stop 2024-09-12T02:39:51.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 -- 172.21.15.184:0/2426791917 shutdown_connections 2024-09-12T02:39:51.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 --2- 172.21.15.184:0/2426791917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c1005f0 0x7fe21c1009f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 -- 172.21.15.184:0/2426791917 >> 172.21.15.184:0/2426791917 conn(0x7fe21c0fbda0 msgr2=0x7fe21c0fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 -- 172.21.15.184:0/2426791917 shutdown_connections 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.860+0000 7fe222bc8640 1 -- 172.21.15.184:0/2426791917 wait complete. 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 Processor -- start 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 -- start start 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe21c1005f0 0x7fe21c197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c197550 0x7fe21c19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c106700 con 0x7fe21c197550 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c106580 con 0x7fe21c1005f0 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe222bc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c106880 con 0x7fe21c1979d0 2024-09-12T02:39:51.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe221165640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe221165640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:45438/0 (socket says 172.21.15.184:45438) 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe221165640 1 -- 172.21.15.184:0/23638441 learned_addr learned my addr 172.21.15.184:0/23638441 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.861+0000 7fe220964640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe21c1005f0 0x7fe21c197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe21bfff640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c197550 0x7fe21c19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 -- 172.21.15.184:0/23638441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe21c1005f0 msgr2=0x7fe21c197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe21c1005f0 0x7fe21c197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 -- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c197550 msgr2=0x7fe21c19c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c197550 0x7fe21c19c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 -- 172.21.15.184:0/23638441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21c19f3a0 con 0x7fe21c1979d0 2024-09-12T02:39:51.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe221165640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe21400ba50 tx=0x7fe21400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe214019070 con 0x7fe21c1979d0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe21400cbe0 con 0x7fe21c1979d0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21c19f690 con 0x7fe21c1979d0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140076a0 con 0x7fe21c1979d0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.862+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21c19fc00 con 0x7fe21c1979d0 2024-09-12T02:39:51.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.864+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe21c1907d0 con 0x7fe21c1979d0 2024-09-12T02:39:51.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.864+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe21401c070 con 0x7fe21c1979d0 2024-09-12T02:39:51.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.864+0000 7fe219ffb640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 0x7fe20c03b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:51.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.864+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe21404bb70 con 0x7fe21c1979d0 2024-09-12T02:39:51.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.864+0000 7fe220964640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 0x7fe20c03b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:51.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.865+0000 7fe220964640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 0x7fe20c03b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe210005ab0 tx=0x7fe2100315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:51.870 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:51.869+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe214007de0 con 0x7fe21c1979d0 2024-09-12T02:39:52.046 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.045+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi033=y;smithi184=z", "target": ["mon-mgr", ""]}) -- 0x7fe21c0626c0 con 0x7fe20c0394f0 2024-09-12T02:39:52.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.050+0000 7fe219ffb640 1 -- 172.21.15.184:0/23638441 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe21c0626c0 con 0x7fe20c0394f0 2024-09-12T02:39:52.051 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mgr update... 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 msgr2=0x7fe20c03b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 0x7fe20c03b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe210005ab0 tx=0x7fe2100315d0 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 msgr2=0x7fe21c19eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe21400ba50 tx=0x7fe21400bf10 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 shutdown_connections 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe21c1979d0 0x7fe21c19eae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe20c0394f0 0x7fe20c03b9b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe21c197550 0x7fe21c19c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 --2- 172.21.15.184:0/23638441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe21c1005f0 0x7fe21c197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 >> 172.21.15.184:0/23638441 conn(0x7fe21c0fbda0 msgr2=0x7fe21c06a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 shutdown_connections 2024-09-12T02:39:52.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:39:52.052+0000 7fe222bc8640 1 -- 172.21.15.184:0/23638441 wait complete. 2024-09-12T02:39:52.653 DEBUG:teuthology.orchestra.run.smithi033:mgr.y> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.y.service 2024-09-12T02:39:52.655 DEBUG:teuthology.orchestra.run.smithi184:mgr.z> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.z.service 2024-09-12T02:39:52.658 INFO:tasks.cephadm:Deploying OSDs... 2024-09-12T02:39:52.658 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:39:52.658 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T02:39:52.679 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T02:39:52.679 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-12T02:39:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T02:39:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-12T02:39:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 02:39:10.974606379 +0000 2024-09-12T02:39:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 02:34:58.460663152 +0000 2024-09-12T02:39:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 02:34:58.460663152 +0000 2024-09-12T02:39:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 02:34:58.460663152 +0000 2024-09-12T02:39:52.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T02:39:52.808 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T02:39:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T02:39:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000131075 s, 3.9 MB/s 2024-09-12T02:39:52.810 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T02:39:52.869 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-12T02:39:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T02:39:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:52.928 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 02:39:11.009605833 +0000 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 02:34:58.786658173 +0000 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 02:34:58.786658173 +0000 2024-09-12T02:39:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 02:34:58.786658173 +0000 2024-09-12T02:39:52.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T02:39:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T02:39:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T02:39:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000121631 s, 4.2 MB/s 2024-09-12T02:39:52.998 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T02:39:53.056 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-12T02:39:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T02:39:53.114 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-12T02:39:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:53.114 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:53.115 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 02:39:11.045605272 +0000 2024-09-12T02:39:53.115 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 02:34:59.109653241 +0000 2024-09-12T02:39:53.115 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 02:34:59.109653241 +0000 2024-09-12T02:39:53.115 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 02:34:59.109653241 +0000 2024-09-12T02:39:53.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T02:39:53.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: Reconfiguring mon.b (monmap changed)... 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: Reconfiguring daemon mon.b on smithi033 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi033=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: Saving service mgr spec with placement smithi016=x;smithi033=y;smithi184=z;count:3 2024-09-12T02:39:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T02:39:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T02:39:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000102655 s, 5.0 MB/s 2024-09-12T02:39:53.181 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T02:39:53.238 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-12T02:39:53.294 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 02:39:11.079604742 +0000 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 02:34:59.431648323 +0000 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 02:34:59.431648323 +0000 2024-09-12T02:39:53.295 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 02:34:59.431648323 +0000 2024-09-12T02:39:53.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T02:39:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: Reconfiguring mon.b (monmap changed)... 2024-09-12T02:39:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: Reconfiguring daemon mon.b on smithi033 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi033=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: Saving service mgr spec with placement smithi016=x;smithi033=y;smithi184=z;count:3 2024-09-12T02:39:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T02:39:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T02:39:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000116754 s, 4.4 MB/s 2024-09-12T02:39:53.362 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T02:39:53.419 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:53.419 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: Reconfiguring mon.b (monmap changed)... 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: Reconfiguring daemon mon.b on smithi033 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi033=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: Saving service mgr spec with placement smithi016=x;smithi033=y;smithi184=z;count:3 2024-09-12T02:39:53.420 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:53.421 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-09-12T02:39:53.421 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T02:39:53.438 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T02:39:53.438 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:53.495 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-12 02:39:42.536535451 +0000 2024-09-12T02:39:53.496 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-12 02:35:20.862266201 +0000 2024-09-12T02:39:53.496 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-12 02:35:20.862266201 +0000 2024-09-12T02:39:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-12 02:35:20.862266201 +0000 2024-09-12T02:39:53.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T02:39:53.561 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-12T02:39:53.561 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-12T02:39:53.561 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000270865 s, 1.9 MB/s 2024-09-12T02:39:53.562 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T02:39:53.619 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-09-12T02:39:53.675 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T02:39:53.675 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:53.675 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-12T02:39:53.675 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:53.676 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:53.676 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-12 02:39:42.537535435 +0000 2024-09-12T02:39:53.676 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-12 02:35:21.189260743 +0000 2024-09-12T02:39:53.676 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-12 02:35:21.189260743 +0000 2024-09-12T02:39:53.676 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-12 02:35:21.189260743 +0000 2024-09-12T02:39:53.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T02:39:53.740 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-12T02:39:53.740 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-12T02:39:53.741 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000191526 s, 2.7 MB/s 2024-09-12T02:39:53.742 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T02:39:53.798 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-12 02:39:42.537535435 +0000 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-12 02:35:21.513255335 +0000 2024-09-12T02:39:53.854 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-12 02:35:21.513255335 +0000 2024-09-12T02:39:53.855 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-12 02:35:21.513255335 +0000 2024-09-12T02:39:53.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T02:39:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-12T02:39:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-12T02:39:53.918 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000198554 s, 2.6 MB/s 2024-09-12T02:39:53.919 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T02:39:53.976 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-09-12 02:39:42.538535418 +0000 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-09-12 02:35:21.843249827 +0000 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-09-12 02:35:21.843249827 +0000 2024-09-12T02:39:54.032 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-09-12 02:35:21.843249827 +0000 2024-09-12T02:39:54.032 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T02:39:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-09-12T02:39:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-09-12T02:39:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000217815 s, 2.4 MB/s 2024-09-12T02:39:54.097 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T02:39:54.153 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-12T02:39:54.153 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T02:39:54.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T02:39:54.170 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-09-12T02:39:54.224 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T02:39:54.224 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-12 02:39:33.701099836 +0000 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-12 02:35:09.259596676 +0000 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-12 02:35:09.259596676 +0000 2024-09-12T02:39:54.225 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-12 02:35:09.259596676 +0000 2024-09-12T02:39:54.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T02:39:54.289 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-12T02:39:54.289 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-12T02:39:54.289 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000267478 s, 1.9 MB/s 2024-09-12T02:39:54.290 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T02:39:54.346 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-09-12T02:39:54.401 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T02:39:54.401 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:54.401 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-12T02:39:54.401 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:54.402 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:54.402 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-12 02:39:33.701099836 +0000 2024-09-12T02:39:54.402 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-12 02:35:09.571590135 +0000 2024-09-12T02:39:54.402 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-12 02:35:09.571590135 +0000 2024-09-12T02:39:54.402 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-12 02:35:09.571590135 +0000 2024-09-12T02:39:54.402 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T02:39:54.464 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-12T02:39:54.464 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-12T02:39:54.464 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000169827 s, 3.0 MB/s 2024-09-12T02:39:54.465 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: Reconfiguring mon.c (monmap changed)... 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:54 smithi033 ceph-mon[35092]: Reconfiguring daemon mon.c on smithi184 2024-09-12T02:39:54.520 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: Reconfiguring mon.c (monmap changed)... 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:54.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:54 smithi184 ceph-mon[35321]: Reconfiguring daemon mon.c on smithi184 2024-09-12T02:39:54.554 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T02:39:54.554 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:54.554 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-12 02:39:33.702099815 +0000 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-12 02:35:09.912582987 +0000 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-12 02:35:09.912582987 +0000 2024-09-12T02:39:54.555 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-12 02:35:09.912582987 +0000 2024-09-12T02:39:54.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T02:39:54.617 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-12T02:39:54.618 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-12T02:39:54.618 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000174728 s, 2.9 MB/s 2024-09-12T02:39:54.619 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T02:39:54.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: Reconfiguring mon.c (monmap changed)... 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:54 smithi016 ceph-mon[30925]: Reconfiguring daemon mon.c on smithi184 2024-09-12T02:39:54.676 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-09-12T02:39:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T02:39:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-12 02:39:33.702099815 +0000 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-12 02:35:10.232576278 +0000 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-12 02:35:10.232576278 +0000 2024-09-12T02:39:54.732 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-12 02:35:10.232576278 +0000 2024-09-12T02:39:54.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T02:39:54.795 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-12T02:39:54.796 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-12T02:39:54.796 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000231334 s, 2.2 MB/s 2024-09-12T02:39:54.797 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T02:39:54.854 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-09-12T02:39:54.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-12T02:39:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:55.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:55.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:55.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:55.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:55 smithi184 ceph-mon[35321]: Deploying daemon mgr.z on smithi184 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:55 smithi016 ceph-mon[30925]: Deploying daemon mgr.z on smithi184 2024-09-12T02:39:55.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:56.004 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:56.005 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:55 smithi033 ceph-mon[35092]: Deploying daemon mgr.z on smithi184 2024-09-12T02:39:56.949 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:56.801+0000 7f8cf6b2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-12T02:39:56.950 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:56.948+0000 7f8cf6b2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-12T02:39:57.295 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.014+0000 7f8cf6b2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-12T02:39:57.295 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.158+0000 7f8cf6b2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:39:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:39:57.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-09-12T02:39:57.610 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.298+0000 7f8cf6b2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-12T02:39:57.610 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.609+0000 7f8cf6b2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-12T02:39:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:57.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:57.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:57.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:57 smithi016 ceph-mon[30925]: Deploying daemon mgr.y on smithi033 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:57.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:57.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:57.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:57 smithi033 ceph-mon[35092]: Deploying daemon mgr.y on smithi033 2024-09-12T02:39:58.045 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.669+0000 7f8cf6b2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-12T02:39:58.045 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:57.851+0000 7f8cf6b2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-12T02:39:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:57 smithi184 ceph-mon[35321]: Deploying daemon mgr.y on smithi033 2024-09-12T02:39:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- 172.21.15.16:0/2130021898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 msgr2=0x7ff6b8109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2130021898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b8109d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009ca0 tx=0x7ff6b002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- 172.21.15.16:0/2130021898 shutdown_connections 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2130021898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b8109d50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2130021898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b8104fd0 0x7ff6b81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2130021898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b81026a0 0x7ff6b8104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- 172.21.15.16:0/2130021898 >> 172.21.15.16:0/2130021898 conn(0x7ff6b80fc2a0 msgr2=0x7ff6b80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- 172.21.15.16:0/2130021898 shutdown_connections 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- 172.21.15.16:0/2130021898 wait complete. 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 Processor -- start 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.227+0000 7ff6bf045640 1 -- start start 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b81026a0 0x7ff6b819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b8104fd0 0x7ff6b819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b810d910 con 0x7ff6b8107900 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b810d790 con 0x7ff6b8104fd0 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b810da90 con 0x7ff6b81026a0 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59742/0 (socket says 172.21.15.16:59742) 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 -- 172.21.15.16:0/2168004062 learned_addr learned my addr 172.21.15.16:0/2168004062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bcde1640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b81026a0 0x7ff6b819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6b7fff640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b8104fd0 0x7ff6b819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 -- 172.21.15.16:0/2168004062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b81026a0 msgr2=0x7ff6b819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b81026a0 0x7ff6b819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 -- 172.21.15.16:0/2168004062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b8104fd0 msgr2=0x7ff6b819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b8104fd0 0x7ff6b819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:39:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 -- 172.21.15.16:0/2168004062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b81a1d90 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bd5e2640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0005e30 tx=0x7ff6b0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0003c30 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b0003dd0 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b81a2020 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.228+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b002fde0 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.229+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b81a2630 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.229+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ff6b0004050 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.230+0000 7ff6b5ffb640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 0x7ff6a003b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.230+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff6b006d510 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.230+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b81093f0 con 0x7ff6b8107900 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.230+0000 7ff6bcde1640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 0x7ff6a003b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:39:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.230+0000 7ff6bcde1640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 0x7ff6a003b780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800bbe0 tx=0x7ff6a800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:39:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.235+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff6b002e270 con 0x7ff6b8107900 2024-09-12T02:39:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:39:58.419+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff6b8062630 con 0x7ff6a00392c0 2024-09-12T02:39:58.747 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:58 smithi033 systemd[1]: Starting Ceph mgr.y for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:39:59.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:39:59.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:39:59.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:59.046 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:58.728+0000 7f8cf6b2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-12T02:39:59.046 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:58.795+0000 7f8cf6b2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-12T02:39:59.058 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:39:59.058 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:39:59.058 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:59.059 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:58 smithi033 podman[36445]: 2024-09-12 02:39:58.723331501 +0000 UTC m=+0.015942663 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:39:59.059 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:58 smithi033 podman[36445]: 2024-09-12 02:39:58.83158155 +0000 UTC m=+0.124192761 container create 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-12T02:39:59.059 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 podman[36445]: 2024-09-12 02:39:59.058710569 +0000 UTC m=+0.351321733 container init 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T02:39:59.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:39:59.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:39:59.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:39:59.413 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 podman[36445]: 2024-09-12 02:39:59.061596314 +0000 UTC m=+0.354207466 container start 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-12T02:39:59.413 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mgr[36465]: -- 172.21.15.33:0/1602387215 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb4a0830e0 con 0x55fb4a060800 2024-09-12T02:39:59.413 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 bash[36445]: 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 2024-09-12T02:39:59.414 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 systemd[1]: Started Ceph mgr.y for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:39:59.414 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.186+0000 7fc1a6f38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-12T02:39:59.414 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.254+0000 7fc1a6f38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-12T02:39:59.414 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.412+0000 7fc1a6f38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-12T02:39:59.747 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.482+0000 7fc1a6f38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-12T02:39:59.747 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.643+0000 7fc1a6f38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:39:59.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:59.521+0000 7f8cf6b2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-12T02:39:59.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:59.650+0000 7f8cf6b2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-12T02:39:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:39:59.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:00.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:00.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:00.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:39:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:59.808+0000 7f8cf6b2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:40:00.097 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:39:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:39:59.870+0000 7f8cf6b2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-12T02:40:00.098 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.028+0000 7f8cf6b2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-12T02:40:00.098 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.097+0000 7f8cf6b2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:39:59.800+0000 7fc1a6f38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.108 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.109 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:00.109 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:39:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:00.395 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.157+0000 7f8cf6b2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-12T02:40:00.395 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.324+0000 7f8cf6b2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-12T02:40:00.395 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.394+0000 7f8cf6b2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-12T02:40:00.401 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:00.137+0000 7fc1a6f38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-12T02:40:00.401 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:00.200+0000 7fc1a6f38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-12T02:40:00.401 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:00.400+0000 7fc1a6f38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-12T02:40:00.667 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.461+0000 7f8cf6b2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-12T02:40:00.667 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.543+0000 7f8cf6b2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-12T02:40:00.667 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.606+0000 7f8cf6b2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-12T02:40:00.667 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:00.665+0000 7f8cf6b2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-12T02:40:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T02:40:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: Reconfiguring mgr.x (unknown last config time)... 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: Reconfiguring daemon mgr.x on smithi016 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]: dispatch 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]': finished 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: osdmap e5: 1 total, 0 up, 1 in 2024-09-12T02:40:01.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: Reconfiguring mgr.x (unknown last config time)... 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: Reconfiguring daemon mgr.x on smithi016 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]: dispatch 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]': finished 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: osdmap e5: 1 total, 0 up, 1 in 2024-09-12T02:40:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: Reconfiguring mgr.x (unknown last config time)... 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: Reconfiguring daemon mgr.x on smithi016 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]: dispatch 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1161905027' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfd4081-15d2-46ed-922c-c65451eff4f1"}]': finished 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: osdmap e5: 1 total, 0 up, 1 in 2024-09-12T02:40:01.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:01.747 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:01.333+0000 7fc1a6f38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-12T02:40:01.747 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:01.403+0000 7fc1a6f38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-12T02:40:01.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z[37132]: 2024-09-12T02:40:01.448+0000 7f8cf6b2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-12T02:40:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:01.975+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7ff6b0030940 con 0x7ff6b8107900 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.163+0000 7fc1a6f38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2277907114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: Standby manager daemon z started 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-12T02:40:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:01 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2277907114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: Standby manager daemon z started 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-12T02:40:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:01 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2277907114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: Standby manager daemon z started 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-12T02:40:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:01 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.184:0/3159162106' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:02.653 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.284+0000 7fc1a6f38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-12T02:40:02.653 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.433+0000 7fc1a6f38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-12T02:40:02.653 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.496+0000 7fc1a6f38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-12T02:40:02.653 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.652+0000 7fc1a6f38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-12T02:40:02.951 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.723+0000 7fc1a6f38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-12T02:40:02.951 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.785+0000 7fc1a6f38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-12T02:40:02.951 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:02.950+0000 7fc1a6f38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-mon[35092]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-mon[35092]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:03.018+0000 7fc1a6f38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:03.088+0000 7fc1a6f38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:03.174+0000 7fc1a6f38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-12T02:40:03.237 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:03.236+0000 7fc1a6f38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-12T02:40:03.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:02 smithi184 ceph-mon[35321]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:03.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:02 smithi184 ceph-mon[35321]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-12T02:40:03.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-12T02:40:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:02 smithi016 ceph-mon[30925]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:02 smithi016 ceph-mon[30925]: mgrmap e14: x(active, since 62s), standbys: z 2024-09-12T02:40:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-12T02:40:03.497 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:03.297+0000 7fc1a6f38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-12T02:40:04.497 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y[36461]: 2024-09-12T02:40:04.126+0000 7fc1a6f38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-12T02:40:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:04.985+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7ff6b00433d0 con 0x7ff6b8107900 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: Standby manager daemon y started 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-12T02:40:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:04 smithi033 ceph-mon[35092]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: Standby manager daemon y started 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-12T02:40:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:04 smithi184 ceph-mon[35321]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:05.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:05.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: Standby manager daemon y started 2024-09-12T02:40:05.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-12T02:40:05.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-12T02:40:05.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-12T02:40:05.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:04 smithi016 ceph-mon[30925]: from='mgr.? 172.21.15.33:0/1249920804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-12T02:40:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:05 smithi033 ceph-mon[35092]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-12T02:40:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-12T02:40:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:05 smithi184 ceph-mon[35321]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-12T02:40:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-12T02:40:06.305 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:05 smithi016 ceph-mon[30925]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-12T02:40:06.305 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-12T02:40:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:06 smithi016 ceph-mon[30925]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:06 smithi033 ceph-mon[35092]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:06 smithi184 ceph-mon[35321]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:09.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:09.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:10.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:09 smithi016 ceph-mon[30925]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:10.073 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:09 smithi033 ceph-mon[35092]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:10.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:09 smithi184 ceph-mon[35321]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:11.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-12T02:40:11.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-12T02:40:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-12T02:40:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:11 smithi016 ceph-mon[30925]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:11 smithi016 ceph-mon[30925]: Deploying daemon osd.0 on smithi016 2024-09-12T02:40:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:11 smithi033 ceph-mon[35092]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:11 smithi033 ceph-mon[35092]: Deploying daemon osd.0 on smithi016 2024-09-12T02:40:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:11 smithi184 ceph-mon[35321]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:11 smithi184 ceph-mon[35321]: Deploying daemon osd.0 on smithi016 2024-09-12T02:40:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:13 smithi016 ceph-mon[30925]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:13 smithi033 ceph-mon[35092]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:13 smithi184 ceph-mon[35321]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:15.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:15 smithi016 ceph-mon[30925]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:15 smithi033 ceph-mon[35092]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:15 smithi184 ceph-mon[35321]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:17.142 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:17.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:17.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:17 smithi016 ceph-mon[30925]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:17 smithi033 ceph-mon[35092]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:17 smithi184 ceph-mon[35321]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:20.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:20.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.733+0000 7ff6b5ffb640 1 -- 172.21.15.16:0/2168004062 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff6b8062630 con 0x7ff6a00392c0 2024-09-12T02:40:20.735 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 msgr2=0x7ff6a003b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 0x7ff6a003b780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800bbe0 tx=0x7ff6a800b390 comp rx=0 tx=0).stop 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 msgr2=0x7ff6b81a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0005e30 tx=0x7ff6b0031ee0 comp rx=0 tx=0).stop 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 shutdown_connections 2024-09-12T02:40:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6a00392c0 0x7ff6a003b780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b8107900 0x7ff6b81a1720 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6b8104fd0 0x7ff6b819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 --2- 172.21.15.16:0/2168004062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b81026a0 0x7ff6b819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 >> 172.21.15.16:0/2168004062 conn(0x7ff6b80fc2a0 msgr2=0x7ff6b8101ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 shutdown_connections 2024-09-12T02:40:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:40:20.736+0000 7ff6bf045640 1 -- 172.21.15.16:0/2168004062 wait complete. 2024-09-12T02:40:21.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:21.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:21.363 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.0.service 2024-09-12T02:40:21.366 INFO:tasks.cephadm:Deploying osd.1 on smithi033 with /dev/vg_nvme/lv_4... 2024-09-12T02:40:21.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-12T02:40:21.620 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:40:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:21 smithi033 ceph-mon[35092]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:21 smithi016 ceph-mon[30925]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:21 smithi184 ceph-mon[35321]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:23.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:40:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T02:40:22.890+0000 7f16b2f2c640 -1 osd.0 0 log_to_monitors true 2024-09-12T02:40:23.917 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:23 smithi033 ceph-mon[35092]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:23.917 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:23 smithi033 ceph-mon[35092]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-12T02:40:23.919 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:40:23.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-09-12T02:40:24.132 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:23 smithi016 ceph-mon[30925]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:24.132 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:23 smithi016 ceph-mon[30925]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-12T02:40:24.197 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:40:24.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:23 smithi184 ceph-mon[35321]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:24.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:23 smithi184 ceph-mon[35321]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-12T02:40:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 -- 172.21.15.33:0/2130428217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c100ad0 msgr2=0x7f775c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 --2- 172.21.15.33:0/2130428217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c100ad0 0x7f775c100ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f774c009900 tx=0x7f774c02a410 comp rx=0 tx=0).stop 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 -- 172.21.15.33:0/2130428217 shutdown_connections 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 --2- 172.21.15.33:0/2130428217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c102690 0x7f775c108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 --2- 172.21.15.33:0/2130428217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f775c101cd0 0x7f775c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 --2- 172.21.15.33:0/2130428217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c100ad0 0x7f775c100ed0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 -- 172.21.15.33:0/2130428217 >> 172.21.15.33:0/2130428217 conn(0x7f775c0fc280 msgr2=0x7f775c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 -- 172.21.15.33:0/2130428217 shutdown_connections 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.664+0000 7f776085d640 1 -- 172.21.15.33:0/2130428217 wait complete. 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 Processor -- start 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 -- start start 2024-09-12T02:40:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 0x7f775c10b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f775c101cd0 0x7f775c109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10d8c0 con 0x7f775c100ad0 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10d740 con 0x7f775c102690 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f776085d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10da40 con 0x7f775c101cd0 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48058/0 (socket says 172.21.15.33:48058) 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 -- 172.21.15.33:0/2638702518 learned_addr learned my addr 172.21.15.33:0/2638702518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775a59c640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 0x7f775c10b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 -- 172.21.15.33:0/2638702518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f775c101cd0 msgr2=0x7f775c109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f775c101cd0 0x7f775c109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 -- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 msgr2=0x7f775c10b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 0x7f775c10b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 -- 172.21.15.33:0/2638702518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c10a8a0 con 0x7f775c102690 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775a59c640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 0x7f775c10b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.665+0000 7f775ad9d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f775400cad0 tx=0x7f775400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.666+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775400ae20 con 0x7f775c102690 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.666+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7754005020 con 0x7f775c102690 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.666+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775c1a2110 con 0x7f775c102690 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.666+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775400f6f0 con 0x7f775c102690 2024-09-12T02:40:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.666+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f775c1a26a0 con 0x7f775c102690 2024-09-12T02:40:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.667+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f775c100ed0 con 0x7f775c102690 2024-09-12T02:40:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.668+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f77540051d0 con 0x7f775c102690 2024-09-12T02:40:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.668+0000 7f774b7fe640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 0x7f77440a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.669+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1361+0+0 (secure 0 0 0) 0x7f77540cd8e0 con 0x7f775c102690 2024-09-12T02:40:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.669+0000 7f775a59c640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 0x7f77440a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:24.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.669+0000 7f775a59c640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 0x7f77440a50b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f774c006010 tx=0x7f774c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:40:24.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.672+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7754014030 con 0x7f775c102690 2024-09-12T02:40:24.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:24.855+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f775c062630 con 0x7f77440a2bf0 2024-09-12T02:40:24.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: osdmap e6: 1 total, 0 up, 1 in 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-12T02:40:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: osdmap e6: 1 total, 0 up, 1 in 2024-09-12T02:40:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:40:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:25.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: osdmap e6: 1 total, 0 up, 1 in 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:25.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:25.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:25.860 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T02:40:25.566+0000 7f16ae68c640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T02:40:25.860 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T02:40:25.570+0000 7f16a6cbe640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi016 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: osdmap e7: 1 total, 0 up, 1 in 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:25 smithi016 ceph-mon[30925]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' 2024-09-12T02:40:26.160 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:26.160 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi016 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: osdmap e7: 1 total, 0 up, 1 in 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.161 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:25 smithi033 ceph-mon[35092]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' 2024-09-12T02:40:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi016 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: osdmap e7: 1 total, 0 up, 1 in 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:26.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:25 smithi184 ceph-mon[35321]: from='osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353]' entity='osd.0' 2024-09-12T02:40:27.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:40:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:40:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] boot 2024-09-12T02:40:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: osdmap e8: 1 total, 1 up, 1 in 2024-09-12T02:40:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:26 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] boot 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: osdmap e8: 1 total, 1 up, 1 in 2024-09-12T02:40:27.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:26 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:40:27.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:40:27.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: osd.0 [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] boot 2024-09-12T02:40:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: osdmap e8: 1 total, 1 up, 1 in 2024-09-12T02:40:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:26 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/3062458185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]': finished 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: osdmap e9: 2 total, 1 up, 2 in 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:27 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/3818988095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/3062458185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]': finished 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: osdmap e9: 2 total, 1 up, 2 in 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/3818988095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-12T02:40:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/3062458185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]: dispatch 2024-09-12T02:40:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0"}]': finished 2024-09-12T02:40:28.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: osdmap e9: 2 total, 1 up, 2 in 2024-09-12T02:40:28.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:28.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:27 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/3818988095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:29 smithi016 ceph-mon[30925]: pgmap v43: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:29 smithi033 ceph-mon[35092]: pgmap v43: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:29 smithi184 ceph-mon[35321]: pgmap v43: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:31 smithi016 ceph-mon[30925]: pgmap v44: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:32.196 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:31 smithi033 ceph-mon[35092]: pgmap v44: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:31 smithi184 ceph-mon[35321]: pgmap v44: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:33 smithi016 ceph-mon[30925]: pgmap v45: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:33 smithi033 ceph-mon[35092]: pgmap v45: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:33 smithi184 ceph-mon[35321]: pgmap v45: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:36.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:35 smithi016 ceph-mon[30925]: pgmap v46: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-12T02:40:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:36.201 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:35 smithi033 ceph-mon[35092]: pgmap v46: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:36.201 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-12T02:40:36.201 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:35 smithi184 ceph-mon[35321]: pgmap v46: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-12T02:40:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:37.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:36 smithi016 ceph-mon[30925]: Deploying daemon osd.1 on smithi033 2024-09-12T02:40:37.205 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:36 smithi033 ceph-mon[35092]: Deploying daemon osd.1 on smithi033 2024-09-12T02:40:37.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:36 smithi184 ceph-mon[35321]: Deploying daemon osd.1 on smithi033 2024-09-12T02:40:38.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:37 smithi016 ceph-mon[30925]: pgmap v47: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:38.168 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:37 smithi033 ceph-mon[35092]: pgmap v47: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:37 smithi184 ceph-mon[35321]: pgmap v47: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:39 smithi016 ceph-mon[30925]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:39 smithi033 ceph-mon[35092]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:39 smithi184 ceph-mon[35321]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:41 smithi033 ceph-mon[35092]: pgmap v49: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:41 smithi016 ceph-mon[30925]: pgmap v49: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:41 smithi184 ceph-mon[35321]: pgmap v49: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:43 smithi016 ceph-mon[30925]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:43 smithi033 ceph-mon[35092]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:43 smithi184 ceph-mon[35321]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:45.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.297+0000 7f774b7fe640 1 -- 172.21.15.33:0/2638702518 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f775c062630 con 0x7f77440a2bf0 2024-09-12T02:40:45.298 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 1 on host 'smithi033' 2024-09-12T02:40:45.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 msgr2=0x7f77440a50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:45.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 0x7f77440a50b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f774c006010 tx=0x7f774c000bc0 comp rx=0 tx=0).stop 2024-09-12T02:40:45.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 msgr2=0x7f775c109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f775400cad0 tx=0x7f775400cf90 comp rx=0 tx=0).stop 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 shutdown_connections 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f775c102690 0x7f775c109fe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f775c101cd0 0x7f775c109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77440a2bf0 0x7f77440a50b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 --2- 172.21.15.33:0/2638702518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f775c100ad0 0x7f775c10b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 >> 172.21.15.33:0/2638702518 conn(0x7f775c0fc280 msgr2=0x7f775c105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 shutdown_connections 2024-09-12T02:40:45.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:40:45.300+0000 7f776085d640 1 -- 172.21.15.33:0/2638702518 wait complete. 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: pgmap v51: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: pgmap v51: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:45.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:45.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: pgmap v51: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:45.862 DEBUG:teuthology.orchestra.run.smithi033:osd.1> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.1.service 2024-09-12T02:40:45.864 INFO:tasks.cephadm:Deploying osd.2 on smithi184 with /dev/vg_nvme/lv_4... 2024-09-12T02:40:45.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-12T02:40:46.115 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:40:47.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:47 smithi184 ceph-mon[35321]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:47 smithi033 ceph-mon[35092]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:47 smithi016 ceph-mon[30925]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:48.432 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:48 smithi033 ceph-mon[35092]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.432 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:48 smithi033 ceph-mon[35092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.432 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:40:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T02:40:48.197+0000 7f7e5367c640 -1 osd.1 0 log_to_monitors true 2024-09-12T02:40:48.508 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:40:48.532 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:48 smithi184 ceph-mon[35321]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.532 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:48 smithi184 ceph-mon[35321]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-09-12T02:40:48.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:48 smithi016 ceph-mon[30925]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:48 smithi016 ceph-mon[30925]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-12T02:40:48.790 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:40:49.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- 172.21.15.184:0/1941352450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8107940 msgr2=0x7f35a8109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:49.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 --2- 172.21.15.184:0/1941352450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8107940 0x7f35a8109d30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009ca0 tx=0x7f35a002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- 172.21.15.184:0/1941352450 shutdown_connections 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 --2- 172.21.15.184:0/1941352450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8107940 0x7f35a8109d30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 --2- 172.21.15.184:0/1941352450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8105010 0x7f35a8107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 --2- 172.21.15.184:0/1941352450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 0x7f35a8104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- 172.21.15.184:0/1941352450 >> 172.21.15.184:0/1941352450 conn(0x7f35a80fc280 msgr2=0x7f35a80fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- 172.21.15.184:0/1941352450 shutdown_connections 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- 172.21.15.184:0/1941352450 wait complete. 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 Processor -- start 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.243+0000 7f35ae942640 1 -- start start 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 0x7f35a8102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8107940 0x7f35a8100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:49.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35a810d880 con 0x7f35a8107940 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35a810d700 con 0x7f35a81026e0 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35ae942640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35a810da00 con 0x7f35a8105010 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:52700/0 (socket says 172.21.15.184:52700) 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 -- 172.21.15.184:0/3077865812 learned_addr learned my addr 172.21.15.184:0/3077865812 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a7fff640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 0x7f35a8102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35acedf640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8107940 0x7f35a8100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 -- 172.21.15.184:0/3077865812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 msgr2=0x7f35a8102040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 0x7f35a8102040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 -- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8107940 msgr2=0x7f35a8100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8107940 0x7f35a8100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:40:49.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 -- 172.21.15.184:0/3077865812 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a8101490 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a77fe640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c00d770 tx=0x7f359c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.244+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359c017070 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.245+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a81a2060 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.245+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f359c009c30 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.245+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359c00f620 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.245+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a81a2570 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.246+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a80fdd70 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.247+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f359c01a070 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.247+0000 7f35a57fa640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 0x7f35900a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.248+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1903+0+0 (secure 0 0 0) 0x7f359c0cd120 con 0x7f35a8105010 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.248+0000 7f35a7fff640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 0x7f35900a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.248+0000 7f35a7fff640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 0x7f35900a4fb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3598005fd0 tx=0x7f359800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:40:49.261 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.252+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f359c009de0 con 0x7f35a8105010 2024-09-12T02:40:49.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:40:49.456+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f35a8062630 con 0x7f35900a2af0 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: osdmap e10: 2 total, 1 up, 2 in 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:49.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi033 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:49.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: osdmap e10: 2 total, 1 up, 2 in 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi033 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: osdmap e10: 2 total, 1 up, 2 in 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi033 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 17092M 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:40:49.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:40:50.497 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T02:40:50.116+0000 7f7e4eddc640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T02:40:50.497 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T02:40:50.122+0000 7f7e4740e640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:40:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: osdmap e11: 2 total, 1 up, 2 in 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:50.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:50 smithi033 ceph-mon[35092]: from='osd.1 ' entity='osd.1' 2024-09-12T02:40:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:40:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: osdmap e11: 2 total, 1 up, 2 in 2024-09-12T02:40:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:50.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:50 smithi184 ceph-mon[35321]: from='osd.1 ' entity='osd.1' 2024-09-12T02:40:50.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: osdmap e11: 2 total, 1 up, 2 in 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:40:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:50 smithi016 ceph-mon[30925]: from='osd.1 ' entity='osd.1' 2024-09-12T02:40:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] boot 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: osdmap e12: 2 total, 2 up, 2 in 2024-09-12T02:40:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] boot 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: osdmap e12: 2 total, 2 up, 2 in 2024-09-12T02:40:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:40:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:40:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-09-12T02:40:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:51.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: osd.1 [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] boot 2024-09-12T02:40:51.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: osdmap e12: 2 total, 2 up, 2 in 2024-09-12T02:40:51.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-12T02:40:52.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/285552511' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]': finished 2024-09-12T02:40:52.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: osdmap e13: 3 total, 2 up, 3 in 2024-09-12T02:40:52.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:40:52.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/3319310029' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/285552511' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]': finished 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: osdmap e13: 3 total, 2 up, 3 in 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:40:52.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/3319310029' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/285552511' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]: dispatch 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c715c5-107d-47dc-9c18-bee07a4bc40e"}]': finished 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: osdmap e13: 3 total, 2 up, 3 in 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:40:52.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/3319310029' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:40:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:53 smithi016 ceph-mon[30925]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:54.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:53 smithi033 ceph-mon[35092]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:53 smithi184 ceph-mon[35321]: pgmap v59: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:55 smithi016 ceph-mon[30925]: pgmap v60: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:55 smithi033 ceph-mon[35092]: pgmap v60: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:55 smithi184 ceph-mon[35321]: pgmap v60: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:57 smithi016 ceph-mon[30925]: pgmap v61: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:58.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:57 smithi184 ceph-mon[35321]: pgmap v61: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:40:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:57 smithi033 ceph-mon[35092]: pgmap v61: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:40:59 smithi016 ceph-mon[30925]: pgmap v62: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:00.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:40:59 smithi184 ceph-mon[35321]: pgmap v62: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:00.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:40:59 smithi033 ceph-mon[35092]: pgmap v62: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:00.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-12T02:41:00.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-12T02:41:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-12T02:41:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:02.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:01 smithi184 ceph-mon[35321]: pgmap v63: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:02.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:01 smithi184 ceph-mon[35321]: Deploying daemon osd.2 on smithi184 2024-09-12T02:41:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:01 smithi016 ceph-mon[30925]: pgmap v63: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:01 smithi016 ceph-mon[30925]: Deploying daemon osd.2 on smithi184 2024-09-12T02:41:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:01 smithi033 ceph-mon[35092]: pgmap v63: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:01 smithi033 ceph-mon[35092]: Deploying daemon osd.2 on smithi184 2024-09-12T02:41:04.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:03 smithi184 ceph-mon[35321]: pgmap v64: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:03 smithi016 ceph-mon[30925]: pgmap v64: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:03 smithi033 ceph-mon[35092]: pgmap v64: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:05 smithi184 ceph-mon[35321]: pgmap v65: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:05 smithi016 ceph-mon[30925]: pgmap v65: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:05 smithi033 ceph-mon[35092]: pgmap v65: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:07.156 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:07.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:07.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:07.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:08.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:07 smithi184 ceph-mon[35321]: pgmap v66: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:07 smithi016 ceph-mon[30925]: pgmap v66: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:07 smithi033 ceph-mon[35092]: pgmap v66: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: pgmap v67: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: pgmap v67: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:10.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: pgmap v67: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:10.535 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 2 on host 'smithi184' 2024-09-12T02:41:10.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.534+0000 7f35a57fa640 1 -- 172.21.15.184:0/3077865812 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f35a8062630 con 0x7f35900a2af0 2024-09-12T02:41:10.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 msgr2=0x7f35900a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:10.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 0x7f35900a4fb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3598005fd0 tx=0x7f359800b390 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 msgr2=0x7f35a8100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c00d770 tx=0x7f359c00dc30 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 shutdown_connections 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35900a2af0 0x7f35900a4fb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a8107940 0x7f35a8100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35a8105010 0x7f35a8100690 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 --2- 172.21.15.184:0/3077865812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a81026e0 0x7f35a8102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 >> 172.21.15.184:0/3077865812 conn(0x7f35a80fc280 msgr2=0x7f35a8102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 shutdown_connections 2024-09-12T02:41:10.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:41:10.537+0000 7f35ae942640 1 -- 172.21.15.184:0/3077865812 wait complete. 2024-09-12T02:41:11.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:11.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.109 DEBUG:teuthology.orchestra.run.smithi184:osd.2> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.2.service 2024-09-12T02:41:11.111 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-09-12T02:41:11.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-12T02:41:11.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:11.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.137 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:11.363 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:41:12.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:11 smithi184 ceph-mon[35321]: pgmap v68: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:11 smithi016 ceph-mon[30925]: pgmap v68: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:11 smithi033 ceph-mon[35092]: pgmap v68: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:13.156 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:12 smithi184 ceph-mon[35321]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.156 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:12 smithi184 ceph-mon[35321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.156 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T02:41:12.796+0000 7f94b449b640 -1 osd.2 0 log_to_monitors true 2024-09-12T02:41:13.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:12 smithi016 ceph-mon[30925]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:12 smithi016 ceph-mon[30925]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:12 smithi033 ceph-mon[35092]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:12 smithi033 ceph-mon[35092]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-12T02:41:13.967 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: pgmap v69: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:13.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-12T02:41:13.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: osdmap e14: 3 total, 2 up, 3 in 2024-09-12T02:41:13.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:13.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:13.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:13 smithi016 ceph-mon[30925]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:41:13.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: pgmap v69: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: osdmap e14: 3 total, 2 up, 3 in 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:13 smithi033 ceph-mon[35092]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: pgmap v69: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: osdmap e14: 3 total, 2 up, 3 in 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: from='osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:13 smithi184 ceph-mon[35321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:41:14.300 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:41:14.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- 172.21.15.16:0/3575327340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e41026d0 msgr2=0x7f02e4108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:14.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 --2- 172.21.15.16:0/3575327340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e41026d0 0x7f02e4108f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009ca0 tx=0x7f02dc02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- 172.21.15.16:0/3575327340 shutdown_connections 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 --2- 172.21.15.16:0/3575327340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e41026d0 0x7f02e4108f60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 --2- 172.21.15.16:0/3575327340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e4101d10 0x7f02e4102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 --2- 172.21.15.16:0/3575327340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 0x7f02e4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- 172.21.15.16:0/3575327340 >> 172.21.15.16:0/3575327340 conn(0x7f02e40fc2a0 msgr2=0x7f02e40fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- 172.21.15.16:0/3575327340 shutdown_connections 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- 172.21.15.16:0/3575327340 wait complete. 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 Processor -- start 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.739+0000 7f02e96f6640 1 -- start start 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 0x7f02e4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 0x7f02e419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410c790 con 0x7f02e4101d10 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410c610 con 0x7f02e41026d0 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e96f6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410c910 con 0x7f02e4100b10 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34190/0 (socket says 172.21.15.16:34190) 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 -- 172.21.15.16:0/1296491616 learned_addr learned my addr 172.21.15.16:0/1296491616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e2ffd640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 0x7f02e4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e37fe640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 0x7f02e419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 -- 172.21.15.16:0/1296491616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 msgr2=0x7f02e4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 0x7f02e4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 -- 172.21.15.16:0/1296491616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 msgr2=0x7f02e419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 0x7f02e419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 -- 172.21.15.16:0/1296491616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419fb00 con 0x7f02e4101d10 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e37fe640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 0x7f02e419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02dbfff640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00d950 tx=0x7f02cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.740+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc017070 con 0x7f02e4101d10 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.741+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e419fdf0 con 0x7f02e4101d10 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.741+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02cc009c30 con 0x7f02e4101d10 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.741+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc00f6b0 con 0x7f02e4101d10 2024-09-12T02:41:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.741+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e41a02f0 con 0x7f02e4101d10 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.742+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e4100f10 con 0x7f02e4101d10 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.743+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f02cc0109d0 con 0x7f02e4101d10 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.743+0000 7f02e0ff9640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 0x7f02d00a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.743+0000 7f02e2ffd640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 0x7f02d00a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.744+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2571+0+0 (secure 0 0 0) 0x7f02cc0cd4a0 con 0x7f02e4101d10 2024-09-12T02:41:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.744+0000 7f02e2ffd640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 0x7f02d00a4fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02e4101b70 tx=0x7f02d4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:41:14.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.748+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f02cc009de0 con 0x7f02e4101d10 2024-09-12T02:41:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:14.936+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f02e40fe4f0 con 0x7f02d00a2b20 2024-09-12T02:41:15.045 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T02:41:14.624+0000 7f94afbfb640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T02:41:15.045 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T02:41:14.632+0000 7f94a822d640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: osdmap e15: 3 total, 2 up, 3 in 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi184 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:14 smithi184 ceph-mon[35321]: from='osd.2 ' entity='osd.2' 2024-09-12T02:41:15.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:41:15.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: osdmap e15: 3 total, 2 up, 3 in 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi184 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:14 smithi016 ceph-mon[30925]: from='osd.2 ' entity='osd.2' 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: osdmap e15: 3 total, 2 up, 3 in 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi184 2024-09-12T02:41:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:15.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:14 smithi033 ceph-mon[35092]: from='osd.2 ' entity='osd.2' 2024-09-12T02:41:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: pgmap v72: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] boot 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: osdmap e16: 3 total, 3 up, 3 in 2024-09-12T02:41:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: pgmap v72: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:16.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] boot 2024-09-12T02:41:16.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: osdmap e16: 3 total, 3 up, 3 in 2024-09-12T02:41:16.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: pgmap v72: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: osd.2 [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] boot 2024-09-12T02:41:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: osdmap e16: 3 total, 3 up, 3 in 2024-09-12T02:41:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-12T02:41:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:16 smithi016 ceph-mon[30925]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:16 smithi033 ceph-mon[35092]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:16 smithi184 ceph-mon[35321]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:18.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:17 smithi016 sudo[49849]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:18.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:17 smithi016 sudo[49849]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:18.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:17 smithi016 sudo[49849]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:18.159 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:17 smithi016 sudo[49849]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:18.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: pgmap v74: 0 pgs: ; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:18.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: osdmap e17: 3 total, 3 up, 3 in 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]: dispatch 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]': finished 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: osdmap e18: 4 total, 3 up, 4 in 2024-09-12T02:41:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: pgmap v74: 0 pgs: ; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: osdmap e17: 3 total, 3 up, 3 in 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]: dispatch 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]': finished 2024-09-12T02:41:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: osdmap e18: 4 total, 3 up, 4 in 2024-09-12T02:41:18.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: pgmap v74: 0 pgs: ; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: osdmap e17: 3 total, 3 up, 3 in 2024-09-12T02:41:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T02:41:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]: dispatch 2024-09-12T02:41:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-12T02:41:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2896956338' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24"}]': finished 2024-09-12T02:41:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: osdmap e18: 4 total, 3 up, 4 in 2024-09-12T02:41:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:18.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:18 smithi016 sudo[49901]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:18.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:18 smithi016 sudo[49901]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:18.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:18 smithi016 sudo[49901]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:18.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 02:41:18 smithi016 sudo[49901]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:18.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:18.880+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f02cc097ac0 con 0x7f02e4101d10 2024-09-12T02:41:18.900 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3811020921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:18.901 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:18 smithi033 ceph-mon[35092]: osdmap e19: 4 total, 3 up, 4 in 2024-09-12T02:41:18.901 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:18.901 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:18.901 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43311]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:18.901 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43311]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:18.901 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43311]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:18.901 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43311]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3811020921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:18 smithi016 ceph-mon[30925]: osdmap e19: 4 total, 3 up, 4 in 2024-09-12T02:41:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:19.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3811020921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:19.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:18 smithi184 ceph-mon[35321]: osdmap e19: 4 total, 3 up, 4 in 2024-09-12T02:41:19.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:19.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:19.247 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:19.247 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:19.247 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43314]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:19.247 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 02:41:18 smithi033 sudo[43314]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:19.428 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44272]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:19.428 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44272]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:19.429 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44272]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:19.429 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44272]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:19.795 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44275]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:19.795 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44275]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:19.795 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44275]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:19.795 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 02:41:19 smithi184 sudo[44275]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:19 smithi016 sudo[50282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:19 smithi016 sudo[50282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:19 smithi016 sudo[50282]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:19 smithi016 sudo[50282]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:20.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 ceph-mon[35321]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:20.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 ceph-mon[35321]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44278]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:20.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44278]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:20.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 sudo[50475]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 sudo[50475]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 sudo[50475]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 sudo[50475]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 ceph-mon[30925]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 ceph-mon[30925]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:20.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:20 smithi033 ceph-mon[35092]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:20.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:20 smithi033 ceph-mon[35092]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-12T02:41:20.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:20.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:21.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44281]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:21.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44281]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:21.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44281]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:21.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:20 smithi184 sudo[44281]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43317]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-12T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43317]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43317]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-12T02:41:21.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43320]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 sudo[43320]: pam_unix(sudo:session): session closed for user root 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 500 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-12T02:41:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-12T02:41:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:22 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:22 smithi016 ceph-mon[30925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:22.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:22 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:22.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:22 smithi033 ceph-mon[35092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:22 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-12T02:41:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:22 smithi184 ceph-mon[35321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-12T02:41:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:23 smithi016 ceph-mon[30925]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:23.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:23 smithi033 ceph-mon[35092]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:23.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:23 smithi184 ceph-mon[35321]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:25 smithi016 ceph-mon[30925]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:25.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:25 smithi033 ceph-mon[35092]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:25.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:25 smithi184 ceph-mon[35321]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:26.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:26 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-12T02:41:26.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:26 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:26.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:26 smithi016 ceph-mon[30925]: Deploying daemon osd.3 on smithi016 2024-09-12T02:41:26.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:26 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-12T02:41:26.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:26 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:26.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:26 smithi033 ceph-mon[35092]: Deploying daemon osd.3 on smithi016 2024-09-12T02:41:26.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:26 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-12T02:41:26.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:26 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:26.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:26 smithi184 ceph-mon[35321]: Deploying daemon osd.3 on smithi016 2024-09-12T02:41:27.490 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:27 smithi016 ceph-mon[30925]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:27.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:27 smithi033 ceph-mon[35092]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:27.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:27 smithi184 ceph-mon[35321]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:29 smithi016 ceph-mon[30925]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-12T02:41:29.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:29 smithi033 ceph-mon[35092]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-12T02:41:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:29 smithi184 ceph-mon[35321]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-12T02:41:31.646 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:31 smithi016 ceph-mon[30925]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-12T02:41:31.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:31 smithi033 ceph-mon[35092]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-12T02:41:31.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:31 smithi184 ceph-mon[35321]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-12T02:41:32.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:32.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:32.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:32.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:32.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:32.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:32.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:32.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:32.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:34.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:33 smithi016 ceph-mon[30925]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-12T02:41:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:33 smithi033 ceph-mon[35092]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-12T02:41:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:33 smithi184 ceph-mon[35321]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-12T02:41:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.169+0000 7f02e0ff9640 1 -- 172.21.15.16:0/1296491616 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f02e40fe4f0 con 0x7f02d00a2b20 2024-09-12T02:41:35.170 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 msgr2=0x7f02d00a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 0x7f02d00a4fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02e4101b70 tx=0x7f02d4005f90 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 msgr2=0x7f02e4198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00d950 tx=0x7f02cc00de10 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 shutdown_connections 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02e41026d0 0x7f02e419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f02d00a2b20 0x7f02d00a4fe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4101d10 0x7f02e4198410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 --2- 172.21.15.16:0/1296491616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02e4100b10 0x7f02e4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 >> 172.21.15.16:0/1296491616 conn(0x7f02e40fc2a0 msgr2=0x7f02e40fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:41:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 shutdown_connections 2024-09-12T02:41:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:41:35.172+0000 7f02e96f6640 1 -- 172.21.15.16:0/1296491616 wait complete. 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:35.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:35.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:35.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:35.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:35.843 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.3.service 2024-09-12T02:41:35.846 INFO:tasks.cephadm:Deploying osd.4 on smithi033 with /dev/vg_nvme/lv_3... 2024-09-12T02:41:35.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-12T02:41:36.105 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:41:37.570 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:37 smithi033 ceph-mon[35092]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:37 smithi016 ceph-mon[30925]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:37 smithi184 ceph-mon[35321]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:38.303 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 02:41:38 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T02:41:38.105+0000 7f0bac255640 -1 osd.3 0 log_to_monitors true 2024-09-12T02:41:38.426 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:41:38.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-09-12T02:41:38.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:38 smithi016 ceph-mon[30925]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-12T02:41:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:41:38.745 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:38 smithi033 ceph-mon[35092]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-12T02:41:38.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:38 smithi184 ceph-mon[35321]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/2148851054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac101cd0 msgr2=0x7ff6ac102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/2148851054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac102150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009900 tx=0x7ff6a002a410 comp rx=0 tx=0).stop 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/2148851054 shutdown_connections 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/2148851054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac102690 0x7ff6ac108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/2148851054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac102150 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/2148851054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/2148851054 >> 172.21.15.33:0/2148851054 conn(0x7ff6ac0fc280 msgr2=0x7ff6ac0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/2148851054 shutdown_connections 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.161+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/2148851054 wait complete. 2024-09-12T02:41:39.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 Processor -- start 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 -- start start 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac102690 0x7ff6ac19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10c730 con 0x7ff6ac101cd0 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10c5b0 con 0x7ff6ac100ad0 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.162+0000 7ff6b1dc4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10c8b0 con 0x7ff6ac102690 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6aaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58138/0 (socket says 172.21.15.33:58138) 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6aaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.33:53594/0 (socket says 172.21.15.33:53594) 2024-09-12T02:41:39.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 -- 172.21.15.33:0/1444670050 learned_addr learned my addr 172.21.15.33:0/1444670050 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6abfff640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac102690 0x7ff6ac19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 -- 172.21.15.33:0/1444670050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac102690 msgr2=0x7ff6ac19f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac102690 0x7ff6ac19f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 -- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 msgr2=0x7ff6ac198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 -- 172.21.15.33:0/1444670050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac19fb30 con 0x7ff6ac100ad0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6ab7fe640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c970 tx=0x7ff69c00cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c004220 con 0x7ff6ac100ad0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0043c0 con 0x7ff6ac100ad0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac19fe20 con 0x7ff6ac100ad0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.163+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00f450 con 0x7ff6ac100ad0 2024-09-12T02:41:39.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.164+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac1a0280 con 0x7ff6ac100ad0 2024-09-12T02:41:39.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.165+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac1a0510 con 0x7ff6ac100ad0 2024-09-12T02:41:39.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.166+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff69c0079b0 con 0x7ff6ac100ad0 2024-09-12T02:41:39.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.167+0000 7ff6a8ff9640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 0x7ff6940a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:41:39.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.167+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3373+0+0 (secure 0 0 0) 0x7ff69c0cdce0 con 0x7ff6ac100ad0 2024-09-12T02:41:39.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.167+0000 7ff6aaffd640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 0x7ff6940a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:41:39.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.168+0000 7ff6aaffd640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 0x7ff6940a50f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00314e0 tx=0x7ff6a0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:41:39.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.170+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff69c014030 con 0x7ff6ac100ad0 2024-09-12T02:41:39.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:39.356+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff6ac1a06e0 con 0x7ff6940a2c30 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: osdmap e20: 4 total, 3 up, 4 in 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi016 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:39.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:39.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:39.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-12T02:41:39.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: osdmap e20: 4 total, 3 up, 4 in 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi016 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:39.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:39.908 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T02:41:39.491+0000 7f0ba79b5640 -1 osd.3 0 waiting for initial osdmap 2024-09-12T02:41:39.908 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T02:41:39.497+0000 7f0ba3fef640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: osdmap e20: 4 total, 3 up, 4 in 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi016 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:41:39.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:40.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:40 smithi033 ceph-mon[35092]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:40.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:40 smithi033 ceph-mon[35092]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:41:40.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:40 smithi033 ceph-mon[35092]: osdmap e21: 4 total, 3 up, 4 in 2024-09-12T02:41:40.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:40.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:40 smithi184 ceph-mon[35321]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:40 smithi184 ceph-mon[35321]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:41:40.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:40 smithi184 ceph-mon[35321]: osdmap e21: 4 total, 3 up, 4 in 2024-09-12T02:41:40.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:40.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:40.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:40 smithi016 ceph-mon[30925]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:41:40.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:40 smithi016 ceph-mon[30925]: from='osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:41:40.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:40 smithi016 ceph-mon[30925]: osdmap e21: 4 total, 3 up, 4 in 2024-09-12T02:41:40.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:40.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] boot 2024-09-12T02:41:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: osdmap e22: 4 total, 4 up, 4 in 2024-09-12T02:41:41.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] boot 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: osdmap e22: 4 total, 4 up, 4 in 2024-09-12T02:41:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: osd.3 [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] boot 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: osdmap e22: 4 total, 4 up, 4 in 2024-09-12T02:41:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: osdmap e23: 4 total, 4 up, 4 in 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/607449307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]': finished 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: osdmap e24: 5 total, 4 up, 5 in 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:42 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/573946419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: osdmap e23: 4 total, 4 up, 4 in 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/607449307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]': finished 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: osdmap e24: 5 total, 4 up, 5 in 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:42 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/573946419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: osdmap e23: 4 total, 4 up, 4 in 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/607449307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]: dispatch 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd26143a-5c8f-47b6-8f10-7bc4480b6c6d"}]': finished 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: osdmap e24: 5 total, 4 up, 5 in 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:42.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:42 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/573946419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:41:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:43 smithi033 ceph-mon[35092]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:43 smithi033 ceph-mon[35092]: osdmap e25: 5 total, 4 up, 5 in 2024-09-12T02:41:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:44.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:43 smithi184 ceph-mon[35321]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:44.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:43 smithi184 ceph-mon[35321]: osdmap e25: 5 total, 4 up, 5 in 2024-09-12T02:41:44.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:43 smithi016 ceph-mon[30925]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:43 smithi016 ceph-mon[30925]: osdmap e25: 5 total, 4 up, 5 in 2024-09-12T02:41:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:44.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:44 smithi033 ceph-mon[35092]: osdmap e26: 5 total, 4 up, 5 in 2024-09-12T02:41:44.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:45.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:44 smithi184 ceph-mon[35321]: osdmap e26: 5 total, 4 up, 5 in 2024-09-12T02:41:45.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:44 smithi016 ceph-mon[30925]: osdmap e26: 5 total, 4 up, 5 in 2024-09-12T02:41:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:41:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:45 smithi033 ceph-mon[35092]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:45 smithi184 ceph-mon[35321]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:45 smithi016 ceph-mon[30925]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:47.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:47 smithi033 ceph-mon[35092]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:47 smithi184 ceph-mon[35321]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:47 smithi016 ceph-mon[30925]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:41:49.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:49 smithi033 ceph-mon[35092]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-12T02:41:50.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:49 smithi184 ceph-mon[35321]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-12T02:41:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:49 smithi016 ceph-mon[30925]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-12T02:41:50.979 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-12T02:41:50.980 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:51.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-12T02:41:51.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-12T02:41:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:51.875 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:51 smithi033 ceph-mon[35092]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:41:51.876 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:51 smithi033 ceph-mon[35092]: Deploying daemon osd.4 on smithi033 2024-09-12T02:41:51.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:51 smithi184 ceph-mon[35321]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:41:51.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:51 smithi184 ceph-mon[35321]: Deploying daemon osd.4 on smithi033 2024-09-12T02:41:52.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:51 smithi016 ceph-mon[30925]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:41:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:51 smithi016 ceph-mon[30925]: Deploying daemon osd.4 on smithi033 2024-09-12T02:41:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:53 smithi033 ceph-mon[35092]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:41:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:53 smithi184 ceph-mon[35321]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:41:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:53 smithi016 ceph-mon[30925]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:41:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:55 smithi033 ceph-mon[35092]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:41:56.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:55 smithi184 ceph-mon[35321]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:41:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:55 smithi016 ceph-mon[30925]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:41:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:57.857 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:57 smithi033 ceph-mon[35092]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:41:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:57 smithi184 ceph-mon[35321]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:41:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:57 smithi016 ceph-mon[30925]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:41:59.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.431+0000 7ff6a8ff9640 1 -- 172.21.15.33:0/1444670050 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff6ac1a06e0 con 0x7ff6940a2c30 2024-09-12T02:41:59.432 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 4 on host 'smithi033' 2024-09-12T02:41:59.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 msgr2=0x7ff6940a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 0x7ff6940a50f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00314e0 tx=0x7ff6a0000bc0 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 msgr2=0x7ff6ac197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c970 tx=0x7ff69c00cd70 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 shutdown_connections 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac102690 0x7ff6ac19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff6940a2c30 0x7ff6940a50f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ac101cd0 0x7ff6ac198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 --2- 172.21.15.33:0/1444670050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6ac100ad0 0x7ff6ac197f00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 >> 172.21.15.33:0/1444670050 conn(0x7ff6ac0fc280 msgr2=0x7ff6ac0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 shutdown_connections 2024-09-12T02:41:59.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:41:59.434+0000 7ff6b1dc4640 1 -- 172.21.15.33:0/1444670050 wait complete. 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:41:59.995 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:41:59.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:41:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.022 DEBUG:teuthology.orchestra.run.smithi033:osd.4> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.4.service 2024-09-12T02:42:00.024 INFO:tasks.cephadm:Deploying osd.5 on smithi184 with /dev/vg_nvme/lv_3... 2024-09-12T02:42:00.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:41:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.277 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:00.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:00.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:41:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:01 smithi033 ceph-mon[35092]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:01 smithi016 ceph-mon[30925]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:02.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:01 smithi184 ceph-mon[35321]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:02.556 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:42:02.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-09-12T02:42:02.632 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 02:42:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T02:42:02.559+0000 7f17fd9c3640 -1 osd.4 0 log_to_monitors true 2024-09-12T02:42:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:42:02.966 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:02 smithi033 ceph-mon[35092]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:02.966 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:02 smithi033 ceph-mon[35092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:03.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:02 smithi184 ceph-mon[35321]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:03.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:02 smithi184 ceph-mon[35321]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:03.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:02 smithi016 ceph-mon[30925]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:02 smithi016 ceph-mon[30925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-12T02:42:03.292 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2324840678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 msgr2=0x7fcc00109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2324840678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc00109d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009ca0 tx=0x7fcbf802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2324840678 shutdown_connections 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2324840678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc00109d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2324840678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc00107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2324840678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 0x7fcc00104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2324840678 >> 172.21.15.184:0/2324840678 conn(0x7fcc000fc280 msgr2=0x7fcc000fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2324840678 shutdown_connections 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.291+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2324840678 wait complete. 2024-09-12T02:42:03.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 Processor -- start 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 -- start start 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 0x7fcc00197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc001983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0010d860 con 0x7fcc00105010 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0010d6e0 con 0x7fcc001026e0 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc06d3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0010d9e0 con 0x7fcc00107940 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc052d9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc052d9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:60850/0 (socket says 172.21.15.184:60850) 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc052d9640 1 -- 172.21.15.184:0/2235742761 learned_addr learned my addr 172.21.15.184:0/2235742761 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:42:03.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcc04ad8640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 0x7fcc00197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:03.295 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.292+0000 7fcbfffff640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc001983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:03.295 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 -- 172.21.15.184:0/2235742761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 msgr2=0x7fcc00197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 0x7fcc00197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 -- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 msgr2=0x7fcc001983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc001983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 -- 172.21.15.184:0/2235742761 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0019fae0 con 0x7fcc00107940 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcbfffff640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc001983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc052d9640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8031ae0 tx=0x7fcbf8031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8003c30 con 0x7fcc00107940 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8003dd0 con 0x7fcc00107940 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0019fd70 con 0x7fcc00107940 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf802fcb0 con 0x7fcc00107940 2024-09-12T02:42:03.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc001a0270 con 0x7fcc00107940 2024-09-12T02:42:03.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.293+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc000fdd70 con 0x7fcc00107940 2024-09-12T02:42:03.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.294+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcbf8004050 con 0x7fcc00107940 2024-09-12T02:42:03.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.295+0000 7fcbfdffb640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 0x7fcbf00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:03.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.295+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3805+0+0 (secure 0 0 0) 0x7fcbf80ef320 con 0x7fcc00107940 2024-09-12T02:42:03.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.296+0000 7fcc04ad8640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 0x7fcbf00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:03.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.296+0000 7fcc04ad8640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 0x7fcbf00a50f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400bbe0 tx=0x7fcbf400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:03.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.299+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcbf80bd420 con 0x7fcc00107940 2024-09-12T02:42:03.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:03.480+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fcc00062630 con 0x7fcbf00a2c30 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: osdmap e27: 5 total, 4 up, 5 in 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: osdmap e27: 5 total, 4 up, 5 in 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:04.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:04.247 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T02:42:03.876+0000 7f17f9123640 -1 osd.4 0 waiting for initial osdmap 2024-09-12T02:42:04.247 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T02:42:03.883+0000 7f17f575d640 -1 osd.4 28 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:42:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-12T02:42:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: osdmap e27: 5 total, 4 up, 5 in 2024-09-12T02:42:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:04.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:04.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi033 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: osdmap e28: 5 total, 4 up, 5 in 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi033 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: osdmap e28: 5 total, 4 up, 5 in 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi033 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 8546M 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: osdmap e28: 5 total, 4 up, 5 in 2024-09-12T02:42:05.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:05.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] boot 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: osdmap e29: 5 total, 5 up, 5 in 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/1268327247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]': finished 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: osdmap e30: 6 total, 5 up, 6 in 2024-09-12T02:42:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] boot 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: osdmap e29: 5 total, 5 up, 5 in 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/1268327247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]': finished 2024-09-12T02:42:06.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: osdmap e30: 6 total, 5 up, 6 in 2024-09-12T02:42:06.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:06.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:42:06.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:42:06.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: osd.4 [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] boot 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: osdmap e29: 5 total, 5 up, 5 in 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/1268327247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]: dispatch 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269f5e13-d530-4741-8b1f-7e6313cd1ccf"}]': finished 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: osdmap e30: 6 total, 5 up, 6 in 2024-09-12T02:42:06.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:06 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/2142200829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:06 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/2142200829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:07.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:06 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/2142200829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:07 smithi033 ceph-mon[35092]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:07 smithi184 ceph-mon[35321]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:07 smithi016 ceph-mon[30925]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:08 smithi033 ceph-mon[35092]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:08 smithi184 ceph-mon[35321]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:09.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:08 smithi016 ceph-mon[30925]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:11 smithi184 ceph-mon[35321]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:11.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:11 smithi016 ceph-mon[30925]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:11.328 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:11 smithi033 ceph-mon[35092]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:13.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:13 smithi184 ceph-mon[35321]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:13.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:13 smithi016 ceph-mon[30925]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:13.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:13 smithi033 ceph-mon[35092]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:15 smithi184 ceph-mon[35321]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-12T02:42:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:15.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:15 smithi184 ceph-mon[35321]: Deploying daemon osd.5 on smithi184 2024-09-12T02:42:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:15 smithi016 ceph-mon[30925]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-12T02:42:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:15 smithi016 ceph-mon[30925]: Deploying daemon osd.5 on smithi184 2024-09-12T02:42:15.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:15 smithi033 ceph-mon[35092]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:15.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-12T02:42:15.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:15.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:15 smithi033 ceph-mon[35092]: Deploying daemon osd.5 on smithi184 2024-09-12T02:42:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:17 smithi184 ceph-mon[35321]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:17.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:17 smithi016 ceph-mon[30925]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:17.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:17 smithi033 ceph-mon[35092]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:19 smithi184 ceph-mon[35321]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:19 smithi016 ceph-mon[30925]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:19 smithi033 ceph-mon[35092]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:21 smithi184 ceph-mon[35321]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:21.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:21.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:21 smithi016 ceph-mon[30925]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:21.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:21 smithi033 ceph-mon[35092]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:21.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:21.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:21.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:23.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:23.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:23.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.968 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.966+0000 7fcbfdffb640 1 -- 172.21.15.184:0/2235742761 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc00062630 con 0x7fcbf00a2c30 2024-09-12T02:42:23.969 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 5 on host 'smithi184' 2024-09-12T02:42:23.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 msgr2=0x7fcbf00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 0x7fcbf00a50f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400bbe0 tx=0x7fcbf400b390 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 msgr2=0x7fcc0019f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8031ae0 tx=0x7fcbf8031ee0 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 shutdown_connections 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc00107940 0x7fcc0019f470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbf00a2c30 0x7fcbf00a50f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00105010 0x7fcc001983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 --2- 172.21.15.184:0/2235742761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcc001026e0 0x7fcc00197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:23.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 >> 172.21.15.184:0/2235742761 conn(0x7fcc000fc280 msgr2=0x7fcc001030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:42:23.971 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 shutdown_connections 2024-09-12T02:42:23.971 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:42:23.968+0000 7fcc06d3c640 1 -- 172.21.15.184:0/2235742761 wait complete. 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.573 DEBUG:teuthology.orchestra.run.smithi184:osd.5> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.5.service 2024-09-12T02:42:24.575 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-09-12T02:42:24.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-12T02:42:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.842 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:42:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:25.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:25 smithi184 ceph-mon[35321]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:25 smithi033 ceph-mon[35092]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:26.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:25 smithi016 ceph-mon[30925]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:26.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:26 smithi184 ceph-mon[35321]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:26.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:26 smithi184 ceph-mon[35321]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:26.967 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 02:42:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T02:42:26.620+0000 7f76e6232640 -1 osd.5 0 log_to_monitors true 2024-09-12T02:42:26.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:26 smithi033 ceph-mon[35092]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:26.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:26 smithi033 ceph-mon[35092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:27.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:26 smithi016 ceph-mon[30925]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:27.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:26 smithi016 ceph-mon[30925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-12T02:42:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:42:27.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-09-12T02:42:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:42:27.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: osdmap e31: 6 total, 5 up, 6 in 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:27.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:27.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:27.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:27.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:27.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:27.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: osdmap e31: 6 total, 5 up, 6 in 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:28.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:28.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 -- 172.21.15.16:0/1722115148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c0074b40 msgr2=0x7f68c0074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 --2- 172.21.15.16:0/1722115148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c0074b40 0x7f68c0074fa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009900 tx=0x7f68bc02a410 comp rx=0 tx=0).stop 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 -- 172.21.15.16:0/1722115148 shutdown_connections 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 --2- 172.21.15.16:0/1722115148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0109a70 0x7f68c010bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 --2- 172.21.15.16:0/1722115148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c0074b40 0x7f68c0074fa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 --2- 172.21.15.16:0/1722115148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c00761a0 0x7f68c0074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 -- 172.21.15.16:0/1722115148 >> 172.21.15.16:0/1722115148 conn(0x7f68c00fd3b0 msgr2=0x7f68c00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 -- 172.21.15.16:0/1722115148 shutdown_connections 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.131+0000 7f68c74c2640 1 -- 172.21.15.16:0/1722115148 wait complete. 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 Processor -- start 2024-09-12T02:42:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 -- start start 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0074b40 0x7f68c019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c0109a70 0x7f68c01a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68c010fad0 con 0x7f68c00761a0 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68c010f950 con 0x7f68c0074b40 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c74c2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68c010fc50 con 0x7f68c0109a70 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53776/0 (socket says 172.21.15.16:53776) 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 -- 172.21.15.16:0/3710541218 learned_addr learned my addr 172.21.15.16:0/3710541218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c5a5f640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c0109a70 0x7f68c01a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c525e640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0074b40 0x7f68c019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 -- 172.21.15.16:0/3710541218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c0109a70 msgr2=0x7f68c01a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c0109a70 0x7f68c01a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 -- 172.21.15.16:0/3710541218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0074b40 msgr2=0x7f68c019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0074b40 0x7f68c019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.132+0000 7f68c4a5d640 1 -- 172.21.15.16:0/3710541218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c01a1d50 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68c4a5d640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009230 tx=0x7f68bc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc03f030 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c01a1fe0 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68bc003ce0 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc035070 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.133+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c01a24b0 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.134+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c0108880 con 0x7f68c00761a0 2024-09-12T02:42:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.136+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f68bc042030 con 0x7f68c00761a0 2024-09-12T02:42:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.137+0000 7f68b27fc640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 0x7f68ac0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.137+0000 7f68c525e640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 0x7f68ac0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.137+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4286+0+0 (secure 0 0 0) 0x7f68bc0eeb50 con 0x7f68c00761a0 2024-09-12T02:42:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.138+0000 7f68c525e640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 0x7f68ac0a5070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68b400bbe0 tx=0x7f68b400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.140+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f68bc0bc380 con 0x7f68c00761a0 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: osdmap e31: 6 total, 5 up, 6 in 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:28.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:28.326+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f68c0062630 con 0x7f68ac0a2bb0 2024-09-12T02:42:28.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T02:42:28.534+0000 7f76e1992640 -1 osd.5 0 waiting for initial osdmap 2024-09-12T02:42:28.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T02:42:28.542+0000 7f76d9fc4640 -1 osd.5 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi184 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: osdmap e32: 6 total, 5 up, 6 in 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: osdmap e33: 6 total, 5 up, 6 in 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:28.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:28 smithi184 ceph-mon[35321]: from='osd.5 ' entity='osd.5' 2024-09-12T02:42:28.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi184 2024-09-12T02:42:28.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: osdmap e32: 6 total, 5 up, 6 in 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: osdmap e33: 6 total, 5 up, 6 in 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:28.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:28 smithi016 ceph-mon[30925]: from='osd.5 ' entity='osd.5' 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi184 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: osdmap e32: 6 total, 5 up, 6 in 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: osdmap e33: 6 total, 5 up, 6 in 2024-09-12T02:42:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:28 smithi033 ceph-mon[35092]: from='osd.5 ' entity='osd.5' 2024-09-12T02:42:30.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] boot 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: osdmap e34: 6 total, 6 up, 6 in 2024-09-12T02:42:30.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:42:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:42:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:30.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:30.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:30.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] boot 2024-09-12T02:42:30.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: osdmap e34: 6 total, 6 up, 6 in 2024-09-12T02:42:30.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: osd.5 [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] boot 2024-09-12T02:42:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: osdmap e34: 6 total, 6 up, 6 in 2024-09-12T02:42:30.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-12T02:42:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:30 smithi016 ceph-mon[30925]: osdmap e35: 6 total, 6 up, 6 in 2024-09-12T02:42:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]: dispatch 2024-09-12T02:42:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]': finished 2024-09-12T02:42:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:30 smithi016 ceph-mon[30925]: osdmap e36: 7 total, 6 up, 7 in 2024-09-12T02:42:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:30 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:31.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:30 smithi033 ceph-mon[35092]: osdmap e35: 6 total, 6 up, 6 in 2024-09-12T02:42:31.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:30 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]: dispatch 2024-09-12T02:42:31.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:30 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]': finished 2024-09-12T02:42:31.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:30 smithi033 ceph-mon[35092]: osdmap e36: 7 total, 6 up, 7 in 2024-09-12T02:42:31.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:30 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:30 smithi184 ceph-mon[35321]: osdmap e35: 6 total, 6 up, 6 in 2024-09-12T02:42:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:30 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]: dispatch 2024-09-12T02:42:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:30 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2525837866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0744cd3c-8a0c-42f5-9e43-bfe91d615dc2"}]': finished 2024-09-12T02:42:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:30 smithi184 ceph-mon[35321]: osdmap e36: 7 total, 6 up, 7 in 2024-09-12T02:42:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:30 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:31 smithi016 ceph-mon[30925]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:31 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2376640277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:31 smithi016 ceph-mon[30925]: osdmap e37: 7 total, 6 up, 7 in 2024-09-12T02:42:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:31 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:31 smithi033 ceph-mon[35092]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:31 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2376640277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:31 smithi033 ceph-mon[35092]: osdmap e37: 7 total, 6 up, 7 in 2024-09-12T02:42:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:31 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:31 smithi184 ceph-mon[35321]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:31 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2376640277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:31 smithi184 ceph-mon[35321]: osdmap e37: 7 total, 6 up, 7 in 2024-09-12T02:42:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:31 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:33 smithi033 ceph-mon[35092]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-12T02:42:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:33 smithi033 ceph-mon[35092]: osdmap e38: 7 total, 6 up, 7 in 2024-09-12T02:42:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:33 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:33 smithi184 ceph-mon[35321]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-12T02:42:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:33 smithi184 ceph-mon[35321]: osdmap e38: 7 total, 6 up, 7 in 2024-09-12T02:42:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:33 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:33 smithi016 ceph-mon[30925]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-12T02:42:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:33 smithi016 ceph-mon[30925]: osdmap e38: 7 total, 6 up, 7 in 2024-09-12T02:42:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:33 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:35 smithi016 ceph-mon[30925]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-09-12T02:42:35.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:35 smithi033 ceph-mon[35092]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-09-12T02:42:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:35 smithi184 ceph-mon[35321]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-09-12T02:42:37.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:37 smithi033 ceph-mon[35092]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:42:37.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:37 smithi016 ceph-mon[30925]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:42:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:37 smithi184 ceph-mon[35321]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:42:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:39 smithi033 ceph-mon[35092]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:42:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-12T02:42:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:40.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:39 smithi016 ceph-mon[30925]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:42:40.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-12T02:42:40.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:39 smithi184 ceph-mon[35321]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:42:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-12T02:42:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:41.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:40 smithi016 ceph-mon[30925]: Deploying daemon osd.6 on smithi016 2024-09-12T02:42:41.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:40 smithi184 ceph-mon[35321]: Deploying daemon osd.6 on smithi016 2024-09-12T02:42:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:40 smithi033 ceph-mon[35092]: Deploying daemon osd.6 on smithi016 2024-09-12T02:42:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:41 smithi184 ceph-mon[35321]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:42.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:41 smithi016 ceph-mon[30925]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:42.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:41 smithi033 ceph-mon[35092]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:43 smithi184 ceph-mon[35321]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:44.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:43 smithi016 ceph-mon[30925]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:43 smithi033 ceph-mon[35092]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:45 smithi184 ceph-mon[35321]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:46.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:46.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:45 smithi016 ceph-mon[30925]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:46.067 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:46.067 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:46.067 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:45 smithi033 ceph-mon[35092]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:47 smithi184 ceph-mon[35321]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:47 smithi016 ceph-mon[30925]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:47 smithi033 ceph-mon[35092]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:48.604 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-09-12T02:42:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.603+0000 7f68b27fc640 1 -- 172.21.15.16:0/3710541218 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f68c0062630 con 0x7f68ac0a2bb0 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 msgr2=0x7f68ac0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 0x7f68ac0a5070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68b400bbe0 tx=0x7f68b400b390 comp rx=0 tx=0).stop 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 msgr2=0x7f68c019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009230 tx=0x7f68bc004360 comp rx=0 tx=0).stop 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 shutdown_connections 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68c0109a70 0x7f68c01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68ac0a2bb0 0x7f68ac0a5070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c00761a0 0x7f68c019a660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 --2- 172.21.15.16:0/3710541218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68c0074b40 0x7f68c019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.605+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 >> 172.21.15.16:0/3710541218 conn(0x7f68c00fd3b0 msgr2=0x7f68c0105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:42:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.606+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 shutdown_connections 2024-09-12T02:42:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:42:48.606+0000 7f68c74c2640 1 -- 172.21.15.16:0/3710541218 wait complete. 2024-09-12T02:42:49.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:49.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:49.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:49.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:48 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:48 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.260 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.6.service 2024-09-12T02:42:49.263 INFO:tasks.cephadm:Deploying osd.7 on smithi033 with /dev/vg_nvme/lv_2... 2024-09-12T02:42:49.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-12T02:42:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:48 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:49.524 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:42:50.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:49 smithi033 ceph-mon[35092]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:49 smithi184 ceph-mon[35321]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:49 smithi016 ceph-mon[30925]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:50 smithi033 ceph-mon[35092]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:50 smithi184 ceph-mon[35321]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:50 smithi016 ceph-mon[30925]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:51.746 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 02:42:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T02:42:51.626+0000 7f634ee45640 -1 osd.6 0 log_to_monitors true 2024-09-12T02:42:51.962 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:42:51.990 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-09-12T02:42:52.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:51 smithi016 ceph-mon[30925]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-12T02:42:52.244 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:42:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:51 smithi033 ceph-mon[35092]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-12T02:42:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:51 smithi184 ceph-mon[35321]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-12T02:42:52.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 -- 172.21.15.33:0/4047261467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8102e00 msgr2=0x7f15e8103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:52.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 --2- 172.21.15.33:0/4047261467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8102e00 0x7f15e8103280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009900 tx=0x7f15d802a410 comp rx=0 tx=0).stop 2024-09-12T02:42:52.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 -- 172.21.15.33:0/4047261467 shutdown_connections 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 --2- 172.21.15.33:0/4047261467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e81037c0 0x7f15e810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 --2- 172.21.15.33:0/4047261467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8102e00 0x7f15e8103280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 --2- 172.21.15.33:0/4047261467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8101c00 0x7f15e8102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 -- 172.21.15.33:0/4047261467 >> 172.21.15.33:0/4047261467 conn(0x7f15e80fd390 msgr2=0x7f15e80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 -- 172.21.15.33:0/4047261467 shutdown_connections 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.803+0000 7f15f0738640 1 -- 172.21.15.33:0/4047261467 wait complete. 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 Processor -- start 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 -- start start 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e8102e00 0x7f15e81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e81037c0 0x7f15e819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:52.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15e810d8e0 con 0x7f15e81037c0 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15e810d760 con 0x7f15e8101c00 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15f0738640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f15e810da60 con 0x7f15e8102e00 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37740/0 (socket says 172.21.15.33:37740) 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 -- 172.21.15.33:0/1102221489 learned_addr learned my addr 172.21.15.33:0/1102221489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15eecd5640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e81037c0 0x7f15e819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 -- 172.21.15.33:0/1102221489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e8102e00 msgr2=0x7f15e81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e8102e00 0x7f15e81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 -- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e81037c0 msgr2=0x7f15e819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.804+0000 7f15ee4d4640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e81037c0 0x7f15e819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15ee4d4640 1 -- 172.21.15.33:0/1102221489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e819fb90 con 0x7f15e8101c00 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15ee4d4640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f15e400d770 tx=0x7f15e400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:52.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4017070 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15e4009c30 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e819fe80 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4003dd0 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.805+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e81a03b0 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.806+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e8108fd0 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.807+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f15e4003f70 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.808+0000 7f15df7fe640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 0x7f15d40a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.808+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4669+0+0 (secure 0 0 0) 0x7f15e40ce070 con 0x7f15e8101c00 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.808+0000 7f15edcd3640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 0x7f15d40a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:42:52.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.809+0000 7f15edcd3640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 0x7f15d40a5070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15d80096c0 tx=0x7f15d8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:42:52.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:52.811+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f15e4009de0 con 0x7f15e8101c00 2024-09-12T02:42:53.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:42:53.000+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f15e8062630 con 0x7f15d40a2bb0 2024-09-12T02:42:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: osdmap e39: 7 total, 6 up, 7 in 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi016 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-12T02:42:53.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:53.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:53.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-12T02:42:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: osdmap e39: 7 total, 6 up, 7 in 2024-09-12T02:42:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:42:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi016 2024-09-12T02:42:53.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T02:42:52.991+0000 7f634a5a5640 -1 osd.6 0 waiting for initial osdmap 2024-09-12T02:42:53.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T02:42:53.003+0000 7f6346bdf640 -1 osd.6 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:42:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-12T02:42:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: osdmap e39: 7 total, 6 up, 7 in 2024-09-12T02:42:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:42:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi016 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:42:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:42:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:42:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: osdmap e40: 7 total, 6 up, 7 in 2024-09-12T02:42:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:54.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:54.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:54.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: osdmap e40: 7 total, 6 up, 7 in 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:54.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: osdmap e40: 7 total, 6 up, 7 in 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:42:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] boot 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: osdmap e41: 7 total, 7 up, 7 in 2024-09-12T02:42:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:55 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:55.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:42:55.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:42:55.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:55.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:55.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] boot 2024-09-12T02:42:55.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: osdmap e41: 7 total, 7 up, 7 in 2024-09-12T02:42:55.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:55 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: osd.6 [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] boot 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: osdmap e41: 7 total, 7 up, 7 in 2024-09-12T02:42:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:55 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-12T02:42:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: osdmap e42: 7 total, 7 up, 7 in 2024-09-12T02:42:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/1670174528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]': finished 2024-09-12T02:42:56.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: osdmap e43: 8 total, 7 up, 8 in 2024-09-12T02:42:56.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:42:56.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:56 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/3368751744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: osdmap e42: 7 total, 7 up, 7 in 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/1670174528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]': finished 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: osdmap e43: 8 total, 7 up, 8 in 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:42:56.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:56 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/3368751744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: osdmap e42: 7 total, 7 up, 7 in 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/1670174528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]: dispatch 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "165df37e-3ffb-4d76-91b3-a15d9eaaa233"}]': finished 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: osdmap e43: 8 total, 7 up, 8 in 2024-09-12T02:42:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:42:56.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:56 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/3368751744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:42:57.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:57 smithi033 ceph-mon[35092]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:42:57.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:57 smithi184 ceph-mon[35321]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:42:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:57 smithi016 ceph-mon[30925]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:42:59.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:42:59 smithi016 ceph-mon[30925]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:42:59.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:42:59 smithi033 ceph-mon[35092]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:42:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:42:59 smithi184 ceph-mon[35321]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:01 smithi033 ceph-mon[35092]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:01 smithi184 ceph-mon[35321]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:01 smithi016 ceph-mon[30925]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:03.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:03 smithi033 ceph-mon[35092]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:03 smithi184 ceph-mon[35321]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:03 smithi016 ceph-mon[30925]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:04.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-12T02:43:04.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:04.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-12T02:43:04.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:04.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-12T02:43:04.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:05 smithi033 ceph-mon[35092]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:05 smithi033 ceph-mon[35092]: Deploying daemon osd.7 on smithi033 2024-09-12T02:43:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:05 smithi184 ceph-mon[35321]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:05 smithi184 ceph-mon[35321]: Deploying daemon osd.7 on smithi033 2024-09-12T02:43:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:05 smithi016 ceph-mon[30925]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:05 smithi016 ceph-mon[30925]: Deploying daemon osd.7 on smithi033 2024-09-12T02:43:07.631 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:07 smithi033 ceph-mon[35092]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:07 smithi184 ceph-mon[35321]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:07 smithi016 ceph-mon[30925]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:09 smithi033 ceph-mon[35092]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:09 smithi184 ceph-mon[35321]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:09.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:09 smithi016 ceph-mon[30925]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:10.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:10.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:10.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:11.931 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:11 smithi033 ceph-mon[35092]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:12.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:11 smithi016 ceph-mon[30925]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:11 smithi184 ceph-mon[35321]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:13.395 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 7 on host 'smithi033' 2024-09-12T02:43:13.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.394+0000 7f15df7fe640 1 -- 172.21.15.33:0/1102221489 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f15e8062630 con 0x7f15d40a2bb0 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 msgr2=0x7f15d40a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 0x7f15d40a5070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15d80096c0 tx=0x7f15d8031400 comp rx=0 tx=0).stop 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 msgr2=0x7f15e8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f15e400d770 tx=0x7f15e400dc30 comp rx=0 tx=0).stop 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 shutdown_connections 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f15d40a2bb0 0x7f15d40a5070 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e81037c0 0x7f15e819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:13.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e8102e00 0x7f15e81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:13.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 --2- 172.21.15.33:0/1102221489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15e8101c00 0x7f15e8197f60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:13.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 >> 172.21.15.33:0/1102221489 conn(0x7f15e80fd390 msgr2=0x7f15e81061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:43:13.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 shutdown_connections 2024-09-12T02:43:13.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:43:13.396+0000 7f15f0738640 1 -- 172.21.15.33:0/1102221489 wait complete. 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:13.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:13.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:14.146 DEBUG:teuthology.orchestra.run.smithi033:osd.7> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.7.service 2024-09-12T02:43:14.148 INFO:tasks.cephadm:Deploying osd.8 on smithi184 with /dev/vg_nvme/lv_2... 2024-09-12T02:43:14.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-12T02:43:14.403 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:43:15.651 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:15 smithi184 ceph-mon[35321]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:15.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:15 smithi033 ceph-mon[35092]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:15 smithi016 ceph-mon[30925]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:16.337 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 02:43:16 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T02:43:16.201+0000 7f21e018a640 -1 osd.7 0 log_to_monitors true 2024-09-12T02:43:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:16 smithi184 ceph-mon[35321]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:16 smithi184 ceph-mon[35321]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:16.596 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:16 smithi033 ceph-mon[35092]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:16.596 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:16 smithi033 ceph-mon[35092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:16.755 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:43:16.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-09-12T02:43:16.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:16 smithi016 ceph-mon[30925]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:16 smithi016 ceph-mon[30925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-12T02:43:17.027 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 -- 172.21.15.184:0/1234767809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b41026d0 msgr2=0x7fa5b4108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 --2- 172.21.15.184:0/1234767809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b41026d0 0x7fa5b4108f60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009ca0 tx=0x7fa5ac02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 -- 172.21.15.184:0/1234767809 shutdown_connections 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 --2- 172.21.15.184:0/1234767809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b41026d0 0x7fa5b4108f60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 --2- 172.21.15.184:0/1234767809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 0x7fa5b4102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 --2- 172.21.15.184:0/1234767809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b4100b10 0x7fa5b4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 -- 172.21.15.184:0/1234767809 >> 172.21.15.184:0/1234767809 conn(0x7fa5b40fc2a0 msgr2=0x7fa5b40fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:43:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 -- 172.21.15.184:0/1234767809 shutdown_connections 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.474+0000 7fa5bb562640 1 -- 172.21.15.184:0/1234767809 wait complete. 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 Processor -- start 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 -- start start 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 0x7fa5b4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b41026d0 0x7fa5b419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b410c7c0 con 0x7fa5b41026d0 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b410c640 con 0x7fa5b4101d10 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5bb562640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b410c940 con 0x7fa5b4100b10 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5b92fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5b92fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:36908/0 (socket says 172.21.15.184:36908) 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5b92fe640 1 -- 172.21.15.184:0/3482935692 learned_addr learned my addr 172.21.15.184:0/3482935692 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:43:17.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5b8afd640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 0x7fa5b4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.475+0000 7fa5b9aff640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b41026d0 0x7fa5b419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 -- 172.21.15.184:0/3482935692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 msgr2=0x7fa5b4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 0x7fa5b4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 -- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b41026d0 msgr2=0x7fa5b419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b41026d0 0x7fa5b419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 -- 172.21.15.184:0/3482935692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b419faa0 con 0x7fa5b4100b10 2024-09-12T02:43:17.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.476+0000 7fa5b92fe640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e9c0 tx=0x7fa5b000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:43:17.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.478+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000cd80 con 0x7fa5b4100b10 2024-09-12T02:43:17.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.478+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5b00035f0 con 0x7fa5b4100b10 2024-09-12T02:43:17.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.478+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b419fd90 con 0x7fa5b4100b10 2024-09-12T02:43:17.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.478+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0010690 con 0x7fa5b4100b10 2024-09-12T02:43:17.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.478+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b41a0240 con 0x7fa5b4100b10 2024-09-12T02:43:17.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.479+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b41a04d0 con 0x7fa5b4100b10 2024-09-12T02:43:17.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.480+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa5b00037a0 con 0x7fa5b4100b10 2024-09-12T02:43:17.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.481+0000 7fa5aa7fc640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 0x7fa5a00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:17.482 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.481+0000 7fa5b8afd640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 0x7fa5a00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:17.482 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.481+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5117+0+0 (secure 0 0 0) 0x7fa5b00ce6c0 con 0x7fa5b4100b10 2024-09-12T02:43:17.482 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.481+0000 7fa5b8afd640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 0x7fa5a00a50f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4199390 tx=0x7fa5a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:43:17.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.485+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa5b009c2a0 con 0x7fa5b4100b10 2024-09-12T02:43:17.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:17.658+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa5b40fe480 con 0x7fa5a00a2c30 2024-09-12T02:43:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-12T02:43:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: osdmap e44: 8 total, 7 up, 8 in 2024-09-12T02:43:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi033 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:17.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:17.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: osdmap e44: 8 total, 7 up, 8 in 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi033 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-12T02:43:17.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: osdmap e44: 8 total, 7 up, 8 in 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:43:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi033 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 5697M 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:18.747 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T02:43:18.261+0000 7f21db8ea640 -1 osd.7 0 waiting for initial osdmap 2024-09-12T02:43:18.747 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T02:43:18.271+0000 7f21d3f1c640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:43:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:43:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: osdmap e45: 8 total, 7 up, 8 in 2024-09-12T02:43:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: osdmap e46: 8 total, 7 up, 8 in 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:18 smithi033 ceph-mon[35092]: from='osd.7 ' entity='osd.7' 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: osdmap e45: 8 total, 7 up, 8 in 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:18.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:18.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: osdmap e46: 8 total, 7 up, 8 in 2024-09-12T02:43:18.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:18 smithi184 ceph-mon[35321]: from='osd.7 ' entity='osd.7' 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: osdmap e45: 8 total, 7 up, 8 in 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:18.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: osdmap e46: 8 total, 7 up, 8 in 2024-09-12T02:43:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:18 smithi016 ceph-mon[30925]: from='osd.7 ' entity='osd.7' 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] boot 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: osdmap e47: 8 total, 8 up, 8 in 2024-09-12T02:43:19.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:19.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:43:19.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:43:19.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:19.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:19.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] boot 2024-09-12T02:43:19.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: osdmap e47: 8 total, 8 up, 8 in 2024-09-12T02:43:19.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:19.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: osd.7 [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] boot 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: osdmap e47: 8 total, 8 up, 8 in 2024-09-12T02:43:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-12T02:43:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: osdmap e48: 8 total, 8 up, 8 in 2024-09-12T02:43:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/2798804068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]': finished 2024-09-12T02:43:21.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: osdmap e49: 9 total, 8 up, 9 in 2024-09-12T02:43:21.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:21.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/2292168561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: osdmap e48: 8 total, 8 up, 8 in 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/2798804068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]': finished 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: osdmap e49: 9 total, 8 up, 9 in 2024-09-12T02:43:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:21.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/2292168561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:21.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: osdmap e48: 8 total, 8 up, 8 in 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/2798804068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]: dispatch 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1980104-b353-42e4-9168-5c1c326da47e"}]': finished 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: osdmap e49: 9 total, 8 up, 9 in 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/2292168561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:21 smithi033 ceph-mon[35092]: pgmap v168: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:21 smithi033 ceph-mon[35092]: osdmap e50: 9 total, 8 up, 9 in 2024-09-12T02:43:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:21 smithi184 ceph-mon[35321]: pgmap v168: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:21 smithi184 ceph-mon[35321]: osdmap e50: 9 total, 8 up, 9 in 2024-09-12T02:43:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:21 smithi016 ceph-mon[30925]: pgmap v168: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:21 smithi016 ceph-mon[30925]: osdmap e50: 9 total, 8 up, 9 in 2024-09-12T02:43:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:23 smithi033 ceph-mon[35092]: pgmap v171: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:23 smithi033 ceph-mon[35092]: osdmap e51: 9 total, 8 up, 9 in 2024-09-12T02:43:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:23.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:23 smithi184 ceph-mon[35321]: pgmap v171: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:23.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:23 smithi184 ceph-mon[35321]: osdmap e51: 9 total, 8 up, 9 in 2024-09-12T02:43:23.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:23.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:23 smithi016 ceph-mon[30925]: pgmap v171: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:23.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:23 smithi016 ceph-mon[30925]: osdmap e51: 9 total, 8 up, 9 in 2024-09-12T02:43:23.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:25 smithi184 ceph-mon[35321]: pgmap v173: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:25.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:25 smithi016 ceph-mon[30925]: pgmap v173: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:25.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:25 smithi033 ceph-mon[35092]: pgmap v173: 1 pgs: 1 remapped; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:27.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:27 smithi184 ceph-mon[35321]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:27.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:27 smithi016 ceph-mon[30925]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:27 smithi033 ceph-mon[35092]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:29.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:29 smithi184 ceph-mon[35321]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:29.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-12T02:43:29.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:29.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:29 smithi184 ceph-mon[35321]: Deploying daemon osd.8 on smithi184 2024-09-12T02:43:29.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:29 smithi016 ceph-mon[30925]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:29.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-12T02:43:29.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:29.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:29 smithi016 ceph-mon[30925]: Deploying daemon osd.8 on smithi184 2024-09-12T02:43:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:29 smithi033 ceph-mon[35092]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-12T02:43:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:29 smithi033 ceph-mon[35092]: Deploying daemon osd.8 on smithi184 2024-09-12T02:43:31.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:31 smithi184 ceph-mon[35321]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:31 smithi016 ceph-mon[30925]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:31 smithi033 ceph-mon[35092]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:33.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:33 smithi184 ceph-mon[35321]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:33 smithi016 ceph-mon[30925]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:33 smithi033 ceph-mon[35092]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:35 smithi184 ceph-mon[35321]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:35 smithi016 ceph-mon[30925]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:35 smithi033 ceph-mon[35092]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:37.757 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 8 on host 'smithi184' 2024-09-12T02:43:37.757 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.756+0000 7fa5aa7fc640 1 -- 172.21.15.184:0/3482935692 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa5b40fe480 con 0x7fa5a00a2c30 2024-09-12T02:43:37.759 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.758+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 msgr2=0x7fa5a00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:37.759 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.758+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 0x7fa5a00a50f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4199390 tx=0x7fa5a4005f90 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 msgr2=0x7fa5b4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e9c0 tx=0x7fa5b000ee80 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 shutdown_connections 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5a00a2c30 0x7fa5a00a50f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5b41026d0 0x7fa5b419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5b4101d10 0x7fa5b4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 --2- 172.21.15.184:0/3482935692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b4100b10 0x7fa5b4197ee0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 >> 172.21.15.184:0/3482935692 conn(0x7fa5b40fc2a0 msgr2=0x7fa5b40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:43:37.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 shutdown_connections 2024-09-12T02:43:37.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:43:37.759+0000 7fa5bb562640 1 -- 172.21.15.184:0/3482935692 wait complete. 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:38.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:38.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:38.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:43:38.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:38.373 DEBUG:teuthology.orchestra.run.smithi184:osd.8> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.8.service 2024-09-12T02:43:38.375 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-09-12T02:43:38.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-12T02:43:38.641 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:43:39.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:39 smithi016 ceph-mon[30925]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:39 smithi184 ceph-mon[35321]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:39 smithi033 ceph-mon[35092]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:40.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 02:43:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T02:43:40.492+0000 7f05aba47640 -1 osd.8 0 log_to_monitors true 2024-09-12T02:43:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:40 smithi016 ceph-mon[30925]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:40 smithi016 ceph-mon[30925]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:40.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:40 smithi184 ceph-mon[35321]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:40.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:40 smithi184 ceph-mon[35321]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:43:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:40 smithi033 ceph-mon[35092]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:40 smithi033 ceph-mon[35092]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-12T02:43:41.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-09-12T02:43:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: osdmap e52: 9 total, 8 up, 9 in 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:41.892 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:41.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 -- 172.21.15.16:0/4164212910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 msgr2=0x7ffb1c102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 --2- 172.21.15.16:0/4164212910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c102150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009900 tx=0x7ffb0402a410 comp rx=0 tx=0).stop 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 -- 172.21.15.16:0/4164212910 shutdown_connections 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 --2- 172.21.15.16:0/4164212910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 --2- 172.21.15.16:0/4164212910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c102150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 --2- 172.21.15.16:0/4164212910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 0x7ffb1c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.977+0000 7ffb230c8640 1 -- 172.21.15.16:0/4164212910 >> 172.21.15.16:0/4164212910 conn(0x7ffb1c0fc280 msgr2=0x7ffb1c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- 172.21.15.16:0/4164212910 shutdown_connections 2024-09-12T02:43:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- 172.21.15.16:0/4164212910 wait complete. 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 Processor -- start 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- start start 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 0x7ffb1c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c10d8e0 con 0x7ffb1c101cd0 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c10d760 con 0x7ffb1c102690 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb230c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c10da60 con 0x7ffb1c100ad0 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57840/0 (socket says 172.21.15.16:57840) 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.978+0000 7ffb1bfff640 1 -- 172.21.15.16:0/986944294 learned_addr learned my addr 172.21.15.16:0/986944294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:43:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb21665640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb20e64640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 0x7ffb1c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 -- 172.21.15.16:0/986944294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 msgr2=0x7ffb1c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 0x7ffb1c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 -- 172.21.15.16:0/986944294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 msgr2=0x7ffb1c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 -- 172.21.15.16:0/986944294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1c19fba0 con 0x7ffb1c101cd0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb21665640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb1bfff640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009230 tx=0x7ffb04004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0403f030 con 0x7ffb1c101cd0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1c19fe30 con 0x7ffb1c101cd0 2024-09-12T02:43:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb04003ce0 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04035070 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.979+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb1c1a0310 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.980+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb1c107f10 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.982+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffb04042030 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.983+0000 7ffb19ffb640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 0x7ffb080a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.983+0000 7ffb20e64640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 0x7ffb080a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.983+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5549+0+0 (secure 0 0 0) 0x7ffb040ef100 con 0x7ffb1c101cd0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.983+0000 7ffb20e64640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 0x7ffb080a5070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c101b30 tx=0x7ffb0c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:43:42.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:41.986+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffb040bcad0 con 0x7ffb1c101cd0 2024-09-12T02:43:42.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: osdmap e52: 9 total, 8 up, 9 in 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:42.149 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:42.150 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:43:42.176+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ffb1c062630 con 0x7ffb080a2bb0 2024-09-12T02:43:42.197 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T02:43:41.797+0000 7f05a71a7640 -1 osd.8 0 waiting for initial osdmap 2024-09-12T02:43:42.197 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T02:43:41.809+0000 7f05a37e1640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: osdmap e52: 9 total, 8 up, 9 in 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:42.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:43:42.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:43:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi184 2024-09-12T02:43:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-12T02:43:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: osdmap e53: 9 total, 8 up, 9 in 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:42.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:42 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi184 2024-09-12T02:43:43.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-12T02:43:43.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:43:43.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: osdmap e53: 9 total, 8 up, 9 in 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:43.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:42 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi184 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: osdmap e53: 9 total, 8 up, 9 in 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:43.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:42 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:43:43.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] boot 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: osdmap e54: 9 total, 9 up, 9 in 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:43 smithi016 ceph-mon[30925]: osdmap e55: 9 total, 9 up, 9 in 2024-09-12T02:43:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:43:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:43:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:44.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] boot 2024-09-12T02:43:44.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: osdmap e54: 9 total, 9 up, 9 in 2024-09-12T02:43:44.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:44.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:43 smithi033 ceph-mon[35092]: osdmap e55: 9 total, 9 up, 9 in 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: osd.8 [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] boot 2024-09-12T02:43:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: osdmap e54: 9 total, 9 up, 9 in 2024-09-12T02:43:44.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-12T02:43:44.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:43 smithi184 ceph-mon[35321]: osdmap e55: 9 total, 9 up, 9 in 2024-09-12T02:43:45.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:44 smithi016 ceph-mon[30925]: osdmap e56: 9 total, 9 up, 9 in 2024-09-12T02:43:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]: dispatch 2024-09-12T02:43:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]': finished 2024-09-12T02:43:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:44 smithi016 ceph-mon[30925]: osdmap e57: 10 total, 9 up, 10 in 2024-09-12T02:43:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:44 smithi033 ceph-mon[35092]: osdmap e56: 9 total, 9 up, 9 in 2024-09-12T02:43:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]: dispatch 2024-09-12T02:43:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]': finished 2024-09-12T02:43:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:44 smithi033 ceph-mon[35092]: osdmap e57: 10 total, 9 up, 10 in 2024-09-12T02:43:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:44 smithi184 ceph-mon[35321]: osdmap e56: 9 total, 9 up, 9 in 2024-09-12T02:43:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]: dispatch 2024-09-12T02:43:45.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3740516271' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1da86df9-e623-41f7-8089-73aef567c50a"}]': finished 2024-09-12T02:43:45.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:44 smithi184 ceph-mon[35321]: osdmap e57: 10 total, 9 up, 10 in 2024-09-12T02:43:45.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:45 smithi016 ceph-mon[30925]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:43:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/652209983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:45 smithi016 ceph-mon[30925]: osdmap e58: 10 total, 9 up, 10 in 2024-09-12T02:43:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:45 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:45 smithi033 ceph-mon[35092]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:43:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/652209983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:45 smithi033 ceph-mon[35092]: osdmap e58: 10 total, 9 up, 10 in 2024-09-12T02:43:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:45 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:45 smithi184 ceph-mon[35321]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:43:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/652209983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:43:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:45 smithi184 ceph-mon[35321]: osdmap e58: 10 total, 9 up, 10 in 2024-09-12T02:43:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:45 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:43:47.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:46 smithi033 ceph-mon[35092]: pgmap v191: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:47.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:46 smithi033 ceph-mon[35092]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-12T02:43:47.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:46 smithi184 ceph-mon[35321]: pgmap v191: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:47.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:46 smithi184 ceph-mon[35321]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-12T02:43:47.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:46 smithi016 ceph-mon[30925]: pgmap v191: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:47.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:46 smithi016 ceph-mon[30925]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-12T02:43:49.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:49 smithi184 ceph-mon[35321]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:49.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:49 smithi016 ceph-mon[30925]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:49 smithi033 ceph-mon[35092]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-09-12T02:43:50.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:50 smithi016 ceph-mon[30925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-12T02:43:50.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:50 smithi016 ceph-mon[30925]: Cluster is now healthy 2024-09-12T02:43:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:50 smithi033 ceph-mon[35092]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-12T02:43:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:50 smithi033 ceph-mon[35092]: Cluster is now healthy 2024-09-12T02:43:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:50 smithi184 ceph-mon[35321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-12T02:43:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:50 smithi184 ceph-mon[35321]: Cluster is now healthy 2024-09-12T02:43:51.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:51 smithi016 ceph-mon[30925]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-12T02:43:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:51 smithi033 ceph-mon[35092]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-12T02:43:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:51 smithi184 ceph-mon[35321]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-12T02:43:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:53 smithi033 ceph-mon[35092]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:43:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:53 smithi016 ceph-mon[30925]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:43:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:53 smithi184 ceph-mon[35321]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-12T02:43:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-12T02:43:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:54 smithi184 ceph-mon[35321]: Deploying daemon osd.9 on smithi016 2024-09-12T02:43:54.317 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-12T02:43:54.317 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:54.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:54 smithi016 ceph-mon[30925]: Deploying daemon osd.9 on smithi016 2024-09-12T02:43:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-12T02:43:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:43:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:54 smithi033 ceph-mon[35092]: Deploying daemon osd.9 on smithi016 2024-09-12T02:43:55.345 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:55 smithi016 ceph-mon[30925]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:43:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:55 smithi033 ceph-mon[35092]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:43:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:55 smithi184 ceph-mon[35321]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-12T02:43:57.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:57 smithi016 ceph-mon[30925]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:43:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:57 smithi033 ceph-mon[35092]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:43:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:57 smithi184 ceph-mon[35321]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-12T02:43:59.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:43:59 smithi016 ceph-mon[30925]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:43:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:43:59 smithi033 ceph-mon[35092]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:43:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:43:59 smithi184 ceph-mon[35321]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:00.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:00.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:00.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:01.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:01 smithi016 ceph-mon[30925]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:01 smithi033 ceph-mon[35092]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:01 smithi184 ceph-mon[35321]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.290+0000 7ffb19ffb640 1 -- 172.21.15.16:0/986944294 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffb1c062630 con 0x7ffb080a2bb0 2024-09-12T02:44:03.291 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-09-12T02:44:03.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 msgr2=0x7ffb080a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:03.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 0x7ffb080a5070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c101b30 tx=0x7ffb0c008040 comp rx=0 tx=0).stop 2024-09-12T02:44:03.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 msgr2=0x7ffb1c1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009230 tx=0x7ffb04004430 comp rx=0 tx=0).stop 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 shutdown_connections 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb1c102690 0x7ffb1c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffb080a2bb0 0x7ffb080a5070 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c101cd0 0x7ffb1c1984b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 --2- 172.21.15.16:0/986944294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb1c100ad0 0x7ffb1c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 >> 172.21.15.16:0/986944294 conn(0x7ffb1c0fc280 msgr2=0x7ffb1c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 shutdown_connections 2024-09-12T02:44:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:03.293+0000 7ffb230c8640 1 -- 172.21.15.16:0/986944294 wait complete. 2024-09-12T02:44:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:03 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:03 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:03.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:03.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:03 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:04.043 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.9.service 2024-09-12T02:44:04.046 INFO:tasks.cephadm:Deploying osd.10 on smithi033 with /dev/vg_nvme/lv_1... 2024-09-12T02:44:04.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-12T02:44:04.310 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:44:05.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:05 smithi016 ceph-mon[30925]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:05 smithi033 ceph-mon[35092]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:05 smithi184 ceph-mon[35321]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:06.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 02:44:06 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T02:44:06.006+0000 7fdfd5728640 -1 osd.9 0 log_to_monitors true 2024-09-12T02:44:06.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:06 smithi016 ceph-mon[30925]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-12T02:44:06.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:06 smithi033 ceph-mon[35092]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-12T02:44:06.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:06 smithi184 ceph-mon[35321]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-12T02:44:07.036 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-09-12T02:44:07.062 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-09-12T02:44:07.316 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: osdmap e59: 10 total, 9 up, 10 in 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:07.703 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:07.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 -- 172.21.15.33:0/4082095711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 msgr2=0x7f7bd41105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 --2- 172.21.15.33:0/4082095711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41105d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009ca0 tx=0x7f7bcc02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 -- 172.21.15.33:0/4082095711 shutdown_connections 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 --2- 172.21.15.33:0/4082095711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41105d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 --2- 172.21.15.33:0/4082095711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 0x7f7bd40750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 --2- 172.21.15.33:0/4082095711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 0x7f7bd4074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 -- 172.21.15.33:0/4082095711 >> 172.21.15.33:0/4082095711 conn(0x7f7bd40fd390 msgr2=0x7f7bd40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 -- 172.21.15.33:0/4082095711 shutdown_connections 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.788+0000 7f7bd8978640 1 -- 172.21.15.33:0/4082095711 wait complete. 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 Processor -- start 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 -- start start 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 0x7f7bd419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:07.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 0x7f7bd419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd4113f00 con 0x7f7bd4076220 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd4113d80 con 0x7f7bd410e1e0 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd8978640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd4114080 con 0x7f7bd4074c50 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50332/0 (socket says 172.21.15.33:50332) 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 -- 172.21.15.33:0/3451088715 learned_addr learned my addr 172.21.15.33:0/3451088715 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd1d9b640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 0x7f7bd419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd259c640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 0x7f7bd419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 -- 172.21.15.33:0/3451088715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 msgr2=0x7f7bd419e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 0x7f7bd419e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 -- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 msgr2=0x7f7bd419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.789+0000 7f7bd2d9d640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 0x7f7bd419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bd2d9d640 1 -- 172.21.15.33:0/3451088715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd41a6120 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bd2d9d640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc005e30 tx=0x7f7bcc031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc003b70 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bcc003d10 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd41a63b0 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc02fd30 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.790+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd41a69c0 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.791+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd41043b0 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.792+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7bcc004050 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.793+0000 7f7bc37fe640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 0x7f7bbc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.793+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5981+0+0 (secure 0 0 0) 0x7f7bcc0efb40 con 0x7f7bd410e1e0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.793+0000 7f7bd259c640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 0x7f7bbc0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:07.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.794+0000 7f7bd259c640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 0x7f7bbc0a50f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005fd0 tx=0x7f7bc4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: osdmap e59: 10 total, 9 up, 10 in 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.796+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7bcc0bd3c0 con 0x7f7bd410e1e0 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: osdmap e59: 10 total, 9 up, 10 in 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:07.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:07.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:07.981+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7bd4062630 con 0x7f7bbc0a2c30 2024-09-12T02:44:08.544 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T02:44:08.256+0000 7fdfd0e88640 -1 osd.9 0 waiting for initial osdmap 2024-09-12T02:44:08.544 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T02:44:08.268+0000 7fdfc94ba640 -1 osd.9 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi016 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: osdmap e60: 10 total, 9 up, 10 in 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:08.673 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:08.674 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:08.674 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' 2024-09-12T02:44:08.674 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi016 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: osdmap e60: 10 total, 9 up, 10 in 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:08.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:08.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' 2024-09-12T02:44:08.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:09.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi016 2024-09-12T02:44:09.322 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: osdmap e60: 10 total, 9 up, 10 in 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039]' entity='osd.9' 2024-09-12T02:44:09.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] boot 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: osdmap e61: 10 total, 10 up, 10 in 2024-09-12T02:44:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] boot 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: osdmap e61: 10 total, 10 up, 10 in 2024-09-12T02:44:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:44:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:44:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-09-12T02:44:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: osd.9 [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] boot 2024-09-12T02:44:10.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: osdmap e61: 10 total, 10 up, 10 in 2024-09-12T02:44:10.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-12T02:44:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:10 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/2997063169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:10 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:10 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]': finished 2024-09-12T02:44:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:10 smithi033 ceph-mon[35092]: osdmap e62: 11 total, 10 up, 11 in 2024-09-12T02:44:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:10 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/2997063169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:10 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:10 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]': finished 2024-09-12T02:44:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:10 smithi016 ceph-mon[30925]: osdmap e62: 11 total, 10 up, 11 in 2024-09-12T02:44:10.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:10 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/2997063169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:10 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]: dispatch 2024-09-12T02:44:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:10 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d052784-3463-4b31-8894-92d1ef1d1ea9"}]': finished 2024-09-12T02:44:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:10 smithi184 ceph-mon[35321]: osdmap e62: 11 total, 10 up, 11 in 2024-09-12T02:44:11.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:11 smithi016 ceph-mon[30925]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:11 smithi016 ceph-mon[30925]: from='client.? 172.21.15.33:0/3391328380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:11 smithi033 ceph-mon[35092]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:11 smithi033 ceph-mon[35092]: from='client.? 172.21.15.33:0/3391328380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:11 smithi184 ceph-mon[35321]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:11 smithi184 ceph-mon[35321]: from='client.? 172.21.15.33:0/3391328380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:13.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:13 smithi184 ceph-mon[35321]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:13 smithi016 ceph-mon[30925]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:13.944 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:13 smithi033 ceph-mon[35092]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:15 smithi016 ceph-mon[30925]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:15.917 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:15 smithi033 ceph-mon[35092]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:15 smithi184 ceph-mon[35321]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:17.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:17 smithi033 ceph-mon[35092]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:17 smithi016 ceph-mon[30925]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:17 smithi184 ceph-mon[35321]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:19 smithi016 ceph-mon[30925]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-12T02:44:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:19.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:19 smithi016 ceph-mon[30925]: Deploying daemon osd.10 on smithi033 2024-09-12T02:44:19.937 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:19 smithi033 ceph-mon[35092]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:19.937 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-12T02:44:19.937 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:19.937 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:19 smithi033 ceph-mon[35092]: Deploying daemon osd.10 on smithi033 2024-09-12T02:44:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:19 smithi184 ceph-mon[35321]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-12T02:44:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:19 smithi184 ceph-mon[35321]: Deploying daemon osd.10 on smithi033 2024-09-12T02:44:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:21 smithi016 ceph-mon[30925]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:21.938 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:21 smithi033 ceph-mon[35092]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:21 smithi184 ceph-mon[35321]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:23.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:23 smithi033 ceph-mon[35092]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:23.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:23 smithi016 ceph-mon[30925]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:23 smithi184 ceph-mon[35321]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:25.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:25 smithi016 ceph-mon[30925]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:25.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:25.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:25.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:25 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:25 smithi033 ceph-mon[35092]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:25 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:25 smithi184 ceph-mon[35321]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:25 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:27.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.103+0000 7f7bc37fe640 1 -- 172.21.15.33:0/3451088715 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7bd4062630 con 0x7f7bbc0a2c30 2024-09-12T02:44:28.104 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 10 on host 'smithi033' 2024-09-12T02:44:28.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 msgr2=0x7f7bbc0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:28.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 0x7f7bbc0a50f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005fd0 tx=0x7f7bc4005850 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 msgr2=0x7f7bd41a5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc005e30 tx=0x7f7bcc031ee0 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 shutdown_connections 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bd410e1e0 0x7f7bd41a5ab0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bbc0a2c30 0x7f7bbc0a50f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd4076220 0x7f7bd419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 --2- 172.21.15.33:0/3451088715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bd4074c50 0x7f7bd419e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:28.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 >> 172.21.15.33:0/3451088715 conn(0x7f7bd40fd390 msgr2=0x7f7bd4101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:28.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 shutdown_connections 2024-09-12T02:44:28.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-09-12T02:44:28.106+0000 7f7bd8978640 1 -- 172.21.15.33:0/3451088715 wait complete. 2024-09-12T02:44:28.692 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:28.693 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.693 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.723 DEBUG:teuthology.orchestra.run.smithi033:osd.10> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.10.service 2024-09-12T02:44:28.725 INFO:tasks.cephadm:Deploying osd.11 on smithi184 with /dev/vg_nvme/lv_1... 2024-09-12T02:44:28.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-12T02:44:28.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:28.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:28.977 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:44:29.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:29.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:29.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:29 smithi184 ceph-mon[35321]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:29.898 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:29 smithi033 ceph-mon[35092]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:29 smithi016 ceph-mon[30925]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:31.213 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 02:44:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T02:44:30.924+0000 7fea4b740640 -1 osd.10 0 log_to_monitors true 2024-09-12T02:44:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-12T02:44:31.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_1 2024-09-12T02:44:31.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:31 smithi184 ceph-mon[35321]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:31.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:31 smithi184 ceph-mon[35321]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:31.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:31 smithi184 ceph-mon[35321]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:31.811 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:44:31.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:31 smithi016 ceph-mon[30925]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:31.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:31 smithi016 ceph-mon[30925]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:31.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:31 smithi016 ceph-mon[30925]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:31.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:31 smithi033 ceph-mon[35092]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:31.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:31 smithi033 ceph-mon[35092]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:31.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:31 smithi033 ceph-mon[35092]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 -- 172.21.15.184:0/301131839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa40103930 msgr2=0x7ffa4010a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 --2- 172.21.15.184:0/301131839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa40103930 0x7ffa4010a1c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009ca0 tx=0x7ffa3802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 -- 172.21.15.184:0/301131839 shutdown_connections 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 --2- 172.21.15.184:0/301131839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa40103930 0x7ffa4010a1c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 --2- 172.21.15.184:0/301131839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40102f70 0x7ffa401033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 --2- 172.21.15.184:0/301131839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa400ffef0 0x7ffa401002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 -- 172.21.15.184:0/301131839 >> 172.21.15.184:0/301131839 conn(0x7ffa400fba40 msgr2=0x7ffa400fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 -- 172.21.15.184:0/301131839 shutdown_connections 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.172+0000 7ffa466da640 1 -- 172.21.15.184:0/301131839 wait complete. 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 Processor -- start 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 -- start start 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa40102f70 0x7ffa401984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40103930 0x7ffa4019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010d920 con 0x7ffa40103930 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010d7a0 con 0x7ffa40102f70 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa466da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010daa0 con 0x7ffa400ffef0 2024-09-12T02:44:32.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:46154/0 (socket says 172.21.15.184:46154) 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.173+0000 7ffa3ffff640 1 -- 172.21.15.184:0/3225809923 learned_addr learned my addr 172.21.15.184:0/3225809923 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa44c77640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40103930 0x7ffa4019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3f7fe640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa40102f70 0x7ffa401984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 -- 172.21.15.184:0/3225809923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa40102f70 msgr2=0x7ffa401984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa40102f70 0x7ffa401984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 -- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40103930 msgr2=0x7ffa4019f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40103930 0x7ffa4019f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 -- 172.21.15.184:0/3225809923 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4019fbd0 con 0x7ffa400ffef0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3ffff640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b450 tx=0x7ffa3000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30015020 con 0x7ffa400ffef0 2024-09-12T02:44:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4019fec0 con 0x7ffa400ffef0 2024-09-12T02:44:32.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa30011d70 con 0x7ffa400ffef0 2024-09-12T02:44:32.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000f340 con 0x7ffa400ffef0 2024-09-12T02:44:32.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.174+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa401a0390 con 0x7ffa400ffef0 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.176+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa40109140 con 0x7ffa400ffef0 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.176+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffa30018020 con 0x7ffa400ffef0 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.177+0000 7ffa3d7fa640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 0x7ffa280a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.177+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6398+0+0 (secure 0 0 0) 0x7ffa300ce360 con 0x7ffa400ffef0 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.177+0000 7ffa3f7fe640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 0x7ffa280a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:32.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.178+0000 7ffa3f7fe640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 0x7ffa280a5070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005fd0 tx=0x7ffa34005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:32.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.185+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa3001f020 con 0x7ffa400ffef0 2024-09-12T02:44:32.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:32.406+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ffa40062630 con 0x7ffa280a2bb0 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: osdmap e63: 11 total, 10 up, 11 in 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi033 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.878 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: osdmap e63: 11 total, 10 up, 11 in 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi033 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:32.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:32.997 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T02:44:32.599+0000 7fea46ea0640 -1 osd.10 0 waiting for initial osdmap 2024-09-12T02:44:32.997 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T02:44:32.613+0000 7fea434da640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:44:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-12T02:44:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: osdmap e63: 11 total, 10 up, 11 in 2024-09-12T02:44:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi033 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 4273M 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:32.999 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-12T02:44:32.999 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-12T02:44:32.999 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: osdmap e64: 11 total, 10 up, 11 in 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] boot 2024-09-12T02:44:33.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: osdmap e65: 11 total, 11 up, 11 in 2024-09-12T02:44:33.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:33 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: osdmap e64: 11 total, 10 up, 11 in 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] boot 2024-09-12T02:44:33.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: osdmap e65: 11 total, 11 up, 11 in 2024-09-12T02:44:33.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:33 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: osdmap e64: 11 total, 10 up, 11 in 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: osd.10 [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] boot 2024-09-12T02:44:33.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: osdmap e65: 11 total, 11 up, 11 in 2024-09-12T02:44:33.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:33 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: osdmap e66: 11 total, 11 up, 11 in 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/1924279248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]': finished 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: osdmap e67: 12 total, 11 up, 12 in 2024-09-12T02:44:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:34 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:35.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:44:35.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: osdmap e66: 11 total, 11 up, 11 in 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/1924279248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]': finished 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: osdmap e67: 12 total, 11 up, 12 in 2024-09-12T02:44:35.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:34 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:44:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: osdmap e66: 11 total, 11 up, 11 in 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/1924279248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]: dispatch 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5b71a4-67ba-479b-83e0-80e5ce8cfec9"}]': finished 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: osdmap e67: 12 total, 11 up, 12 in 2024-09-12T02:44:35.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:34 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:36.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:35 smithi016 ceph-mon[30925]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.184:0/886034447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:35 smithi016 ceph-mon[30925]: osdmap e68: 12 total, 11 up, 12 in 2024-09-12T02:44:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:35 smithi033 ceph-mon[35092]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.184:0/886034447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:35 smithi033 ceph-mon[35092]: osdmap e68: 12 total, 11 up, 12 in 2024-09-12T02:44:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:35 smithi184 ceph-mon[35321]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.184:0/886034447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-12T02:44:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:35 smithi184 ceph-mon[35321]: osdmap e68: 12 total, 11 up, 12 in 2024-09-12T02:44:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:37 smithi184 ceph-mon[35321]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:37 smithi184 ceph-mon[35321]: osdmap e69: 12 total, 11 up, 12 in 2024-09-12T02:44:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:37 smithi016 ceph-mon[30925]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:37 smithi016 ceph-mon[30925]: osdmap e69: 12 total, 11 up, 12 in 2024-09-12T02:44:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:37 smithi033 ceph-mon[35092]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:37 smithi033 ceph-mon[35092]: osdmap e69: 12 total, 11 up, 12 in 2024-09-12T02:44:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:39 smithi184 ceph-mon[35321]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:39 smithi016 ceph-mon[30925]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:39 smithi033 ceph-mon[35092]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:41 smithi016 ceph-mon[30925]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:44:41.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:41 smithi184 ceph-mon[35321]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:44:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:41 smithi033 ceph-mon[35092]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-12T02:44:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:43 smithi184 ceph-mon[35321]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:44:44.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-12T02:44:44.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:44.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:43 smithi033 ceph-mon[35092]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:44:44.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-12T02:44:44.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:43 smithi016 ceph-mon[30925]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-12T02:44:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-12T02:44:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:45.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:44 smithi184 ceph-mon[35321]: Deploying daemon osd.11 on smithi184 2024-09-12T02:44:45.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:44 smithi016 ceph-mon[30925]: Deploying daemon osd.11 on smithi184 2024-09-12T02:44:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:44 smithi033 ceph-mon[35092]: Deploying daemon osd.11 on smithi184 2024-09-12T02:44:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:45 smithi184 ceph-mon[35321]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-12T02:44:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:45 smithi016 ceph-mon[30925]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-12T02:44:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:45 smithi033 ceph-mon[35092]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-12T02:44:48.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:47 smithi184 ceph-mon[35321]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-12T02:44:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:47 smithi016 ceph-mon[30925]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-12T02:44:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:47 smithi033 ceph-mon[35092]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-12T02:44:50.132 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:49 smithi184 ceph-mon[35321]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-12T02:44:50.132 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:50.132 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:50.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:49 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:49 smithi016 ceph-mon[30925]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-12T02:44:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:49 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:49 smithi033 ceph-mon[35092]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-12T02:44:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:49 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:52.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:51 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:51 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:51 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:52.341 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.340+0000 7ffa3d7fa640 1 -- 172.21.15.184:0/3225809923 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ffa40062630 con 0x7ffa280a2bb0 2024-09-12T02:44:52.342 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 11 on host 'smithi184' 2024-09-12T02:44:52.343 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 msgr2=0x7ffa280a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:52.343 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 0x7ffa280a5070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005fd0 tx=0x7ffa34005850 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 msgr2=0x7ffa40197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b450 tx=0x7ffa3000b910 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 shutdown_connections 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa280a2bb0 0x7ffa280a5070 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa40103930 0x7ffa4019f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa40102f70 0x7ffa401984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 --2- 172.21.15.184:0/3225809923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa400ffef0 0x7ffa40197fa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 >> 172.21.15.184:0/3225809923 conn(0x7ffa400fba40 msgr2=0x7ffa40106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 shutdown_connections 2024-09-12T02:44:52.344 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:44:52.343+0000 7ffa466da640 1 -- 172.21.15.184:0/3225809923 wait complete. 2024-09-12T02:44:53.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:53.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:52 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.036 DEBUG:teuthology.orchestra.run.smithi184:osd.11> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.11.service 2024-09-12T02:44:53.041 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-12T02:44:53.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-12T02:44:53.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:53.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:52 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:44:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:52 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:44:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 -- 172.21.15.16:0/685231978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9001026c0 msgr2=0x7fa900104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 --2- 172.21.15.16:0/685231978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9001026c0 0x7fa900104ab0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009900 tx=0x7fa8fc02a410 comp rx=0 tx=0).stop 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 -- 172.21.15.16:0/685231978 shutdown_connections 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 --2- 172.21.15.16:0/685231978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa900107920 0x7fa900109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 --2- 172.21.15.16:0/685231978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900104ff0 0x7fa9001073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 --2- 172.21.15.16:0/685231978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9001026c0 0x7fa900104ab0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 -- 172.21.15.16:0/685231978 >> 172.21.15.16:0/685231978 conn(0x7fa9000fc280 msgr2=0x7fa9000fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 -- 172.21.15.16:0/685231978 shutdown_connections 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.773+0000 7fa90848b640 1 -- 172.21.15.16:0/685231978 wait complete. 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 Processor -- start 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 -- start start 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 0x7fa900197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900107920 0x7fa90019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa90010d8d0 con 0x7fa900104ff0 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa90010d750 con 0x7fa9001026c0 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa90848b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa90010da50 con 0x7fa900107920 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa905a26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa905a26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36090/0 (socket says 172.21.15.16:36090) 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa906227640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 0x7fa900197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa905a26640 1 -- 172.21.15.16:0/3633887507 learned_addr learned my addr 172.21.15.16:0/3633887507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa906a28640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900107920 0x7fa90019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.774+0000 7fa905a26640 1 -- 172.21.15.16:0/3633887507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900107920 msgr2=0x7fa90019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa905a26640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900107920 0x7fa90019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa905a26640 1 -- 172.21.15.16:0/3633887507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 msgr2=0x7fa900197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa905a26640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 0x7fa900197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa905a26640 1 -- 172.21.15.16:0/3633887507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90019fb80 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa906227640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 0x7fa900197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa905a26640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b450 tx=0x7fa8f000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0015020 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90019fe70 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8f0011d70 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f000f340 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.775+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9001a0350 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.776+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9000fdd70 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.777+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa8f0018020 con 0x7fa900104ff0 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.778+0000 7fa8f77fe640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 0x7fa8ec0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.778+0000 7fa906227640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 0x7fa8ec0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.778+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6809+0+0 (secure 0 0 0) 0x7fa8f00ce4f0 con 0x7fa900104ff0 2024-09-12T02:44:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.778+0000 7fa906227640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 0x7fa8ec0a5070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0314e0 tx=0x7fa8fc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.782+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa8f001f020 con 0x7fa900104ff0 2024-09-12T02:44:53.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:53 smithi016 ceph-mon[30925]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.960+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa900062df0 con 0x7fa900104ff0 2024-09-12T02:44:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.961+0000 7fa8f77fe640 1 -- 172.21.15.16:0/3633887507 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) ==== 74+0+133 (secure 0 0 0) 0x7fa8f000d040 con 0x7fa900104ff0 2024-09-12T02:44:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:44:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 msgr2=0x7fa8ec0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 0x7fa8ec0a5070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0314e0 tx=0x7fa8fc000bc0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 msgr2=0x7fa900198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b450 tx=0x7fa8f000b910 comp rx=0 tx=0).stop 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 shutdown_connections 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa900107920 0x7fa90019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8ec0a2bb0 0x7fa8ec0a5070 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa900104ff0 0x7fa900198490 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 --2- 172.21.15.16:0/3633887507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9001026c0 0x7fa900197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 >> 172.21.15.16:0/3633887507 conn(0x7fa9000fc280 msgr2=0x7fa9001030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 shutdown_connections 2024-09-12T02:44:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:53.963+0000 7fa90848b640 1 -- 172.21.15.16:0/3633887507 wait complete. 2024-09-12T02:44:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:53 smithi184 ceph-mon[35321]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:54.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:53 smithi033 ceph-mon[35092]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:54.843 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"num_osds":12,"num_up_osds":11,"osd_up_since":1726109072,"num_in_osds":12,"osd_in_since":1726109074,"num_remapped_pgs":0} 2024-09-12T02:44:55.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:54 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3633887507' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:55.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3633887507' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:54 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3633887507' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:55.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 02:44:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T02:44:55.426+0000 7fbcb547a640 -1 osd.11 0 log_to_monitors true 2024-09-12T02:44:55.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-12T02:44:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:44:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:55 smithi016 ceph-mon[30925]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:55 smithi016 ceph-mon[30925]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:55 smithi016 ceph-mon[30925]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:55 smithi033 ceph-mon[35092]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:55 smithi033 ceph-mon[35092]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:55 smithi033 ceph-mon[35092]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:55 smithi184 ceph-mon[35321]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:55 smithi184 ceph-mon[35321]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:55 smithi184 ceph-mon[35321]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-12T02:44:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 -- 172.21.15.16:0/3353390003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378107900 msgr2=0x7f4378109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 --2- 172.21.15.16:0/3353390003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378107900 0x7f4378109d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4370009ca0 tx=0x7f437002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 -- 172.21.15.16:0/3353390003 shutdown_connections 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 --2- 172.21.15.16:0/3353390003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4378107900 0x7f4378109d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 --2- 172.21.15.16:0/3353390003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 0x7f43781073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 --2- 172.21.15.16:0/3353390003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43781026a0 0x7f4378104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.703+0000 7f437e98f640 1 -- 172.21.15.16:0/3353390003 >> 172.21.15.16:0/3353390003 conn(0x7f43780fc2a0 msgr2=0x7f43780fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 -- 172.21.15.16:0/3353390003 shutdown_connections 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 -- 172.21.15.16:0/3353390003 wait complete. 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 Processor -- start 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 -- start start 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:56.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.704+0000 7f437e98f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 0x7f4378100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437e98f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4378107900 0x7f4378100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437e98f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f437810d8a0 con 0x7f43781026a0 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437e98f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f437810d720 con 0x7f4378104fd0 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437e98f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f437810da20 con 0x7f4378107900 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37768/0 (socket says 172.21.15.16:37768) 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 -- 172.21.15.16:0/2375901217 learned_addr learned my addr 172.21.15.16:0/2375901217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f43777fe640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 0x7f4378100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437cf2c640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4378107900 0x7f4378100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 -- 172.21.15.16:0/2375901217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4378107900 msgr2=0x7f4378100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4378107900 0x7f4378100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 -- 172.21.15.16:0/2375901217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 msgr2=0x7f4378100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 0x7f4378100650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 -- 172.21.15.16:0/2375901217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4378101450 con 0x7f43781026a0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f4377fff640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f436800d950 tx=0x7f436800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368017070 con 0x7f43781026a0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4368009c30 con 0x7f43781026a0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43781a2060 con 0x7f43781026a0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.705+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436800f6b0 con 0x7f43781026a0 2024-09-12T02:44:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.706+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781a2570 con 0x7f43781026a0 2024-09-12T02:44:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.706+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43780fdd90 con 0x7f43781026a0 2024-09-12T02:44:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.708+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f43680109d0 con 0x7f43781026a0 2024-09-12T02:44:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.708+0000 7f43757fa640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 0x7f43600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.708+0000 7f43777fe640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 0x7f43600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.708+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6831+0+0 (secure 0 0 0) 0x7f43680ce6a0 con 0x7f43781026a0 2024-09-12T02:44:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.709+0000 7f43777fe640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 0x7f43600a5040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4378101d80 tx=0x7f436c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.713+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4368009de0 con 0x7f43781026a0 2024-09-12T02:44:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.890+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f4378062e20 con 0x7f43781026a0 2024-09-12T02:44:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.891+0000 7f43757fa640 1 -- 172.21.15.16:0/2375901217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7f436809bbd0 con 0x7f43781026a0 2024-09-12T02:44:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 msgr2=0x7f43600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 0x7f43600a5040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4378101d80 tx=0x7f436c00b390 comp rx=0 tx=0).stop 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 msgr2=0x7f4378102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f436800d950 tx=0x7f436800de10 comp rx=0 tx=0).stop 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 shutdown_connections 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4378107900 0x7f4378100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4378104fd0 0x7f4378100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f43600a2b80 0x7f43600a5040 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 --2- 172.21.15.16:0/2375901217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43781026a0 0x7f4378102000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 >> 172.21.15.16:0/2375901217 conn(0x7f43780fc2a0 msgr2=0x7f4378102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 shutdown_connections 2024-09-12T02:44:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:56.893+0000 7f437e98f640 1 -- 172.21.15.16:0/2375901217 wait complete. 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: osdmap e70: 12 total, 11 up, 12 in 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: Detected new or changed devices on smithi184 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: osdmap e70: 12 total, 11 up, 12 in 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: Detected new or changed devices on smithi184 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:57.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-12T02:44:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: osdmap e70: 12 total, 11 up, 12 in 2024-09-12T02:44:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: Detected new or changed devices on smithi184 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:44:57.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-12T02:44:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:44:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:44:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:44:57.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"num_osds":12,"num_up_osds":11,"osd_up_since":1726109072,"num_in_osds":12,"osd_in_since":1726109074,"num_remapped_pgs":0} 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: osdmap e71: 12 total, 11 up, 12 in 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2375901217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:57 smithi016 ceph-mon[30925]: from='osd.11 ' entity='osd.11' 2024-09-12T02:44:58.045 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T02:44:57.669+0000 7fbcb0bda640 -1 osd.11 0 waiting for initial osdmap 2024-09-12T02:44:58.045 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T02:44:57.684+0000 7fbca920c640 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T02:44:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:44:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: osdmap e71: 12 total, 11 up, 12 in 2024-09-12T02:44:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2375901217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:58.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:57 smithi184 ceph-mon[35321]: from='osd.11 ' entity='osd.11' 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: osdmap e71: 12 total, 11 up, 12 in 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2375901217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:44:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:57 smithi033 ceph-mon[35092]: from='osd.11 ' entity='osd.11' 2024-09-12T02:44:58.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-12T02:44:58.957 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: purged_snaps scrub starts 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: purged_snaps scrub ok 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] boot 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: osdmap e72: 12 total, 12 up, 12 in 2024-09-12T02:44:59.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: purged_snaps scrub starts 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: purged_snaps scrub ok 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] boot 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: osdmap e72: 12 total, 12 up, 12 in 2024-09-12T02:44:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: purged_snaps scrub starts 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: purged_snaps scrub ok 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: osd.11 [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] boot 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: osdmap e72: 12 total, 12 up, 12 in 2024-09-12T02:44:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.402+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4253207992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90102700 msgr2=0x7f5b90104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.402+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4253207992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90102700 0x7f5b90104af0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009900 tx=0x7f5b8002a410 comp rx=0 tx=0).stop 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4253207992 shutdown_connections 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4253207992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90107960 0x7f5b90109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4253207992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90105030 0x7f5b90107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4253207992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90102700 0x7f5b90104af0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4253207992 >> 172.21.15.16:0/4253207992 conn(0x7f5b900fc280 msgr2=0x7f5b900fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4253207992 shutdown_connections 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4253207992 wait complete. 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 Processor -- start 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.403+0000 7f5b94ef5640 1 -- start start 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 0x7f5b90197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90105030 0x7f5b901983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d860 con 0x7f5b90107960 2024-09-12T02:44:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d6e0 con 0x7f5b90102700 2024-09-12T02:44:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b94ef5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d9e0 con 0x7f5b90105030 2024-09-12T02:44:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37774/0 (socket says 172.21.15.16:37774) 2024-09-12T02:44:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 -- 172.21.15.16:0/4060535828 learned_addr learned my addr 172.21.15.16:0/4060535828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:44:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8dd9b640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90105030 0x7f5b901983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8e59c640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 0x7f5b90197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 -- 172.21.15.16:0/4060535828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90105030 msgr2=0x7f5b901983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90105030 0x7f5b901983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 -- 172.21.15.16:0/4060535828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 msgr2=0x7f5b90197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 0x7f5b90197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 -- 172.21.15.16:0/4060535828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9019fae0 con 0x7f5b90107960 2024-09-12T02:44:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8e59c640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 0x7f5b90197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:44:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.404+0000 7f5b8ed9d640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800cce0 tx=0x7f5b8800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.405+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8800f450 con 0x7f5b90107960 2024-09-12T02:44:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.405+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9019fdd0 con 0x7f5b90107960 2024-09-12T02:44:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.405+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b8800fad0 con 0x7f5b90107960 2024-09-12T02:44:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.405+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b88018c10 con 0x7f5b90107960 2024-09-12T02:44:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.405+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b901a0230 con 0x7f5b90107960 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.406+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b900fdd70 con 0x7f5b90107960 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.407+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5b8800f5f0 con 0x7f5b90107960 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.408+0000 7f5b7f7fe640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 0x7f5b780a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.408+0000 7f5b8e59c640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 0x7f5b780a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.408+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f5b880cf290 con 0x7f5b90107960 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.408+0000 7f5b8e59c640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 0x7f5b780a5070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b80000c00 tx=0x7f5b800095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:44:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.411+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5b88014030 con 0x7f5b90107960 2024-09-12T02:44:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.588+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5b9002da10 con 0x7f5b90107960 2024-09-12T02:44:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.589+0000 7f5b7f7fe640 1 -- 172.21.15.16:0/4060535828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f5b8809c720 con 0x7f5b90107960 2024-09-12T02:44:59.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.590+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 msgr2=0x7f5b780a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.590+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 0x7f5b780a5070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b80000c00 tx=0x7f5b800095f0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.590+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 msgr2=0x7f5b9019f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.590+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800cce0 tx=0x7f5b8800a620 comp rx=0 tx=0).stop 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 shutdown_connections 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2bb0 0x7f5b780a5070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b90107960 0x7f5b9019f470 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b90105030 0x7f5b901983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 --2- 172.21.15.16:0/4060535828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b90102700 0x7f5b90197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:44:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 >> 172.21.15.16:0/4060535828 conn(0x7f5b900fc280 msgr2=0x7f5b90103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:44:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 shutdown_connections 2024-09-12T02:44:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:44:59.591+0000 7f5b94ef5640 1 -- 172.21.15.16:0/4060535828 wait complete. 2024-09-12T02:45:00.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:59 smithi016 ceph-mon[30925]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:45:00.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:44:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4060535828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:45:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:59 smithi033 ceph-mon[35092]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:45:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:44:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4060535828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:45:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:59 smithi184 ceph-mon[35321]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-09-12T02:45:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:44:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4060535828' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-12T02:45:00.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1726109098,"num_in_osds":12,"osd_in_since":1726109074,"num_remapped_pgs":0} 2024-09-12T02:45:00.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:45:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:01.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:00 smithi016 ceph-mon[30925]: osdmap e73: 12 total, 12 up, 12 in 2024-09-12T02:45:01.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 -- 172.21.15.16:0/4210876892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3370103660 msgr2=0x7f337010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:01.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 --2- 172.21.15.16:0/4210876892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3370103660 0x7f337010bb50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3360009900 tx=0x7f336002a480 comp rx=0 tx=0).stop 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 -- 172.21.15.16:0/4210876892 shutdown_connections 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 --2- 172.21.15.16:0/4210876892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f337010c170 0x7f337010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 --2- 172.21.15.16:0/4210876892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3370103660 0x7f337010bb50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 --2- 172.21.15.16:0/4210876892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 0x7f3370103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 -- 172.21.15.16:0/4210876892 >> 172.21.15.16:0/4210876892 conn(0x7f33700fc7a0 msgr2=0x7f33700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 -- 172.21.15.16:0/4210876892 shutdown_connections 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.036+0000 7f33785d5640 1 -- 172.21.15.16:0/4210876892 wait complete. 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 Processor -- start 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 -- start start 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 0x7f33701984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3370103660 0x7f33701989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:01.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3370111fb0 con 0x7f337010c170 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3370111e30 con 0x7f3370103660 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f33785d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3370112130 con 0x7f3370102d40 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f3376b72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f3376b72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37784/0 (socket says 172.21.15.16:37784) 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f3375b70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3370103660 0x7f33701989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f3376b72640 1 -- 172.21.15.16:0/3465168837 learned_addr learned my addr 172.21.15.16:0/3465168837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.037+0000 7f3376371640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 0x7f33701984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 -- 172.21.15.16:0/3465168837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 msgr2=0x7f33701984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 0x7f33701984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 -- 172.21.15.16:0/3465168837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3370103660 msgr2=0x7f33701989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3370103660 0x7f33701989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 -- 172.21.15.16:0/3465168837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337019d460 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f3376b72640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f336800eaa0 tx=0x7f336800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336800cdb0 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337019d750 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368005020 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680116b0 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.038+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a5030 con 0x7f337010c170 2024-09-12T02:45:01.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.039+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337006b070 con 0x7f337010c170 2024-09-12T02:45:01.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.040+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3368005560 con 0x7f337010c170 2024-09-12T02:45:01.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.041+0000 7f33677fe640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 0x7f335c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:01.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.041+0000 7f3376371640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 0x7f335c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:01.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.041+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f33680ceea0 con 0x7f337010c170 2024-09-12T02:45:01.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.042+0000 7f3376371640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 0x7f335c0a5040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f336c009890 tx=0x7f336c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:01.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.045+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3368016030 con 0x7f337010c170 2024-09-12T02:45:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.220+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3370107210 con 0x7f337010c170 2024-09-12T02:45:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.221+0000 7f33677fe640 1 -- 172.21.15.16:0/3465168837 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18428 (secure 0 0 0) 0x7f336809c200 con 0x7f337010c170 2024-09-12T02:45:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:01.222 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:44:59.842404+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:45:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 msgr2=0x7f335c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 0x7f335c0a5040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f336c009890 tx=0x7f336c008040 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 msgr2=0x7f337019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f336800eaa0 tx=0x7f336800ef60 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 shutdown_connections 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f335c0a2b80 0x7f335c0a5040 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f337010c170 0x7f337019cd80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3370103660 0x7f33701989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 --2- 172.21.15.16:0/3465168837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3370102d40 0x7f33701984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:01.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 >> 172.21.15.16:0/3465168837 conn(0x7f33700fc7a0 msgr2=0x7f33700fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:01.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 shutdown_connections 2024-09-12T02:45:01.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:01.223+0000 7f33785d5640 1 -- 172.21.15.16:0/3465168837 wait complete. 2024-09-12T02:45:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:00 smithi033 ceph-mon[35092]: osdmap e73: 12 total, 12 up, 12 in 2024-09-12T02:45:01.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:00 smithi184 ceph-mon[35321]: osdmap e73: 12 total, 12 up, 12 in 2024-09-12T02:45:01.892 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-12T02:41:16.432997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-12T02:45:01.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-12T02:45:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:01 smithi016 ceph-mon[30925]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:01 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3465168837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:01 smithi033 ceph-mon[35092]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:01 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3465168837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:01 smithi184 ceph-mon[35321]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:01 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3465168837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 -- 172.21.15.16:0/3586686907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128108f90 msgr2=0x7f2128110b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 --2- 172.21.15.16:0/3586686907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128108f90 0x7f2128110b10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2120009ca0 tx=0x7f212002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 -- 172.21.15.16:0/3586686907 shutdown_connections 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 --2- 172.21.15.16:0/3586686907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128108f90 0x7f2128110b10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 --2- 172.21.15.16:0/3586686907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128104810 0x7f2128108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 --2- 172.21.15.16:0/3586686907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128103e60 0x7f2128104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 -- 172.21.15.16:0/3586686907 >> 172.21.15.16:0/3586686907 conn(0x7f21280ff8e0 msgr2=0x7f2128101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 -- 172.21.15.16:0/3586686907 shutdown_connections 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.612+0000 7f212d803640 1 -- 172.21.15.16:0/3586686907 wait complete. 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 Processor -- start 2024-09-12T02:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 -- start start 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128103e60 0x7f212819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 0x7f212819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2128114010 con 0x7f2128104810 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2128113e90 con 0x7f2128108f90 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f212d803640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2128114190 con 0x7f2128103e60 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37804/0 (socket says 172.21.15.16:37804) 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 -- 172.21.15.16:0/1427611993 learned_addr learned my addr 172.21.15.16:0/1427611993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f21277fe640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 0x7f212819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f2126ffd640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128103e60 0x7f212819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 -- 172.21.15.16:0/1427611993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128103e60 msgr2=0x7f212819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128103e60 0x7f212819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 -- 172.21.15.16:0/1427611993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 msgr2=0x7f212819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 0x7f212819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 -- 172.21.15.16:0/1427611993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212819f480 con 0x7f2128104810 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f21277fe640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 0x7f212819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.613+0000 7f211ffff640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f211000b4a0 tx=0x7f211000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.614+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110014070 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.614+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2110010af0 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.614+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212819f770 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.614+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110017d10 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.614+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21281a7050 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.615+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2128104240 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.616+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2110017070 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.617+0000 7f212c801640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 0x7f21140a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.617+0000 7f2126ffd640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 0x7f21140a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.617+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f21100ce450 con 0x7f2128104810 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.617+0000 7f2126ffd640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 0x7f21140a50c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2118005fd0 tx=0x7f2118005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.620+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f211009b7b0 con 0x7f2128104810 2024-09-12T02:45:02.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.795+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f212802d7c0 con 0x7f2128104810 2024-09-12T02:45:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.795+0000 7f212c801640 1 -- 172.21.15.16:0/1427611993 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f211009b0b0 con 0x7f2128104810 2024-09-12T02:45:02.796 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-12T02:45:02.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 msgr2=0x7f21140a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 0x7f21140a50c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2118005fd0 tx=0x7f2118005850 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 msgr2=0x7f212819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f211000b4a0 tx=0x7f211000b960 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 shutdown_connections 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2128108f90 0x7f212819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21140a2c00 0x7f21140a50c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2128104810 0x7f212819aa10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 --2- 172.21.15.16:0/1427611993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2128103e60 0x7f212819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 >> 172.21.15.16:0/1427611993 conn(0x7f21280ff8e0 msgr2=0x7f2128100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 shutdown_connections 2024-09-12T02:45:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:02.797+0000 7f212d803640 1 -- 172.21.15.16:0/1427611993 wait complete. 2024-09-12T02:45:03.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:02 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1427611993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:45:03.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:02 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1427611993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:45:03.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:02 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1427611993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.b on smithi033 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.e on smithi033 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.h on smithi033 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.k on smithi033 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.c on smithi184 2024-09-12T02:45:03.394 INFO:tasks.cephadm:Adding mds.f on smithi184 2024-09-12T02:45:03.395 INFO:tasks.cephadm:Adding mds.i on smithi184 2024-09-12T02:45:03.395 INFO:tasks.cephadm:Adding mds.l on smithi184 2024-09-12T02:45:03.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l' 2024-09-12T02:45:03.655 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T02:45:04.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:03 smithi184 ceph-mon[35321]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 -- 172.21.15.184:0/1677335472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0102e00 msgr2=0x7f8ec0103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 --2- 172.21.15.184:0/1677335472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0102e00 0x7f8ec0103280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009900 tx=0x7f8eb402a410 comp rx=0 tx=0).stop 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 -- 172.21.15.184:0/1677335472 shutdown_connections 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 --2- 172.21.15.184:0/1677335472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 0x7f8ec010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 --2- 172.21.15.184:0/1677335472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0102e00 0x7f8ec0103280 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 --2- 172.21.15.184:0/1677335472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0101c00 0x7f8ec0102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 -- 172.21.15.184:0/1677335472 >> 172.21.15.184:0/1677335472 conn(0x7f8ec00fd390 msgr2=0x7f8ec00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 -- 172.21.15.184:0/1677335472 shutdown_connections 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.108+0000 7f8ec5937640 1 -- 172.21.15.184:0/1677335472 wait complete. 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.109+0000 7f8ec5937640 1 Processor -- start 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.109+0000 7f8ec5937640 1 -- start start 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.109+0000 7f8ec5937640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0102e00 0x7f8ec0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 0x7f8ec019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010d880 con 0x7f8ec01037c0 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010d700 con 0x7f8ec0102e00 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010da00 con 0x7f8ec0101c00 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:34998/0 (socket says 172.21.15.184:34998) 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 -- 172.21.15.184:0/2611525240 learned_addr learned my addr 172.21.15.184:0/2611525240 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebf7fe640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 0x7f8ec019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebe7fc640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0102e00 0x7f8ec0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 -- 172.21.15.184:0/2611525240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0102e00 msgr2=0x7f8ec0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0102e00 0x7f8ec0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 -- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 msgr2=0x7f8ec019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 0x7f8ec019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 -- 172.21.15.184:0/2611525240 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec019fb30 con 0x7f8ec0101c00 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ebeffd640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d950 tx=0x7f8eb000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:04.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0017070 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec019fe20 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb0009c30 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000f600 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.110+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec01a0220 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.111+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec0109050 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.113+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8eb001a070 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.114+0000 7f8ec4935640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 0x7f8ea80a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.114+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f8eb00ce5b0 con 0x7f8ec0101c00 2024-09-12T02:45:04.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.115+0000 7f8ebe7fc640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 0x7f8ea80a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:04.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.115+0000 7f8ebe7fc640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 0x7f8ea80a5070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb40315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:04.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.117+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8eb009b9b0 con 0x7f8ec0101c00 2024-09-12T02:45:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:03 smithi016 ceph-mon[30925]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:03 smithi033 ceph-mon[35092]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:04.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.300+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}) -- 0x7f8ec00627f0 con 0x7f8ea80a2bb0 2024-09-12T02:45:04.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.308+0000 7f8ec4935640 1 -- 172.21.15.184:0/2611525240 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f8ec00627f0 con 0x7f8ea80a2bb0 2024-09-12T02:45:04.308 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mds.all update... 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 msgr2=0x7f8ea80a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 0x7f8ea80a5070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb40315d0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 msgr2=0x7f8ec0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d950 tx=0x7f8eb000de10 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 shutdown_connections 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ea80a2bb0 0x7f8ea80a5070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec01037c0 0x7f8ec019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ec0102e00 0x7f8ec0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 --2- 172.21.15.184:0/2611525240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ec0101c00 0x7f8ec0197f00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 >> 172.21.15.184:0/2611525240 conn(0x7f8ec00fd390 msgr2=0x7f8ec0106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 shutdown_connections 2024-09-12T02:45:04.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-12T02:45:04.310+0000 7f8ec5937640 1 -- 172.21.15.184:0/2611525240 wait complete. 2024-09-12T02:45:04.915 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.a.service 2024-09-12T02:45:04.918 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.d.service 2024-09-12T02:45:04.961 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.g.service 2024-09-12T02:45:05.004 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.j.service 2024-09-12T02:45:05.047 DEBUG:teuthology.orchestra.run.smithi033:mds.b> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.b.service 2024-09-12T02:45:05.050 DEBUG:teuthology.orchestra.run.smithi033:mds.e> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.e.service 2024-09-12T02:45:05.094 DEBUG:teuthology.orchestra.run.smithi033:mds.h> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.h.service 2024-09-12T02:45:05.137 DEBUG:teuthology.orchestra.run.smithi033:mds.k> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.k.service 2024-09-12T02:45:05.180 DEBUG:teuthology.orchestra.run.smithi184:mds.c> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.c.service 2024-09-12T02:45:05.183 DEBUG:teuthology.orchestra.run.smithi184:mds.f> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.f.service 2024-09-12T02:45:05.225 DEBUG:teuthology.orchestra.run.smithi184:mds.i> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.i.service 2024-09-12T02:45:05.268 DEBUG:teuthology.orchestra.run.smithi184:mds.l> sudo journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.l.service 2024-09-12T02:45:05.311 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-12T02:45:05.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:05.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:05.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:05.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:05 smithi184 ceph-mon[35321]: Deploying daemon mds.l on smithi184 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:05.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:05.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:05 smithi033 ceph-mon[35092]: Deploying daemon mds.l on smithi184 2024-09-12T02:45:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:05 smithi016 ceph-mon[30925]: Deploying daemon mds.l on smithi184 2024-09-12T02:45:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 -- 172.21.15.16:0/3834724039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08106f10 msgr2=0x7f8a0810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 --2- 172.21.15.16:0/3834724039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08106f10 0x7f8a0810ea90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009ca0 tx=0x7f8a0002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 -- 172.21.15.16:0/3834724039 shutdown_connections 2024-09-12T02:45:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 --2- 172.21.15.16:0/3834724039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08106f10 0x7f8a0810ea90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 --2- 172.21.15.16:0/3834724039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a081067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 --2- 172.21.15.16:0/3834724039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08101de0 0x7f8a081021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 -- 172.21.15.16:0/3834724039 >> 172.21.15.16:0/3834724039 conn(0x7f8a080fd8c0 msgr2=0x7f8a080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 -- 172.21.15.16:0/3834724039 shutdown_connections 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.000+0000 7f8a0cada640 1 -- 172.21.15.16:0/3834724039 wait complete. 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 Processor -- start 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 -- start start 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a0819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 0x7f8a0819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a08111f70 con 0x7f8a08101de0 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a08111df0 con 0x7f8a08106f10 2024-09-12T02:45:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0cada640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a081120f0 con 0x7f8a08102790 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47520/0 (socket says 172.21.15.16:47520) 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a0659c640 1 -- 172.21.15.16:0/1927932984 learned_addr learned my addr 172.21.15.16:0/1927932984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.001+0000 7f8a05d9b640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a0819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a06d9d640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 0x7f8a0819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 -- 172.21.15.16:0/1927932984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 msgr2=0x7f8a0819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a0819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 -- 172.21.15.16:0/1927932984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 msgr2=0x7f8a0819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 0x7f8a0819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 -- 172.21.15.16:0/1927932984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0819f5d0 con 0x7f8a08101de0 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a05d9b640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a0819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a06d9d640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 0x7f8a0819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0659c640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f89f800c890 tx=0x7f89f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8004220 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0819f8c0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f80043c0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8004a00 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.002+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a081a71a0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.003+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a081021c0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.004+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f89f80079b0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.005+0000 7f89f77fe640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.005+0000 7f8a05d9b640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.005+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f89f80ce9f0 con 0x7f8a08101de0 2024-09-12T02:45:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.005+0000 7f8a05d9b640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89fc005fd0 tx=0x7f89fc00b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.009+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f89f8014030 con 0x7f8a08101de0 2024-09-12T02:45:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.222+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a081a7400 con 0x7f8a08101de0 2024-09-12T02:45:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.222+0000 7f89f77fe640 1 -- 172.21.15.16:0/1927932984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f89f809bd50 con 0x7f8a08101de0 2024-09-12T02:45:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:06.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"btime":"2024-09-12T02:38:12:619852+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-12T02:45:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 msgr2=0x7f89f00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89fc005fd0 tx=0x7f89fc00b5d0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 msgr2=0x7f8a0819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f89f800c890 tx=0x7f89f800cd50 comp rx=0 tx=0).stop 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 shutdown_connections 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.224+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a08106f10 0x7f8a0819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a08102790 0x7f8a0819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 --2- 172.21.15.16:0/1927932984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a08101de0 0x7f8a0819a620 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 >> 172.21.15.16:0/1927932984 conn(0x7f8a080fd8c0 msgr2=0x7f8a080febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 shutdown_connections 2024-09-12T02:45:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:06.225+0000 7f8a0cada640 1 -- 172.21.15.16:0/1927932984 wait complete. 2024-09-12T02:45:06.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:06 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1927932984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:06.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:06 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1927932984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:06.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:06 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1927932984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:06.756 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:06 smithi184 systemd[1]: Starting Ceph mds.l for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:07.007 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:06 smithi184 podman[65531]: 2024-09-12 02:45:06.658390113 +0000 UTC m=+0.015343674 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:07.008 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:06 smithi184 podman[65531]: 2024-09-12 02:45:06.777824884 +0000 UTC m=+0.134778473 container create 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-12T02:45:07.008 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:06 smithi184 podman[65531]: 2024-09-12 02:45:06.980105078 +0000 UTC m=+0.337058630 container init 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-12T02:45:07.008 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:06 smithi184 podman[65531]: 2024-09-12 02:45:06.982849735 +0000 UTC m=+0.339803294 container start 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-12T02:45:07.008 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mds[65550]: -- 172.21.15.184:0/588062316 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ecfd73b860 con 0x55ecfd719400 2024-09-12T02:45:07.008 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: starting mds.l at 2024-09-12T02:45:07.295 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:07 smithi184 bash[65531]: 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 2024-09-12T02:45:07.295 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 02:45:07 smithi184 systemd[1]: Started Ceph mds.l for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6832/897342399,v1:172.21.15.184:6833/897342399] up:boot 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: fsmap 1 up:standby 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:07.575 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6832/897342399,v1:172.21.15.184:6833/897342399] up:boot 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: fsmap 1 up:standby 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:07.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:07.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6832/897342399,v1:172.21.15.184:6833/897342399] up:boot 2024-09-12T02:45:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: fsmap 1 up:standby 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:08.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:08 smithi016 ceph-mon[30925]: Deploying daemon mds.k on smithi033 2024-09-12T02:45:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:08 smithi033 ceph-mon[35092]: Deploying daemon mds.k on smithi033 2024-09-12T02:45:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:08 smithi184 ceph-mon[35321]: Deploying daemon mds.k on smithi033 2024-09-12T02:45:08.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:09.321 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 systemd[1]: Starting Ceph mds.k for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:09.592 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:09 smithi033 ceph-mon[35092]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:09.592 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 podman[64217]: 2024-09-12 02:45:09.347223943 +0000 UTC m=+0.016268821 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:09.592 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 podman[64217]: 2024-09-12 02:45:09.47824281 +0000 UTC m=+0.147287718 container create 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T02:45:09.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:09 smithi016 ceph-mon[30925]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 -- 172.21.15.16:0/133133099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148101420 msgr2=0x7f014810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 --2- 172.21.15.16:0/133133099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148101420 0x7f014810d950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0140009cd0 tx=0x7f014002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 -- 172.21.15.16:0/133133099 shutdown_connections 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 --2- 172.21.15.16:0/133133099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148101420 0x7f014810d950 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 --2- 172.21.15.16:0/133133099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148100a80 0x7f0148100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 --2- 172.21.15.16:0/133133099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148106a80 0x7f0148106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 -- 172.21.15.16:0/133133099 >> 172.21.15.16:0/133133099 conn(0x7f01480fc7a0 msgr2=0x7f01480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 -- 172.21.15.16:0/133133099 shutdown_connections 2024-09-12T02:45:09.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.684+0000 7f014ccff640 1 -- 172.21.15.16:0/133133099 wait complete. 2024-09-12T02:45:09.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 Processor -- start 2024-09-12T02:45:09.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 -- start start 2024-09-12T02:45:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148101420 0x7f0148197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0148110e80 con 0x7f0148106a80 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0148110d00 con 0x7f0148100a80 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014ccff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0148111000 con 0x7f0148101420 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f0146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41662/0 (socket says 172.21.15.16:41662) 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f0146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47528/0 (socket says 172.21.15.16:47528) 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f014659c640 1 -- 172.21.15.16:0/3026046985 learned_addr learned my addr 172.21.15.16:0/3026046985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.685+0000 7f0145d9b640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148101420 0x7f0148197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 -- 172.21.15.16:0/3026046985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148101420 msgr2=0x7f0148197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148101420 0x7f0148197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 -- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 msgr2=0x7f014819bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 -- 172.21.15.16:0/3026046985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014819c2b0 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f0146d9d640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014659c640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f013800e9f0 tx=0x7f013800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013800cd80 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0138010910 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014819c5a0 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138018990 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.686+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014806a790 con 0x7f0148100a80 2024-09-12T02:45:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.687+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0148100ee0 con 0x7f0148100a80 2024-09-12T02:45:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.688+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0138010ac0 con 0x7f0148100a80 2024-09-12T02:45:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.689+0000 7f013f7fe640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 0x7f01340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.689+0000 7f0145d9b640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 0x7f01340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.689+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f01380cf450 con 0x7f0148100a80 2024-09-12T02:45:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.690+0000 7f0145d9b640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 0x7f01340a5040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01480fed40 tx=0x7f013000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:09.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.692+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f013809c160 con 0x7f0148100a80 2024-09-12T02:45:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:09 smithi184 ceph-mon[35321]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:09.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.906+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f014806b1b0 con 0x7f0148100a80 2024-09-12T02:45:09.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.907+0000 7f013f7fe640 1 -- 172.21.15.16:0/3026046985 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1435 (secure 0 0 0) 0x7f01380a0460 con 0x7f0148100a80 2024-09-12T02:45:09.908 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:09.908 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"btime":"2024-09-12T02:45:07:010989+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-12T02:45:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 msgr2=0x7f01340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 0x7f01340a5040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01480fed40 tx=0x7f013000a250 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 msgr2=0x7f0148197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f013800e9f0 tx=0x7f013800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 shutdown_connections 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01340a2b80 0x7f01340a5040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0148106a80 0x7f014819bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0148101420 0x7f0148197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 --2- 172.21.15.16:0/3026046985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0148100a80 0x7f0148197300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 >> 172.21.15.16:0/3026046985 conn(0x7f01480fc7a0 msgr2=0x7f01480fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 shutdown_connections 2024-09-12T02:45:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:09.910+0000 7f014ccff640 1 -- 172.21.15.16:0/3026046985 wait complete. 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 podman[64217]: 2024-09-12 02:45:09.697272958 +0000 UTC m=+0.366317825 container init 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 podman[64217]: 2024-09-12 02:45:09.700090358 +0000 UTC m=+0.369135225 container start 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 ceph-mds[64237]: -- 172.21.15.33:0/3379995974 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563787bd7860 con 0x563787bb5400 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: starting mds.k at 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 bash[64217]: 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc 2024-09-12T02:45:09.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 02:45:09 smithi033 systemd[1]: Started Ceph mds.k for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:10.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:10.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:10.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:10.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: Deploying daemon mds.j on smithi016 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3026046985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6832/1900353998,v1:172.21.15.33:6833/1900353998] up:boot 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: fsmap 2 up:standby 2024-09-12T02:45:10.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:11.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: Deploying daemon mds.j on smithi016 2024-09-12T02:45:11.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3026046985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:11.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6832/1900353998,v1:172.21.15.33:6833/1900353998] up:boot 2024-09-12T02:45:11.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: fsmap 2 up:standby 2024-09-12T02:45:11.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T02:45:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: Deploying daemon mds.j on smithi016 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3026046985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6832/1900353998,v1:172.21.15.33:6833/1900353998] up:boot 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: fsmap 2 up:standby 2024-09-12T02:45:11.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T02:45:12.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:11 smithi016 ceph-mon[30925]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:12.132 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:11 smithi033 ceph-mon[35092]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:11 smithi184 ceph-mon[35321]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:12.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 systemd[1]: Starting Ceph mds.j for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:12.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:12.769 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 podman[70235]: 2024-09-12 02:45:12.407243401 +0000 UTC m=+0.016904303 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:12.769 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 podman[70235]: 2024-09-12 02:45:12.524023655 +0000 UTC m=+0.133684559 container create 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-12T02:45:12.769 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 podman[70235]: 2024-09-12 02:45:12.768670863 +0000 UTC m=+0.378331780 container init 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T02:45:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:13.097 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 podman[70235]: 2024-09-12 02:45:12.773443408 +0000 UTC m=+0.383104313 container start 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, io.buildah.version=1.37.2) 2024-09-12T02:45:13.097 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 ceph-mds[70272]: -- 172.21.15.16:0/4195694170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8f562d860 con 0x55d8f560b000 2024-09-12T02:45:13.097 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j[70250]: starting mds.j at 2024-09-12T02:45:13.097 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 bash[70235]: 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 2024-09-12T02:45:13.097 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 02:45:12 smithi016 systemd[1]: Started Ceph mds.j for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 -- 172.21.15.16:0/3354038835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724102790 msgr2=0x7fb7241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3354038835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724102790 0x7fb7241067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb718009900 tx=0x7fb71802a410 comp rx=0 tx=0).stop 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 -- 172.21.15.16:0/3354038835 shutdown_connections 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3354038835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724106f10 0x7fb72410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3354038835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724102790 0x7fb7241067e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3354038835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724101de0 0x7fb7241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 -- 172.21.15.16:0/3354038835 >> 172.21.15.16:0/3354038835 conn(0x7fb7240fd8c0 msgr2=0x7fb7240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 -- 172.21.15.16:0/3354038835 shutdown_connections 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.462+0000 7fb7295a5640 1 -- 172.21.15.16:0/3354038835 wait complete. 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 Processor -- start 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 -- start start 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724102790 0x7fb72419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 0x7fb72419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb724111f30 con 0x7fb724101de0 2024-09-12T02:45:13.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb724111db0 con 0x7fb724106f10 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7295a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb7241120b0 con 0x7fb724102790 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb722ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb722ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47550/0 (socket says 172.21.15.16:47550) 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb722ffd640 1 -- 172.21.15.16:0/3333769941 learned_addr learned my addr 172.21.15.16:0/3333769941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.463+0000 7fb7237fe640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 0x7fb72419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 -- 172.21.15.16:0/3333769941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724102790 msgr2=0x7fb72419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb7227fc640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724102790 0x7fb72419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724102790 0x7fb72419ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 -- 172.21.15.16:0/3333769941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 msgr2=0x7fb72419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 0x7fb72419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 -- 172.21.15.16:0/3333769941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72419f6a0 con 0x7fb724101de0 2024-09-12T02:45:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb7237fe640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 0x7fb72419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb722ffd640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d950 tx=0x7fb71400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714017030 con 0x7fb724101de0 2024-09-12T02:45:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72419f990 con 0x7fb724101de0 2024-09-12T02:45:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb714004020 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140047a0 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.464+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7241a7270 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.465+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7241021c0 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.466+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb7140041d0 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.467+0000 7fb70bfff640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 0x7fb70c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.467+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fb7140ce640 con 0x7fb724101de0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.467+0000 7fb7227fc640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 0x7fb70c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.467+0000 7fb7227fc640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 0x7fb70c0a5040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb7180096c0 tx=0x7fb718000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.471+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb71409ba40 con 0x7fb724101de0 2024-09-12T02:45:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.689+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb72419ba50 con 0x7fb724101de0 2024-09-12T02:45:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.690+0000 7fb70bfff640 1 -- 172.21.15.16:0/3333769941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3085 (secure 0 0 0) 0x7fb71409b340 con 0x7fb724101de0 2024-09-12T02:45:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"btime":"2024-09-12T02:45:12:860094+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-12T02:45:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-09-12T02:45:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 msgr2=0x7fb70c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 0x7fb70c0a5040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb7180096c0 tx=0x7fb718000bc0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 msgr2=0x7fb72419a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d950 tx=0x7fb71400de10 comp rx=0 tx=0).stop 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 shutdown_connections 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb724106f10 0x7fb72419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb724102790 0x7fb72419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb70c0a2b80 0x7fb70c0a5040 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 --2- 172.21.15.16:0/3333769941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb724101de0 0x7fb72419a6f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 >> 172.21.15.16:0/3333769941 conn(0x7fb7240fd8c0 msgr2=0x7fb7240fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 shutdown_connections 2024-09-12T02:45:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:13.692+0000 7fb7295a5640 1 -- 172.21.15.16:0/3333769941 wait complete. 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6834/1858508080,v1:172.21.15.16:6835/1858508080] up:boot 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: fsmap 3 up:standby 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:13.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:13 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3333769941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6834/1858508080,v1:172.21.15.16:6835/1858508080] up:boot 2024-09-12T02:45:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: fsmap 3 up:standby 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:13 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3333769941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6834/1858508080,v1:172.21.15.16:6835/1858508080] up:boot 2024-09-12T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: fsmap 3 up:standby 2024-09-12T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:13 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3333769941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:15.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:14 smithi184 ceph-mon[35321]: Deploying daemon mds.d on smithi016 2024-09-12T02:45:15.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:14 smithi016 ceph-mon[30925]: Deploying daemon mds.d on smithi016 2024-09-12T02:45:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:14 smithi033 ceph-mon[35092]: Deploying daemon mds.d on smithi016 2024-09-12T02:45:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:15 smithi016 ceph-mon[30925]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:16.158 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 systemd[1]: Starting Ceph mds.d for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:15 smithi033 ceph-mon[35092]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:15 smithi184 ceph-mon[35321]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:16.483 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 podman[70651]: 2024-09-12 02:45:16.130647171 +0000 UTC m=+0.016515623 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:16.483 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 podman[70651]: 2024-09-12 02:45:16.252688888 +0000 UTC m=+0.138557340 container create 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d, RELEASE=squid-b53b6d3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-12T02:45:16.483 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 podman[70651]: 2024-09-12 02:45:16.455080482 +0000 UTC m=+0.340948933 container init 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3) 2024-09-12T02:45:16.483 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 podman[70651]: 2024-09-12 02:45:16.458416557 +0000 UTC m=+0.344285011 container start 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-12T02:45:16.483 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 ceph-mds[70670]: -- 172.21.15.16:0/297170647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627bc6bd860 con 0x5627bc69b400 2024-09-12T02:45:16.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:16.908 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d[70666]: starting mds.d at 2024-09-12T02:45:16.908 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 bash[70651]: 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 2024-09-12T02:45:16.908 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 02:45:16 smithi016 systemd[1]: Started Ceph mds.d for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:17.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 -- 172.21.15.16:0/3830124065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102d40 msgr2=0x7fa038103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 --2- 172.21.15.16:0/3830124065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102d40 0x7fa038103120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa028009900 tx=0x7fa02802a480 comp rx=0 tx=0).stop 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 -- 172.21.15.16:0/3830124065 shutdown_connections 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 --2- 172.21.15.16:0/3830124065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa03810c170 0x7fa03810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 --2- 172.21.15.16:0/3830124065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa03810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 --2- 172.21.15.16:0/3830124065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102d40 0x7fa038103120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 -- 172.21.15.16:0/3830124065 >> 172.21.15.16:0/3830124065 conn(0x7fa0380fc7a0 msgr2=0x7fa0380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 -- 172.21.15.16:0/3830124065 shutdown_connections 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.240+0000 7fa03e11e640 1 -- 172.21.15.16:0/3830124065 wait complete. 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.241+0000 7fa03e11e640 1 Processor -- start 2024-09-12T02:45:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.241+0000 7fa03e11e640 1 -- start start 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa038102d40 0x7fa0381009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa038100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa038111f70 con 0x7fa03810c170 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa038111df0 con 0x7fa038103660 2024-09-12T02:45:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa03e11e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381120f0 con 0x7fa038102d40 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55856/0 (socket says 172.21.15.16:55856) 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 -- 172.21.15.16:0/1845363260 learned_addr learned my addr 172.21.15.16:0/1845363260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa0377fe640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa038102d40 0x7fa0381009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa036ffd640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa038100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 -- 172.21.15.16:0/1845363260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa038102d40 msgr2=0x7fa0381009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa038102d40 0x7fa0381009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 -- 172.21.15.16:0/1845363260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 msgr2=0x7fa038100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa038100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 -- 172.21.15.16:0/1845363260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0381a68d0 con 0x7fa03810c170 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa036ffd640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa038100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.242+0000 7fa037fff640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cad0 tx=0x7fa03000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.243+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03000ae20 con 0x7fa03810c170 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.243+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa030005020 con 0x7fa03810c170 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.243+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03000f6f0 con 0x7fa03810c170 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.243+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0381a6bc0 con 0x7fa03810c170 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.243+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0381a70f0 con 0x7fa03810c170 2024-09-12T02:45:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.244+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa038103120 con 0x7fa03810c170 2024-09-12T02:45:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.245+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa0300051d0 con 0x7fa03810c170 2024-09-12T02:45:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.246+0000 7fa034ff9640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 0x7fa0200a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.246+0000 7fa0377fe640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 0x7fa0200a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.246+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fa0300cee00 con 0x7fa03810c170 2024-09-12T02:45:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.246+0000 7fa0377fe640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 0x7fa0200a50c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa0280058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.250+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa030014030 con 0x7fa03810c170 2024-09-12T02:45:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.463+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0381a7560 con 0x7fa03810c170 2024-09-12T02:45:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.463+0000 7fa034ff9640 1 -- 172.21.15.16:0/1845363260 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7fa03009c200 con 0x7fa03810c170 2024-09-12T02:45:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:17.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"btime":"2024-09-12T02:45:16:873088+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-12T02:45:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-09-12T02:45:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 msgr2=0x7fa0200a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 0x7fa0200a50c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa0280058d0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 msgr2=0x7fa038101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cad0 tx=0x7fa03000cf90 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 shutdown_connections 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0200a2c00 0x7fa0200a50c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03810c170 0x7fa038101420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa038103660 0x7fa038100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 --2- 172.21.15.16:0/1845363260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa038102d40 0x7fa0381009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 >> 172.21.15.16:0/1845363260 conn(0x7fa0380fc7a0 msgr2=0x7fa0380fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 shutdown_connections 2024-09-12T02:45:17.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:17.465+0000 7fa03e11e640 1 -- 172.21.15.16:0/1845363260 wait complete. 2024-09-12T02:45:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: Deploying daemon mds.e on smithi033 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6836/3544362280,v1:172.21.15.16:6837/3544362280] up:boot 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: fsmap 4 up:standby 2024-09-12T02:45:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-12T02:45:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:17 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1845363260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: Deploying daemon mds.e on smithi033 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6836/3544362280,v1:172.21.15.16:6837/3544362280] up:boot 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: fsmap 4 up:standby 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-12T02:45:17.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:17 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1845363260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: Deploying daemon mds.e on smithi033 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6836/3544362280,v1:172.21.15.16:6837/3544362280] up:boot 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: fsmap 4 up:standby 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-12T02:45:18.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:17 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1845363260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:18.673 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 systemd[1]: Starting Ceph mds.e for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:18.934 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 podman[64531]: 2024-09-12 02:45:18.575674758 +0000 UTC m=+0.015685543 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:18.934 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 podman[64531]: 2024-09-12 02:45:18.721994439 +0000 UTC m=+0.162005215 container create 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-12T02:45:18.934 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 podman[64531]: 2024-09-12 02:45:18.907255338 +0000 UTC m=+0.347266117 container init 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-12T02:45:18.934 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 podman[64531]: 2024-09-12 02:45:18.910052196 +0000 UTC m=+0.350062975 container start 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-12T02:45:18.934 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 ceph-mds[64550]: -- 172.21.15.33:0/840228948 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c3767b860 con 0x556c37659400 2024-09-12T02:45:19.247 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:18 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: starting mds.e at 2024-09-12T02:45:19.247 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:19 smithi033 bash[64531]: 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 2024-09-12T02:45:19.247 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 02:45:19 smithi033 systemd[1]: Started Ceph mds.e for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:19.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:19.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:19.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:19.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:19.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:20.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:20.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:20 smithi016 ceph-mon[30925]: Deploying daemon mds.f on smithi184 2024-09-12T02:45:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:20 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:boot 2024-09-12T02:45:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:20 smithi016 ceph-mon[30925]: fsmap 5 up:standby 2024-09-12T02:45:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:20 smithi033 ceph-mon[35092]: Deploying daemon mds.f on smithi184 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:20 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:boot 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:20 smithi033 ceph-mon[35092]: fsmap 5 up:standby 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:20 smithi184 ceph-mon[35321]: Deploying daemon mds.f on smithi184 2024-09-12T02:45:20.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:20 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:boot 2024-09-12T02:45:20.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:20 smithi184 ceph-mon[35321]: fsmap 5 up:standby 2024-09-12T02:45:20.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T02:45:20.998 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:20 smithi184 systemd[1]: Starting Ceph mds.f for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 -- 172.21.15.16:0/3608920265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0102790 msgr2=0x7f01f01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/3608920265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0102790 0x7f01f01067e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009900 tx=0x7f01e002a410 comp rx=0 tx=0).stop 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 -- 172.21.15.16:0/3608920265 shutdown_connections 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/3608920265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0106f10 0x7f01f010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/3608920265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0102790 0x7f01f01067e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/3608920265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 0x7f01f01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 -- 172.21.15.16:0/3608920265 >> 172.21.15.16:0/3608920265 conn(0x7f01f00fd8c0 msgr2=0x7f01f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 -- 172.21.15.16:0/3608920265 shutdown_connections 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.995+0000 7f01f6cb2640 1 -- 172.21.15.16:0/3608920265 wait complete. 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 Processor -- start 2024-09-12T02:45:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 -- start start 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 0x7f01f00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0102790 0x7f01f0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0111f90 con 0x7f01f0106f10 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0111e10 con 0x7f01f0101de0 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f6cb2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0112110 con 0x7f01f0102790 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0102790 0x7f01f0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55884/0 (socket says 172.21.15.16:55884) 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 -- 172.21.15.16:0/215390582 learned_addr learned my addr 172.21.15.16:0/215390582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f4a4e640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 0x7f01f00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 -- 172.21.15.16:0/215390582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0102790 msgr2=0x7f01f0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0102790 0x7f01f0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 -- 172.21.15.16:0/215390582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 msgr2=0x7f01f00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 0x7f01f00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.996+0000 7f01f524f640 1 -- 172.21.15.16:0/215390582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f01a69e0 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01f524f640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b970 tx=0x7f01e800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8019070 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01e800cb40 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f01a6cd0 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8007650 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.997+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f01a7230 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.998+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f01021c0 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:20.999+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f01e801c070 con 0x7f01f0106f10 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.000+0000 7f01e5ffb640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 0x7f01d80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.000+0000 7f01f4a4e640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 0x7f01d80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.000+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f01e80ce150 con 0x7f01f0106f10 2024-09-12T02:45:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.000+0000 7f01f4a4e640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 0x7f01d80a50c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005fd0 tx=0x7f01ec005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.004+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f01e809ae60 con 0x7f01f0106f10 2024-09-12T02:45:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.214+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01f010a150 con 0x7f01f0106f10 2024-09-12T02:45:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.215+0000 7f01e5ffb640 1 -- 172.21.15.16:0/215390582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4735 (secure 0 0 0) 0x7f01e809f160 con 0x7f01f0106f10 2024-09-12T02:45:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:21.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"btime":"2024-09-12T02:45:19:594058+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-12T02:45:21.217 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-09-12T02:45:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 msgr2=0x7f01d80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 0x7f01d80a50c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005fd0 tx=0x7f01ec005850 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 msgr2=0x7f01f0074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b970 tx=0x7f01e800be30 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 shutdown_connections 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01d80a2c00 0x7f01d80a50c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f0106f10 0x7f01f0074f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f0102790 0x7f01f0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 --2- 172.21.15.16:0/215390582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f0101de0 0x7f01f00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 >> 172.21.15.16:0/215390582 conn(0x7f01f00fd8c0 msgr2=0x7f01f00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 shutdown_connections 2024-09-12T02:45:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:21.217+0000 7f01f6cb2640 1 -- 172.21.15.16:0/215390582 wait complete. 2024-09-12T02:45:21.249 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:20 smithi184 podman[65842]: 2024-09-12 02:45:20.996792207 +0000 UTC m=+0.099021813 container create 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-12T02:45:21.250 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 podman[65842]: 2024-09-12 02:45:20.913298727 +0000 UTC m=+0.015528333 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:21.250 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 podman[65842]: 2024-09-12 02:45:21.249372273 +0000 UTC m=+0.351601871 container init 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-12T02:45:21.545 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 podman[65842]: 2024-09-12 02:45:21.252410995 +0000 UTC m=+0.354640602 container start 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-12T02:45:21.545 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mds[65861]: -- 172.21.15.184:0/1959304423 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae760e9860 con 0x55ae760c7400 2024-09-12T02:45:21.545 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: starting mds.f at 2024-09-12T02:45:21.545 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 bash[65842]: 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 2024-09-12T02:45:21.545 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 02:45:21 smithi184 systemd[1]: Started Ceph mds.f for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/215390582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/215390582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:21.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/215390582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:22 smithi184 ceph-mon[35321]: Deploying daemon mds.c on smithi184 2024-09-12T02:45:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:22 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6834/1041505607,v1:172.21.15.184:6835/1041505607] up:boot 2024-09-12T02:45:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:22 smithi184 ceph-mon[35321]: fsmap 6 up:standby 2024-09-12T02:45:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:22 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T02:45:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:22 smithi016 ceph-mon[30925]: Deploying daemon mds.c on smithi184 2024-09-12T02:45:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:22 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6834/1041505607,v1:172.21.15.184:6835/1041505607] up:boot 2024-09-12T02:45:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:22 smithi016 ceph-mon[30925]: fsmap 6 up:standby 2024-09-12T02:45:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T02:45:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:22 smithi033 ceph-mon[35092]: Deploying daemon mds.c on smithi184 2024-09-12T02:45:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:22 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6834/1041505607,v1:172.21.15.184:6835/1041505607] up:boot 2024-09-12T02:45:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:22 smithi033 ceph-mon[35092]: fsmap 6 up:standby 2024-09-12T02:45:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:22 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T02:45:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:23 smithi016 ceph-mon[30925]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:23.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:23.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:23 smithi033 ceph-mon[35092]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:23 smithi184 ceph-mon[35321]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:24.045 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 02:45:23 smithi184 systemd[1]: Started Ceph mds.c for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:24.261 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.731+0000 7fb0f1871640 1 -- 172.21.15.16:0/1366489774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 msgr2=0x7fb0ec10ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.731+0000 7fb0f1871640 1 --2- 172.21.15.16:0/1366489774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec10ea20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009ca0 tx=0x7fb0e402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 -- 172.21.15.16:0/1366489774 shutdown_connections 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 --2- 172.21.15.16:0/1366489774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec10ea20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 --2- 172.21.15.16:0/1366489774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100100 0x7fb0ec100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 --2- 172.21.15.16:0/1366489774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec106b80 0x7fb0ec106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 -- 172.21.15.16:0/1366489774 >> 172.21.15.16:0/1366489774 conn(0x7fb0ec077230 msgr2=0x7fb0ec0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 -- 172.21.15.16:0/1366489774 shutdown_connections 2024-09-12T02:45:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 -- 172.21.15.16:0/1366489774 wait complete. 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 Processor -- start 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.732+0000 7fb0f1871640 1 -- start start 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec100100 0x7fb0ec198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec106b80 0x7fb0ec19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec112050 con 0x7fb0ec100aa0 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec111ed0 con 0x7fb0ec106b80 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0f1871640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec1121d0 con 0x7fb0ec100100 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55898/0 (socket says 172.21.15.16:55898) 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 -- 172.21.15.16:0/2537169321 learned_addr learned my addr 172.21.15.16:0/2537169321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0eaffd640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec100100 0x7fb0ec198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0eb7fe640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec106b80 0x7fb0ec19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 -- 172.21.15.16:0/2537169321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec100100 msgr2=0x7fb0ec198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec100100 0x7fb0ec198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 -- 172.21.15.16:0/2537169321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec106b80 msgr2=0x7fb0ec19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec106b80 0x7fb0ec19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 -- 172.21.15.16:0/2537169321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec19d420 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.733+0000 7fb0ea7fc640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000ea60 tx=0x7fb0e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.734+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000cde0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.734+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ec19d710 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.734+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0e00035f0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.734+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0010670 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.734+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ec1a4ff0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.735+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0ec100560 con 0x7fb0ec100aa0 2024-09-12T02:45:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.736+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb0e00037a0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.737+0000 7fb0f086f640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 0x7fb0d40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.737+0000 7fb0eaffd640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 0x7fb0d40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.737+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fb0e00cef10 con 0x7fb0ec100aa0 2024-09-12T02:45:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.737+0000 7fb0eaffd640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 0x7fb0d40a5040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc005d00 tx=0x7fb0dc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.744+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0e009c3a0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: Deploying daemon mds.h on smithi033 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6836/2803727914,v1:172.21.15.184:6837/2803727914] up:boot 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: fsmap 7 up:standby 2024-09-12T02:45:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:24 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T02:45:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.956+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0ec10a010 con 0x7fb0ec100aa0 2024-09-12T02:45:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.957+0000 7fb0f086f640 1 -- 172.21.15.16:0/2537169321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6391 (secure 0 0 0) 0x7fb0e009bca0 con 0x7fb0ec100aa0 2024-09-12T02:45:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:24.959 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"btime":"2024-09-12T02:45:24:620585+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-12T02:45:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.959+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 msgr2=0x7fb0d40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.959+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 0x7fb0d40a5040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc005d00 tx=0x7fb0dc00b410 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.959+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 msgr2=0x7fb0ec1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.959+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000ea60 tx=0x7fb0e000ef20 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 shutdown_connections 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec106b80 0x7fb0ec19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0d40a2b80 0x7fb0d40a5040 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec100aa0 0x7fb0ec1989b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 --2- 172.21.15.16:0/2537169321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec100100 0x7fb0ec198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 >> 172.21.15.16:0/2537169321 conn(0x7fb0ec077230 msgr2=0x7fb0ec0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 shutdown_connections 2024-09-12T02:45:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:24.960+0000 7fb0f1871640 1 -- 172.21.15.16:0/2537169321 wait complete. 2024-09-12T02:45:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:24.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:24.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: Deploying daemon mds.h on smithi033 2024-09-12T02:45:24.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6836/2803727914,v1:172.21.15.184:6837/2803727914] up:boot 2024-09-12T02:45:24.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: fsmap 7 up:standby 2024-09-12T02:45:24.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:24 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T02:45:25.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:25.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: Deploying daemon mds.h on smithi033 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6836/2803727914,v1:172.21.15.184:6837/2803727914] up:boot 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: fsmap 7 up:standby 2024-09-12T02:45:25.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:24 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T02:45:25.938 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:25 smithi033 ceph-mon[35092]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:25.938 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:25 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2537169321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:25.951 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:25 smithi016 ceph-mon[30925]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:25.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:25 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2537169321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:25 smithi184 ceph-mon[35321]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:25 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2537169321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:26.211 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 02:45:26 smithi033 podman[64832]: 2024-09-12 02:45:26.188032826 +0000 UTC m=+0.364380418 container start f10876fe397a1a812e8bef25c943c808b5a84b042239710194de9e9932b9e33e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-12T02:45:26.497 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 02:45:26 smithi033 ceph-mds[64851]: -- 172.21.15.33:0/1601417220 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559d2eb07860 con 0x559d2eae5400 2024-09-12T02:45:26.497 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 02:45:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: starting mds.h at 2024-09-12T02:45:26.497 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 02:45:26 smithi033 bash[64832]: f10876fe397a1a812e8bef25c943c808b5a84b042239710194de9e9932b9e33e 2024-09-12T02:45:26.497 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 02:45:26 smithi033 systemd[1]: Started Ceph mds.h for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: Deploying daemon mds.i on smithi184 2024-09-12T02:45:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:boot 2024-09-12T02:45:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: fsmap 8 up:standby 2024-09-12T02:45:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:27 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: Deploying daemon mds.i on smithi184 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:boot 2024-09-12T02:45:27.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: fsmap 8 up:standby 2024-09-12T02:45:27.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:27 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T02:45:27.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: Deploying daemon mds.i on smithi184 2024-09-12T02:45:27.750 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:boot 2024-09-12T02:45:27.751 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: fsmap 8 up:standby 2024-09-12T02:45:27.751 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:27 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T02:45:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:28.295 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 systemd[1]: Starting Ceph mds.i for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:28.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 -- 172.21.15.16:0/809926026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 msgr2=0x7fd1681003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:28.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/809926026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681003d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd164009900 tx=0x7fd16402a400 comp rx=0 tx=0).stop 2024-09-12T02:45:28.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 -- 172.21.15.16:0/809926026 shutdown_connections 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/809926026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168104f30 0x7fd1681053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/809926026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168100910 0x7fd1681049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/809926026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681003d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 -- 172.21.15.16:0/809926026 >> 172.21.15.16:0/809926026 conn(0x7fd1680fba90 msgr2=0x7fd1680fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 -- 172.21.15.16:0/809926026 shutdown_connections 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 -- 172.21.15.16:0/809926026 wait complete. 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.414+0000 7fd16f6ef640 1 Processor -- start 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 -- start start 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168100910 0x7fd168199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 0x7fd1681a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd168110e80 con 0x7fd1680ffff0 2024-09-12T02:45:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd168110d00 con 0x7fd168104f30 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16f6ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd168111000 con 0x7fd168100910 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55154/0 (socket says 172.21.15.16:55154) 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 -- 172.21.15.16:0/539873847 learned_addr learned my addr 172.21.15.16:0/539873847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16dc8c640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 0x7fd1681a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 -- 172.21.15.16:0/539873847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168100910 msgr2=0x7fd168199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16cc8a640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168100910 0x7fd168199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168100910 0x7fd168199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 -- 172.21.15.16:0/539873847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 msgr2=0x7fd1681a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 0x7fd1681a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16d48b640 1 -- 172.21.15.16:0/539873847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1681a49b0 con 0x7fd1680ffff0 2024-09-12T02:45:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.415+0000 7fd16dc8c640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 0x7fd1681a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd16d48b640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd164006010 tx=0x7fd164003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16403f070 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd164031c30 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164035030 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1681a4b80 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.416+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1681a4fd0 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.417+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1681049f0 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.418+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd164042070 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.419+0000 7fd15e7fc640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 0x7fd1540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.419+0000 7fd16cc8a640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 0x7fd1540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.419+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fd1640ef190 con 0x7fd1680ffff0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.419+0000 7fd16cc8a640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 0x7fd1540a5040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd158005fd0 tx=0x7fd158009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.423+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd164031de0 con 0x7fd1680ffff0 2024-09-12T02:45:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.637+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1681a5200 con 0x7fd1680ffff0 2024-09-12T02:45:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.638+0000 7fd15e7fc640 1 -- 172.21.15.16:0/539873847 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7216 (secure 0 0 0) 0x7fd1640bc590 con 0x7fd1680ffff0 2024-09-12T02:45:28.640 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"btime":"2024-09-12T02:45:26:774876+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-12T02:45:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 msgr2=0x7fd1540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 0x7fd1540a5040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd158005fd0 tx=0x7fd158009470 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 msgr2=0x7fd1681993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd164006010 tx=0x7fd164003c10 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 shutdown_connections 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd168104f30 0x7fd1681a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd168100910 0x7fd168199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1540a2b80 0x7fd1540a5040 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 --2- 172.21.15.16:0/539873847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1680ffff0 0x7fd1681993f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.640+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 >> 172.21.15.16:0/539873847 conn(0x7fd1680fba90 msgr2=0x7fd168107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.641+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 shutdown_connections 2024-09-12T02:45:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:28.641+0000 7fd16f6ef640 1 -- 172.21.15.16:0/539873847 wait complete. 2024-09-12T02:45:28.706 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 podman[66439]: 2024-09-12 02:45:28.340999978 +0000 UTC m=+0.016457986 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:28.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 podman[66439]: 2024-09-12 02:45:28.453501626 +0000 UTC m=+0.128959643 container create fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-12T02:45:28.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 podman[66439]: 2024-09-12 02:45:28.680883692 +0000 UTC m=+0.356341704 container init fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, ceph=True) 2024-09-12T02:45:28.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 podman[66439]: 2024-09-12 02:45:28.683719712 +0000 UTC m=+0.359177722 container start fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-12T02:45:29.045 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 ceph-mds[66458]: -- 172.21.15.184:0/2267047708 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5561e4541860 con 0x5561e451e800 2024-09-12T02:45:29.045 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: starting mds.i at 2024-09-12T02:45:29.045 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 bash[66439]: fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 2024-09-12T02:45:29.045 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 02:45:28 smithi184 systemd[1]: Started Ceph mds.i for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/539873847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:29.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:29.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:29.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:29 smithi016 ceph-mon[30925]: Deploying daemon mds.g on smithi016 2024-09-12T02:45:29.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:29.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/539873847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:29.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:29.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:29.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:29.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:29 smithi033 ceph-mon[35092]: Deploying daemon mds.g on smithi016 2024-09-12T02:45:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/539873847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:29.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:29.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:29.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:29.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:29 smithi184 ceph-mon[35321]: Deploying daemon mds.g on smithi016 2024-09-12T02:45:30.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:30 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:boot 2024-09-12T02:45:30.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:30 smithi016 ceph-mon[30925]: fsmap 9 up:standby 2024-09-12T02:45:30.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:30 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T02:45:30.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:30 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:boot 2024-09-12T02:45:30.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:30 smithi033 ceph-mon[35092]: fsmap 9 up:standby 2024-09-12T02:45:30.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:30 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T02:45:30.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:30 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:boot 2024-09-12T02:45:30.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:30 smithi184 ceph-mon[35321]: fsmap 9 up:standby 2024-09-12T02:45:30.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:30 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T02:45:31.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:31.341 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 podman[71422]: 2024-09-12 02:45:31.271195854 +0000 UTC m=+0.120660981 container create 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-12T02:45:31.593 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 podman[71422]: 2024-09-12 02:45:31.490295872 +0000 UTC m=+0.339760996 container init 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-12T02:45:31.593 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 podman[71422]: 2024-09-12 02:45:31.493613483 +0000 UTC m=+0.343078607 container start 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-12T02:45:31.594 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 ceph-mds[71459]: -- 172.21.15.16:0/3786877468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2c407fc20 con 0x55b2c405d400 2024-09-12T02:45:31.594 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: starting mds.g at 2024-09-12T02:45:31.594 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 bash[71422]: 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 2024-09-12T02:45:31.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:31 smithi016 ceph-mon[30925]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:31.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:31 smithi033 ceph-mon[35092]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:31.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:31 smithi184 ceph-mon[35321]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:31.907 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 02:45:31 smithi016 systemd[1]: Started Ceph mds.g for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 -- 172.21.15.16:0/165503812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880102790 msgr2=0x7f48801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 --2- 172.21.15.16:0/165503812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880102790 0x7f48801067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4870009900 tx=0x7f487002a410 comp rx=0 tx=0).stop 2024-09-12T02:45:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 -- 172.21.15.16:0/165503812 shutdown_connections 2024-09-12T02:45:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 --2- 172.21.15.16:0/165503812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880106f10 0x7f488010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 --2- 172.21.15.16:0/165503812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880102790 0x7f48801067e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 --2- 172.21.15.16:0/165503812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880101de0 0x7f48801021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 -- 172.21.15.16:0/165503812 >> 172.21.15.16:0/165503812 conn(0x7f48800fd8c0 msgr2=0x7f48800ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 -- 172.21.15.16:0/165503812 shutdown_connections 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.118+0000 7f4886de5640 1 -- 172.21.15.16:0/165503812 wait complete. 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 Processor -- start 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 -- start start 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880101de0 0x7f4880198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 0x7f4880198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4880111fc0 con 0x7f4880106f10 2024-09-12T02:45:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4880111e40 con 0x7f4880101de0 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4886de5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4880112140 con 0x7f4880102790 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4885382640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4885382640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55184/0 (socket says 172.21.15.16:55184) 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4885382640 1 -- 172.21.15.16:0/838202182 learned_addr learned my addr 172.21.15.16:0/838202182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4884b81640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880101de0 0x7f4880198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.119+0000 7f4877fff640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 0x7f4880198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 -- 172.21.15.16:0/838202182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 msgr2=0x7f4880198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 0x7f4880198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 -- 172.21.15.16:0/838202182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880101de0 msgr2=0x7f4880198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880101de0 0x7f4880198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 -- 172.21.15.16:0/838202182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488019d3f0 con 0x7f4880106f10 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4877fff640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 0x7f4880198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4885382640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f487800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878017070 con 0x7f4880106f10 2024-09-12T02:45:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488019d6e0 con 0x7f4880106f10 2024-09-12T02:45:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4878003e50 con 0x7f4880106f10 2024-09-12T02:45:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48780106c0 con 0x7f4880106f10 2024-09-12T02:45:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.120+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48801a4fc0 con 0x7f4880106f10 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.121+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48801021c0 con 0x7f4880106f10 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.122+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4878003970 con 0x7f4880106f10 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.123+0000 7f4875ffb640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 0x7f48680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.123+0000 7f4884b81640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 0x7f48680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.123+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f48780ce910 con 0x7f4880106f10 2024-09-12T02:45:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.123+0000 7f4884b81640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 0x7f48680a5040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f487c005fd0 tx=0x7f487c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.127+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f487809bd10 con 0x7f4880106f10 2024-09-12T02:45:32.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.347+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f488010a150 con 0x7f4880106f10 2024-09-12T02:45:32.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.347+0000 7f4875ffb640 1 -- 172.21.15.16:0/838202182 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8869 (secure 0 0 0) 0x7f487809b610 con 0x7f4880106f10 2024-09-12T02:45:32.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:32.349 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"btime":"2024-09-12T02:45:32:341592+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-12T02:45:32.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 msgr2=0x7f48680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 0x7f48680a5040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f487c005fd0 tx=0x7f487c00b390 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 msgr2=0x7f488019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f487800b260 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 shutdown_connections 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48680a2b80 0x7f48680a5040 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4880106f10 0x7f488019cd10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4880102790 0x7f4880198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.349+0000 7f4886de5640 1 --2- 172.21.15.16:0/838202182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4880101de0 0x7f4880198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.350+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 >> 172.21.15.16:0/838202182 conn(0x7f48800fd8c0 msgr2=0x7f48800fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:32.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.350+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 shutdown_connections 2024-09-12T02:45:32.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:32.350+0000 7f4886de5640 1 -- 172.21.15.16:0/838202182 wait complete. 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: Deploying daemon mds.a on smithi016 2024-09-12T02:45:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6838/2216233994,v1:172.21.15.16:6839/2216233994] up:boot 2024-09-12T02:45:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: fsmap 10 up:standby 2024-09-12T02:45:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T02:45:32.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/838202182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: Deploying daemon mds.a on smithi016 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6838/2216233994,v1:172.21.15.16:6839/2216233994] up:boot 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: fsmap 10 up:standby 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T02:45:32.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/838202182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:33.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:33.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:33.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:33.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:33.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:33.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: Deploying daemon mds.a on smithi016 2024-09-12T02:45:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6838/2216233994,v1:172.21.15.16:6839/2216233994] up:boot 2024-09-12T02:45:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: fsmap 10 up:standby 2024-09-12T02:45:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T02:45:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/838202182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:33 smithi016 ceph-mon[30925]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:33.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:33 smithi033 ceph-mon[35092]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:33 smithi184 ceph-mon[35321]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:34.579 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 systemd[1]: Starting Ceph mds.a for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T02:45:34.851 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 podman[71836]: 2024-09-12 02:45:34.481740401 +0000 UTC m=+0.017066004 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T02:45:34.851 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 podman[71836]: 2024-09-12 02:45:34.607198797 +0000 UTC m=+0.142524385 container create 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a, RELEASE=squid-b53b6d3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-12T02:45:34.851 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 podman[71836]: 2024-09-12 02:45:34.851390434 +0000 UTC m=+0.386716025 container init 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T02:45:35.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:35.157 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 podman[71836]: 2024-09-12 02:45:34.854526497 +0000 UTC m=+0.389852095 container start 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True) 2024-09-12T02:45:35.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 ceph-mds[71855]: -- 172.21.15.16:0/3006309038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f68d417860 con 0x55f68d3f4800 2024-09-12T02:45:35.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: starting mds.a at 2024-09-12T02:45:35.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 bash[71836]: 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 2024-09-12T02:45:35.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 02:45:34 smithi016 systemd[1]: Started Ceph mds.a for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T02:45:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:35.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:35 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:35.700 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:35.700 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.700 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.700 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:35.701 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:35.701 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:35.701 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:35 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/1699100991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a0103660 msgr2=0x7ff0a010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/1699100991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a0103660 0x7ff0a010bb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff090009900 tx=0x7ff09002a480 comp rx=0 tx=0).stop 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/1699100991 shutdown_connections 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/1699100991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a010c170 0x7ff0a010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/1699100991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a0103660 0x7ff0a010bb50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/1699100991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0102d40 0x7ff0a0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/1699100991 >> 172.21.15.16:0/1699100991 conn(0x7ff0a00fc7a0 msgr2=0x7ff0a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/1699100991 shutdown_connections 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.762+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/1699100991 wait complete. 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 Processor -- start 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 -- start start 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0102d40 0x7ff0a0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0103660 0x7ff0a0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0111fb0 con 0x7ff0a010c170 2024-09-12T02:45:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0111e30 con 0x7ff0a0102d40 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a6cf1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0112130 con 0x7ff0a0103660 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a528e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a528e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54142/0 (socket says 172.21.15.16:54142) 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a528e640 1 -- 172.21.15.16:0/4130869621 learned_addr learned my addr 172.21.15.16:0/4130869621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.763+0000 7ff0a4a8d640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0102d40 0x7ff0a0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff097fff640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0103660 0x7ff0a0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 -- 172.21.15.16:0/4130869621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0103660 msgr2=0x7ff0a0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0103660 0x7ff0a0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 -- 172.21.15.16:0/4130869621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0102d40 msgr2=0x7ff0a0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0102d40 0x7ff0a0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 -- 172.21.15.16:0/4130869621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a019d520 con 0x7ff0a010c170 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a528e640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff098009fd0 tx=0x7ff09800def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098017070 con 0x7ff0a010c170 2024-09-12T02:45:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a019d810 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff098003e50 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098010460 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.764+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a01a50f0 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.765+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a0103120 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.767+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff098003970 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.767+0000 7ff095ffb640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 0x7ff0880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.767+0000 7ff0a4a8d640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 0x7ff0880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.767+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff0980ce870 con 0x7ff0a010c170 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.768+0000 7ff0a4a8d640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 0x7ff0880a5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009990 tx=0x7ff09c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.771+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff09809bc70 con 0x7ff0a010c170 2024-09-12T02:45:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.982+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0a01071e0 con 0x7ff0a010c170 2024-09-12T02:45:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.983+0000 7ff095ffb640 1 -- 172.21.15.16:0/4130869621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9695 (secure 0 0 0) 0x7ff0980bb070 con 0x7ff0a010c170 2024-09-12T02:45:35.984 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:35.985 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"btime":"2024-09-12T02:45:35:672156+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-12T02:45:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.985+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 msgr2=0x7ff0880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.985+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 0x7ff0880a5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009990 tx=0x7ff09c009110 comp rx=0 tx=0).stop 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.985+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 msgr2=0x7ff0a019ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.985+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff098009fd0 tx=0x7ff09800def0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 shutdown_connections 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0880a2b80 0x7ff0880a5040 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a010c170 0x7ff0a019ce40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0a0103660 0x7ff0a0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 --2- 172.21.15.16:0/4130869621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0102d40 0x7ff0a0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 >> 172.21.15.16:0/4130869621 conn(0x7ff0a00fc7a0 msgr2=0x7ff0a00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 shutdown_connections 2024-09-12T02:45:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:35.986+0000 7ff0a6cf1640 1 -- 172.21.15.16:0/4130869621 wait complete. 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-12T02:45:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-12T02:45:36.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:35 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:36.725 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:36 smithi033 ceph-mon[35092]: Deploying daemon mds.b on smithi033 2024-09-12T02:45:36.726 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:36 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:boot 2024-09-12T02:45:36.726 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:36 smithi033 ceph-mon[35092]: fsmap 11 up:standby 2024-09-12T02:45:36.726 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:36 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T02:45:36.726 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:36 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4130869621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:36 smithi016 ceph-mon[30925]: Deploying daemon mds.b on smithi033 2024-09-12T02:45:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:36 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:boot 2024-09-12T02:45:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:36 smithi016 ceph-mon[30925]: fsmap 11 up:standby 2024-09-12T02:45:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:36 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T02:45:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:36 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4130869621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:36 smithi184 ceph-mon[35321]: Deploying daemon mds.b on smithi033 2024-09-12T02:45:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:36 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:boot 2024-09-12T02:45:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:36 smithi184 ceph-mon[35321]: fsmap 11 up:standby 2024-09-12T02:45:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:36 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T02:45:37.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:36 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4130869621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:37.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:37 smithi033 ceph-mon[35092]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:37.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:37 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:37 smithi184 ceph-mon[35321]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:37 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:37 smithi016 ceph-mon[30925]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:37 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:45:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:boot 2024-09-12T02:45:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: fsmap 12 up:standby 2024-09-12T02:45:38.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T02:45:38.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:38 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:boot 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: fsmap 12 up:standby 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:38.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:38 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:boot 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: fsmap 12 up:standby 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:38 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 -- 172.21.15.16:0/2729891945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418101420 msgr2=0x7fb41810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2729891945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418101420 0x7fb41810d950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb410009cd0 tx=0x7fb41002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 -- 172.21.15.16:0/2729891945 shutdown_connections 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2729891945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418101420 0x7fb41810d950 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2729891945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418100a80 0x7fb418100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2729891945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418106a80 0x7fb418106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.366+0000 7fb41cc93640 1 -- 172.21.15.16:0/2729891945 >> 172.21.15.16:0/2729891945 conn(0x7fb4180fc7a0 msgr2=0x7fb4180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.367+0000 7fb41cc93640 1 -- 172.21.15.16:0/2729891945 shutdown_connections 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.367+0000 7fb41cc93640 1 -- 172.21.15.16:0/2729891945 wait complete. 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.367+0000 7fb41cc93640 1 Processor -- start 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.367+0000 7fb41cc93640 1 -- start start 2024-09-12T02:45:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 0x7fb418102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 0x7fb4181034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb418110e80 con 0x7fb418106a80 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb418110d00 con 0x7fb418101420 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41cc93640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb418111000 con 0x7fb418100a80 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 0x7fb4181034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52202/0 (socket says 172.21.15.16:52202) 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41659c640 1 -- 172.21.15.16:0/2942296705 learned_addr learned my addr 172.21.15.16:0/2942296705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb415d9b640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 0x7fb418102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 -- 172.21.15.16:0/2942296705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 msgr2=0x7fb418102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 -- 172.21.15.16:0/2942296705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 msgr2=0x7fb418102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 0x7fb418102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb416d9d640 1 -- 172.21.15.16:0/2942296705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4181a68e0 con 0x7fb418106a80 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb41659c640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:45:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.368+0000 7fb415d9b640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 0x7fb418102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb416d9d640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 0x7fb4181034b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb41002ae00 tx=0x7fb410031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410004040 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4100046c0 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4181a6b70 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410030e40 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.369+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4181a7100 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.370+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb418100ee0 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.371+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb4100305a0 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.372+0000 7fb4077fe640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 0x7fb4000a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.372+0000 7fb41659c640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 0x7fb4000a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.372+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fb41002e030 con 0x7fb418106a80 2024-09-12T02:45:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.373+0000 7fb41659c640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 0x7fb4000a50c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb408005fd0 tx=0x7fb40800b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.375+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb41004b060 con 0x7fb418106a80 2024-09-12T02:45:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.616+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb418103f60 con 0x7fb418106a80 2024-09-12T02:45:39.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.616+0000 7fb4077fe640 1 -- 172.21.15.16:0/2942296705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10521 (secure 0 0 0) 0x7fb418103f60 con 0x7fb418106a80 2024-09-12T02:45:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:39.618 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"btime":"2024-09-12T02:45:37:685211+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-12T02:45:39.618 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-09-12T02:45:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 msgr2=0x7fb4000a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 0x7fb4000a50c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb408005fd0 tx=0x7fb40800b4a0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 msgr2=0x7fb4181034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 0x7fb4181034b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb41002ae00 tx=0x7fb410031e60 comp rx=0 tx=0).stop 2024-09-12T02:45:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 shutdown_connections 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb4000a2c00 0x7fb4000a50c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb418106a80 0x7fb4181034b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb418101420 0x7fb418102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 --2- 172.21.15.16:0/2942296705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb418100a80 0x7fb418102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 >> 172.21.15.16:0/2942296705 conn(0x7fb4180fc7a0 msgr2=0x7fb418104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 shutdown_connections 2024-09-12T02:45:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:39.619+0000 7fb41cc93640 1 -- 172.21.15.16:0/2942296705 wait complete. 2024-09-12T02:45:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:39 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2942296705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:39 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2942296705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:39 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2942296705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:45:40.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-12T02:45:40.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:40.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:40.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:40 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:40.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:40.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:40 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:41.178 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:40 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 -- 172.21.15.16:0/530893923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 msgr2=0x7fecd81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 --2- 172.21.15.16:0/530893923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd81021c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009900 tx=0x7fecc802a410 comp rx=0 tx=0).stop 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 -- 172.21.15.16:0/530893923 shutdown_connections 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 --2- 172.21.15.16:0/530893923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecd8106f10 0x7fecd810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 --2- 172.21.15.16:0/530893923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 0x7fecd81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 --2- 172.21.15.16:0/530893923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd81021c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 -- 172.21.15.16:0/530893923 >> 172.21.15.16:0/530893923 conn(0x7fecd80fd8c0 msgr2=0x7fecd80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 -- 172.21.15.16:0/530893923 shutdown_connections 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.646+0000 7fecde13c640 1 -- 172.21.15.16:0/530893923 wait complete. 2024-09-12T02:45:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 Processor -- start 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 -- start start 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 0x7fecd81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecd8106f10 0x7fecd819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fecd8111f30 con 0x7fecd8101de0 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecd8111db0 con 0x7fecd8102790 2024-09-12T02:45:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecde13c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81120b0 con 0x7fecd8106f10 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54172/0 (socket says 172.21.15.16:54172) 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 -- 172.21.15.16:0/2065392977 learned_addr learned my addr 172.21.15.16:0/2065392977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd6ffd640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 0x7fecd81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 -- 172.21.15.16:0/2065392977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecd8106f10 msgr2=0x7fecd819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecd8106f10 0x7fecd819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 -- 172.21.15.16:0/2065392977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 msgr2=0x7fecd81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 0x7fecd81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.647+0000 7fecd77fe640 1 -- 172.21.15.16:0/2065392977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd819d340 con 0x7fecd8101de0 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecd77fe640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fecc8006010 tx=0x7fecc80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc803f030 con 0x7fecd8101de0 2024-09-12T02:45:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc8003ce0 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8035070 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd819d5d0 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.648+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd81a4f10 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.649+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd81032c0 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.650+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fecc8042030 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.651+0000 7fecd4ff9640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 0x7fecc00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.651+0000 7fecd6ffd640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 0x7fecc00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.651+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fecc80eec60 con 0x7fecd8101de0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.651+0000 7fecd6ffd640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 0x7fecc00a5040 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecd81088e0 tx=0x7feccc00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.655+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fecc80bc060 con 0x7fecd8101de0 2024-09-12T02:45:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.852+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fecd810a0f0 con 0x7fecd8101de0 2024-09-12T02:45:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.852+0000 7fecd4ff9640 1 -- 172.21.15.16:0/2065392977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+65 (secure 0 0 0) 0x7fecc80bb960 con 0x7fecd8101de0 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:45:41.854 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-09-12T02:45:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 msgr2=0x7fecc00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 0x7fecc00a5040 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecd81088e0 tx=0x7feccc00a2d0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 msgr2=0x7fecd8198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fecc8006010 tx=0x7fecc80042f0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 shutdown_connections 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecd8106f10 0x7fecd819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecd8102790 0x7fecd81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fecc00a2b80 0x7fecc00a5040 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 --2- 172.21.15.16:0/2065392977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecd8101de0 0x7fecd8198390 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 >> 172.21.15.16:0/2065392977 conn(0x7fecd80fd8c0 msgr2=0x7fecd80fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 shutdown_connections 2024-09-12T02:45:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:41.855+0000 7fecde13c640 1 -- 172.21.15.16:0/2065392977 wait complete. 2024-09-12T02:45:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:41 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2065392977' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:41 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2065392977' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-12T02:45:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:42.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:41 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2065392977' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-12T02:45:42.532 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-12T02:45:42.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-12T02:45:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 -- 172.21.15.16:0/2497937420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc03810e480 msgr2=0x7fc038110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 --2- 172.21.15.16:0/2497937420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc03810e480 0x7fc038110870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ca0 tx=0x7fc03002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 -- 172.21.15.16:0/2497937420 shutdown_connections 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 --2- 172.21.15.16:0/2497937420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc03810e480 0x7fc038110870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 --2- 172.21.15.16:0/2497937420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 --2- 172.21.15.16:0/2497937420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0381008d0 0x7fc038100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 -- 172.21.15.16:0/2497937420 >> 172.21.15.16:0/2497937420 conn(0x7fc0380fc7a0 msgr2=0x7fc0380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 -- 172.21.15.16:0/2497937420 shutdown_connections 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.250+0000 7fc03cc30640 1 -- 172.21.15.16:0/2497937420 wait complete. 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 Processor -- start 2024-09-12T02:45:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 -- start start 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc03810e480 0x7fc03819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc038114160 con 0x7fc0381008d0 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc038113fe0 con 0x7fc03810e480 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03cc30640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381142e0 con 0x7fc038101280 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54190/0 (socket says 172.21.15.16:54190) 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 -- 172.21.15.16:0/1038642339 learned_addr learned my addr 172.21.15.16:0/1038642339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc036d9d640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc03810e480 0x7fc03819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc035d9b640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 -- 172.21.15.16:0/1038642339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 msgr2=0x7fc03819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 -- 172.21.15.16:0/1038642339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc03810e480 msgr2=0x7fc03819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc03810e480 0x7fc03819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 -- 172.21.15.16:0/1038642339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03819f670 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc035d9b640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.251+0000 7fc03659c640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc02800d950 tx=0x7fc02800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.252+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028017070 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.252+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc028009c30 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.252+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03819f960 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.252+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02800f6b0 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.252+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0381a7240 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.253+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc038100cb0 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.254+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc0280109d0 con 0x7fc0381008d0 2024-09-12T02:45:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.255+0000 7fc0277fe640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 0x7fc0200a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.255+0000 7fc035d9b640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 0x7fc0200a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.255+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fc0280ce790 con 0x7fc0381008d0 2024-09-12T02:45:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.255+0000 7fc035d9b640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 0x7fc0200a50c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc02c005e10 tx=0x7fc02c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.259+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc028009de0 con 0x7fc0381008d0 2024-09-12T02:45:43.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:43.434+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc03802d7c0 con 0x7fc0381008d0 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:43.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:45:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:45:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.117+0000 7fc0277fe640 1 -- 172.21.15.16:0/1038642339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) ==== 138+0+0 (secure 0 0 0) 0x7fc02809bb90 con 0x7fc0381008d0 2024-09-12T02:45:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 msgr2=0x7fc0200a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 0x7fc0200a50c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc02c005e10 tx=0x7fc02c005da0 comp rx=0 tx=0).stop 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 msgr2=0x7fc03819a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc02800d950 tx=0x7fc02800de10 comp rx=0 tx=0).stop 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 shutdown_connections 2024-09-12T02:45:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc03810e480 0x7fc03819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc038101280 0x7fc03819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0200a2c00 0x7fc0200a50c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 --2- 172.21.15.16:0/1038642339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0381008d0 0x7fc03819a6c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 >> 172.21.15.16:0/1038642339 conn(0x7fc0380fc7a0 msgr2=0x7fc0380feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 shutdown_connections 2024-09-12T02:45:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:44.120+0000 7fc03cc30640 1 -- 172.21.15.16:0/1038642339 wait complete. 2024-09-12T02:45:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-12T02:45:44.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-12T02:45:44.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-12T02:45:44.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-12T02:45:45.027 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:45.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:45 smithi016 ceph-mon[30925]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-12T02:45:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:45 smithi016 ceph-mon[30925]: osdmap e74: 12 total, 12 up, 12 in 2024-09-12T02:45:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:45 smithi033 ceph-mon[35092]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-12T02:45:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:45 smithi033 ceph-mon[35092]: osdmap e74: 12 total, 12 up, 12 in 2024-09-12T02:45:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- 172.21.15.16:0/4085052342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798100e20 msgr2=0x7f9798101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 --2- 172.21.15.16:0/4085052342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798100e20 0x7f9798101200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9794009900 tx=0x7f979402a480 comp rx=0 tx=0).stop 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- 172.21.15.16:0/4085052342 shutdown_connections 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 --2- 172.21.15.16:0/4085052342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 0x7f979810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 --2- 172.21.15.16:0/4085052342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798102570 0x7f97981029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 --2- 172.21.15.16:0/4085052342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798100e20 0x7f9798101200 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- 172.21.15.16:0/4085052342 >> 172.21.15.16:0/4085052342 conn(0x7f97980fc7d0 msgr2=0x7f97980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- 172.21.15.16:0/4085052342 shutdown_connections 2024-09-12T02:45:45.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- 172.21.15.16:0/4085052342 wait complete. 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 Processor -- start 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.536+0000 7f979f8fb640 1 -- start start 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 0x7f97980fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 0x7f97980fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9798110f80 con 0x7f9798102570 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9798110e00 con 0x7f9798100e20 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979f8fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9798111100 con 0x7f9798102ef0 2024-09-12T02:45:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54688/0 (socket says 172.21.15.16:54688) 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 -- 172.21.15.16:0/3803725723 learned_addr learned my addr 172.21.15.16:0/3803725723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979d697640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 0x7f97980fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979de98640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 0x7f97980fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 -- 172.21.15.16:0/3803725723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 msgr2=0x7f97980fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 0x7f97980fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 -- 172.21.15.16:0/3803725723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 msgr2=0x7f97980fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 0x7f97980fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 -- 172.21.15.16:0/3803725723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97981a69a0 con 0x7f9798102570 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979d697640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 0x7f97980fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.537+0000 7f979ce96640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f978800e9f0 tx=0x7f978800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.538+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978800cd80 con 0x7f9798102570 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.538+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97981a6c90 con 0x7f9798102570 2024-09-12T02:45:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.538+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9788010910 con 0x7f9798102570 2024-09-12T02:45:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.538+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880188f0 con 0x7f9798102570 2024-09-12T02:45:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.538+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97981a71c0 con 0x7f9798102570 2024-09-12T02:45:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.539+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9798101200 con 0x7f9798102570 2024-09-12T02:45:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.540+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9788010ac0 con 0x7f9798102570 2024-09-12T02:45:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.541+0000 7f978e7fc640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 0x7f97840a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.541+0000 7f979d697640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 0x7f97840a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.541+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f97880cf0f0 con 0x7f9798102570 2024-09-12T02:45:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.541+0000 7f979d697640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 0x7f97840a5040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9794009230 tx=0x7f97940058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:45.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:45 smithi184 ceph-mon[35321]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:45.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1038642339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-12T02:45:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:45 smithi184 ceph-mon[35321]: osdmap e74: 12 total, 12 up, 12 in 2024-09-12T02:45:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.546+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f978809bd30 con 0x7f9798102570 2024-09-12T02:45:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:45.720+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f97981a74b0 con 0x7f9798102570 2024-09-12T02:45:46.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.158+0000 7f978e7fc640 1 -- 172.21.15.16:0/3803725723 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f97880a0030 con 0x7f9798102570 2024-09-12T02:45:46.159 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-09-12T02:45:46.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 msgr2=0x7f97840a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:46.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 0x7f97840a5040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9794009230 tx=0x7f97940058d0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 msgr2=0x7f97980fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:46.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f978800e9f0 tx=0x7f978800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 shutdown_connections 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9798102ef0 0x7f97980fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f97840a2b80 0x7f97840a5040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9798102570 0x7f97980fd1c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 --2- 172.21.15.16:0/3803725723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9798100e20 0x7f97980fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 >> 172.21.15.16:0/3803725723 conn(0x7f97980fc7d0 msgr2=0x7f9798069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 shutdown_connections 2024-09-12T02:45:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:46.161+0000 7f979f8fb640 1 -- 172.21.15.16:0/3803725723 wait complete. 2024-09-12T02:45:46.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:46 smithi016 ceph-mon[30925]: osdmap e75: 12 total, 12 up, 12 in 2024-09-12T02:45:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:46 smithi033 ceph-mon[35092]: osdmap e75: 12 total, 12 up, 12 in 2024-09-12T02:45:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:46 smithi184 ceph-mon[35321]: osdmap e75: 12 total, 12 up, 12 in 2024-09-12T02:45:46.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-12T02:45:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: pgmap v268: 33 pgs: 1 creating+activating, 14 creating+peering, 6 active+clean, 12 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: osdmap e76: 12 total, 12 up, 12 in 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-12T02:45:47.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:47 smithi016 ceph-mon[30925]: osdmap e77: 12 total, 12 up, 12 in 2024-09-12T02:45:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- 172.21.15.16:0/291412376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0348102d30 msgr2=0x7f0348103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 --2- 172.21.15.16:0/291412376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0348102d30 0x7f0348103110 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0338009cd0 tx=0x7f033802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- 172.21.15.16:0/291412376 shutdown_connections 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 --2- 172.21.15.16:0/291412376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03481082a0 0x7f034810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 --2- 172.21.15.16:0/291412376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03481036e0 0x7f0348107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 --2- 172.21.15.16:0/291412376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0348102d30 0x7f0348103110 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- 172.21.15.16:0/291412376 >> 172.21.15.16:0/291412376 conn(0x7f03480fc7d0 msgr2=0x7f03480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- 172.21.15.16:0/291412376 shutdown_connections 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- 172.21.15.16:0/291412376 wait complete. 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 Processor -- start 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.466+0000 7f034f83a640 1 -- start start 2024-09-12T02:45:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348102d30 0x7f0348100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03481082a0 0x7f0348101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03481120a0 con 0x7f03481036e0 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0348111f20 con 0x7f0348102d30 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0348112220 con 0x7f03481082a0 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54708/0 (socket says 172.21.15.16:54708) 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 -- 172.21.15.16:0/3753564179 learned_addr learned my addr 172.21.15.16:0/3753564179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 -- 172.21.15.16:0/3753564179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03481082a0 msgr2=0x7f0348101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03481082a0 0x7f0348101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 -- 172.21.15.16:0/3753564179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348102d30 msgr2=0x7f0348100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348102d30 0x7f0348100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 -- 172.21.15.16:0/3753564179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03481a6a30 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034cdd5640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d750 tx=0x7f033c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c017070 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c005500 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03481a6d20 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c004e00 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.467+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03481a71b0 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.469+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034806a850 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.471+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f033c004fa0 con 0x7f03481036e0 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.471+0000 7f03467fc640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 0x7f03300a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.471+0000 7f034d5d6640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 0x7f03300a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:47.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.471+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7879+0+0 (secure 0 0 0) 0x7f033c0cecc0 con 0x7f03481036e0 2024-09-12T02:45:47.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.472+0000 7f034d5d6640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 0x7f03300a5040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f033802ae00 tx=0x7f03380093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:47.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.475+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f033c09bdd0 con 0x7f03481036e0 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: pgmap v268: 33 pgs: 1 creating+activating, 14 creating+peering, 6 active+clean, 12 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: osdmap e76: 12 total, 12 up, 12 in 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-12T02:45:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:47 smithi033 ceph-mon[35092]: osdmap e77: 12 total, 12 up, 12 in 2024-09-12T02:45:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-12T02:45:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: pgmap v268: 33 pgs: 1 creating+activating, 14 creating+peering, 6 active+clean, 12 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3803725723' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-12T02:45:47.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: osdmap e76: 12 total, 12 up, 12 in 2024-09-12T02:45:47.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-12T02:45:47.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:47 smithi184 ceph-mon[35321]: osdmap e77: 12 total, 12 up, 12 in 2024-09-12T02:45:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:47.689+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0348103110 con 0x7f03481036e0 2024-09-12T02:45:48.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.179+0000 7f03467fc640 1 -- 172.21.15.16:0/3753564179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f033c09b6d0 con 0x7f03481036e0 2024-09-12T02:45:48.180 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 msgr2=0x7f03300a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 0x7f03300a5040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f033802ae00 tx=0x7f03380093f0 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 msgr2=0x7f0348100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d750 tx=0x7f033c00dc10 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 shutdown_connections 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03481082a0 0x7f0348101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03300a2b80 0x7f03300a5040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03481036e0 0x7f0348100ed0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 --2- 172.21.15.16:0/3753564179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0348102d30 0x7f0348100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.183+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 >> 172.21.15.16:0/3753564179 conn(0x7f03480fc7d0 msgr2=0x7f0348105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.184+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 shutdown_connections 2024-09-12T02:45:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:48.184+0000 7f034f83a640 1 -- 172.21.15.16:0/3753564179 wait complete. 2024-09-12T02:45:48.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:48 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-12T02:45:48.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:48 smithi016 ceph-mon[30925]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-12T02:45:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:48 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-12T02:45:48.539 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:48 smithi033 ceph-mon[35092]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-12T02:45:48.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:48 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-12T02:45:48.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:48 smithi184 ceph-mon[35321]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-12T02:45:48.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:45:49.103 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:49 smithi033 ceph-mon[35092]: pgmap v271: 97 pgs: 1 creating+activating, 36 creating+peering, 9 active+clean, 51 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:49 smithi033 ceph-mon[35092]: osdmap e78: 12 total, 12 up, 12 in 2024-09-12T02:45:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:49 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-12T02:45:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:49 smithi033 ceph-mon[35092]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-12T02:45:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:49 smithi033 ceph-mon[35092]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-12T02:45:49.514 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:49 smithi016 ceph-mon[30925]: pgmap v271: 97 pgs: 1 creating+activating, 36 creating+peering, 9 active+clean, 51 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:49.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:49 smithi016 ceph-mon[30925]: osdmap e78: 12 total, 12 up, 12 in 2024-09-12T02:45:49.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:49 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-12T02:45:49.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:49 smithi016 ceph-mon[30925]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-12T02:45:49.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:49 smithi016 ceph-mon[30925]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-12T02:45:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:49 smithi184 ceph-mon[35321]: pgmap v271: 97 pgs: 1 creating+activating, 36 creating+peering, 9 active+clean, 51 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T02:45:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:49 smithi184 ceph-mon[35321]: osdmap e78: 12 total, 12 up, 12 in 2024-09-12T02:45:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:49 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3753564179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-12T02:45:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:49 smithi184 ceph-mon[35321]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-12T02:45:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:49 smithi184 ceph-mon[35321]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-12T02:45:49.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 -- 172.21.15.16:0/761212261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c10e480 msgr2=0x7fe17c110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:49.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 --2- 172.21.15.16:0/761212261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c10e480 0x7fe17c110870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe174009ca0 tx=0x7fe17402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 -- 172.21.15.16:0/761212261 shutdown_connections 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 --2- 172.21.15.16:0/761212261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c10e480 0x7fe17c110870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 --2- 172.21.15.16:0/761212261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c101280 0x7fe17c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 --2- 172.21.15.16:0/761212261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c1008d0 0x7fe17c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.596+0000 7fe18146e640 1 -- 172.21.15.16:0/761212261 >> 172.21.15.16:0/761212261 conn(0x7fe17c0fc7a0 msgr2=0x7fe17c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- 172.21.15.16:0/761212261 shutdown_connections 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- 172.21.15.16:0/761212261 wait complete. 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 Processor -- start 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- start start 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 0x7fe17c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:49.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c10e480 0x7fe17c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c1141d0 con 0x7fe17c101280 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c114050 con 0x7fe17c1008d0 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe18146e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c114350 con 0x7fe17c10e480 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.597+0000 7fe173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54726/0 (socket says 172.21.15.16:54726) 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 -- 172.21.15.16:0/4195513289 learned_addr learned my addr 172.21.15.16:0/4195513289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe17affd640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 0x7fe17c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe17b7fe640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c10e480 0x7fe17c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 -- 172.21.15.16:0/4195513289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c10e480 msgr2=0x7fe17c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:49.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c10e480 0x7fe17c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 -- 172.21.15.16:0/4195513289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 msgr2=0x7fe17c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 0x7fe17c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 -- 172.21.15.16:0/4195513289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c19f660 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe17affd640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 0x7fe17c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe173fff640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c890 tx=0x7fe16400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164004220 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c19f950 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1640043c0 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164004a00 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.598+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c1a7230 con 0x7fe17c101280 2024-09-12T02:45:49.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.599+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c100cb0 con 0x7fe17c101280 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.600+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe1640079b0 con 0x7fe17c101280 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.601+0000 7fe178ff9640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 0x7fe1680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.601+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fe1640ceca0 con 0x7fe17c101280 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.601+0000 7fe17affd640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 0x7fe1680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.602+0000 7fe17affd640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 0x7fe1680a5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009930 tx=0x7fe16c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:49.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.605+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe164014030 con 0x7fe17c101280 2024-09-12T02:45:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.780+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe17c109080 con 0x7fe17c101280 2024-09-12T02:45:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.782+0000 7fe178ff9640 1 -- 172.21.15.16:0/4195513289 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22239 (secure 0 0 0) 0x7fe16409bd50 con 0x7fe17c101280 2024-09-12T02:45:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:49.783 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:45:47.690378+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:45:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 msgr2=0x7fe1680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 0x7fe1680a5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009930 tx=0x7fe16c008040 comp rx=0 tx=0).stop 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 msgr2=0x7fe17c19abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c890 tx=0x7fe16400cd50 comp rx=0 tx=0).stop 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 shutdown_connections 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe17c10e480 0x7fe17c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1680a2b80 0x7fe1680a5040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe17c101280 0x7fe17c19abf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 --2- 172.21.15.16:0/4195513289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe17c1008d0 0x7fe17c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:49.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.784+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 >> 172.21.15.16:0/4195513289 conn(0x7fe17c0fc7a0 msgr2=0x7fe17c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:49.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.785+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 shutdown_connections 2024-09-12T02:45:49.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:49.785+0000 7fe18146e640 1 -- 172.21.15.16:0/4195513289 wait complete. 2024-09-12T02:45:50.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:50 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:active 2024-09-12T02:45:50.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:50 smithi016 ceph-mon[30925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:50 smithi016 ceph-mon[30925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:50 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4195513289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:50.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:45:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:50 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:active 2024-09-12T02:45:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:50 smithi033 ceph-mon[35092]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:50 smithi033 ceph-mon[35092]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:50 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4195513289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:50 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] up:active 2024-09-12T02:45:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:50 smithi184 ceph-mon[35321]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:50 smithi184 ceph-mon[35321]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:50 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4195513289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:50.667 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 -- 172.21.15.16:0/3215867623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c60100e30 msgr2=0x7f6c60104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 --2- 172.21.15.16:0/3215867623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c60100e30 0x7f6c60104e80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009900 tx=0x7f6c5002a410 comp rx=0 tx=0).stop 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 -- 172.21.15.16:0/3215867623 shutdown_connections 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 --2- 172.21.15.16:0/3215867623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c601055b0 0x7f6c6010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 --2- 172.21.15.16:0/3215867623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c60100e30 0x7f6c60104e80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 --2- 172.21.15.16:0/3215867623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 0x7f6c60100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.120+0000 7f6c67eef640 1 -- 172.21.15.16:0/3215867623 >> 172.21.15.16:0/3215867623 conn(0x7f6c60077210 msgr2=0x7f6c600fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- 172.21.15.16:0/3215867623 shutdown_connections 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- 172.21.15.16:0/3215867623 wait complete. 2024-09-12T02:45:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 Processor -- start 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- start start 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 0x7f6c60198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 0x7f6c60198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c60111fb0 con 0x7f6c601055b0 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c60111e30 con 0x7f6c60100e30 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.121+0000 7f6c67eef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c60112130 con 0x7f6c60100480 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54730/0 (socket says 172.21.15.16:54730) 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 -- 172.21.15.16:0/2366166437 learned_addr learned my addr 172.21.15.16:0/2366166437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6548a640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 0x7f6c60198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 -- 172.21.15.16:0/2366166437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 msgr2=0x7f6c60198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c65c8b640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 0x7f6c60198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 0x7f6c60198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 -- 172.21.15.16:0/2366166437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 msgr2=0x7f6c60198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 0x7f6c60198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 -- 172.21.15.16:0/2366166437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6019d520 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6548a640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 0x7f6c60198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c6648c640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800cad0 tx=0x7f6c5800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800ae20 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6019d810 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c58005020 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c601a50f0 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.122+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5801d060 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.123+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c60100860 con 0x7f6c601055b0 2024-09-12T02:45:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.124+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c580051d0 con 0x7f6c601055b0 2024-09-12T02:45:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.125+0000 7f6c56ffd640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 0x7f6c4c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.125+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f6c580cf3f0 con 0x7f6c601055b0 2024-09-12T02:45:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.125+0000 7f6c65c8b640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 0x7f6c4c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.126+0000 7f6c65c8b640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 0x7f6c4c0a5040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00a810 tx=0x7f6c5c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.128+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6c58014030 con 0x7f6c601055b0 2024-09-12T02:45:51.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.309+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c601998d0 con 0x7f6c601055b0 2024-09-12T02:45:51.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:51 smithi016 ceph-mon[30925]: pgmap v273: 97 pgs: 1 creating+activating, 25 creating+peering, 59 active+clean, 12 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2024-09-12T02:45:51.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:51 smithi016 ceph-mon[30925]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-12T02:45:51.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:51 smithi016 ceph-mon[30925]: Cluster is now healthy 2024-09-12T02:45:51.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.311+0000 7f6c56ffd640 1 -- 172.21.15.16:0/2366166437 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22239 (secure 0 0 0) 0x7f6c601998d0 con 0x7f6c601055b0 2024-09-12T02:45:51.312 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:45:51.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:45:47.690378+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 msgr2=0x7f6c4c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 0x7f6c4c0a5040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00a810 tx=0x7f6c5c008040 comp rx=0 tx=0).stop 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 msgr2=0x7f6c6019ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800cad0 tx=0x7f6c5800cf90 comp rx=0 tx=0).stop 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 shutdown_connections 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c4c0a2b80 0x7f6c4c0a5040 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c601055b0 0x7f6c6019ce40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c60100e30 0x7f6c60198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 --2- 172.21.15.16:0/2366166437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c60100480 0x7f6c60198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:51.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 >> 172.21.15.16:0/2366166437 conn(0x7f6c60077210 msgr2=0x7f6c600fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:51.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 shutdown_connections 2024-09-12T02:45:51.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:51.314+0000 7f6c67eef640 1 -- 172.21.15.16:0/2366166437 wait complete. 2024-09-12T02:45:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:51 smithi033 ceph-mon[35092]: pgmap v273: 97 pgs: 1 creating+activating, 25 creating+peering, 59 active+clean, 12 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2024-09-12T02:45:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:51 smithi033 ceph-mon[35092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-12T02:45:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:51 smithi033 ceph-mon[35092]: Cluster is now healthy 2024-09-12T02:45:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:51 smithi184 ceph-mon[35321]: pgmap v273: 97 pgs: 1 creating+activating, 25 creating+peering, 59 active+clean, 12 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2024-09-12T02:45:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:51 smithi184 ceph-mon[35321]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-12T02:45:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:51 smithi184 ceph-mon[35321]: Cluster is now healthy 2024-09-12T02:45:51.917 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-12T02:45:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:52.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2366166437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:52.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2366166437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2366166437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 -- 172.21.15.16:0/3716970786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18106f10 msgr2=0x7f6c1810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/3716970786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18106f10 0x7f6c1810ea90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009ca0 tx=0x7f6c1002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 -- 172.21.15.16:0/3716970786 shutdown_connections 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/3716970786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18106f10 0x7f6c1810ea90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/3716970786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 0x7f6c181067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/3716970786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18101de0 0x7f6c181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 -- 172.21.15.16:0/3716970786 >> 172.21.15.16:0/3716970786 conn(0x7f6c180fd8c0 msgr2=0x7f6c180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.609+0000 7f6c1cc18640 1 -- 172.21.15.16:0/3716970786 shutdown_connections 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.610+0000 7f6c1cc18640 1 -- 172.21.15.16:0/3716970786 wait complete. 2024-09-12T02:45:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.610+0000 7f6c1cc18640 1 Processor -- start 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.610+0000 7f6c1cc18640 1 -- start start 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.610+0000 7f6c1cc18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.610+0000 7f6c1cc18640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 0x7f6c18198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1cc18640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18106f10 0x7f6c1819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1cc18640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c18111f70 con 0x7f6c18101de0 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1cc18640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c18111df0 con 0x7f6c18102790 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1cc18640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c181120f0 con 0x7f6c18106f10 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54760/0 (socket says 172.21.15.16:54760) 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 -- 172.21.15.16:0/596553855 learned_addr learned my addr 172.21.15.16:0/596553855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c16d9d640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18106f10 0x7f6c1819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c0ffff640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 0x7f6c18198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 -- 172.21.15.16:0/596553855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18106f10 msgr2=0x7f6c1819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18106f10 0x7f6c1819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 -- 172.21.15.16:0/596553855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 msgr2=0x7f6c18198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 0x7f6c18198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 -- 172.21.15.16:0/596553855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1819d380 con 0x7f6c18101de0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1659c640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800d770 tx=0x7f6c0800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08017070 con 0x7f6c18101de0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c08005500 con 0x7f6c18101de0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1819d670 con 0x7f6c18101de0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.611+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08004e00 con 0x7f6c18101de0 2024-09-12T02:45:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.612+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c181a4f50 con 0x7f6c18101de0 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.612+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c181021c0 con 0x7f6c18101de0 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.614+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c08004fa0 con 0x7f6c18101de0 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.614+0000 7f6c0f7fe640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 0x7f6c040a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.614+0000 7f6c0ffff640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 0x7f6c040a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.614+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f6c080cecd0 con 0x7f6c18101de0 2024-09-12T02:45:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.615+0000 7f6c0ffff640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 0x7f6c040a50c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6c00005fd0 tx=0x7f6c00005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.618+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6c0809be00 con 0x7f6c18101de0 2024-09-12T02:45:52.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:52.830+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6c1802d7c0 con 0x7f6c18101de0 2024-09-12T02:45:52.997 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 02:45:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: 2024-09-12T02:45:52.698+0000 7fb29591f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-12T02:45:53.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.203+0000 7f6c0f7fe640 1 -- 172.21.15.16:0/596553855 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f6c0809b700 con 0x7f6c18101de0 2024-09-12T02:45:53.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 msgr2=0x7f6c040a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 0x7f6c040a50c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6c00005fd0 tx=0x7f6c00005850 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 msgr2=0x7f6c181983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800d770 tx=0x7f6c0800dc30 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 shutdown_connections 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c18106f10 0x7f6c1819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c18102790 0x7f6c18198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c040a2c00 0x7f6c040a50c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 --2- 172.21.15.16:0/596553855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18101de0 0x7f6c181983d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 >> 172.21.15.16:0/596553855 conn(0x7f6c180fd8c0 msgr2=0x7f6c180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.206+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 shutdown_connections 2024-09-12T02:45:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:53.207+0000 7f6c1cc18640 1 -- 172.21.15.16:0/596553855 wait complete. 2024-09-12T02:45:53.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:53 smithi016 ceph-mon[30925]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-12T02:45:53.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-12T02:45:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:53 smithi033 ceph-mon[35092]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-12T02:45:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-12T02:45:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:53 smithi184 ceph-mon[35321]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-12T02:45:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-12T02:45:53.813 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-12T02:45:53.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-09-12T02:45:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:54.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-12T02:45:54.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:54 smithi016 ceph-mon[30925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:54 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-12T02:45:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:54 smithi033 ceph-mon[35092]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:54 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/596553855' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-12T02:45:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:54 smithi184 ceph-mon[35321]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 -- 172.21.15.16:0/1355283479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 msgr2=0x7efc0010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 --2- 172.21.15.16:0/1355283479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc0010d950 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009cd0 tx=0x7efbf802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 -- 172.21.15.16:0/1355283479 shutdown_connections 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 --2- 172.21.15.16:0/1355283479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc0010d950 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 --2- 172.21.15.16:0/1355283479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc00100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 --2- 172.21.15.16:0/1355283479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc00106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 -- 172.21.15.16:0/1355283479 >> 172.21.15.16:0/1355283479 conn(0x7efc000fc7a0 msgr2=0x7efc000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 -- 172.21.15.16:0/1355283479 shutdown_connections 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 -- 172.21.15.16:0/1355283479 wait complete. 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.556+0000 7efc048e1640 1 Processor -- start 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 -- start start 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc000fce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc000fd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efc00110e80 con 0x7efc00101420 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efc00110d00 con 0x7efc00100a80 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efc048e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc00111000 con 0x7efc00106a80 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54770/0 (socket says 172.21.15.16:54770) 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 -- 172.21.15.16:0/2752577147 learned_addr learned my addr 172.21.15.16:0/2752577147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbfe59c640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc000fce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbfed9d640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc000fd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 -- 172.21.15.16:0/2752577147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 msgr2=0x7efc000fd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc000fd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 -- 172.21.15.16:0/2752577147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 msgr2=0x7efc000fce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc000fce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 -- 172.21.15.16:0/2752577147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc001a68f0 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbfe59c640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc000fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbfed9d640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc000fd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.557+0000 7efbf5d9b640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efbe800d950 tx=0x7efbe800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.558+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe8017070 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.558+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbe8009c30 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.558+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc001a6be0 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.558+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe800f6b0 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.558+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc001a7170 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.559+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc00100ee0 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.560+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efbe80109d0 con 0x7efc00101420 2024-09-12T02:45:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.561+0000 7efbf7fff640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 0x7efbec0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.561+0000 7efbfe59c640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 0x7efbec0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.561+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7efbe80ceb30 con 0x7efc00101420 2024-09-12T02:45:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.561+0000 7efbfe59c640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 0x7efbec0a50c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005fd0 tx=0x7efbf000b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.565+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efbe8009de0 con 0x7efc00101420 2024-09-12T02:45:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:54.780+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7efc00109a70 con 0x7efc00101420 2024-09-12T02:45:55.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.213+0000 7efbf7fff640 1 -- 172.21.15.16:0/2752577147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7efbe809bbe0 con 0x7efc00101420 2024-09-12T02:45:55.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 msgr2=0x7efbec0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 0x7efbec0a50c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005fd0 tx=0x7efbf000b5d0 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 msgr2=0x7efc000fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efbe800d950 tx=0x7efbe800de10 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 shutdown_connections 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc00106a80 0x7efc000fd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efbec0a2c00 0x7efbec0a50c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc00101420 0x7efc000fd340 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 --2- 172.21.15.16:0/2752577147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efc00100a80 0x7efc000fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 >> 172.21.15.16:0/2752577147 conn(0x7efc000fc7a0 msgr2=0x7efc0006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 shutdown_connections 2024-09-12T02:45:55.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:55.216+0000 7efc048e1640 1 -- 172.21.15.16:0/2752577147 wait complete. 2024-09-12T02:45:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:55 smithi033 ceph-mon[35092]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-12T02:45:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:55 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-12T02:45:55.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:55 smithi016 ceph-mon[30925]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-12T02:45:55.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:55 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-12T02:45:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:55 smithi184 ceph-mon[35321]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-09-12T02:45:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:55 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-12T02:45:55.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-12T02:45:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-12T02:45:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:56 smithi033 ceph-mon[35092]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-12T02:45:56.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:56 smithi016 ceph-mon[30925]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2752577147' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-12T02:45:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:56 smithi184 ceph-mon[35321]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.668+0000 7f56b9040640 1 -- 172.21.15.16:0/3108369082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 msgr2=0x7f56b410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.668+0000 7f56b9040640 1 --2- 172.21.15.16:0/3108369082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b410bb50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f569c009900 tx=0x7f569c02a480 comp rx=0 tx=0).stop 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 -- 172.21.15.16:0/3108369082 shutdown_connections 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 --2- 172.21.15.16:0/3108369082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b410c170 0x7f56b410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 --2- 172.21.15.16:0/3108369082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b410bb50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 --2- 172.21.15.16:0/3108369082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b4102d40 0x7f56b4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 -- 172.21.15.16:0/3108369082 >> 172.21.15.16:0/3108369082 conn(0x7f56b40fc7a0 msgr2=0x7f56b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 -- 172.21.15.16:0/3108369082 shutdown_connections 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 -- 172.21.15.16:0/3108369082 wait complete. 2024-09-12T02:45:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 Processor -- start 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.669+0000 7f56b9040640 1 -- start start 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b4102d40 0x7f56b4198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b410c170 0x7f56b419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4111fb0 con 0x7f56b4103660 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4111e30 con 0x7f56b4102d40 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b9040640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4112130 con 0x7f56b410c170 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36920/0 (socket says 172.21.15.16:36920) 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 -- 172.21.15.16:0/798026032 learned_addr learned my addr 172.21.15.16:0/798026032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b2d9d640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b4102d40 0x7f56b4198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b359e640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b410c170 0x7f56b419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 -- 172.21.15.16:0/798026032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b410c170 msgr2=0x7f56b419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b410c170 0x7f56b419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 -- 172.21.15.16:0/798026032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b4102d40 msgr2=0x7f56b4198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b4102d40 0x7f56b4198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 -- 172.21.15.16:0/798026032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b419d440 con 0x7f56b4103660 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.670+0000 7f56b259c640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f569c005ab0 tx=0x7f569c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.671+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c03f030 con 0x7f56b4103660 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.671+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b419d6d0 con 0x7f56b4103660 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.671+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569c003ce0 con 0x7f56b4103660 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.671+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c035070 con 0x7f56b4103660 2024-09-12T02:45:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.671+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b41a5010 con 0x7f56b4103660 2024-09-12T02:45:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.672+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b4103120 con 0x7f56b4103660 2024-09-12T02:45:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.673+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f569c042030 con 0x7f56b4103660 2024-09-12T02:45:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.674+0000 7f56abfff640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 0x7f56a00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.675+0000 7f56b2d9d640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 0x7f56a00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.675+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f569c0ef7a0 con 0x7f56b4103660 2024-09-12T02:45:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.675+0000 7f56b2d9d640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 0x7f56a00a5040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56a40058c0 tx=0x7f56a400a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.679+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f569c0bc850 con 0x7f56b4103660 2024-09-12T02:45:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:56.891+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f56b4107210 con 0x7f56b4103660 2024-09-12T02:45:57.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.232+0000 7f56abfff640 1 -- 172.21.15.16:0/798026032 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f569c0bc150 con 0x7f56b4103660 2024-09-12T02:45:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 msgr2=0x7f56a00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 0x7f56a00a5040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56a40058c0 tx=0x7f56a400a4c0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 msgr2=0x7f56b41989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f569c005ab0 tx=0x7f569c0042f0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 shutdown_connections 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56b410c170 0x7f56b419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56a00a2b80 0x7f56a00a5040 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56b4103660 0x7f56b41989d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 --2- 172.21.15.16:0/798026032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56b4102d40 0x7f56b4198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 >> 172.21.15.16:0/798026032 conn(0x7f56b40fc7a0 msgr2=0x7f56b40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 shutdown_connections 2024-09-12T02:45:57.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:57.234+0000 7f56b9040640 1 -- 172.21.15.16:0/798026032 wait complete. 2024-09-12T02:45:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-12T02:45:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:active 2024-09-12T02:45:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-12T02:45:57.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-12T02:45:57.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-12T02:45:57.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-12T02:45:57.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-12T02:45:57.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-12T02:45:57.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:active 2024-09-12T02:45:57.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-12T02:45:57.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-12T02:45:57.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-12T02:45:57.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-12T02:45:57.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:57 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-12T02:45:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-12T02:45:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] up:active 2024-09-12T02:45:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-12T02:45:57.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-12T02:45:57.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-12T02:45:57.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-12T02:45:57.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:57 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-12T02:45:57.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-12T02:45:57.997 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: 2024-09-12T02:45:57.698+0000 7fb29591f640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-12T02:45:57.997 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 02:45:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: 2024-09-12T02:45:57.698+0000 7fb29591f640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-12T02:45:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:active 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-12T02:45:58.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:active 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: Cluster is now healthy 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:58 smithi016 ceph-mon[30925]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-12T02:45:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:active 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:active 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: Cluster is now healthy 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-09-12T02:45:58.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:58 smithi033 ceph-mon[35092]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/798026032' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] up:active 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-12T02:45:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] up:active 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: Cluster is now healthy 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-09-12T02:45:58.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:58 smithi184 ceph-mon[35321]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-09-12T02:45:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- 172.21.15.16:0/1996944689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60102570 msgr2=0x7fab601029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 --2- 172.21.15.16:0/1996944689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60102570 0x7fab601029b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fab54009900 tx=0x7fab5402a480 comp rx=0 tx=0).stop 2024-09-12T02:45:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- 172.21.15.16:0/1996944689 shutdown_connections 2024-09-12T02:45:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 --2- 172.21.15.16:0/1996944689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102ef0 0x7fab6010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 --2- 172.21.15.16:0/1996944689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60102570 0x7fab601029b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 --2- 172.21.15.16:0/1996944689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60100e20 0x7fab60101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- 172.21.15.16:0/1996944689 >> 172.21.15.16:0/1996944689 conn(0x7fab600fc7d0 msgr2=0x7fab600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- 172.21.15.16:0/1996944689 shutdown_connections 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- 172.21.15.16:0/1996944689 wait complete. 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 Processor -- start 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.582+0000 7fab6552b640 1 -- start start 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102570 0x7fab601998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60102ef0 0x7fab601a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab60110f80 con 0x7fab60100e20 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab60110e00 con 0x7fab60102570 2024-09-12T02:45:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab6552b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab60111100 con 0x7fab60102ef0 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36940/0 (socket says 172.21.15.16:36940) 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 -- 172.21.15.16:0/3789647442 learned_addr learned my addr 172.21.15.16:0/3789647442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5e7fc640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102570 0x7fab601998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5f7fe640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60102ef0 0x7fab601a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 -- 172.21.15.16:0/3789647442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60102ef0 msgr2=0x7fab601a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60102ef0 0x7fab601a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 -- 172.21.15.16:0/3789647442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102570 msgr2=0x7fab601998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102570 0x7fab601998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 -- 172.21.15.16:0/3789647442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab601a4950 con 0x7fab60100e20 2024-09-12T02:45:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab5effd640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c890 tx=0x7fab5000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.583+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50004220 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.584+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab601a4b20 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.584+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab500043c0 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.584+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab500049e0 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.584+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab601a4fe0 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.585+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab60101200 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.586+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fab500079b0 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.586+0000 7fab47fff640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 0x7fab480a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.586+0000 7fab5e7fc640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 0x7fab480a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.587+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fab500ceea0 con 0x7fab60100e20 2024-09-12T02:45:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.587+0000 7fab5e7fc640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 0x7fab480a5040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab540058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:45:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.591+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab50014030 con 0x7fab60100e20 2024-09-12T02:45:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.803+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fab60109090 con 0x7fab60100e20 2024-09-12T02:45:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.922+0000 7fab47fff640 1 -- 172.21.15.16:0/3789647442 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7fab5009b900 con 0x7fab60100e20 2024-09-12T02:45:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 msgr2=0x7fab480a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 0x7fab480a5040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab540058d0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 msgr2=0x7fab60199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c890 tx=0x7fab5000cd50 comp rx=0 tx=0).stop 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 shutdown_connections 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab60102ef0 0x7fab601a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab60102570 0x7fab601998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab480a2b80 0x7fab480a5040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 --2- 172.21.15.16:0/3789647442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab60100e20 0x7fab60199390 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:45:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 >> 172.21.15.16:0/3789647442 conn(0x7fab600fc7d0 msgr2=0x7fab600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:45:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.925+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 shutdown_connections 2024-09-12T02:45:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:45:58.926+0000 7fab6552b640 1 -- 172.21.15.16:0/3789647442 wait complete. 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:active 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-12T02:45:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:45:59 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:active 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-12T02:45:59.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:45:59 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:45:59.529 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-09-12T02:45:59.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3789647442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] up:active 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-09-12T02:45:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:45:59 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:45:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 -- 172.21.15.16:0/1710616520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90106a80 msgr2=0x7fec90106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 --2- 172.21.15.16:0/1710616520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90106a80 0x7fec90106e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fec80009930 tx=0x7fec8002a420 comp rx=0 tx=0).stop 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 -- 172.21.15.16:0/1710616520 shutdown_connections 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 --2- 172.21.15.16:0/1710616520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec9010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 --2- 172.21.15.16:0/1710616520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec90100a80 0x7fec90100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 --2- 172.21.15.16:0/1710616520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90106a80 0x7fec90106e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 -- 172.21.15.16:0/1710616520 >> 172.21.15.16:0/1710616520 conn(0x7fec900fc7a0 msgr2=0x7fec900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 -- 172.21.15.16:0/1710616520 shutdown_connections 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.212+0000 7fec96b47640 1 -- 172.21.15.16:0/1710616520 wait complete. 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 Processor -- start 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 -- start start 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec901975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec90106a80 0x7fec90197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec90110e80 con 0x7fec90100a80 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec90110d00 con 0x7fec90101420 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec96b47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec90111000 con 0x7fec90106a80 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36948/0 (socket says 172.21.15.16:36948) 2024-09-12T02:46:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 -- 172.21.15.16:0/1216515658 learned_addr learned my addr 172.21.15.16:0/1216515658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec8ffff640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec901975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 -- 172.21.15.16:0/1216515658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec90106a80 msgr2=0x7fec90197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec90106a80 0x7fec90197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 -- 172.21.15.16:0/1216515658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 msgr2=0x7fec901975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec901975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec948e3640 1 -- 172.21.15.16:0/1216515658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec901a68d0 con 0x7fec90100a80 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.213+0000 7fec8ffff640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec901975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec948e3640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec80009230 tx=0x7fec80004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8003f030 con 0x7fec90100a80 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec80003ad0 con 0x7fec90100a80 2024-09-12T02:46:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec901a6b60 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80035070 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.214+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec901a7040 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.215+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90100ee0 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.216+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fec80042030 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.217+0000 7fec8dffb640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 0x7fec7c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.217+0000 7fec8ffff640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 0x7fec7c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.217+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fec800ef8f0 con 0x7fec90100a80 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.218+0000 7fec8ffff640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 0x7fec7c0a5040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec780058c0 tx=0x7fec78005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.220+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fec800bc900 con 0x7fec90100a80 2024-09-12T02:46:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.427+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fec90062630 con 0x7fec7c0a2b80 2024-09-12T02:46:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.455+0000 7fec8dffb640 1 -- 172.21.15.16:0/1216515658 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fec90062630 con 0x7fec7c0a2b80 2024-09-12T02:46:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 msgr2=0x7fec7c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 0x7fec7c0a5040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec780058c0 tx=0x7fec78005850 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 msgr2=0x7fec90197060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec80009230 tx=0x7fec80004270 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 shutdown_connections 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec90106a80 0x7fec90197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90101420 0x7fec901975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec7c0a2b80 0x7fec7c0a5040 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 --2- 172.21.15.16:0/1216515658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec90100a80 0x7fec90197060 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 >> 172.21.15.16:0/1216515658 conn(0x7fec900fc7a0 msgr2=0x7fec900fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 shutdown_connections 2024-09-12T02:46:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:00.457+0000 7fec96b47640 1 -- 172.21.15.16:0/1216515658 wait complete. 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-12T02:46:00.467 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-09-12T02:46:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:46:00.431+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:46:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:46:00.431+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:46:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:46:00.431+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:46:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:46:00.431+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:46:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:46:00 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:46:00.431+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-12T02:46:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-09-12T02:46:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-09-12T02:46:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-09-12T02:46:00.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-09-12T02:46:00.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-12T02:46:00.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-09-12T02:46:00.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-12T02:46:00.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-09-12T02:46:01.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-12T02:46:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-09-12T02:46:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-12T02:46:01.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-09-12T02:46:01.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-12T02:46:01.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-12T02:46:01.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: osdmap e79: 12 total, 12 up, 12 in 2024-09-12T02:46:01.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:01 smithi016 ceph-mon[30925]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-12T02:46:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: osdmap e79: 12 total, 12 up, 12 in 2024-09-12T02:46:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:01 smithi184 ceph-mon[35321]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-12T02:46:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: osdmap e79: 12 total, 12 up, 12 in 2024-09-12T02:46:01.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:01 smithi033 ceph-mon[35092]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 -- 172.21.15.16:0/1151037117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 msgr2=0x7f196410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 --2- 172.21.15.16:0/1151037117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196410da40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f195c009ca0 tx=0x7f195c02a950 comp rx=0 tx=0).stop 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 -- 172.21.15.16:0/1151037117 shutdown_connections 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 --2- 172.21.15.16:0/1151037117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196410da40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 --2- 172.21.15.16:0/1151037117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f19641029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 --2- 172.21.15.16:0/1151037117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 0x7f1964101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.807+0000 7f196905d640 1 -- 172.21.15.16:0/1151037117 >> 172.21.15.16:0/1151037117 conn(0x7f19640fc7d0 msgr2=0x7f19640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.808+0000 7f196905d640 1 -- 172.21.15.16:0/1151037117 shutdown_connections 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.808+0000 7f196905d640 1 -- 172.21.15.16:0/1151037117 wait complete. 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.808+0000 7f196905d640 1 Processor -- start 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.808+0000 7f196905d640 1 -- start start 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 0x7f1964198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f1964198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19641120a0 con 0x7f1964102ef0 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1964111f20 con 0x7f1964102570 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196905d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1964112220 con 0x7f1964100e20 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f1964198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36986/0 (socket says 172.21.15.16:36986) 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f1964198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34122/0 (socket says 172.21.15.16:34122) 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 -- 172.21.15.16:0/2872406734 learned_addr learned my addr 172.21.15.16:0/2872406734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f1962d9d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 0x7f1964198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 -- 172.21.15.16:0/2872406734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 msgr2=0x7f1964198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 0x7f1964198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 -- 172.21.15.16:0/2872406734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 msgr2=0x7f1964198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f1964198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 -- 172.21.15.16:0/2872406734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196419d3f0 con 0x7f1964102ef0 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.809+0000 7f196359e640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f195c02ae70 tx=0x7f195c02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.810+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c034040 con 0x7f1964102ef0 2024-09-12T02:46:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.810+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195c030920 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.810+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196419d680 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.810+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c045480 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.810+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19641a4fc0 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.811+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1964101200 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.812+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f195c037020 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.812+0000 7f194bfff640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 0x7f194c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.813+0000 7f1962d9d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 0x7f194c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.813+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f195c0eff70 con 0x7f1964102ef0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.813+0000 7f1962d9d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 0x7f194c0a50c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f195400bbe0 tx=0x7f195400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:01.816+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f195c0811b0 con 0x7f1964102ef0 2024-09-12T02:46:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.018+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1964106f40 con 0x7f194c0a2c00 2024-09-12T02:46:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.046+0000 7f194bfff640 1 -- 172.21.15.16:0/2872406734 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1964106f40 con 0x7f194c0a2c00 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 msgr2=0x7f194c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 0x7f194c0a50c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f195400bbe0 tx=0x7f195400b390 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 msgr2=0x7f196419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f195c02ae70 tx=0x7f195c02f5d0 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 shutdown_connections 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f194c0a2c00 0x7f194c0a50c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1964102ef0 0x7f196419cd10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1964102570 0x7f1964198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 --2- 172.21.15.16:0/2872406734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1964100e20 0x7f1964198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 >> 172.21.15.16:0/2872406734 conn(0x7f19640fc7d0 msgr2=0x7f1964106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.048+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 shutdown_connections 2024-09-12T02:46:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:02.049+0000 7f196905d640 1 -- 172.21.15.16:0/2872406734 wait complete. 2024-09-12T02:46:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:02 smithi016 ceph-mon[30925]: osdmap e80: 12 total, 12 up, 12 in 2024-09-12T02:46:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:02 smithi184 ceph-mon[35321]: osdmap e80: 12 total, 12 up, 12 in 2024-09-12T02:46:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:02 smithi033 ceph-mon[35092]: osdmap e80: 12 total, 12 up, 12 in 2024-09-12T02:46:02.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-12T02:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:03 smithi016 ceph-mon[30925]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.1 KiB/s wr, 25 op/s 2024-09-12T02:46:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:03 smithi016 ceph-mon[30925]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:03.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:03 smithi016 ceph-mon[30925]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/3022488856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 msgr2=0x7f6c58103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/3022488856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c58103180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009900 tx=0x7f6c4802a480 comp rx=0 tx=0).stop 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/3022488856 shutdown_connections 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/3022488856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c5810c180 0x7f6c5810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/3022488856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c581036c0 0x7f6c5810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/3022488856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c58103180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/3022488856 >> 172.21.15.16:0/3022488856 conn(0x7f6c580fc7a0 msgr2=0x7f6c580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.499+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/3022488856 shutdown_connections 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/3022488856 wait complete. 2024-09-12T02:46:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 Processor -- start 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 -- start start 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c581036c0 0x7f6c5819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5810c180 0x7f6c5819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58111f60 con 0x7f6c58102da0 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58111de0 con 0x7f6c581036c0 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.500+0000 7f6c5dfc4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c581120e0 con 0x7f6c5810c180 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37006/0 (socket says 172.21.15.16:37006) 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 -- 172.21.15.16:0/1292702315 learned_addr learned my addr 172.21.15.16:0/1292702315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c56ffd640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c581036c0 0x7f6c5819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c57fff640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5810c180 0x7f6c5819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 -- 172.21.15.16:0/1292702315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5810c180 msgr2=0x7f6c5819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5810c180 0x7f6c5819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 -- 172.21.15.16:0/1292702315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c581036c0 msgr2=0x7f6c5819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c581036c0 0x7f6c5819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 -- 172.21.15.16:0/1292702315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5819f600 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c577fe640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6c48006010 tx=0x7f6c480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4803f030 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5819f890 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c48003ce0 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c48035070 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.501+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c581a71d0 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.503+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c58103180 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.504+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6c48042030 con 0x7f6c58102da0 2024-09-12T02:46:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.504+0000 7f6c54ff9640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.504+0000 7f6c56ffd640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.504+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6c480ef8a0 con 0x7f6c58102da0 2024-09-12T02:46:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.505+0000 7f6c56ffd640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 0x7f6c400a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c580febe0 tx=0x7f6c4c00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.509+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6c480bc770 con 0x7f6c58102da0 2024-09-12T02:46:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:03 smithi184 ceph-mon[35321]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.1 KiB/s wr, 25 op/s 2024-09-12T02:46:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:03 smithi184 ceph-mon[35321]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:03 smithi184 ceph-mon[35321]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-12T02:46:03.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.715+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f6c581020c0 con 0x7f6c400a2c10 2024-09-12T02:46:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.720+0000 7f6c54ff9640 1 -- 172.21.15.16:0/1292702315 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6c581020c0 con 0x7f6c400a2c10 2024-09-12T02:46:03.720 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/ea81b183-7644-4ee5-b31e-69e82ea2d449 2024-09-12T02:46:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 msgr2=0x7f6c400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 0x7f6c400a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c580febe0 tx=0x7f6c4c00a4c0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 msgr2=0x7f6c5819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6c48006010 tx=0x7f6c480042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 shutdown_connections 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5810c180 0x7f6c5819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c581036c0 0x7f6c5819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 --2- 172.21.15.16:0/1292702315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c58102da0 0x7f6c5819a650 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 >> 172.21.15.16:0/1292702315 conn(0x7f6c580fc7a0 msgr2=0x7f6c581019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 shutdown_connections 2024-09-12T02:46:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:03.722+0000 7f6c5dfc4640 1 -- 172.21.15.16:0/1292702315 wait complete. 2024-09-12T02:46:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:03 smithi033 ceph-mon[35092]: pgmap v281: 97 pgs: 97 active+clean; 585 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.1 KiB/s wr, 25 op/s 2024-09-12T02:46:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:03 smithi033 ceph-mon[35092]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:03 smithi033 ceph-mon[35092]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-12T02:46:04.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-12T02:46:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:04 smithi184 ceph-mon[35321]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:04.625 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:04.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:04 smithi016 ceph-mon[30925]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:04.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:04 smithi033 ceph-mon[35092]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 -- 172.21.15.16:0/1631657164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 msgr2=0x7f3dcc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1631657164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc1029b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009900 tx=0x7f3db402a480 comp rx=0 tx=0).stop 2024-09-12T02:46:05.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 -- 172.21.15.16:0/1631657164 shutdown_connections 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1631657164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dcc102ef0 0x7f3dcc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1631657164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc1029b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1631657164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 -- 172.21.15.16:0/1631657164 >> 172.21.15.16:0/1631657164 conn(0x7f3dcc0fc7d0 msgr2=0x7f3dcc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 -- 172.21.15.16:0/1631657164 shutdown_connections 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.114+0000 7f3dd25da640 1 -- 172.21.15.16:0/1631657164 wait complete. 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 Processor -- start 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 -- start start 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:05.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dcc102ef0 0x7f3dcc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc110f80 con 0x7f3dcc102570 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc110e00 con 0x7f3dcc100e20 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dd25da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc111100 con 0x7f3dcc102ef0 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37018/0 (socket says 172.21.15.16:37018) 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.115+0000 7f3dc3fff640 1 -- 172.21.15.16:0/1823725162 learned_addr learned my addr 172.21.15.16:0/1823725162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dcbfff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 -- 172.21.15.16:0/1823725162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dcc102ef0 msgr2=0x7f3dcc19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dcc102ef0 0x7f3dcc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 -- 172.21.15.16:0/1823725162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 msgr2=0x7f3dcc19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 -- 172.21.15.16:0/1823725162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc19f660 con 0x7f3dcc102570 2024-09-12T02:46:05.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dcbfff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc3fff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3db4006010 tx=0x7f3db4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db403f070 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3db4031c30 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc19f8f0 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4035030 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.116+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc1a7230 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.117+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc101200 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.118+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3db4042070 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.119+0000 7f3dc9ffb640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 0x7f3db80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.119+0000 7f3dcbfff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 0x7f3db80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.119+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3db40efb20 con 0x7f3dcc102570 2024-09-12T02:46:05.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.119+0000 7f3dcbfff640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 0x7f3db80a5150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005d00 tx=0x7f3dbc005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:05.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.123+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3db4031de0 con 0x7f3dcc102570 2024-09-12T02:46:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.329+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3dcc06a760 con 0x7f3db80a2c90 2024-09-12T02:46:05.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:05 smithi016 ceph-mon[30925]: pgmap v282: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 28 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.356+0000 7f3dc9ffb640 1 -- 172.21.15.16:0/1823725162 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3dcc06a760 con 0x7f3db80a2c90 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 msgr2=0x7f3db80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 0x7f3db80a5150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005d00 tx=0x7f3dbc005c70 comp rx=0 tx=0).stop 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 msgr2=0x7f3dcc19abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3db4006010 tx=0x7f3db4003c10 comp rx=0 tx=0).stop 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 shutdown_connections 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dcc102ef0 0x7f3dcc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3db80a2c90 0x7f3db80a5150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dcc102570 0x7f3dcc19abf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 --2- 172.21.15.16:0/1823725162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dcc100e20 0x7f3dcc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:05.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 >> 172.21.15.16:0/1823725162 conn(0x7f3dcc0fc7d0 msgr2=0x7f3dcc06a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:05.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 shutdown_connections 2024-09-12T02:46:05.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:05.359+0000 7f3dd25da640 1 -- 172.21.15.16:0/1823725162 wait complete. 2024-09-12T02:46:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:05 smithi184 ceph-mon[35321]: pgmap v282: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 28 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:05 smithi033 ceph-mon[35092]: pgmap v282: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 28 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:06.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-12T02:46:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:06 smithi016 ceph-mon[30925]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:06.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:06 smithi184 ceph-mon[35321]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:06.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:06 smithi033 ceph-mon[35092]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:06.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 -- 172.21.15.16:0/4239524016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df0102d30 msgr2=0x7f6df0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 --2- 172.21.15.16:0/4239524016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df0102d30 0x7f6df0103110 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009930 tx=0x7f6dec02a420 comp rx=0 tx=0).stop 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 -- 172.21.15.16:0/4239524016 shutdown_connections 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 --2- 172.21.15.16:0/4239524016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 0x7f6df010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 --2- 172.21.15.16:0/4239524016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df01036e0 0x7f6df0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 --2- 172.21.15.16:0/4239524016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df0102d30 0x7f6df0103110 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 -- 172.21.15.16:0/4239524016 >> 172.21.15.16:0/4239524016 conn(0x7f6df00fc7d0 msgr2=0x7f6df00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 -- 172.21.15.16:0/4239524016 shutdown_connections 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.764+0000 7f6df6aba640 1 -- 172.21.15.16:0/4239524016 wait complete. 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 Processor -- start 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 -- start start 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 0x7f6df01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 0x7f6df019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:06.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01120a0 con 0x7f6df01036e0 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0111f20 con 0x7f6df01082a0 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df6aba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0112220 con 0x7f6df0102d30 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42638/0 (socket says 172.21.15.16:42638) 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 -- 172.21.15.16:0/3857836778 learned_addr learned my addr 172.21.15.16:0/3857836778 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df5057640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 0x7f6df019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df4856640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 0x7f6df01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:06.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 -- 172.21.15.16:0/3857836778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 msgr2=0x7f6df01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:06.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 0x7f6df01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 -- 172.21.15.16:0/3857836778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 msgr2=0x7f6df019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:06.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 0x7f6df019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6de7fff640 1 -- 172.21.15.16:0/3857836778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df019d550 con 0x7f6df01036e0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.765+0000 7f6df4856640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 0x7f6df01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6de7fff640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6de000c8c0 tx=0x7f6de000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0007bb0 con 0x7f6df01036e0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df019d840 con 0x7f6df01036e0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6de0007d50 con 0x7f6df01036e0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0018830 con 0x7f6df01036e0 2024-09-12T02:46:06.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.766+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df01a5120 con 0x7f6df01036e0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.767+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df006a850 con 0x7f6df01036e0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.768+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6de0018a10 con 0x7f6df01036e0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.769+0000 7f6de5ffb640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 0x7f6dd80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.769+0000 7f6df4856640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 0x7f6dd80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.769+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6de00cf3a0 con 0x7f6df01036e0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.769+0000 7f6df4856640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 0x7f6dd80a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6dec000c00 tx=0x7f6dec032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.773+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6de0014030 con 0x7f6df01036e0 2024-09-12T02:46:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.975+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6df0062630 con 0x7f6dd80a2c10 2024-09-12T02:46:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.980+0000 7f6de5ffb640 1 -- 172.21.15.16:0/3857836778 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6df0062630 con 0x7f6dd80a2c10 2024-09-12T02:46:06.981 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/f393359a-6719-4728-81d5-f56f7e21ccf3 2024-09-12T02:46:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 msgr2=0x7f6dd80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 0x7f6dd80a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6dec000c00 tx=0x7f6dec032d50 comp rx=0 tx=0).stop 2024-09-12T02:46:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 msgr2=0x7f6df0198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6de000c8c0 tx=0x7f6de000cd80 comp rx=0 tx=0).stop 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 shutdown_connections 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6df01082a0 0x7f6df019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6dd80a2c10 0x7f6dd80a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df01036e0 0x7f6df0198ae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 --2- 172.21.15.16:0/3857836778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df0102d30 0x7f6df01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 >> 172.21.15.16:0/3857836778 conn(0x7f6df00fc7d0 msgr2=0x7f6df0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 shutdown_connections 2024-09-12T02:46:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:06.983+0000 7f6df6aba640 1 -- 172.21.15.16:0/3857836778 wait complete. 2024-09-12T02:46:07.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:07 smithi184 ceph-mon[35321]: pgmap v283: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 32 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:07 smithi016 ceph-mon[30925]: pgmap v283: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 32 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:07.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:46:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:07 smithi033 ceph-mon[35092]: pgmap v283: 97 pgs: 97 active+clean; 610 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 32 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:08.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:08 smithi016 ceph-mon[30925]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 -- 172.21.15.16:0/1461350727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54106f10 msgr2=0x7f3b5410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 --2- 172.21.15.16:0/1461350727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54106f10 0x7f3b5410ea90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009ca0 tx=0x7f3b4c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 -- 172.21.15.16:0/1461350727 shutdown_connections 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 --2- 172.21.15.16:0/1461350727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54106f10 0x7f3b5410ea90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 --2- 172.21.15.16:0/1461350727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54102790 0x7f3b541067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 --2- 172.21.15.16:0/1461350727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54101de0 0x7f3b541021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 -- 172.21.15.16:0/1461350727 >> 172.21.15.16:0/1461350727 conn(0x7f3b540fd8c0 msgr2=0x7f3b540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 -- 172.21.15.16:0/1461350727 shutdown_connections 2024-09-12T02:46:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.392+0000 7f3b58e58640 1 -- 172.21.15.16:0/1461350727 wait complete. 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 Processor -- start 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 -- start start 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54102790 0x7f3b54074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 0x7f3b54074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54111fb0 con 0x7f3b54101de0 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54111e30 con 0x7f3b54102790 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b58e58640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54112130 con 0x7f3b54106f10 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42668/0 (socket says 172.21.15.16:42668) 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b5259c640 1 -- 172.21.15.16:0/3540171065 learned_addr learned my addr 172.21.15.16:0/3540171065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.393+0000 7f3b4bfff640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54102790 0x7f3b54074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b52d9d640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 0x7f3b54074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 -- 172.21.15.16:0/3540171065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 msgr2=0x7f3b54074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 0x7f3b54074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 -- 172.21.15.16:0/3540171065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54102790 msgr2=0x7f3b54074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54102790 0x7f3b54074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 -- 172.21.15.16:0/3540171065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b541a6950 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b52d9d640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 0x7f3b54074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b5259c640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b4a0 tx=0x7f3b4400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44014070 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b541a6c40 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b44010af0 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44017d10 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.394+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541a7180 con 0x7f3b54101de0 2024-09-12T02:46:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.395+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b541021c0 con 0x7f3b54101de0 2024-09-12T02:46:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.396+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3b44017070 con 0x7f3b54101de0 2024-09-12T02:46:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.397+0000 7f3b4b7fe640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 0x7f3b400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.397+0000 7f3b4bfff640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 0x7f3b400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:08.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.397+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3b440ce420 con 0x7f3b54101de0 2024-09-12T02:46:08.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.397+0000 7f3b4bfff640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 0x7f3b400a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b54075f80 tx=0x7f3b3c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.401+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3b4409b340 con 0x7f3b54101de0 2024-09-12T02:46:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:08 smithi184 ceph-mon[35321]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.612+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b54109b60 con 0x7f3b54101de0 2024-09-12T02:46:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.613+0000 7f3b4b7fe640 1 -- 172.21.15.16:0/3540171065 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7f3b4409ac40 con 0x7f3b54101de0 2024-09-12T02:46:08.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:08.613 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:46:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 msgr2=0x7f3b400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 0x7f3b400a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b54075f80 tx=0x7f3b3c00b390 comp rx=0 tx=0).stop 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 msgr2=0x7f3b540744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b4a0 tx=0x7f3b4400b960 comp rx=0 tx=0).stop 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 shutdown_connections 2024-09-12T02:46:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b54106f10 0x7f3b54074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b54102790 0x7f3b54074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b400a2c10 0x7f3b400a50d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 --2- 172.21.15.16:0/3540171065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b54101de0 0x7f3b540744f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 >> 172.21.15.16:0/3540171065 conn(0x7f3b540fd8c0 msgr2=0x7f3b540fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 shutdown_connections 2024-09-12T02:46:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:08.615+0000 7f3b58e58640 1 -- 172.21.15.16:0/3540171065 wait complete. 2024-09-12T02:46:08.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:08 smithi033 ceph-mon[35092]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:46:09.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:46:09.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:09 smithi016 ceph-mon[30925]: pgmap v284: 97 pgs: 97 active+clean; 610 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.1 KiB/s wr, 17 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:09 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3540171065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:09.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:09 smithi184 ceph-mon[35321]: pgmap v284: 97 pgs: 97 active+clean; 610 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.1 KiB/s wr, 17 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:09.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:09 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3540171065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:09 smithi033 ceph-mon[35092]: pgmap v284: 97 pgs: 97 active+clean; 610 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.1 KiB/s wr, 17 op/s; 16 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:09 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3540171065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:10.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 -- 172.21.15.16:0/4003217740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc241002f0 msgr2=0x7fbc241006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:10.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 --2- 172.21.15.16:0/4003217740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc241002f0 0x7fbc241006d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009930 tx=0x7fbc1402a420 comp rx=0 tx=0).stop 2024-09-12T02:46:10.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 -- 172.21.15.16:0/4003217740 shutdown_connections 2024-09-12T02:46:10.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 --2- 172.21.15.16:0/4003217740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc2410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 --2- 172.21.15.16:0/4003217740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc24100ca0 0x7fbc24106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 --2- 172.21.15.16:0/4003217740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc241002f0 0x7fbc241006d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.982+0000 7fbc293ef640 1 -- 172.21.15.16:0/4003217740 >> 172.21.15.16:0/4003217740 conn(0x7fbc24077210 msgr2=0x7fbc240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:10.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 -- 172.21.15.16:0/4003217740 shutdown_connections 2024-09-12T02:46:10.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 -- 172.21.15.16:0/4003217740 wait complete. 2024-09-12T02:46:10.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 Processor -- start 2024-09-12T02:46:10.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 -- start start 2024-09-12T02:46:10.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc241002f0 0x7fbc24101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:10.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:10.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.983+0000 7fbc293ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc24101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:10.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc293ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24112080 con 0x7fbc24100ca0 2024-09-12T02:46:10.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc293ef640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24111f00 con 0x7fbc241002f0 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc293ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24112200 con 0x7fbc24106f50 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42694/0 (socket says 172.21.15.16:42694) 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 -- 172.21.15.16:0/2660190877 learned_addr learned my addr 172.21.15.16:0/2660190877 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc237fe640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc24101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:10.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc22ffd640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc241002f0 0x7fbc24101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:10.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 -- 172.21.15.16:0/2660190877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 msgr2=0x7fbc24101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:10.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc24101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 -- 172.21.15.16:0/2660190877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc241002f0 msgr2=0x7fbc24101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:10.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc241002f0 0x7fbc24101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 -- 172.21.15.16:0/2660190877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc241a6b00 con 0x7fbc24100ca0 2024-09-12T02:46:10.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc237fe640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc24101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:10.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc227fc640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b450 tx=0x7fbc1800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:10.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18015020 con 0x7fbc24100ca0 2024-09-12T02:46:10.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc241a6df0 con 0x7fbc24100ca0 2024-09-12T02:46:10.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc18011d70 con 0x7fbc24100ca0 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800f340 con 0x7fbc24100ca0 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.984+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc241a72a0 con 0x7fbc24100ca0 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.985+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc24100750 con 0x7fbc24100ca0 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.987+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbc18018020 con 0x7fbc24100ca0 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.987+0000 7fbc0bfff640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 0x7fbc0c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:10.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.988+0000 7fbc22ffd640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 0x7fbc0c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:10.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.988+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbc180cebf0 con 0x7fbc24100ca0 2024-09-12T02:46:10.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.988+0000 7fbc22ffd640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 0x7fbc0c0a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc14000c00 tx=0x7fbc14031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:10.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:09.992+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbc1801f020 con 0x7fbc24100ca0 2024-09-12T02:46:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.197+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc24109bb0 con 0x7fbc24100ca0 2024-09-12T02:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.198+0000 7fbc0bfff640 1 -- 172.21.15.16:0/2660190877 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7fbc1800d040 con 0x7fbc24100ca0 2024-09-12T02:46:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:10.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 msgr2=0x7fbc0c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 0x7fbc0c0a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc14000c00 tx=0x7fbc14031490 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 msgr2=0x7fbc241017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b450 tx=0x7fbc1800b910 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 shutdown_connections 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc24106f50 0x7fbc24101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc0c0a2c10 0x7fbc0c0a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc24100ca0 0x7fbc241017b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 --2- 172.21.15.16:0/2660190877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc241002f0 0x7fbc24101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 >> 172.21.15.16:0/2660190877 conn(0x7fbc24077210 msgr2=0x7fbc24105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 shutdown_connections 2024-09-12T02:46:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:10.200+0000 7fbc293ef640 1 -- 172.21.15.16:0/2660190877 wait complete. 2024-09-12T02:46:10.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:10 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2660190877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:10.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:10 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2660190877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:10 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2660190877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:10.846 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-12T02:46:10.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:46:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:11.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:11 smithi016 ceph-mon[30925]: pgmap v285: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 18 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:11 smithi184 ceph-mon[35321]: pgmap v285: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 18 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 -- 172.21.15.16:0/2705239293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c1019e0 msgr2=0x7f9b6c101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2705239293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c101e40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009930 tx=0x7f9b6002a420 comp rx=0 tx=0).stop 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 -- 172.21.15.16:0/2705239293 shutdown_connections 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2705239293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102380 0x7f9b6c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2705239293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c101e40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2705239293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c107ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 -- 172.21.15.16:0/2705239293 >> 172.21.15.16:0/2705239293 conn(0x7f9b6c0fd700 msgr2=0x7f9b6c0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 -- 172.21.15.16:0/2705239293 shutdown_connections 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 -- 172.21.15.16:0/2705239293 wait complete. 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.566+0000 7f9b715a1640 1 Processor -- start 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 -- start start 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c111eb0 con 0x7f9b6c102380 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c111d30 con 0x7f9b6c1019e0 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b715a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c112030 con 0x7f9b6c107af0 2024-09-12T02:46:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42710/0 (socket says 172.21.15.16:42710) 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 -- 172.21.15.16:0/2294150267 learned_addr learned my addr 172.21.15.16:0/2294150267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6bfff640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b70da0640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 -- 172.21.15.16:0/2294150267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 msgr2=0x7f9b6c19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 -- 172.21.15.16:0/2294150267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c1019e0 msgr2=0x7f9b6c1982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c1982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b6b7fe640 1 -- 172.21.15.16:0/2294150267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c19d2a0 con 0x7f9b6c102380 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.567+0000 7f9b70da0640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b6b7fe640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009a50 tx=0x7f9b60004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6003f030 con 0x7f9b6c102380 2024-09-12T02:46:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c19d530 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b60003ad0 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60035070 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.568+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c1a4e70 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.569+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c101e40 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.570+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9b60042030 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.571+0000 7f9b697fa640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 0x7f9b540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.571+0000 7f9b6bfff640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 0x7f9b540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.571+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b600ef9a0 con 0x7f9b6c102380 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.571+0000 7f9b6bfff640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 0x7f9b540a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00ab60 tx=0x7f9b5c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.575+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9b600bc7b0 con 0x7f9b6c102380 2024-09-12T02:46:11.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.746+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b6c0fe310 con 0x7f9b540a2c10 2024-09-12T02:46:11.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:11 smithi033 ceph-mon[35092]: pgmap v285: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 18 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-12T02:46:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.753+0000 7f9b697fa640 1 -- 172.21.15.16:0/2294150267 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258772 (secure 0 0 0) 0x7f9b6c0fe310 con 0x7f9b540a2c10 2024-09-12T02:46:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 msgr2=0x7f9b540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 0x7f9b540a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00ab60 tx=0x7f9b5c00a2d0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 msgr2=0x7f9b6c198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009a50 tx=0x7f9b60004270 comp rx=0 tx=0).stop 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 shutdown_connections 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c107af0 0x7f9b6c19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2c10 0x7f9b540a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102380 0x7f9b6c198830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 --2- 172.21.15.16:0/2294150267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c1019e0 0x7f9b6c1982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 >> 172.21.15.16:0/2294150267 conn(0x7f9b6c0fd700 msgr2=0x7f9b6c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 shutdown_connections 2024-09-12T02:46:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:11.756+0000 7f9b715a1640 1 -- 172.21.15.16:0/2294150267 wait complete. 2024-09-12T02:46:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:12 smithi016 ceph-mon[30925]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:12.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:12 smithi184 ceph-mon[35321]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:12.569 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-12T02:46:10.001929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648315,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":265,"num_read_kb":243,"num_write":419,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378404,"kb_used_data":12492,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563420,"statfs":{"total":1151940427776,"available":1151552942080,"internally_reserved":0,"allocated":12791808,"data_stored":7067375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":374453565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51795,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":109,"num_read_kb":108,"num_write":68,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.759947"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245601+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:46:01.245601+0000","last_peered":"2024-09-12T02:46:01.245601+0000","last_clean":"2024-09-12T02:46:01.245601+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:46:01.245601+0000","last_undegraded":"2024-09-12T02:46:01.245601+0000","last_fullsized":"2024-09-12T02:46:01.245601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247959+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:46:01.247959+0000","last_peered":"2024-09-12T02:46:01.247959+0000","last_clean":"2024-09-12T02:46:01.247959+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:46:01.247959+0000","last_undegraded":"2024-09-12T02:46:01.247959+0000","last_fullsized":"2024-09-12T02:46:01.247959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247397+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:46:01.247397+0000","last_peered":"2024-09-12T02:46:01.247397+0000","last_clean":"2024-09-12T02:46:01.247397+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:46:01.247397+0000","last_undegraded":"2024-09-12T02:46:01.247397+0000","last_fullsized":"2024-09-12T02:46:01.247397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247671+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:46:01.247671+0000","last_peered":"2024-09-12T02:46:01.247671+0000","last_clean":"2024-09-12T02:46:01.247671+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:46:01.247671+0000","last_undegraded":"2024-09-12T02:46:01.247671+0000","last_fullsized":"2024-09-12T02:46:01.247671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247675+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:46:01.247675+0000","last_peered":"2024-09-12T02:46:01.247675+0000","last_clean":"2024-09-12T02:46:01.247675+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:46:01.247675+0000","last_undegraded":"2024-09-12T02:46:01.247675+0000","last_fullsized":"2024-09-12T02:46:01.247675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245784+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:46:01.245784+0000","last_peered":"2024-09-12T02:46:01.245784+0000","last_clean":"2024-09-12T02:46:01.245784+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:46:01.245784+0000","last_undegraded":"2024-09-12T02:46:01.245784+0000","last_fullsized":"2024-09-12T02:46:01.245784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249851+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:46:01.249851+0000","last_peered":"2024-09-12T02:46:01.249851+0000","last_clean":"2024-09-12T02:46:01.249851+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:46:01.249851+0000","last_undegraded":"2024-09-12T02:46:01.249851+0000","last_fullsized":"2024-09-12T02:46:01.249851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244102+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:46:01.244102+0000","last_peered":"2024-09-12T02:46:01.244102+0000","last_clean":"2024-09-12T02:46:01.244102+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:46:01.244102+0000","last_undegraded":"2024-09-12T02:46:01.244102+0000","last_fullsized":"2024-09-12T02:46:01.244102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255221+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:46:01.255221+0000","last_peered":"2024-09-12T02:46:01.255221+0000","last_clean":"2024-09-12T02:46:01.255221+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:46:01.255221+0000","last_undegraded":"2024-09-12T02:46:01.255221+0000","last_fullsized":"2024-09-12T02:46:01.255221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255844+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:46:01.255844+0000","last_peered":"2024-09-12T02:46:01.255844+0000","last_clean":"2024-09-12T02:46:01.255844+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:46:01.255844+0000","last_undegraded":"2024-09-12T02:46:01.255844+0000","last_fullsized":"2024-09-12T02:46:01.255844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250813+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:46:01.250813+0000","last_peered":"2024-09-12T02:46:01.250813+0000","last_clean":"2024-09-12T02:46:01.250813+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:46:01.250813+0000","last_undegraded":"2024-09-12T02:46:01.250813+0000","last_fullsized":"2024-09-12T02:46:01.250813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247697+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:46:01.247697+0000","last_peered":"2024-09-12T02:46:01.247697+0000","last_clean":"2024-09-12T02:46:01.247697+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:46:01.247697+0000","last_undegraded":"2024-09-12T02:46:01.247697+0000","last_fullsized":"2024-09-12T02:46:01.247697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251093+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:46:01.251093+0000","last_peered":"2024-09-12T02:46:01.251093+0000","last_clean":"2024-09-12T02:46:01.251093+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:46:01.251093+0000","last_undegraded":"2024-09-12T02:46:01.251093+0000","last_fullsized":"2024-09-12T02:46:01.251093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247142+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:46:01.247142+0000","last_peered":"2024-09-12T02:46:01.247142+0000","last_clean":"2024-09-12T02:46:01.247142+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:46:01.247142+0000","last_undegraded":"2024-09-12T02:46:01.247142+0000","last_fullsized":"2024-09-12T02:46:01.247142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247412+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:46:01.247412+0000","last_peered":"2024-09-12T02:46:01.247412+0000","last_clean":"2024-09-12T02:46:01.247412+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:46:01.247412+0000","last_undegraded":"2024-09-12T02:46:01.247412+0000","last_fullsized":"2024-09-12T02:46:01.247412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254313+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:46:01.254313+0000","last_peered":"2024-09-12T02:46:01.254313+0000","last_clean":"2024-09-12T02:46:01.254313+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:46:01.254313+0000","last_undegraded":"2024-09-12T02:46:01.254313+0000","last_fullsized":"2024-09-12T02:46:01.254313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245720+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:46:01.245720+0000","last_peered":"2024-09-12T02:46:01.245720+0000","last_clean":"2024-09-12T02:46:01.245720+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:46:01.245720+0000","last_undegraded":"2024-09-12T02:46:01.245720+0000","last_fullsized":"2024-09-12T02:46:01.245720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250415+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:46:01.250415+0000","last_peered":"2024-09-12T02:46:01.250415+0000","last_clean":"2024-09-12T02:46:01.250415+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:46:01.250415+0000","last_undegraded":"2024-09-12T02:46:01.250415+0000","last_fullsized":"2024-09-12T02:46:01.250415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252979+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:46:01.252979+0000","last_peered":"2024-09-12T02:46:01.252979+0000","last_clean":"2024-09-12T02:46:01.252979+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:46:01.252979+0000","last_undegraded":"2024-09-12T02:46:01.252979+0000","last_fullsized":"2024-09-12T02:46:01.252979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245213+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:46:01.245213+0000","last_peered":"2024-09-12T02:46:01.245213+0000","last_clean":"2024-09-12T02:46:01.245213+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:46:01.245213+0000","last_undegraded":"2024-09-12T02:46:01.245213+0000","last_fullsized":"2024-09-12T02:46:01.245213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247833+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:46:01.247833+0000","last_peered":"2024-09-12T02:46:01.247833+0000","last_clean":"2024-09-12T02:46:01.247833+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:46:01.247833+0000","last_undegraded":"2024-09-12T02:46:01.247833+0000","last_fullsized":"2024-09-12T02:46:01.247833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254746+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:46:01.254746+0000","last_peered":"2024-09-12T02:46:01.254746+0000","last_clean":"2024-09-12T02:46:01.254746+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:46:01.254746+0000","last_undegraded":"2024-09-12T02:46:01.254746+0000","last_fullsized":"2024-09-12T02:46:01.254746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244169+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:46:01.244169+0000","last_peered":"2024-09-12T02:46:01.244169+0000","last_clean":"2024-09-12T02:46:01.244169+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:46:01.244169+0000","last_undegraded":"2024-09-12T02:46:01.244169+0000","last_fullsized":"2024-09-12T02:46:01.244169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248907+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:46:01.248907+0000","last_peered":"2024-09-12T02:46:01.248907+0000","last_clean":"2024-09-12T02:46:01.248907+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:46:01.248907+0000","last_undegraded":"2024-09-12T02:46:01.248907+0000","last_fullsized":"2024-09-12T02:46:01.248907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244381+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:46:01.244381+0000","last_peered":"2024-09-12T02:46:01.244381+0000","last_clean":"2024-09-12T02:46:01.244381+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:46:01.244381+0000","last_undegraded":"2024-09-12T02:46:01.244381+0000","last_fullsized":"2024-09-12T02:46:01.244381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253636+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:46:01.253636+0000","last_peered":"2024-09-12T02:46:01.253636+0000","last_clean":"2024-09-12T02:46:01.253636+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:46:01.253636+0000","last_undegraded":"2024-09-12T02:46:01.253636+0000","last_fullsized":"2024-09-12T02:46:01.253636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245512+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:46:01.245512+0000","last_peered":"2024-09-12T02:46:01.245512+0000","last_clean":"2024-09-12T02:46:01.245512+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:46:01.245512+0000","last_undegraded":"2024-09-12T02:46:01.245512+0000","last_fullsized":"2024-09-12T02:46:01.245512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247679+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:46:01.247679+0000","last_peered":"2024-09-12T02:46:01.247679+0000","last_clean":"2024-09-12T02:46:01.247679+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:46:01.247679+0000","last_undegraded":"2024-09-12T02:46:01.247679+0000","last_fullsized":"2024-09-12T02:46:01.247679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247302+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:46:01.247302+0000","last_peered":"2024-09-12T02:46:01.247302+0000","last_clean":"2024-09-12T02:46:01.247302+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:46:01.247302+0000","last_undegraded":"2024-09-12T02:46:01.247302+0000","last_fullsized":"2024-09-12T02:46:01.247302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252658+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:46:01.252658+0000","last_peered":"2024-09-12T02:46:01.252658+0000","last_clean":"2024-09-12T02:46:01.252658+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:46:01.252658+0000","last_undegraded":"2024-09-12T02:46:01.252658+0000","last_fullsized":"2024-09-12T02:46:01.252658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252938+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:46:01.252938+0000","last_peered":"2024-09-12T02:46:01.252938+0000","last_clean":"2024-09-12T02:46:01.252938+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:46:01.252938+0000","last_undegraded":"2024-09-12T02:46:01.252938+0000","last_fullsized":"2024-09-12T02:46:01.252938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245099+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:46:01.245099+0000","last_peered":"2024-09-12T02:46:01.245099+0000","last_clean":"2024-09-12T02:46:01.245099+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:46:01.245099+0000","last_undegraded":"2024-09-12T02:46:01.245099+0000","last_fullsized":"2024-09-12T02:46:01.245099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248039+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:46:01.248039+0000","last_peered":"2024-09-12T02:46:01.248039+0000","last_clean":"2024-09-12T02:46:01.248039+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:46:01.248039+0000","last_undegraded":"2024-09-12T02:46:01.248039+0000","last_fullsized":"2024-09-12T02:46:01.248039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.945648+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:46:07.945648+0000","last_peered":"2024-09-12T02:46:07.945648+0000","last_clean":"2024-09-12T02:46:07.945648+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:46:07.945648+0000","last_undegraded":"2024-09-12T02:46:07.945648+0000","last_fullsized":"2024-09-12T02:46:07.945648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248424+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:46:01.248424+0000","last_peered":"2024-09-12T02:46:01.248424+0000","last_clean":"2024-09-12T02:46:01.248424+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:46:01.248424+0000","last_undegraded":"2024-09-12T02:46:01.248424+0000","last_fullsized":"2024-09-12T02:46:01.248424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244158+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:46:01.244158+0000","last_peered":"2024-09-12T02:46:01.244158+0000","last_clean":"2024-09-12T02:46:01.244158+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:46:01.244158+0000","last_undegraded":"2024-09-12T02:46:01.244158+0000","last_fullsized":"2024-09-12T02:46:01.244158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248649+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:46:01.248649+0000","last_peered":"2024-09-12T02:46:01.248649+0000","last_clean":"2024-09-12T02:46:01.248649+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:46:01.248649+0000","last_undegraded":"2024-09-12T02:46:01.248649+0000","last_fullsized":"2024-09-12T02:46:01.248649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.944499+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:46:07.944499+0000","last_peered":"2024-09-12T02:46:07.944499+0000","last_clean":"2024-09-12T02:46:07.944499+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:46:07.944499+0000","last_undegraded":"2024-09-12T02:46:07.944499+0000","last_fullsized":"2024-09-12T02:46:07.944499+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255352+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:46:01.255352+0000","last_peered":"2024-09-12T02:46:01.255352+0000","last_clean":"2024-09-12T02:46:01.255352+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:46:01.255352+0000","last_undegraded":"2024-09-12T02:46:01.255352+0000","last_fullsized":"2024-09-12T02:46:01.255352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253009+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:46:01.253009+0000","last_peered":"2024-09-12T02:46:01.253009+0000","last_clean":"2024-09-12T02:46:01.253009+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:46:01.253009+0000","last_undegraded":"2024-09-12T02:46:01.253009+0000","last_fullsized":"2024-09-12T02:46:01.253009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254608+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:46:01.254608+0000","last_peered":"2024-09-12T02:46:01.254608+0000","last_clean":"2024-09-12T02:46:01.254608+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:46:01.254608+0000","last_undegraded":"2024-09-12T02:46:01.254608+0000","last_fullsized":"2024-09-12T02:46:01.254608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'27","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.947949+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:46:07.947949+0000","last_peered":"2024-09-12T02:46:07.947949+0000","last_clean":"2024-09-12T02:46:07.947949+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:46:07.947949+0000","last_undegraded":"2024-09-12T02:46:07.947949+0000","last_fullsized":"2024-09-12T02:46:07.947949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50112,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":62,"num_write":29,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248286+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:46:01.248286+0000","last_peered":"2024-09-12T02:46:01.248286+0000","last_clean":"2024-09-12T02:46:01.248286+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:46:01.248286+0000","last_undegraded":"2024-09-12T02:46:01.248286+0000","last_fullsized":"2024-09-12T02:46:01.248286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247592+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:46:01.247592+0000","last_peered":"2024-09-12T02:46:01.247592+0000","last_clean":"2024-09-12T02:46:01.247592+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:46:01.247592+0000","last_undegraded":"2024-09-12T02:46:01.247592+0000","last_fullsized":"2024-09-12T02:46:01.247592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250385+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:46:01.250385+0000","last_peered":"2024-09-12T02:46:01.250385+0000","last_clean":"2024-09-12T02:46:01.250385+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:46:01.250385+0000","last_undegraded":"2024-09-12T02:46:01.250385+0000","last_fullsized":"2024-09-12T02:46:01.250385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'6","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:03.940909+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:46:03.940909+0000","last_peered":"2024-09-12T02:46:03.940909+0000","last_clean":"2024-09-12T02:46:03.940909+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:46:03.940909+0000","last_undegraded":"2024-09-12T02:46:03.940909+0000","last_fullsized":"2024-09-12T02:46:03.940909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249800+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:46:01.249800+0000","last_peered":"2024-09-12T02:46:01.249800+0000","last_clean":"2024-09-12T02:46:01.249800+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:46:01.249800+0000","last_undegraded":"2024-09-12T02:46:01.249800+0000","last_fullsized":"2024-09-12T02:46:01.249800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247633+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:46:01.247633+0000","last_peered":"2024-09-12T02:46:01.247633+0000","last_clean":"2024-09-12T02:46:01.247633+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:46:01.247633+0000","last_undegraded":"2024-09-12T02:46:01.247633+0000","last_fullsized":"2024-09-12T02:46:01.247633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245314+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:46:01.245314+0000","last_peered":"2024-09-12T02:46:01.245314+0000","last_clean":"2024-09-12T02:46:01.245314+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:46:01.245314+0000","last_undegraded":"2024-09-12T02:46:01.245314+0000","last_fullsized":"2024-09-12T02:46:01.245314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251385+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:46:01.251385+0000","last_peered":"2024-09-12T02:46:01.251385+0000","last_clean":"2024-09-12T02:46:01.251385+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:46:01.251385+0000","last_undegraded":"2024-09-12T02:46:01.251385+0000","last_fullsized":"2024-09-12T02:46:01.251385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250603+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:46:01.250603+0000","last_peered":"2024-09-12T02:46:01.250603+0000","last_clean":"2024-09-12T02:46:01.250603+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:46:01.250603+0000","last_undegraded":"2024-09-12T02:46:01.250603+0000","last_fullsized":"2024-09-12T02:46:01.250603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:03.939260+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:46:03.939260+0000","last_peered":"2024-09-12T02:46:03.939260+0000","last_clean":"2024-09-12T02:46:03.939260+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:46:03.939260+0000","last_undegraded":"2024-09-12T02:46:03.939260+0000","last_fullsized":"2024-09-12T02:46:03.939260+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250931+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:46:01.250931+0000","last_peered":"2024-09-12T02:46:01.250931+0000","last_clean":"2024-09-12T02:46:01.250931+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:46:01.250931+0000","last_undegraded":"2024-09-12T02:46:01.250931+0000","last_fullsized":"2024-09-12T02:46:01.250931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":334,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250005+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250005+0000","last_peered":"2024-09-12T02:46:01.250005+0000","last_clean":"2024-09-12T02:46:01.250005+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250005+0000","last_undegraded":"2024-09-12T02:46:01.250005+0000","last_fullsized":"2024-09-12T02:46:01.250005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244482+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:46:01.244482+0000","last_peered":"2024-09-12T02:46:01.244482+0000","last_clean":"2024-09-12T02:46:01.244482+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:46:01.244482+0000","last_undegraded":"2024-09-12T02:46:01.244482+0000","last_fullsized":"2024-09-12T02:46:01.244482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247535+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:46:01.247535+0000","last_peered":"2024-09-12T02:46:01.247535+0000","last_clean":"2024-09-12T02:46:01.247535+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:46:01.247535+0000","last_undegraded":"2024-09-12T02:46:01.247535+0000","last_fullsized":"2024-09-12T02:46:01.247535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:03.940245+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:46:03.940245+0000","last_peered":"2024-09-12T02:46:03.940245+0000","last_clean":"2024-09-12T02:46:03.940245+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:46:03.940245+0000","last_undegraded":"2024-09-12T02:46:03.940245+0000","last_fullsized":"2024-09-12T02:46:03.940245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249000+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:46:01.249000+0000","last_peered":"2024-09-12T02:46:01.249000+0000","last_clean":"2024-09-12T02:46:01.249000+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:46:01.249000+0000","last_undegraded":"2024-09-12T02:46:01.249000+0000","last_fullsized":"2024-09-12T02:46:01.249000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.947287+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:46:07.947287+0000","last_peered":"2024-09-12T02:46:07.947287+0000","last_clean":"2024-09-12T02:46:07.947287+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:46:07.947287+0000","last_undegraded":"2024-09-12T02:46:07.947287+0000","last_fullsized":"2024-09-12T02:46:07.947287+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251067+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:46:01.251067+0000","last_peered":"2024-09-12T02:46:01.251067+0000","last_clean":"2024-09-12T02:46:01.251067+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:46:01.251067+0000","last_undegraded":"2024-09-12T02:46:01.251067+0000","last_fullsized":"2024-09-12T02:46:01.251067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245843+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:46:01.245843+0000","last_peered":"2024-09-12T02:46:01.245843+0000","last_clean":"2024-09-12T02:46:01.245843+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:46:01.245843+0000","last_undegraded":"2024-09-12T02:46:01.245843+0000","last_fullsized":"2024-09-12T02:46:01.245843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247794+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:46:01.247794+0000","last_peered":"2024-09-12T02:46:01.247794+0000","last_clean":"2024-09-12T02:46:01.247794+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:46:01.247794+0000","last_undegraded":"2024-09-12T02:46:01.247794+0000","last_fullsized":"2024-09-12T02:46:01.247794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:06.189681+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:46:06.189681+0000","last_peered":"2024-09-12T02:46:06.189681+0000","last_clean":"2024-09-12T02:46:06.189681+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:46:06.189681+0000","last_undegraded":"2024-09-12T02:46:06.189681+0000","last_fullsized":"2024-09-12T02:46:06.189681+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244430+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:46:01.244430+0000","last_peered":"2024-09-12T02:46:01.244430+0000","last_clean":"2024-09-12T02:46:01.244430+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:46:01.244430+0000","last_undegraded":"2024-09-12T02:46:01.244430+0000","last_fullsized":"2024-09-12T02:46:01.244430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247494+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:46:01.247494+0000","last_peered":"2024-09-12T02:46:01.247494+0000","last_clean":"2024-09-12T02:46:01.247494+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:46:01.247494+0000","last_undegraded":"2024-09-12T02:46:01.247494+0000","last_fullsized":"2024-09-12T02:46:01.247494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248263+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:46:01.248263+0000","last_peered":"2024-09-12T02:46:01.248263+0000","last_clean":"2024-09-12T02:46:01.248263+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:46:01.248263+0000","last_undegraded":"2024-09-12T02:46:01.248263+0000","last_fullsized":"2024-09-12T02:46:01.248263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245197+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:46:01.245197+0000","last_peered":"2024-09-12T02:46:01.245197+0000","last_clean":"2024-09-12T02:46:01.245197+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:46:01.245197+0000","last_undegraded":"2024-09-12T02:46:01.245197+0000","last_fullsized":"2024-09-12T02:46:01.245197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244361+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:46:01.244361+0000","last_peered":"2024-09-12T02:46:01.244361+0000","last_clean":"2024-09-12T02:46:01.244361+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:46:01.244361+0000","last_undegraded":"2024-09-12T02:46:01.244361+0000","last_fullsized":"2024-09-12T02:46:01.244361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250135+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:46:01.250135+0000","last_peered":"2024-09-12T02:46:01.250135+0000","last_clean":"2024-09-12T02:46:01.250135+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:46:01.250135+0000","last_undegraded":"2024-09-12T02:46:01.250135+0000","last_fullsized":"2024-09-12T02:46:01.250135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247854+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:46:01.247854+0000","last_peered":"2024-09-12T02:46:01.247854+0000","last_clean":"2024-09-12T02:46:01.247854+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:46:01.247854+0000","last_undegraded":"2024-09-12T02:46:01.247854+0000","last_fullsized":"2024-09-12T02:46:01.247854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254491+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:46:01.254491+0000","last_peered":"2024-09-12T02:46:01.254491+0000","last_clean":"2024-09-12T02:46:01.254491+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:46:01.254491+0000","last_undegraded":"2024-09-12T02:46:01.254491+0000","last_fullsized":"2024-09-12T02:46:01.254491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247259+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:46:01.247259+0000","last_peered":"2024-09-12T02:46:01.247259+0000","last_clean":"2024-09-12T02:46:01.247259+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:46:01.247259+0000","last_undegraded":"2024-09-12T02:46:01.247259+0000","last_fullsized":"2024-09-12T02:46:01.247259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244615+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:46:01.244615+0000","last_peered":"2024-09-12T02:46:01.244615+0000","last_clean":"2024-09-12T02:46:01.244615+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:46:01.244615+0000","last_undegraded":"2024-09-12T02:46:01.244615+0000","last_fullsized":"2024-09-12T02:46:01.244615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.946540+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:46:07.946540+0000","last_peered":"2024-09-12T02:46:07.946540+0000","last_clean":"2024-09-12T02:46:07.946540+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:46:07.946540+0000","last_undegraded":"2024-09-12T02:46:07.946540+0000","last_fullsized":"2024-09-12T02:46:07.946540+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57713,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":236,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":155045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645329,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1071103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234016,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659177,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":429609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888111,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051634,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":892928,"data_stored":429470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280568,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476797,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":471807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607618,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":933888,"data_stored":471587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40404,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704748,"statfs":{"total":95995035648,"available":95953661952,"internally_reserved":0,"allocated":913408,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":43430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":52241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:46:12.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:46:12.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:12 smithi033 ceph-mon[35092]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 -- 172.21.15.16:0/1156702515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4100aa0 msgr2=0x7fc2b410ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 --2- 172.21.15.16:0/1156702515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4100aa0 0x7fc2b410ea20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009ca0 tx=0x7fc2ac02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 -- 172.21.15.16:0/1156702515 shutdown_connections 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 --2- 172.21.15.16:0/1156702515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4100aa0 0x7fc2b410ea20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 --2- 172.21.15.16:0/1156702515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 0x7fc2b4100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 --2- 172.21.15.16:0/1156702515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4106b80 0x7fc2b4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 -- 172.21.15.16:0/1156702515 >> 172.21.15.16:0/1156702515 conn(0x7fc2b4077230 msgr2=0x7fc2b40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 -- 172.21.15.16:0/1156702515 shutdown_connections 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 -- 172.21.15.16:0/1156702515 wait complete. 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.318+0000 7fc2bc067640 1 Processor -- start 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 -- start start 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 0x7fc2b41012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41120a0 con 0x7fc2b4106b80 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4111f20 con 0x7fc2b4100aa0 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2bc067640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4112220 con 0x7fc2b4100100 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42724/0 (socket says 172.21.15.16:42724) 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 -- 172.21.15.16:0/2044202028 learned_addr learned my addr 172.21.15.16:0/2044202028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2b9602640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2b9e03640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 0x7fc2b41012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 -- 172.21.15.16:0/2044202028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 msgr2=0x7fc2b41012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 0x7fc2b41012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 -- 172.21.15.16:0/2044202028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 msgr2=0x7fc2b4101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 -- 172.21.15.16:0/2044202028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b41a69a0 con 0x7fc2b4106b80 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2b9602640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.319+0000 7fc2ba604640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005e30 tx=0x7fc2ac003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.320+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac031650 con 0x7fc2b4106b80 2024-09-12T02:46:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.320+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b41a6c30 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.320+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac0317f0 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.320+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac02fa40 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.320+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b41a7170 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.321+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b4100560 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.322+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc2ac02fc00 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.322+0000 7fc2aaffd640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 0x7fc2a00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.323+0000 7fc2b9e03640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 0x7fc2a00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.323+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc2ac0f0660 con 0x7fc2b4106b80 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.323+0000 7fc2b9e03640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 0x7fc2a00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009ae0 tx=0x7fc2b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.326+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc2ac0bd580 con 0x7fc2b4106b80 2024-09-12T02:46:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.497+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2b4105270 con 0x7fc2a00a2c10 2024-09-12T02:46:13.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.504+0000 7fc2aaffd640 1 -- 172.21.15.16:0/2044202028 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258772 (secure 0 0 0) 0x7fc2b4105270 con 0x7fc2a00a2c10 2024-09-12T02:46:13.506 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:13.507 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 msgr2=0x7fc2a00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 0x7fc2a00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009ae0 tx=0x7fc2b0005f90 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 msgr2=0x7fc2b4101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005e30 tx=0x7fc2ac003f10 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 shutdown_connections 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2a00a2c10 0x7fc2a00a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2b4106b80 0x7fc2b4101d60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 --2- 172.21.15.16:0/2044202028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2b4100100 0x7fc2b41012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 >> 172.21.15.16:0/2044202028 conn(0x7fc2b4077230 msgr2=0x7fc2b4104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 shutdown_connections 2024-09-12T02:46:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:13.508+0000 7fc2bc067640 1 -- 172.21.15.16:0/2044202028 wait complete. 2024-09-12T02:46:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:13 smithi016 ceph-mon[30925]: pgmap v286: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 KiB/s wr, 13 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:13 smithi033 ceph-mon[35092]: pgmap v286: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 KiB/s wr, 13 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:14.278 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-12T02:46:12.002473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648315,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":280,"num_read_kb":248,"num_write":419,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378428,"kb_used_data":12516,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563396,"statfs":{"total":1151940427776,"available":1151552917504,"internally_reserved":0,"allocated":12816384,"data_stored":7085469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":374453565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49520,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":101,"num_write":39,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.759471"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245601+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:46:01.245601+0000","last_peered":"2024-09-12T02:46:01.245601+0000","last_clean":"2024-09-12T02:46:01.245601+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:46:01.245601+0000","last_undegraded":"2024-09-12T02:46:01.245601+0000","last_fullsized":"2024-09-12T02:46:01.245601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247959+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:46:01.247959+0000","last_peered":"2024-09-12T02:46:01.247959+0000","last_clean":"2024-09-12T02:46:01.247959+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:46:01.247959+0000","last_undegraded":"2024-09-12T02:46:01.247959+0000","last_fullsized":"2024-09-12T02:46:01.247959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247397+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:46:01.247397+0000","last_peered":"2024-09-12T02:46:01.247397+0000","last_clean":"2024-09-12T02:46:01.247397+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:46:01.247397+0000","last_undegraded":"2024-09-12T02:46:01.247397+0000","last_fullsized":"2024-09-12T02:46:01.247397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247671+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:46:01.247671+0000","last_peered":"2024-09-12T02:46:01.247671+0000","last_clean":"2024-09-12T02:46:01.247671+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:46:01.247671+0000","last_undegraded":"2024-09-12T02:46:01.247671+0000","last_fullsized":"2024-09-12T02:46:01.247671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247675+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:46:01.247675+0000","last_peered":"2024-09-12T02:46:01.247675+0000","last_clean":"2024-09-12T02:46:01.247675+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:46:01.247675+0000","last_undegraded":"2024-09-12T02:46:01.247675+0000","last_fullsized":"2024-09-12T02:46:01.247675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245784+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:46:01.245784+0000","last_peered":"2024-09-12T02:46:01.245784+0000","last_clean":"2024-09-12T02:46:01.245784+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:46:01.245784+0000","last_undegraded":"2024-09-12T02:46:01.245784+0000","last_fullsized":"2024-09-12T02:46:01.245784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249851+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:46:01.249851+0000","last_peered":"2024-09-12T02:46:01.249851+0000","last_clean":"2024-09-12T02:46:01.249851+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:46:01.249851+0000","last_undegraded":"2024-09-12T02:46:01.249851+0000","last_fullsized":"2024-09-12T02:46:01.249851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244102+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:46:01.244102+0000","last_peered":"2024-09-12T02:46:01.244102+0000","last_clean":"2024-09-12T02:46:01.244102+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:46:01.244102+0000","last_undegraded":"2024-09-12T02:46:01.244102+0000","last_fullsized":"2024-09-12T02:46:01.244102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255221+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:46:01.255221+0000","last_peered":"2024-09-12T02:46:01.255221+0000","last_clean":"2024-09-12T02:46:01.255221+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:46:01.255221+0000","last_undegraded":"2024-09-12T02:46:01.255221+0000","last_fullsized":"2024-09-12T02:46:01.255221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255844+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:46:01.255844+0000","last_peered":"2024-09-12T02:46:01.255844+0000","last_clean":"2024-09-12T02:46:01.255844+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:46:01.255844+0000","last_undegraded":"2024-09-12T02:46:01.255844+0000","last_fullsized":"2024-09-12T02:46:01.255844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250813+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:46:01.250813+0000","last_peered":"2024-09-12T02:46:01.250813+0000","last_clean":"2024-09-12T02:46:01.250813+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:46:01.250813+0000","last_undegraded":"2024-09-12T02:46:01.250813+0000","last_fullsized":"2024-09-12T02:46:01.250813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247697+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:46:01.247697+0000","last_peered":"2024-09-12T02:46:01.247697+0000","last_clean":"2024-09-12T02:46:01.247697+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:46:01.247697+0000","last_undegraded":"2024-09-12T02:46:01.247697+0000","last_fullsized":"2024-09-12T02:46:01.247697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251093+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:46:01.251093+0000","last_peered":"2024-09-12T02:46:01.251093+0000","last_clean":"2024-09-12T02:46:01.251093+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:46:01.251093+0000","last_undegraded":"2024-09-12T02:46:01.251093+0000","last_fullsized":"2024-09-12T02:46:01.251093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247142+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:46:01.247142+0000","last_peered":"2024-09-12T02:46:01.247142+0000","last_clean":"2024-09-12T02:46:01.247142+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:46:01.247142+0000","last_undegraded":"2024-09-12T02:46:01.247142+0000","last_fullsized":"2024-09-12T02:46:01.247142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247412+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:46:01.247412+0000","last_peered":"2024-09-12T02:46:01.247412+0000","last_clean":"2024-09-12T02:46:01.247412+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:46:01.247412+0000","last_undegraded":"2024-09-12T02:46:01.247412+0000","last_fullsized":"2024-09-12T02:46:01.247412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254313+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:46:01.254313+0000","last_peered":"2024-09-12T02:46:01.254313+0000","last_clean":"2024-09-12T02:46:01.254313+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:46:01.254313+0000","last_undegraded":"2024-09-12T02:46:01.254313+0000","last_fullsized":"2024-09-12T02:46:01.254313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245720+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:46:01.245720+0000","last_peered":"2024-09-12T02:46:01.245720+0000","last_clean":"2024-09-12T02:46:01.245720+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:46:01.245720+0000","last_undegraded":"2024-09-12T02:46:01.245720+0000","last_fullsized":"2024-09-12T02:46:01.245720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250415+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:46:01.250415+0000","last_peered":"2024-09-12T02:46:01.250415+0000","last_clean":"2024-09-12T02:46:01.250415+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:46:01.250415+0000","last_undegraded":"2024-09-12T02:46:01.250415+0000","last_fullsized":"2024-09-12T02:46:01.250415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252979+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:46:01.252979+0000","last_peered":"2024-09-12T02:46:01.252979+0000","last_clean":"2024-09-12T02:46:01.252979+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:46:01.252979+0000","last_undegraded":"2024-09-12T02:46:01.252979+0000","last_fullsized":"2024-09-12T02:46:01.252979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245213+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:46:01.245213+0000","last_peered":"2024-09-12T02:46:01.245213+0000","last_clean":"2024-09-12T02:46:01.245213+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:46:01.245213+0000","last_undegraded":"2024-09-12T02:46:01.245213+0000","last_fullsized":"2024-09-12T02:46:01.245213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247833+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:46:01.247833+0000","last_peered":"2024-09-12T02:46:01.247833+0000","last_clean":"2024-09-12T02:46:01.247833+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:46:01.247833+0000","last_undegraded":"2024-09-12T02:46:01.247833+0000","last_fullsized":"2024-09-12T02:46:01.247833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254746+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:46:01.254746+0000","last_peered":"2024-09-12T02:46:01.254746+0000","last_clean":"2024-09-12T02:46:01.254746+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:46:01.254746+0000","last_undegraded":"2024-09-12T02:46:01.254746+0000","last_fullsized":"2024-09-12T02:46:01.254746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244169+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:46:01.244169+0000","last_peered":"2024-09-12T02:46:01.244169+0000","last_clean":"2024-09-12T02:46:01.244169+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:46:01.244169+0000","last_undegraded":"2024-09-12T02:46:01.244169+0000","last_fullsized":"2024-09-12T02:46:01.244169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248907+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:46:01.248907+0000","last_peered":"2024-09-12T02:46:01.248907+0000","last_clean":"2024-09-12T02:46:01.248907+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:46:01.248907+0000","last_undegraded":"2024-09-12T02:46:01.248907+0000","last_fullsized":"2024-09-12T02:46:01.248907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244381+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:46:01.244381+0000","last_peered":"2024-09-12T02:46:01.244381+0000","last_clean":"2024-09-12T02:46:01.244381+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:46:01.244381+0000","last_undegraded":"2024-09-12T02:46:01.244381+0000","last_fullsized":"2024-09-12T02:46:01.244381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253636+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:46:01.253636+0000","last_peered":"2024-09-12T02:46:01.253636+0000","last_clean":"2024-09-12T02:46:01.253636+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:46:01.253636+0000","last_undegraded":"2024-09-12T02:46:01.253636+0000","last_fullsized":"2024-09-12T02:46:01.253636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245512+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:46:01.245512+0000","last_peered":"2024-09-12T02:46:01.245512+0000","last_clean":"2024-09-12T02:46:01.245512+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:46:01.245512+0000","last_undegraded":"2024-09-12T02:46:01.245512+0000","last_fullsized":"2024-09-12T02:46:01.245512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247679+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:46:01.247679+0000","last_peered":"2024-09-12T02:46:01.247679+0000","last_clean":"2024-09-12T02:46:01.247679+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:46:01.247679+0000","last_undegraded":"2024-09-12T02:46:01.247679+0000","last_fullsized":"2024-09-12T02:46:01.247679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247302+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:46:01.247302+0000","last_peered":"2024-09-12T02:46:01.247302+0000","last_clean":"2024-09-12T02:46:01.247302+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:46:01.247302+0000","last_undegraded":"2024-09-12T02:46:01.247302+0000","last_fullsized":"2024-09-12T02:46:01.247302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252658+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:46:01.252658+0000","last_peered":"2024-09-12T02:46:01.252658+0000","last_clean":"2024-09-12T02:46:01.252658+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:46:01.252658+0000","last_undegraded":"2024-09-12T02:46:01.252658+0000","last_fullsized":"2024-09-12T02:46:01.252658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252938+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:46:01.252938+0000","last_peered":"2024-09-12T02:46:01.252938+0000","last_clean":"2024-09-12T02:46:01.252938+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:46:01.252938+0000","last_undegraded":"2024-09-12T02:46:01.252938+0000","last_fullsized":"2024-09-12T02:46:01.252938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245099+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:46:01.245099+0000","last_peered":"2024-09-12T02:46:01.245099+0000","last_clean":"2024-09-12T02:46:01.245099+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:46:01.245099+0000","last_undegraded":"2024-09-12T02:46:01.245099+0000","last_fullsized":"2024-09-12T02:46:01.245099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248039+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:46:01.248039+0000","last_peered":"2024-09-12T02:46:01.248039+0000","last_clean":"2024-09-12T02:46:01.248039+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:46:01.248039+0000","last_undegraded":"2024-09-12T02:46:01.248039+0000","last_fullsized":"2024-09-12T02:46:01.248039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.945648+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:46:07.945648+0000","last_peered":"2024-09-12T02:46:07.945648+0000","last_clean":"2024-09-12T02:46:07.945648+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:46:07.945648+0000","last_undegraded":"2024-09-12T02:46:07.945648+0000","last_fullsized":"2024-09-12T02:46:07.945648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248424+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:46:01.248424+0000","last_peered":"2024-09-12T02:46:01.248424+0000","last_clean":"2024-09-12T02:46:01.248424+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:46:01.248424+0000","last_undegraded":"2024-09-12T02:46:01.248424+0000","last_fullsized":"2024-09-12T02:46:01.248424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244158+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:46:01.244158+0000","last_peered":"2024-09-12T02:46:01.244158+0000","last_clean":"2024-09-12T02:46:01.244158+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:46:01.244158+0000","last_undegraded":"2024-09-12T02:46:01.244158+0000","last_fullsized":"2024-09-12T02:46:01.244158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248649+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:46:01.248649+0000","last_peered":"2024-09-12T02:46:01.248649+0000","last_clean":"2024-09-12T02:46:01.248649+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:46:01.248649+0000","last_undegraded":"2024-09-12T02:46:01.248649+0000","last_fullsized":"2024-09-12T02:46:01.248649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.944499+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:46:07.944499+0000","last_peered":"2024-09-12T02:46:07.944499+0000","last_clean":"2024-09-12T02:46:07.944499+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:46:07.944499+0000","last_undegraded":"2024-09-12T02:46:07.944499+0000","last_fullsized":"2024-09-12T02:46:07.944499+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255352+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:46:01.255352+0000","last_peered":"2024-09-12T02:46:01.255352+0000","last_clean":"2024-09-12T02:46:01.255352+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:46:01.255352+0000","last_undegraded":"2024-09-12T02:46:01.255352+0000","last_fullsized":"2024-09-12T02:46:01.255352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253009+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:46:01.253009+0000","last_peered":"2024-09-12T02:46:01.253009+0000","last_clean":"2024-09-12T02:46:01.253009+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:46:01.253009+0000","last_undegraded":"2024-09-12T02:46:01.253009+0000","last_fullsized":"2024-09-12T02:46:01.253009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254608+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:46:01.254608+0000","last_peered":"2024-09-12T02:46:01.254608+0000","last_clean":"2024-09-12T02:46:01.254608+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:46:01.254608+0000","last_undegraded":"2024-09-12T02:46:01.254608+0000","last_fullsized":"2024-09-12T02:46:01.254608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'27","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.947949+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:46:07.947949+0000","last_peered":"2024-09-12T02:46:07.947949+0000","last_clean":"2024-09-12T02:46:07.947949+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:46:07.947949+0000","last_undegraded":"2024-09-12T02:46:07.947949+0000","last_fullsized":"2024-09-12T02:46:07.947949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50112,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":62,"num_write":29,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248286+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:46:01.248286+0000","last_peered":"2024-09-12T02:46:01.248286+0000","last_clean":"2024-09-12T02:46:01.248286+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:46:01.248286+0000","last_undegraded":"2024-09-12T02:46:01.248286+0000","last_fullsized":"2024-09-12T02:46:01.248286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247592+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:46:01.247592+0000","last_peered":"2024-09-12T02:46:01.247592+0000","last_clean":"2024-09-12T02:46:01.247592+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:46:01.247592+0000","last_undegraded":"2024-09-12T02:46:01.247592+0000","last_fullsized":"2024-09-12T02:46:01.247592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250385+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:46:01.250385+0000","last_peered":"2024-09-12T02:46:01.250385+0000","last_clean":"2024-09-12T02:46:01.250385+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:46:01.250385+0000","last_undegraded":"2024-09-12T02:46:01.250385+0000","last_fullsized":"2024-09-12T02:46:01.250385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'6","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:08.953430+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:46:08.953430+0000","last_peered":"2024-09-12T02:46:08.953430+0000","last_clean":"2024-09-12T02:46:08.953430+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:46:08.953430+0000","last_undegraded":"2024-09-12T02:46:08.953430+0000","last_fullsized":"2024-09-12T02:46:08.953430+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249800+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:46:01.249800+0000","last_peered":"2024-09-12T02:46:01.249800+0000","last_clean":"2024-09-12T02:46:01.249800+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:46:01.249800+0000","last_undegraded":"2024-09-12T02:46:01.249800+0000","last_fullsized":"2024-09-12T02:46:01.249800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247633+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:46:01.247633+0000","last_peered":"2024-09-12T02:46:01.247633+0000","last_clean":"2024-09-12T02:46:01.247633+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:46:01.247633+0000","last_undegraded":"2024-09-12T02:46:01.247633+0000","last_fullsized":"2024-09-12T02:46:01.247633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245314+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:46:01.245314+0000","last_peered":"2024-09-12T02:46:01.245314+0000","last_clean":"2024-09-12T02:46:01.245314+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:46:01.245314+0000","last_undegraded":"2024-09-12T02:46:01.245314+0000","last_fullsized":"2024-09-12T02:46:01.245314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251385+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:46:01.251385+0000","last_peered":"2024-09-12T02:46:01.251385+0000","last_clean":"2024-09-12T02:46:01.251385+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:46:01.251385+0000","last_undegraded":"2024-09-12T02:46:01.251385+0000","last_fullsized":"2024-09-12T02:46:01.251385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250603+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:46:01.250603+0000","last_peered":"2024-09-12T02:46:01.250603+0000","last_clean":"2024-09-12T02:46:01.250603+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:46:01.250603+0000","last_undegraded":"2024-09-12T02:46:01.250603+0000","last_fullsized":"2024-09-12T02:46:01.250603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:08.946903+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:46:08.946903+0000","last_peered":"2024-09-12T02:46:08.946903+0000","last_clean":"2024-09-12T02:46:08.946903+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:46:08.946903+0000","last_undegraded":"2024-09-12T02:46:08.946903+0000","last_fullsized":"2024-09-12T02:46:08.946903+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250931+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:46:01.250931+0000","last_peered":"2024-09-12T02:46:01.250931+0000","last_clean":"2024-09-12T02:46:01.250931+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:46:01.250931+0000","last_undegraded":"2024-09-12T02:46:01.250931+0000","last_fullsized":"2024-09-12T02:46:01.250931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":334,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250005+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250005+0000","last_peered":"2024-09-12T02:46:01.250005+0000","last_clean":"2024-09-12T02:46:01.250005+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250005+0000","last_undegraded":"2024-09-12T02:46:01.250005+0000","last_fullsized":"2024-09-12T02:46:01.250005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244482+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:46:01.244482+0000","last_peered":"2024-09-12T02:46:01.244482+0000","last_clean":"2024-09-12T02:46:01.244482+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:46:01.244482+0000","last_undegraded":"2024-09-12T02:46:01.244482+0000","last_fullsized":"2024-09-12T02:46:01.244482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247535+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:46:01.247535+0000","last_peered":"2024-09-12T02:46:01.247535+0000","last_clean":"2024-09-12T02:46:01.247535+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:46:01.247535+0000","last_undegraded":"2024-09-12T02:46:01.247535+0000","last_fullsized":"2024-09-12T02:46:01.247535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:08.952911+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:46:08.952911+0000","last_peered":"2024-09-12T02:46:08.952911+0000","last_clean":"2024-09-12T02:46:08.952911+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:46:08.952911+0000","last_undegraded":"2024-09-12T02:46:08.952911+0000","last_fullsized":"2024-09-12T02:46:08.952911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249000+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:46:01.249000+0000","last_peered":"2024-09-12T02:46:01.249000+0000","last_clean":"2024-09-12T02:46:01.249000+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:46:01.249000+0000","last_undegraded":"2024-09-12T02:46:01.249000+0000","last_fullsized":"2024-09-12T02:46:01.249000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.947287+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:46:07.947287+0000","last_peered":"2024-09-12T02:46:07.947287+0000","last_clean":"2024-09-12T02:46:07.947287+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:46:07.947287+0000","last_undegraded":"2024-09-12T02:46:07.947287+0000","last_fullsized":"2024-09-12T02:46:07.947287+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251067+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:46:01.251067+0000","last_peered":"2024-09-12T02:46:01.251067+0000","last_clean":"2024-09-12T02:46:01.251067+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:46:01.251067+0000","last_undegraded":"2024-09-12T02:46:01.251067+0000","last_fullsized":"2024-09-12T02:46:01.251067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245843+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:46:01.245843+0000","last_peered":"2024-09-12T02:46:01.245843+0000","last_clean":"2024-09-12T02:46:01.245843+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:46:01.245843+0000","last_undegraded":"2024-09-12T02:46:01.245843+0000","last_fullsized":"2024-09-12T02:46:01.245843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247794+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:46:01.247794+0000","last_peered":"2024-09-12T02:46:01.247794+0000","last_clean":"2024-09-12T02:46:01.247794+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:46:01.247794+0000","last_undegraded":"2024-09-12T02:46:01.247794+0000","last_fullsized":"2024-09-12T02:46:01.247794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:06.189681+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:46:06.189681+0000","last_peered":"2024-09-12T02:46:06.189681+0000","last_clean":"2024-09-12T02:46:06.189681+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:46:06.189681+0000","last_undegraded":"2024-09-12T02:46:06.189681+0000","last_fullsized":"2024-09-12T02:46:06.189681+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244430+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:46:01.244430+0000","last_peered":"2024-09-12T02:46:01.244430+0000","last_clean":"2024-09-12T02:46:01.244430+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:46:01.244430+0000","last_undegraded":"2024-09-12T02:46:01.244430+0000","last_fullsized":"2024-09-12T02:46:01.244430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247494+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:46:01.247494+0000","last_peered":"2024-09-12T02:46:01.247494+0000","last_clean":"2024-09-12T02:46:01.247494+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:46:01.247494+0000","last_undegraded":"2024-09-12T02:46:01.247494+0000","last_fullsized":"2024-09-12T02:46:01.247494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248263+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:46:01.248263+0000","last_peered":"2024-09-12T02:46:01.248263+0000","last_clean":"2024-09-12T02:46:01.248263+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:46:01.248263+0000","last_undegraded":"2024-09-12T02:46:01.248263+0000","last_fullsized":"2024-09-12T02:46:01.248263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245197+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:46:01.245197+0000","last_peered":"2024-09-12T02:46:01.245197+0000","last_clean":"2024-09-12T02:46:01.245197+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:46:01.245197+0000","last_undegraded":"2024-09-12T02:46:01.245197+0000","last_fullsized":"2024-09-12T02:46:01.245197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244361+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:46:01.244361+0000","last_peered":"2024-09-12T02:46:01.244361+0000","last_clean":"2024-09-12T02:46:01.244361+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:46:01.244361+0000","last_undegraded":"2024-09-12T02:46:01.244361+0000","last_fullsized":"2024-09-12T02:46:01.244361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250135+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:46:01.250135+0000","last_peered":"2024-09-12T02:46:01.250135+0000","last_clean":"2024-09-12T02:46:01.250135+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:46:01.250135+0000","last_undegraded":"2024-09-12T02:46:01.250135+0000","last_fullsized":"2024-09-12T02:46:01.250135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247854+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:46:01.247854+0000","last_peered":"2024-09-12T02:46:01.247854+0000","last_clean":"2024-09-12T02:46:01.247854+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:46:01.247854+0000","last_undegraded":"2024-09-12T02:46:01.247854+0000","last_fullsized":"2024-09-12T02:46:01.247854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254491+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:46:01.254491+0000","last_peered":"2024-09-12T02:46:01.254491+0000","last_clean":"2024-09-12T02:46:01.254491+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:46:01.254491+0000","last_undegraded":"2024-09-12T02:46:01.254491+0000","last_fullsized":"2024-09-12T02:46:01.254491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247259+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:46:01.247259+0000","last_peered":"2024-09-12T02:46:01.247259+0000","last_clean":"2024-09-12T02:46:01.247259+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:46:01.247259+0000","last_undegraded":"2024-09-12T02:46:01.247259+0000","last_fullsized":"2024-09-12T02:46:01.247259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244615+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:46:01.244615+0000","last_peered":"2024-09-12T02:46:01.244615+0000","last_clean":"2024-09-12T02:46:01.244615+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:46:01.244615+0000","last_undegraded":"2024-09-12T02:46:01.244615+0000","last_fullsized":"2024-09-12T02:46:01.244615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:07.946540+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:46:07.946540+0000","last_peered":"2024-09-12T02:46:07.946540+0000","last_clean":"2024-09-12T02:46:07.946540+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:46:07.946540+0000","last_undegraded":"2024-09-12T02:46:07.946540+0000","last_fullsized":"2024-09-12T02:46:07.946540+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57713,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":154,"num_read_kb":139,"num_write":236,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":173139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645329,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1071103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234016,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659178,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":429609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888111,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051635,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":892928,"data_stored":429470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476798,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":480854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607619,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":946176,"data_stored":480634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40404,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704748,"statfs":{"total":95995035648,"available":95953661952,"internally_reserved":0,"allocated":913408,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":52257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":52477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":52241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:46:14.282 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-12T02:46:14.282 INFO:tasks.cephadm:Setting up client nodes... 2024-09-12T02:46:14.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-12T02:46:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:13 smithi184 ceph-mon[35321]: pgmap v286: 97 pgs: 97 active+clean; 633 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 KiB/s wr, 13 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 -- 172.21.15.16:0/3846186339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 msgr2=0x7f067c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 --2- 172.21.15.16:0/3846186339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c101c20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0674009ca0 tx=0x7f067402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 -- 172.21.15.16:0/3846186339 shutdown_connections 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 --2- 172.21.15.16:0/3846186339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c101c20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 --2- 172.21.15.16:0/3846186339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100dd0 0x7f067c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 --2- 172.21.15.16:0/3846186339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100490 0x7f067c100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 -- 172.21.15.16:0/3846186339 >> 172.21.15.16:0/3846186339 conn(0x7f067c0fc2a0 msgr2=0x7f067c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 -- 172.21.15.16:0/3846186339 shutdown_connections 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.994+0000 7f06813a7640 1 -- 172.21.15.16:0/3846186339 wait complete. 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 Processor -- start 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 -- start start 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 0x7f067c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100dd0 0x7f067c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10fa80 con 0x7f067c101770 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10f900 con 0x7f067c100dd0 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f06813a7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10fc00 con 0x7f067c100490 2024-09-12T02:46:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f067b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f067b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42744/0 (socket says 172.21.15.16:42744) 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f067b7fe640 1 -- 172.21.15.16:0/3512025523 learned_addr learned my addr 172.21.15.16:0/3512025523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.995+0000 7f067a7fc640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100dd0 0x7f067c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067affd640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 0x7f067c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 -- 172.21.15.16:0/3512025523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 msgr2=0x7f067c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 0x7f067c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 -- 172.21.15.16:0/3512025523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100dd0 msgr2=0x7f067c19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100dd0 0x7f067c19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 -- 172.21.15.16:0/3512025523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c1a1cd0 con 0x7f067c101770 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067affd640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 0x7f067c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f067b7fe640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0674005e30 tx=0x7f0674031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674003c30 con 0x7f067c101770 2024-09-12T02:46:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674003dd0 con 0x7f067c101770 2024-09-12T02:46:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c1a1f60 con 0x7f067c101770 2024-09-12T02:46:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067402fde0 con 0x7f067c101770 2024-09-12T02:46:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.996+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c1a2410 con 0x7f067c101770 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.997+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067c10b1e0 con 0x7f067c101770 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.998+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0674004050 con 0x7f067c101770 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.999+0000 7f0673fff640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 0x7f06680a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.999+0000 7f067affd640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 0x7f06680a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:14.999+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f06740f02e0 con 0x7f067c101770 2024-09-12T02:46:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.000+0000 7f067affd640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 0x7f06680a5100 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0664009b40 tx=0x7f0664005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:15.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.003+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f06740bd200 con 0x7f067c101770 2024-09-12T02:46:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.244+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f067c062e90 con 0x7f067c101770 2024-09-12T02:46:15.245 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:14 smithi016 ceph-mon[30925]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.246+0000 7f0673fff640 1 -- 172.21.15.16:0/3512025523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f06740bcb00 con 0x7f067c101770 2024-09-12T02:46:15.247 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-12T02:46:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD3VeJm+iueDhAAURjyd7ObF2haAj9l+DA6TQ== 2024-09-12T02:46:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:14 smithi033 ceph-mon[35092]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 msgr2=0x7f06680a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 0x7f06680a5100 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0664009b40 tx=0x7f0664005e50 comp rx=0 tx=0).stop 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 msgr2=0x7f067c1a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0674005e30 tx=0x7f0674031ee0 comp rx=0 tx=0).stop 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 shutdown_connections 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06680a2c40 0x7f06680a5100 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f067c101770 0x7f067c1a1660 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f067c100dd0 0x7f067c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 --2- 172.21.15.16:0/3512025523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f067c100490 0x7f067c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.248+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 >> 172.21.15.16:0/3512025523 conn(0x7f067c0fc2a0 msgr2=0x7f067c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.249+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 shutdown_connections 2024-09-12T02:46:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:15.249+0000 7f06813a7640 1 -- 172.21.15.16:0/3512025523 wait complete. 2024-09-12T02:46:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:14 smithi184 ceph-mon[35321]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:15.969 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:46:15.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-12T02:46:15.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-12T02:46:16.011 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-12T02:46:16.011 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-12T02:46:16.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-12T02:46:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:15 smithi033 ceph-mon[35092]: pgmap v287: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 13 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:15 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-12T02:46:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:15 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-12T02:46:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:15 smithi184 ceph-mon[35321]: pgmap v287: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 13 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:15 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-12T02:46:16.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:15 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-12T02:46:16.310 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:15 smithi016 ceph-mon[30925]: pgmap v287: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 KiB/s wr, 13 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-12T02:46:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:15 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-12T02:46:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:15 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3512025523' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-12T02:46:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 -- 172.21.15.16:0/811060167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc100a80 msgr2=0x7fa6bc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/811060167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc100a80 0x7fa6bc100ee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009930 tx=0x7fa6ac02a420 comp rx=0 tx=0).stop 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 -- 172.21.15.16:0/811060167 shutdown_connections 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/811060167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc101420 0x7fa6bc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/811060167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc100a80 0x7fa6bc100ee0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/811060167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 0x7fa6bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 -- 172.21.15.16:0/811060167 >> 172.21.15.16:0/811060167 conn(0x7fa6bc0fc7a0 msgr2=0x7fa6bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.767+0000 7fa6c42a2640 1 -- 172.21.15.16:0/811060167 shutdown_connections 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 -- 172.21.15.16:0/811060167 wait complete. 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 Processor -- start 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 -- start start 2024-09-12T02:46:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc100a80 0x7fa6bc10e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 0x7fa6bc10ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc111fe0 con 0x7fa6bc101420 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc111e60 con 0x7fa6bc100a80 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.768+0000 7fa6c42a2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc112160 con 0x7fa6bc106a80 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43024/0 (socket says 172.21.15.16:43024) 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 -- 172.21.15.16:0/2110382157 learned_addr learned my addr 172.21.15.16:0/2110382157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c203e640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc100a80 0x7fa6bc10e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c283f640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 0x7fa6bc10ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 -- 172.21.15.16:0/2110382157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 msgr2=0x7fa6bc10ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 0x7fa6bc10ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 -- 172.21.15.16:0/2110382157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc100a80 msgr2=0x7fa6bc10e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc100a80 0x7fa6bc10e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 -- 172.21.15.16:0/2110382157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6bc1a6900 con 0x7fa6bc101420 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c183d640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac02a940 tx=0x7fa6ac003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac03f070 con 0x7fa6bc101420 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6bc1a6b90 con 0x7fa6bc101420 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6ac03b460 con 0x7fa6bc101420 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac035030 con 0x7fa6bc101420 2024-09-12T02:46:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.769+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6bc1a7070 con 0x7fa6bc101420 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.771+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6bc100ee0 con 0x7fa6bc101420 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.772+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa6ac042070 con 0x7fa6bc101420 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.772+0000 7fa6b37fe640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 0x7fa6a80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.772+0000 7fa6c203e640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 0x7fa6a80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.772+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa6ac0ef9c0 con 0x7fa6bc101420 2024-09-12T02:46:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.773+0000 7fa6c203e640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 0x7fa6a80a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80058c0 tx=0x7fa6b800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.776+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa6ac0bc180 con 0x7fa6bc101420 2024-09-12T02:46:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:16.999+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa6bc108ff0 con 0x7fa6bc101420 2024-09-12T02:46:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.004+0000 7fa6b37fe640 1 -- 172.21.15.16:0/2110382157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263199 (secure 0 0 0) 0x7fa6ac0c0480 con 0x7fa6bc101420 2024-09-12T02:46:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 msgr2=0x7fa6a80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 0x7fa6a80a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80058c0 tx=0x7fa6b800a2d0 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 msgr2=0x7fa6bc10e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac02a940 tx=0x7fa6ac003c00 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 shutdown_connections 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6bc106a80 0x7fa6bc10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6a80a2c10 0x7fa6a80a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6bc101420 0x7fa6bc10e920 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 --2- 172.21.15.16:0/2110382157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6bc100a80 0x7fa6bc10e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 >> 172.21.15.16:0/2110382157 conn(0x7fa6bc0fc7a0 msgr2=0x7fa6bc0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 shutdown_connections 2024-09-12T02:46:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:17.008+0000 7fa6c42a2640 1 -- 172.21.15.16:0/2110382157 wait complete. 2024-09-12T02:46:17.689 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2564464416},{"type":"v1","addr":"172.21.15.16:6801","nonce":2564464416}]},"active_addr":"172.21.15.16:6801/2564464416","active_change":"2024-09-12T02:38:59.915343+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4118962901}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3465030604}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":56395524}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1304938134}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4178483604}]}]} 2024-09-12T02:46:17.698 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-12T02:46:17.698 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-12T02:46:17.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:46:17.957 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:18.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:17 smithi016 ceph-mon[30925]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-12T02:46:18.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:17 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2110382157' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-12T02:46:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:17 smithi033 ceph-mon[35092]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-12T02:46:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:17 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2110382157' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-12T02:46:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:17 smithi184 ceph-mon[35321]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-12T02:46:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:17 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2110382157' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-12T02:46:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- 172.21.15.16:0/263057203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 msgr2=0x7f884810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 --2- 172.21.15.16:0/263057203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884810eaa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8838009900 tx=0x7f883802a410 comp rx=0 tx=0).stop 2024-09-12T02:46:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- 172.21.15.16:0/263057203 shutdown_connections 2024-09-12T02:46:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 --2- 172.21.15.16:0/263057203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884810eaa0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 --2- 172.21.15.16:0/263057203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88481027a0 0x7f88481067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 --2- 172.21.15.16:0/263057203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8848101df0 0x7f88481021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- 172.21.15.16:0/263057203 >> 172.21.15.16:0/263057203 conn(0x7f88480fd8f0 msgr2=0x7f88480ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- 172.21.15.16:0/263057203 shutdown_connections 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- 172.21.15.16:0/263057203 wait complete. 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 Processor -- start 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.468+0000 7f884ee72640 1 -- start start 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8848101df0 0x7f884819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88481027a0 0x7f884819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8848111f80 con 0x7f8848106f20 2024-09-12T02:46:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8848111e00 con 0x7f88481027a0 2024-09-12T02:46:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884ee72640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8848112100 con 0x7f8848101df0 2024-09-12T02:46:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43044/0 (socket says 172.21.15.16:43044) 2024-09-12T02:46:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 -- 172.21.15.16:0/107513246 learned_addr learned my addr 172.21.15.16:0/107513246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f8847fff640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88481027a0 0x7f884819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884cc0e640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8848101df0 0x7f884819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 -- 172.21.15.16:0/107513246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8848101df0 msgr2=0x7f884819a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8848101df0 0x7f884819a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 -- 172.21.15.16:0/107513246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88481027a0 msgr2=0x7f884819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88481027a0 0x7f884819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 -- 172.21.15.16:0/107513246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884819f4e0 con 0x7f8848106f20 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.469+0000 7f884d40f640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8838005ab0 tx=0x7f88380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.470+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883803f030 con 0x7f8848106f20 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.470+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8838003ce0 con 0x7f8848106f20 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.470+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884819f770 con 0x7f8848106f20 2024-09-12T02:46:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.470+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838035070 con 0x7f8848106f20 2024-09-12T02:46:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.470+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88481a70b0 con 0x7f8848106f20 2024-09-12T02:46:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.471+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88481021d0 con 0x7f8848106f20 2024-09-12T02:46:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.472+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8838042030 con 0x7f8848106f20 2024-09-12T02:46:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.473+0000 7f8845ffb640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 0x7f88340a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.473+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f88380ef8f0 con 0x7f8848106f20 2024-09-12T02:46:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.473+0000 7f884cc0e640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 0x7f88340a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:18.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.474+0000 7f884cc0e640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 0x7f88340a5150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8840007ec0 tx=0x7f88400075b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.477+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f88380bc050 con 0x7f8848106f20 2024-09-12T02:46:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.654+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88481a7310 con 0x7f8848106f20 2024-09-12T02:46:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.655+0000 7f8845ffb640 1 -- 172.21.15.16:0/107513246 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f88380c0350 con 0x7f8848106f20 2024-09-12T02:46:18.656 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:18.656 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:46:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.657+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 msgr2=0x7f88340a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.657+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 0x7f88340a5150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8840007ec0 tx=0x7f88400075b0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.657+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 msgr2=0x7f884819ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.657+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8838005ab0 tx=0x7f88380042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.657+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 shutdown_connections 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88340a2c90 0x7f88340a5150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8848106f20 0x7f884819ee00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88481027a0 0x7f884819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 --2- 172.21.15.16:0/107513246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8848101df0 0x7f884819a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 >> 172.21.15.16:0/107513246 conn(0x7f88480fd8f0 msgr2=0x7f88480fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 shutdown_connections 2024-09-12T02:46:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:18.658+0000 7f884ee72640 1 -- 172.21.15.16:0/107513246 wait complete. 2024-09-12T02:46:19.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/107513246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/107513246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:19.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/107513246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:19.347 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-12T02:46:19.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:46:19.606 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:20.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:19 smithi016 ceph-mon[30925]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-09-12T02:46:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 -- 172.21.15.16:0/1590942769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228810bfc0 msgr2=0x7f228810e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 --2- 172.21.15.16:0/1590942769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228810bfc0 0x7f228810e460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2280009ca0 tx=0x7f228002a950 comp rx=0 tx=0).stop 2024-09-12T02:46:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 -- 172.21.15.16:0/1590942769 shutdown_connections 2024-09-12T02:46:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 --2- 172.21.15.16:0/1590942769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228810bfc0 0x7f228810e460 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 --2- 172.21.15.16:0/1590942769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22881034b0 0x7f228810b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 --2- 172.21.15.16:0/1590942769 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2288102b90 0x7f2288102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 -- 172.21.15.16:0/1590942769 >> 172.21.15.16:0/1590942769 conn(0x7f22880fc610 msgr2=0x7f22880fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 -- 172.21.15.16:0/1590942769 shutdown_connections 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.045+0000 7f228dd9e640 1 -- 172.21.15.16:0/1590942769 wait complete. 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 Processor -- start 2024-09-12T02:46:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 -- start start 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22881034b0 0x7f22881987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 0x7f228819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2288111df0 con 0x7f2288102b90 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2288111c70 con 0x7f22881034b0 2024-09-12T02:46:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228dd9e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2288111f70 con 0x7f228810bfc0 2024-09-12T02:46:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43066/0 (socket says 172.21.15.16:43066) 2024-09-12T02:46:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 -- 172.21.15.16:0/2212920467 learned_addr learned my addr 172.21.15.16:0/2212920467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f2287fff640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22881034b0 0x7f22881987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228d59d640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 0x7f228819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 -- 172.21.15.16:0/2212920467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 msgr2=0x7f228819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 0x7f228819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 -- 172.21.15.16:0/2212920467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22881034b0 msgr2=0x7f22881987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22881034b0 0x7f22881987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 -- 172.21.15.16:0/2212920467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228819d220 con 0x7f2288102b90 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228d59d640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 0x7f228819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.046+0000 7f228cd9c640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c890 tx=0x7f227c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.047+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c004220 con 0x7f2288102b90 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.047+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f227c0043c0 con 0x7f2288102b90 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.047+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228819d510 con 0x7f2288102b90 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.047+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0049e0 con 0x7f2288102b90 2024-09-12T02:46:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.047+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22881a4df0 con 0x7f2288102b90 2024-09-12T02:46:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.048+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2288102f70 con 0x7f2288102b90 2024-09-12T02:46:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.050+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f227c0079b0 con 0x7f2288102b90 2024-09-12T02:46:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.050+0000 7f2285ffb640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 0x7f22780a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.051+0000 7f2287fff640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 0x7f22780a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.051+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f227c0cf1c0 con 0x7f2288102b90 2024-09-12T02:46:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.051+0000 7f2287fff640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 0x7f22780a5150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f228810d270 tx=0x7f227400b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.054+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f227c014030 con 0x7f2288102b90 2024-09-12T02:46:20.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.232+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22881995d0 con 0x7f2288102b90 2024-09-12T02:46:20.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.233+0000 7f2285ffb640 1 -- 172.21.15.16:0/2212920467 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f227c09c050 con 0x7f2288102b90 2024-09-12T02:46:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:20.235 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:46:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 msgr2=0x7f22780a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 0x7f22780a5150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f228810d270 tx=0x7f227400b2e0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 msgr2=0x7f2288198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c890 tx=0x7f227c00cd50 comp rx=0 tx=0).stop 2024-09-12T02:46:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 shutdown_connections 2024-09-12T02:46:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228810bfc0 0x7f228819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22881034b0 0x7f22881987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22780a2c90 0x7f22780a5150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 --2- 172.21.15.16:0/2212920467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2288102b90 0x7f2288198270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.235+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 >> 172.21.15.16:0/2212920467 conn(0x7f22880fc610 msgr2=0x7f22880fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.236+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 shutdown_connections 2024-09-12T02:46:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:20.236+0000 7f228dd9e640 1 -- 172.21.15.16:0/2212920467 wait complete. 2024-09-12T02:46:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:19 smithi033 ceph-mon[35092]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-09-12T02:46:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:19 smithi184 ceph-mon[35321]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-09-12T02:46:20.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-12T02:46:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-12T02:46:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-12T02:46:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-12T02:46:20.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-12T02:46:20.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-12T02:46:20.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-12T02:46:20.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-12T02:46:20.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-12T02:46:20.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-12T02:46:20.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-12T02:46:20.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-12T02:46:21.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2212920467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2212920467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:21.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2212920467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.967 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:21.972 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:22.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:21 smithi016 ceph-mon[30925]: pgmap v290: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-09-12T02:46:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:21 smithi033 ceph-mon[35092]: pgmap v290: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-09-12T02:46:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:21 smithi184 ceph-mon[35321]: pgmap v290: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 -- 172.21.15.16:0/1929332293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829010e3c0 msgr2=0x7f8290110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 --2- 172.21.15.16:0/1929332293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829010e3c0 0x7f8290110810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8288009ca0 tx=0x7f828802a950 comp rx=0 tx=0).stop 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 -- 172.21.15.16:0/1929332293 shutdown_connections 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 --2- 172.21.15.16:0/1929332293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829010e3c0 0x7f8290110810 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 --2- 172.21.15.16:0/1929332293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8290074f40 0x7f829010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 --2- 172.21.15.16:0/1929332293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8290074620 0x7f8290074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.591+0000 7f829879b640 1 -- 172.21.15.16:0/1929332293 >> 172.21.15.16:0/1929332293 conn(0x7f82900fd8c0 msgr2=0x7f82900ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 -- 172.21.15.16:0/1929332293 shutdown_connections 2024-09-12T02:46:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 -- 172.21.15.16:0/1929332293 wait complete. 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 Processor -- start 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 -- start start 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8290074620 0x7f829019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 0x7f829019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.592+0000 7f829879b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829010e3c0 0x7f829019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 0x7f829019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f829879b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82901141a0 con 0x7f8290074f40 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 0x7f829019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43074/0 (socket says 172.21.15.16:43074) 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f829879b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8290114020 con 0x7f8290074620 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 -- 172.21.15.16:0/3305981110 learned_addr learned my addr 172.21.15.16:0/3305981110 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8290114320 con 0x7f829010e3c0 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8296537640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8290074620 0x7f829019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8296d38640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829010e3c0 0x7f829019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 -- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829010e3c0 msgr2=0x7f829019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829010e3c0 0x7f829019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 -- 172.21.15.16:0/3305981110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8290074620 msgr2=0x7f829019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8290074620 0x7f829019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829019f570 con 0x7f8290074f40 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8296d38640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829010e3c0 0x7f829019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f8295d36640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 0x7f829019ab00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f828000c920 tx=0x7f828000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280007c70 con 0x7f8290074f40 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829019f860 con 0x7f8290074f40 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8280007e10 con 0x7f8290074f40 2024-09-12T02:46:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280018890 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.593+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82901a7140 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.595+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82900fff30 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.596+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8280018a30 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f82877fe640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f827c0a2c90 0x7f827c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f8296537640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f827c0a2c90 0x7f827c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f82800cf710 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f82877fe640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] conn(0x7f827c0ac840 0x7f827c0aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f827c0af3b0 con 0x7f827c0ac840 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f82800cfb60 con 0x7f8290074f40 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f8296537640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f827c0a2c90 0x7f827c0a5150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f828c005fd0 tx=0x7f828c00b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.597+0000 7f8296d38640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] conn(0x7f827c0ac840 0x7f827c0aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.598+0000 7f8296d38640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] conn(0x7f827c0ac840 0x7f827c0aece0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.599+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== osd.8 v2:172.21.15.184:6816/1742401135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f827c0af3b0 con 0x7f827c0ac840 2024-09-12T02:46:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.621+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 --> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8290075cd0 con 0x7f827c0ac840 2024-09-12T02:46:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.622+0000 7f82877fe640 1 -- 172.21.15.16:0/3305981110 <== osd.8 v2:172.21.15.184:6816/1742401135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8290075cd0 con 0x7f827c0ac840 2024-09-12T02:46:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] conn(0x7f827c0ac840 msgr2=0x7f827c0aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] conn(0x7f827c0ac840 0x7f827c0aece0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f827c0a2c90 msgr2=0x7f827c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f827c0a2c90 0x7f827c0a5150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f828c005fd0 tx=0x7f828c00b590 comp rx=0 tx=0).stop 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 msgr2=0x7f829019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 --2- 172.21.15.16:0/3305981110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8290074f40 0x7f829019ab00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f828000c920 tx=0x7f828000cde0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f8296d38640 1 -- 172.21.15.16:0/3305981110 reap_dead start 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 shutdown_connections 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 >> 172.21.15.16:0/3305981110 conn(0x7f82900fd8c0 msgr2=0x7f82900fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 shutdown_connections 2024-09-12T02:46:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.623+0000 7f829879b640 1 -- 172.21.15.16:0/3305981110 wait complete. 2024-09-12T02:46:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 -- 172.21.15.16:0/354569931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 msgr2=0x7f294c110790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 --2- 172.21.15.16:0/354569931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c110790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2944009ca0 tx=0x7f294402a950 comp rx=0 tx=0).stop 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 -- 172.21.15.16:0/354569931 shutdown_connections 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 --2- 172.21.15.16:0/354569931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c110790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 --2- 172.21.15.16:0/354569931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f294c075110 0x7f294c10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 --2- 172.21.15.16:0/354569931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f294c0747f0 0x7f294c074bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 -- 172.21.15.16:0/354569931 >> 172.21.15.16:0/354569931 conn(0x7f294c0fd700 msgr2=0x7f294c0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 -- 172.21.15.16:0/354569931 shutdown_connections 2024-09-12T02:46:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.784+0000 7f29529a1640 1 -- 172.21.15.16:0/354569931 wait complete. 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 Processor -- start 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 -- start start 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f294c0747f0 0x7f294c10bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f294c075110 0x7f294c10c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c10c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1022c0 con 0x7f294c10e300 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f294c102140 con 0x7f294c075110 2024-09-12T02:46:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29529a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f294c102440 con 0x7f294c0747f0 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29521a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c10c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29521a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c10c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43090/0 (socket says 172.21.15.16:43090) 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f29521a0640 1 -- 172.21.15.16:0/2477074723 learned_addr learned my addr 172.21.15.16:0/2477074723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f295199f640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f294c0747f0 0x7f294c10bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.785+0000 7f295119e640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f294c075110 0x7f294c10c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 -- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f294c0747f0 msgr2=0x7f294c10bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f294c0747f0 0x7f294c10bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 -- 172.21.15.16:0/2477074723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f294c075110 msgr2=0x7f294c10c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f294c075110 0x7f294c10c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c1aacc0 con 0x7f294c10e300 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29521a0640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c10c920 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2944004af0 tx=0x7f294402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944034040 con 0x7f294c10e300 2024-09-12T02:46:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1aaf50 con 0x7f294c10e300 2024-09-12T02:46:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2944030920 con 0x7f294c10e300 2024-09-12T02:46:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29440047b0 con 0x7f294c10e300 2024-09-12T02:46:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.786+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c1ab3f0 con 0x7f294c10e300 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.787+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f294c062900 con 0x7f294c10e300 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.788+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2944037020 con 0x7f294c10e300 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f2942ffd640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29380a2c10 0x7f29380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f29440f0220 con 0x7f294c10e300 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f295199f640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29380a2c10 0x7f29380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f2942ffd640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] conn(0x7f29380ac7c0 0x7f29380aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2944043e50 con 0x7f29380ac7c0 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f29440f0670 con 0x7f294c10e300 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.789+0000 7f295199f640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29380a2c10 0x7f29380a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2948009bd0 tx=0x7f2948005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.790+0000 7f295119e640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] conn(0x7f29380ac7c0 0x7f29380aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.790+0000 7f295119e640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] conn(0x7f29380ac7c0 0x7f29380aec60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.791+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== osd.11 v2:172.21.15.184:6824/3211758447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2944043e50 con 0x7f29380ac7c0 2024-09-12T02:46:22.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.805+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 --> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f294c062d00 con 0x7f29380ac7c0 2024-09-12T02:46:22.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.805+0000 7f2942ffd640 1 -- 172.21.15.16:0/2477074723 <== osd.11 v2:172.21.15.184:6824/3211758447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f294c062d00 con 0x7f29380ac7c0 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.805+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] conn(0x7f29380ac7c0 msgr2=0x7f29380aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.805+0000 7f29529a1640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] conn(0x7f29380ac7c0 0x7f29380aec60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29380a2c10 msgr2=0x7f29380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29380a2c10 0x7f29380a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2948009bd0 tx=0x7f2948005f90 comp rx=0 tx=0).stop 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 msgr2=0x7f294c10c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 --2- 172.21.15.16:0/2477074723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294c10e300 0x7f294c10c920 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2944004af0 tx=0x7f294402f5b0 comp rx=0 tx=0).stop 2024-09-12T02:46:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29521a0640 1 -- 172.21.15.16:0/2477074723 reap_dead start 2024-09-12T02:46:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 shutdown_connections 2024-09-12T02:46:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 >> 172.21.15.16:0/2477074723 conn(0x7f294c0fd700 msgr2=0x7f294c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 shutdown_connections 2024-09-12T02:46:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:22.806+0000 7f29529a1640 1 -- 172.21.15.16:0/2477074723 wait complete. 2024-09-12T02:46:23.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 -- 172.21.15.16:0/2753030370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 msgr2=0x7f4a9410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2753030370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9410da40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009ca0 tx=0x7f4a8c02a950 comp rx=0 tx=0).stop 2024-09-12T02:46:23.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 -- 172.21.15.16:0/2753030370 shutdown_connections 2024-09-12T02:46:23.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2753030370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9410da40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2753030370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94102570 0x7f4a941029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2753030370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94100e20 0x7f4a94101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 -- 172.21.15.16:0/2753030370 >> 172.21.15.16:0/2753030370 conn(0x7f4a940fc7d0 msgr2=0x7f4a940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 -- 172.21.15.16:0/2753030370 shutdown_connections 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.399+0000 7f4a9c602640 1 -- 172.21.15.16:0/2753030370 wait complete. 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 Processor -- start 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 -- start start 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94100e20 0x7f4a9419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94102570 0x7f4a9419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94110f80 con 0x7f4a94102ef0 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94110e00 con 0x7f4a94100e20 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9c602640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94111100 con 0x7f4a94102570 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9ab9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9ab9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43100/0 (socket says 172.21.15.16:43100) 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9ab9f640 1 -- 172.21.15.16:0/2631421008 learned_addr learned my addr 172.21.15.16:0/2631421008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a9a39e640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94100e20 0x7f4a9419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.400+0000 7f4a99b9d640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94102570 0x7f4a9419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 -- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94102570 msgr2=0x7f4a9419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94102570 0x7f4a9419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 -- 172.21.15.16:0/2631421008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94100e20 msgr2=0x7f4a9419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94100e20 0x7f4a9419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9419f5b0 con 0x7f4a94102ef0 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a99b9d640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a94102570 0x7f4a9419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9a39e640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a94100e20 0x7f4a9419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9ab9f640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9419eed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c004af0 tx=0x7f4a8c02f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c034040 con 0x7f4a94102ef0 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a8c030920 con 0x7f4a94102ef0 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9419f840 con 0x7f4a94102ef0 2024-09-12T02:46:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c004820 con 0x7f4a94102ef0 2024-09-12T02:46:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.401+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a7180 con 0x7f4a94102ef0 2024-09-12T02:46:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.402+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a940fdf30 con 0x7f4a94102ef0 2024-09-12T02:46:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4a8c037020 con 0x7f4a94102ef0 2024-09-12T02:46:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4a800a2c90 0x7f4a800a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a9a39e640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4a800a2c90 0x7f4a800a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4a8c0f0300 con 0x7f4a94102ef0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] conn(0x7f4a800ac840 0x7f4a800aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a8c02c030 con 0x7f4a800ac840 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.404+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f4a8c0f0750 con 0x7f4a94102ef0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.405+0000 7f4a9a39e640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4a800a2c90 0x7f4a800a5150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000abe0 tx=0x7f4a9000a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.405+0000 7f4a99b9d640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] conn(0x7f4a800ac840 0x7f4a800aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.405+0000 7f4a99b9d640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] conn(0x7f4a800ac840 0x7f4a800aece0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.406+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== osd.5 v2:172.21.15.184:6808/746947299 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a8c02c030 con 0x7f4a800ac840 2024-09-12T02:46:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.420+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 --> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4a941021b0 con 0x7f4a800ac840 2024-09-12T02:46:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.420+0000 7f4a8b7fe640 1 -- 172.21.15.16:0/2631421008 <== osd.5 v2:172.21.15.184:6808/746947299 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4a941021b0 con 0x7f4a800ac840 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] conn(0x7f4a800ac840 msgr2=0x7f4a800aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] conn(0x7f4a800ac840 0x7f4a800aece0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4a800a2c90 msgr2=0x7f4a800a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4a800a2c90 0x7f4a800a5150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000abe0 tx=0x7f4a9000a5c0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 msgr2=0x7f4a9419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 --2- 172.21.15.16:0/2631421008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a94102ef0 0x7f4a9419eed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c004af0 tx=0x7f4a8c02f5b0 comp rx=0 tx=0).stop 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9ab9f640 1 -- 172.21.15.16:0/2631421008 reap_dead start 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 shutdown_connections 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 >> 172.21.15.16:0/2631421008 conn(0x7f4a940fc7d0 msgr2=0x7f4a9406a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 shutdown_connections 2024-09-12T02:46:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:23.421+0000 7f4a9c602640 1 -- 172.21.15.16:0/2631421008 wait complete. 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 -- 172.21.15.16:0/3356808806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4106f10 msgr2=0x7f74a410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/3356808806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4106f10 0x7f74a410ea90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f749c009ca0 tx=0x7f749c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 -- 172.21.15.16:0/3356808806 shutdown_connections 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/3356808806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4106f10 0x7f74a410ea90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/3356808806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74a4102790 0x7f74a41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/3356808806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74a4101de0 0x7f74a41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 -- 172.21.15.16:0/3356808806 >> 172.21.15.16:0/3356808806 conn(0x7f74a40fd8c0 msgr2=0x7f74a40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 -- 172.21.15.16:0/3356808806 shutdown_connections 2024-09-12T02:46:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.025+0000 7f74a8ae8640 1 -- 172.21.15.16:0/3356808806 wait complete. 2024-09-12T02:46:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 Processor -- start 2024-09-12T02:46:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 -- start start 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74a4101de0 0x7f74a419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 0x7f74a419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74a4106f10 0x7f74a419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4111f70 con 0x7f74a4102790 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4111df0 con 0x7f74a4101de0 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a8ae8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41120f0 con 0x7f74a4106f10 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 0x7f74a419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 0x7f74a419aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43112/0 (socket says 172.21.15.16:43112) 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a1d9b640 1 -- 172.21.15.16:0/296221366 learned_addr learned my addr 172.21.15.16:0/296221366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a259c640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74a4101de0 0x7f74a419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.026+0000 7f74a2d9d640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74a4106f10 0x7f74a419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 -- 172.21.15.16:0/296221366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74a4106f10 msgr2=0x7f74a419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74a4106f10 0x7f74a419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 -- 172.21.15.16:0/296221366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74a4101de0 msgr2=0x7f74a419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74a4101de0 0x7f74a419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a419f500 con 0x7f74a4102790 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a1d9b640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 0x7f74a419aa90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f749800d950 tx=0x7f749800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498017070 con 0x7f74a4102790 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a419f7f0 con 0x7f74a4102790 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749800de80 con 0x7f74a4102790 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749800f620 con 0x7f74a4102790 2024-09-12T02:46:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.027+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a41a70d0 con 0x7f74a4102790 2024-09-12T02:46:24.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.028+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f74a410e150 con 0x7f74a4102790 2024-09-12T02:46:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.029+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7498009c30 con 0x7f74a4102790 2024-09-12T02:46:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74937fe640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f748c0a2c10 0x7f748c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74a259c640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f748c0a2c10 0x7f748c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74980cee60 con 0x7f74a4102790 2024-09-12T02:46:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74937fe640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] conn(0x7f748c0ac7c0 0x7f748c0aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7498014ec0 con 0x7f748c0ac7c0 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f74980d1480 con 0x7f74a4102790 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74a2d9d640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] conn(0x7f748c0ac7c0 0x7f748c0aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.030+0000 7f74a259c640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f748c0a2c10 0x7f748c0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f749400abe0 tx=0x7f749400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.031+0000 7f74a2d9d640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] conn(0x7f748c0ac7c0 0x7f748c0aec60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.031+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== osd.6 v2:172.21.15.16:6818/2942447777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7498014ec0 con 0x7f748c0ac7c0 2024-09-12T02:46:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.045+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 --> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f74a41032c0 con 0x7f748c0ac7c0 2024-09-12T02:46:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74937fe640 1 -- 172.21.15.16:0/296221366 <== osd.6 v2:172.21.15.16:6818/2942447777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f74a41032c0 con 0x7f748c0ac7c0 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] conn(0x7f748c0ac7c0 msgr2=0x7f748c0aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] conn(0x7f748c0ac7c0 0x7f748c0aec60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f748c0a2c10 msgr2=0x7f748c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f748c0a2c10 0x7f748c0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f749400abe0 tx=0x7f749400a2d0 comp rx=0 tx=0).stop 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 msgr2=0x7f74a419aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 --2- 172.21.15.16:0/296221366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4102790 0x7f74a419aa90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f749800d950 tx=0x7f749800dd50 comp rx=0 tx=0).stop 2024-09-12T02:46:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a2d9d640 1 -- 172.21.15.16:0/296221366 reap_dead start 2024-09-12T02:46:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 shutdown_connections 2024-09-12T02:46:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 >> 172.21.15.16:0/296221366 conn(0x7f74a40fd8c0 msgr2=0x7f74a40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 shutdown_connections 2024-09-12T02:46:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.046+0000 7f74a8ae8640 1 -- 172.21.15.16:0/296221366 wait complete. 2024-09-12T02:46:24.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:23 smithi016 ceph-mon[30925]: pgmap v291: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:46:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:23 smithi033 ceph-mon[35092]: pgmap v291: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:46:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:23 smithi184 ceph-mon[35321]: pgmap v291: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:46:24.893 INFO:teuthology.orchestra.run.smithi016.stdout:146028888114 2024-09-12T02:46:24.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-12T02:46:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- 172.21.15.16:0/3525040134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 msgr2=0x7fa8a410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- 172.21.15.16:0/3525040134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a410bc00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa890009900 tx=0x7fa89002a410 comp rx=0 tx=0).stop 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- 172.21.15.16:0/3525040134 shutdown_connections 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- 172.21.15.16:0/3525040134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a410c2f0 0x7fa8a410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- 172.21.15.16:0/3525040134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a410bc00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- 172.21.15.16:0/3525040134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8a4102df0 0x7fa8a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- 172.21.15.16:0/3525040134 >> 172.21.15.16:0/3525040134 conn(0x7fa8a40fc7d0 msgr2=0x7fa8a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- 172.21.15.16:0/3525040134 shutdown_connections 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- 172.21.15.16:0/3525040134 wait complete. 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 Processor -- start 2024-09-12T02:46:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 -- start start 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a4102df0 0x7fa8a41a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.050+0000 7fa8aae16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a41a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8aae16640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8a410c2f0 0x7fa8a419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8aae16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a41120d0 con 0x7fa8a41037a0 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8aae16640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a4111f50 con 0x7fa8a4102df0 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8aae16640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a4112250 con 0x7fa8a410c2f0 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a41a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a41a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43138/0 (socket says 172.21.15.16:43138) 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 -- 172.21.15.16:0/2435738748 learned_addr learned my addr 172.21.15.16:0/2435738748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a8bb2640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a4102df0 0x7fa8a41a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 -- 172.21.15.16:0/2435738748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8a410c2f0 msgr2=0x7fa8a419a740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8a410c2f0 0x7fa8a419a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 -- 172.21.15.16:0/2435738748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a4102df0 msgr2=0x7fa8a41a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a4102df0 0x7fa8a41a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a419afb0 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a8bb2640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a4102df0 0x7fa8a41a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a3fff640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a41a0b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa890005940 tx=0x7fa8900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89003f030 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa890003ce0 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa890035070 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.051+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a419b240 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.052+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a41a7380 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.052+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa8a4102440 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.054+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa890042030 con 0x7fa8a41037a0 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.054+0000 7fa8a1ffb640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8940a2c90 0x7fa8940a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8900efe00 con 0x7fa8a41037a0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a8bb2640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8940a2c90 0x7fa8940a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a1ffb640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] conn(0x7fa8940ac840 0x7fa8940aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa890001d30 con 0x7fa8940ac840 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fa8900f0250 con 0x7fa8a41037a0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a93b3640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] conn(0x7fa8940ac840 0x7fa8940aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a93b3640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] conn(0x7fa8940ac840 0x7fa8940aece0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.055+0000 7fa8a8bb2640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8940a2c90 0x7fa8940a5150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa898005fd0 tx=0x7fa898005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.056+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== osd.9 v2:172.21.15.16:6826/3409413039 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa890001d30 con 0x7fa8940ac840 2024-09-12T02:46:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 --> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa8a406a790 con 0x7fa8940ac840 2024-09-12T02:46:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8a1ffb640 1 -- 172.21.15.16:0/2435738748 <== osd.9 v2:172.21.15.16:6826/3409413039 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa8a406a790 con 0x7fa8940ac840 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] conn(0x7fa8940ac840 msgr2=0x7fa8940aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] conn(0x7fa8940ac840 0x7fa8940aece0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8940a2c90 msgr2=0x7fa8940a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8940a2c90 0x7fa8940a5150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa898005fd0 tx=0x7fa898005ea0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 msgr2=0x7fa8a41a0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.070+0000 7fa8aae16640 1 --2- 172.21.15.16:0/2435738748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8a41037a0 0x7fa8a41a0b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa890005940 tx=0x7fa8900042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.071+0000 7fa8a93b3640 1 -- 172.21.15.16:0/2435738748 reap_dead start 2024-09-12T02:46:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.071+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 shutdown_connections 2024-09-12T02:46:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.071+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 >> 172.21.15.16:0/2435738748 conn(0x7fa8a40fc7d0 msgr2=0x7fa8a410d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.071+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 shutdown_connections 2024-09-12T02:46:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:25.071+0000 7fa8aae16640 1 -- 172.21.15.16:0/2435738748 wait complete. 2024-09-12T02:46:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:25 smithi016 ceph-mon[30925]: pgmap v292: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:46:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:25 smithi033 ceph-mon[35092]: pgmap v292: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:46:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:25 smithi184 ceph-mon[35321]: pgmap v292: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:46:26.462 INFO:teuthology.orchestra.run.smithi016.stdout:261993005087 2024-09-12T02:46:26.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-12T02:46:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 -- 172.21.15.16:0/3166715742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 msgr2=0x7f3300107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 --2- 172.21.15.16:0/3166715742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300107b90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009930 tx=0x7f32f402a420 comp rx=0 tx=0).stop 2024-09-12T02:46:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 -- 172.21.15.16:0/3166715742 shutdown_connections 2024-09-12T02:46:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 --2- 172.21.15.16:0/3166715742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33001082a0 0x7f330010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 --2- 172.21.15.16:0/3166715742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300107b90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 --2- 172.21.15.16:0/3166715742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3300102d30 0x7f3300103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 -- 172.21.15.16:0/3166715742 >> 172.21.15.16:0/3166715742 conn(0x7f33000fc7d0 msgr2=0x7f33000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 -- 172.21.15.16:0/3166715742 shutdown_connections 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 -- 172.21.15.16:0/3166715742 wait complete. 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.620+0000 7f3305906640 1 Processor -- start 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 -- start start 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3300102d30 0x7f33001985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:26.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33001082a0 0x7f330019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33001120c0 con 0x7f33001036e0 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3300111f40 con 0x7f33001082a0 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f3305906640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3300112240 con 0x7f3300102d30 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43346/0 (socket says 172.21.15.16:43346) 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 -- 172.21.15.16:0/2100111223 learned_addr learned my addr 172.21.15.16:0/2100111223 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32ff7fe640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33001082a0 0x7f330019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32feffd640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3300102d30 0x7f33001985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 -- 172.21.15.16:0/2100111223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3300102d30 msgr2=0x7f33001985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3300102d30 0x7f33001985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 -- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33001082a0 msgr2=0x7f330019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33001082a0 0x7f330019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.621+0000 7f32fe7fc640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330019d580 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f32fe7fc640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300198b10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f32f4005940 tx=0x7f32f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f403f030 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330019d810 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f4003ce0 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4035070 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.622+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33001a5150 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.623+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3300062980 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.625+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f32f4042030 con 0x7f33001036e0 2024-09-12T02:46:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f3304904640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32e80a2c10 0x7f32e80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f32feffd640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32e80a2c10 0x7f32e80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f32f40efe20 con 0x7f33001036e0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f3304904640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] conn(0x7f32e80ac7c0 0x7f32e80aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32e80af330 con 0x7f32e80ac7c0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f32f40f01d0 con 0x7f33001036e0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.626+0000 7f32ff7fe640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] conn(0x7f32e80ac7c0 0x7f32e80aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.627+0000 7f32feffd640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32e80a2c10 0x7f32e80a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f32f0009990 tx=0x7f32f0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.627+0000 7f32ff7fe640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] conn(0x7f32e80ac7c0 0x7f32e80aec60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.628+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== osd.7 v2:172.21.15.33:6816/1053213399 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f32e80af330 con 0x7f32e80ac7c0 2024-09-12T02:46:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.642+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 --> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3300062d80 con 0x7f32e80ac7c0 2024-09-12T02:46:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.642+0000 7f3304904640 1 -- 172.21.15.16:0/2100111223 <== osd.7 v2:172.21.15.33:6816/1053213399 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3300062d80 con 0x7f32e80ac7c0 2024-09-12T02:46:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] conn(0x7f32e80ac7c0 msgr2=0x7f32e80aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] conn(0x7f32e80ac7c0 0x7f32e80aec60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32e80a2c10 msgr2=0x7f32e80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32e80a2c10 0x7f32e80a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f32f0009990 tx=0x7f32f0009110 comp rx=0 tx=0).stop 2024-09-12T02:46:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 msgr2=0x7f3300198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 --2- 172.21.15.16:0/2100111223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33001036e0 0x7f3300198b10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f32f4005940 tx=0x7f32f40042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f32ff7fe640 1 -- 172.21.15.16:0/2100111223 reap_dead start 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 shutdown_connections 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 >> 172.21.15.16:0/2100111223 conn(0x7f33000fc7d0 msgr2=0x7f3300105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 shutdown_connections 2024-09-12T02:46:26.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:26.643+0000 7f3305906640 1 -- 172.21.15.16:0/2100111223 wait complete. 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 -- 172.21.15.16:0/896213756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308102d30 msgr2=0x7f6308103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 --2- 172.21.15.16:0/896213756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308102d30 0x7f6308103110 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009cd0 tx=0x7f62f802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 -- 172.21.15.16:0/896213756 shutdown_connections 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 --2- 172.21.15.16:0/896213756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63081082a0 0x7f630810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 --2- 172.21.15.16:0/896213756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63081036e0 0x7f6308107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 --2- 172.21.15.16:0/896213756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308102d30 0x7f6308103110 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 -- 172.21.15.16:0/896213756 >> 172.21.15.16:0/896213756 conn(0x7f63080fc7d0 msgr2=0x7f63080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 -- 172.21.15.16:0/896213756 shutdown_connections 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.243+0000 7f6310243640 1 -- 172.21.15.16:0/896213756 wait complete. 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 Processor -- start 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 -- start start 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6308102d30 0x7f63081985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 0x7f6308198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63081082a0 0x7f630819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f63081120a0 con 0x7f63081036e0 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6308111f20 con 0x7f6308102d30 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f6310243640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6308112220 con 0x7f63081082a0 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f630d7de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 0x7f6308198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f630d7de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 0x7f6308198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43134/0 (socket says 172.21.15.16:43134) 2024-09-12T02:46:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f630d7de640 1 -- 172.21.15.16:0/3272595571 learned_addr learned my addr 172.21.15.16:0/3272595571 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.244+0000 7f630dfdf640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6308102d30 0x7f63081985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630e7e0640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63081082a0 0x7f630819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 -- 172.21.15.16:0/3272595571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63081082a0 msgr2=0x7f630819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63081082a0 0x7f630819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 -- 172.21.15.16:0/3272595571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6308102d30 msgr2=0x7f63081985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6308102d30 0x7f63081985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630819d5a0 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630dfdf640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6308102d30 0x7f63081985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f630d7de640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 0x7f6308198b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00d740 tx=0x7f62fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc017070 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630819d890 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc00de50 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00f6b0 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.245+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63081a5170 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.246+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6308062980 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.247+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f62fc01a070 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f6306ffd640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f62f00a2c10 0x7f62f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f62fc0cefb0 con 0x7f63081036e0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f630dfdf640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f62f00a2c10 0x7f62f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f6306ffd640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] conn(0x7f62f00ac7c0 0x7f62f00aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62fc01de50 con 0x7f62f00ac7c0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f62fc0cf400 con 0x7f63081036e0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.248+0000 7f630e7e0640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] conn(0x7f62f00ac7c0 0x7f62f00aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.249+0000 7f630dfdf640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f62f00a2c10 0x7f62f00a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006010 tx=0x7f62f8005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.249+0000 7f630e7e0640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] conn(0x7f62f00ac7c0 0x7f62f00aec60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.249+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== osd.0 v2:172.21.15.16:6802/245735353 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62fc01de50 con 0x7f62f00ac7c0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.263+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 --> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6308062d80 con 0x7f62f00ac7c0 2024-09-12T02:46:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.263+0000 7f6306ffd640 1 -- 172.21.15.16:0/3272595571 <== osd.0 v2:172.21.15.16:6802/245735353 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6308062d80 con 0x7f62f00ac7c0 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] conn(0x7f62f00ac7c0 msgr2=0x7f62f00aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] conn(0x7f62f00ac7c0 0x7f62f00aec60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f62f00a2c10 msgr2=0x7f62f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f62f00a2c10 0x7f62f00a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006010 tx=0x7f62f8005ef0 comp rx=0 tx=0).stop 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 msgr2=0x7f6308198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 --2- 172.21.15.16:0/3272595571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081036e0 0x7f6308198b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00d740 tx=0x7f62fc00dc00 comp rx=0 tx=0).stop 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f630e7e0640 1 -- 172.21.15.16:0/3272595571 reap_dead start 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 shutdown_connections 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 >> 172.21.15.16:0/3272595571 conn(0x7f63080fc7d0 msgr2=0x7f6308105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 shutdown_connections 2024-09-12T02:46:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:24.264+0000 7f6310243640 1 -- 172.21.15.16:0/3272595571 wait complete. 2024-09-12T02:46:27.909 INFO:teuthology.orchestra.run.smithi016.stdout:34359738443 2024-09-12T02:46:27.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-12T02:46:28.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:27 smithi016 ceph-mon[30925]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:27 smithi033 ceph-mon[35092]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:27 smithi184 ceph-mon[35321]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:28.537 INFO:teuthology.orchestra.run.smithi016.stdout:176093659181 2024-09-12T02:46:28.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-12T02:46:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 -- 172.21.15.16:0/3429967855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401082a0 msgr2=0x7f7a4010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 --2- 172.21.15.16:0/3429967855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401082a0 0x7f7a4010ec40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009cd0 tx=0x7f7a3802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 -- 172.21.15.16:0/3429967855 shutdown_connections 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 --2- 172.21.15.16:0/3429967855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401082a0 0x7f7a4010ec40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 --2- 172.21.15.16:0/3429967855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a401036e0 0x7f7a40107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 --2- 172.21.15.16:0/3429967855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a40102d30 0x7f7a40103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 -- 172.21.15.16:0/3429967855 >> 172.21.15.16:0/3429967855 conn(0x7f7a400fc7d0 msgr2=0x7f7a400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 -- 172.21.15.16:0/3429967855 shutdown_connections 2024-09-12T02:46:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 -- 172.21.15.16:0/3429967855 wait complete. 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.695+0000 7f7a4620f640 1 Processor -- start 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 -- start start 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a40102d30 0x7f7a401984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 0x7f7a40198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a401082a0 0x7f7a4019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401120d0 con 0x7f7a401036e0 2024-09-12T02:46:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a40111f50 con 0x7f7a401082a0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a4620f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a40112250 con 0x7f7a40102d30 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 0x7f7a40198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 0x7f7a40198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43370/0 (socket says 172.21.15.16:43370) 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a3effd640 1 -- 172.21.15.16:0/1850706096 learned_addr learned my addr 172.21.15.16:0/1850706096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a3ffff640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a401082a0 0x7f7a4019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.696+0000 7f7a3f7fe640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a40102d30 0x7f7a401984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 -- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a40102d30 msgr2=0x7f7a401984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a40102d30 0x7f7a401984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 -- 172.21.15.16:0/1850706096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a401082a0 msgr2=0x7f7a4019cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a401082a0 0x7f7a4019cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4019d470 con 0x7f7a401036e0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3effd640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 0x7f7a40198a00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800d740 tx=0x7f7a2800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28017070 con 0x7f7a401036e0 2024-09-12T02:46:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4019d760 con 0x7f7a401036e0 2024-09-12T02:46:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a2800de50 con 0x7f7a401036e0 2024-09-12T02:46:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a401a5040 con 0x7f7a401036e0 2024-09-12T02:46:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.697+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2800f6b0 con 0x7f7a401036e0 2024-09-12T02:46:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.698+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a400fedb0 con 0x7f7a401036e0 2024-09-12T02:46:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.700+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7a2801a070 con 0x7f7a401036e0 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.700+0000 7f7a3cff9640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a2c0a2c90 0x7f7a2c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7a280cf100 con 0x7f7a401036e0 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3f7fe640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a2c0a2c90 0x7f7a2c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3cff9640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] conn(0x7f7a2c0ac840 0x7f7a2c0aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a28009dd0 con 0x7f7a2c0ac840 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f7a280cf550 con 0x7f7a401036e0 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3f7fe640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a2c0a2c90 0x7f7a2c0a5150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a30005e10 tx=0x7f7a30005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.701+0000 7f7a3ffff640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] conn(0x7f7a2c0ac840 0x7f7a2c0aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.702+0000 7f7a3ffff640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] conn(0x7f7a2c0ac840 0x7f7a2c0aece0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.702+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== osd.2 v2:172.21.15.184:6800/867017914 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7a28009dd0 con 0x7f7a2c0ac840 2024-09-12T02:46:28.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.719+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 --> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7a4006a850 con 0x7f7a2c0ac840 2024-09-12T02:46:28.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.720+0000 7f7a3cff9640 1 -- 172.21.15.16:0/1850706096 <== osd.2 v2:172.21.15.184:6800/867017914 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7a4006a850 con 0x7f7a2c0ac840 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.720+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] conn(0x7f7a2c0ac840 msgr2=0x7f7a2c0aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.720+0000 7f7a4620f640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] conn(0x7f7a2c0ac840 0x7f7a2c0aece0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a2c0a2c90 msgr2=0x7f7a2c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a2c0a2c90 0x7f7a2c0a5150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a30005e10 tx=0x7f7a30005da0 comp rx=0 tx=0).stop 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 msgr2=0x7f7a40198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 --2- 172.21.15.16:0/1850706096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a401036e0 0x7f7a40198a00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800d740 tx=0x7f7a2800dc00 comp rx=0 tx=0).stop 2024-09-12T02:46:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a3ffff640 1 -- 172.21.15.16:0/1850706096 reap_dead start 2024-09-12T02:46:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 shutdown_connections 2024-09-12T02:46:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 >> 172.21.15.16:0/1850706096 conn(0x7f7a400fc7d0 msgr2=0x7f7a400fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 shutdown_connections 2024-09-12T02:46:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:28.721+0000 7f7a4620f640 1 -- 172.21.15.16:0/1850706096 wait complete. 2024-09-12T02:46:29.082 INFO:teuthology.orchestra.run.smithi016.stdout:309237645332 2024-09-12T02:46:29.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-12T02:46:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 -- 172.21.15.16:0/3152569215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102570 msgr2=0x7f77c41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 --2- 172.21.15.16:0/3152569215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102570 0x7f77c41029b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009900 tx=0x7f77b402a480 comp rx=0 tx=0).stop 2024-09-12T02:46:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 -- 172.21.15.16:0/3152569215 shutdown_connections 2024-09-12T02:46:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 --2- 172.21.15.16:0/3152569215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4102ef0 0x7f77c410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 --2- 172.21.15.16:0/3152569215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102570 0x7f77c41029b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 --2- 172.21.15.16:0/3152569215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77c4100e20 0x7f77c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 -- 172.21.15.16:0/3152569215 >> 172.21.15.16:0/3152569215 conn(0x7f77c40fc7d0 msgr2=0x7f77c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.440+0000 7f77cc24c640 1 -- 172.21.15.16:0/3152569215 shutdown_connections 2024-09-12T02:46:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.441+0000 7f77cc24c640 1 -- 172.21.15.16:0/3152569215 wait complete. 2024-09-12T02:46:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.441+0000 7f77cc24c640 1 Processor -- start 2024-09-12T02:46:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.441+0000 7f77cc24c640 1 -- start start 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4100e20 0x7f77c41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77c4102570 0x7f77c4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 0x7f77c419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4110f80 con 0x7f77c4102ef0 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4110e00 con 0x7f77c4102570 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77cc24c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4111100 con 0x7f77c4100e20 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 0x7f77c419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 0x7f77c419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43390/0 (socket says 172.21.15.16:43390) 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 -- 172.21.15.16:0/853357188 learned_addr learned my addr 172.21.15.16:0/853357188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77c97e7640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77c4102570 0x7f77c4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77c9fe8640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4100e20 0x7f77c41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 -- 172.21.15.16:0/853357188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4100e20 msgr2=0x7f77c41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4100e20 0x7f77c41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 -- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77c4102570 msgr2=0x7f77c4198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77c4102570 0x7f77c4198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77ca7e9640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c419d490 con 0x7f77c4102ef0 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.442+0000 7f77c9fe8640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c4100e20 0x7f77c41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77ca7e9640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 0x7f77c419cdb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00cce0 tx=0x7f77bc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00f450 con 0x7f77c4102ef0 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77bc00fad0 con 0x7f77c4102ef0 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c419d780 con 0x7f77c4102ef0 2024-09-12T02:46:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc018c10 con 0x7f77c4102ef0 2024-09-12T02:46:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.443+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c41a5060 con 0x7f77c4102ef0 2024-09-12T02:46:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.444+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77c4062980 con 0x7f77c4102ef0 2024-09-12T02:46:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.445+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f77bc00f5f0 con 0x7f77c4102ef0 2024-09-12T02:46:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.446+0000 7f77baffd640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77b00a2c90 0x7f77b00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.446+0000 7f77c9fe8640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77b00a2c90 0x7f77b00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.446+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f77bc0cf7f0 con 0x7f77c4102ef0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77baffd640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] conn(0x7f77b00ac840 0x7f77b00aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77bc01ee50 con 0x7f77b00ac840 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f77bc0d6050 con 0x7f77c4102ef0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77c9fe8640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77b00a2c90 0x7f77b00a5150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f77c0005d00 tx=0x7f77c0009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77c97e7640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] conn(0x7f77b00ac840 0x7f77b00aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.447+0000 7f77c97e7640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] conn(0x7f77b00ac840 0x7f77b00aece0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.448+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== osd.4 v2:172.21.15.33:6808/3318678506 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77bc01ee50 con 0x7f77b00ac840 2024-09-12T02:46:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.463+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 --> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f77c4062db0 con 0x7f77b00ac840 2024-09-12T02:46:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77baffd640 1 -- 172.21.15.16:0/853357188 <== osd.4 v2:172.21.15.33:6808/3318678506 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f77c4062db0 con 0x7f77b00ac840 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] conn(0x7f77b00ac840 msgr2=0x7f77b00aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] conn(0x7f77b00ac840 0x7f77b00aece0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77b00a2c90 msgr2=0x7f77b00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77b00a2c90 0x7f77b00a5150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f77c0005d00 tx=0x7f77c0009550 comp rx=0 tx=0).stop 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 msgr2=0x7f77c419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 --2- 172.21.15.16:0/853357188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c4102ef0 0x7f77c419cdb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00cce0 tx=0x7f77bc00a620 comp rx=0 tx=0).stop 2024-09-12T02:46:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77ca7e9640 1 -- 172.21.15.16:0/853357188 reap_dead start 2024-09-12T02:46:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 shutdown_connections 2024-09-12T02:46:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 >> 172.21.15.16:0/853357188 conn(0x7f77c40fc7d0 msgr2=0x7f77c406a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 shutdown_connections 2024-09-12T02:46:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:29.464+0000 7f77cc24c640 1 -- 172.21.15.16:0/853357188 wait complete. 2024-09-12T02:46:29.898 INFO:teuthology.orchestra.run.smithi016.stdout:201863462953 2024-09-12T02:46:29.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-12T02:46:30.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 -- 172.21.15.16:0/2735829512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794102ef0 msgr2=0x7ff79410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 --2- 172.21.15.16:0/2735829512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794102ef0 0x7ff79410da40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009ca0 tx=0x7ff78c02a950 comp rx=0 tx=0).stop 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 -- 172.21.15.16:0/2735829512 shutdown_connections 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 --2- 172.21.15.16:0/2735829512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794102ef0 0x7ff79410da40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 --2- 172.21.15.16:0/2735829512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff794102570 0x7ff7941029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 --2- 172.21.15.16:0/2735829512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794100e20 0x7ff794101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 -- 172.21.15.16:0/2735829512 >> 172.21.15.16:0/2735829512 conn(0x7ff7940fc7d0 msgr2=0x7ff7940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 -- 172.21.15.16:0/2735829512 shutdown_connections 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.054+0000 7ff79c367640 1 -- 172.21.15.16:0/2735829512 wait complete. 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 Processor -- start 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 -- start start 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 0x7ff794198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff794102570 0x7ff7941989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794102ef0 0x7ff79419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff794110f80 con 0x7ff794100e20 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff794110e00 con 0x7ff794102570 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79c367640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff794111100 con 0x7ff794102ef0 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 0x7ff794198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 0x7ff794198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43414/0 (socket says 172.21.15.16:43414) 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 -- 172.21.15.16:0/2746998846 learned_addr learned my addr 172.21.15.16:0/2746998846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a904640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794102ef0 0x7ff79419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff799902640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff794102570 0x7ff7941989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 -- 172.21.15.16:0/2746998846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794102ef0 msgr2=0x7ff79419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794102ef0 0x7ff79419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 -- 172.21.15.16:0/2746998846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff794102570 msgr2=0x7ff7941989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff794102570 0x7ff7941989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a103640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79419d420 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.055+0000 7ff79a904640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff794102ef0 0x7ff79419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff79a103640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 0x7ff794198470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d770 tx=0x7ff79000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790017070 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff790005500 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79419d710 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790004e00 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.056+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7941a4ff0 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.057+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff794062980 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.059+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff790004fa0 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.059+0000 7ff78b7fe640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff7800a2c10 0x7ff7800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff7900cf0d0 con 0x7ff794100e20 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff799902640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff7800a2c10 0x7ff7800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff78b7fe640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] conn(0x7ff7800ac7c0 0x7ff7800aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff790001d30 con 0x7ff7800ac7c0 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7ff7900cf510 con 0x7ff794100e20 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff79a904640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] conn(0x7ff7800ac7c0 0x7ff7800aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff799902640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff7800a2c10 0x7ff7800a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff794105220 tx=0x7ff78400b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.060+0000 7ff79a904640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] conn(0x7ff7800ac7c0 0x7ff7800aec60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.061+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== osd.3 v2:172.21.15.16:6810/3579605426 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff790001d30 con 0x7ff7800ac7c0 2024-09-12T02:46:30.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.075+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 --> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff794062e00 con 0x7ff7800ac7c0 2024-09-12T02:46:30.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.075+0000 7ff78b7fe640 1 -- 172.21.15.16:0/2746998846 <== osd.3 v2:172.21.15.16:6810/3579605426 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff794062e00 con 0x7ff7800ac7c0 2024-09-12T02:46:30.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.075+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] conn(0x7ff7800ac7c0 msgr2=0x7ff7800aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] conn(0x7ff7800ac7c0 0x7ff7800aec60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:30.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff7800a2c10 msgr2=0x7ff7800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff7800a2c10 0x7ff7800a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff794105220 tx=0x7ff78400b590 comp rx=0 tx=0).stop 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 msgr2=0x7ff794198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 --2- 172.21.15.16:0/2746998846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff794100e20 0x7ff794198470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d770 tx=0x7ff79000dc30 comp rx=0 tx=0).stop 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79a904640 1 -- 172.21.15.16:0/2746998846 reap_dead start 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 shutdown_connections 2024-09-12T02:46:30.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 >> 172.21.15.16:0/2746998846 conn(0x7ff7940fc7d0 msgr2=0x7ff79406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:30.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 shutdown_connections 2024-09-12T02:46:30.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:30.076+0000 7ff79c367640 1 -- 172.21.15.16:0/2746998846 wait complete. 2024-09-12T02:46:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:29 smithi016 ceph-mon[30925]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-12T02:46:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:29 smithi033 ceph-mon[35092]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-12T02:46:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:29 smithi184 ceph-mon[35321]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-12T02:46:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:30.981 INFO:teuthology.orchestra.run.smithi016.stdout:231928234019 2024-09-12T02:46:30.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-12T02:46:31.565 INFO:teuthology.orchestra.run.smithi016.stdout:68719476802 2024-09-12T02:46:31.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-12T02:46:32.186 INFO:teuthology.orchestra.run.smithi016.stdout:124554051639 2024-09-12T02:46:32.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-12T02:46:32.210 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:31 smithi016 ceph-mon[30925]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:32.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:31 smithi033 ceph-mon[35092]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:31 smithi184 ceph-mon[35321]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-12T02:46:32.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 -- 172.21.15.16:0/831023718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118106a80 msgr2=0x7f8118106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 --2- 172.21.15.16:0/831023718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118106a80 0x7f8118106e60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8108009900 tx=0x7f810802a410 comp rx=0 tx=0).stop 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 -- 172.21.15.16:0/831023718 shutdown_connections 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 --2- 172.21.15.16:0/831023718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8118101420 0x7f811810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 --2- 172.21.15.16:0/831023718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8118100a80 0x7f8118100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 --2- 172.21.15.16:0/831023718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118106a80 0x7f8118106e60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 -- 172.21.15.16:0/831023718 >> 172.21.15.16:0/831023718 conn(0x7f81180fc7a0 msgr2=0x7f81180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 -- 172.21.15.16:0/831023718 shutdown_connections 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.473+0000 7f811eefc640 1 -- 172.21.15.16:0/831023718 wait complete. 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 Processor -- start 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 -- start start 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8118100a80 0x7f811819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 0x7f811819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:32.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8118106a80 0x7f811810aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8118111fa0 con 0x7f8118101420 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8118111e20 con 0x7f8118106a80 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f811eefc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8118112120 con 0x7f8118100a80 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f8117fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 0x7f811819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f8117fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 0x7f811819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43436/0 (socket says 172.21.15.16:43436) 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.474+0000 7f8117fff640 1 -- 172.21.15.16:0/887077875 learned_addr learned my addr 172.21.15.16:0/887077875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f811cc98640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8118100a80 0x7f811819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f811d499640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8118106a80 0x7f811810aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 -- 172.21.15.16:0/887077875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8118100a80 msgr2=0x7f811819f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8118100a80 0x7f811819f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 -- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8118106a80 msgr2=0x7f811810aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8118106a80 0x7f811810aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811810b3a0 con 0x7f8118101420 2024-09-12T02:46:32.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8117fff640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 0x7f811819f6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f810000c890 tx=0x7f810000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:32.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100004220 con 0x7f8118101420 2024-09-12T02:46:32.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81000043c0 con 0x7f8118101420 2024-09-12T02:46:32.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.475+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811810b690 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.476+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81000049e0 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.476+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811819ad60 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.476+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f811810c800 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.477+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f81000079b0 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f8115ffb640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f81040a2c90 0x7f81040a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f81000cf290 con 0x7f8118101420 2024-09-12T02:46:32.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f811cc98640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f81040a2c90 0x7f81040a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f8115ffb640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] conn(0x7f81040ac840 0x7f81040aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8100012ec0 con 0x7f81040ac840 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.478+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f81000d6050 con 0x7f8118101420 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.479+0000 7f811cc98640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f81040a2c90 0x7f81040a5150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81080096c0 tx=0x7f8108031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.479+0000 7f811d499640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] conn(0x7f81040ac840 0x7f81040aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.479+0000 7f811d499640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] conn(0x7f81040ac840 0x7f81040aece0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.480+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== osd.10 v2:172.21.15.33:6824/3476378344 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8100012ec0 con 0x7f81040ac840 2024-09-12T02:46:32.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.495+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 --> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81181021c0 con 0x7f81040ac840 2024-09-12T02:46:32.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.495+0000 7f8115ffb640 1 -- 172.21.15.16:0/887077875 <== osd.10 v2:172.21.15.33:6824/3476378344 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f81181021c0 con 0x7f81040ac840 2024-09-12T02:46:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] conn(0x7f81040ac840 msgr2=0x7f81040aece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] conn(0x7f81040ac840 0x7f81040aece0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f81040a2c90 msgr2=0x7f81040a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f81040a2c90 0x7f81040a5150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81080096c0 tx=0x7f8108031490 comp rx=0 tx=0).stop 2024-09-12T02:46:32.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 msgr2=0x7f811819f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 --2- 172.21.15.16:0/887077875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8118101420 0x7f811819f6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f810000c890 tx=0x7f810000cd50 comp rx=0 tx=0).stop 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811d499640 1 -- 172.21.15.16:0/887077875 reap_dead start 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 shutdown_connections 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 >> 172.21.15.16:0/887077875 conn(0x7f81180fc7a0 msgr2=0x7f811810bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 shutdown_connections 2024-09-12T02:46:32.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:32.496+0000 7f811eefc640 1 -- 172.21.15.16:0/887077875 wait complete. 2024-09-12T02:46:33.247 INFO:teuthology.orchestra.run.smithi016.stdout:94489280573 2024-09-12T02:46:33.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-12T02:46:33.892 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:33.893 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:33.893 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:33.907 INFO:teuthology.orchestra.run.smithi016.stdout:279172874266 2024-09-12T02:46:33.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-12T02:46:34.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 -- 172.21.15.16:0/1253249590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4701082a0 msgr2=0x7fb47010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 --2- 172.21.15.16:0/1253249590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4701082a0 0x7fb47010ec40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb468009cd0 tx=0x7fb46802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 -- 172.21.15.16:0/1253249590 shutdown_connections 2024-09-12T02:46:34.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 --2- 172.21.15.16:0/1253249590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4701082a0 0x7fb47010ec40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 --2- 172.21.15.16:0/1253249590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701036e0 0x7fb470107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 --2- 172.21.15.16:0/1253249590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb470102d30 0x7fb470103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 -- 172.21.15.16:0/1253249590 >> 172.21.15.16:0/1253249590 conn(0x7fb4700fc7d0 msgr2=0x7fb4700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 -- 172.21.15.16:0/1253249590 shutdown_connections 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 -- 172.21.15.16:0/1253249590 wait complete. 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.070+0000 7fb47853c640 1 Processor -- start 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 -- start start 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 0x7fb470198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4701036e0 0x7fb470198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701082a0 0x7fb47019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4701120a0 con 0x7fb470102d30 2024-09-12T02:46:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb470111f20 con 0x7fb4701036e0 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb47853c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb470112220 con 0x7fb4701082a0 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 0x7fb470198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 0x7fb470198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43458/0 (socket says 172.21.15.16:43458) 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 -- 172.21.15.16:0/3954329672 learned_addr learned my addr 172.21.15.16:0/3954329672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb475ad7640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4701036e0 0x7fb470198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb476ad9640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701082a0 0x7fb47019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 -- 172.21.15.16:0/3954329672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701082a0 msgr2=0x7fb47019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701082a0 0x7fb47019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 -- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4701036e0 msgr2=0x7fb470198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb4701036e0 0x7fb470198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47019d400 con 0x7fb470102d30 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb476ad9640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4701082a0 0x7fb47019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.071+0000 7fb4762d8640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 0x7fb470198450 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c920 tx=0x7fb46c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.072+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c007bb0 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.072+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47019d6f0 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.072+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb46c007d50 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.072+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c018780 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.072+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4701a4fd0 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.073+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4700fede0 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.074+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb46c00f450 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb4677fe640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb45c0a2c10 0x7fb45c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb46c0cf740 con 0x7fb470102d30 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb475ad7640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb45c0a2c10 0x7fb45c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb4677fe640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] conn(0x7fb45c0ac7c0 0x7fb45c0aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb46c015e70 con 0x7fb45c0ac7c0 2024-09-12T02:46:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.075+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fb46c0cfb90 con 0x7fb470102d30 2024-09-12T02:46:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.076+0000 7fb475ad7640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb45c0a2c10 0x7fb45c0a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb4700fdaf0 tx=0x7fb46000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.076+0000 7fb476ad9640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] conn(0x7fb45c0ac7c0 0x7fb45c0aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.077+0000 7fb476ad9640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] conn(0x7fb45c0ac7c0 0x7fb45c0aec60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.077+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== osd.1 v2:172.21.15.33:6800/2781270412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb46c015e70 con 0x7fb45c0ac7c0 2024-09-12T02:46:34.085 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:34.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.094+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 --> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb47006a850 con 0x7fb45c0ac7c0 2024-09-12T02:46:34.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.094+0000 7fb4677fe640 1 -- 172.21.15.16:0/3954329672 <== osd.1 v2:172.21.15.33:6800/2781270412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb47006a850 con 0x7fb45c0ac7c0 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] conn(0x7fb45c0ac7c0 msgr2=0x7fb45c0aec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] conn(0x7fb45c0ac7c0 0x7fb45c0aec60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb45c0a2c10 msgr2=0x7fb45c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb45c0a2c10 0x7fb45c0a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb4700fdaf0 tx=0x7fb46000a2d0 comp rx=0 tx=0).stop 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 msgr2=0x7fb470198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb47853c640 1 --2- 172.21.15.16:0/3954329672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb470102d30 0x7fb470198450 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c920 tx=0x7fb46c00cd20 comp rx=0 tx=0).stop 2024-09-12T02:46:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.095+0000 7fb476ad9640 1 -- 172.21.15.16:0/3954329672 reap_dead start 2024-09-12T02:46:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.096+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 shutdown_connections 2024-09-12T02:46:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.096+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 >> 172.21.15.16:0/3954329672 conn(0x7fb4700fc7d0 msgr2=0x7fb4700fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.096+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 shutdown_connections 2024-09-12T02:46:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:34.096+0000 7fb47853c640 1 -- 172.21.15.16:0/3954329672 wait complete. 2024-09-12T02:46:34.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:33 smithi016 ceph-mon[30925]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:33 smithi033 ceph-mon[35092]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:33 smithi184 ceph-mon[35321]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:34.747 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:35.315 INFO:teuthology.orchestra.run.smithi016.stdout:51539607624 2024-09-12T02:46:35.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 -- 172.21.15.16:0/608457381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 msgr2=0x7fc990106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 --2- 172.21.15.16:0/608457381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc990106840 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc984009900 tx=0x7fc98402a410 comp rx=0 tx=0).stop 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 -- 172.21.15.16:0/608457381 shutdown_connections 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 --2- 172.21.15.16:0/608457381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990106f70 0x7fc99010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 --2- 172.21.15.16:0/608457381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc990106840 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 --2- 172.21.15.16:0/608457381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990100500 0x7fc9901008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 -- 172.21.15.16:0/608457381 >> 172.21.15.16:0/608457381 conn(0x7fc9900fc5c0 msgr2=0x7fc9900fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 -- 172.21.15.16:0/608457381 shutdown_connections 2024-09-12T02:46:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.772+0000 7fc99517a640 1 -- 172.21.15.16:0/608457381 wait complete. 2024-09-12T02:46:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 Processor -- start 2024-09-12T02:46:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 -- start start 2024-09-12T02:46:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 0x7fc9901984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 0x7fc99019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc990111fd0 con 0x7fc990100eb0 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc990111e50 con 0x7fc990106f70 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc99517a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc990112150 con 0x7fc990100500 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.773+0000 7fc98e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35230/0 (socket says 172.21.15.16:35230) 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 -- 172.21.15.16:0/4082882770 learned_addr learned my addr 172.21.15.16:0/4082882770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98f59e640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 0x7fc99019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98ed9d640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 0x7fc9901984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 -- 172.21.15.16:0/4082882770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 msgr2=0x7fc9901984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 0x7fc9901984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 -- 172.21.15.16:0/4082882770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 msgr2=0x7fc99019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 0x7fc99019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 -- 172.21.15.16:0/4082882770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99019d460 con 0x7fc990100eb0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98ed9d640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 0x7fc9901984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98f59e640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 0x7fc99019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc98e59c640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc984006010 tx=0x7fc9840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98403f030 con 0x7fc990100eb0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99019d6f0 con 0x7fc990100eb0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984003ce0 con 0x7fc990100eb0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984035070 con 0x7fc990100eb0 2024-09-12T02:46:35.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.774+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9901a5030 con 0x7fc990100eb0 2024-09-12T02:46:35.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.775+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9901008e0 con 0x7fc990100eb0 2024-09-12T02:46:35.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.777+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc984042030 con 0x7fc990100eb0 2024-09-12T02:46:35.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.777+0000 7fc977fff640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 0x7fc9780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:35.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.777+0000 7fc98ed9d640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 0x7fc9780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:35.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.777+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc9840efad0 con 0x7fc990100eb0 2024-09-12T02:46:35.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.778+0000 7fc98ed9d640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 0x7fc9780a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc990069fc0 tx=0x7fc9800073c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:35.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.780+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc9840bc8e0 con 0x7fc990100eb0 2024-09-12T02:46:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.958+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc99010a110 con 0x7fc990100eb0 2024-09-12T02:46:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.958+0000 7fc977fff640 1 -- 172.21.15.16:0/4082882770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc9840dae20 con 0x7fc990100eb0 2024-09-12T02:46:35.960 INFO:teuthology.orchestra.run.smithi016.stdout:261993005088 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 msgr2=0x7fc9780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 0x7fc9780a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc990069fc0 tx=0x7fc9800073c0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 msgr2=0x7fc9901989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc984006010 tx=0x7fc9840042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 shutdown_connections 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc990106f70 0x7fc99019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9780a2c10 0x7fc9780a50d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc990100eb0 0x7fc9901989f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 --2- 172.21.15.16:0/4082882770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc990100500 0x7fc9901984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 >> 172.21.15.16:0/4082882770 conn(0x7fc9900fc5c0 msgr2=0x7fc990105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 shutdown_connections 2024-09-12T02:46:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:35.960+0000 7fc99517a640 1 -- 172.21.15.16:0/4082882770 wait complete. 2024-09-12T02:46:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:35 smithi016 ceph-mon[30925]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4082882770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-12T02:46:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:35 smithi033 ceph-mon[35092]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4082882770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-12T02:46:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:35 smithi184 ceph-mon[35321]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4082882770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-12T02:46:36.662 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005087 got 261993005088 for osd.9 2024-09-12T02:46:36.662 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 -- 172.21.15.16:0/2248255468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81062b0 msgr2=0x7f79e8106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 --2- 172.21.15.16:0/2248255468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81062b0 0x7f79e8106690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009930 tx=0x7f79d802a420 comp rx=0 tx=0).stop 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 -- 172.21.15.16:0/2248255468 shutdown_connections 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 --2- 172.21.15.16:0/2248255468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 0x7f79e810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 --2- 172.21.15.16:0/2248255468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81002a0 0x7f79e81006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 --2- 172.21.15.16:0/2248255468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81062b0 0x7f79e8106690 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 -- 172.21.15.16:0/2248255468 >> 172.21.15.16:0/2248255468 conn(0x7f79e80fc5c0 msgr2=0x7f79e80fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 -- 172.21.15.16:0/2248255468 shutdown_connections 2024-09-12T02:46:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.934+0000 7f79ec93b640 1 -- 172.21.15.16:0/2248255468 wait complete. 2024-09-12T02:46:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 Processor -- start 2024-09-12T02:46:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 -- start start 2024-09-12T02:46:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 0x7f79e819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81062b0 0x7f79e819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810fd90 con 0x7f79e81002a0 2024-09-12T02:46:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810fc10 con 0x7f79e81062b0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79ec93b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810ff10 con 0x7f79e8100c00 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35252/0 (socket says 172.21.15.16:35252) 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79e659c640 1 -- 172.21.15.16:0/3739312953 learned_addr learned my addr 172.21.15.16:0/3739312953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79e5d9b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 0x7f79e819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.935+0000 7f79e6d9d640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81062b0 0x7f79e819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 -- 172.21.15.16:0/3739312953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 msgr2=0x7f79e819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 0x7f79e819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 -- 172.21.15.16:0/3739312953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81062b0 msgr2=0x7f79e819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81062b0 0x7f79e819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 -- 172.21.15.16:0/3739312953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e819f620 con 0x7f79e81002a0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79e659c640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009230 tx=0x7f79d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d803f030 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d8003ad0 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e819f8b0 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8035070 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.936+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e81a71f0 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.937+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e81006c0 con 0x7f79e81002a0 2024-09-12T02:46:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.938+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f79d8042030 con 0x7f79e81002a0 2024-09-12T02:46:36.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.939+0000 7f79d77fe640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 0x7f79d00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:36.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.939+0000 7f79e5d9b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 0x7f79d00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:36.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.939+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f79d80ef200 con 0x7f79e81002a0 2024-09-12T02:46:36.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.940+0000 7f79e5d9b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 0x7f79d00a5150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79e80fff20 tx=0x7f79dc00b490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:36.943+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f79d80bb9c0 con 0x7f79e81002a0 2024-09-12T02:46:37.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.120+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f79e819b9d0 con 0x7f79e81002a0 2024-09-12T02:46:37.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.120+0000 7f79d77fe640 1 -- 172.21.15.16:0/3739312953 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f79d80bfcc0 con 0x7f79e81002a0 2024-09-12T02:46:37.122 INFO:teuthology.orchestra.run.smithi016.stdout:124554051641 2024-09-12T02:46:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 msgr2=0x7f79d00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 0x7f79d00a5150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79e80fff20 tx=0x7f79dc00b490 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 msgr2=0x7f79e819a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009230 tx=0x7f79d8004270 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 shutdown_connections 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79e81062b0 0x7f79e819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79e8100c00 0x7f79e819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f79d00a2c90 0x7f79d00a5150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 --2- 172.21.15.16:0/3739312953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79e81002a0 0x7f79e819a670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 >> 172.21.15.16:0/3739312953 conn(0x7f79e80fc5c0 msgr2=0x7f79e81045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 shutdown_connections 2024-09-12T02:46:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.122+0000 7f79ec93b640 1 -- 172.21.15.16:0/3739312953 wait complete. 2024-09-12T02:46:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:37.538 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 -- 172.21.15.16:0/3853483104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 msgr2=0x7f991810df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 --2- 172.21.15.16:0/3853483104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f991810df90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9900009900 tx=0x7f990002a410 comp rx=0 tx=0).stop 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 -- 172.21.15.16:0/3853483104 shutdown_connections 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 --2- 172.21.15.16:0/3853483104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991810e4d0 0x7f99181108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 --2- 172.21.15.16:0/3853483104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f991810df90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 --2- 172.21.15.16:0/3853483104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99181009b0 0x7f9918100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 -- 172.21.15.16:0/3853483104 >> 172.21.15.16:0/3853483104 conn(0x7f99180fc7d0 msgr2=0x7f99180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 -- 172.21.15.16:0/3853483104 shutdown_connections 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.624+0000 7f991dbd7640 1 -- 172.21.15.16:0/3853483104 wait complete. 2024-09-12T02:46:37.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 Processor -- start 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 -- start start 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99181009b0 0x7f99181a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f991810e4d0 0x7f991819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9918114220 con 0x7f9918101360 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99181140a0 con 0x7f991810e4d0 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f991dbd7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99181143a0 con 0x7f99181009b0 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35272/0 (socket says 172.21.15.16:35272) 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 -- 172.21.15.16:0/728735535 learned_addr learned my addr 172.21.15.16:0/728735535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f9917fff640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f991810e4d0 0x7f991819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f99177fe640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99181009b0 0x7f99181a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 -- 172.21.15.16:0/728735535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99181009b0 msgr2=0x7f99181a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99181009b0 0x7f99181a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 -- 172.21.15.16:0/728735535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f991810e4d0 msgr2=0x7f991819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f991810e4d0 0x7f991819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.625+0000 7f990ffff640 1 -- 172.21.15.16:0/728735535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991819ace0 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f990ffff640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9900006010 tx=0x7f99000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990003f030 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991819af70 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9900003ce0 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900035070 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.626+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99181a72c0 con 0x7f9918101360 2024-09-12T02:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.627+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9918100d90 con 0x7f9918101360 2024-09-12T02:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.628+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9900042030 con 0x7f9918101360 2024-09-12T02:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.629+0000 7f99157fa640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 0x7f99040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.629+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f99000efb20 con 0x7f9918101360 2024-09-12T02:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.629+0000 7f99177fe640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 0x7f99040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.629+0000 7f99177fe640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 0x7f99040a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9908005fd0 tx=0x7f990800b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.632+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f99000bc9f0 con 0x7f9918101360 2024-09-12T02:46:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.818+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f991819bae0 con 0x7f9918101360 2024-09-12T02:46:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.818+0000 7f99157fa640 1 -- 172.21.15.16:0/728735535 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f991819bae0 con 0x7f9918101360 2024-09-12T02:46:37.820 INFO:teuthology.orchestra.run.smithi016.stdout:309237645335 2024-09-12T02:46:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 msgr2=0x7f99040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 0x7f99040a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9908005fd0 tx=0x7f990800b590 comp rx=0 tx=0).stop 2024-09-12T02:46:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 msgr2=0x7f99181a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9900006010 tx=0x7f99000042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 shutdown_connections 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f991810e4d0 0x7f991819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99040a2c10 0x7f99040a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9918101360 0x7f99181a0930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 --2- 172.21.15.16:0/728735535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99181009b0 0x7f99181a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 >> 172.21.15.16:0/728735535 conn(0x7f99180fc7d0 msgr2=0x7f99180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 shutdown_connections 2024-09-12T02:46:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:37.820+0000 7f991dbd7640 1 -- 172.21.15.16:0/728735535 wait complete. 2024-09-12T02:46:38.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:37 smithi016 ceph-mon[30925]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:37 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3739312953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-12T02:46:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:37 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/728735535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-12T02:46:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:37 smithi033 ceph-mon[35092]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:37 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3739312953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-12T02:46:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:37 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/728735535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-12T02:46:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:37 smithi184 ceph-mon[35321]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:37 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3739312953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-12T02:46:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:37 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/728735535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-12T02:46:38.545 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645332 got 309237645335 for osd.11 2024-09-12T02:46:38.545 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:38.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 -- 172.21.15.16:0/934267681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102570 msgr2=0x7f3af41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 --2- 172.21.15.16:0/934267681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102570 0x7f3af41029b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009900 tx=0x7f3ae802a480 comp rx=0 tx=0).stop 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 -- 172.21.15.16:0/934267681 shutdown_connections 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 --2- 172.21.15.16:0/934267681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4102ef0 0x7f3af410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 --2- 172.21.15.16:0/934267681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102570 0x7f3af41029b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 --2- 172.21.15.16:0/934267681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4100e20 0x7f3af4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 -- 172.21.15.16:0/934267681 >> 172.21.15.16:0/934267681 conn(0x7f3af40fc7d0 msgr2=0x7f3af40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 -- 172.21.15.16:0/934267681 shutdown_connections 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.722+0000 7f3af9e34640 1 -- 172.21.15.16:0/934267681 wait complete. 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 Processor -- start 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 -- start start 2024-09-12T02:46:38.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4100e20 0x7f3af40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 0x7f3af40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4110fd0 con 0x7f3af4102ef0 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4110e50 con 0x7f3af4100e20 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af9e34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4111150 con 0x7f3af4102570 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35282/0 (socket says 172.21.15.16:35282) 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.723+0000 7f3af3fff640 1 -- 172.21.15.16:0/3831823133 learned_addr learned my addr 172.21.15.16:0/3831823133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af2ffd640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 0x7f3af40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af37fe640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4100e20 0x7f3af40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 -- 172.21.15.16:0/3831823133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 msgr2=0x7f3af40fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:38.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 0x7f3af40fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 -- 172.21.15.16:0/3831823133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4100e20 msgr2=0x7f3af40fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4100e20 0x7f3af40fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 -- 172.21.15.16:0/3831823133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af41a6a10 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af2ffd640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 0x7f3af40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af3fff640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00cad0 tx=0x7f3aec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00ae20 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aec005020 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af41a6d00 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00f6f0 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.724+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af41a71c0 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.725+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af4101200 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.726+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3aec0051d0 con 0x7f3af4102ef0 2024-09-12T02:46:38.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.727+0000 7f3af0ff9640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 0x7f3adc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:38.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.727+0000 7f3af37fe640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 0x7f3adc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:38.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.727+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3aec0cf2c0 con 0x7f3af4102ef0 2024-09-12T02:46:38.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.727+0000 7f3af37fe640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 0x7f3adc0a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4005fd0 tx=0x7f3ae4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:38.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.731+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3aec014030 con 0x7f3af4102ef0 2024-09-12T02:46:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.909+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3af40fe1b0 con 0x7f3af4102ef0 2024-09-12T02:46:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.910+0000 7f3af0ff9640 1 -- 172.21.15.16:0/3831823133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3af40fe1b0 con 0x7f3af4102ef0 2024-09-12T02:46:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:34359738445 2024-09-12T02:46:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 msgr2=0x7f3adc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 0x7f3adc0a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4005fd0 tx=0x7f3ae4005850 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 msgr2=0x7f3af40fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00cad0 tx=0x7f3aec00cf90 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 shutdown_connections 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3adc0a2c10 0x7f3adc0a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3af4102ef0 0x7f3af40fd700 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4102570 0x7f3af40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 --2- 172.21.15.16:0/3831823133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3af4100e20 0x7f3af40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 >> 172.21.15.16:0/3831823133 conn(0x7f3af40fc7d0 msgr2=0x7f3af406a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 shutdown_connections 2024-09-12T02:46:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:38.912+0000 7f3af9e34640 1 -- 172.21.15.16:0/3831823133 wait complete. 2024-09-12T02:46:39.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:38 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3831823133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-12T02:46:39.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051639 got 124554051641 for osd.4 2024-09-12T02:46:39.136 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:39.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:38 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3831823133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-12T02:46:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:38 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3831823133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-12T02:46:39.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 -- 172.21.15.16:0/1699520405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074106f10 msgr2=0x7f907410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1699520405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074106f10 0x7f907410ea90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f906c009ca0 tx=0x7f906c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 -- 172.21.15.16:0/1699520405 shutdown_connections 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1699520405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074106f10 0x7f907410ea90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1699520405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074102790 0x7f90741067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1699520405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 0x7f90741021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 -- 172.21.15.16:0/1699520405 >> 172.21.15.16:0/1699520405 conn(0x7f90740fd8c0 msgr2=0x7f90740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 -- 172.21.15.16:0/1699520405 shutdown_connections 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.299+0000 7f9078ff7640 1 -- 172.21.15.16:0/1699520405 wait complete. 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 Processor -- start 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 -- start start 2024-09-12T02:46:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 0x7f90741983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074106f10 0x7f907419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9074111f70 con 0x7f9074102790 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9074111df0 con 0x7f9074101de0 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9078ff7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90741120f0 con 0x7f9074106f10 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35304/0 (socket says 172.21.15.16:35304) 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f907259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 0x7f90741983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 -- 172.21.15.16:0/1328398701 learned_addr learned my addr 172.21.15.16:0/1328398701 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f9072d9d640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074106f10 0x7f907419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 -- 172.21.15.16:0/1328398701 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074106f10 msgr2=0x7f907419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074106f10 0x7f907419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 -- 172.21.15.16:0/1328398701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 msgr2=0x7f90741983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 0x7f90741983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.300+0000 7f906bfff640 1 -- 172.21.15.16:0/1328398701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907419d360 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f906bfff640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f905c00d770 tx=0x7f905c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c017070 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c005500 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907419d650 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c004e00 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.301+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90741a4f30 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.302+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90741032c0 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.303+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f905c004fa0 con 0x7f9074102790 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.304+0000 7f906b7fe640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 0x7f90600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:39.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.304+0000 7f907259c640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 0x7f90600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.304+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f905c0cebe0 con 0x7f9074102790 2024-09-12T02:46:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.304+0000 7f907259c640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 0x7f90600a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f906400bbe0 tx=0x7f906400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.308+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f905c09b9f0 con 0x7f9074102790 2024-09-12T02:46:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.487+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f907410a0f0 con 0x7f9074102790 2024-09-12T02:46:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.488+0000 7f906b7fe640 1 -- 172.21.15.16:0/1328398701 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f905c09b2f0 con 0x7f9074102790 2024-09-12T02:46:39.489 INFO:teuthology.orchestra.run.smithi016.stdout:176093659184 2024-09-12T02:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 msgr2=0x7f90600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 0x7f90600a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f906400bbe0 tx=0x7f906400b390 comp rx=0 tx=0).stop 2024-09-12T02:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 msgr2=0x7f90741988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f905c00d770 tx=0x7f905c00dc30 comp rx=0 tx=0).stop 2024-09-12T02:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 shutdown_connections 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9074106f10 0x7f907419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90600a2c10 0x7f90600a50d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102790 0x7f90741988f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 --2- 172.21.15.16:0/1328398701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074101de0 0x7f90741983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 >> 172.21.15.16:0/1328398701 conn(0x7f90740fd8c0 msgr2=0x7f90740fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 shutdown_connections 2024-09-12T02:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:39.490+0000 7f9078ff7640 1 -- 172.21.15.16:0/1328398701 wait complete. 2024-09-12T02:46:40.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:39 smithi016 ceph-mon[30925]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:46:40.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:39 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1328398701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-12T02:46:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:39 smithi033 ceph-mon[35092]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:46:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:39 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1328398701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 -- 172.21.15.16:0/2022822773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08102d40 msgr2=0x7fdc08103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 --2- 172.21.15.16:0/2022822773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08102d40 0x7fdc08103120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009900 tx=0x7fdbfc02a480 comp rx=0 tx=0).stop 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 -- 172.21.15.16:0/2022822773 shutdown_connections 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 --2- 172.21.15.16:0/2022822773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc0810c170 0x7fdc0810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 --2- 172.21.15.16:0/2022822773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc08103660 0x7fdc0810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 --2- 172.21.15.16:0/2022822773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08102d40 0x7fdc08103120 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 -- 172.21.15.16:0/2022822773 >> 172.21.15.16:0/2022822773 conn(0x7fdc080fc7a0 msgr2=0x7fdc080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 -- 172.21.15.16:0/2022822773 shutdown_connections 2024-09-12T02:46:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.281+0000 7fdc0f196640 1 -- 172.21.15.16:0/2022822773 wait complete. 2024-09-12T02:46:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 Processor -- start 2024-09-12T02:46:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 -- start start 2024-09-12T02:46:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 0x7fdc081984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc0810c170 0x7fdc0819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08111fb0 con 0x7fdc08103660 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08111e30 con 0x7fdc0810c170 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0f196640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08112130 con 0x7fdc08102d40 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35326/0 (socket says 172.21.15.16:35326) 2024-09-12T02:46:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 -- 172.21.15.16:0/1818657311 learned_addr learned my addr 172.21.15.16:0/1818657311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0cf32640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 0x7fdc081984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0d733640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc0810c170 0x7fdc0819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 -- 172.21.15.16:0/1818657311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 msgr2=0x7fdc081984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 0x7fdc081984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 -- 172.21.15.16:0/1818657311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc0810c170 msgr2=0x7fdc0819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc0810c170 0x7fdc0819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc07fff640 1 -- 172.21.15.16:0/1818657311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0819d460 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.282+0000 7fdc0cf32640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 0x7fdc081984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc07fff640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400b3f0 tx=0x7fdbf400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4015020 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0819d750 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf4011d10 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400f2e0 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.283+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc081a5030 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.284+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc08103120 con 0x7fdc08103660 2024-09-12T02:46:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.286+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdbf4018020 con 0x7fdc08103660 2024-09-12T02:46:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.287+0000 7fdc05ffb640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 0x7fdbf80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.287+0000 7fdc0cf32640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 0x7fdbf80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.287+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdbf40ceef0 con 0x7fdc08103660 2024-09-12T02:46:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.287+0000 7fdc0cf32640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 0x7fdbf80a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005ab0 tx=0x7fdbfc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.290+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdbf40fe1b0 con 0x7fdc08103660 2024-09-12T02:46:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:39 smithi184 ceph-mon[35321]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:46:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:39 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1328398701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-12T02:46:40.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.469+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdc081a5290 con 0x7fdc08103660 2024-09-12T02:46:40.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.469+0000 7fdc05ffb640 1 -- 172.21.15.16:0/1818657311 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdbf401f020 con 0x7fdc08103660 2024-09-12T02:46:40.472 INFO:teuthology.orchestra.run.smithi016.stdout:201863462955 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.471+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 msgr2=0x7fdbf80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.471+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 0x7fdbf80a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005ab0 tx=0x7fdbfc005a20 comp rx=0 tx=0).stop 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.471+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 msgr2=0x7fdc081989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.471+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400b3f0 tx=0x7fdbf400b8b0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 shutdown_connections 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc0810c170 0x7fdc0819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdbf80a2c10 0x7fdbf80a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08103660 0x7fdc081989f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 --2- 172.21.15.16:0/1818657311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc08102d40 0x7fdc081984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 >> 172.21.15.16:0/1818657311 conn(0x7fdc080fc7a0 msgr2=0x7fdc080fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:40.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 shutdown_connections 2024-09-12T02:46:40.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.472+0000 7fdc0f196640 1 -- 172.21.15.16:0/1818657311 wait complete. 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 -- 172.21.15.16:0/680553159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c101420 msgr2=0x7f472c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 --2- 172.21.15.16:0/680553159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c101420 0x7f472c10d950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4724009cd0 tx=0x7f472402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 -- 172.21.15.16:0/680553159 shutdown_connections 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 --2- 172.21.15.16:0/680553159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c101420 0x7f472c10d950 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 --2- 172.21.15.16:0/680553159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 0x7f472c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 --2- 172.21.15.16:0/680553159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c106a80 0x7f472c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 -- 172.21.15.16:0/680553159 >> 172.21.15.16:0/680553159 conn(0x7f472c0fc7a0 msgr2=0x7f472c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 -- 172.21.15.16:0/680553159 shutdown_connections 2024-09-12T02:46:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 -- 172.21.15.16:0/680553159 wait complete. 2024-09-12T02:46:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.576+0000 7f47310e3640 1 Processor -- start 2024-09-12T02:46:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 -- start start 2024-09-12T02:46:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 0x7f472c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c101420 0x7f472c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f472c110f20 con 0x7f472c106a80 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f472c110da0 con 0x7f472c100a80 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f47310e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1110a0 con 0x7f472c101420 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35332/0 (socket says 172.21.15.16:35332) 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 -- 172.21.15.16:0/1000366678 learned_addr learned my addr 172.21.15.16:0/1000366678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472ad9d640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 0x7f472c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472a59c640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c101420 0x7f472c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 -- 172.21.15.16:0/1000366678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c101420 msgr2=0x7f472c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c101420 0x7f472c198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 -- 172.21.15.16:0/1000366678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 msgr2=0x7f472c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.577+0000 7f472b59e640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 0x7f472c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f472b59e640 1 -- 172.21.15.16:0/1000366678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c19d4f0 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f472b59e640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4724006010 tx=0x7f4724003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724004450 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4724003e10 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c19d780 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724046480 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.578+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c1a50c0 con 0x7f472c106a80 2024-09-12T02:46:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.579+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f472c100f60 con 0x7f472c106a80 2024-09-12T02:46:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.581+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f47240045f0 con 0x7f472c106a80 2024-09-12T02:46:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.582+0000 7f4723fff640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 0x7f47180a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.582+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f472402e070 con 0x7f472c106a80 2024-09-12T02:46:40.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.583+0000 7f472ad9d640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 0x7f47180a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:40.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.583+0000 7f472ad9d640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 0x7f47180a5150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f471c009990 tx=0x7f471c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.586+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f47240bd4c0 con 0x7f472c106a80 2024-09-12T02:46:40.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.765+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f472c10a080 con 0x7f472c106a80 2024-09-12T02:46:40.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.765+0000 7f4723fff640 1 -- 172.21.15.16:0/1000366678 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f47240bcdc0 con 0x7f472c106a80 2024-09-12T02:46:40.767 INFO:teuthology.orchestra.run.smithi016.stdout:231928234022 2024-09-12T02:46:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 msgr2=0x7f47180a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 0x7f47180a5150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f471c009990 tx=0x7f471c009110 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 msgr2=0x7f472c19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4724006010 tx=0x7f4724003c20 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 shutdown_connections 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47180a2c90 0x7f47180a5150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c106a80 0x7f472c19ce10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f472c101420 0x7f472c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 --2- 172.21.15.16:0/1000366678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f472c100a80 0x7f472c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 >> 172.21.15.16:0/1000366678 conn(0x7f472c0fc7a0 msgr2=0x7f472c06aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 shutdown_connections 2024-09-12T02:46:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:40.767+0000 7f47310e3640 1 -- 172.21.15.16:0/1000366678 wait complete. 2024-09-12T02:46:41.016 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:41.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:40 smithi016 ceph-mon[30925]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-12T02:46:41.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:40 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1818657311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-12T02:46:41.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:40 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1000366678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-12T02:46:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:40 smithi033 ceph-mon[35092]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-12T02:46:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:40 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1818657311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-12T02:46:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:40 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1000366678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-12T02:46:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:40 smithi184 ceph-mon[35321]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-12T02:46:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:40 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1818657311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-12T02:46:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:40 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1000366678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-12T02:46:41.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738445 for osd.0 2024-09-12T02:46:41.422 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:41.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234019 got 231928234022 for osd.8 2024-09-12T02:46:41.869 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 -- 172.21.15.16:0/1289015619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0102790 msgr2=0x7fa7e01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/1289015619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0102790 0x7fa7e01067e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009900 tx=0x7fa7d402a410 comp rx=0 tx=0).stop 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 -- 172.21.15.16:0/1289015619 shutdown_connections 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/1289015619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 0x7fa7e010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/1289015619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0102790 0x7fa7e01067e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/1289015619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0101de0 0x7fa7e01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 -- 172.21.15.16:0/1289015619 >> 172.21.15.16:0/1289015619 conn(0x7fa7e00fd8c0 msgr2=0x7fa7e00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 -- 172.21.15.16:0/1289015619 shutdown_connections 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.032+0000 7fa7e63b8640 1 -- 172.21.15.16:0/1289015619 wait complete. 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.033+0000 7fa7e63b8640 1 Processor -- start 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.033+0000 7fa7e63b8640 1 -- start start 2024-09-12T02:46:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0102790 0x7fa7e01989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 0x7fa7e019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0111f70 con 0x7fa7e0101de0 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0111df0 con 0x7fa7e0106f10 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e63b8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e01120f0 con 0x7fa7e0102790 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35344/0 (socket says 172.21.15.16:35344) 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 -- 172.21.15.16:0/3859600752 learned_addr learned my addr 172.21.15.16:0/3859600752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7df7fe640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0102790 0x7fa7e01989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7e4955640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 0x7fa7e019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 -- 172.21.15.16:0/3859600752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0102790 msgr2=0x7fa7e01989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0102790 0x7fa7e01989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 -- 172.21.15.16:0/3859600752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 msgr2=0x7fa7e019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 0x7fa7e019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 -- 172.21.15.16:0/3859600752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e019d440 con 0x7fa7e0101de0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dffff640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000d770 tx=0x7fa7d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.034+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0017070 con 0x7fa7e0101de0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.035+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e019d730 con 0x7fa7e0101de0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.035+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d0005500 con 0x7fa7e0101de0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.035+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0004e00 con 0x7fa7e0101de0 2024-09-12T02:46:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.035+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e01a5010 con 0x7fa7e0101de0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.036+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e01021c0 con 0x7fa7e0101de0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.037+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa7d0004fa0 con 0x7fa7e0101de0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.037+0000 7fa7dd7fa640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 0x7fa7c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.038+0000 7fa7df7fe640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 0x7fa7c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.038+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa7d00ceb90 con 0x7fa7e0101de0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.038+0000 7fa7df7fe640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 0x7fa7c80a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.042+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa7d009b350 con 0x7fa7e0101de0 2024-09-12T02:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.223+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa7e010a150 con 0x7fa7e0101de0 2024-09-12T02:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.223+0000 7fa7dd7fa640 1 -- 172.21.15.16:0/3859600752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa7d009f650 con 0x7fa7e0101de0 2024-09-12T02:46:42.225 INFO:teuthology.orchestra.run.smithi016.stdout:146028888118 2024-09-12T02:46:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.225+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 msgr2=0x7fa7c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.225+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 0x7fa7c80a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.225+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 msgr2=0x7fa7e0198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.225+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000d770 tx=0x7fa7d000dc30 comp rx=0 tx=0).stop 2024-09-12T02:46:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 shutdown_connections 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7e0106f10 0x7fa7e019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7e0102790 0x7fa7e01989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7c80a2c10 0x7fa7c80a50d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 --2- 172.21.15.16:0/3859600752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e0101de0 0x7fa7e0198490 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 >> 172.21.15.16:0/3859600752 conn(0x7fa7e00fd8c0 msgr2=0x7fa7e00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 shutdown_connections 2024-09-12T02:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:42.226+0000 7fa7e63b8640 1 -- 172.21.15.16:0/3859600752 wait complete. 2024-09-12T02:46:42.915 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888114 got 146028888118 for osd.5 2024-09-12T02:46:42.916 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:43.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:43 smithi033 ceph-mon[35092]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-12T02:46:43.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3859600752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-12T02:46:43.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:43 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:46:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:43 smithi016 ceph-mon[30925]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-12T02:46:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:43 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3859600752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-12T02:46:43.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:43 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:46:43.287 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659181 got 176093659184 for osd.6 2024-09-12T02:46:43.288 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:43 smithi184 ceph-mon[35321]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-12T02:46:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:43 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3859600752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-12T02:46:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:43 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 -- 172.21.15.16:0/3802406553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab6810e480 msgr2=0x7fab68110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/3802406553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab6810e480 0x7fab68110870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab60009ca0 tx=0x7fab6002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 -- 172.21.15.16:0/3802406553 shutdown_connections 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/3802406553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab6810e480 0x7fab68110870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/3802406553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 0x7fab6810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/3802406553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab681008d0 0x7fab68100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 -- 172.21.15.16:0/3802406553 >> 172.21.15.16:0/3802406553 conn(0x7fab680fc7a0 msgr2=0x7fab680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 -- 172.21.15.16:0/3802406553 shutdown_connections 2024-09-12T02:46:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 -- 172.21.15.16:0/3802406553 wait complete. 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.418+0000 7fab6fc9d640 1 Processor -- start 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 -- start start 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 0x7fab6819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab6810e480 0x7fab6819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab68114160 con 0x7fab681008d0 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab68113fe0 con 0x7fab6810e480 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6fc9d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab681142e0 con 0x7fab68101280 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35368/0 (socket says 172.21.15.16:35368) 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 -- 172.21.15.16:0/2854552778 learned_addr learned my addr 172.21.15.16:0/2854552778 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6d238640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 0x7fab6819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6e23a640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab6810e480 0x7fab6819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 -- 172.21.15.16:0/2854552778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 msgr2=0x7fab6819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 0x7fab6819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 -- 172.21.15.16:0/2854552778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab6810e480 msgr2=0x7fab6819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab6810e480 0x7fab6819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 -- 172.21.15.16:0/2854552778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6819f610 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab6da39640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9f0 tx=0x7fab6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6400cd80 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.419+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab64010910 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.420+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6819f900 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.420+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab64018990 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.420+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab681a71e0 con 0x7fab681008d0 2024-09-12T02:46:43.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.420+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab68100cb0 con 0x7fab681008d0 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.422+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fab64010ac0 con 0x7fab681008d0 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.423+0000 7fab5effd640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 0x7fab540a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.423+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fab640cf850 con 0x7fab681008d0 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.423+0000 7fab6d238640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 0x7fab540a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.423+0000 7fab6d238640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 0x7fab540a5150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab6810ed80 tx=0x7fab58005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:43.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.426+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab6409c010 con 0x7fab681008d0 2024-09-12T02:46:43.493 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:43.494 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.622+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fab6819b9c0 con 0x7fab681008d0 2024-09-12T02:46:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.622+0000 7fab5effd640 1 -- 172.21.15.16:0/2854552778 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fab640a0310 con 0x7fab681008d0 2024-09-12T02:46:43.624 INFO:teuthology.orchestra.run.smithi016.stdout:68719476805 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 msgr2=0x7fab540a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 0x7fab540a5150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab6810ed80 tx=0x7fab58005f90 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 msgr2=0x7fab6819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9f0 tx=0x7fab6400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 shutdown_connections 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab6810e480 0x7fab6819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab68101280 0x7fab6819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab540a2c90 0x7fab540a5150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 --2- 172.21.15.16:0/2854552778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab681008d0 0x7fab6819a660 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 >> 172.21.15.16:0/2854552778 conn(0x7fab680fc7a0 msgr2=0x7fab680feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 shutdown_connections 2024-09-12T02:46:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:43.625+0000 7fab6fc9d640 1 -- 172.21.15.16:0/2854552778 wait complete. 2024-09-12T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2854552778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-12T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:44 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462953 got 201863462955 for osd.7 2024-09-12T02:46:44.287 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2854552778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-12T02:46:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:44 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2854552778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-12T02:46:44.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:44 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:44.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476802 got 68719476805 for osd.2 2024-09-12T02:46:44.693 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 -- 172.21.15.16:0/19031703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102790 msgr2=0x7f53581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 --2- 172.21.15.16:0/19031703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102790 0x7f53581067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f534c009900 tx=0x7f534c02a410 comp rx=0 tx=0).stop 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 -- 172.21.15.16:0/19031703 shutdown_connections 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 --2- 172.21.15.16:0/19031703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358106f10 0x7f535810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 --2- 172.21.15.16:0/19031703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102790 0x7f53581067e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 --2- 172.21.15.16:0/19031703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358101de0 0x7f53581021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 -- 172.21.15.16:0/19031703 >> 172.21.15.16:0/19031703 conn(0x7f53580fd8c0 msgr2=0x7f53580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 -- 172.21.15.16:0/19031703 shutdown_connections 2024-09-12T02:46:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.099+0000 7f535e291640 1 -- 172.21.15.16:0/19031703 wait complete. 2024-09-12T02:46:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 Processor -- start 2024-09-12T02:46:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 -- start start 2024-09-12T02:46:45.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358102790 0x7f5358074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358106f10 0x7f5358074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5358111f70 con 0x7f5358101de0 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5358111df0 con 0x7f5358106f10 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535e291640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f53581120f0 con 0x7f5358102790 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f5357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f5357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35384/0 (socket says 172.21.15.16:35384) 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f5357fff640 1 -- 172.21.15.16:0/1119229743 learned_addr learned my addr 172.21.15.16:0/1119229743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f535c82e640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358106f10 0x7f5358074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.100+0000 7f53577fe640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358102790 0x7f5358074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 -- 172.21.15.16:0/1119229743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358102790 msgr2=0x7f5358074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358102790 0x7f5358074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 -- 172.21.15.16:0/1119229743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358106f10 msgr2=0x7f5358074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358106f10 0x7f5358074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 -- 172.21.15.16:0/1119229743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53581a68f0 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f5357fff640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f534800d950 tx=0x7f534800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5348017030 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5348004020 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53581a6be0 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480047a0 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.101+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53581a70f0 con 0x7f5358101de0 2024-09-12T02:46:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.102+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53581021c0 con 0x7f5358101de0 2024-09-12T02:46:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.103+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f53480041d0 con 0x7f5358101de0 2024-09-12T02:46:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.104+0000 7f53557fa640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 0x7f53400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.104+0000 7f53577fe640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 0x7f53400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.104+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f53480ceae0 con 0x7f5358101de0 2024-09-12T02:46:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.104+0000 7f53577fe640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 0x7f53400a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f534c009230 tx=0x7f534c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.108+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f534809b8f0 con 0x7f5358101de0 2024-09-12T02:46:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.286+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f535802d7c0 con 0x7f5358101de0 2024-09-12T02:46:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.287+0000 7f53557fa640 1 -- 172.21.15.16:0/1119229743 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f534809b1f0 con 0x7f5358101de0 2024-09-12T02:46:45.289 INFO:teuthology.orchestra.run.smithi016.stdout:94489280576 2024-09-12T02:46:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 msgr2=0x7f53400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 0x7f53400a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f534c009230 tx=0x7f534c0058d0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 msgr2=0x7f53580744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f534800d950 tx=0x7f534800de10 comp rx=0 tx=0).stop 2024-09-12T02:46:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 shutdown_connections 2024-09-12T02:46:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5358106f10 0x7f5358074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5358102790 0x7f5358074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53400a2c10 0x7f53400a50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 --2- 172.21.15.16:0/1119229743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101de0 0x7f53580744f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 >> 172.21.15.16:0/1119229743 conn(0x7f53580fd8c0 msgr2=0x7f53580fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 shutdown_connections 2024-09-12T02:46:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.289+0000 7f535e291640 1 -- 172.21.15.16:0/1119229743 wait complete. 2024-09-12T02:46:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 -- 172.21.15.16:0/231374072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4106f10 msgr2=0x7faef410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 --2- 172.21.15.16:0/231374072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4106f10 0x7faef410ea90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faeec009ca0 tx=0x7faeec02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 -- 172.21.15.16:0/231374072 shutdown_connections 2024-09-12T02:46:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 --2- 172.21.15.16:0/231374072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4106f10 0x7faef410ea90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 --2- 172.21.15.16:0/231374072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4102790 0x7faef41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 --2- 172.21.15.16:0/231374072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4101de0 0x7faef41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 -- 172.21.15.16:0/231374072 >> 172.21.15.16:0/231374072 conn(0x7faef40fd8c0 msgr2=0x7faef40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 -- 172.21.15.16:0/231374072 shutdown_connections 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.291+0000 7faefc2ce640 1 -- 172.21.15.16:0/231374072 wait complete. 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 Processor -- start 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 -- start start 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 0x7faef41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4106f10 0x7faef419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faef4111f70 con 0x7faef4102790 2024-09-12T02:46:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faef4111df0 con 0x7faef4101de0 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefc2ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faef41120f0 con 0x7faef4106f10 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faef9869640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faef9869640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54836/0 (socket says 172.21.15.16:54836) 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faef9869640 1 -- 172.21.15.16:0/1380106797 learned_addr learned my addr 172.21.15.16:0/1380106797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 0x7faef41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa86b640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4106f10 0x7faef419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 -- 172.21.15.16:0/1380106797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4106f10 msgr2=0x7faef419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4106f10 0x7faef419cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 -- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 msgr2=0x7faef4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faefa06a640 1 -- 172.21.15.16:0/1380106797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef419d380 con 0x7faef4101de0 2024-09-12T02:46:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.292+0000 7faef9869640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faefa06a640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 0x7faef41983d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faef000e9f0 tx=0x7faef000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef000ccc0 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef0010910 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef419d670 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0018960 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.293+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef41a4f50 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.294+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef41021c0 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.295+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7faef0010ac0 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.296+0000 7faeeb7fe640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 0x7faee00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.296+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faef00cf5b0 con 0x7faef4101de0 2024-09-12T02:46:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.296+0000 7faef9869640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 0x7faee00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.297+0000 7faef9869640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 0x7faee00a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faef4108880 tx=0x7faee4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.301+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faef009bd10 con 0x7faef4101de0 2024-09-12T02:46:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.479+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7faef410a150 con 0x7faef4101de0 2024-09-12T02:46:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.479+0000 7faeeb7fe640 1 -- 172.21.15.16:0/1380106797 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7faef00a0010 con 0x7faef4101de0 2024-09-12T02:46:45.481 INFO:teuthology.orchestra.run.smithi016.stdout:279172874269 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 msgr2=0x7faee00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 0x7faee00a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faef4108880 tx=0x7faee4005f90 comp rx=0 tx=0).stop 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 msgr2=0x7faef41983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 0x7faef41983d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faef000e9f0 tx=0x7faef000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 shutdown_connections 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faef4106f10 0x7faef419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faee00a2c10 0x7faee00a50d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4102790 0x7faef4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 --2- 172.21.15.16:0/1380106797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faef4101de0 0x7faef41983d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 >> 172.21.15.16:0/1380106797 conn(0x7faef40fd8c0 msgr2=0x7faef40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 shutdown_connections 2024-09-12T02:46:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:45.481+0000 7faefc2ce640 1 -- 172.21.15.16:0/1380106797 wait complete. 2024-09-12T02:46:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:45 smithi033 ceph-mon[35092]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1119229743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-12T02:46:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1380106797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-12T02:46:46.042 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:45 smithi016 ceph-mon[30925]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:46.042 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1119229743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-12T02:46:46.042 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1380106797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-12T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:45 smithi184 ceph-mon[35321]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1119229743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-12T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1380106797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-12T02:46:46.388 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874266 got 279172874269 for osd.10 2024-09-12T02:46:46.388 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 -- 172.21.15.16:0/3891489005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 msgr2=0x7f73c810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 --2- 172.21.15.16:0/3891489005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c810e610 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009ca0 tx=0x7f73c002a950 comp rx=0 tx=0).stop 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 -- 172.21.15.16:0/3891489005 shutdown_connections 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 --2- 172.21.15.16:0/3891489005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c810e610 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 --2- 172.21.15.16:0/3891489005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 0x7f73c810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 --2- 172.21.15.16:0/3891489005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 -- 172.21.15.16:0/3891489005 >> 172.21.15.16:0/3891489005 conn(0x7f73c80fc7a0 msgr2=0x7f73c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 -- 172.21.15.16:0/3891489005 shutdown_connections 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 -- 172.21.15.16:0/3891489005 wait complete. 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.593+0000 7f73cd73f640 1 Processor -- start 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 -- start start 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c81009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 0x7f73c8100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8111fb0 con 0x7f73c810c170 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8111e30 con 0x7f73c8102d40 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73cd73f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8112130 con 0x7f73c8103660 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54852/0 (socket says 172.21.15.16:54852) 2024-09-12T02:46:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c81009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 -- 172.21.15.16:0/2902902074 learned_addr learned my addr 172.21.15.16:0/2902902074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73bffff640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 0x7f73c8100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 -- 172.21.15.16:0/2902902074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 msgr2=0x7f73c8100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 0x7f73c8100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 -- 172.21.15.16:0/2902902074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 msgr2=0x7f73c81009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c81009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 -- 172.21.15.16:0/2902902074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c81a6930 con 0x7f73c810c170 2024-09-12T02:46:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c6ffd640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c81009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.594+0000 7f73c77fe640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f73c0004af0 tx=0x7f73c002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.595+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0034040 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.595+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c81a6bc0 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.595+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73c0030920 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.595+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c00047b0 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.595+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c81a70f0 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.596+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c8103120 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.597+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f73c0037020 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.597+0000 7f73c4ff9640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 0x7f73b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.598+0000 7f73c6ffd640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 0x7f73b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.598+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f73c00efdf0 con 0x7f73c810c170 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.598+0000 7f73c6ffd640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 0x7f73b40a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009fc0 tx=0x7f73b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.602+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f73c00bcd10 con 0x7f73c810c170 2024-09-12T02:46:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.778+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f73c81071e0 con 0x7f73c810c170 2024-09-12T02:46:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.778+0000 7f73c4ff9640 1 -- 172.21.15.16:0/2902902074 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f73c00bc610 con 0x7f73c810c170 2024-09-12T02:46:46.780 INFO:teuthology.orchestra.run.smithi016.stdout:51539607626 2024-09-12T02:46:46.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 msgr2=0x7f73b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:46.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 0x7f73b40a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009fc0 tx=0x7f73b8005f90 comp rx=0 tx=0).stop 2024-09-12T02:46:46.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 msgr2=0x7f73c8101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:46.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f73c0004af0 tx=0x7f73c002f5b0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 shutdown_connections 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73b40a2c10 0x7f73b40a50d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c810c170 0x7f73c8101420 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73c8103660 0x7f73c8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 --2- 172.21.15.16:0/2902902074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f73c8102d40 0x7f73c81009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 >> 172.21.15.16:0/2902902074 conn(0x7f73c80fc7a0 msgr2=0x7f73c80fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 shutdown_connections 2024-09-12T02:46:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:46.781+0000 7f73cd73f640 1 -- 172.21.15.16:0/2902902074 wait complete. 2024-09-12T02:46:46.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:46 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2902902074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-12T02:46:46.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:46 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2902902074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-12T02:46:47.120 INFO:tasks.cephadm.ceph_manager.ceph:need seq 94489280573 got 94489280576 for osd.3 2024-09-12T02:46:47.120 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:46 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2902902074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-12T02:46:47.766 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607624 got 51539607626 for osd.1 2024-09-12T02:46:47.767 DEBUG:teuthology.parallel:result is None 2024-09-12T02:46:47.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-12T02:46:47.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:46:48.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:47 smithi016 ceph-mon[30925]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:47 smithi033 ceph-mon[35092]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:47 smithi184 ceph-mon[35321]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:46:48.550 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 -- 172.21.15.16:0/393393017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 msgr2=0x7f37d0114f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/393393017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d0114f80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009ca0 tx=0x7f37c802a950 comp rx=0 tx=0).stop 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 -- 172.21.15.16:0/393393017 shutdown_connections 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/393393017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d0114f80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/393393017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 0x7f37d0112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/393393017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 0x7f37d0074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.126+0000 7f37d6c4c640 1 -- 172.21.15.16:0/393393017 >> 172.21.15.16:0/393393017 conn(0x7f37d00fd8c0 msgr2=0x7f37d00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.127+0000 7f37d6c4c640 1 -- 172.21.15.16:0/393393017 shutdown_connections 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.127+0000 7f37d6c4c640 1 -- 172.21.15.16:0/393393017 wait complete. 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.127+0000 7f37d6c4c640 1 Processor -- start 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.127+0000 7f37d6c4c640 1 -- start start 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 0x7f37d01a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 0x7f37d01a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0118550 con 0x7f37d0112b90 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01183d0 con 0x7f37d0074eb0 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d6c4c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01186d0 con 0x7f37d0074590 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54878/0 (socket says 172.21.15.16:54878) 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 -- 172.21.15.16:0/151019488 learned_addr learned my addr 172.21.15.16:0/151019488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d49e8640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 0x7f37d01a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37cffff640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 0x7f37d01a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 -- 172.21.15.16:0/151019488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 msgr2=0x7f37d01a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 0x7f37d01a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 -- 172.21.15.16:0/151019488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 msgr2=0x7f37d01a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 0x7f37d01a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 -- 172.21.15.16:0/151019488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d019efc0 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.128+0000 7f37d51e9640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006010 tx=0x7f37c802f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.129+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8034040 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.129+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d019f250 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.129+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37c8003e00 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.129+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8045480 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.129+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d01ab500 con 0x7f37d0112b90 2024-09-12T02:46:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.130+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d0075cd0 con 0x7f37d0112b90 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.131+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f37c8037020 con 0x7f37d0112b90 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.132+0000 7f37cdffb640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 0x7f37c00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.132+0000 7f37d49e8640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 0x7f37c00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.132+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f37c80f0110 con 0x7f37d0112b90 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.133+0000 7f37d49e8640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 0x7f37c00a5150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f37c400abe0 tx=0x7f37c400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.137+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f37c80bd0c0 con 0x7f37d0112b90 2024-09-12T02:46:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.325+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37d0062630 con 0x7f37c00a2c90 2024-09-12T02:46:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.332+0000 7f37cdffb640 1 -- 172.21.15.16:0/151019488 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258765 (secure 0 0 0) 0x7f37d0062630 con 0x7f37c00a2c90 2024-09-12T02:46:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:49.339 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:46:49.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 msgr2=0x7f37c00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:49.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 0x7f37c00a5150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f37c400abe0 tx=0x7f37c400a2d0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 msgr2=0x7f37d019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:49.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006010 tx=0x7f37c802f710 comp rx=0 tx=0).stop 2024-09-12T02:46:49.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 shutdown_connections 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f37c00a2c90 0x7f37c00a5150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37d0112b90 0x7f37d019e820 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37d0074eb0 0x7f37d01a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 --2- 172.21.15.16:0/151019488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37d0074590 0x7f37d01a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.336+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 >> 172.21.15.16:0/151019488 conn(0x7f37d00fd8c0 msgr2=0x7f37d00fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.337+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 shutdown_connections 2024-09-12T02:46:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:49.337+0000 7f37d6c4c640 1 -- 172.21.15.16:0/151019488 wait complete. 2024-09-12T02:46:49.991 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:49 smithi016 ceph-mon[30925]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:46:50.019 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-12T02:46:48.011353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657967,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":632,"num_read_kb":434,"num_write":427,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378452,"kb_used_data":12540,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563372,"statfs":{"total":1151940427776,"available":1151552892928,"internally_reserved":0,"allocated":12840960,"data_stored":7114425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":374453565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245601+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:46:01.245601+0000","last_peered":"2024-09-12T02:46:01.245601+0000","last_clean":"2024-09-12T02:46:01.245601+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:46:01.245601+0000","last_undegraded":"2024-09-12T02:46:01.245601+0000","last_fullsized":"2024-09-12T02:46:01.245601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247959+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:46:01.247959+0000","last_peered":"2024-09-12T02:46:01.247959+0000","last_clean":"2024-09-12T02:46:01.247959+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:46:01.247959+0000","last_undegraded":"2024-09-12T02:46:01.247959+0000","last_fullsized":"2024-09-12T02:46:01.247959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247397+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:46:01.247397+0000","last_peered":"2024-09-12T02:46:01.247397+0000","last_clean":"2024-09-12T02:46:01.247397+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:46:01.247397+0000","last_undegraded":"2024-09-12T02:46:01.247397+0000","last_fullsized":"2024-09-12T02:46:01.247397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247671+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:46:01.247671+0000","last_peered":"2024-09-12T02:46:01.247671+0000","last_clean":"2024-09-12T02:46:01.247671+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:46:01.247671+0000","last_undegraded":"2024-09-12T02:46:01.247671+0000","last_fullsized":"2024-09-12T02:46:01.247671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247675+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:46:01.247675+0000","last_peered":"2024-09-12T02:46:01.247675+0000","last_clean":"2024-09-12T02:46:01.247675+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:46:01.247675+0000","last_undegraded":"2024-09-12T02:46:01.247675+0000","last_fullsized":"2024-09-12T02:46:01.247675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245784+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:46:01.245784+0000","last_peered":"2024-09-12T02:46:01.245784+0000","last_clean":"2024-09-12T02:46:01.245784+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:46:01.245784+0000","last_undegraded":"2024-09-12T02:46:01.245784+0000","last_fullsized":"2024-09-12T02:46:01.245784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249851+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:46:01.249851+0000","last_peered":"2024-09-12T02:46:01.249851+0000","last_clean":"2024-09-12T02:46:01.249851+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:46:01.249851+0000","last_undegraded":"2024-09-12T02:46:01.249851+0000","last_fullsized":"2024-09-12T02:46:01.249851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244102+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:46:01.244102+0000","last_peered":"2024-09-12T02:46:01.244102+0000","last_clean":"2024-09-12T02:46:01.244102+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:46:01.244102+0000","last_undegraded":"2024-09-12T02:46:01.244102+0000","last_fullsized":"2024-09-12T02:46:01.244102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255221+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:46:01.255221+0000","last_peered":"2024-09-12T02:46:01.255221+0000","last_clean":"2024-09-12T02:46:01.255221+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:46:01.255221+0000","last_undegraded":"2024-09-12T02:46:01.255221+0000","last_fullsized":"2024-09-12T02:46:01.255221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255844+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:46:01.255844+0000","last_peered":"2024-09-12T02:46:01.255844+0000","last_clean":"2024-09-12T02:46:01.255844+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:46:01.255844+0000","last_undegraded":"2024-09-12T02:46:01.255844+0000","last_fullsized":"2024-09-12T02:46:01.255844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250813+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:46:01.250813+0000","last_peered":"2024-09-12T02:46:01.250813+0000","last_clean":"2024-09-12T02:46:01.250813+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:46:01.250813+0000","last_undegraded":"2024-09-12T02:46:01.250813+0000","last_fullsized":"2024-09-12T02:46:01.250813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247697+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:46:01.247697+0000","last_peered":"2024-09-12T02:46:01.247697+0000","last_clean":"2024-09-12T02:46:01.247697+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:46:01.247697+0000","last_undegraded":"2024-09-12T02:46:01.247697+0000","last_fullsized":"2024-09-12T02:46:01.247697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251093+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:46:01.251093+0000","last_peered":"2024-09-12T02:46:01.251093+0000","last_clean":"2024-09-12T02:46:01.251093+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:46:01.251093+0000","last_undegraded":"2024-09-12T02:46:01.251093+0000","last_fullsized":"2024-09-12T02:46:01.251093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247142+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:46:01.247142+0000","last_peered":"2024-09-12T02:46:01.247142+0000","last_clean":"2024-09-12T02:46:01.247142+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:46:01.247142+0000","last_undegraded":"2024-09-12T02:46:01.247142+0000","last_fullsized":"2024-09-12T02:46:01.247142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247412+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:46:01.247412+0000","last_peered":"2024-09-12T02:46:01.247412+0000","last_clean":"2024-09-12T02:46:01.247412+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:46:01.247412+0000","last_undegraded":"2024-09-12T02:46:01.247412+0000","last_fullsized":"2024-09-12T02:46:01.247412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254313+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:46:01.254313+0000","last_peered":"2024-09-12T02:46:01.254313+0000","last_clean":"2024-09-12T02:46:01.254313+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:46:01.254313+0000","last_undegraded":"2024-09-12T02:46:01.254313+0000","last_fullsized":"2024-09-12T02:46:01.254313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245720+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:46:01.245720+0000","last_peered":"2024-09-12T02:46:01.245720+0000","last_clean":"2024-09-12T02:46:01.245720+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:46:01.245720+0000","last_undegraded":"2024-09-12T02:46:01.245720+0000","last_fullsized":"2024-09-12T02:46:01.245720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250415+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:46:01.250415+0000","last_peered":"2024-09-12T02:46:01.250415+0000","last_clean":"2024-09-12T02:46:01.250415+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:46:01.250415+0000","last_undegraded":"2024-09-12T02:46:01.250415+0000","last_fullsized":"2024-09-12T02:46:01.250415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252979+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:46:01.252979+0000","last_peered":"2024-09-12T02:46:01.252979+0000","last_clean":"2024-09-12T02:46:01.252979+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:46:01.252979+0000","last_undegraded":"2024-09-12T02:46:01.252979+0000","last_fullsized":"2024-09-12T02:46:01.252979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245213+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:46:01.245213+0000","last_peered":"2024-09-12T02:46:01.245213+0000","last_clean":"2024-09-12T02:46:01.245213+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:46:01.245213+0000","last_undegraded":"2024-09-12T02:46:01.245213+0000","last_fullsized":"2024-09-12T02:46:01.245213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247833+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:46:01.247833+0000","last_peered":"2024-09-12T02:46:01.247833+0000","last_clean":"2024-09-12T02:46:01.247833+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:46:01.247833+0000","last_undegraded":"2024-09-12T02:46:01.247833+0000","last_fullsized":"2024-09-12T02:46:01.247833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254746+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:46:01.254746+0000","last_peered":"2024-09-12T02:46:01.254746+0000","last_clean":"2024-09-12T02:46:01.254746+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:46:01.254746+0000","last_undegraded":"2024-09-12T02:46:01.254746+0000","last_fullsized":"2024-09-12T02:46:01.254746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244169+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:46:01.244169+0000","last_peered":"2024-09-12T02:46:01.244169+0000","last_clean":"2024-09-12T02:46:01.244169+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:46:01.244169+0000","last_undegraded":"2024-09-12T02:46:01.244169+0000","last_fullsized":"2024-09-12T02:46:01.244169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248907+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:46:01.248907+0000","last_peered":"2024-09-12T02:46:01.248907+0000","last_clean":"2024-09-12T02:46:01.248907+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:46:01.248907+0000","last_undegraded":"2024-09-12T02:46:01.248907+0000","last_fullsized":"2024-09-12T02:46:01.248907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244381+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:46:01.244381+0000","last_peered":"2024-09-12T02:46:01.244381+0000","last_clean":"2024-09-12T02:46:01.244381+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:46:01.244381+0000","last_undegraded":"2024-09-12T02:46:01.244381+0000","last_fullsized":"2024-09-12T02:46:01.244381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253636+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:46:01.253636+0000","last_peered":"2024-09-12T02:46:01.253636+0000","last_clean":"2024-09-12T02:46:01.253636+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:46:01.253636+0000","last_undegraded":"2024-09-12T02:46:01.253636+0000","last_fullsized":"2024-09-12T02:46:01.253636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245512+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:46:01.245512+0000","last_peered":"2024-09-12T02:46:01.245512+0000","last_clean":"2024-09-12T02:46:01.245512+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:46:01.245512+0000","last_undegraded":"2024-09-12T02:46:01.245512+0000","last_fullsized":"2024-09-12T02:46:01.245512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247679+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:46:01.247679+0000","last_peered":"2024-09-12T02:46:01.247679+0000","last_clean":"2024-09-12T02:46:01.247679+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:46:01.247679+0000","last_undegraded":"2024-09-12T02:46:01.247679+0000","last_fullsized":"2024-09-12T02:46:01.247679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247302+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:46:01.247302+0000","last_peered":"2024-09-12T02:46:01.247302+0000","last_clean":"2024-09-12T02:46:01.247302+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:46:01.247302+0000","last_undegraded":"2024-09-12T02:46:01.247302+0000","last_fullsized":"2024-09-12T02:46:01.247302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252658+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:46:01.252658+0000","last_peered":"2024-09-12T02:46:01.252658+0000","last_clean":"2024-09-12T02:46:01.252658+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:46:01.252658+0000","last_undegraded":"2024-09-12T02:46:01.252658+0000","last_fullsized":"2024-09-12T02:46:01.252658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252938+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:46:01.252938+0000","last_peered":"2024-09-12T02:46:01.252938+0000","last_clean":"2024-09-12T02:46:01.252938+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:46:01.252938+0000","last_undegraded":"2024-09-12T02:46:01.252938+0000","last_fullsized":"2024-09-12T02:46:01.252938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245099+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:46:01.245099+0000","last_peered":"2024-09-12T02:46:01.245099+0000","last_clean":"2024-09-12T02:46:01.245099+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:46:01.245099+0000","last_undegraded":"2024-09-12T02:46:01.245099+0000","last_fullsized":"2024-09-12T02:46:01.245099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248039+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:46:01.248039+0000","last_peered":"2024-09-12T02:46:01.248039+0000","last_clean":"2024-09-12T02:46:01.248039+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:46:01.248039+0000","last_undegraded":"2024-09-12T02:46:01.248039+0000","last_fullsized":"2024-09-12T02:46:01.248039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":78,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:42.996944+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:46:42.996944+0000","last_peered":"2024-09-12T02:46:42.996944+0000","last_clean":"2024-09-12T02:46:42.996944+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:46:42.996944+0000","last_undegraded":"2024-09-12T02:46:42.996944+0000","last_fullsized":"2024-09-12T02:46:42.996944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248424+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:46:01.248424+0000","last_peered":"2024-09-12T02:46:01.248424+0000","last_clean":"2024-09-12T02:46:01.248424+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:46:01.248424+0000","last_undegraded":"2024-09-12T02:46:01.248424+0000","last_fullsized":"2024-09-12T02:46:01.248424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244158+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:46:01.244158+0000","last_peered":"2024-09-12T02:46:01.244158+0000","last_clean":"2024-09-12T02:46:01.244158+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:46:01.244158+0000","last_undegraded":"2024-09-12T02:46:01.244158+0000","last_fullsized":"2024-09-12T02:46:01.244158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248649+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:46:01.248649+0000","last_peered":"2024-09-12T02:46:01.248649+0000","last_clean":"2024-09-12T02:46:01.248649+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:46:01.248649+0000","last_undegraded":"2024-09-12T02:46:01.248649+0000","last_fullsized":"2024-09-12T02:46:01.248649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":118,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:42.996263+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:46:42.996263+0000","last_peered":"2024-09-12T02:46:42.996263+0000","last_clean":"2024-09-12T02:46:42.996263+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:46:42.996263+0000","last_undegraded":"2024-09-12T02:46:42.996263+0000","last_fullsized":"2024-09-12T02:46:42.996263+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255352+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:46:01.255352+0000","last_peered":"2024-09-12T02:46:01.255352+0000","last_clean":"2024-09-12T02:46:01.255352+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:46:01.255352+0000","last_undegraded":"2024-09-12T02:46:01.255352+0000","last_fullsized":"2024-09-12T02:46:01.255352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253009+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:46:01.253009+0000","last_peered":"2024-09-12T02:46:01.253009+0000","last_clean":"2024-09-12T02:46:01.253009+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:46:01.253009+0000","last_undegraded":"2024-09-12T02:46:01.253009+0000","last_fullsized":"2024-09-12T02:46:01.253009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254608+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:46:01.254608+0000","last_peered":"2024-09-12T02:46:01.254608+0000","last_clean":"2024-09-12T02:46:01.254608+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:46:01.254608+0000","last_undegraded":"2024-09-12T02:46:01.254608+0000","last_fullsized":"2024-09-12T02:46:01.254608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'29","reported_seq":151,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:42.996330+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:46:42.996330+0000","last_peered":"2024-09-12T02:46:42.996330+0000","last_clean":"2024-09-12T02:46:42.996330+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:46:42.996330+0000","last_undegraded":"2024-09-12T02:46:42.996330+0000","last_fullsized":"2024-09-12T02:46:42.996330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59764,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":100,"num_read_kb":108,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248286+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:46:01.248286+0000","last_peered":"2024-09-12T02:46:01.248286+0000","last_clean":"2024-09-12T02:46:01.248286+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:46:01.248286+0000","last_undegraded":"2024-09-12T02:46:01.248286+0000","last_fullsized":"2024-09-12T02:46:01.248286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247592+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:46:01.247592+0000","last_peered":"2024-09-12T02:46:01.247592+0000","last_clean":"2024-09-12T02:46:01.247592+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:46:01.247592+0000","last_undegraded":"2024-09-12T02:46:01.247592+0000","last_fullsized":"2024-09-12T02:46:01.247592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250385+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:46:01.250385+0000","last_peered":"2024-09-12T02:46:01.250385+0000","last_clean":"2024-09-12T02:46:01.250385+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:46:01.250385+0000","last_undegraded":"2024-09-12T02:46:01.250385+0000","last_fullsized":"2024-09-12T02:46:01.250385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'6","reported_seq":79,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:44.001619+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:46:44.001619+0000","last_peered":"2024-09-12T02:46:44.001619+0000","last_clean":"2024-09-12T02:46:44.001619+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:46:44.001619+0000","last_undegraded":"2024-09-12T02:46:44.001619+0000","last_fullsized":"2024-09-12T02:46:44.001619+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249800+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:46:01.249800+0000","last_peered":"2024-09-12T02:46:01.249800+0000","last_clean":"2024-09-12T02:46:01.249800+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:46:01.249800+0000","last_undegraded":"2024-09-12T02:46:01.249800+0000","last_fullsized":"2024-09-12T02:46:01.249800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247633+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:46:01.247633+0000","last_peered":"2024-09-12T02:46:01.247633+0000","last_clean":"2024-09-12T02:46:01.247633+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:46:01.247633+0000","last_undegraded":"2024-09-12T02:46:01.247633+0000","last_fullsized":"2024-09-12T02:46:01.247633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245314+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:46:01.245314+0000","last_peered":"2024-09-12T02:46:01.245314+0000","last_clean":"2024-09-12T02:46:01.245314+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:46:01.245314+0000","last_undegraded":"2024-09-12T02:46:01.245314+0000","last_fullsized":"2024-09-12T02:46:01.245314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251385+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:46:01.251385+0000","last_peered":"2024-09-12T02:46:01.251385+0000","last_clean":"2024-09-12T02:46:01.251385+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:46:01.251385+0000","last_undegraded":"2024-09-12T02:46:01.251385+0000","last_fullsized":"2024-09-12T02:46:01.251385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250603+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:46:01.250603+0000","last_peered":"2024-09-12T02:46:01.250603+0000","last_clean":"2024-09-12T02:46:01.250603+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:46:01.250603+0000","last_undegraded":"2024-09-12T02:46:01.250603+0000","last_fullsized":"2024-09-12T02:46:01.250603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:43.997939+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:46:43.997939+0000","last_peered":"2024-09-12T02:46:43.997939+0000","last_clean":"2024-09-12T02:46:43.997939+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:46:43.997939+0000","last_undegraded":"2024-09-12T02:46:43.997939+0000","last_fullsized":"2024-09-12T02:46:43.997939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250931+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:46:01.250931+0000","last_peered":"2024-09-12T02:46:01.250931+0000","last_clean":"2024-09-12T02:46:01.250931+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:46:01.250931+0000","last_undegraded":"2024-09-12T02:46:01.250931+0000","last_fullsized":"2024-09-12T02:46:01.250931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":334,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250005+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250005+0000","last_peered":"2024-09-12T02:46:01.250005+0000","last_clean":"2024-09-12T02:46:01.250005+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250005+0000","last_undegraded":"2024-09-12T02:46:01.250005+0000","last_fullsized":"2024-09-12T02:46:01.250005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244482+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:46:01.244482+0000","last_peered":"2024-09-12T02:46:01.244482+0000","last_clean":"2024-09-12T02:46:01.244482+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:46:01.244482+0000","last_undegraded":"2024-09-12T02:46:01.244482+0000","last_fullsized":"2024-09-12T02:46:01.244482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247535+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:46:01.247535+0000","last_peered":"2024-09-12T02:46:01.247535+0000","last_clean":"2024-09-12T02:46:01.247535+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:46:01.247535+0000","last_undegraded":"2024-09-12T02:46:01.247535+0000","last_fullsized":"2024-09-12T02:46:01.247535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":76,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:44.001080+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:46:44.001080+0000","last_peered":"2024-09-12T02:46:44.001080+0000","last_clean":"2024-09-12T02:46:44.001080+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:46:44.001080+0000","last_undegraded":"2024-09-12T02:46:44.001080+0000","last_fullsized":"2024-09-12T02:46:44.001080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":49,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249000+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:46:01.249000+0000","last_peered":"2024-09-12T02:46:01.249000+0000","last_clean":"2024-09-12T02:46:01.249000+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:46:01.249000+0000","last_undegraded":"2024-09-12T02:46:01.249000+0000","last_fullsized":"2024-09-12T02:46:01.249000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":76,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:42.993502+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:46:42.993502+0000","last_peered":"2024-09-12T02:46:42.993502+0000","last_clean":"2024-09-12T02:46:42.993502+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:46:42.993502+0000","last_undegraded":"2024-09-12T02:46:42.993502+0000","last_fullsized":"2024-09-12T02:46:42.993502+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251067+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:46:01.251067+0000","last_peered":"2024-09-12T02:46:01.251067+0000","last_clean":"2024-09-12T02:46:01.251067+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:46:01.251067+0000","last_undegraded":"2024-09-12T02:46:01.251067+0000","last_fullsized":"2024-09-12T02:46:01.251067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245843+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:46:01.245843+0000","last_peered":"2024-09-12T02:46:01.245843+0000","last_clean":"2024-09-12T02:46:01.245843+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:46:01.245843+0000","last_undegraded":"2024-09-12T02:46:01.245843+0000","last_fullsized":"2024-09-12T02:46:01.245843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247794+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:46:01.247794+0000","last_peered":"2024-09-12T02:46:01.247794+0000","last_clean":"2024-09-12T02:46:01.247794+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:46:01.247794+0000","last_undegraded":"2024-09-12T02:46:01.247794+0000","last_fullsized":"2024-09-12T02:46:01.247794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:11.190259+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:46:11.190259+0000","last_peered":"2024-09-12T02:46:11.190259+0000","last_clean":"2024-09-12T02:46:11.190259+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:46:11.190259+0000","last_undegraded":"2024-09-12T02:46:11.190259+0000","last_fullsized":"2024-09-12T02:46:11.190259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244430+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:46:01.244430+0000","last_peered":"2024-09-12T02:46:01.244430+0000","last_clean":"2024-09-12T02:46:01.244430+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:46:01.244430+0000","last_undegraded":"2024-09-12T02:46:01.244430+0000","last_fullsized":"2024-09-12T02:46:01.244430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247494+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:46:01.247494+0000","last_peered":"2024-09-12T02:46:01.247494+0000","last_clean":"2024-09-12T02:46:01.247494+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:46:01.247494+0000","last_undegraded":"2024-09-12T02:46:01.247494+0000","last_fullsized":"2024-09-12T02:46:01.247494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248263+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:46:01.248263+0000","last_peered":"2024-09-12T02:46:01.248263+0000","last_clean":"2024-09-12T02:46:01.248263+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:46:01.248263+0000","last_undegraded":"2024-09-12T02:46:01.248263+0000","last_fullsized":"2024-09-12T02:46:01.248263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245197+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:46:01.245197+0000","last_peered":"2024-09-12T02:46:01.245197+0000","last_clean":"2024-09-12T02:46:01.245197+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:46:01.245197+0000","last_undegraded":"2024-09-12T02:46:01.245197+0000","last_fullsized":"2024-09-12T02:46:01.245197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244361+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:46:01.244361+0000","last_peered":"2024-09-12T02:46:01.244361+0000","last_clean":"2024-09-12T02:46:01.244361+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:46:01.244361+0000","last_undegraded":"2024-09-12T02:46:01.244361+0000","last_fullsized":"2024-09-12T02:46:01.244361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250135+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:46:01.250135+0000","last_peered":"2024-09-12T02:46:01.250135+0000","last_clean":"2024-09-12T02:46:01.250135+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:46:01.250135+0000","last_undegraded":"2024-09-12T02:46:01.250135+0000","last_fullsized":"2024-09-12T02:46:01.250135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247854+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:46:01.247854+0000","last_peered":"2024-09-12T02:46:01.247854+0000","last_clean":"2024-09-12T02:46:01.247854+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:46:01.247854+0000","last_undegraded":"2024-09-12T02:46:01.247854+0000","last_fullsized":"2024-09-12T02:46:01.247854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254491+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:46:01.254491+0000","last_peered":"2024-09-12T02:46:01.254491+0000","last_clean":"2024-09-12T02:46:01.254491+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:46:01.254491+0000","last_undegraded":"2024-09-12T02:46:01.254491+0000","last_fullsized":"2024-09-12T02:46:01.254491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247259+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:46:01.247259+0000","last_peered":"2024-09-12T02:46:01.247259+0000","last_clean":"2024-09-12T02:46:01.247259+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:46:01.247259+0000","last_undegraded":"2024-09-12T02:46:01.247259+0000","last_fullsized":"2024-09-12T02:46:01.247259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244615+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:46:01.244615+0000","last_peered":"2024-09-12T02:46:01.244615+0000","last_clean":"2024-09-12T02:46:01.244615+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:46:01.244615+0000","last_undegraded":"2024-09-12T02:46:01.244615+0000","last_fullsized":"2024-09-12T02:46:01.244615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":82,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:42.992845+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:46:42.992845+0000","last_peered":"2024-09-12T02:46:42.992845+0000","last_clean":"2024-09-12T02:46:42.992845+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:46:42.992845+0000","last_undegraded":"2024-09-12T02:46:42.992845+0000","last_fullsized":"2024-09-12T02:46:42.992845+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":47,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67365,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":506,"num_read_kb":325,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645337,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1080755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005091,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462957,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659186,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":429609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888119,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051643,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":892928,"data_stored":429470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280577,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476806,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":490506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607627,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":490286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40404,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704748,"statfs":{"total":95995035648,"available":95953661952,"internally_reserved":0,"allocated":913408,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:46:50.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:46:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:49 smithi033 ceph-mon[35092]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:46:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:49 smithi184 ceph-mon[35321]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:46:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:50.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- 172.21.15.16:0/3912757025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 msgr2=0x7f2470106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:50.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- 172.21.15.16:0/3912757025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f2470106e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f246c009930 tx=0x7f246c02a420 comp rx=0 tx=0).stop 2024-09-12T02:46:50.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- 172.21.15.16:0/3912757025 shutdown_connections 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- 172.21.15.16:0/3912757025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470101420 0x7f247010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- 172.21.15.16:0/3912757025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470100a80 0x7f2470100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- 172.21.15.16:0/3912757025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f2470106e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- 172.21.15.16:0/3912757025 >> 172.21.15.16:0/3912757025 conn(0x7f24700fc7a0 msgr2=0x7f24700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- 172.21.15.16:0/3912757025 shutdown_connections 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- 172.21.15.16:0/3912757025 wait complete. 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 Processor -- start 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- start start 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470100a80 0x7f24701991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470101420 0x7f2470199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2470110e80 con 0x7f2470106a80 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2470110d00 con 0x7f2470100a80 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2477e09640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2470111000 con 0x7f2470101420 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f24763a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f24763a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54896/0 (socket says 172.21.15.16:54896) 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f24763a6640 1 -- 172.21.15.16:0/2588367562 learned_addr learned my addr 172.21.15.16:0/2588367562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f24753a4640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470101420 0x7f2470199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.748+0000 7f2475ba5640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470100a80 0x7f24701991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 -- 172.21.15.16:0/2588367562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470101420 msgr2=0x7f2470199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470101420 0x7f2470199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 -- 172.21.15.16:0/2588367562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470100a80 msgr2=0x7f24701991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470100a80 0x7f24701991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 -- 172.21.15.16:0/2588367562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24701a47b0 con 0x7f2470106a80 2024-09-12T02:46:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f24763a6640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f246800cce0 tx=0x7f246800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246800f450 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f246800fad0 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468018c10 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24701a4980 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.749+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24701a4ec0 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.750+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2470100ee0 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.752+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f246800f5f0 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.753+0000 7f2466ffd640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 0x7f245c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.753+0000 7f2475ba5640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 0x7f245c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.753+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f24680cf7d0 con 0x7f2470106a80 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.753+0000 7f2475ba5640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 0x7f245c0a5150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f246c000c00 tx=0x7f246c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.757+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2468014030 con 0x7f2470106a80 2024-09-12T02:46:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.945+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f24700fe5f0 con 0x7f245c0a2c90 2024-09-12T02:46:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.951+0000 7f2466ffd640 1 -- 172.21.15.16:0/2588367562 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258767 (secure 0 0 0) 0x7f24700fe5f0 con 0x7f245c0a2c90 2024-09-12T02:46:50.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 msgr2=0x7f245c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 0x7f245c0a5150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f246c000c00 tx=0x7f246c005a20 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 msgr2=0x7f24701a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f246800cce0 tx=0x7f246800a620 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 shutdown_connections 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f245c0a2c90 0x7f245c0a5150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2470106a80 0x7f24701a4270 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2470101420 0x7f2470199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 --2- 172.21.15.16:0/2588367562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2470100a80 0x7f24701991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 >> 172.21.15.16:0/2588367562 conn(0x7f24700fc7a0 msgr2=0x7f24700fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 shutdown_connections 2024-09-12T02:46:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:50.955+0000 7f2477e09640 1 -- 172.21.15.16:0/2588367562 wait complete. 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:46:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:46:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:46:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:46:51.619 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-12T02:46:50.011770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657967,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":672,"num_read_kb":459,"num_write":427,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378452,"kb_used_data":12540,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563372,"statfs":{"total":1151940427776,"available":1151552892928,"internally_reserved":0,"allocated":12840960,"data_stored":7114425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":374453565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":75,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245601+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:46:01.245601+0000","last_peered":"2024-09-12T02:46:01.245601+0000","last_clean":"2024-09-12T02:46:01.245601+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:46:01.245601+0000","last_undegraded":"2024-09-12T02:46:01.245601+0000","last_fullsized":"2024-09-12T02:46:01.245601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247959+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:46:01.247959+0000","last_peered":"2024-09-12T02:46:01.247959+0000","last_clean":"2024-09-12T02:46:01.247959+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:46:01.247959+0000","last_undegraded":"2024-09-12T02:46:01.247959+0000","last_fullsized":"2024-09-12T02:46:01.247959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247397+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:46:01.247397+0000","last_peered":"2024-09-12T02:46:01.247397+0000","last_clean":"2024-09-12T02:46:01.247397+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:46:01.247397+0000","last_undegraded":"2024-09-12T02:46:01.247397+0000","last_fullsized":"2024-09-12T02:46:01.247397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247671+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:46:01.247671+0000","last_peered":"2024-09-12T02:46:01.247671+0000","last_clean":"2024-09-12T02:46:01.247671+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:46:01.247671+0000","last_undegraded":"2024-09-12T02:46:01.247671+0000","last_fullsized":"2024-09-12T02:46:01.247671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247675+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:46:01.247675+0000","last_peered":"2024-09-12T02:46:01.247675+0000","last_clean":"2024-09-12T02:46:01.247675+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:46:01.247675+0000","last_undegraded":"2024-09-12T02:46:01.247675+0000","last_fullsized":"2024-09-12T02:46:01.247675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245784+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:46:01.245784+0000","last_peered":"2024-09-12T02:46:01.245784+0000","last_clean":"2024-09-12T02:46:01.245784+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:46:01.245784+0000","last_undegraded":"2024-09-12T02:46:01.245784+0000","last_fullsized":"2024-09-12T02:46:01.245784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249851+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:46:01.249851+0000","last_peered":"2024-09-12T02:46:01.249851+0000","last_clean":"2024-09-12T02:46:01.249851+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:46:01.249851+0000","last_undegraded":"2024-09-12T02:46:01.249851+0000","last_fullsized":"2024-09-12T02:46:01.249851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244102+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:46:01.244102+0000","last_peered":"2024-09-12T02:46:01.244102+0000","last_clean":"2024-09-12T02:46:01.244102+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:46:01.244102+0000","last_undegraded":"2024-09-12T02:46:01.244102+0000","last_fullsized":"2024-09-12T02:46:01.244102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255221+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:46:01.255221+0000","last_peered":"2024-09-12T02:46:01.255221+0000","last_clean":"2024-09-12T02:46:01.255221+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:46:01.255221+0000","last_undegraded":"2024-09-12T02:46:01.255221+0000","last_fullsized":"2024-09-12T02:46:01.255221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255844+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:46:01.255844+0000","last_peered":"2024-09-12T02:46:01.255844+0000","last_clean":"2024-09-12T02:46:01.255844+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:46:01.255844+0000","last_undegraded":"2024-09-12T02:46:01.255844+0000","last_fullsized":"2024-09-12T02:46:01.255844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250813+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:46:01.250813+0000","last_peered":"2024-09-12T02:46:01.250813+0000","last_clean":"2024-09-12T02:46:01.250813+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:46:01.250813+0000","last_undegraded":"2024-09-12T02:46:01.250813+0000","last_fullsized":"2024-09-12T02:46:01.250813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247697+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:46:01.247697+0000","last_peered":"2024-09-12T02:46:01.247697+0000","last_clean":"2024-09-12T02:46:01.247697+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:46:01.247697+0000","last_undegraded":"2024-09-12T02:46:01.247697+0000","last_fullsized":"2024-09-12T02:46:01.247697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251093+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:46:01.251093+0000","last_peered":"2024-09-12T02:46:01.251093+0000","last_clean":"2024-09-12T02:46:01.251093+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:46:01.251093+0000","last_undegraded":"2024-09-12T02:46:01.251093+0000","last_fullsized":"2024-09-12T02:46:01.251093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247142+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:46:01.247142+0000","last_peered":"2024-09-12T02:46:01.247142+0000","last_clean":"2024-09-12T02:46:01.247142+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:46:01.247142+0000","last_undegraded":"2024-09-12T02:46:01.247142+0000","last_fullsized":"2024-09-12T02:46:01.247142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247412+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:46:01.247412+0000","last_peered":"2024-09-12T02:46:01.247412+0000","last_clean":"2024-09-12T02:46:01.247412+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:46:01.247412+0000","last_undegraded":"2024-09-12T02:46:01.247412+0000","last_fullsized":"2024-09-12T02:46:01.247412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254313+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:46:01.254313+0000","last_peered":"2024-09-12T02:46:01.254313+0000","last_clean":"2024-09-12T02:46:01.254313+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:46:01.254313+0000","last_undegraded":"2024-09-12T02:46:01.254313+0000","last_fullsized":"2024-09-12T02:46:01.254313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245720+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:46:01.245720+0000","last_peered":"2024-09-12T02:46:01.245720+0000","last_clean":"2024-09-12T02:46:01.245720+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:46:01.245720+0000","last_undegraded":"2024-09-12T02:46:01.245720+0000","last_fullsized":"2024-09-12T02:46:01.245720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250415+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:46:01.250415+0000","last_peered":"2024-09-12T02:46:01.250415+0000","last_clean":"2024-09-12T02:46:01.250415+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:46:01.250415+0000","last_undegraded":"2024-09-12T02:46:01.250415+0000","last_fullsized":"2024-09-12T02:46:01.250415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252979+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:46:01.252979+0000","last_peered":"2024-09-12T02:46:01.252979+0000","last_clean":"2024-09-12T02:46:01.252979+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:46:01.252979+0000","last_undegraded":"2024-09-12T02:46:01.252979+0000","last_fullsized":"2024-09-12T02:46:01.252979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245213+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:46:01.245213+0000","last_peered":"2024-09-12T02:46:01.245213+0000","last_clean":"2024-09-12T02:46:01.245213+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:46:01.245213+0000","last_undegraded":"2024-09-12T02:46:01.245213+0000","last_fullsized":"2024-09-12T02:46:01.245213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247833+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:46:01.247833+0000","last_peered":"2024-09-12T02:46:01.247833+0000","last_clean":"2024-09-12T02:46:01.247833+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:46:01.247833+0000","last_undegraded":"2024-09-12T02:46:01.247833+0000","last_fullsized":"2024-09-12T02:46:01.247833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254746+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:46:01.254746+0000","last_peered":"2024-09-12T02:46:01.254746+0000","last_clean":"2024-09-12T02:46:01.254746+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:46:01.254746+0000","last_undegraded":"2024-09-12T02:46:01.254746+0000","last_fullsized":"2024-09-12T02:46:01.254746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244169+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:46:01.244169+0000","last_peered":"2024-09-12T02:46:01.244169+0000","last_clean":"2024-09-12T02:46:01.244169+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:46:01.244169+0000","last_undegraded":"2024-09-12T02:46:01.244169+0000","last_fullsized":"2024-09-12T02:46:01.244169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248907+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:46:01.248907+0000","last_peered":"2024-09-12T02:46:01.248907+0000","last_clean":"2024-09-12T02:46:01.248907+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:46:01.248907+0000","last_undegraded":"2024-09-12T02:46:01.248907+0000","last_fullsized":"2024-09-12T02:46:01.248907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244381+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:46:01.244381+0000","last_peered":"2024-09-12T02:46:01.244381+0000","last_clean":"2024-09-12T02:46:01.244381+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:46:01.244381+0000","last_undegraded":"2024-09-12T02:46:01.244381+0000","last_fullsized":"2024-09-12T02:46:01.244381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253636+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:46:01.253636+0000","last_peered":"2024-09-12T02:46:01.253636+0000","last_clean":"2024-09-12T02:46:01.253636+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:46:01.253636+0000","last_undegraded":"2024-09-12T02:46:01.253636+0000","last_fullsized":"2024-09-12T02:46:01.253636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245512+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:46:01.245512+0000","last_peered":"2024-09-12T02:46:01.245512+0000","last_clean":"2024-09-12T02:46:01.245512+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:46:01.245512+0000","last_undegraded":"2024-09-12T02:46:01.245512+0000","last_fullsized":"2024-09-12T02:46:01.245512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247679+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:46:01.247679+0000","last_peered":"2024-09-12T02:46:01.247679+0000","last_clean":"2024-09-12T02:46:01.247679+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:46:01.247679+0000","last_undegraded":"2024-09-12T02:46:01.247679+0000","last_fullsized":"2024-09-12T02:46:01.247679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247302+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:46:01.247302+0000","last_peered":"2024-09-12T02:46:01.247302+0000","last_clean":"2024-09-12T02:46:01.247302+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:46:01.247302+0000","last_undegraded":"2024-09-12T02:46:01.247302+0000","last_fullsized":"2024-09-12T02:46:01.247302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252658+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:46:01.252658+0000","last_peered":"2024-09-12T02:46:01.252658+0000","last_clean":"2024-09-12T02:46:01.252658+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:46:01.252658+0000","last_undegraded":"2024-09-12T02:46:01.252658+0000","last_fullsized":"2024-09-12T02:46:01.252658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.252938+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:46:01.252938+0000","last_peered":"2024-09-12T02:46:01.252938+0000","last_clean":"2024-09-12T02:46:01.252938+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:46:01.252938+0000","last_undegraded":"2024-09-12T02:46:01.252938+0000","last_fullsized":"2024-09-12T02:46:01.252938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245099+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:46:01.245099+0000","last_peered":"2024-09-12T02:46:01.245099+0000","last_clean":"2024-09-12T02:46:01.245099+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:46:01.245099+0000","last_undegraded":"2024-09-12T02:46:01.245099+0000","last_fullsized":"2024-09-12T02:46:01.245099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248039+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:46:01.248039+0000","last_peered":"2024-09-12T02:46:01.248039+0000","last_clean":"2024-09-12T02:46:01.248039+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:46:01.248039+0000","last_undegraded":"2024-09-12T02:46:01.248039+0000","last_fullsized":"2024-09-12T02:46:01.248039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:48.003345+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:46:48.003345+0000","last_peered":"2024-09-12T02:46:48.003345+0000","last_clean":"2024-09-12T02:46:48.003345+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:46:48.003345+0000","last_undegraded":"2024-09-12T02:46:48.003345+0000","last_fullsized":"2024-09-12T02:46:48.003345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248424+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:46:01.248424+0000","last_peered":"2024-09-12T02:46:01.248424+0000","last_clean":"2024-09-12T02:46:01.248424+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:46:01.248424+0000","last_undegraded":"2024-09-12T02:46:01.248424+0000","last_fullsized":"2024-09-12T02:46:01.248424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244158+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:46:01.244158+0000","last_peered":"2024-09-12T02:46:01.244158+0000","last_clean":"2024-09-12T02:46:01.244158+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:46:01.244158+0000","last_undegraded":"2024-09-12T02:46:01.244158+0000","last_fullsized":"2024-09-12T02:46:01.244158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248649+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:46:01.248649+0000","last_peered":"2024-09-12T02:46:01.248649+0000","last_clean":"2024-09-12T02:46:01.248649+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:46:01.248649+0000","last_undegraded":"2024-09-12T02:46:01.248649+0000","last_fullsized":"2024-09-12T02:46:01.248649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:48.002586+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:46:48.002586+0000","last_peered":"2024-09-12T02:46:48.002586+0000","last_clean":"2024-09-12T02:46:48.002586+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:46:48.002586+0000","last_undegraded":"2024-09-12T02:46:48.002586+0000","last_fullsized":"2024-09-12T02:46:48.002586+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.255352+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:46:01.255352+0000","last_peered":"2024-09-12T02:46:01.255352+0000","last_clean":"2024-09-12T02:46:01.255352+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:46:01.255352+0000","last_undegraded":"2024-09-12T02:46:01.255352+0000","last_fullsized":"2024-09-12T02:46:01.255352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253009+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:46:01.253009+0000","last_peered":"2024-09-12T02:46:01.253009+0000","last_clean":"2024-09-12T02:46:01.253009+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:46:01.253009+0000","last_undegraded":"2024-09-12T02:46:01.253009+0000","last_fullsized":"2024-09-12T02:46:01.253009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254608+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:46:01.254608+0000","last_peered":"2024-09-12T02:46:01.254608+0000","last_clean":"2024-09-12T02:46:01.254608+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:46:01.254608+0000","last_undegraded":"2024-09-12T02:46:01.254608+0000","last_fullsized":"2024-09-12T02:46:01.254608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'29","reported_seq":161,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:48.002500+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:46:48.002500+0000","last_peered":"2024-09-12T02:46:48.002500+0000","last_clean":"2024-09-12T02:46:48.002500+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:46:48.002500+0000","last_undegraded":"2024-09-12T02:46:48.002500+0000","last_fullsized":"2024-09-12T02:46:48.002500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59764,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":113,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248286+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:46:01.248286+0000","last_peered":"2024-09-12T02:46:01.248286+0000","last_clean":"2024-09-12T02:46:01.248286+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:46:01.248286+0000","last_undegraded":"2024-09-12T02:46:01.248286+0000","last_fullsized":"2024-09-12T02:46:01.248286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247592+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:46:01.247592+0000","last_peered":"2024-09-12T02:46:01.247592+0000","last_clean":"2024-09-12T02:46:01.247592+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:46:01.247592+0000","last_undegraded":"2024-09-12T02:46:01.247592+0000","last_fullsized":"2024-09-12T02:46:01.247592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250385+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:46:01.250385+0000","last_peered":"2024-09-12T02:46:01.250385+0000","last_clean":"2024-09-12T02:46:01.250385+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:46:01.250385+0000","last_undegraded":"2024-09-12T02:46:01.250385+0000","last_fullsized":"2024-09-12T02:46:01.250385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'6","reported_seq":79,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:44.001619+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:46:44.001619+0000","last_peered":"2024-09-12T02:46:44.001619+0000","last_clean":"2024-09-12T02:46:44.001619+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:46:44.001619+0000","last_undegraded":"2024-09-12T02:46:44.001619+0000","last_fullsized":"2024-09-12T02:46:44.001619+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249800+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:46:01.249800+0000","last_peered":"2024-09-12T02:46:01.249800+0000","last_clean":"2024-09-12T02:46:01.249800+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:46:01.249800+0000","last_undegraded":"2024-09-12T02:46:01.249800+0000","last_fullsized":"2024-09-12T02:46:01.249800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247633+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:46:01.247633+0000","last_peered":"2024-09-12T02:46:01.247633+0000","last_clean":"2024-09-12T02:46:01.247633+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:46:01.247633+0000","last_undegraded":"2024-09-12T02:46:01.247633+0000","last_fullsized":"2024-09-12T02:46:01.247633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245314+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:46:01.245314+0000","last_peered":"2024-09-12T02:46:01.245314+0000","last_clean":"2024-09-12T02:46:01.245314+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:46:01.245314+0000","last_undegraded":"2024-09-12T02:46:01.245314+0000","last_fullsized":"2024-09-12T02:46:01.245314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251385+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:46:01.251385+0000","last_peered":"2024-09-12T02:46:01.251385+0000","last_clean":"2024-09-12T02:46:01.251385+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:46:01.251385+0000","last_undegraded":"2024-09-12T02:46:01.251385+0000","last_fullsized":"2024-09-12T02:46:01.251385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250603+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:46:01.250603+0000","last_peered":"2024-09-12T02:46:01.250603+0000","last_clean":"2024-09-12T02:46:01.250603+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:46:01.250603+0000","last_undegraded":"2024-09-12T02:46:01.250603+0000","last_fullsized":"2024-09-12T02:46:01.250603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:43.997939+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:46:43.997939+0000","last_peered":"2024-09-12T02:46:43.997939+0000","last_clean":"2024-09-12T02:46:43.997939+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:46:43.997939+0000","last_undegraded":"2024-09-12T02:46:43.997939+0000","last_fullsized":"2024-09-12T02:46:43.997939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250931+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:46:01.250931+0000","last_peered":"2024-09-12T02:46:01.250931+0000","last_clean":"2024-09-12T02:46:01.250931+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:46:01.250931+0000","last_undegraded":"2024-09-12T02:46:01.250931+0000","last_fullsized":"2024-09-12T02:46:01.250931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":334,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250005+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250005+0000","last_peered":"2024-09-12T02:46:01.250005+0000","last_clean":"2024-09-12T02:46:01.250005+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250005+0000","last_undegraded":"2024-09-12T02:46:01.250005+0000","last_fullsized":"2024-09-12T02:46:01.250005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244482+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:46:01.244482+0000","last_peered":"2024-09-12T02:46:01.244482+0000","last_clean":"2024-09-12T02:46:01.244482+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:46:01.244482+0000","last_undegraded":"2024-09-12T02:46:01.244482+0000","last_fullsized":"2024-09-12T02:46:01.244482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247535+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:46:01.247535+0000","last_peered":"2024-09-12T02:46:01.247535+0000","last_clean":"2024-09-12T02:46:01.247535+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:46:01.247535+0000","last_undegraded":"2024-09-12T02:46:01.247535+0000","last_fullsized":"2024-09-12T02:46:01.247535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"78'5","reported_seq":81,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:49.007530+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:46:49.007530+0000","last_peered":"2024-09-12T02:46:49.007530+0000","last_clean":"2024-09-12T02:46:49.007530+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:46:49.007530+0000","last_undegraded":"2024-09-12T02:46:49.007530+0000","last_fullsized":"2024-09-12T02:46:49.007530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249000+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:46:01.249000+0000","last_peered":"2024-09-12T02:46:01.249000+0000","last_clean":"2024-09-12T02:46:01.249000+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:46:01.249000+0000","last_undegraded":"2024-09-12T02:46:01.249000+0000","last_fullsized":"2024-09-12T02:46:01.249000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":81,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:48.000264+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:46:48.000264+0000","last_peered":"2024-09-12T02:46:48.000264+0000","last_clean":"2024-09-12T02:46:48.000264+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:46:48.000264+0000","last_undegraded":"2024-09-12T02:46:48.000264+0000","last_fullsized":"2024-09-12T02:46:48.000264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.251067+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:46:01.251067+0000","last_peered":"2024-09-12T02:46:01.251067+0000","last_clean":"2024-09-12T02:46:01.251067+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:46:01.251067+0000","last_undegraded":"2024-09-12T02:46:01.251067+0000","last_fullsized":"2024-09-12T02:46:01.251067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245843+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:46:01.245843+0000","last_peered":"2024-09-12T02:46:01.245843+0000","last_clean":"2024-09-12T02:46:01.245843+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:46:01.245843+0000","last_undegraded":"2024-09-12T02:46:01.245843+0000","last_fullsized":"2024-09-12T02:46:01.245843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247794+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:46:01.247794+0000","last_peered":"2024-09-12T02:46:01.247794+0000","last_clean":"2024-09-12T02:46:01.247794+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:46:01.247794+0000","last_undegraded":"2024-09-12T02:46:01.247794+0000","last_fullsized":"2024-09-12T02:46:01.247794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:11.190259+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:46:11.190259+0000","last_peered":"2024-09-12T02:46:11.190259+0000","last_clean":"2024-09-12T02:46:11.190259+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:46:11.190259+0000","last_undegraded":"2024-09-12T02:46:11.190259+0000","last_fullsized":"2024-09-12T02:46:11.190259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244430+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:46:01.244430+0000","last_peered":"2024-09-12T02:46:01.244430+0000","last_clean":"2024-09-12T02:46:01.244430+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:46:01.244430+0000","last_undegraded":"2024-09-12T02:46:01.244430+0000","last_fullsized":"2024-09-12T02:46:01.244430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247494+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:46:01.247494+0000","last_peered":"2024-09-12T02:46:01.247494+0000","last_clean":"2024-09-12T02:46:01.247494+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:46:01.247494+0000","last_undegraded":"2024-09-12T02:46:01.247494+0000","last_fullsized":"2024-09-12T02:46:01.247494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248263+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:46:01.248263+0000","last_peered":"2024-09-12T02:46:01.248263+0000","last_clean":"2024-09-12T02:46:01.248263+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:46:01.248263+0000","last_undegraded":"2024-09-12T02:46:01.248263+0000","last_fullsized":"2024-09-12T02:46:01.248263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245197+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:46:01.245197+0000","last_peered":"2024-09-12T02:46:01.245197+0000","last_clean":"2024-09-12T02:46:01.245197+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:46:01.245197+0000","last_undegraded":"2024-09-12T02:46:01.245197+0000","last_fullsized":"2024-09-12T02:46:01.245197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244361+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:46:01.244361+0000","last_peered":"2024-09-12T02:46:01.244361+0000","last_clean":"2024-09-12T02:46:01.244361+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:46:01.244361+0000","last_undegraded":"2024-09-12T02:46:01.244361+0000","last_fullsized":"2024-09-12T02:46:01.244361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250135+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:46:01.250135+0000","last_peered":"2024-09-12T02:46:01.250135+0000","last_clean":"2024-09-12T02:46:01.250135+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:46:01.250135+0000","last_undegraded":"2024-09-12T02:46:01.250135+0000","last_fullsized":"2024-09-12T02:46:01.250135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247854+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:46:01.247854+0000","last_peered":"2024-09-12T02:46:01.247854+0000","last_clean":"2024-09-12T02:46:01.247854+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:46:01.247854+0000","last_undegraded":"2024-09-12T02:46:01.247854+0000","last_fullsized":"2024-09-12T02:46:01.247854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254491+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:46:01.254491+0000","last_peered":"2024-09-12T02:46:01.254491+0000","last_clean":"2024-09-12T02:46:01.254491+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:46:01.254491+0000","last_undegraded":"2024-09-12T02:46:01.254491+0000","last_fullsized":"2024-09-12T02:46:01.254491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247259+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:46:01.247259+0000","last_peered":"2024-09-12T02:46:01.247259+0000","last_clean":"2024-09-12T02:46:01.247259+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:46:01.247259+0000","last_undegraded":"2024-09-12T02:46:01.247259+0000","last_fullsized":"2024-09-12T02:46:01.247259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244615+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:46:01.244615+0000","last_peered":"2024-09-12T02:46:01.244615+0000","last_clean":"2024-09-12T02:46:01.244615+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:46:01.244615+0000","last_undegraded":"2024-09-12T02:46:01.244615+0000","last_fullsized":"2024-09-12T02:46:01.244615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"78'13","reported_seq":87,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:47.999440+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:46:47.999440+0000","last_peered":"2024-09-12T02:46:47.999440+0000","last_clean":"2024-09-12T02:46:47.999440+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:46:47.999440+0000","last_undegraded":"2024-09-12T02:46:47.999440+0000","last_fullsized":"2024-09-12T02:46:47.999440+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67365,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":546,"num_read_kb":350,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645338,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1080755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005092,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462958,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659187,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":429609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888120,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051643,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":892928,"data_stored":429470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280577,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476806,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":490506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607627,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":490286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40404,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704748,"statfs":{"total":95995035648,"available":95953661952,"internally_reserved":0,"allocated":913408,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:46:51.624 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-12T02:46:51.624 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-12T02:46:51.624 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-12T02:46:51.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-12T02:46:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:52.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:51 smithi016 ceph-mon[30925]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:46:52.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:51 smithi016 ceph-mon[30925]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:51 smithi033 ceph-mon[35092]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:46:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:51 smithi033 ceph-mon[35092]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:51 smithi184 ceph-mon[35321]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:46:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:51 smithi184 ceph-mon[35321]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 -- 172.21.15.16:0/3551781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 msgr2=0x7f2f201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 --2- 172.21.15.16:0/3551781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f201067e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0802a410 comp rx=0 tx=0).stop 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 -- 172.21.15.16:0/3551781664 shutdown_connections 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 --2- 172.21.15.16:0/3551781664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 0x7f2f2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 --2- 172.21.15.16:0/3551781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f201067e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 --2- 172.21.15.16:0/3551781664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 0x7f2f201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.332+0000 7f2f26327640 1 -- 172.21.15.16:0/3551781664 >> 172.21.15.16:0/3551781664 conn(0x7f2f200fd8c0 msgr2=0x7f2f200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- 172.21.15.16:0/3551781664 shutdown_connections 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- 172.21.15.16:0/3551781664 wait complete. 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 Processor -- start 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- start start 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 0x7f2f20198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 0x7f2f2019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f20111f70 con 0x7f2f20102790 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f20111df0 con 0x7f2f20101de0 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f26327640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f201120f0 con 0x7f2f20106f10 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54914/0 (socket says 172.21.15.16:54914) 2024-09-12T02:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.333+0000 7f2f17fff640 1 -- 172.21.15.16:0/2800091042 learned_addr learned my addr 172.21.15.16:0/2800091042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f1ffff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 0x7f2f20198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f248c4640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 0x7f2f2019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 -- 172.21.15.16:0/2800091042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 msgr2=0x7f2f2019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 0x7f2f2019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 -- 172.21.15.16:0/2800091042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 msgr2=0x7f2f20198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 0x7f2f20198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 -- 172.21.15.16:0/2800091042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2019d3c0 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006010 tx=0x7f2f080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0803f030 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f08003ce0 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2019d650 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08035070 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.334+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f201a4f90 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.335+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f201032c0 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.337+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2f08042030 con 0x7f2f20102790 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.337+0000 7f2f1dffb640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 0x7f2f0c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.337+0000 7f2f1ffff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 0x7f2f0c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.337+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2f080ef9e0 con 0x7f2f20102790 2024-09-12T02:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.338+0000 7f2f1ffff640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 0x7f2f0c0a5150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000bbe0 tx=0x7f2f1000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.340+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2f080bc260 con 0x7f2f20102790 2024-09-12T02:46:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.554+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2f201021c0 con 0x7f2f20102790 2024-09-12T02:46:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.555+0000 7f2f1dffb640 1 -- 172.21.15.16:0/2800091042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f2f201021c0 con 0x7f2f20102790 2024-09-12T02:46:52.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:52.555 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 msgr2=0x7f2f0c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 0x7f2f0c0a5150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000bbe0 tx=0x7f2f1000b390 comp rx=0 tx=0).stop 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 msgr2=0x7f2f20198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006010 tx=0x7f2f080042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 shutdown_connections 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f20106f10 0x7f2f2019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f0c0a2c90 0x7f2f0c0a5150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f20102790 0x7f2f20198950 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 --2- 172.21.15.16:0/2800091042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f20101de0 0x7f2f20198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 >> 172.21.15.16:0/2800091042 conn(0x7f2f200fd8c0 msgr2=0x7f2f200fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 shutdown_connections 2024-09-12T02:46:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:52.557+0000 7f2f26327640 1 -- 172.21.15.16:0/2800091042 wait complete. 2024-09-12T02:46:53.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2800091042' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-12T02:46:53.210 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-12T02:46:53.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:46:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2800091042' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-12T02:46:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2800091042' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-12T02:46:53.468 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:53.843 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:53 smithi016 ceph-mon[30925]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:53.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 -- 172.21.15.16:0/1496031952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 msgr2=0x7fdadc103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:53.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 --2- 172.21.15.16:0/1496031952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc103120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009900 tx=0x7fdacc02a480 comp rx=0 tx=0).stop 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 -- 172.21.15.16:0/1496031952 shutdown_connections 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 --2- 172.21.15.16:0/1496031952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc10c170 0x7fdadc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 --2- 172.21.15.16:0/1496031952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc103660 0x7fdadc10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 --2- 172.21.15.16:0/1496031952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc103120 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 -- 172.21.15.16:0/1496031952 >> 172.21.15.16:0/1496031952 conn(0x7fdadc0fc7a0 msgr2=0x7fdadc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 -- 172.21.15.16:0/1496031952 shutdown_connections 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.924+0000 7fdae0877640 1 -- 172.21.15.16:0/1496031952 wait complete. 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 Processor -- start 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 -- start start 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc103660 0x7fdadc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc10c170 0x7fdadc19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc111f70 con 0x7fdadc102d40 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc111df0 con 0x7fdadc10c170 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdae0877640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc1120f0 con 0x7fdadc103660 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.925+0000 7fdada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54938/0 (socket says 172.21.15.16:54938) 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 -- 172.21.15.16:0/3278241105 learned_addr learned my addr 172.21.15.16:0/3278241105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdadad9d640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc10c170 0x7fdadc19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdad9d9b640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc103660 0x7fdadc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 -- 172.21.15.16:0/3278241105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc103660 msgr2=0x7fdadc1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc103660 0x7fdadc1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 -- 172.21.15.16:0/3278241105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc10c170 msgr2=0x7fdadc19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc10c170 0x7fdadc19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 -- 172.21.15.16:0/3278241105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc19d420 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdada59c640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdacc006010 tx=0x7fdacc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc03f030 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdacc003ce0 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdadc19d6b0 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc035070 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.926+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdadc1a4ff0 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.927+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdadc06b070 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.929+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdacc042030 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.929+0000 7fdacb7fe640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 0x7fdac40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.929+0000 7fdad9d9b640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 0x7fdac40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.929+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdacc0ef780 con 0x7fdadc102d40 2024-09-12T02:46:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.930+0000 7fdad9d9b640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 0x7fdac40a50d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdad00058c0 tx=0x7fdad0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:53.934+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdacc0bc650 con 0x7fdadc102d40 2024-09-12T02:46:54.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.149+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdadc107210 con 0x7fdadc102d40 2024-09-12T02:46:54.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.150+0000 7fdacb7fe640 1 -- 172.21.15.16:0/3278241105 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7fdacc0bbf50 con 0x7fdadc102d40 2024-09-12T02:46:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:54.151 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:46:54.151 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.152+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 msgr2=0x7fdac40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 0x7fdac40a50d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdad00058c0 tx=0x7fdad0005850 comp rx=0 tx=0).stop 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 msgr2=0x7fdadc198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdacc006010 tx=0x7fdacc0042f0 comp rx=0 tx=0).stop 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 shutdown_connections 2024-09-12T02:46:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdadc10c170 0x7fdadc19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc103660 0x7fdadc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdac40a2c10 0x7fdac40a50d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 --2- 172.21.15.16:0/3278241105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdadc102d40 0x7fdadc198470 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 >> 172.21.15.16:0/3278241105 conn(0x7fdadc0fc7a0 msgr2=0x7fdadc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 shutdown_connections 2024-09-12T02:46:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:54.153+0000 7fdae0877640 1 -- 172.21.15.16:0/3278241105 wait complete. 2024-09-12T02:46:54.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:53 smithi033 ceph-mon[35092]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:53 smithi184 ceph-mon[35321]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-12T02:46:54.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:46:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:55.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3278241105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:55.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:54 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3278241105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:55.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:54 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3278241105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:55.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 -- 172.21.15.16:0/4183378677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4106a80 msgr2=0x7f1cf4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 --2- 172.21.15.16:0/4183378677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4106a80 0x7f1cf4106e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009930 tx=0x7f1ce802a420 comp rx=0 tx=0).stop 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 -- 172.21.15.16:0/4183378677 shutdown_connections 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 --2- 172.21.15.16:0/4183378677 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4101420 0x7f1cf410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 --2- 172.21.15.16:0/4183378677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 0x7f1cf4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 --2- 172.21.15.16:0/4183378677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4106a80 0x7f1cf4106e60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 -- 172.21.15.16:0/4183378677 >> 172.21.15.16:0/4183378677 conn(0x7f1cf40fc7a0 msgr2=0x7f1cf40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 -- 172.21.15.16:0/4183378677 shutdown_connections 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.567+0000 7f1cfacad640 1 -- 172.21.15.16:0/4183378677 wait complete. 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 Processor -- start 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 -- start start 2024-09-12T02:46:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 0x7f1cf4197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 0x7f1cf419bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf4110e40 con 0x7f1cf4101420 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf4110cc0 con 0x7f1cf4106a80 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.568+0000 7f1cfacad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf4110fc0 con 0x7f1cf4100a80 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47308/0 (socket says 172.21.15.16:47308) 2024-09-12T02:46:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 -- 172.21.15.16:0/3933332931 learned_addr learned my addr 172.21.15.16:0/3933332931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf8a49640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 0x7f1cf4197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf924a640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 0x7f1cf419bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 -- 172.21.15.16:0/3933332931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 msgr2=0x7f1cf4197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 0x7f1cf4197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 -- 172.21.15.16:0/3933332931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 msgr2=0x7f1cf419bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 0x7f1cf419bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 -- 172.21.15.16:0/3933332931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf419c2b0 con 0x7f1cf4101420 2024-09-12T02:46:55.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf924a640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 0x7f1cf419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf3fff640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c920 tx=0x7f1ce000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0007c70 con 0x7f1cf4101420 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf419c5a0 con 0x7f1cf4101420 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0007e10 con 0x7f1cf4101420 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.569+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0018890 con 0x7f1cf4101420 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.570+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf41a3e80 con 0x7f1cf4101420 2024-09-12T02:46:55.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.570+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf4100ee0 con 0x7f1cf4101420 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.572+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1ce0018a50 con 0x7f1cf4101420 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.572+0000 7f1cf1ffb640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 0x7f1ce40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.572+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1ce00cf330 con 0x7f1cf4101420 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.572+0000 7f1cf8a49640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 0x7f1ce40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.573+0000 7f1cf8a49640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 0x7f1ce40a50d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009900 tx=0x7f1ce80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.576+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ce0014030 con 0x7f1cf4101420 2024-09-12T02:46:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.786+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cf4198660 con 0x7f1cf4101420 2024-09-12T02:46:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.787+0000 7f1cf1ffb640 1 -- 172.21.15.16:0/3933332931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7f1cf4198660 con 0x7f1cf4101420 2024-09-12T02:46:55.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:55.789 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:46:55.790 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:46:55.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 msgr2=0x7f1ce40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:55.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 0x7f1ce40a50d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009900 tx=0x7f1ce80058d0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 msgr2=0x7f1cf4197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c920 tx=0x7f1ce000cde0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 shutdown_connections 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf4106a80 0x7f1cf419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1ce40a2c10 0x7f1ce40a50d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf4101420 0x7f1cf4197840 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 --2- 172.21.15.16:0/3933332931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf4100a80 0x7f1cf4197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 >> 172.21.15.16:0/3933332931 conn(0x7f1cf40fc7a0 msgr2=0x7f1cf40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 shutdown_connections 2024-09-12T02:46:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:55.789+0000 7f1cfacad640 1 -- 172.21.15.16:0/3933332931 wait complete. 2024-09-12T02:46:56.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:55 smithi016 ceph-mon[30925]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:55 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3933332931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:56.164 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:55 smithi033 ceph-mon[35092]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:56.164 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:55 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3933332931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:55 smithi184 ceph-mon[35321]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:46:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:55 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3933332931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:46:56.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:46:56.795 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 -- 172.21.15.16:0/1200364065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8100b20 msgr2=0x7f14f8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 --2- 172.21.15.16:0/1200364065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8100b20 0x7f14f8100f00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009900 tx=0x7f14f402a410 comp rx=0 tx=0).stop 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 -- 172.21.15.16:0/1200364065 shutdown_connections 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 --2- 172.21.15.16:0/1200364065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f810e500 0x7f14f8110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 --2- 172.21.15.16:0/1200364065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f810df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 --2- 172.21.15.16:0/1200364065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8100b20 0x7f14f8100f00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 -- 172.21.15.16:0/1200364065 >> 172.21.15.16:0/1200364065 conn(0x7f14f80fc7d0 msgr2=0x7f14f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 -- 172.21.15.16:0/1200364065 shutdown_connections 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.269+0000 7f1500419640 1 -- 172.21.15.16:0/1200364065 wait complete. 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 Processor -- start 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 -- start start 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 0x7f14f81a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14f8114260 con 0x7f14f810e500 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f14f81140e0 con 0x7f14f8100b20 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.270+0000 7f1500419640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14f81143e0 con 0x7f14f8101440 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe9b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe1b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 0x7f14f81a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe9b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47318/0 (socket says 172.21.15.16:47318) 2024-09-12T02:46:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe9b6640 1 -- 172.21.15.16:0/3159419085 learned_addr learned my addr 172.21.15.16:0/3159419085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 -- 172.21.15.16:0/3159419085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 msgr2=0x7f14f81a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 0x7f14f81a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 -- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 msgr2=0x7f14f819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 -- 172.21.15.16:0/3159419085 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f819ace0 con 0x7f14f8101440 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe9b6640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:46:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fe1b5640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 0x7f14f81a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.271+0000 7f14fd9b4640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f81a0930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14e800d6a0 tx=0x7f14e800db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.272+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8017070 con 0x7f14f8101440 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.272+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14e8009d00 con 0x7f14f8101440 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.272+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8003c60 con 0x7f14f8101440 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.272+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f819afd0 con 0x7f14f8101440 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.272+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f81a72c0 con 0x7f14f8101440 2024-09-12T02:46:57.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.273+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f806a850 con 0x7f14f8101440 2024-09-12T02:46:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.274+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f14e8009820 con 0x7f14f8101440 2024-09-12T02:46:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.275+0000 7f14ef7fe640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 0x7f14e40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.275+0000 7f14fe1b5640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 0x7f14e40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.275+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14e80cee90 con 0x7f14f8101440 2024-09-12T02:46:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.275+0000 7f14fe1b5640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 0x7f14e40a5150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009230 tx=0x7f14f40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.279+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f14e809b670 con 0x7f14f8101440 2024-09-12T02:46:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.457+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14f8100f00 con 0x7f14f8101440 2024-09-12T02:46:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.458+0000 7f14ef7fe640 1 -- 172.21.15.16:0/3159419085 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f14f8100f00 con 0x7f14f8101440 2024-09-12T02:46:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:57.459 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:46:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 msgr2=0x7f14e40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 0x7f14e40a5150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009230 tx=0x7f14f40058d0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 msgr2=0x7f14f81a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f81a0930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14e800d6a0 tx=0x7f14e800db60 comp rx=0 tx=0).stop 2024-09-12T02:46:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 shutdown_connections 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14e40a2c90 0x7f14e40a5150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f810e500 0x7f14f819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14f8101440 0x7f14f81a0930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 --2- 172.21.15.16:0/3159419085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14f8100b20 0x7f14f81a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 >> 172.21.15.16:0/3159419085 conn(0x7f14f80fc7d0 msgr2=0x7f14f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 shutdown_connections 2024-09-12T02:46:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:57.461+0000 7f1500419640 1 -- 172.21.15.16:0/3159419085 wait complete. 2024-09-12T02:46:58.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:57 smithi016 ceph-mon[30925]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:46:58.132 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:57 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3159419085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:58.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:46:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:57 smithi033 ceph-mon[35092]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:46:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:57 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3159419085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:57 smithi184 ceph-mon[35321]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:46:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:57 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3159419085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:46:58.418 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- 172.21.15.16:0/2570013039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95301008d0 msgr2=0x7f9530100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 --2- 172.21.15.16:0/2570013039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95301008d0 0x7f9530100cb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f952c009900 tx=0x7f952c02a410 comp rx=0 tx=0).stop 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- 172.21.15.16:0/2570013039 shutdown_connections 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 --2- 172.21.15.16:0/2570013039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f953010e480 0x7f9530110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 --2- 172.21.15.16:0/2570013039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 0x7f953010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 --2- 172.21.15.16:0/2570013039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95301008d0 0x7f9530100cb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- 172.21.15.16:0/2570013039 >> 172.21.15.16:0/2570013039 conn(0x7f95300fc7a0 msgr2=0x7f95300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- 172.21.15.16:0/2570013039 shutdown_connections 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- 172.21.15.16:0/2570013039 wait complete. 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 Processor -- start 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.860+0000 7f9538091640 1 -- start start 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95301008d0 0x7f953019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 0x7f953019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95301141f0 con 0x7f953010e480 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9530114070 con 0x7f9530101280 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9530114370 con 0x7f95301008d0 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47336/0 (socket says 172.21.15.16:47336) 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953562c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 0x7f953019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 -- 172.21.15.16:0/650648471 learned_addr learned my addr 172.21.15.16:0/650648471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9535e2d640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95301008d0 0x7f953019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 -- 172.21.15.16:0/650648471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95301008d0 msgr2=0x7f953019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95301008d0 0x7f953019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 -- 172.21.15.16:0/650648471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 msgr2=0x7f953019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 0x7f953019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 -- 172.21.15.16:0/650648471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953019f610 con 0x7f953010e480 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f953662e640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952800b9e0 tx=0x7f952800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528019070 con 0x7f953010e480 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.861+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f953019f900 con 0x7f953010e480 2024-09-12T02:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.862+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9528014440 con 0x7f953010e480 2024-09-12T02:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.862+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95301a71e0 con 0x7f953010e480 2024-09-12T02:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.862+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95280125a0 con 0x7f953010e480 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.863+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9530100cb0 con 0x7f953010e480 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.864+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f952801c070 con 0x7f953010e480 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.864+0000 7f9526ffd640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 0x7f951c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.865+0000 7f9535e2d640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 0x7f951c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.865+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f95280ce600 con 0x7f953010e480 2024-09-12T02:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.865+0000 7f9535e2d640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 0x7f951c0a50d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f952c031550 tx=0x7f952c009560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:46:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:58.869+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f952809b520 con 0x7f953010e480 2024-09-12T02:46:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.083+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f953002d7c0 con 0x7f953010e480 2024-09-12T02:46:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.084+0000 7f9526ffd640 1 -- 172.21.15.16:0/650648471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7f952809ae20 con 0x7f953010e480 2024-09-12T02:46:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:46:59.085 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:46:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 msgr2=0x7f951c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 0x7f951c0a50d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f952c031550 tx=0x7f952c009560 comp rx=0 tx=0).stop 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 msgr2=0x7f953019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952800b9e0 tx=0x7f952800bde0 comp rx=0 tx=0).stop 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 shutdown_connections 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f951c0a2c10 0x7f951c0a50d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f953010e480 0x7f953019ef30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9530101280 0x7f953019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 --2- 172.21.15.16:0/650648471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95301008d0 0x7f953019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.086+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 >> 172.21.15.16:0/650648471 conn(0x7f95300fc7a0 msgr2=0x7f95300feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.087+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 shutdown_connections 2024-09-12T02:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:46:59.087+0000 7f9538091640 1 -- 172.21.15.16:0/650648471 wait complete. 2024-09-12T02:46:59.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-12T02:45:47.690653+0000', 'modified': '2024-09-12T02:45:58.926336+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24467, 'mds_2': 24445, 'mds_3': 24443, 'mds_4': 14595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14583': {'gid': 14583, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/2216233994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 2216233994}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 2216233994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14595': {'gid': 14595, 'name': 'a', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.16:6841/1593859053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 1593859053}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 1593859053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6833/897342399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 897342399}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 897342399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6833/1900353998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6832', 'nonce': 1900353998}, {'type': 'v1', 'addr': '172.21.15.33:6833', 'nonce': 1900353998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6835/1041505607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 1041505607}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 1041505607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6837/2803727914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 2803727914}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 2803727914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.33:6835/2990713107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 2990713107}, {'type': 'v1', 'addr': '172.21.15.33:6835', 'nonce': 2990713107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.184:6839/598714982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6838', 'nonce': 598714982}, {'type': 'v1', 'addr': '172.21.15.184:6839', 'nonce': 598714982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.33:6837/4242490884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6836', 'nonce': 4242490884}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 4242490884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.33:6839/3603617308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 3603617308}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 3603617308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24500, 24467, 24445, 24443, 14595]} 2024-09-12T02:46:59.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-12T02:46:59.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.a status 2024-09-12T02:47:00.050 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:59 smithi016 ceph-mon[30925]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:47:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:46:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/650648471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:59 smithi033 ceph-mon[35092]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:47:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:46:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/650648471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:59 smithi184 ceph-mon[35321]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-09-12T02:47:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:46:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/650648471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:00.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.507+0000 7feb7373f640 1 -- 172.21.15.16:0/2088832682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c106f20 msgr2=0x7feb6c10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.507+0000 7feb7373f640 1 --2- 172.21.15.16:0/2088832682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c106f20 0x7feb6c10eaa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feb64009ca0 tx=0x7feb6402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 -- 172.21.15.16:0/2088832682 shutdown_connections 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 --2- 172.21.15.16:0/2088832682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c106f20 0x7feb6c10eaa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 --2- 172.21.15.16:0/2088832682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c1027a0 0x7feb6c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 --2- 172.21.15.16:0/2088832682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c101df0 0x7feb6c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 -- 172.21.15.16:0/2088832682 >> 172.21.15.16:0/2088832682 conn(0x7feb6c0fd8f0 msgr2=0x7feb6c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 -- 172.21.15.16:0/2088832682 shutdown_connections 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 -- 172.21.15.16:0/2088832682 wait complete. 2024-09-12T02:47:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 Processor -- start 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.508+0000 7feb7373f640 1 -- start start 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 0x7feb6c1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c106f20 0x7feb6c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c111f80 con 0x7feb6c101df0 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c111e00 con 0x7feb6c1027a0 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb7373f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c112100 con 0x7feb6c106f20 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47350/0 (socket says 172.21.15.16:47350) 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 -- 172.21.15.16:0/821341297 learned_addr learned my addr 172.21.15.16:0/821341297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb70cda640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 0x7feb6c1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb71cdc640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c106f20 0x7feb6c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 -- 172.21.15.16:0/821341297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c106f20 msgr2=0x7feb6c19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c106f20 0x7feb6c19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 -- 172.21.15.16:0/821341297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 msgr2=0x7feb6c1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 0x7feb6c1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 -- 172.21.15.16:0/821341297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c19d310 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb70cda640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 0x7feb6c1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb714db640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb6800d950 tx=0x7feb6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.509+0000 7feb627fc640 1 -- 172.21.15.16:0/821341297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68017070 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.510+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c19d600 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.510+0000 7feb627fc640 1 -- 172.21.15.16:0/821341297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb68009c30 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.510+0000 7feb627fc640 1 -- 172.21.15.16:0/821341297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6800f6b0 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.510+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb6c1a4ee0 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.512+0000 7feb627fc640 1 -- 172.21.15.16:0/821341297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7feb680109d0 con 0x7feb6c101df0 2024-09-12T02:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.512+0000 7feb75b18740 1 Processor -- start 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.512+0000 7feb75b18740 1 -- start start 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aefaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aefd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56291ae52090 con 0x56291aeeb8b0 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56291adf9a70 con 0x56291aefafc0 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb75b18740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56291ae4e050 con 0x56291aeec1d0 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47358/0 (socket says 172.21.15.16:47358) 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 -- 172.21.15.16:0/4259778100 learned_addr learned my addr 172.21.15.16:0/4259778100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb627fc640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 0x7feb580a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb714db640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aefaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 0x7feb580a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb70cda640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aefd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 -- 172.21.15.16:0/4259778100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 msgr2=0x56291aefaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aefaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb627fc640 1 -- 172.21.15.16:0/821341297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7feb680cef40 con 0x7feb6c101df0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 -- 172.21.15.16:0/4259778100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 msgr2=0x56291aefd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aefd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 -- 172.21.15.16:0/4259778100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56291ae06130 con 0x56291aeeb8b0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb70cda640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aefd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.513+0000 7feb71cdc640 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feb64005cc0 tx=0x7feb640039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc810d15c7374b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb577fe640 1 -- 172.21.15.16:0/4259778100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640319a0 con 0x56291aeeb8b0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb577fe640 1 -- 172.21.15.16:0/4259778100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6404e070 con 0x56291aeeb8b0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb577fe640 1 -- 172.21.15.16:0/4259778100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6402fca0 con 0x56291aeeb8b0 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 -- 172.21.15.16:0/4259778100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 msgr2=0x56291aeebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb71cdc640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 0x7feb580a50d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb6404d000 tx=0x7feb6404b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feb64005cc0 tx=0x7feb640039f0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 -- 172.21.15.16:0/4259778100 shutdown_connections 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aefd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aefaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.514+0000 7feb75b18740 1 --2- 172.21.15.16:0/4259778100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aeebc90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- 172.21.15.16:0/4259778100 >> 172.21.15.16:0/4259778100 conn(0x56291aee7f60 msgr2=0x56291aee8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- 172.21.15.16:0/4259778100 shutdown_connections 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- 172.21.15.16:0/4259778100 wait complete. 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- start start 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 Processor -- start 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aef7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aef8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56291aefda10 con 0x56291aeeb8b0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56291ae0cf20 con 0x56291aefafc0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56291aefdb90 con 0x56291aeec1d0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb75b18740 10 client.0 _ll_register_callbacks cb 0x56291add6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb714db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb714db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47366/0 (socket says 172.21.15.16:47366) 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.515+0000 7feb714db640 1 -- 172.21.15.16:0/3666667669 learned_addr learned my addr 172.21.15.16:0/3666667669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb70cda640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aef7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 -- 172.21.15.16:0/3666667669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 msgr2=0x56291aef7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aef7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 -- 172.21.15.16:0/3666667669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 msgr2=0x56291aef8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aef8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 -- 172.21.15.16:0/3666667669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56291abc32b0 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb70cda640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aef7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb714db640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb68097fc0 tx=0x7feb6806a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef82c9d1293fd0c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68020070 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7feb680ccaf0 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb6404e070 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb680666d0 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb75b18740 10 client.14952 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56291af09390 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.516+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6404e070 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7feb6809b850 con 0x56291aeeb8b0 2024-09-12T02:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 10 client.14952 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 10 client.14952 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 20 client.14952 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af0b9f0 0x56291af0ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 4 client.14952 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56291ae5f600 con 0x56291af0b9f0 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb70cda640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af0b9f0 0x56291af0ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb54ff9640 10 client.14952 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.517+0000 7feb70cda640 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af0b9f0 0x56291af0ce60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3b85e4f9b8e0de9f server_cookie=e6615d6e6af84be4 in_seq=0 out_seq=0 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb54ff9640 1 -- 172.21.15.16:0/3666667669 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56291ae5f600 con 0x56291af0b9f0 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb54ff9640 10 client.14952 handle_command_reply: tid=0 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 client.14952 shutdown 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 msgr2=0x56291aefa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb68097fc0 tx=0x7feb6806a570 comp rx=0 tx=0).stop 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 shutdown_connections 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af0b9f0 0x56291af0ce60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aefafc0 0x56291aef8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291aef7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 --2- 172.21.15.16:0/3666667669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aeeb8b0 0x56291aefa870 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb54ff9640 0 client.14952 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 >> 172.21.15.16:0/3666667669 conn(0x56291aee7f60 msgr2=0x56291aeedcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 shutdown_connections 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 1 -- 172.21.15.16:0/3666667669 wait complete. 2024-09-12T02:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.518+0000 7feb75b18740 20 client.14952 trim_cache size 0 max 16384 2024-09-12T02:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 Processor -- start 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 -- start start 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 0x56291af0c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeec1d0 0x56291aef6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56291af0b1a0 con 0x56291aefafc0 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56291af0b870 con 0x56291aeec1d0 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.536+0000 7feb75b18740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56291af0bf40 con 0x56291aeeb8b0 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47368/0 (socket says 172.21.15.16:47368) 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 -- 172.21.15.16:0/3947895152 learned_addr learned my addr 172.21.15.16:0/3947895152 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb70cda640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeec1d0 0x56291aef6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb714db640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 0x56291af0c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 -- 172.21.15.16:0/3947895152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 msgr2=0x56291af0c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 0x56291af0c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 -- 172.21.15.16:0/3947895152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeec1d0 msgr2=0x56291aef6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeec1d0 0x56291aef6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 -- 172.21.15.16:0/3947895152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56291aefdb80 con 0x56291aefafc0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb714db640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 0x56291af0c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb71cdc640 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7feb6402b000 tx=0x7feb64003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fab625a5b03cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb56ffd640 1 -- 172.21.15.16:0/3947895152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64060070 con 0x56291aefafc0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb56ffd640 1 -- 172.21.15.16:0/3947895152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6404e050 con 0x56291aefafc0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.537+0000 7feb56ffd640 1 -- 172.21.15.16:0/3947895152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6402f850 con 0x56291aefafc0 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- 172.21.15.16:0/3947895152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 msgr2=0x56291aef69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7feb6402b000 tx=0x7feb64003970 comp rx=0 tx=0).stop 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- 172.21.15.16:0/3947895152 shutdown_connections 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291aef69d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeec1d0 0x56291aef6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 --2- 172.21.15.16:0/3947895152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeeb8b0 0x56291af0c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- 172.21.15.16:0/3947895152 >> 172.21.15.16:0/3947895152 conn(0x56291aee7f60 msgr2=0x56291aee2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- 172.21.15.16:0/3947895152 shutdown_connections 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- 172.21.15.16:0/3947895152 wait complete. 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 -- start start 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.538+0000 7feb75b18740 1 Processor -- start 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeeb8b0 0x56291af0e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291af0e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56291af0cce0 con 0x56291aefafc0 2024-09-12T02:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56291af0c610 con 0x56291aeeb8b0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56291af0d3b0 con 0x56291aeec1d0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb75b18740 10 client.0 _ll_register_callbacks cb 0x7feb6800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47376/0 (socket says 172.21.15.16:47376) 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 -- 172.21.15.16:0/991703652 learned_addr learned my addr 172.21.15.16:0/991703652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb714db640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeeb8b0 0x56291af0e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb70cda640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291af0e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 -- 172.21.15.16:0/991703652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 msgr2=0x56291af0e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291af0e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 -- 172.21.15.16:0/991703652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeeb8b0 msgr2=0x56291af0e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeeb8b0 0x56291af0e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 -- 172.21.15.16:0/991703652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56291af74390 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb71cdc640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb6402ade0 tx=0x7feb64003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda5fe49c31f35b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.539+0000 7feb54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64060040 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7feb6402f810 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb6404e050 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb6402fa10 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 10 client.14958 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56291af75600 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6404e050 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7feb64063070 con 0x56291aefafc0 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 10 client.14958 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 10 client.14958 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 20 client.14958 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af79e20 0x56291af78b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 4 client.14958 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "status"}) -- 0x56291ae59b10 con 0x56291af79e20 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.540+0000 7feb714db640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af79e20 0x56291af78b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb54ff9640 10 client.14958 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb714db640 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af79e20 0x56291af78b30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d5f9a0371727e0df server_cookie=ef51c3b204ca6f12 in_seq=0 out_seq=0 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb54ff9640 1 -- 172.21.15.16:0/991703652 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56291ae59b10 con 0x56291af79e20 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb54ff9640 10 client.14958 handle_command_reply: tid=0 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 client.14958 shutdown 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 msgr2=0x56291af0eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb6402ade0 tx=0x7feb64003f30 comp rx=0 tx=0).stop 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 shutdown_connections 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56291aefafc0 0x56291af0eba0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56291aeec1d0 0x56291af0e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56291af79e20 0x56291af78b30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 --2- 172.21.15.16:0/991703652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56291aeeb8b0 0x56291af0e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 >> 172.21.15.16:0/991703652 conn(0x56291aee7f60 msgr2=0x56291aef7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb54ff9640 0 client.14958 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 shutdown_connections 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 1 -- 172.21.15.16:0/991703652 wait complete. 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.541+0000 7feb75b18740 20 client.14958 trim_cache size 0 max 16384 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:00.555 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14595, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 62.618984224999998, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 85.660279227999993 2024-09-12T02:47:00.556 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 msgr2=0x7feb580a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 0x7feb580a50d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb6404d000 tx=0x7feb6404b000 comp rx=0 tx=0).stop 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 msgr2=0x7feb6c198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb6800d950 tx=0x7feb6800de10 comp rx=0 tx=0).stop 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 shutdown_connections 2024-09-12T02:47:00.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb6c106f20 0x7feb6c19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb6c1027a0 0x7feb6c1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7feb580a2c10 0x7feb580a50d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 --2- 172.21.15.16:0/821341297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6c101df0 0x7feb6c198360 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 >> 172.21.15.16:0/821341297 conn(0x7feb6c0fd8f0 msgr2=0x7feb6c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 shutdown_connections 2024-09-12T02:47:00.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:00.542+0000 7feb7373f640 1 -- 172.21.15.16:0/821341297 wait complete. 2024-09-12T02:47:01.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.e status 2024-09-12T02:47:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:02.018 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:01 smithi016 ceph-mon[30925]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 -- 172.21.15.16:0/3941603900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74101de0 msgr2=0x7f8e741021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3941603900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74101de0 0x7f8e741021c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009900 tx=0x7f8e6402a410 comp rx=0 tx=0).stop 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 -- 172.21.15.16:0/3941603900 shutdown_connections 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3941603900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74106f10 0x7f8e7410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3941603900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74102790 0x7f8e741067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3941603900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74101de0 0x7f8e741021c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 -- 172.21.15.16:0/3941603900 >> 172.21.15.16:0/3941603900 conn(0x7f8e740fd8c0 msgr2=0x7f8e740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 -- 172.21.15.16:0/3941603900 shutdown_connections 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.017+0000 7f8e78dea640 1 -- 172.21.15.16:0/3941603900 wait complete. 2024-09-12T02:47:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 Processor -- start 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 -- start start 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74101de0 0x7f8e740744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74106f10 0x7f8e74074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74111f90 con 0x7f8e74102790 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74111e10 con 0x7f8e74106f10 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.018+0000 7f8e78dea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74112110 con 0x7f8e74101de0 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47400/0 (socket says 172.21.15.16:47400) 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 -- 172.21.15.16:0/3774064349 learned_addr learned my addr 172.21.15.16:0/3774064349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74106f10 0x7f8e74074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e7259c640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74101de0 0x7f8e740744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 -- 172.21.15.16:0/3774064349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74101de0 msgr2=0x7f8e740744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74101de0 0x7f8e740744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 -- 172.21.15.16:0/3774064349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74106f10 msgr2=0x7f8e74074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74106f10 0x7f8e74074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 -- 172.21.15.16:0/3774064349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e741a6980 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d950 tx=0x7f8e6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e637fe640 1 -- 172.21.15.16:0/3774064349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68017070 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e637fe640 1 -- 172.21.15.16:0/3774064349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e68009c30 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e741a6c70 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e637fe640 1 -- 172.21.15.16:0/3774064349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800f6b0 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.019+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e741a71b0 con 0x7f8e74102790 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 Processor -- start 2024-09-12T02:47:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e637fe640 1 -- 172.21.15.16:0/3774064349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8e6800f850 con 0x7f8e74102790 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 -- start start 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a92670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a94fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ac99ee090 con 0x559ac9a879d0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ac9995a70 con 0x559ac9a882f0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ac99ea050 con 0x559ac9a92bb0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e72d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e72d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47414/0 (socket says 172.21.15.16:47414) 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e72d9d640 1 -- 172.21.15.16:0/3729793220 learned_addr learned my addr 172.21.15.16:0/3729793220 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.022+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a94fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e7259c640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a92670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 -- 172.21.15.16:0/3729793220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 msgr2=0x559ac9a94fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a94fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 -- 172.21.15.16:0/3729793220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 msgr2=0x559ac9a92670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a92670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 -- 172.21.15.16:0/3729793220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac99a2130 con 0x559ac9a879d0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a94fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e7259c640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a92670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e637fe640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 0x7f8e5c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00a430 tx=0x7f8e6c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef55742e692b2893 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e7259c640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 0x7f8e5c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e637fe640 1 -- 172.21.15.16:0/3774064349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e680cee20 con 0x7f8e74102790 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e60ff9640 1 -- 172.21.15.16:0/3729793220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c013790 con 0x559ac9a879d0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e60ff9640 1 -- 172.21.15.16:0/3729793220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c019070 con 0x559ac9a879d0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.023+0000 7f8e60ff9640 1 -- 172.21.15.16:0/3729793220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c014de0 con 0x559ac9a879d0 2024-09-12T02:47:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/3729793220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 msgr2=0x559ac9a87db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00a430 tx=0x7f8e6c009650 comp rx=0 tx=0).stop 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7259c640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 0x7f8e5c0a50d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x559ac9a84dc0 tx=0x7f8e640058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/3729793220 shutdown_connections 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a94fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a92670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/3729793220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9a87db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/3729793220 >> 172.21.15.16:0/3729793220 conn(0x559ac9a83f60 msgr2=0x559ac9a84340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/3729793220 shutdown_connections 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/3729793220 wait complete. 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.024+0000 7f8e7b1c3740 1 -- start start 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 Processor -- start 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a92430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ac9a95780 con 0x559ac9a882f0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ac99a8f20 con 0x559ac9a879d0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ac9a95900 con 0x559ac9a92bb0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7b1c3740 10 client.0 _ll_register_callbacks cb 0x559ac9972030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47426/0 (socket says 172.21.15.16:47426) 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 -- 172.21.15.16:0/1451486790 learned_addr learned my addr 172.21.15.16:0/1451486790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e7259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a92430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 -- 172.21.15.16:0/1451486790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 msgr2=0x559ac9a8fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 -- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 msgr2=0x559ac9a92430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a92430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.025+0000 7f8e71d9b640 1 -- 172.21.15.16:0/1451486790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac975f2b0 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e6809b4c0 tx=0x7f8e68010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b80ba7b72dcf9c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68020050 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e68097ba0 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6c019070 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8e680bb040 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e7b1c3740 10 client.14970 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559ac9aa1120 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c019070 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.026+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f8e680109d0 con 0x559ac9a882f0 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 10 client.14970 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 10 client.14970 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:47:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 20 client.14970 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9aa36d0 0x559ac9aa4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 4 client.14970 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ac99fb600 con 0x559ac9aa36d0 2024-09-12T02:47:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7259c640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9aa36d0 0x559ac9aa4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e59ffb640 10 client.14970 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.027+0000 7f8e7259c640 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9aa36d0 0x559ac9aa4b40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1503d01b869891a8 server_cookie=d0e116008f560464 in_seq=0 out_seq=0 2024-09-12T02:47:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e59ffb640 1 -- 172.21.15.16:0/1451486790 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ac99fb600 con 0x559ac9aa36d0 2024-09-12T02:47:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e59ffb640 10 client.14970 handle_command_reply: tid=0 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 client.14970 shutdown 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 msgr2=0x559ac9a8f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8e6809b4c0 tx=0x7f8e68010d70 comp rx=0 tx=0).stop 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 shutdown_connections 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9aa36d0 0x559ac9aa4b40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8f780 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1451486790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a92430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 >> 172.21.15.16:0/1451486790 conn(0x559ac9a83f60 msgr2=0x559ac9a89c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.028+0000 7f8e59ffb640 0 client.14970 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.029+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 shutdown_connections 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.029+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1451486790 wait complete. 2024-09-12T02:47:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.029+0000 7f8e7b1c3740 20 client.14970 trim_cache size 0 max 16384 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.046+0000 7f8e7b1c3740 1 Processor -- start 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 -- start start 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a8e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa2e80 con 0x559ac9a879d0 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa3550 con 0x559ac9a882f0 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa3c20 con 0x559ac9a92bb0 2024-09-12T02:47:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47442/0 (socket says 172.21.15.16:47442) 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 -- 172.21.15.16:0/1807661387 learned_addr learned my addr 172.21.15.16:0/1807661387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a8e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 -- 172.21.15.16:0/1807661387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 msgr2=0x559ac9a8f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 -- 172.21.15.16:0/1807661387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 msgr2=0x559ac9a8e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a8e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 -- 172.21.15.16:0/1807661387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac9a84bf0 con 0x559ac9a879d0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a8e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.047+0000 7f8e7259c640 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e6402a930 tx=0x7f8e6403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b69a20e7a77a80e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e5bfff640 1 -- 172.21.15.16:0/1807661387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64055070 con 0x559ac9a879d0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e5bfff640 1 -- 172.21.15.16:0/1807661387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64004300 con 0x559ac9a879d0 2024-09-12T02:47:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e5bfff640 1 -- 172.21.15.16:0/1807661387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64035050 con 0x559ac9a879d0 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1807661387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 msgr2=0x559ac9aa3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e6402a930 tx=0x7f8e6403d000 comp rx=0 tx=0).stop 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1807661387 shutdown_connections 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9a8f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a882f0 0x559ac9a8e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/1807661387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a879d0 0x559ac9aa3da0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1807661387 >> 172.21.15.16:0/1807661387 conn(0x559ac9a83f60 msgr2=0x559ac9a7ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1807661387 shutdown_connections 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.048+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/1807661387 wait complete. 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 -- start start 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 Processor -- start 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9aabd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa49c0 con 0x559ac9a882f0 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa42f0 con 0x559ac9a879d0 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ac9aa5090 con 0x559ac9a92bb0 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7b1c3740 10 client.0 _ll_register_callbacks cb 0x7f8e6800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47452/0 (socket says 172.21.15.16:47452) 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 -- 172.21.15.16:0/879557367 learned_addr learned my addr 172.21.15.16:0/879557367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e72d9d640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9aabd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7259c640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 -- 172.21.15.16:0/879557367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 msgr2=0x559ac9aabd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9aabd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 -- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 msgr2=0x559ac9a8c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a8c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e71d9b640 1 -- 172.21.15.16:0/879557367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac9aac350 con 0x559ac9a882f0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.049+0000 7f8e7259c640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a8c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e71d9b640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e68097f00 tx=0x7f8e68009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6cd7d4f3eb3210 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68020070 con 0x559ac9a882f0 2024-09-12T02:47:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e680b4230 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e64004300 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8e6809ae90 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 10 client.14976 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559ac9aad650 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64004300 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f8e680d1040 con 0x559ac9a882f0 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 10 client.14976 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 10 client.14976 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 20 client.14976 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9b12390 0x559ac9b110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 4 client.14976 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.050+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "status"}) -- 0x559ac99f5b10 con 0x559ac9b12390 2024-09-12T02:47:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e7259c640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9b12390 0x559ac9b110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e59ffb640 10 client.14976 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e7259c640 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9b12390 0x559ac9b110a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=44b5bffcddc9c69 server_cookie=1561529e7b6954e6 in_seq=0 out_seq=0 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e59ffb640 1 -- 172.21.15.16:0/879557367 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559ac99f5b10 con 0x559ac9b12390 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e59ffb640 10 client.14976 handle_command_reply: tid=0 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.051+0000 7f8e7b1c3740 1 client.14976 shutdown 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 msgr2=0x559ac9a8c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e68097f00 tx=0x7f8e68009740 comp rx=0 tx=0).stop 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 shutdown_connections 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ac9b12390 0x559ac9b110a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ac9a92bb0 0x559ac9aabd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ac9a882f0 0x559ac9a8c710 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 --2- 172.21.15.16:0/879557367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ac9a879d0 0x559ac9a8c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 >> 172.21.15.16:0/879557367 conn(0x559ac9a83f60 msgr2=0x559ac9a8e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e59ffb640 0 client.14976 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 shutdown_connections 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 1 -- 172.21.15.16:0/879557367 wait complete. 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e7b1c3740 20 client.14976 trim_cache size 0 max 16384 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24443, 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 64.814190710000005, 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 103.115678437 2024-09-12T02:47:02.059 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 msgr2=0x7f8e5c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 0x7f8e5c0a50d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x559ac9a84dc0 tx=0x7f8e640058d0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 msgr2=0x7f8e74074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.052+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d950 tx=0x7f8e6800de10 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 shutdown_connections 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e74106f10 0x7f8e74074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e5c0a2c10 0x7f8e5c0a50d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e74102790 0x7f8e74074a30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 --2- 172.21.15.16:0/3774064349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e74101de0 0x7f8e740744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 >> 172.21.15.16:0/3774064349 conn(0x7f8e740fd8c0 msgr2=0x7f8e740fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 shutdown_connections 2024-09-12T02:47:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:02.053+0000 7f8e78dea640 1 -- 172.21.15.16:0/3774064349 wait complete. 2024-09-12T02:47:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:01 smithi033 ceph-mon[35092]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:01 smithi184 ceph-mon[35321]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:02.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.i status 2024-09-12T02:47:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 -- 172.21.15.16:0/1073679636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100a80 msgr2=0x7f6dac100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 --2- 172.21.15.16:0/1073679636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100a80 0x7f6dac100ee0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009930 tx=0x7f6d9c02a420 comp rx=0 tx=0).stop 2024-09-12T02:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 -- 172.21.15.16:0/1073679636 shutdown_connections 2024-09-12T02:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 --2- 172.21.15.16:0/1073679636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac101420 0x7f6dac10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 --2- 172.21.15.16:0/1073679636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100a80 0x7f6dac100ee0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 --2- 172.21.15.16:0/1073679636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac106a80 0x7f6dac106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 -- 172.21.15.16:0/1073679636 >> 172.21.15.16:0/1073679636 conn(0x7f6dac0fc7a0 msgr2=0x7f6dac0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 -- 172.21.15.16:0/1073679636 shutdown_connections 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.507+0000 7f6db43eb640 1 -- 172.21.15.16:0/1073679636 wait complete. 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 Processor -- start 2024-09-12T02:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 -- start start 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac100a80 0x7f6dac1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac101420 0x7f6dac198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac110e80 con 0x7f6dac106a80 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac110d00 con 0x7f6dac100a80 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db43eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac111000 con 0x7f6dac101420 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db2988640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db2988640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47472/0 (socket says 172.21.15.16:47472) 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.508+0000 7f6db2988640 1 -- 172.21.15.16:0/4023497971 learned_addr learned my addr 172.21.15.16:0/4023497971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db1986640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac101420 0x7f6dac198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2187640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac100a80 0x7f6dac1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 -- 172.21.15.16:0/4023497971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac101420 msgr2=0x7f6dac198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac101420 0x7f6dac198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 -- 172.21.15.16:0/4023497971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac100a80 msgr2=0x7f6dac1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac100a80 0x7f6dac1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 -- 172.21.15.16:0/4023497971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac19d470 con 0x7f6dac106a80 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db2988640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6da400cce0 tx=0x7f6da400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6da37fe640 1 -- 172.21.15.16:0/4023497971 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da400f450 con 0x7f6dac106a80 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6da37fe640 1 -- 172.21.15.16:0/4023497971 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da400fad0 con 0x7f6dac106a80 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac19d760 con 0x7f6dac106a80 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6da37fe640 1 -- 172.21.15.16:0/4023497971 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4018c10 con 0x7f6dac106a80 2024-09-12T02:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.509+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac1a5040 con 0x7f6dac106a80 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6da37fe640 1 -- 172.21.15.16:0/4023497971 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6da400f5f0 con 0x7f6dac106a80 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 Processor -- start 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 -- start start 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf59cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf6b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dadcec0090 con 0x55dadcf5a1f0 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dadce67a70 con 0x55dadcf68fc0 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db67c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dadcebc050 con 0x55dadcf598d0 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47488/0 (socket says 172.21.15.16:47488) 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.512+0000 7f6db2187640 1 -- 172.21.15.16:0/3914088615 learned_addr learned my addr 172.21.15.16:0/3914088615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db1986640 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf6b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6da37fe640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 0x7f6d980a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 -- 172.21.15.16:0/3914088615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 msgr2=0x55dadcf59cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf59cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 -- 172.21.15.16:0/3914088615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 msgr2=0x55dadcf6b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf6b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 -- 172.21.15.16:0/3914088615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dadce74130 con 0x55dadcf5a1f0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2988640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 0x7f6d980a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6da37fe640 1 -- 172.21.15.16:0/4023497971 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6da40cf760 con 0x7f6dac106a80 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6db2187640 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6da800ab90 tx=0x7f6da800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2687a2bb8b7ba785 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6da0dea640 1 -- 172.21.15.16:0/3914088615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da801d070 con 0x55dadcf5a1f0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6da0dea640 1 -- 172.21.15.16:0/3914088615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8018b20 con 0x55dadcf5a1f0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.513+0000 7f6da0dea640 1 -- 172.21.15.16:0/3914088615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8010b10 con 0x55dadcf5a1f0 2024-09-12T02:47:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 -- 172.21.15.16:0/3914088615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 msgr2=0x55dadcf68a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db2988640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 0x7f6d980a5150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6da4098840 tx=0x7f6da406de80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6da800ab90 tx=0x7f6da800f790 comp rx=0 tx=0).stop 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 -- 172.21.15.16:0/3914088615 shutdown_connections 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf6b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf68a80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 --2- 172.21.15.16:0/3914088615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf59cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 -- 172.21.15.16:0/3914088615 >> 172.21.15.16:0/3914088615 conn(0x55dadcf55f60 msgr2=0x55dadcf56340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 -- 172.21.15.16:0/3914088615 shutdown_connections 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.514+0000 7f6db67c4740 1 -- 172.21.15.16:0/3914088615 wait complete. 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 -- start start 2024-09-12T02:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 Processor -- start 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf68890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf66120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf6bbf0 con 0x55dadcf5a1f0 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dadce7af20 con 0x55dadcf68fc0 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf6bd70 con 0x55dadcf598d0 2024-09-12T02:47:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db67c4740 10 client.0 _ll_register_callbacks cb 0x55dadce44030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 -- 172.21.15.16:0/2130612568 learned_addr learned my addr 172.21.15.16:0/2130612568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db1986640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf68890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db1986640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf66120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 -- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 msgr2=0x55dadcf68890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf68890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 -- 172.21.15.16:0/2130612568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 msgr2=0x55dadcf66120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf66120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db2187640 1 -- 172.21.15.16:0/2130612568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dadcc312b0 con 0x55dadcf5a1f0 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.515+0000 7f6db1986640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf66120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db2187640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6da80107f0 tx=0x7f6da80105a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516664116ccf3ba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da801d050 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6da8018910 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da8018b20 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6da8004020 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 10 client.14988 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dadcf77580 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8018b20 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6da8005200 con 0x55dadcf5a1f0 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 10 client.14988 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 10 client.14988 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.516+0000 7f6db67c4740 20 client.14988 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcf797c0 0x55dadcf7ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6db67c4740 4 client.14988 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dadcecd600 con 0x55dadcf797c0 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6db1986640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcf797c0 0x55dadcf7ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6db1986640 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcf797c0 0x55dadcf7ac30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8382e7a30061b08f server_cookie=87d90773538e5721 in_seq=0 out_seq=0 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.517+0000 7f6d95ffb640 10 client.14988 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6d95ffb640 1 -- 172.21.15.16:0/2130612568 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dadcecd600 con 0x55dadcf797c0 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6d95ffb640 10 client.14988 handle_command_reply: tid=0 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 client.14988 shutdown 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 msgr2=0x55dadcf65be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6da80107f0 tx=0x7f6da80105a0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 shutdown_connections 2024-09-12T02:47:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf68fc0 0x55dadcf66120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf65be0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcf797c0 0x55dadcf7ac30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 --2- 172.21.15.16:0/2130612568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf598d0 0x55dadcf68890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 >> 172.21.15.16:0/2130612568 conn(0x55dadcf55f60 msgr2=0x55dadcf5bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6d95ffb640 0 client.14988 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 shutdown_connections 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 1 -- 172.21.15.16:0/2130612568 wait complete. 2024-09-12T02:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.518+0000 7f6db67c4740 20 client.14988 trim_cache size 0 max 16384 2024-09-12T02:47:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 Processor -- start 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 -- start start 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf598d0 0x55dadcf790f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf61240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf78f70 con 0x55dadcf5a1f0 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf79640 con 0x55dadcf598d0 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.536+0000 7f6db67c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf79d10 con 0x55dadcf68fc0 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47516/0 (socket says 172.21.15.16:47516) 2024-09-12T02:47:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 -- 172.21.15.16:0/1298730681 learned_addr learned my addr 172.21.15.16:0/1298730681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db1986640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf598d0 0x55dadcf790f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 -- 172.21.15.16:0/1298730681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 msgr2=0x55dadcf61240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db1986640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf61240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf61240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 -- 172.21.15.16:0/1298730681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf598d0 msgr2=0x55dadcf790f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf598d0 0x55dadcf790f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 -- 172.21.15.16:0/1298730681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dadcf6b750 con 0x55dadcf5a1f0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db1986640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf61240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6db2187640 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005650 tx=0x7f6da800a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6fe6cfef7785c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6d97fff640 1 -- 172.21.15.16:0/1298730681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da803c070 con 0x55dadcf5a1f0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6d97fff640 1 -- 172.21.15.16:0/1298730681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8029070 con 0x55dadcf5a1f0 2024-09-12T02:47:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.537+0000 7f6d97fff640 1 -- 172.21.15.16:0/1298730681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da801d040 con 0x55dadcf5a1f0 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- 172.21.15.16:0/1298730681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 msgr2=0x55dadcf7a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005650 tx=0x7f6da800a5e0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- 172.21.15.16:0/1298730681 shutdown_connections 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf61240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf5a1f0 0x55dadcf7a560 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1298730681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf598d0 0x55dadcf790f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- 172.21.15.16:0/1298730681 >> 172.21.15.16:0/1298730681 conn(0x55dadcf55f60 msgr2=0x55dadcf56340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- 172.21.15.16:0/1298730681 shutdown_connections 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- 172.21.15.16:0/1298730681 wait complete. 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 -- start start 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.538+0000 7f6db67c4740 1 Processor -- start 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf5a1f0 0x55dadcf81d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf822b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf7aab0 con 0x55dadcf598d0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf7a3e0 con 0x55dadcf5a1f0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dadcf7b180 con 0x55dadcf68fc0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 10 client.0 _ll_register_callbacks cb 0x7f6da801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47532/0 (socket says 172.21.15.16:47532) 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 -- 172.21.15.16:0/1849266929 learned_addr learned my addr 172.21.15.16:0/1849266929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db2187640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf5a1f0 0x55dadcf81d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf822b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 -- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 msgr2=0x55dadcf822b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf822b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 -- 172.21.15.16:0/1849266929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf5a1f0 msgr2=0x55dadcf81d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf5a1f0 0x55dadcf81d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 -- 172.21.15.16:0/1849266929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dadcf7bfe0 con 0x55dadcf598d0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db1986640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c02a940 tx=0x7f6d9c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83b70a42bd68dd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c035040 con 0x55dadcf598d0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6d9c031670 con 0x55dadcf598d0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.539+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da8029070 con 0x55dadcf598d0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d9c031870 con 0x55dadcf598d0 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 10 client.14994 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dadcf809a0 con 0x55dadcf598d0 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8029070 con 0x55dadcf598d0 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6da8019ce0 con 0x55dadcf598d0 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 10 client.14994 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 10 client.14994 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 20 client.14994 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcfe7ee0 0x55dadcfe6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 4 client.14994 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "status"}) -- 0x55dadcec7b10 con 0x55dadcfe7ee0 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.540+0000 7f6db2187640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcfe7ee0 0x55dadcfe6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6d95ffb640 10 client.14994 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db2187640 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcfe7ee0 0x55dadcfe6bf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c64485b66061880 server_cookie=245626c596beee29 in_seq=0 out_seq=0 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6d95ffb640 1 -- 172.21.15.16:0/1849266929 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dadcec7b10 con 0x55dadcfe7ee0 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6d95ffb640 10 client.14994 handle_command_reply: tid=0 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 client.14994 shutdown 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 msgr2=0x55dadcf81830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c02a940 tx=0x7f6d9c03f040 comp rx=0 tx=0).stop 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 shutdown_connections 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55dadcfe7ee0 0x55dadcfe6bf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dadcf68fc0 0x55dadcf822b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dadcf5a1f0 0x55dadcf81d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 --2- 172.21.15.16:0/1849266929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dadcf598d0 0x55dadcf81830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 >> 172.21.15.16:0/1849266929 conn(0x55dadcf55f60 msgr2=0x55dadcf61240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6d95ffb640 0 client.14994 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 shutdown_connections 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.541+0000 7f6db67c4740 1 -- 172.21.15.16:0/1849266929 wait complete. 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db67c4740 20 client.14994 trim_cache size 0 max 16384 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24445, 2024-09-12T02:47:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 67.311906762000007, 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 94.832437056000003 2024-09-12T02:47:03.551 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 msgr2=0x7f6d980a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 0x7f6d980a5150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6da4098840 tx=0x7f6da406de80 comp rx=0 tx=0).stop 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 msgr2=0x7f6dac19cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6da400cce0 tx=0x7f6da400a620 comp rx=0 tx=0).stop 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 shutdown_connections 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d980a2c90 0x7f6d980a5150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac106a80 0x7f6dac19cd90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dac101420 0x7f6dac198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 --2- 172.21.15.16:0/4023497971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dac100a80 0x7f6dac1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:03.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 >> 172.21.15.16:0/4023497971 conn(0x7f6dac0fc7a0 msgr2=0x7f6dac06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:03.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 shutdown_connections 2024-09-12T02:47:03.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:03.542+0000 7f6db43eb640 1 -- 172.21.15.16:0/4023497971 wait complete. 2024-09-12T02:47:04.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:03 smithi016 ceph-mon[30925]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-12T02:47:04.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.h status 2024-09-12T02:47:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:03 smithi033 ceph-mon[35092]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-12T02:47:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:03 smithi184 ceph-mon[35321]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-12T02:47:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 -- 172.21.15.16:0/4143519338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4106a90 msgr2=0x7fc8f4106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/4143519338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4106a90 0x7fc8f4106e70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009930 tx=0x7fc8e402a420 comp rx=0 tx=0).stop 2024-09-12T02:47:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 -- 172.21.15.16:0/4143519338 shutdown_connections 2024-09-12T02:47:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/4143519338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/4143519338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4100a90 0x7fc8f4100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/4143519338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4106a90 0x7fc8f4106e70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 -- 172.21.15.16:0/4143519338 >> 172.21.15.16:0/4143519338 conn(0x7fc8f40fc7d0 msgr2=0x7fc8f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.898+0000 7fc8f97e6640 1 -- 172.21.15.16:0/4143519338 shutdown_connections 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.899+0000 7fc8f97e6640 1 -- 172.21.15.16:0/4143519338 wait complete. 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.899+0000 7fc8f97e6640 1 Processor -- start 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.899+0000 7fc8f97e6640 1 -- start start 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 0x7fc8f419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f4110e90 con 0x7fc8f4100a90 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f4110d10 con 0x7fc8f4106a90 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f4111010 con 0x7fc8f4101430 2024-09-12T02:47:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47550/0 (socket says 172.21.15.16:47550) 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2731295400 learned_addr learned my addr 172.21.15.16:0/2731295400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 0x7fc8f419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2731295400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 msgr2=0x7fc8f419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2731295400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 msgr2=0x7fc8f419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 0x7fc8f419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2731295400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f419f5d0 con 0x7fc8f4100a90 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 0x7fc8f419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009230 tx=0x7fc8e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8dbfff640 1 -- 172.21.15.16:0/2731295400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e403f030 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f419f860 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.900+0000 7fc8dbfff640 1 -- 172.21.15.16:0/2731295400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8e4003ad0 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.901+0000 7fc8dbfff640 1 -- 172.21.15.16:0/2731295400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4035070 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.901+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f41a71a0 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8dbfff640 1 -- 172.21.15.16:0/2731295400 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc8e4042030 con 0x7fc8f4100a90 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 Processor -- start 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 -- start start 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 0x55af9b83edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b849bb0 0x55af9b84bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af9b7a5090 con 0x55af9b83f2f0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af9b74ca70 con 0x55af9b849bb0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.903+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af9b7a1050 con 0x55af9b83e9d0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47554/0 (socket says 172.21.15.16:47554) 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/4037054803 learned_addr learned my addr 172.21.15.16:0/4037054803 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8dbfff640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 0x7fc8dc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b849bb0 0x55af9b84bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 0x55af9b83edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 0x7fc8dc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/4037054803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 msgr2=0x55af9b83edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 0x55af9b83edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/4037054803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b849bb0 msgr2=0x55af9b84bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b849bb0 0x55af9b84bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8dbfff640 1 -- 172.21.15.16:0/2731295400 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc8e40ef7c0 con 0x7fc8f4100a90 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/4037054803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af9b759130 con 0x55af9b83f2f0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 0x55af9b83edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40ed060 tx=0x7fc8e40045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541fcf111828eacd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 0x7fc8dc0a50d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec005c40 tx=0x7fc8ec009200 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8d97fa640 1 -- 172.21.15.16:0/4037054803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4049020 con 0x55af9b83f2f0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8d97fa640 1 -- 172.21.15.16:0/4037054803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8e40c0220 con 0x55af9b83f2f0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.904+0000 7fc8d97fa640 1 -- 172.21.15.16:0/4037054803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4003c80 con 0x55af9b83f2f0 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/4037054803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 msgr2=0x55af9b849670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40ed060 tx=0x7fc8e40045f0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/4037054803 shutdown_connections 2024-09-12T02:47:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b849bb0 0x55af9b84bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b849670 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/4037054803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b83e9d0 0x55af9b83edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/4037054803 >> 172.21.15.16:0/4037054803 conn(0x55af9b83af60 msgr2=0x55af9b83b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/4037054803 shutdown_connections 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/4037054803 wait complete. 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.905+0000 7fc8fbbbf740 1 -- start start 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 Processor -- start 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b849400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b846c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af9b84c540 con 0x55af9b83f2f0 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af9b75ff20 con 0x55af9b83e9d0 2024-09-12T02:47:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af9b84c6c0 con 0x55af9b849bb0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8fbbbf740 10 client.0 _ll_register_callbacks cb 0x55af9b729030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47556/0 (socket says 172.21.15.16:47556) 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 -- 172.21.15.16:0/155816102 learned_addr learned my addr 172.21.15.16:0/155816102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b849400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b846c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 -- 172.21.15.16:0/155816102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 msgr2=0x55af9b846c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b846c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 -- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 msgr2=0x55af9b849400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b849400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 -- 172.21.15.16:0/155816102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9b5162b0 con 0x55af9b83f2f0 2024-09-12T02:47:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8011a60 tx=0x7fc8e8011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c41b53dcaef7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.906+0000 7fc8d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800fe70 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc8e8005020 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8e40c0220 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc8e8005220 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8fbbbf740 10 client.15006 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af9b857de0 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8e40c0220 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc8e801b590 con 0x55af9b83f2f0 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8fbbbf740 10 client.15006 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.907+0000 7fc8fbbbf740 10 client.15006 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8fbbbf740 20 client.15006 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:04.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b85a4a0 0x55af9b85b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8fbbbf740 4 client.15006 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:04.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af9b7b2600 con 0x55af9b85a4a0 2024-09-12T02:47:04.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b85a4a0 0x55af9b85b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8d6ff5640 10 client.15006 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.908+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b85a4a0 0x55af9b85b910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acf2066f5743a952 server_cookie=be533e1b9cdb189 in_seq=0 out_seq=0 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/155816102 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55af9b7b2600 con 0x55af9b85a4a0 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8d6ff5640 10 client.15006 handle_command_reply: tid=0 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 client.15006 shutdown 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 msgr2=0x55af9b846750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8011a60 tx=0x7fc8e8011f20 comp rx=0 tx=0).stop 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 shutdown_connections 2024-09-12T02:47:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b85a4a0 0x55af9b85b910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b846c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b846750 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/155816102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b849400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 >> 172.21.15.16:0/155816102 conn(0x55af9b83af60 msgr2=0x55af9b840c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8d6ff5640 0 client.15006 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 shutdown_connections 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.909+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/155816102 wait complete. 2024-09-12T02:47:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.910+0000 7fc8fbbbf740 20 client.15006 trim_cache size 0 max 16384 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.927+0000 7fc8fbbbf740 1 Processor -- start 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.927+0000 7fc8fbbbf740 1 -- start start 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83f2f0 0x55af9b845020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b8550a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af9b859c50 con 0x55af9b83e9d0 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af9b85a320 con 0x55af9b83f2f0 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af9b85a9f0 con 0x55af9b849bb0 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47572/0 (socket says 172.21.15.16:47572) 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2575431697 learned_addr learned my addr 172.21.15.16:0/2575431697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83f2f0 0x55af9b845020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2575431697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 msgr2=0x55af9b8550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b8550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b8550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2575431697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83f2f0 msgr2=0x55af9b845020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83f2f0 0x55af9b845020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 -- 172.21.15.16:0/2575431697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af9b83f8c0 con 0x55af9b83e9d0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009a50 tx=0x7fc8e40dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5269992fdc15416 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8d8ff9640 1 -- 172.21.15.16:0/2575431697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4049040 con 0x55af9b83e9d0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8d8ff9640 1 -- 172.21.15.16:0/2575431697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8e40b84e0 con 0x55af9b83e9d0 2024-09-12T02:47:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.928+0000 7fc8d8ff9640 1 -- 172.21.15.16:0/2575431697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40ed520 con 0x55af9b83e9d0 2024-09-12T02:47:04.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2575431697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 msgr2=0x55af9b85a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009a50 tx=0x7fc8e40dc040 comp rx=0 tx=0).stop 2024-09-12T02:47:04.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2575431697 shutdown_connections 2024-09-12T02:47:04.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b8550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83f2f0 0x55af9b845020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2575431697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83e9d0 0x55af9b85a4a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2575431697 >> 172.21.15.16:0/2575431697 conn(0x55af9b83af60 msgr2=0x55af9b849400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2575431697 shutdown_connections 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2575431697 wait complete. 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.929+0000 7fc8fbbbf740 1 -- start start 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 Processor -- start 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b864470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b85ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af9b85b790 con 0x55af9b83f2f0 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af9b85b0c0 con 0x55af9b83e9d0 2024-09-12T02:47:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af9b85be60 con 0x55af9b849bb0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8fbbbf740 10 client.0 _ll_register_callbacks cb 0x7fc8e800c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47576/0 (socket says 172.21.15.16:47576) 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 -- 172.21.15.16:0/2767871041 learned_addr learned my addr 172.21.15.16:0/2767871041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b864470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b85ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 -- 172.21.15.16:0/2767871041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 msgr2=0x55af9b85ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b85ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 -- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 msgr2=0x55af9b864470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b864470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 -- 172.21.15.16:0/2767871041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9b8c2ba0 con 0x55af9b83f2f0 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f37fe640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b85ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b864470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8f27fc640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8011a30 tx=0x7fc8e8005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3654f89a2fc7a731 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.930+0000 7fc8d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8033060 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc8e801b600 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8e40b84e0 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc8e802a040 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 10 client.15012 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af9b8c3d70 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8e40b84e0 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc8e802d070 con 0x55af9b83f2f0 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 10 client.15012 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 10 client.15012 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:47:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 20 client.15012 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b8c8980 0x55af9b8c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 4 client.15012 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.931+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "status"}) -- 0x55af9b7acb10 con 0x55af9b8c8980 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b8c8980 0x55af9b8c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8d6ff5640 10 client.15012 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8f2ffd640 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b8c8980 0x55af9b8c7690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66a5dda50c5f323b server_cookie=b3afcb3665d066c2 in_seq=0 out_seq=0 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8d6ff5640 1 -- 172.21.15.16:0/2767871041 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55af9b7acb10 con 0x55af9b8c8980 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8d6ff5640 10 client.15012 handle_command_reply: tid=0 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 client.15012 shutdown 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 msgr2=0x55af9b85e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8011a30 tx=0x7fc8e8005670 comp rx=0 tx=0).stop 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 shutdown_connections 2024-09-12T02:47:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55af9b8c8980 0x55af9b8c7690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af9b849bb0 0x55af9b85ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af9b83f2f0 0x55af9b85e830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 --2- 172.21.15.16:0/2767871041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af9b83e9d0 0x55af9b864470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 >> 172.21.15.16:0/2767871041 conn(0x55af9b83af60 msgr2=0x55af9b83b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8d6ff5640 0 client.15012 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 shutdown_connections 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.932+0000 7fc8fbbbf740 1 -- 172.21.15.16:0/2767871041 wait complete. 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8fbbbf740 20 client.15012 trim_cache size 0 max 16384 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:04.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24467, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 69.713152385000001, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 98.718842623 2024-09-12T02:47:04.940 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 msgr2=0x7fc8dc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 0x7fc8dc0a50d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec005c40 tx=0x7fc8ec009200 comp rx=0 tx=0).stop 2024-09-12T02:47:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 msgr2=0x7fc8f419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009230 tx=0x7fc8e4004270 comp rx=0 tx=0).stop 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 shutdown_connections 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8f4106a90 0x7fc8f419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8f4101430 0x7fc8f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8dc0a2c10 0x7fc8dc0a50d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 --2- 172.21.15.16:0/2731295400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8f4100a90 0x7fc8f419a620 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 >> 172.21.15.16:0/2731295400 conn(0x7fc8f40fc7d0 msgr2=0x7fc8f410c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 shutdown_connections 2024-09-12T02:47:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:04.933+0000 7fc8f97e6640 1 -- 172.21.15.16:0/2731295400 wait complete. 2024-09-12T02:47:05.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.b status 2024-09-12T02:47:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:06.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:05 smithi016 ceph-mon[30925]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-12T02:47:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:05 smithi033 ceph-mon[35092]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-12T02:47:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- 172.21.15.16:0/2005042709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc106f10 msgr2=0x7f3ffc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 --2- 172.21.15.16:0/2005042709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc106f10 0x7f3ffc10ea90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009ca0 tx=0x7f3ff402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- 172.21.15.16:0/2005042709 shutdown_connections 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 --2- 172.21.15.16:0/2005042709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc106f10 0x7f3ffc10ea90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 --2- 172.21.15.16:0/2005042709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 0x7f3ffc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 --2- 172.21.15.16:0/2005042709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc101de0 0x7f3ffc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- 172.21.15.16:0/2005042709 >> 172.21.15.16:0/2005042709 conn(0x7f3ffc0fd8c0 msgr2=0x7f3ffc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- 172.21.15.16:0/2005042709 shutdown_connections 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- 172.21.15.16:0/2005042709 wait complete. 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 Processor -- start 2024-09-12T02:47:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.291+0000 7f40041a3640 1 -- start start 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 0x7f3ffc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc106f10 0x7f3ffc19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc111f70 con 0x7f3ffc101de0 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc111df0 con 0x7f3ffc106f10 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f40041a3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc1120f0 con 0x7f3ffc102790 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50806/0 (socket says 172.21.15.16:50806) 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 -- 172.21.15.16:0/979877235 learned_addr learned my addr 172.21.15.16:0/979877235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4002740640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc106f10 0x7f3ffc19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f400173e640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 0x7f3ffc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 -- 172.21.15.16:0/979877235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 msgr2=0x7f3ffc1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 0x7f3ffc1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 -- 172.21.15.16:0/979877235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc106f10 msgr2=0x7f3ffc19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc106f10 0x7f3ffc19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 -- 172.21.15.16:0/979877235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc19d420 con 0x7f3ffc101de0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.292+0000 7f4001f3f640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800d950 tx=0x7f3ff800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.293+0000 7f3ff2ffd640 1 -- 172.21.15.16:0/979877235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8017070 con 0x7f3ffc101de0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.293+0000 7f3ff2ffd640 1 -- 172.21.15.16:0/979877235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ff8009c30 con 0x7f3ffc101de0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.293+0000 7f3ff2ffd640 1 -- 172.21.15.16:0/979877235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff800f6b0 con 0x7f3ffc101de0 2024-09-12T02:47:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.293+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc19d710 con 0x7f3ffc101de0 2024-09-12T02:47:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.293+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1a4ff0 con 0x7f3ffc101de0 2024-09-12T02:47:06.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:05 smithi184 ceph-mon[35321]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-12T02:47:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f3ff2ffd640 1 -- 172.21.15.16:0/979877235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3ff80109d0 con 0x7f3ffc101de0 2024-09-12T02:47:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 Processor -- start 2024-09-12T02:47:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 -- start start 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde031cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde040a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bbddf98090 con 0x55bbde040fc0 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbddf3fa70 con 0x55bbde0318f0 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400657c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bbddf94050 con 0x55bbde032210 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.296+0000 7f400173e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50818/0 (socket says 172.21.15.16:50818) 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 -- 172.21.15.16:0/3994439031 learned_addr learned my addr 172.21.15.16:0/3994439031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f4002740640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde031cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f4001f3f640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 -- 172.21.15.16:0/3994439031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 msgr2=0x55bbde040a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f3ff2ffd640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 0x7f3fe80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde040a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 -- 172.21.15.16:0/3994439031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 msgr2=0x55bbde031cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde031cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 -- 172.21.15.16:0/3994439031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbddf4c130 con 0x55bbde040fc0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f4001f3f640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde040a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f4002740640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 0x7f3fe80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f3ff2ffd640 1 -- 172.21.15.16:0/979877235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ff80ceca0 con 0x7f3ffc101de0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f400173e640 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00bc30 tx=0x7f3fec0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4464dcd5649b55 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.297+0000 7f3fe7fff640 1 -- 172.21.15.16:0/3994439031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec01c480 con 0x55bbde040fc0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f4002740640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 0x7f3fe80a5150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4004af0 tx=0x7f3ff40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f3fe7fff640 1 -- 172.21.15.16:0/3994439031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fec01cb00 con 0x55bbde040fc0 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f3fe7fff640 1 -- 172.21.15.16:0/3994439031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec011b50 con 0x55bbde040fc0 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 -- 172.21.15.16:0/3994439031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 msgr2=0x55bbde0433b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00bc30 tx=0x7f3fec0107a0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 -- 172.21.15.16:0/3994439031 shutdown_connections 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde040fc0 0x55bbde0433b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde040a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 --2- 172.21.15.16:0/3994439031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde031cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.298+0000 7f400657c740 1 -- 172.21.15.16:0/3994439031 >> 172.21.15.16:0/3994439031 conn(0x55bbde02df60 msgr2=0x55bbde02e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- 172.21.15.16:0/3994439031 shutdown_connections 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- 172.21.15.16:0/3994439031 wait complete. 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- start start 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 Processor -- start 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde0408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde040fc0 0x55bbde03e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bbde043b30 con 0x55bbde032210 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbddf52f20 con 0x55bbde0318f0 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bbde043cb0 con 0x55bbde040fc0 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400173e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.299+0000 7f400657c740 10 client.0 _ll_register_callbacks cb 0x55bbddf1c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50834/0 (socket says 172.21.15.16:50834) 2024-09-12T02:47:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 -- 172.21.15.16:0/1445452962 learned_addr learned my addr 172.21.15.16:0/1445452962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f4001f3f640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde0408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f4002740640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde040fc0 0x55bbde03e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 -- 172.21.15.16:0/1445452962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde040fc0 msgr2=0x55bbde03e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde040fc0 0x55bbde03e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 -- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 msgr2=0x55bbde0408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde0408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 -- 172.21.15.16:0/1445452962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbddd092b0 con 0x55bbde032210 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400173e640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3fec011690 tx=0x7f3fec00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a587831a3144ea7f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f3fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec02b060 con 0x55bbde032210 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3fec01c690 con 0x55bbde032210 2024-09-12T02:47:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fec01cb00 con 0x55bbde032210 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3fec01c890 con 0x55bbde032210 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400657c740 10 client.15024 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.300+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbde04f450 con 0x55bbde032210 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fec01cb00 con 0x55bbde032210 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3fec0046c0 con 0x55bbde032210 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 10 client.15024 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 10 client.15024 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 20 client.15024 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde051640 0x55bbde052ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 4 client.15024 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbddfa5600 con 0x55bbde051640 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.301+0000 7f4001f3f640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde051640 0x55bbde052ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.302+0000 7f3fe57fa640 10 client.15024 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.302+0000 7f4001f3f640 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde051640 0x55bbde052ab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff276534ea3510f server_cookie=c071fda275c08cb7 in_seq=0 out_seq=0 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.302+0000 7f3fe57fa640 1 -- 172.21.15.16:0/1445452962 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bbddfa5600 con 0x55bbde051640 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.302+0000 7f3fe57fa640 10 client.15024 handle_command_reply: tid=0 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.302+0000 7f400657c740 1 client.15024 shutdown 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 msgr2=0x55bbde03dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3fec011690 tx=0x7f3fec00bf50 comp rx=0 tx=0).stop 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 shutdown_connections 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde051640 0x55bbde052ab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde040fc0 0x55bbde03e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde032210 0x55bbde03dc10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f3fe57fa640 0 client.15024 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 --2- 172.21.15.16:0/1445452962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde0318f0 0x55bbde0408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 >> 172.21.15.16:0/1445452962 conn(0x55bbde02df60 msgr2=0x55bbde033bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 shutdown_connections 2024-09-12T02:47:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 1 -- 172.21.15.16:0/1445452962 wait complete. 2024-09-12T02:47:06.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.303+0000 7f400657c740 20 client.15024 trim_cache size 0 max 16384 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 Processor -- start 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 -- start start 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde03ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde03b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bbde050df0 con 0x55bbde0318f0 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbde0514c0 con 0x55bbde040fc0 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400657c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bbde051b90 con 0x55bbde032210 2024-09-12T02:47:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50850/0 (socket says 172.21.15.16:50850) 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 -- 172.21.15.16:0/3531765689 learned_addr learned my addr 172.21.15.16:0/3531765689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4002740640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde03b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400173e640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde03ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 -- 172.21.15.16:0/3531765689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 msgr2=0x55bbde03ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde03ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 -- 172.21.15.16:0/3531765689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 msgr2=0x55bbde03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde03b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 -- 172.21.15.16:0/3531765689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbde032000 con 0x55bbde0318f0 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f400173e640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde03ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4002740640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde03b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.321+0000 7f4001f3f640 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80978c0 tx=0x7f3ff80d1200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b26b977ca1b7932 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f3fe77fe640 1 -- 172.21.15.16:0/3531765689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8020040 con 0x55bbde0318f0 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f3fe77fe640 1 -- 172.21.15.16:0/3531765689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ff8082050 con 0x55bbde0318f0 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f3fe77fe640 1 -- 172.21.15.16:0/3531765689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff809a8b0 con 0x55bbde0318f0 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 -- 172.21.15.16:0/3531765689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 msgr2=0x55bbde052ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80978c0 tx=0x7f3ff80d1200 comp rx=0 tx=0).stop 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 -- 172.21.15.16:0/3531765689 shutdown_connections 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde03b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde03ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 --2- 172.21.15.16:0/3531765689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde052ab0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 -- 172.21.15.16:0/3531765689 >> 172.21.15.16:0/3531765689 conn(0x55bbde02df60 msgr2=0x55bbde039240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 -- 172.21.15.16:0/3531765689 shutdown_connections 2024-09-12T02:47:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.322+0000 7f400657c740 1 -- 172.21.15.16:0/3531765689 wait complete. 2024-09-12T02:47:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 -- start start 2024-09-12T02:47:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 Processor -- start 2024-09-12T02:47:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde05b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde055cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bbde052930 con 0x55bbde0318f0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bbde052260 con 0x55bbde040fc0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bbde053000 con 0x55bbde032210 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400657c740 10 client.0 _ll_register_callbacks cb 0x7f3ff4009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50860/0 (socket says 172.21.15.16:50860) 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 -- 172.21.15.16:0/91375383 learned_addr learned my addr 172.21.15.16:0/91375383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4002740640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde055cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f400173e640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde05b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 -- 172.21.15.16:0/91375383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 msgr2=0x55bbde05b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde05b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 -- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 msgr2=0x55bbde055cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.323+0000 7f4001f3f640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde055cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f4001f3f640 1 -- 172.21.15.16:0/91375383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbde056200 con 0x55bbde0318f0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f400173e640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde05b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f4002740640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde055cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f4001f3f640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8097e30 tx=0x7f3ff80d1190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd1114b064acc0d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80b9060 con 0x55bbde0318f0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ff80b4260 con 0x55bbde0318f0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff8082050 con 0x55bbde0318f0 2024-09-12T02:47:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ff8020070 con 0x55bbde0318f0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f400657c740 10 client.15030 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbde0bc3d0 con 0x55bbde0318f0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ff8082050 con 0x55bbde0318f0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3ff80a6070 con 0x55bbde0318f0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.324+0000 7f400657c740 10 client.15030 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 10 client.15030 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 20 client.15030 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde0c0a00 0x55bbde0bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 4 client.15030 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "status"}) -- 0x55bbddf9fb10 con 0x55bbde0c0a00 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400173e640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde0c0a00 0x55bbde0bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400173e640 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde0c0a00 0x55bbde0bf710 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=537b7606cf170f41 server_cookie=7ad6dba1f4812913 in_seq=0 out_seq=0 2024-09-12T02:47:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f3fe57fa640 10 client.15030 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f3fe57fa640 1 -- 172.21.15.16:0/91375383 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55bbddf9fb10 con 0x55bbde0c0a00 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f3fe57fa640 10 client.15030 handle_command_reply: tid=0 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 1 client.15030 shutdown 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.325+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 msgr2=0x55bbde05ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8097e30 tx=0x7f3ff80d1190 comp rx=0 tx=0).stop 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 shutdown_connections 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bbde0c0a00 0x55bbde0bf710 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bbde040fc0 0x55bbde055cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bbde032210 0x55bbde05b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f3fe57fa640 0 client.15030 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 --2- 172.21.15.16:0/91375383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bbde0318f0 0x55bbde05ad10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 >> 172.21.15.16:0/91375383 conn(0x55bbde02df60 msgr2=0x55bbde03d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 shutdown_connections 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 1 -- 172.21.15.16:0/91375383 wait complete. 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f400657c740 20 client.15030 trim_cache size 0 max 16384 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:06.333 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 78.144675196999998, 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 88.768000779999994 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 msgr2=0x7f3fe80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 0x7f3fe80a5150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4004af0 tx=0x7f3ff40095c0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 msgr2=0x7f3ffc198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800d950 tx=0x7f3ff800de10 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 shutdown_connections 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ffc106f10 0x7f3ffc19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc102790 0x7f3ffc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3fe80a2c90 0x7f3fe80a5150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 --2- 172.21.15.16:0/979877235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc101de0 0x7f3ffc198470 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.326+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 >> 172.21.15.16:0/979877235 conn(0x7f3ffc0fd8c0 msgr2=0x7f3ffc0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.327+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 shutdown_connections 2024-09-12T02:47:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:06.327+0000 7f40041a3640 1 -- 172.21.15.16:0/979877235 wait complete. 2024-09-12T02:47:07.142 INFO:tasks.cephadm:Setup complete, yielding 2024-09-12T02:47:07.142 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-12T02:47:07.388 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T02:47:07.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph orch status' 2024-09-12T02:47:07.646 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:08.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:07 smithi016 ceph-mon[30925]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:47:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/1834334906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8101c00 msgr2=0x7fbdc8102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/1834334906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8101c00 0x7fbdc8102000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009900 tx=0x7fbdbc02a410 comp rx=0 tx=0).stop 2024-09-12T02:47:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/1834334906 shutdown_connections 2024-09-12T02:47:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/1834334906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 0x7fbdc810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/1834334906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8102e00 0x7fbdc8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/1834334906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8101c00 0x7fbdc8102000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/1834334906 >> 172.21.15.16:0/1834334906 conn(0x7fbdc80fd390 msgr2=0x7fbdc80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/1834334906 shutdown_connections 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.082+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/1834334906 wait complete. 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 Processor -- start 2024-09-12T02:47:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 -- start start 2024-09-12T02:47:08.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8101c00 0x7fbdc8075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:08.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:08.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 0x7fbdc8074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:08.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc810d880 con 0x7fbdc8102e00 2024-09-12T02:47:08.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc810d700 con 0x7fbdc81037c0 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcf0ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc810da00 con 0x7fbdc8101c00 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50882/0 (socket says 172.21.15.16:50882) 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 -- 172.21.15.16:0/2374414195 learned_addr learned my addr 172.21.15.16:0/2374414195 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcce6a640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8101c00 0x7fbdc8075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdcd66b640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 0x7fbdc8074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 -- 172.21.15.16:0/2374414195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8101c00 msgr2=0x7fbdc8075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8101c00 0x7fbdc8075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 -- 172.21.15.16:0/2374414195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 msgr2=0x7fbdc8074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.083+0000 7fbdc7fff640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 0x7fbdc8074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdc7fff640 1 -- 172.21.15.16:0/2374414195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc80753e0 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdc7fff640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400d770 tx=0x7fbdb400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4017070 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb4005500 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc81a20a0 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4004e00 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.084+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc81a25b0 con 0x7fbdc8102e00 2024-09-12T02:47:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.085+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc8108ff0 con 0x7fbdc8102e00 2024-09-12T02:47:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.086+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbdb4004fa0 con 0x7fbdc8102e00 2024-09-12T02:47:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.087+0000 7fbdc5ffb640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 0x7fbdb80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.087+0000 7fbdcce6a640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 0x7fbdb80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.087+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbdb40ceb60 con 0x7fbdc8102e00 2024-09-12T02:47:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.088+0000 7fbdcce6a640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 0x7fbdb80a5150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc005ab0 tx=0x7fbdbc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.091+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbdb409b970 con 0x7fbdc8102e00 2024-09-12T02:47:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:07 smithi033 ceph-mon[35092]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:47:08.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.283+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fbdc8062630 con 0x7fbdb80a2c90 2024-09-12T02:47:08.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.285+0000 7fbdc5ffb640 1 -- 172.21.15.16:0/2374414195 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fbdc8062630 con 0x7fbdb80a2c90 2024-09-12T02:47:08.285 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-09-12T02:47:08.286 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-09-12T02:47:08.286 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 msgr2=0x7fbdb80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 0x7fbdb80a5150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc005ab0 tx=0x7fbdbc000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 msgr2=0x7fbdc80745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400d770 tx=0x7fbdb400dc30 comp rx=0 tx=0).stop 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 shutdown_connections 2024-09-12T02:47:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdc81037c0 0x7fbdc8074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbdb80a2c90 0x7fbdb80a5150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdc8102e00 0x7fbdc80745e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 --2- 172.21.15.16:0/2374414195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdc8101c00 0x7fbdc8075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 >> 172.21.15.16:0/2374414195 conn(0x7fbdc80fd390 msgr2=0x7fbdc81061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 shutdown_connections 2024-09-12T02:47:08.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:08.287+0000 7fbdcf0ce640 1 -- 172.21.15.16:0/2374414195 wait complete. 2024-09-12T02:47:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:07 smithi184 ceph-mon[35321]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-09-12T02:47:08.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-12T02:47:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 -- 172.21.15.16:0/1903748404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c076220 msgr2=0x7fd90c074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 --2- 172.21.15.16:0/1903748404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c076220 0x7fd90c074680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd908009900 tx=0x7fd90802a410 comp rx=0 tx=0).stop 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 -- 172.21.15.16:0/1903748404 shutdown_connections 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 --2- 172.21.15.16:0/1903748404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 --2- 172.21.15.16:0/1903748404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 0x7fd90c0750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 --2- 172.21.15.16:0/1903748404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c076220 0x7fd90c074680 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 -- 172.21.15.16:0/1903748404 >> 172.21.15.16:0/1903748404 conn(0x7fd90c0fd390 msgr2=0x7fd90c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 -- 172.21.15.16:0/1903748404 shutdown_connections 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.721+0000 7fd91380f640 1 -- 172.21.15.16:0/1903748404 wait complete. 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 Processor -- start 2024-09-12T02:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 -- start start 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 0x7fd90c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c076220 0x7fd90c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c113ea0 con 0x7fd90c10e1e0 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c113d20 con 0x7fd90c076220 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd91380f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c114020 con 0x7fd90c074c50 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd911dac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd911dac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50908/0 (socket says 172.21.15.16:50908) 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd911dac640 1 -- 172.21.15.16:0/4114330246 learned_addr learned my addr 172.21.15.16:0/4114330246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd9115ab640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 0x7fd90c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.722+0000 7fd910daa640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c076220 0x7fd90c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 -- 172.21.15.16:0/4114330246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 msgr2=0x7fd90c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 0x7fd90c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 -- 172.21.15.16:0/4114330246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c076220 msgr2=0x7fd90c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c076220 0x7fd90c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 -- 172.21.15.16:0/4114330246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90c1a6140 con 0x7fd90c10e1e0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd911dac640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd90400cad0 tx=0x7fd90400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90400ae20 con 0x7fd90c10e1e0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90c1a6430 con 0x7fd90c10e1e0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904005020 con 0x7fd90c10e1e0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90400f6f0 con 0x7fd90c10e1e0 2024-09-12T02:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.723+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90c1a6970 con 0x7fd90c10e1e0 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.724+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd90c103b80 con 0x7fd90c10e1e0 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.725+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd9040051d0 con 0x7fd90c10e1e0 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.726+0000 7fd9027fc640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 0x7fd8f80a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.726+0000 7fd9115ab640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 0x7fd8f80a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.726+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd9040cf240 con 0x7fd90c10e1e0 2024-09-12T02:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.726+0000 7fd9115ab640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 0x7fd8f80a5180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd908005ab0 tx=0x7fd908000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.730+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd904014030 con 0x7fd90c10e1e0 2024-09-12T02:47:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:09 smithi016 ceph-mon[30925]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:09 smithi016 ceph-mon[30925]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.916+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fd90c062630 con 0x7fd8f80a2cc0 2024-09-12T02:47:09.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.928+0000 7fd9027fc640 1 -- 172.21.15.16:0/4114330246 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4154 (secure 0 0 0) 0x7fd90c062630 con 0x7fd8f80a2cc0 2024-09-12T02:47:09.928 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-12T02:47:09.928 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (95s) 19s ago 94s 20.2M - 19.1.1-298-gb53b6d3e 54467abc55f8 4e06ccc6ffd6 2024-09-12T02:47:09.928 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi033 running (92s) 26s ago 92s 26.6M - 19.1.1-298-gb53b6d3e 54467abc55f8 e7aac3a46c26 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi184 running (106s) 26s ago 106s 20.0M - 19.1.1-298-gb53b6d3e 54467abc55f8 be66189974ff 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (113s) 19s ago 113s 15.7M - 19.1.1-298-gb53b6d3e 54467abc55f8 28b3f8ce1d55 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi033 running (111s) 26s ago 110s 19.4M - 19.1.1-298-gb53b6d3e 54467abc55f8 2620755496c0 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi184 running (109s) 26s ago 108s 16.0M - 19.1.1-298-gb53b6d3e 54467abc55f8 4d7ae7378743 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (98s) 19s ago 98s 18.4M - 19.1.1-298-gb53b6d3e 54467abc55f8 84add6388aeb 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi033 running (104s) 26s ago 103s 17.2M - 19.1.1-298-gb53b6d3e 54467abc55f8 f10876fe397a 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi184 running (101s) 26s ago 101s 18.1M - 19.1.1-298-gb53b6d3e 54467abc55f8 fa46e3df1bc2 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (117s) 19s ago 117s 14.4M - 19.1.1-298-gb53b6d3e 54467abc55f8 761ab1bbce20 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi033 running (2m) 26s ago 2m 19.2M - 19.1.1-298-gb53b6d3e 54467abc55f8 4492eb4f0e6f 2024-09-12T02:47:09.929 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi184 running (2m) 26s ago 2m 17.0M - 19.1.1-298-gb53b6d3e 54467abc55f8 6f1e9195617f 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (8m) 19s ago 8m 509M - 19.1.1-298-gb53b6d3e 54467abc55f8 0b9f0e7e01b5 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi033 *:8443,8765 running (7m) 26s ago 7m 443M - 19.1.1-298-gb53b6d3e 54467abc55f8 5c319f05c857 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi184 *:8443,8765 running (7m) 26s ago 7m 444M - 19.1.1-298-gb53b6d3e 54467abc55f8 de85bfae89f5 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (8m) 19s ago 8m 50.7M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 6c85c08c3f0e 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi033 running (7m) 26s ago 7m 39.6M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 067da0701270 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi184 running (7m) 26s ago 7m 42.0M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 d0a3a7b75127 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 19s ago 6m 55.0M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 b10a65f87769 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi033 running (6m) 26s ago 6m 55.6M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 86dee804a9e9 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi184 running (6m) 26s ago 6m 57.3M 4272M 19.1.1-298-gb53b6d3e 54467abc55f8 dfe3b104dfed 2024-09-12T02:47:09.930 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (5m) 19s ago 5m 48.2M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 60dac1a7e993 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi033 running (5m) 26s ago 5m 44.7M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 2f407a9cb368 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi184 running (4m) 26s ago 4m 56.8M 4272M 19.1.1-298-gb53b6d3e 54467abc55f8 c80d02f9edaa 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (4m) 19s ago 4m 47.2M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 8976210906fc 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi033 running (4m) 26s ago 4m 55.0M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 71d64fcf79b4 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi184 running (3m) 26s ago 3m 45.5M 4272M 19.1.1-298-gb53b6d3e 54467abc55f8 84ba08f1808d 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (3m) 19s ago 3m 50.4M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 f0cca31c98cc 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi033 running (2m) 26s ago 2m 48.0M 4273M 19.1.1-298-gb53b6d3e 54467abc55f8 fad088127149 2024-09-12T02:47:09.931 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi184 running (2m) 26s ago 2m 50.2M 4272M 19.1.1-298-gb53b6d3e 54467abc55f8 2fd6c260a3bd 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 msgr2=0x7fd8f80a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 0x7fd8f80a5180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd908005ab0 tx=0x7fd908000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 msgr2=0x7fd90c1a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd90400cad0 tx=0x7fd90400cf90 comp rx=0 tx=0).stop 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 shutdown_connections 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd8f80a2cc0 0x7fd8f80a5180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c10e1e0 0x7fd90c1a5ad0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd90c076220 0x7fd90c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 --2- 172.21.15.16:0/4114330246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd90c074c50 0x7fd90c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 >> 172.21.15.16:0/4114330246 conn(0x7fd90c0fd390 msgr2=0x7fd90c0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 shutdown_connections 2024-09-12T02:47:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:09.930+0000 7fd91380f640 1 -- 172.21.15.16:0/4114330246 wait complete. 2024-09-12T02:47:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:09 smithi033 ceph-mon[35092]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:09 smithi033 ceph-mon[35092]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:09 smithi184 ceph-mon[35321]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:09 smithi184 ceph-mon[35321]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:10.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-12T02:47:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:11.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:10 smithi016 ceph-mon[30925]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:11.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:10 smithi033 ceph-mon[35092]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:11.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:10 smithi184 ceph-mon[35321]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- 172.21.15.16:0/33089777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec104fd0 msgr2=0x7fc7ec1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/33089777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec1073c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009900 tx=0x7fc7d402a410 comp rx=0 tx=0).stop 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- 172.21.15.16:0/33089777 shutdown_connections 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/33089777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec107900 0x7fc7ec109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/33089777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec1073c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/33089777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- 172.21.15.16:0/33089777 >> 172.21.15.16:0/33089777 conn(0x7fc7ec0fc2a0 msgr2=0x7fc7ec0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- 172.21.15.16:0/33089777 shutdown_connections 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- 172.21.15.16:0/33089777 wait complete. 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 Processor -- start 2024-09-12T02:47:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.337+0000 7fc7f1e38640 1 -- start start 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec10d910 con 0x7fc7ec107900 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec10d790 con 0x7fc7ec104fd0 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7f1e38640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec10da90 con 0x7fc7ec1026a0 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7eb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7eb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42980/0 (socket says 172.21.15.16:42980) 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50922/0 (socket says 172.21.15.16:50922) 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7eb7fe640 1 -- 172.21.15.16:0/1837905584 learned_addr learned my addr 172.21.15.16:0/1837905584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7e2ffd640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 -- 172.21.15.16:0/1837905584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 msgr2=0x7fc7ec197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 -- 172.21.15.16:0/1837905584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 msgr2=0x7fc7ec198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7ebfff640 1 -- 172.21.15.16:0/1837905584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec19fbf0 con 0x7fc7ec107900 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7eb7fe640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.338+0000 7fc7e2ffd640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7ebfff640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400be20 tx=0x7fc7e400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4019070 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ec19fee0 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e4014920 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4012920 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.339+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ec1a03c0 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.340+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ec0fdd90 con 0x7fc7ec107900 2024-09-12T02:47:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.341+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc7e401c070 con 0x7fc7ec107900 2024-09-12T02:47:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.342+0000 7fc7e97fa640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 0x7fc7d80a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.342+0000 7fc7eb7fe640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 0x7fc7d80a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.343+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc7e40ce8d0 con 0x7fc7ec107900 2024-09-12T02:47:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.343+0000 7fc7eb7fe640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 0x7fc7d80a5100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0104e0 tx=0x7fc7dc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:11.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.347+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc7e409b1a0 con 0x7fc7ec107900 2024-09-12T02:47:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.531+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fc7ec062630 con 0x7fc7d80a2c40 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.536+0000 7fc7e97fa640 1 -- 172.21.15.16:0/1837905584 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fc7ec062630 con 0x7fc7d80a2c40 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 27s ago 2m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 27s ago 7m smithi016=x;smithi033=y;smithi184=z;count:3 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 27s ago 7m smithi016:172.21.15.16=a;smithi033:172.21.15.33=b;smithi184:172.21.15.184=c;count:3 2024-09-12T02:47:11.537 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 27s ago - 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.538+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 msgr2=0x7fc7d80a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.538+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 0x7fc7d80a5100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0104e0 tx=0x7fc7dc00b410 comp rx=0 tx=0).stop 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.538+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 msgr2=0x7fc7ec19f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.538+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400be20 tx=0x7fc7e400bf10 comp rx=0 tx=0).stop 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 shutdown_connections 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7d80a2c40 0x7fc7d80a5100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7ec107900 0x7fc7ec19f580 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7ec104fd0 0x7fc7ec198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 --2- 172.21.15.16:0/1837905584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7ec1026a0 0x7fc7ec197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 >> 172.21.15.16:0/1837905584 conn(0x7fc7ec0fc2a0 msgr2=0x7fc7ec102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 shutdown_connections 2024-09-12T02:47:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:11.539+0000 7fc7f1e38640 1 -- 172.21.15.16:0/1837905584 wait complete. 2024-09-12T02:47:12.134 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:11 smithi016 ceph-mon[30925]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:12.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-12T02:47:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:11 smithi033 ceph-mon[35092]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:11 smithi184 ceph-mon[35321]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 -- 172.21.15.16:0/3067511059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 msgr2=0x7fcbc410a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 --2- 172.21.15.16:0/3067511059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc410a050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009ca0 tx=0x7fcbbc02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 -- 172.21.15.16:0/3067511059 shutdown_connections 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 --2- 172.21.15.16:0/3067511059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc410a050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 --2- 172.21.15.16:0/3067511059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4102e00 0x7fcbc4103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 --2- 172.21.15.16:0/3067511059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4101c00 0x7fcbc4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 -- 172.21.15.16:0/3067511059 >> 172.21.15.16:0/3067511059 conn(0x7fcbc40fd390 msgr2=0x7fcbc40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 -- 172.21.15.16:0/3067511059 shutdown_connections 2024-09-12T02:47:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.888+0000 7fcbca039640 1 -- 172.21.15.16:0/3067511059 wait complete. 2024-09-12T02:47:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 Processor -- start 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 -- start start 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 0x7fcbc4075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4102e00 0x7fcbc40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410d860 con 0x7fcbc41037c0 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410d6e0 con 0x7fcbc4101c00 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.889+0000 7fcbca039640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410d9e0 con 0x7fcbc4102e00 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50944/0 (socket says 172.21.15.16:50944) 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 -- 172.21.15.16:0/1060960149 learned_addr learned my addr 172.21.15.16:0/1060960149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:12.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc2ffd640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4102e00 0x7fcbc40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc37fe640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 0x7fcbc4075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 -- 172.21.15.16:0/1060960149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4102e00 msgr2=0x7fcbc40745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4102e00 0x7fcbc40745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 -- 172.21.15.16:0/1060960149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 msgr2=0x7fcbc4075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 0x7fcbc4075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 -- 172.21.15.16:0/1060960149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc40753e0 con 0x7fcbc41037c0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc37fe640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 0x7fcbc4075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc3fff640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc031ae0 tx=0x7fcbbc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc003c30 con 0x7fcbc41037c0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc003dd0 con 0x7fcbc41037c0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc41a1f90 con 0x7fcbc41037c0 2024-09-12T02:47:12.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc02fcb0 con 0x7fcbc41037c0 2024-09-12T02:47:12.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.890+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc41a2440 con 0x7fcbc41037c0 2024-09-12T02:47:12.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.891+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc4108f80 con 0x7fcbc41037c0 2024-09-12T02:47:12.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.893+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcbbc004050 con 0x7fcbc41037c0 2024-09-12T02:47:12.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.893+0000 7fcbc0ff9640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 0x7fcbac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:12.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.893+0000 7fcbc37fe640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 0x7fcbac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:12.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.893+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbbc0f0510 con 0x7fcbc41037c0 2024-09-12T02:47:12.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.894+0000 7fcbc37fe640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 0x7fcbac0a50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005d00 tx=0x7fcbb4005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:12.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:12.897+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcbbc0bd430 con 0x7fcbc41037c0 2024-09-12T02:47:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.079+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fcbc4062630 con 0x7fcbac0a2c10 2024-09-12T02:47:13.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:12 smithi016 ceph-mon[30925]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.081+0000 7fcbc0ff9640 1 -- 172.21.15.16:0/1060960149 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fcbc4062630 con 0x7fcbac0a2c10 2024-09-12T02:47:13.081 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-09-12T02:47:13.081 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-09-12T02:47:13.081 INFO:teuthology.orchestra.run.smithi016.stdout:smithi033 172.21.15.33 2024-09-12T02:47:13.082 INFO:teuthology.orchestra.run.smithi016.stdout:smithi184 172.21.15.184 2024-09-12T02:47:13.082 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-09-12T02:47:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 msgr2=0x7fcbac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 0x7fcbac0a50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005d00 tx=0x7fcbb4005c90 comp rx=0 tx=0).stop 2024-09-12T02:47:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 msgr2=0x7fcbc4074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc031ae0 tx=0x7fcbbc031ee0 comp rx=0 tx=0).stop 2024-09-12T02:47:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 shutdown_connections 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbac0a2c10 0x7fcbac0a50d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41037c0 0x7fcbc4074b20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4102e00 0x7fcbc40745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 --2- 172.21.15.16:0/1060960149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101c00 0x7fcbc4075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.083+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 >> 172.21.15.16:0/1060960149 conn(0x7fcbc40fd390 msgr2=0x7fcbc4106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.084+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 shutdown_connections 2024-09-12T02:47:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:13.084+0000 7fcbca039640 1 -- 172.21.15.16:0/1060960149 wait complete. 2024-09-12T02:47:13.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:12 smithi033 ceph-mon[35092]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:13.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:12 smithi184 ceph-mon[35321]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:13.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-12T02:47:13.978 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:14.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:13 smithi033 ceph-mon[35092]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:14.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:13 smithi016 ceph-mon[30925]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:14.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:13 smithi184 ceph-mon[35321]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:14.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 -- 172.21.15.16:0/3929316270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8101c00 msgr2=0x7f9ef8102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:14.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 --2- 172.21.15.16:0/3929316270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8101c00 0x7f9ef8102000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009900 tx=0x7f9eec02a410 comp rx=0 tx=0).stop 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 -- 172.21.15.16:0/3929316270 shutdown_connections 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 --2- 172.21.15.16:0/3929316270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 --2- 172.21.15.16:0/3929316270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102e00 0x7f9ef8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 --2- 172.21.15.16:0/3929316270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8101c00 0x7f9ef8102000 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 -- 172.21.15.16:0/3929316270 >> 172.21.15.16:0/3929316270 conn(0x7f9ef80fd390 msgr2=0x7f9ef80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 -- 172.21.15.16:0/3929316270 shutdown_connections 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.444+0000 7f9efebf0640 1 -- 172.21.15.16:0/3929316270 wait complete. 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 Processor -- start 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 -- start start 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8101c00 0x7f9ef8197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:14.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d890 con 0x7f9ef8102e00 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d710 con 0x7f9ef8101c00 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9efebf0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810da10 con 0x7f9ef81037c0 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9ef7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9ef7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50960/0 (socket says 172.21.15.16:50960) 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.445+0000 7f9ef7fff640 1 -- 172.21.15.16:0/2914604662 learned_addr learned my addr 172.21.15.16:0/2914604662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9efd18d640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9efc98c640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8101c00 0x7f9ef8197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 -- 172.21.15.16:0/2914604662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 msgr2=0x7f9ef819f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef819f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 -- 172.21.15.16:0/2914604662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8101c00 msgr2=0x7f9ef8197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8101c00 0x7f9ef8197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 -- 172.21.15.16:0/2914604662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef819fab0 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9efd18d640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400c920 tx=0x7f9ee400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4007c70 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4007e10 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef819fda0 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4018890 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.446+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef81a0250 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.447+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef8108f30 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.448+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9ee4018af0 con 0x7f9ef8102e00 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.449+0000 7f9ef5ffb640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 0x7f9ee80a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.449+0000 7f9efc98c640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 0x7f9ee80a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.449+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9ee40cf370 con 0x7f9ef8102e00 2024-09-12T02:47:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.450+0000 7f9efc98c640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 0x7f9ee80a5100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006010 tx=0x7f9eec000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.454+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9ee4014030 con 0x7f9ef8102e00 2024-09-12T02:47:14.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.635+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f9ef8062630 con 0x7f9ee80a2c40 2024-09-12T02:47:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.637+0000 7f9ef5ffb640 1 -- 172.21.15.16:0/2914604662 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f9ef8062630 con 0x7f9ee80a2c40 2024-09-12T02:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-12T02:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-12T02:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-12T02:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout:smithi184 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT626500CE400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 msgr2=0x7f9ee80a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 0x7f9ee80a5100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006010 tx=0x7f9eec000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 msgr2=0x7f9ef81983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400c920 tx=0x7f9ee400cde0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 shutdown_connections 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef81037c0 0x7f9ef819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee80a2c40 0x7f9ee80a5100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8102e00 0x7f9ef81983c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 --2- 172.21.15.16:0/2914604662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8101c00 0x7f9ef8197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 >> 172.21.15.16:0/2914604662 conn(0x7f9ef80fd390 msgr2=0x7f9ef8106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 shutdown_connections 2024-09-12T02:47:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:14.639+0000 7f9efebf0640 1 -- 172.21.15.16:0/2914604662 wait complete. 2024-09-12T02:47:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:14 smithi016 ceph-mon[30925]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:14 smithi033 ceph-mon[35092]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:14 smithi184 ceph-mon[35321]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:15.297 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-12T02:47:15.306 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T02:47:15.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-12T02:47:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:15.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:15 smithi016 ceph-mon[30925]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:15.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:15 smithi016 ceph-mon[30925]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 -- 172.21.15.16:0/1526223236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 msgr2=0x7f6a8c10a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1526223236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c10a050 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009ca0 tx=0x7f6a8402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 -- 172.21.15.16:0/1526223236 shutdown_connections 2024-09-12T02:47:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1526223236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c10a050 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1526223236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c102e00 0x7f6a8c103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1526223236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c101c00 0x7f6a8c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 -- 172.21.15.16:0/1526223236 >> 172.21.15.16:0/1526223236 conn(0x7f6a8c0fd390 msgr2=0x7f6a8c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 -- 172.21.15.16:0/1526223236 shutdown_connections 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.020+0000 7f6a9280e640 1 -- 172.21.15.16:0/1526223236 wait complete. 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 Processor -- start 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 -- start start 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c101c00 0x7f6a8c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c102e00 0x7f6a8c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c10d860 con 0x7f6a8c1037c0 2024-09-12T02:47:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c10d6e0 con 0x7f6a8c101c00 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a9280e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c10d9e0 con 0x7f6a8c102e00 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50276/0 (socket says 172.21.15.16:50276) 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 -- 172.21.15.16:0/1530104164 learned_addr learned my addr 172.21.15.16:0/1530104164 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a8bfff640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c101c00 0x7f6a8c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a83fff640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c102e00 0x7f6a8c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 -- 172.21.15.16:0/1530104164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c102e00 msgr2=0x7f6a8c1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c102e00 0x7f6a8c1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 -- 172.21.15.16:0/1530104164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c101c00 msgr2=0x7f6a8c197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c101c00 0x7f6a8c197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.021+0000 7f6a90dab640 1 -- 172.21.15.16:0/1530104164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c19fab0 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a90dab640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005e30 tx=0x7f6a84031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84003c30 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a84003dd0 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8c19fd40 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8402fde0 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.022+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8c1a0270 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.023+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a8c109000 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.024+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6a84004050 con 0x7f6a8c1037c0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.025+0000 7f6a89ffb640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 0x7f6a780a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.025+0000 7f6a8bfff640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 0x7f6a780a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.025+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6a840f0490 con 0x7f6a8c1037c0 2024-09-12T02:47:16.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.026+0000 7f6a8bfff640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 0x7f6a780a5180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c102c60 tx=0x7f6a7c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:16.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.028+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6a840bd350 con 0x7f6a8c1037c0 2024-09-12T02:47:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.241+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f6a8c02d7c0 con 0x7f6a8c1037c0 2024-09-12T02:47:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.242+0000 7f6a89ffb640 1 -- 172.21.15.16:0/1530104164 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3125 (secure 0 0 0) 0x7f6a840bcc50 con 0x7f6a8c1037c0 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:e27 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:btime 2024-09-12T02:45:58:926357+0000 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 27 2024-09-12T02:47:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-12T02:45:47.690653+0000 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-12T02:45:58.926336+0000 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-09-12T02:47:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 5 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2,3,4 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24500,1=24467,2=24445,3=24443,4=14595} 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-09-12T02:47:16.245 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24500 members: 24500,24467,24445,24443,14595 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24500} state up:active seq 4 addr [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{0:24430} state up:standby-replay seq 1 addr [v2:172.21.15.184:6836/2803727914,v1:172.21.15.184:6837/2803727914] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{1:24467} state up:active seq 9 addr [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{1:24424} state up:standby-replay seq 1 addr [v2:172.21.15.184:6834/1041505607,v1:172.21.15.184:6835/1041505607] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{2:24445} state up:active seq 8 addr [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{2:24422} state up:standby-replay seq 1 addr [v2:172.21.15.33:6832/1900353998,v1:172.21.15.33:6833/1900353998] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{3:24443} state up:active seq 11 addr [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{3:24400} state up:standby-replay seq 1 addr [v2:172.21.15.184:6832/897342399,v1:172.21.15.184:6833/897342399] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{4:14595} state up:active seq 7 addr [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{4:14583} state up:standby-replay seq 1 addr [v2:172.21.15.16:6838/2216233994,v1:172.21.15.16:6839/2216233994] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.16:6834/1858508080,v1:172.21.15.16:6835/1858508080] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.16:6836/3544362280,v1:172.21.15.16:6837/3544362280] compat {c=[1],r=[1],i=[1fff]}] 2024-09-12T02:47:16.247 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.244+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 msgr2=0x7f6a780a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 0x7f6a780a5180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c102c60 tx=0x7f6a7c00b390 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 msgr2=0x7f6a8c19f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005e30 tx=0x7f6a84031ee0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 shutdown_connections 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a780a2cc0 0x7f6a780a5180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c1037c0 0x7f6a8c19f440 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a8c102e00 0x7f6a8c1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 --2- 172.21.15.16:0/1530104164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a8c101c00 0x7f6a8c197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 >> 172.21.15.16:0/1530104164 conn(0x7f6a8c0fd390 msgr2=0x7f6a8c1061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 shutdown_connections 2024-09-12T02:47:16.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:16.245+0000 7f6a9280e640 1 -- 172.21.15.16:0/1530104164 wait complete. 2024-09-12T02:47:16.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:15 smithi033 ceph-mon[35092]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:16.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:15 smithi033 ceph-mon[35092]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:15 smithi184 ceph-mon[35321]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:15 smithi184 ceph-mon[35321]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:16.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:16 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1530104164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-12T02:47:16.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-12T02:47:17.210 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:16 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1530104164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-12T02:47:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:16 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1530104164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/1810931293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0101c10 msgr2=0x7f0fe0102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/1810931293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0101c10 0x7f0fe0102010 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009900 tx=0x7f0fc802a3f0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/1810931293 shutdown_connections 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/1810931293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe01037d0 0x7f0fe010a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/1810931293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 0x7f0fe0103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/1810931293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0101c10 0x7f0fe0102010 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/1810931293 >> 172.21.15.16:0/1810931293 conn(0x7f0fe00fd3c0 msgr2=0x7f0fe00ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/1810931293 shutdown_connections 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.676+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/1810931293 wait complete. 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 Processor -- start 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 -- start start 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe0101c10 0x7f0fe0197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 0x7f0fe0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010d870 con 0x7f0fe01037d0 2024-09-12T02:47:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010d6f0 con 0x7f0fe0102e10 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe6e5b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010d9f0 con 0x7f0fe0101c10 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50304/0 (socket says 172.21.15.16:50304) 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 -- 172.21.15.16:0/105920573 learned_addr learned my addr 172.21.15.16:0/105920573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fdffff640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 0x7f0fe0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe4bf7640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe0101c10 0x7f0fe0197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 -- 172.21.15.16:0/105920573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe0101c10 msgr2=0x7f0fe0197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe0101c10 0x7f0fe0197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 -- 172.21.15.16:0/105920573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 msgr2=0x7f0fe0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 0x7f0fe0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.677+0000 7f0fe53f8640 1 -- 172.21.15.16:0/105920573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe019fa30 con 0x7f0fe01037d0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fe53f8640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800cce0 tx=0x7f0fd800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd800f450 con 0x7f0fe01037d0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fd800fad0 con 0x7f0fe01037d0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe019fd20 con 0x7f0fe01037d0 2024-09-12T02:47:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8018b60 con 0x7f0fe01037d0 2024-09-12T02:47:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.678+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe01a01b0 con 0x7f0fe01037d0 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.679+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe0108fc0 con 0x7f0fe01037d0 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.680+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0fd800fc80 con 0x7f0fe01037d0 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.681+0000 7f0fddffb640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 0x7f0fcc0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.681+0000 7f0fe4bf7640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 0x7f0fcc0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.681+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0fd80cf690 con 0x7f0fe01037d0 2024-09-12T02:47:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.681+0000 7f0fe4bf7640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 0x7f0fcc0a5100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8005ab0 tx=0x7f0fc8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.685+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0fd8014030 con 0x7f0fe01037d0 2024-09-12T02:47:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.862+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0fe002d7c0 con 0x7f0fe01037d0 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.866+0000 7f0fddffb640 1 -- 172.21.15.16:0/105920573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v80) ==== 56+0+5263 (secure 0 0 0) 0x7f0fd809c440 con 0x7f0fe01037d0 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 80 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:fsid ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-12T02:38:12.620009+0000 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-12T02:46:01.241215+0000 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-12T02:47:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 26 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 19 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-12T02:47:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 8 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/245735353,v1:172.21.15.16:6803/245735353] [v2:172.21.15.16:6804/245735353,v1:172.21.15.16:6805/245735353] exists,up 1bfd4081-15d2-46ed-922c-c65451eff4f1 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 12 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/2781270412,v1:172.21.15.33:6801/2781270412] [v2:172.21.15.33:6802/2781270412,v1:172.21.15.33:6803/2781270412] exists,up 4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 16 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6800/867017914,v1:172.21.15.184:6801/867017914] [v2:172.21.15.184:6802/867017914,v1:172.21.15.184:6803/867017914] exists,up 83c715c5-107d-47dc-9c18-bee07a4bc40e 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 22 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/3579605426,v1:172.21.15.16:6811/3579605426] [v2:172.21.15.16:6812/3579605426,v1:172.21.15.16:6813/3579605426] exists,up 9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 29 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/3318678506,v1:172.21.15.33:6809/3318678506] [v2:172.21.15.33:6810/3318678506,v1:172.21.15.33:6811/3318678506] exists,up dd26143a-5c8f-47b6-8f10-7bc4480b6c6d 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 34 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6808/746947299,v1:172.21.15.184:6809/746947299] [v2:172.21.15.184:6810/746947299,v1:172.21.15.184:6811/746947299] exists,up 269f5e13-d530-4741-8b1f-7e6313cd1ccf 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 41 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/2942447777,v1:172.21.15.16:6819/2942447777] [v2:172.21.15.16:6820/2942447777,v1:172.21.15.16:6821/2942447777] exists,up 0744cd3c-8a0c-42f5-9e43-bfe91d615dc2 2024-09-12T02:47:17.868 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 47 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/1053213399,v1:172.21.15.33:6817/1053213399] [v2:172.21.15.33:6818/1053213399,v1:172.21.15.33:6819/1053213399] exists,up 165df37e-3ffb-4d76-91b3-a15d9eaaa233 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 54 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6816/1742401135,v1:172.21.15.184:6817/1742401135] [v2:172.21.15.184:6818/1742401135,v1:172.21.15.184:6819/1742401135] exists,up f1980104-b353-42e4-9168-5c1c326da47e 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 61 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/3409413039,v1:172.21.15.16:6827/3409413039] [v2:172.21.15.16:6828/3409413039,v1:172.21.15.16:6829/3409413039] exists,up 1da86df9-e623-41f7-8089-73aef567c50a 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 65 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/3476378344,v1:172.21.15.33:6825/3476378344] [v2:172.21.15.33:6826/3476378344,v1:172.21.15.33:6827/3476378344] exists,up 5d052784-3463-4b31-8894-92d1ef1d1ea9 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 72 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6824/3211758447,v1:172.21.15.184:6825/3211758447] [v2:172.21.15.184:6826/3211758447,v1:172.21.15.184:6827/3211758447] exists,up ed5b71a4-67ba-479b-83e0-80e5ce8cfec9 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.5 [8,11] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.f [8,11] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.16 [2,4] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.1d [2,1] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.3 [4,9] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.17 [4,8] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.33 [5,9] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.36 [4,9] 2024-09-12T02:47:17.869 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3425705974 expires 2024-09-13T02:38:59.915197+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/744312028 expires 2024-09-13T02:38:59.915197+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/744312028 expires 2024-09-13T02:38:59.915197+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/540700595 expires 2024-09-13T02:38:59.915197+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2390815323 expires 2024-09-13T02:38:59.915197+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2593008088 expires 2024-09-13T02:38:34.945193+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/2594473615 expires 2024-09-13T02:38:34.945193+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/2594473615 expires 2024-09-13T02:38:34.945193+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1418983 expires 2024-09-13T02:38:34.945193+0000 2024-09-12T02:47:17.870 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3307112498 expires 2024-09-13T02:38:34.945193+0000 2024-09-12T02:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 msgr2=0x7f0fcc0a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 0x7f0fcc0a5100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8005ab0 tx=0x7f0fc8000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 msgr2=0x7f0fe019f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800cce0 tx=0x7f0fd800a620 comp rx=0 tx=0).stop 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 shutdown_connections 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0fcc0a2c40 0x7f0fcc0a5100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe01037d0 0x7f0fe019f3c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0fe0102e10 0x7f0fe0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 --2- 172.21.15.16:0/105920573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fe0101c10 0x7f0fe0197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 >> 172.21.15.16:0/105920573 conn(0x7f0fe00fd3c0 msgr2=0x7f0fe0106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 shutdown_connections 2024-09-12T02:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:17.868+0000 7f0fe6e5b640 1 -- 172.21.15.16:0/105920573 wait complete. 2024-09-12T02:47:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:17 smithi016 ceph-mon[30925]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:17 smithi033 ceph-mon[35092]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:17 smithi184 ceph-mon[35321]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:18.513 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-12T02:47:18.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:19.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/105920573' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-12T02:47:19.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/105920573' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-12T02:47:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 -- 172.21.15.16:0/388800006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 msgr2=0x7f538010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 --2- 172.21.15.16:0/388800006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f538010d950 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5378009cd0 tx=0x7f537802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 -- 172.21.15.16:0/388800006 shutdown_connections 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 --2- 172.21.15.16:0/388800006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f538010d950 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 --2- 172.21.15.16:0/388800006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380100a80 0x7f5380100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 --2- 172.21.15.16:0/388800006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380106a80 0x7f5380106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 -- 172.21.15.16:0/388800006 >> 172.21.15.16:0/388800006 conn(0x7f53800fc7a0 msgr2=0x7f53800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 -- 172.21.15.16:0/388800006 shutdown_connections 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.217+0000 7f5385f27640 1 -- 172.21.15.16:0/388800006 wait complete. 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 Processor -- start 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 -- start start 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380100a80 0x7f53801992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380106a80 0x7f53801a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5380110e80 con 0x7f5380101420 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5380110d00 con 0x7f5380100a80 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f5385f27640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5380111000 con 0x7f5380106a80 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f537effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f537effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50336/0 (socket says 172.21.15.16:50336) 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f537effd640 1 -- 172.21.15.16:0/578594203 learned_addr learned my addr 172.21.15.16:0/578594203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.218+0000 7f537ffff640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380106a80 0x7f53801a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537f7fe640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380100a80 0x7f53801992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 -- 172.21.15.16:0/578594203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380106a80 msgr2=0x7f53801a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380106a80 0x7f53801a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 -- 172.21.15.16:0/578594203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380100a80 msgr2=0x7f53801992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380100a80 0x7f53801992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 -- 172.21.15.16:0/578594203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53801a4870 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537effd640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f537400d770 tx=0x7f537400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374017070 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5374005500 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53801a4a40 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374004e00 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.219+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53801a4f50 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.220+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5380100ee0 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.221+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5374004fa0 con 0x7f5380101420 2024-09-12T02:47:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.222+0000 7f537cff9640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 0x7f53680a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.222+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f53740cec40 con 0x7f5380101420 2024-09-12T02:47:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.222+0000 7f537f7fe640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 0x7f53680a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.222+0000 7f537f7fe640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 0x7f53680a5150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f53700099e0 tx=0x7f5370005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.226+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f53740d5080 con 0x7f5380101420 2024-09-12T02:47:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/105920573' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-12T02:47:19.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.440+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5380109010 con 0x7f5380101420 2024-09-12T02:47:19.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.441+0000 7f537cff9640 1 -- 172.21.15.16:0/578594203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7f537409ba50 con 0x7f5380101420 2024-09-12T02:47:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:19.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:47:19.444 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 msgr2=0x7f53680a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 0x7f53680a5150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f53700099e0 tx=0x7f5370005ed0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 msgr2=0x7f53801997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f537400d770 tx=0x7f537400dc30 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 shutdown_connections 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5380106a80 0x7f53801a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53680a2c90 0x7f53680a5150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5380101420 0x7f53801997f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 --2- 172.21.15.16:0/578594203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5380100a80 0x7f53801992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.443+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 >> 172.21.15.16:0/578594203 conn(0x7f53800fc7a0 msgr2=0x7f53800fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.444+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 shutdown_connections 2024-09-12T02:47:19.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:19.444+0000 7f5385f27640 1 -- 172.21.15.16:0/578594203 wait complete. 2024-09-12T02:47:20.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:19 smithi016 ceph-mon[30925]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:20.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:19 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/578594203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:20.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:19 smithi033 ceph-mon[35092]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:19 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/578594203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:19 smithi184 ceph-mon[35321]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:19 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/578594203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:20.400 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 -- 172.21.15.16:0/2629546688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa53410c170 msgr2=0x7fa53410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 --2- 172.21.15.16:0/2629546688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa53410c170 0x7fa53410e610 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009ca0 tx=0x7fa52c02a950 comp rx=0 tx=0).stop 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 -- 172.21.15.16:0/2629546688 shutdown_connections 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 --2- 172.21.15.16:0/2629546688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa53410c170 0x7fa53410e610 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 --2- 172.21.15.16:0/2629546688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534103660 0x7fa53410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 --2- 172.21.15.16:0/2629546688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa534102d40 0x7fa534103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 -- 172.21.15.16:0/2629546688 >> 172.21.15.16:0/2629546688 conn(0x7fa5340fc7a0 msgr2=0x7fa5340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 -- 172.21.15.16:0/2629546688 shutdown_connections 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.858+0000 7fa53994e640 1 -- 172.21.15.16:0/2629546688 wait complete. 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 Processor -- start 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 -- start start 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534102d40 0x7fa5341983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa53410c170 0x7fa53419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa534111f70 con 0x7fa534103660 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa534111df0 con 0x7fa534102d40 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.859+0000 7fa53994e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5341120f0 con 0x7fa53410c170 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa532ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534102d40 0x7fa5341983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50346/0 (socket says 172.21.15.16:50346) 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 -- 172.21.15.16:0/1936840995 learned_addr learned my addr 172.21.15.16:0/1936840995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5337fe640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa53410c170 0x7fa53419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 -- 172.21.15.16:0/1936840995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa53410c170 msgr2=0x7fa53419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa53410c170 0x7fa53419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 -- 172.21.15.16:0/1936840995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534102d40 msgr2=0x7fa5341983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534102d40 0x7fa5341983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 -- 172.21.15.16:0/1936840995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53419d3a0 con 0x7fa534103660 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa5327fc640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa52800c970 tx=0x7fa52800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528007bb0 con 0x7fa534103660 2024-09-12T02:47:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa528007d50 con 0x7fa534103660 2024-09-12T02:47:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53419d690 con 0x7fa534103660 2024-09-12T02:47:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528018820 con 0x7fa534103660 2024-09-12T02:47:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.860+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5341a4f70 con 0x7fa534103660 2024-09-12T02:47:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.861+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa534103120 con 0x7fa534103660 2024-09-12T02:47:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.862+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa5280189c0 con 0x7fa534103660 2024-09-12T02:47:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.863+0000 7fa53894c640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 0x7fa51c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.863+0000 7fa532ffd640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 0x7fa51c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.863+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa5280cf140 con 0x7fa534103660 2024-09-12T02:47:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.864+0000 7fa532ffd640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 0x7fa51c0a50d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa524005fd0 tx=0x7fa52400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:20.867+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa528014030 con 0x7fa534103660 2024-09-12T02:47:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.077+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa534106c80 con 0x7fa534103660 2024-09-12T02:47:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.078+0000 7fa53894c640 1 -- 172.21.15.16:0/1936840995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11969 (secure 0 0 0) 0x7fa52809bf50 con 0x7fa534103660 2024-09-12T02:47:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:21.082 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-12T02:45:58:926357+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:45:58.926336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:47:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-12T02:47:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 msgr2=0x7fa51c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 0x7fa51c0a50d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa524005fd0 tx=0x7fa52400b390 comp rx=0 tx=0).stop 2024-09-12T02:47:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 msgr2=0x7fa534198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa52800c970 tx=0x7fa52800ce30 comp rx=0 tx=0).stop 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 shutdown_connections 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa53410c170 0x7fa53419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa51c0a2c10 0x7fa51c0a50d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa534103660 0x7fa534198930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 --2- 172.21.15.16:0/1936840995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa534102d40 0x7fa5341983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 >> 172.21.15.16:0/1936840995 conn(0x7fa5340fc7a0 msgr2=0x7fa5340fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 shutdown_connections 2024-09-12T02:47:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:21.081+0000 7fa53994e640 1 -- 172.21.15.16:0/1936840995 wait complete. 2024-09-12T02:47:21.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:47:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:22.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:21 smithi016 ceph-mon[30925]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:22.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:21 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1936840995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:21 smithi033 ceph-mon[35092]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:21 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1936840995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:21 smithi184 ceph-mon[35321]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:47:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:21 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1936840995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 -- 172.21.15.16:0/760329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29041008d0 msgr2=0x7f2904100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 --2- 172.21.15.16:0/760329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29041008d0 0x7f2904100cb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009900 tx=0x7f28f802a410 comp rx=0 tx=0).stop 2024-09-12T02:47:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 -- 172.21.15.16:0/760329998 shutdown_connections 2024-09-12T02:47:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 --2- 172.21.15.16:0/760329998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 0x7f2904110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 --2- 172.21.15.16:0/760329998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2904101280 0x7f290410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 --2- 172.21.15.16:0/760329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29041008d0 0x7f2904100cb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 -- 172.21.15.16:0/760329998 >> 172.21.15.16:0/760329998 conn(0x7f29040fc7a0 msgr2=0x7f29040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 -- 172.21.15.16:0/760329998 shutdown_connections 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.526+0000 7f290af6a640 1 -- 172.21.15.16:0/760329998 wait complete. 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 Processor -- start 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 -- start start 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29041008d0 0x7f290419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 0x7f290419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2904114200 con 0x7f2904101280 2024-09-12T02:47:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2904114080 con 0x7f290410e480 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f290af6a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2904114380 con 0x7f29041008d0 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.527+0000 7f2903fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50366/0 (socket says 172.21.15.16:50366) 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 -- 172.21.15.16:0/2867561464 learned_addr learned my addr 172.21.15.16:0/2867561464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2908d06640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29041008d0 0x7f290419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 -- 172.21.15.16:0/2867561464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29041008d0 msgr2=0x7f290419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29041008d0 0x7f290419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2909507640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 0x7f290419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 -- 172.21.15.16:0/2867561464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 msgr2=0x7f290419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 0x7f290419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 -- 172.21.15.16:0/2867561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290419f6d0 con 0x7f2904101280 2024-09-12T02:47:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2903fff640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b4a0 tx=0x7f28f000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0014070 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f0010af0 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290419f9c0 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0017d10 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.528+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29041a72a0 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.529+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2904100cb0 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.530+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f28f0017070 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.531+0000 7f2901ffb640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 0x7f28f40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.531+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f28f00ce450 con 0x7f2904101280 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.531+0000 7f2908d06640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 0x7f28f40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.532+0000 7f2908d06640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 0x7f28f40a5150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28f80096c0 tx=0x7f28f8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.535+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f28f009b370 con 0x7f2904101280 2024-09-12T02:47:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.711+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f290402d7c0 con 0x7f2904101280 2024-09-12T02:47:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.713+0000 7f2901ffb640 1 -- 172.21.15.16:0/2867561464 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f28f009ac70 con 0x7f2904101280 2024-09-12T02:47:22.713 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:22.713 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:47:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 msgr2=0x7f28f40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 0x7f28f40a5150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28f80096c0 tx=0x7f28f8000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 msgr2=0x7f290419ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b4a0 tx=0x7f28f000b960 comp rx=0 tx=0).stop 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 shutdown_connections 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f290410e480 0x7f290419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28f40a2c90 0x7f28f40a5150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2904101280 0x7f290419ac60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 --2- 172.21.15.16:0/2867561464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29041008d0 0x7f290419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 >> 172.21.15.16:0/2867561464 conn(0x7f29040fc7a0 msgr2=0x7f29040feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 shutdown_connections 2024-09-12T02:47:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:22.715+0000 7f290af6a640 1 -- 172.21.15.16:0/2867561464 wait complete. 2024-09-12T02:47:23.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:22 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2867561464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:22 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2867561464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:23.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:22 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2867561464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:23.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-12T02:45:47.690653+0000', 'modified': '2024-09-12T02:45:58.926336+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24467, 'mds_2': 24445, 'mds_3': 24443, 'mds_4': 14595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14583': {'gid': 14583, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/2216233994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 2216233994}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 2216233994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14595': {'gid': 14595, 'name': 'a', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.16:6841/1593859053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 1593859053}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 1593859053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6833/897342399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 897342399}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 897342399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6833/1900353998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6832', 'nonce': 1900353998}, {'type': 'v1', 'addr': '172.21.15.33:6833', 'nonce': 1900353998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6835/1041505607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 1041505607}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 1041505607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6837/2803727914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 2803727914}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 2803727914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.33:6835/2990713107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 2990713107}, {'type': 'v1', 'addr': '172.21.15.33:6835', 'nonce': 2990713107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.184:6839/598714982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6838', 'nonce': 598714982}, {'type': 'v1', 'addr': '172.21.15.184:6839', 'nonce': 598714982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.33:6837/4242490884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6836', 'nonce': 4242490884}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 4242490884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.33:6839/3603617308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 3603617308}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 3603617308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24500, 24467, 24445, 24443, 14595]} 2024-09-12T02:47:23.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-12T02:47:23.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.a status 2024-09-12T02:47:23.683 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:24.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:23 smithi016 ceph-mon[30925]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:24.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 -- 172.21.15.16:0/2960152992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684101420 msgr2=0x7f868410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 --2- 172.21.15.16:0/2960152992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684101420 0x7f868410d950 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f867c009cd0 tx=0x7f867c02a8e0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 -- 172.21.15.16:0/2960152992 shutdown_connections 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 --2- 172.21.15.16:0/2960152992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684101420 0x7f868410d950 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 --2- 172.21.15.16:0/2960152992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684100a80 0x7f8684100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 --2- 172.21.15.16:0/2960152992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684106a80 0x7f8684106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 -- 172.21.15.16:0/2960152992 >> 172.21.15.16:0/2960152992 conn(0x7f86840fc7a0 msgr2=0x7f86840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 -- 172.21.15.16:0/2960152992 shutdown_connections 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.141+0000 7f8689737640 1 -- 172.21.15.16:0/2960152992 wait complete. 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 Processor -- start 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 -- start start 2024-09-12T02:47:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684101420 0x7f86840fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 0x7f86840fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8684110e80 con 0x7f8684100a80 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8684110d00 con 0x7f8684101420 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.142+0000 7f8689737640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8684111000 con 0x7f8684106a80 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50384/0 (socket says 172.21.15.16:50384) 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 -- 172.21.15.16:0/4198530405 learned_addr learned my addr 172.21.15.16:0/4198530405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f867bfff640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684101420 0x7f86840fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f86837fe640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 0x7f86840fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 -- 172.21.15.16:0/4198530405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 msgr2=0x7f86840fd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 0x7f86840fd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 -- 172.21.15.16:0/4198530405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684101420 msgr2=0x7f86840fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684101420 0x7f86840fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 -- 172.21.15.16:0/4198530405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86841a68c0 con 0x7f8684100a80 2024-09-12T02:47:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f86837fe640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 0x7f86840fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8682ffd640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f867400d950 tx=0x7f867400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8680ff9640 1 -- 172.21.15.16:0/4198530405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674017030 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8680ff9640 1 -- 172.21.15.16:0/4198530405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8674004020 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86841a6bb0 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8680ff9640 1 -- 172.21.15.16:0/4198530405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86740047a0 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.143+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86841a7140 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.145+0000 7f8680ff9640 1 -- 172.21.15.16:0/4198530405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f86740041d0 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f868bb10740 1 Processor -- start 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f868bb10740 1 -- start start 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f8680ff9640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 0x7f86700a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f8680ff9640 1 -- 172.21.15.16:0/4198530405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f86740cea90 con 0x7f8684100a80 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f86837fe640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 0x7f86700a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.146+0000 7f868bb10740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 0x5559cd5fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f868bb10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f868bb10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 0x5559cd609fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f868bb10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5559cd563090 con 0x5559cd5fd2f0 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f868bb10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559cd50aa70 con 0x5559cd607bb0 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f868bb10740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5559cd55f050 con 0x5559cd5fc9d0 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 0x5559cd5fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50394/0 (socket says 172.21.15.16:50394) 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 0x5559cd5fcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51070/0 (socket says 172.21.15.16:51070) 2024-09-12T02:47:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 -- 172.21.15.16:0/663183857 learned_addr learned my addr 172.21.15.16:0/663183857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86837fe640 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 0x5559cd609fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 -- 172.21.15.16:0/663183857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 msgr2=0x5559cd5fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 0x5559cd5fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 -- 172.21.15.16:0/663183857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 msgr2=0x5559cd609fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 0x5559cd609fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 -- 172.21.15.16:0/663183857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559cd517130 con 0x5559cd5fd2f0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86837fe640 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 0x5559cd609fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86837fe640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 0x7f86700a5150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f867c009430 tx=0x7f867c005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f867bfff640 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b740 tx=0x7f866c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af8d79d22444959 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86797fa640 1 -- 172.21.15.16:0/663183857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c003dd0 con 0x5559cd5fd2f0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86797fa640 1 -- 172.21.15.16:0/663183857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f866c015070 con 0x5559cd5fd2f0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.147+0000 7f86797fa640 1 -- 172.21.15.16:0/663183857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c004e50 con 0x5559cd5fd2f0 2024-09-12T02:47:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 -- 172.21.15.16:0/663183857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 msgr2=0x5559cd607670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b740 tx=0x7f866c00bc00 comp rx=0 tx=0).stop 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 -- 172.21.15.16:0/663183857 shutdown_connections 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd607bb0 0x5559cd609fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd607670 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 --2- 172.21.15.16:0/663183857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fc9d0 0x5559cd5fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 -- 172.21.15.16:0/663183857 >> 172.21.15.16:0/663183857 conn(0x5559cd5f8f60 msgr2=0x5559cd5f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 -- 172.21.15.16:0/663183857 shutdown_connections 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.148+0000 7f868bb10740 1 -- 172.21.15.16:0/663183857 wait complete. 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 -- start start 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 Processor -- start 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd607420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd604cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5559cd60a780 con 0x5559cd5fd2f0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559cd51df20 con 0x5559cd5fc9d0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5559cd60a900 con 0x5559cd607bb0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f868bb10740 10 client.0 _ll_register_callbacks cb 0x5559cd4e7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50402/0 (socket says 172.21.15.16:50402) 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.149+0000 7f867bfff640 1 -- 172.21.15.16:0/258043044 learned_addr learned my addr 172.21.15.16:0/258043044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f8682ffd640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd607420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f86837fe640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd604cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 -- 172.21.15.16:0/258043044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 msgr2=0x5559cd604cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd604cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 -- 172.21.15.16:0/258043044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 msgr2=0x5559cd607420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd607420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 -- 172.21.15.16:0/258043044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559cd2d42b0 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f86837fe640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd604cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f867bfff640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b710 tx=0x7f866c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a0e05a13514d0a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c028060 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f866c011640 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f866c015070 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f866c003ea0 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f868bb10740 10 client.15102 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559cd616010 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f866c015070 con 0x5559cd5fd2f0 2024-09-12T02:47:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.150+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f866c022070 con 0x5559cd5fd2f0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 10 client.15102 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 10 client.15102 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 20 client.15102 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd6182a0 0x5559cd619710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 4 client.15102 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559cd570600 con 0x5559cd6182a0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f8682ffd640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd6182a0 0x5559cd619710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f8682ffd640 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd6182a0 0x5559cd619710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=57cc1e63d48f5590 server_cookie=38cf6d6690f1d0b3 in_seq=0 out_seq=0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.151+0000 7f866a7fc640 10 client.15102 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f866a7fc640 1 -- 172.21.15.16:0/258043044 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5559cd570600 con 0x5559cd6182a0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f866a7fc640 10 client.15102 handle_command_reply: tid=0 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 client.15102 shutdown 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 msgr2=0x5559cd604770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b710 tx=0x7f866c004410 comp rx=0 tx=0).stop 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 shutdown_connections 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd604cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd604770 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd6182a0 0x5559cd619710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 --2- 172.21.15.16:0/258043044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd607420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f866a7fc640 0 client.15102 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 >> 172.21.15.16:0/258043044 conn(0x5559cd5f8f60 msgr2=0x5559cd5fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 shutdown_connections 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 1 -- 172.21.15.16:0/258043044 wait complete. 2024-09-12T02:47:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.152+0000 7f868bb10740 20 client.15102 trim_cache size 0 max 16384 2024-09-12T02:47:24.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.170+0000 7f868bb10740 1 Processor -- start 2024-09-12T02:47:24.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 -- start start 2024-09-12T02:47:24.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd619710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd601f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5559cd617a50 con 0x5559cd5fd2f0 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559cd618120 con 0x5559cd5fc9d0 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f868bb10740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5559cd6187f0 con 0x5559cd607bb0 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50404/0 (socket says 172.21.15.16:50404) 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 -- 172.21.15.16:0/1864392065 learned_addr learned my addr 172.21.15.16:0/1864392065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f8682ffd640 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd619710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 -- 172.21.15.16:0/1864392065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 msgr2=0x5559cd601f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd601f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 -- 172.21.15.16:0/1864392065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 msgr2=0x5559cd619710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd619710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.171+0000 7f867bfff640 1 -- 172.21.15.16:0/1864392065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559cd60a890 con 0x5559cd5fd2f0 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f867bfff640 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f866c005350 tx=0x7f866c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b31c68a550f9b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f8678ff9640 1 -- 172.21.15.16:0/1864392065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c033030 con 0x5559cd5fd2f0 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f8678ff9640 1 -- 172.21.15.16:0/1864392065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f866c015070 con 0x5559cd5fd2f0 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f8678ff9640 1 -- 172.21.15.16:0/1864392065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c019700 con 0x5559cd5fd2f0 2024-09-12T02:47:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f868bb10740 1 -- 172.21.15.16:0/1864392065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 msgr2=0x5559cd6019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.172+0000 7f868bb10740 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f866c005350 tx=0x7f866c005c50 comp rx=0 tx=0).stop 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- 172.21.15.16:0/1864392065 shutdown_connections 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd607bb0 0x5559cd601f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd5fd2f0 0x5559cd6019d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- 172.21.15.16:0/1864392065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd619710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- 172.21.15.16:0/1864392065 >> 172.21.15.16:0/1864392065 conn(0x5559cd5f8f60 msgr2=0x5559cd5f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- 172.21.15.16:0/1864392065 shutdown_connections 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- 172.21.15.16:0/1864392065 wait complete. 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- start start 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 Processor -- start 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd61ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fd2f0 0x5559cd61d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5559cd619590 con 0x5559cd607bb0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5559cd618ec0 con 0x5559cd5fc9d0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5559cd619c60 con 0x5559cd5fd2f0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.173+0000 7f868bb10740 10 client.0 _ll_register_callbacks cb 0x7f86740051b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50406/0 (socket says 172.21.15.16:50406) 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 -- 172.21.15.16:0/1370327567 learned_addr learned my addr 172.21.15.16:0/1370327567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f8682ffd640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd61ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f867bfff640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fd2f0 0x5559cd61d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 -- 172.21.15.16:0/1370327567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fd2f0 msgr2=0x5559cd61d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fd2f0 0x5559cd61d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 -- 172.21.15.16:0/1370327567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 msgr2=0x5559cd61ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd61ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 -- 172.21.15.16:0/1370327567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559cd680f60 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f86837fe640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f867c0099a0 tx=0x7f867c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b171645e51408bd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f866a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c04c070 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f867c004740 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f866c015070 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f867c058050 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f868bb10740 10 client.15108 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.174+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559cd682200 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f866c015070 con 0x5559cd607bb0 2024-09-12T02:47:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f867c030070 con 0x5559cd607bb0 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 10 client.15108 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 10 client.15108 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 20 client.15108 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd686e20 0x5559cd685b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 4 client.15108 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "status"}) -- 0x5559cd56ab10 con 0x5559cd686e20 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f8682ffd640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd686e20 0x5559cd685b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f8682ffd640 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd686e20 0x5559cd685b30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4e470168f6407a3e server_cookie=f53ede078fcdd25d in_seq=0 out_seq=0 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.175+0000 7f866a7fc640 10 client.15108 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f866a7fc640 1 -- 172.21.15.16:0/1370327567 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5559cd56ab10 con 0x5559cd686e20 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f866a7fc640 10 client.15108 handle_command_reply: tid=0 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 client.15108 shutdown 2024-09-12T02:47:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 msgr2=0x5559cd6809a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f867c0099a0 tx=0x7f867c0095c0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 shutdown_connections 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559cd607bb0 0x5559cd6809a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5559cd5fd2f0 0x5559cd61d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5559cd686e20 0x5559cd685b30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 --2- 172.21.15.16:0/1370327567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5559cd5fc9d0 0x5559cd61ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 >> 172.21.15.16:0/1370327567 conn(0x5559cd5f8f60 msgr2=0x5559cd5f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f866a7fc640 0 client.15108 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 shutdown_connections 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 1 -- 172.21.15.16:0/1370327567 wait complete. 2024-09-12T02:47:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f868bb10740 20 client.15108 trim_cache size 0 max 16384 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14595, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 86.253638702999993, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 109.294933517 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 msgr2=0x7f86700a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.176+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 0x7f86700a5150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f867c009430 tx=0x7f867c005f60 comp rx=0 tx=0).stop 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 msgr2=0x7f86840fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f867400d950 tx=0x7f867400de10 comp rx=0 tx=0).stop 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 shutdown_connections 2024-09-12T02:47:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8684106a80 0x7f86840fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8684101420 0x7f86840fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f86700a2c90 0x7f86700a5150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 --2- 172.21.15.16:0/4198530405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8684100a80 0x7f86840fcd90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 >> 172.21.15.16:0/4198530405 conn(0x7f86840fc7a0 msgr2=0x7f868406a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 shutdown_connections 2024-09-12T02:47:24.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:24.177+0000 7f8689737640 1 -- 172.21.15.16:0/4198530405 wait complete. 2024-09-12T02:47:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:23 smithi033 ceph-mon[35092]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:23 smithi184 ceph-mon[35321]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:47:24.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.e status 2024-09-12T02:47:25.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:25.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 -- 172.21.15.16:0/337859700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0106a70 msgr2=0x7f9be0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 --2- 172.21.15.16:0/337859700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0106a70 0x7f9be0106e50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009930 tx=0x7f9bd402a420 comp rx=0 tx=0).stop 2024-09-12T02:47:25.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 -- 172.21.15.16:0/337859700 shutdown_connections 2024-09-12T02:47:25.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 --2- 172.21.15.16:0/337859700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 0x7f9be010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 --2- 172.21.15.16:0/337859700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0100a70 0x7f9be0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 --2- 172.21.15.16:0/337859700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0106a70 0x7f9be0106e50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 -- 172.21.15.16:0/337859700 >> 172.21.15.16:0/337859700 conn(0x7f9be00fc790 msgr2=0x7f9be00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 -- 172.21.15.16:0/337859700 shutdown_connections 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.610+0000 7f9be7012640 1 -- 172.21.15.16:0/337859700 wait complete. 2024-09-12T02:47:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 Processor -- start 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 -- start start 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 0x7f9be01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 0x7f9be01a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0110f10 con 0x7f9be0100a70 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0110d90 con 0x7f9be0101410 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be7012640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0111090 con 0x7f9be0106a70 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58676/0 (socket says 172.21.15.16:58676) 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.611+0000 7f9be4dae640 1 -- 172.21.15.16:0/303090338 learned_addr learned my addr 172.21.15.16:0/303090338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9bdffff640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 0x7f9be01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be55af640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 0x7f9be01a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 -- 172.21.15.16:0/303090338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 msgr2=0x7f9be01a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 0x7f9be01a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 -- 172.21.15.16:0/303090338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 msgr2=0x7f9be01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 0x7f9be01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 -- 172.21.15.16:0/303090338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be01a4a40 con 0x7f9be0100a70 2024-09-12T02:47:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be55af640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 0x7f9be01a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be4dae640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009a50 tx=0x7f9bd4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9bddffb640 1 -- 172.21.15.16:0/303090338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd403f030 con 0x7f9be0100a70 2024-09-12T02:47:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be01a4c10 con 0x7f9be0100a70 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9bddffb640 1 -- 172.21.15.16:0/303090338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd4003ce0 con 0x7f9be0100a70 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9bddffb640 1 -- 172.21.15.16:0/303090338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4035070 con 0x7f9be0100a70 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.612+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be01a50c0 con 0x7f9be0100a70 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9bddffb640 1 -- 172.21.15.16:0/303090338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9bd4042030 con 0x7f9be0100a70 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 Processor -- start 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 -- start start 2024-09-12T02:47:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf96670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55829bef2090 con 0x55829bf8b9d0 2024-09-12T02:47:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55829be99a70 con 0x55829bf96bb0 2024-09-12T02:47:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be93eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55829beee050 con 0x55829bf8c2f0 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be55af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be55af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58690/0 (socket says 172.21.15.16:58690) 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.615+0000 7f9be55af640 1 -- 172.21.15.16:0/1293967790 learned_addr learned my addr 172.21.15.16:0/1293967790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bddffb640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 0x7f9bd00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be4dae640 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf96670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bdffff640 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 -- 172.21.15.16:0/1293967790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 msgr2=0x55829bf96670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf96670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 -- 172.21.15.16:0/1293967790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 msgr2=0x55829bf98fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf98fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 -- 172.21.15.16:0/1293967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829bea6130 con 0x55829bf8b9d0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 0x7f9bd00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bddffb640 1 -- 172.21.15.16:0/303090338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9bd40ef890 con 0x7f9be0100a70 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9be55af640 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c370 tx=0x7f9bd800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbf10cbaad65ece server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bcaffd640 1 -- 172.21.15.16:0/1293967790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8016070 con 0x55829bf8b9d0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bcaffd640 1 -- 172.21.15.16:0/1293967790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd8011d00 con 0x55829bf8b9d0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.616+0000 7f9bcaffd640 1 -- 172.21.15.16:0/1293967790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd801bc70 con 0x55829bf8b9d0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- 172.21.15.16:0/1293967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 msgr2=0x55829bf8bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be55af640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 0x7f9bd00a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9bd801d000 tx=0x7f9bd801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c370 tx=0x7f9bd800c830 comp rx=0 tx=0).stop 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- 172.21.15.16:0/1293967790 shutdown_connections 2024-09-12T02:47:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf98fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf96670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1293967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf8bdb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- 172.21.15.16:0/1293967790 >> 172.21.15.16:0/1293967790 conn(0x55829bf87f60 msgr2=0x55829bf88340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- 172.21.15.16:0/1293967790 shutdown_connections 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- 172.21.15.16:0/1293967790 wait complete. 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 -- start start 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.617+0000 7f9be93eb740 1 Processor -- start 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf93cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55829bf99780 con 0x55829bf8b9d0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55829beacf20 con 0x55829bf96bb0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55829bf99900 con 0x55829bf8c2f0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be93eb740 10 client.0 _ll_register_callbacks cb 0x55829be76030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58696/0 (socket says 172.21.15.16:58696) 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 -- 172.21.15.16:0/776818817 learned_addr learned my addr 172.21.15.16:0/776818817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9bdffff640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be55af640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf93cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 -- 172.21.15.16:0/776818817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 msgr2=0x55829bf937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 -- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 msgr2=0x55829bf93cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf93cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.618+0000 7f9be4dae640 1 -- 172.21.15.16:0/776818817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829bc632b0 con 0x55829bf8b9d0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be4dae640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4031550 tx=0x7f9bd402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2360e1034d263b74 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4049040 con 0x55829bf8b9d0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9bd40ed440 con 0x55829bf8b9d0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd8011d00 con 0x55829bf8b9d0 2024-09-12T02:47:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bd40bc640 con 0x55829bf8b9d0 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 10 client.15120 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55829bfa50d0 con 0x55829bf8b9d0 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd8011d00 con 0x55829bf8b9d0 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9bd403b460 con 0x55829bf8b9d0 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 10 client.15120 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 10 client.15120 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 20 client.15120 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829bfa7780 0x55829bfa8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 4 client.15120 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.619+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55829beff600 con 0x55829bfa7780 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.620+0000 7f9bdffff640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829bfa7780 0x55829bfa8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.620+0000 7f9bdffff640 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829bfa7780 0x55829bfa8bf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=198fd3e2fd00c47d server_cookie=c2d288bb0cd68f62 in_seq=0 out_seq=0 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.620+0000 7f9bbffff640 10 client.15120 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9bbffff640 1 -- 172.21.15.16:0/776818817 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55829beff600 con 0x55829bfa7780 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9bbffff640 10 client.15120 handle_command_reply: tid=0 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 client.15120 shutdown 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 msgr2=0x55829bf96460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4031550 tx=0x7f9bd402a420 comp rx=0 tx=0).stop 2024-09-12T02:47:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 shutdown_connections 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829bfa7780 0x55829bfa8bf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bf93cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9bbffff640 0 client.15120 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 --2- 172.21.15.16:0/776818817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf96460 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 >> 172.21.15.16:0/776818817 conn(0x55829bf87f60 msgr2=0x55829bf8dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 shutdown_connections 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 1 -- 172.21.15.16:0/776818817 wait complete. 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.621+0000 7f9be93eb740 20 client.15120 trim_cache size 0 max 16384 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 Processor -- start 2024-09-12T02:47:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 -- start start 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8b9d0 0x55829bfa8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf8c2f0 0x55829bf82ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa6f30 con 0x55829bf96bb0 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa7600 con 0x55829bf8c2f0 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be93eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa7cd0 con 0x55829bf8b9d0 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be55af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be55af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58708/0 (socket says 172.21.15.16:58708) 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be55af640 1 -- 172.21.15.16:0/4284742947 learned_addr learned my addr 172.21.15.16:0/4284742947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9bdffff640 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf8c2f0 0x55829bf82ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.639+0000 7f9be4dae640 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8b9d0 0x55829bfa8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 -- 172.21.15.16:0/4284742947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8b9d0 msgr2=0x55829bfa8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8b9d0 0x55829bfa8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 -- 172.21.15.16:0/4284742947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf8c2f0 msgr2=0x55829bf82ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf8c2f0 0x55829bf82ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 -- 172.21.15.16:0/4284742947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829bf96080 con 0x55829bf96bb0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be55af640 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c370 tx=0x7f9bd800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd856e0b0fe58803 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9bca7fc640 1 -- 172.21.15.16:0/4284742947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8016050 con 0x55829bf96bb0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9bca7fc640 1 -- 172.21.15.16:0/4284742947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd8010600 con 0x55829bf96bb0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9bca7fc640 1 -- 172.21.15.16:0/4284742947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8011960 con 0x55829bf96bb0 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be93eb740 1 -- 172.21.15.16:0/4284742947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 msgr2=0x55829bf920a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.640+0000 7f9be93eb740 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c370 tx=0x7f9bd800f000 comp rx=0 tx=0).stop 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- 172.21.15.16:0/4284742947 shutdown_connections 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf96bb0 0x55829bf920a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf8c2f0 0x55829bf82ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- 172.21.15.16:0/4284742947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8b9d0 0x55829bfa8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- 172.21.15.16:0/4284742947 >> 172.21.15.16:0/4284742947 conn(0x55829bf87f60 msgr2=0x55829bfa6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- 172.21.15.16:0/4284742947 shutdown_connections 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- 172.21.15.16:0/4284742947 wait complete. 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- start start 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 Processor -- start 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf90710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bfafeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa8a70 con 0x55829bf8b9d0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa83a0 con 0x55829bf96bb0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55829bfa9140 con 0x55829bf8c2f0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be93eb740 10 client.0 _ll_register_callbacks cb 0x7f9bd4004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.641+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58712/0 (socket says 172.21.15.16:58712) 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 -- 172.21.15.16:0/1939087636 learned_addr learned my addr 172.21.15.16:0/1939087636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be55af640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bfafeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bdffff640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf90710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 -- 172.21.15.16:0/1939087636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 msgr2=0x55829bf90710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf90710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 -- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 msgr2=0x55829bfafeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bfafeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 -- 172.21.15.16:0/1939087636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829bfb03f0 con 0x55829bf8b9d0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bdffff640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf90710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be4dae640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40c4680 tx=0x7f9bd4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1244954a6072900 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bbffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4085030 con 0x55829bf8b9d0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9bd4086070 con 0x55829bf8b9d0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55829bef9b10 con 0x55829bf8b9d0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bd4049070 con 0x55829bf8b9d0 2024-09-12T02:47:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be93eb740 10 client.15126 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55829bfb16d0 con 0x55829bf8b9d0 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.642+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55829bef9b10 con 0x55829bf8b9d0 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9bd40b85b0 con 0x55829bf8b9d0 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 10 client.15126 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 10 client.15126 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 20 client.15126 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829c0161c0 0x55829c014ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24443, 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 88.407392771999994, 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 126.708880507 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 4 client.15126 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "status"}) -- 0x55829be760e0 con 0x55829c0161c0 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9bdffff640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829c0161c0 0x55829c014ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9bdffff640 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829c0161c0 0x55829c014ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=283882e68b5276af server_cookie=2ccc616488eb45cd in_seq=0 out_seq=0 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.643+0000 7f9bbffff640 10 client.15126 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9bbffff640 1 -- 172.21.15.16:0/1939087636 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55829be760e0 con 0x55829c0161c0 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9bbffff640 10 client.15126 handle_command_reply: tid=0 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 client.15126 shutdown 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 msgr2=0x55829bf901d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40c4680 tx=0x7f9bd4005bd0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 shutdown_connections 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55829c0161c0 0x55829c014ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55829bf96bb0 0x55829bfafeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55829bf8c2f0 0x55829bf90710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9bbffff640 0 client.15126 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 --2- 172.21.15.16:0/1939087636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55829bf8b9d0 0x55829bf901d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 >> 172.21.15.16:0/1939087636 conn(0x55829bf87f60 msgr2=0x55829bf8d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 shutdown_connections 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 1 -- 172.21.15.16:0/1939087636 wait complete. 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.644+0000 7f9be93eb740 20 client.15126 trim_cache size 0 max 16384 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 msgr2=0x7f9bd00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 0x7f9bd00a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9bd801d000 tx=0x7f9bd801e000 comp rx=0 tx=0).stop 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 msgr2=0x7f9be0199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009a50 tx=0x7f9bd4004270 comp rx=0 tx=0).stop 2024-09-12T02:47:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 shutdown_connections 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be0106a70 0x7f9be01a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9be0101410 0x7f9be01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bd00a2c10 0x7f9bd00a50d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 --2- 172.21.15.16:0/303090338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be0100a70 0x7f9be0199480 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 >> 172.21.15.16:0/303090338 conn(0x7f9be00fc790 msgr2=0x7f9be00fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 shutdown_connections 2024-09-12T02:47:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:25.645+0000 7f9be7012640 1 -- 172.21.15.16:0/303090338 wait complete. 2024-09-12T02:47:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:25 smithi016 ceph-mon[30925]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:25 smithi033 ceph-mon[35092]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:25 smithi184 ceph-mon[35321]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:47:26.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.i status 2024-09-12T02:47:26.728 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 -- 172.21.15.16:0/349860327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818101430 msgr2=0x7fa81810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 --2- 172.21.15.16:0/349860327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818101430 0x7fa81810d960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa810009cd0 tx=0x7fa81002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 -- 172.21.15.16:0/349860327 shutdown_connections 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 --2- 172.21.15.16:0/349860327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818101430 0x7fa81810d960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 --2- 172.21.15.16:0/349860327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 --2- 172.21.15.16:0/349860327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818106a90 0x7fa818106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 -- 172.21.15.16:0/349860327 >> 172.21.15.16:0/349860327 conn(0x7fa8180fc7d0 msgr2=0x7fa8180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 -- 172.21.15.16:0/349860327 shutdown_connections 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.196+0000 7fa81e44a640 1 -- 172.21.15.16:0/349860327 wait complete. 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 Processor -- start 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 -- start start 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818101430 0x7fa818198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa818110e90 con 0x7fa818106a90 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa818110d10 con 0x7fa818100a90 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.197+0000 7fa81e44a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa818111010 con 0x7fa818101430 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58732/0 (socket says 172.21.15.16:58732) 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 -- 172.21.15.16:0/2803855126 learned_addr learned my addr 172.21.15.16:0/2803855126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa8177fe640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818101430 0x7fa818198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa817fff640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 -- 172.21.15.16:0/2803855126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818101430 msgr2=0x7fa818198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818101430 0x7fa818198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 -- 172.21.15.16:0/2803855126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 msgr2=0x7fa818198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 -- 172.21.15.16:0/2803855126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81819d3b0 con 0x7fa818106a90 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa817fff640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81c9e7640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa810005cc0 tx=0x7fa810003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa8157fa640 1 -- 172.21.15.16:0/2803855126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810004040 con 0x7fa818106a90 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81819d640 con 0x7fa818106a90 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.198+0000 7fa8157fa640 1 -- 172.21.15.16:0/2803855126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8100046c0 con 0x7fa818106a90 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.199+0000 7fa8157fa640 1 -- 172.21.15.16:0/2803855126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810046480 con 0x7fa818106a90 2024-09-12T02:47:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.199+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8181a4f80 con 0x7fa818106a90 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.201+0000 7fa8157fa640 1 -- 172.21.15.16:0/2803855126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa8100305a0 con 0x7fa818106a90 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.201+0000 7fa820823740 1 Processor -- start 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.201+0000 7fa820823740 1 -- start start 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140b9d0 0x55976140bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140c2f0 0x559761416670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559761372090 con 0x559761416bb0 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559761319a70 con 0x55976140c2f0 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa820823740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55976136e050 con 0x55976140b9d0 2024-09-12T02:47:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58744/0 (socket says 172.21.15.16:58744) 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 -- 172.21.15.16:0/2864127154 learned_addr learned my addr 172.21.15.16:0/2864127154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8157fa640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa817fff640 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140c2f0 0x559761416670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140b9d0 0x55976140bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa81c9e7640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 -- 172.21.15.16:0/2864127154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140b9d0 msgr2=0x55976140bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140b9d0 0x55976140bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 -- 172.21.15.16:0/2864127154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140c2f0 msgr2=0x559761416670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8157fa640 1 -- 172.21.15.16:0/2803855126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa81002e030 con 0x7fa818106a90 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140c2f0 0x559761416670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.202+0000 7fa8177fe640 1 -- 172.21.15.16:0/2864127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559761326130 con 0x559761416bb0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.203+0000 7fa8177fe640 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa80c011740 tx=0x7fa80c00a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9166863f84269 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.203+0000 7fa81c9e7640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 0x7fa8000a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa8100b9550 tx=0x7fa81008f100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.203+0000 7fa7fe7fc640 1 -- 172.21.15.16:0/2864127154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c004020 con 0x559761416bb0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.203+0000 7fa7fe7fc640 1 -- 172.21.15.16:0/2864127154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c0041c0 con 0x559761416bb0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.203+0000 7fa7fe7fc640 1 -- 172.21.15.16:0/2864127154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c005400 con 0x559761416bb0 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- 172.21.15.16:0/2864127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 msgr2=0x559761418fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa80c011740 tx=0x7fa80c00a750 comp rx=0 tx=0).stop 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- 172.21.15.16:0/2864127154 shutdown_connections 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761418fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140c2f0 0x559761416670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 --2- 172.21.15.16:0/2864127154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140b9d0 0x55976140bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- 172.21.15.16:0/2864127154 >> 172.21.15.16:0/2864127154 conn(0x559761407f60 msgr2=0x559761408340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- 172.21.15.16:0/2864127154 shutdown_connections 2024-09-12T02:47:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- 172.21.15.16:0/2864127154 wait complete. 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 -- start start 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.204+0000 7fa820823740 1 Processor -- start 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x559761416420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761413770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559761419780 con 0x559761416bb0 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55976132cf20 con 0x55976140b9d0 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559761419900 con 0x55976140c2f0 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa820823740 10 client.0 _ll_register_callbacks cb 0x5597612f6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58750/0 (socket says 172.21.15.16:58750) 2024-09-12T02:47:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 -- 172.21.15.16:0/1013787561 learned_addr learned my addr 172.21.15.16:0/1013787561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x559761416420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa817fff640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761413770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 -- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 msgr2=0x559761413770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761413770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 -- 172.21.15.16:0/1013787561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 msgr2=0x559761416420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x559761416420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 -- 172.21.15.16:0/1013787561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597610e32b0 con 0x559761416bb0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00f790 tx=0x7fa80c0050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1abfdf23eeabe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.205+0000 7fa7f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00fbd0 con 0x559761416bb0 2024-09-12T02:47:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa80c00fd70 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c0041c0 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa80c011020 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 10 client.15138 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559761425070 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c0041c0 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa80c01e500 con 0x559761416bb0 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 10 client.15138 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 10 client.15138 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 20 client.15138 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x5597614288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 4 client.15138 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.206+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55976137f600 con 0x559761427430 2024-09-12T02:47:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.207+0000 7fa817fff640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x5597614288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.207+0000 7fa7f77fe640 10 client.15138 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.207+0000 7fa817fff640 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x5597614288a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3e45c983b67413d server_cookie=726e7d911e333c89 in_seq=0 out_seq=0 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1013787561 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55976137f600 con 0x559761427430 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa7f77fe640 10 client.15138 handle_command_reply: tid=0 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 client.15138 shutdown 2024-09-12T02:47:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 msgr2=0x559761413cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00f790 tx=0x7fa80c0050c0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 shutdown_connections 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559761416bb0 0x559761413cb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x5597614288a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761413770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa7f77fe640 0 client.15138 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 --2- 172.21.15.16:0/1013787561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x559761416420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 >> 172.21.15.16:0/1013787561 conn(0x559761407f60 msgr2=0x55976140dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 shutdown_connections 2024-09-12T02:47:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 1 -- 172.21.15.16:0/1013787561 wait complete. 2024-09-12T02:47:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.208+0000 7fa820823740 20 client.15138 trim_cache size 0 max 16384 2024-09-12T02:47:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 Processor -- start 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 -- start start 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761427b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559761416bb0 0x559761413030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559761426be0 con 0x55976140b9d0 2024-09-12T02:47:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5597614272b0 con 0x559761416bb0 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.226+0000 7fa820823740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559761427980 con 0x55976140c2f0 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58758/0 (socket says 172.21.15.16:58758) 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 -- 172.21.15.16:0/1860309268 learned_addr learned my addr 172.21.15.16:0/1860309268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa817fff640 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761427b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 -- 172.21.15.16:0/1860309268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 msgr2=0x559761427b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761427b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 -- 172.21.15.16:0/1860309268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559761416bb0 msgr2=0x559761413030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559761416bb0 0x559761413030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 -- 172.21.15.16:0/1860309268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559761416040 con 0x55976140b9d0 2024-09-12T02:47:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa817fff640 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761427b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa80c011550 tx=0x7fa80c00a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a31ad00fe197a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa7fdffb640 1 -- 172.21.15.16:0/1860309268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c03d070 con 0x55976140b9d0 2024-09-12T02:47:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa7fdffb640 1 -- 172.21.15.16:0/1860309268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c00a6a0 con 0x55976140b9d0 2024-09-12T02:47:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.227+0000 7fa7fdffb640 1 -- 172.21.15.16:0/1860309268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c01e470 con 0x55976140b9d0 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 -- 172.21.15.16:0/1860309268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 msgr2=0x559761426d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa80c011550 tx=0x7fa80c00a370 comp rx=0 tx=0).stop 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 -- 172.21.15.16:0/1860309268 shutdown_connections 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559761416bb0 0x559761413030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55976140c2f0 0x559761427b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 --2- 172.21.15.16:0/1860309268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140b9d0 0x559761426d60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.228+0000 7fa820823740 1 -- 172.21.15.16:0/1860309268 >> 172.21.15.16:0/1860309268 conn(0x559761407f60 msgr2=0x559761408340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- 172.21.15.16:0/1860309268 shutdown_connections 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- 172.21.15.16:0/1860309268 wait complete. 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- start start 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 Processor -- start 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x55976142c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559761416bb0 0x55976148fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559761428720 con 0x55976140c2f0 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559761428050 con 0x55976140b9d0 2024-09-12T02:47:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559761428df0 con 0x559761416bb0 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa820823740 10 client.0 _ll_register_callbacks cb 0x7fa80800b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.229+0000 7fa817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58770/0 (socket says 172.21.15.16:58770) 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 -- 172.21.15.16:0/1064006395 learned_addr learned my addr 172.21.15.16:0/1064006395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x55976142c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559761416bb0 0x55976148fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 -- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559761416bb0 msgr2=0x55976148fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559761416bb0 0x55976148fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 -- 172.21.15.16:0/1064006395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 msgr2=0x55976142c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x55976142c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 -- 172.21.15.16:0/1064006395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559761490170 con 0x55976140c2f0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa817fff640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x559761413000 tx=0x7fa8080119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a41053d7cb1cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa7f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80801d940 con 0x55976140c2f0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa80801dae0 con 0x55976140c2f0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c00a6a0 con 0x55976140c2f0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa80801dce0 con 0x55976140c2f0 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa820823740 10 client.15144 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.230+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597614914e0 con 0x55976140c2f0 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c00a6a0 con 0x55976140c2f0 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa808010700 con 0x55976140c2f0 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 10 client.15144 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 10 client.15144 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 20 client.15144 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x559761494e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 4 client.15144 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "status"}) -- 0x559761379b10 con 0x559761427430 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.231+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x559761494e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa7f77fe640 10 client.15144 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa8177fe640 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x559761494e30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d89fd082e5e8bd8f server_cookie=26ef1eb39dd5659f in_seq=0 out_seq=0 2024-09-12T02:47:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa7f77fe640 1 -- 172.21.15.16:0/1064006395 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559761379b10 con 0x559761427430 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa7f77fe640 10 client.15144 handle_command_reply: tid=0 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 client.15144 shutdown 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 msgr2=0x55976142c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x559761413000 tx=0x7fa8080119b0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 shutdown_connections 2024-09-12T02:47:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x559761427430 0x559761494e30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559761416bb0 0x55976148fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55976140c2f0 0x55976142c630 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 --2- 172.21.15.16:0/1064006395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55976140b9d0 0x55976142c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 >> 172.21.15.16:0/1064006395 conn(0x559761407f60 msgr2=0x559761412000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa7f77fe640 0 client.15144 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 shutdown_connections 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 1 -- 172.21.15.16:0/1064006395 wait complete. 2024-09-12T02:47:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.232+0000 7fa820823740 20 client.15144 trim_cache size 0 max 16384 2024-09-12T02:47:27.246 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-12T02:47:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24445, 2024-09-12T02:47:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 91.003583688999996, 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 118.524114737 2024-09-12T02:47:27.247 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 msgr2=0x7fa8000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 0x7fa8000a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa8100b9550 tx=0x7fa81008f100 comp rx=0 tx=0).stop 2024-09-12T02:47:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 msgr2=0x7fa81819ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa810005cc0 tx=0x7fa810003970 comp rx=0 tx=0).stop 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 shutdown_connections 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818106a90 0x7fa81819ccd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa818101430 0x7fa818198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 --2- 172.21.15.16:0/2803855126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa818100a90 0x7fa818198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 >> 172.21.15.16:0/2803855126 conn(0x7fa8180fc7d0 msgr2=0x7fa818104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 shutdown_connections 2024-09-12T02:47:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:27.233+0000 7fa81e44a640 1 -- 172.21.15.16:0/2803855126 wait complete. 2024-09-12T02:47:27.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.h status 2024-09-12T02:47:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:27 smithi016 ceph-mon[30925]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:27 smithi033 ceph-mon[35092]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:27 smithi184 ceph-mon[35321]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 -- 172.21.15.16:0/2457999207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481082a0 msgr2=0x7f444810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 --2- 172.21.15.16:0/2457999207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481082a0 0x7f444810ec40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4440009cd0 tx=0x7f444002a910 comp rx=0 tx=0).stop 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 -- 172.21.15.16:0/2457999207 shutdown_connections 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 --2- 172.21.15.16:0/2457999207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481082a0 0x7f444810ec40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 --2- 172.21.15.16:0/2457999207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481036e0 0x7f4448107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 --2- 172.21.15.16:0/2457999207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 0x7f4448103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 -- 172.21.15.16:0/2457999207 >> 172.21.15.16:0/2457999207 conn(0x7f44480fc7d0 msgr2=0x7f44480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 -- 172.21.15.16:0/2457999207 shutdown_connections 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.620+0000 7f444dca7640 1 -- 172.21.15.16:0/2457999207 wait complete. 2024-09-12T02:47:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 Processor -- start 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 -- start start 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 0x7f44481986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481082a0 0x7f444819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4448112160 con 0x7f44481036e0 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4448111fe0 con 0x7f4448102d30 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f444dca7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44481122e0 con 0x7f44481082a0 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58802/0 (socket says 172.21.15.16:58802) 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f4446ffd640 1 -- 172.21.15.16:0/2538103754 learned_addr learned my addr 172.21.15.16:0/2538103754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f44477fe640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 0x7f44481986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.621+0000 7f4447fff640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481082a0 0x7f444819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 -- 172.21.15.16:0/2538103754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481082a0 msgr2=0x7f444819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481082a0 0x7f444819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 -- 172.21.15.16:0/2538103754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 msgr2=0x7f44481986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 0x7f44481986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 -- 172.21.15.16:0/2538103754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444819d660 con 0x7f44481036e0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4446ffd640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4430006fd0 tx=0x7f443000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4444ff9640 1 -- 172.21.15.16:0/2538103754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430015020 con 0x7f44481036e0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444819d950 con 0x7f44481036e0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4444ff9640 1 -- 172.21.15.16:0/2538103754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4430011d10 con 0x7f44481036e0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f4444ff9640 1 -- 172.21.15.16:0/2538103754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443000f2e0 con 0x7f44481036e0 2024-09-12T02:47:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.622+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44481a5230 con 0x7f44481036e0 2024-09-12T02:47:28.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.624+0000 7f4444ff9640 1 -- 172.21.15.16:0/2538103754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4430018020 con 0x7f44481036e0 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 Processor -- start 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 -- start start 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132dbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556513242090 con 0x5565132dc2f0 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565131e9a70 con 0x5565132db9d0 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4450080740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55651323e050 con 0x5565132e6bb0 2024-09-12T02:47:28.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58814/0 (socket says 172.21.15.16:58814) 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 -- 172.21.15.16:0/704413272 learned_addr learned my addr 172.21.15.16:0/704413272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4447fff640 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132dbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f4446ffd640 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 -- 172.21.15.16:0/704413272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 msgr2=0x5565132e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 -- 172.21.15.16:0/704413272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 msgr2=0x5565132dbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132dbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.625+0000 7f44477fe640 1 -- 172.21.15.16:0/704413272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565131f6130 con 0x5565132dc2f0 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f44477fe640 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4438009a70 tx=0x7f4438009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efa13aa4c5c4cfd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f443dffb640 1 -- 172.21.15.16:0/704413272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4438017070 con 0x5565132dc2f0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f443dffb640 1 -- 172.21.15.16:0/704413272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4438013470 con 0x5565132dc2f0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f443dffb640 1 -- 172.21.15.16:0/704413272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4438012630 con 0x5565132dc2f0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f4444ff9640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 0x7f44340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f4447fff640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 0x7f44340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.626+0000 7f4444ff9640 1 -- 172.21.15.16:0/2538103754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44300ce940 con 0x7f44481036e0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- 172.21.15.16:0/704413272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 msgr2=0x5565132e6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4447fff640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 0x7f44340a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f44400098d0 tx=0x7f44400099e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4438009a70 tx=0x7f4438009e70 comp rx=0 tx=0).stop 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- 172.21.15.16:0/704413272 shutdown_connections 2024-09-12T02:47:28.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e6670 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 --2- 172.21.15.16:0/704413272 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132dbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- 172.21.15.16:0/704413272 >> 172.21.15.16:0/704413272 conn(0x5565132d7f60 msgr2=0x5565132d8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- 172.21.15.16:0/704413272 shutdown_connections 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- 172.21.15.16:0/704413272 wait complete. 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 -- start start 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.627+0000 7f4450080740 1 Processor -- start 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565132e9780 con 0x5565132dc2f0 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565131fcf20 con 0x5565132db9d0 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5565132e9900 con 0x5565132e6bb0 2024-09-12T02:47:28.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 10 client.0 _ll_register_callbacks cb 0x5565131c6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58822/0 (socket says 172.21.15.16:58822) 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 -- 172.21.15.16:0/549907750 learned_addr learned my addr 172.21.15.16:0/549907750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f44477fe640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4447fff640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 -- 172.21.15.16:0/549907750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 msgr2=0x5565132e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 -- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 msgr2=0x5565132e6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 -- 172.21.15.16:0/549907750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556512fb32b0 con 0x5565132dc2f0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4446ffd640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f443000bd20 tx=0x7f44300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a83ceef9d0004d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:28.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443009b160 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44300bb0e0 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.628+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4438013470 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f443009f510 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 10 client.15156 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5565132f50c0 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4438013470 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f443009a5b0 con 0x5565132dc2f0 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 10 client.15156 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 10 client.15156 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 20 client.15156 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5565132f7670 0x5565132f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 4 client.15156 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:28.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55651324f600 con 0x5565132f7670 2024-09-12T02:47:28.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.629+0000 7f44477fe640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5565132f7670 0x5565132f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.630+0000 7f44477fe640 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5565132f7670 0x5565132f8ae0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f49e4fd438a09dfb server_cookie=ab25c2f7cbb1bec9 in_seq=0 out_seq=0 2024-09-12T02:47:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.630+0000 7f4426ffd640 10 client.15156 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.630+0000 7f4426ffd640 1 -- 172.21.15.16:0/549907750 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55651324f600 con 0x5565132f7670 2024-09-12T02:47:28.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.630+0000 7f4426ffd640 10 client.15156 handle_command_reply: tid=0 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 client.15156 shutdown 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 msgr2=0x5565132e3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f443000bd20 tx=0x7f44300046c0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 shutdown_connections 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5565132f7670 0x5565132f8ae0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132e6bb0 0x5565132e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132dc2f0 0x5565132e3790 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 --2- 172.21.15.16:0/549907750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 >> 172.21.15.16:0/549907750 conn(0x5565132d7f60 msgr2=0x5565132ddc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4426ffd640 0 client.15156 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 shutdown_connections 2024-09-12T02:47:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 1 -- 172.21.15.16:0/549907750 wait complete. 2024-09-12T02:47:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.631+0000 7f4450080740 20 client.15156 trim_cache size 0 max 16384 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 Processor -- start 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 -- start start 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132db9d0 0x5565132f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 0x5565133011b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565132f6e20 con 0x5565132e6bb0 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565132f74f0 con 0x5565132dc2f0 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4450080740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5565132f7bc0 con 0x5565132db9d0 2024-09-12T02:47:28.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58824/0 (socket says 172.21.15.16:58824) 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 -- 172.21.15.16:0/129369925 learned_addr learned my addr 172.21.15.16:0/129369925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f44477fe640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132db9d0 0x5565132f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4446ffd640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 0x5565133011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 -- 172.21.15.16:0/129369925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132db9d0 msgr2=0x5565132f8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132db9d0 0x5565132f8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 -- 172.21.15.16:0/129369925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 msgr2=0x5565133011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 0x5565133011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4447fff640 1 -- 172.21.15.16:0/129369925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565132e67b0 con 0x5565132e6bb0 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.649+0000 7f4446ffd640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 0x5565133011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4447fff640 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4440032dc0 tx=0x7f4440041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eb08ecc9844c1d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f443d7fa640 1 -- 172.21.15.16:0/129369925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440032eb0 con 0x5565132e6bb0 2024-09-12T02:47:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f443d7fa640 1 -- 172.21.15.16:0/129369925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4440036050 con 0x5565132e6bb0 2024-09-12T02:47:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f443d7fa640 1 -- 172.21.15.16:0/129369925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444002fdd0 con 0x5565132e6bb0 2024-09-12T02:47:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4450080740 1 -- 172.21.15.16:0/129369925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 msgr2=0x5565132e2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4450080740 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4440032dc0 tx=0x7f4440041000 comp rx=0 tx=0).stop 2024-09-12T02:47:28.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4450080740 1 -- 172.21.15.16:0/129369925 shutdown_connections 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4450080740 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x5565132e2060 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.650+0000 7f4450080740 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132dc2f0 0x5565133011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 --2- 172.21.15.16:0/129369925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132db9d0 0x5565132f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- 172.21.15.16:0/129369925 >> 172.21.15.16:0/129369925 conn(0x5565132d7f60 msgr2=0x5565132d2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- 172.21.15.16:0/129369925 shutdown_connections 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- 172.21.15.16:0/129369925 wait complete. 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- start start 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 Processor -- start 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 0x5565132e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565132f8960 con 0x5565132e6bb0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.651+0000 7f4450080740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565132f8290 con 0x5565132db9d0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4450080740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5565132f9030 con 0x5565132dc2f0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4450080740 10 client.0 _ll_register_callbacks cb 0x7f44300107e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58826/0 (socket says 172.21.15.16:58826) 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 -- 172.21.15.16:0/469863311 learned_addr learned my addr 172.21.15.16:0/469863311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4446ffd640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 0x5565132e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f44477fe640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 -- 172.21.15.16:0/469863311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 msgr2=0x5565132e0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 0x5565132e0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 -- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 msgr2=0x5565132e09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 -- 172.21.15.16:0/469863311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55651335d9b0 con 0x5565132e6bb0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f44477fe640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4446ffd640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 0x5565132e0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4447fff640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f444002a950 tx=0x7f4440031950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41ad0e2e8a7a29d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444002fdb0 con 0x5565132e6bb0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44400305d0 con 0x5565132e6bb0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4440036050 con 0x5565132e6bb0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f44400307d0 con 0x5565132e6bb0 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.652+0000 7f4450080740 10 client.15162 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556513365eb0 con 0x5565132e6bb0 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4440036050 con 0x5565132e6bb0 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4440004410 con 0x5565132e6bb0 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 10 client.15162 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 10 client.15162 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 20 client.15162 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55651336a550 0x556513369260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 4 client.15162 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "status"}) -- 0x556513249b10 con 0x55651336a550 2024-09-12T02:47:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.653+0000 7f44477fe640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55651336a550 0x556513369260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f44477fe640 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55651336a550 0x556513369260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c583e852180678cb server_cookie=55fab8cda70bf2b1 in_seq=0 out_seq=0 2024-09-12T02:47:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4426ffd640 10 client.15162 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:28.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4426ffd640 1 -- 172.21.15.16:0/469863311 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556513249b10 con 0x55651336a550 2024-09-12T02:47:28.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4426ffd640 10 client.15162 handle_command_reply: tid=0 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 client.15162 shutdown 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 msgr2=0x55651335d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f444002a950 tx=0x7f4440031950 comp rx=0 tx=0).stop 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 shutdown_connections 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55651336a550 0x556513369260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565132e6bb0 0x55651335d310 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565132dc2f0 0x5565132e0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 --2- 172.21.15.16:0/469863311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565132db9d0 0x5565132e09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 >> 172.21.15.16:0/469863311 conn(0x5565132d7f60 msgr2=0x5565132e1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4426ffd640 0 client.15162 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 shutdown_connections 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.654+0000 7f4450080740 1 -- 172.21.15.16:0/469863311 wait complete. 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f4450080740 20 client.15162 trim_cache size 0 max 16384 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24467, 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:28.665 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 93.435983816000004, 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 122.44167377799999 2024-09-12T02:47:28.666 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 msgr2=0x7f44340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 0x7f44340a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f44400098d0 tx=0x7f44400099e0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 msgr2=0x7f4448198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4430006fd0 tx=0x7f443000b820 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 shutdown_connections 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44481082a0 0x7f444819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44340a2c10 0x7f44340a50d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44481036e0 0x7f4448198bf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 --2- 172.21.15.16:0/2538103754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4448102d30 0x7f44481986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.655+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 >> 172.21.15.16:0/2538103754 conn(0x7f44480fc7d0 msgr2=0x7f4448105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.656+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 shutdown_connections 2024-09-12T02:47:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:28.656+0000 7f444dca7640 1 -- 172.21.15.16:0/2538103754 wait complete. 2024-09-12T02:47:29.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.b status 2024-09-12T02:47:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:30.140 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:29 smithi016 ceph-mon[30925]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 -- 172.21.15.16:0/946290148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 msgr2=0x7facf010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 --2- 172.21.15.16:0/946290148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 0x7facf010da40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7face8009900 tx=0x7face802a480 comp rx=0 tx=0).stop 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 -- 172.21.15.16:0/946290148 shutdown_connections 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 --2- 172.21.15.16:0/946290148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 0x7facf010da40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 --2- 172.21.15.16:0/946290148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 --2- 172.21.15.16:0/946290148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0100e20 0x7facf0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 -- 172.21.15.16:0/946290148 >> 172.21.15.16:0/946290148 conn(0x7facf00fc7d0 msgr2=0x7facf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 -- 172.21.15.16:0/946290148 shutdown_connections 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 -- 172.21.15.16:0/946290148 wait complete. 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.139+0000 7facf618a640 1 Processor -- start 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 -- start start 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 0x7facf0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111020 con 0x7facf0102570 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf0110ea0 con 0x7facf0102ef0 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facf618a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facf01111a0 con 0x7facf0100e20 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58850/0 (socket says 172.21.15.16:58850) 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7faceeffd640 1 -- 172.21.15.16:0/3561204422 learned_addr learned my addr 172.21.15.16:0/3561204422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7faceffff640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.140+0000 7facef7fe640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 0x7facf0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 -- 172.21.15.16:0/3561204422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 msgr2=0x7facf0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 0x7facf0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 -- 172.21.15.16:0/3561204422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 msgr2=0x7facf019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 -- 172.21.15.16:0/3561204422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019d630 con 0x7facf0102570 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7faceeffd640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7face400b4a0 tx=0x7face400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7facecff9640 1 -- 172.21.15.16:0/3561204422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4014070 con 0x7facf0102570 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7facecff9640 1 -- 172.21.15.16:0/3561204422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face4010af0 con 0x7facf0102570 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf019d920 con 0x7facf0102570 2024-09-12T02:47:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7facecff9640 1 -- 172.21.15.16:0/3561204422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4017d10 con 0x7facf0102570 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.141+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a5200 con 0x7facf0102570 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.143+0000 7facecff9640 1 -- 172.21.15.16:0/3561204422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7face4017070 con 0x7facf0102570 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 Processor -- start 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 -- start start 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facecff9640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 0x7facd80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3e1d0 0x56126cc4ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56126cba4090 con 0x56126cc3d8b0 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56126cb4ba70 con 0x56126cc4cfc0 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facf8563740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56126cba0050 con 0x56126cc3e1d0 2024-09-12T02:47:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7faceffff640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 0x7facd80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facecff9640 1 -- 172.21.15.16:0/3561204422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face40ce3d0 con 0x7facf0102570 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58856/0 (socket says 172.21.15.16:58856) 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.144+0000 7facef7fe640 1 -- 172.21.15.16:0/3499392945 learned_addr learned my addr 172.21.15.16:0/3499392945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7faceeffd640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3e1d0 0x56126cc4ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7faceffff640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 -- 172.21.15.16:0/3499392945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3e1d0 msgr2=0x56126cc4ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3e1d0 0x56126cc4ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 -- 172.21.15.16:0/3499392945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 msgr2=0x56126cc4f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 -- 172.21.15.16:0/3499392945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56126cb58130 con 0x56126cc3d8b0 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7faceffff640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7faceffff640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 0x7facd80a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7face80096c0 tx=0x7face8004240 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facef7fe640 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7face000a4b0 tx=0x7face00096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64515c4ea1fd848 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facd5ffb640 1 -- 172.21.15.16:0/3499392945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0018020 con 0x56126cc3d8b0 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facd5ffb640 1 -- 172.21.15.16:0/3499392945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face0014cf0 con 0x56126cc3d8b0 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.145+0000 7facd5ffb640 1 -- 172.21.15.16:0/3499392945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0013d20 con 0x56126cc3d8b0 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- 172.21.15.16:0/3499392945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 msgr2=0x56126cc3dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7face000a4b0 tx=0x7face00096d0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- 172.21.15.16:0/3499392945 shutdown_connections 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3e1d0 0x56126cc4ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 --2- 172.21.15.16:0/3499392945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3d8b0 0x56126cc3dc90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- 172.21.15.16:0/3499392945 >> 172.21.15.16:0/3499392945 conn(0x56126cc39f60 msgr2=0x56126cc3a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- 172.21.15.16:0/3499392945 shutdown_connections 2024-09-12T02:47:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- 172.21.15.16:0/3499392945 wait complete. 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 -- start start 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.146+0000 7facf8563740 1 Processor -- start 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc4c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56126cc4fb90 con 0x56126cc3e1d0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56126cb5ef20 con 0x56126cc4cfc0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56126cc4fd10 con 0x56126cc3d8b0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facf8563740 10 client.0 _ll_register_callbacks cb 0x56126cb28030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58870/0 (socket says 172.21.15.16:58870) 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facef7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc4c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 -- 172.21.15.16:0/1297221935 learned_addr learned my addr 172.21.15.16:0/1297221935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceffff640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 -- 172.21.15.16:0/1297221935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 msgr2=0x56126cc4c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc4c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 -- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 msgr2=0x56126cc4a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 -- 172.21.15.16:0/1297221935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126c9152b0 con 0x56126cc3e1d0 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facef7fe640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc4c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7faceeffd640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7face40cbed0 tx=0x7face4009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8338934f61be53fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.147+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face401e070 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face4017360 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face0014cf0 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7face4010ca0 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 10 client.15174 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56126cc5b440 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face0014cf0 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7face409a210 con 0x56126cc3e1d0 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 10 client.15174 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 10 client.15174 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 20 client.15174 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cc5da40 0x56126cc5eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 4 client.15174 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56126cbb1600 con 0x56126cc5da40 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.148+0000 7facef7fe640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cc5da40 0x56126cc5eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.149+0000 7facef7fe640 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cc5da40 0x56126cc5eeb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15b1e8ca4f8d41f1 server_cookie=b1b9e3ad7f43b1a in_seq=0 out_seq=0 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.149+0000 7facceffd640 10 client.15174 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.149+0000 7facceffd640 1 -- 172.21.15.16:0/1297221935 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56126cbb1600 con 0x56126cc5da40 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.149+0000 7facceffd640 10 client.15174 handle_command_reply: tid=0 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.149+0000 7facf8563740 1 client.15174 shutdown 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 msgr2=0x56126cc49b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7face40cbed0 tx=0x7face4009370 comp rx=0 tx=0).stop 2024-09-12T02:47:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 shutdown_connections 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cc5da40 0x56126cc5eeb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc4a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc49b40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facceffd640 0 client.15174 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 --2- 172.21.15.16:0/1297221935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc4c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 >> 172.21.15.16:0/1297221935 conn(0x56126cc39f60 msgr2=0x56126cc3fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 shutdown_connections 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 1 -- 172.21.15.16:0/1297221935 wait complete. 2024-09-12T02:47:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.150+0000 7facf8563740 20 client.15174 trim_cache size 0 max 16384 2024-09-12T02:47:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 Processor -- start 2024-09-12T02:47:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 -- start start 2024-09-12T02:47:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc5eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 0x56126cc5d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 0x56126cc67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5d1f0 con 0x56126cc4cfc0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5d8c0 con 0x56126cc3e1d0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facf8563740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5df90 con 0x56126cc3d8b0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7faceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 0x56126cc67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7faceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 0x56126cc67190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58882/0 (socket says 172.21.15.16:58882) 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7faceffff640 1 -- 172.21.15.16:0/807696182 learned_addr learned my addr 172.21.15.16:0/807696182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7faceeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 0x56126cc5d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.170+0000 7facef7fe640 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc5eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 -- 172.21.15.16:0/807696182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 msgr2=0x56126cc5eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc5eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 -- 172.21.15.16:0/807696182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 msgr2=0x56126cc67190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 0x56126cc67190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 -- 172.21.15.16:0/807696182 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56126cc4cb10 con 0x56126cc3e1d0 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7faceeffd640 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 0x56126cc5d370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7face40cbf20 tx=0x7face40991b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c83162c6a3d57eb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7facd57fa640 1 -- 172.21.15.16:0/807696182 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face401e070 con 0x56126cc3e1d0 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7facd57fa640 1 -- 172.21.15.16:0/807696182 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face40cbf50 con 0x56126cc3e1d0 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.171+0000 7facd57fa640 1 -- 172.21.15.16:0/807696182 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face409a0f0 con 0x56126cc3e1d0 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- 172.21.15.16:0/807696182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 msgr2=0x56126cc5d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 0x56126cc5d370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7face40cbf20 tx=0x7face40991b0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- 172.21.15.16:0/807696182 shutdown_connections 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc4cfc0 0x56126cc67190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc3e1d0 0x56126cc5d370 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 --2- 172.21.15.16:0/807696182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc5eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- 172.21.15.16:0/807696182 >> 172.21.15.16:0/807696182 conn(0x56126cc39f60 msgr2=0x56126cc48470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- 172.21.15.16:0/807696182 shutdown_connections 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- 172.21.15.16:0/807696182 wait complete. 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.172+0000 7facf8563740 1 -- start start 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 Processor -- start 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc67670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc46d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5ed30 con 0x56126cc3e1d0 2024-09-12T02:47:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5e660 con 0x56126cc4cfc0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56126cc5f400 con 0x56126cc3d8b0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facf8563740 10 client.0 _ll_register_callbacks cb 0x7face4017f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7faceeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58896/0 (socket says 172.21.15.16:58896) 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7faceeffd640 1 -- 172.21.15.16:0/2630290084 learned_addr learned my addr 172.21.15.16:0/2630290084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7faceffff640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc46d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.173+0000 7facef7fe640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc67670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 -- 172.21.15.16:0/2630290084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 msgr2=0x56126cc67670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc67670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 -- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 msgr2=0x56126cc46d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc46d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 -- 172.21.15.16:0/2630290084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126cc47370 con 0x56126cc3e1d0 2024-09-12T02:47:30.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7faceeffd640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7face4098f50 tx=0x7face40975e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73510012c8757d06 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face401e040 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face409a0f0 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face40cbf50 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7face4097310 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facf8563740 10 client.15180 fetch_fsmap learned FSMap version 27 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56126cc66050 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.174+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face40a9020 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7face40a3ab0 con 0x56126cc3e1d0 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 10 client.15180 fetch_fsmap finished waiting for FSMap version 27 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 10 client.15180 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 20 client.15180 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:30.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cccc430 0x56126cccb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 4 client.15180 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "status"}) -- 0x7face40cbf50 con 0x56126cccc430 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facef7fe640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cccc430 0x56126cccb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facef7fe640 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cccc430 0x56126cccb140 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a89eb01171fa93b6 server_cookie=edc1ca10e46ee1ee in_seq=0 out_seq=0 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.175+0000 7facceffd640 10 client.15180 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facceffd640 1 -- 172.21.15.16:0/2630290084 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x7face40cbf50 con 0x56126cccc430 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facceffd640 10 client.15180 handle_command_reply: tid=0 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 client.15180 shutdown 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 msgr2=0x56126cc467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7face4098f50 tx=0x7face40975e0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 shutdown_connections 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56126cccc430 0x56126cccb140 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56126cc4cfc0 0x56126cc46d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56126cc3e1d0 0x56126cc467f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 --2- 172.21.15.16:0/2630290084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56126cc3d8b0 0x56126cc67670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facceffd640 0 client.15180 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 >> 172.21.15.16:0/2630290084 conn(0x56126cc39f60 msgr2=0x56126cc44a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 shutdown_connections 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 1 -- 172.21.15.16:0/2630290084 wait complete. 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.176+0000 7facf8563740 20 client.15180 trim_cache size 0 max 16384 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 101.99586072, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 78, 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 112.61918536500001 2024-09-12T02:47:30.190 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 msgr2=0x7facd80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 0x7facd80a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7face80096c0 tx=0x7face8004240 comp rx=0 tx=0).stop 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 msgr2=0x7facf0198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7face400b4a0 tx=0x7face400b960 comp rx=0 tx=0).stop 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 shutdown_connections 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2c10 0x7facd80a50d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102570 0x7facf0198bc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 --2- 172.21.15.16:0/3561204422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 0x7facf0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 >> 172.21.15.16:0/3561204422 conn(0x7facf00fc7d0 msgr2=0x7facf006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 shutdown_connections 2024-09-12T02:47:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:30.177+0000 7facf618a640 1 -- 172.21.15.16:0/3561204422 wait complete. 2024-09-12T02:47:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:29 smithi033 ceph-mon[35092]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:29 smithi184 ceph-mon[35321]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:47:30.782 INFO:teuthology.run_tasks:Running task sequential... 2024-09-12T02:47:30.792 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-12T02:47:30.793 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-12T02:47:30.793 INFO:teuthology.task.print:Enabling mgr modules 2024-09-12T02:47:30.793 INFO:teuthology.run_tasks:Running task kclient... 2024-09-12T02:47:30.834 INFO:tasks.kclient:Mounting kernel clients... 2024-09-12T02:47:30.834 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-12T02:47:30.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:30.835 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-12T02:47:30.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:30.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-09-12T02:47:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-09-12T02:47:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T02:47:30.927 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-09-12T02:47:30.927 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-12T02:47:30.927 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:30.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs ls 2024-09-12T02:47:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:31.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 -- 172.21.15.16:0/4181970761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd694106f20 msgr2=0x7fd69410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:31.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/4181970761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd694106f20 0x7fd69410eaa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009ca0 tx=0x7fd68c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 -- 172.21.15.16:0/4181970761 shutdown_connections 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/4181970761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd694106f20 0x7fd69410eaa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/4181970761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6941027a0 0x7fd6941067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/4181970761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 0x7fd6941021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 -- 172.21.15.16:0/4181970761 >> 172.21.15.16:0/4181970761 conn(0x7fd6940fd8f0 msgr2=0x7fd6940ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 -- 172.21.15.16:0/4181970761 shutdown_connections 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.689+0000 7fd69a3e1640 1 -- 172.21.15.16:0/4181970761 wait complete. 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 Processor -- start 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 -- start start 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 0x7fd6941983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd694106f20 0x7fd69419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd694111f80 con 0x7fd6941027a0 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd694111e00 con 0x7fd694106f20 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69a3e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd694112100 con 0x7fd694101df0 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58922/0 (socket says 172.21.15.16:58922) 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 -- 172.21.15.16:0/2691209352 learned_addr learned my addr 172.21.15.16:0/2691209352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd693fff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 0x7fd6941983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd69897e640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd694106f20 0x7fd69419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 -- 172.21.15.16:0/2691209352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 msgr2=0x7fd6941983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 0x7fd6941983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 -- 172.21.15.16:0/2691209352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd694106f20 msgr2=0x7fd69419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:31.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd694106f20 0x7fd69419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 -- 172.21.15.16:0/2691209352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69419d390 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.690+0000 7fd68bfff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d950 tx=0x7fd67c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.691+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c017070 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.691+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67c009c30 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.691+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69419d680 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.691+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00f6b0 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.691+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6941a4f60 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.692+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6941021d0 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.693+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd67c0109d0 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.693+0000 7fd691ffb640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 0x7fd6800a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.694+0000 7fd693fff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 0x7fd6800a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.694+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd67c0ceaa0 con 0x7fd6941027a0 2024-09-12T02:47:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.694+0000 7fd693fff640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 0x7fd6800a50c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd684005fd0 tx=0x7fd68400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:31.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.698+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd67c009de0 con 0x7fd6941027a0 2024-09-12T02:47:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.911+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd69402d7c0 con 0x7fd6941027a0 2024-09-12T02:47:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.911+0000 7fd691ffb640 1 -- 172.21.15.16:0/2691209352 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7fd67c09b3c0 con 0x7fd6941027a0 2024-09-12T02:47:31.913 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-12T02:47:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 msgr2=0x7fd6800a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 0x7fd6800a50c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd684005fd0 tx=0x7fd68400b390 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 msgr2=0x7fd694198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d950 tx=0x7fd67c00de10 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 shutdown_connections 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd694106f20 0x7fd69419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6800a2c00 0x7fd6800a50c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6941027a0 0x7fd694198920 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 --2- 172.21.15.16:0/2691209352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd694101df0 0x7fd6941983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 >> 172.21.15.16:0/2691209352 conn(0x7fd6940fd8f0 msgr2=0x7fd6940fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 shutdown_connections 2024-09-12T02:47:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:31.913+0000 7fd69a3e1640 1 -- 172.21.15.16:0/2691209352 wait complete. 2024-09-12T02:47:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:31 smithi016 ceph-mon[30925]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:47:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:31 smithi016 ceph-mon[30925]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-12T02:47:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:31 smithi033 ceph-mon[35092]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:47:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:31 smithi033 ceph-mon[35092]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-12T02:47:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:31 smithi184 ceph-mon[35321]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:47:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:31 smithi184 ceph-mon[35321]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-12T02:47:32.584 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-12T02:47:32.585 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-12T02:47:32.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:32.585 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-09-12T02:47:32.602 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-12T02:47:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-12T02:47:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-12T02:47:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42222sec preferred_lft 42222sec 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-09-12T02:47:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-09-12T02:47:32.605 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-12T02:47:32.605 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-12T02:47:32.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:32.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:32.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:32.827 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-12T02:47:32.855 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-09-12T02:47:32.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:32.857 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-09-12T02:47:32.927 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-09-12T02:47:32.928 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-12T02:47:32.928 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-12T02:47:32.928 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-12T02:47:32.928 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-12T02:47:32.928 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-12T02:47:32.928 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-12T02:47:32.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-12T02:47:32.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-12T02:47:32.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-12T02:47:32.929 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-12T02:47:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:33.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2691209352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-12T02:47:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:33.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:33.231 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-12T02:47:33.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2691209352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-12T02:47:33.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:33.248 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-09-12T02:47:33.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2691209352' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-12T02:47:33.304 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-12T02:47:33.305 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-12T02:47:33.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-12T02:47:33.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-12T02:47:33.305 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-12T02:47:33.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:33.425 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-12T02:47:33.426 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-12T02:47:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:33.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:33.559 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-12T02:47:33.559 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-12T02:47:33.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-09-12T02:47:33.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-09-12T02:47:33.559 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-12T02:47:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:33.639 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-12T02:47:33.639 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:33.655 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-12T02:47:33.656 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-12T02:47:33.656 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:33.712 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-12T02:47:33.712 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-12T02:47:33.712 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/f393359a-6719-4728-81d5-f56f7e21ccf3 2024-09-12T02:47:33.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:33.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/f393359a-6719-4728-81d5-f56f7e21ccf3 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-09-12T02:47:33.816 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-09-12T02:47:33.817 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-09-12T02:47:33.817 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.33:3300,172.21.15.184:3300" 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:3300,172.21.15.33:3300,172.21.15.184:3300:/volumes/qa/sv_1/f393359a-6719-4728-81d5-f56f7e21ccf3 2024-09-12T02:47:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=ef86fd2a-70af-11ef-bcea-c7b262605968" will pass to kernel 2024-09-12T02:47:33.949 INFO:teuthology.orchestra.run.smithi016.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-12T02:47:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: You just mounted a file system that supports labels which does not 2024-09-12T02:47:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-12T02:47:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: applications will generate AVC messages and not be allowed access to 2024-09-12T02:47:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-12T02:47:33.950 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-12T02:47:33.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T02:47:33.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:33.978 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-12T02:47:34.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-12T02:47:34.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:34.077 INFO:teuthology.orchestra.run.smithi016.stdout:client24841 2024-09-12T02:47:34.077 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:47:34.077 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-12T02:47:34.095 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-12T02:47:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:33 smithi016 ceph-mon[30925]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-12T02:47:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-12T02:47:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:34.194 INFO:teuthology.orchestra.run.smithi016.stdout:client24841 2024-09-12T02:47:34.194 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-12T02:47:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:33 smithi033 ceph-mon[35092]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-12T02:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-12T02:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:34.292 INFO:teuthology.orchestra.run.smithi016.stdout:client24841 2024-09-12T02:47:34.292 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968.client24841/status') 2024-09-12T02:47:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:33 smithi184 ceph-mon[35321]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-12T02:47:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-12T02:47:34.352 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.24841 (3)172.21.15.16:0/4238878852 2024-09-12T02:47:34.352 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2024-09-12T02:47:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2024-09-12T02:47:34.353 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2024-09-12T02:47:34.353 INFO:teuthology.orchestra.run.smithi016.stderr:71 bytes copied, 2.8554e-05 s, 2.5 MB/s 2024-09-12T02:47:34.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-12T02:47:34.360 INFO:teuthology.run_tasks:Running task exec... 2024-09-12T02:47:34.371 INFO:teuthology.task.exec:Executing custom commands... 2024-09-12T02:47:34.371 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T02:47:34.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-12T02:47:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- 172.21.15.16:0/4067085518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 msgr2=0x7f9d1c1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 --2- 172.21.15.16:0/4067085518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c1020d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009900 tx=0x7f9d0402a410 comp rx=0 tx=0).stop 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- 172.21.15.16:0/4067085518 shutdown_connections 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 --2- 172.21.15.16:0/4067085518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 0x7f9d1c108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 --2- 172.21.15.16:0/4067085518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c1020d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 --2- 172.21.15.16:0/4067085518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- 172.21.15.16:0/4067085518 >> 172.21.15.16:0/4067085518 conn(0x7f9d1c0fc1e0 msgr2=0x7f9d1c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- 172.21.15.16:0/4067085518 shutdown_connections 2024-09-12T02:47:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- 172.21.15.16:0/4067085518 wait complete. 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 Processor -- start 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.474+0000 7f9d220b7640 1 -- start start 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 0x7f9d1c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c10c7e0 con 0x7f9d1c101c50 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c10c660 con 0x7f9d1c100a60 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d220b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c10c960 con 0x7f9d1c102610 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58954/0 (socket says 172.21.15.16:58954) 2024-09-12T02:47:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 -- 172.21.15.16:0/2482232850 learned_addr learned my addr 172.21.15.16:0/2482232850 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d1bfff640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 0x7f9d1c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d1b7fe640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 -- 172.21.15.16:0/2482232850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 msgr2=0x7f9d1c1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 0x7f9d1c1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 -- 172.21.15.16:0/2482232850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 msgr2=0x7f9d1c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 -- 172.21.15.16:0/2482232850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c1a1d90 con 0x7f9d1c101c50 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d1b7fe640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d13fff640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009230 tx=0x7f9d04004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.475+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0403f030 con 0x7f9d1c101c50 2024-09-12T02:47:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.476+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d04003ce0 con 0x7f9d1c101c50 2024-09-12T02:47:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.476+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c1a2020 con 0x7f9d1c101c50 2024-09-12T02:47:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.476+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04035070 con 0x7f9d1c101c50 2024-09-12T02:47:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.476+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c1a2590 con 0x7f9d1c101c50 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.477+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1c107da0 con 0x7f9d1c101c50 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.478+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9d04042030 con 0x7f9d1c101c50 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.478+0000 7f9d197fa640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 0x7f9d080a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.479+0000 7f9d1b7fe640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 0x7f9d080a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.479+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9d040ef7d0 con 0x7f9d1c101c50 2024-09-12T02:47:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.479+0000 7f9d1b7fe640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 0x7f9d080a50f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c101ab0 tx=0x7f9d0c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.483+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9d040bc740 con 0x7f9d1c101c50 2024-09-12T02:47:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.699+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f9d1c062df0 con 0x7f9d1c101c50 2024-09-12T02:47:34.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.904+0000 7f9d197fa640 1 -- 172.21.15.16:0/2482232850 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7f9d040dc100 con 0x7f9d1c101c50 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 msgr2=0x7f9d080a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 0x7f9d080a50f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c101ab0 tx=0x7f9d0c008040 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 msgr2=0x7f9d1c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009230 tx=0x7f9d04004430 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 shutdown_connections 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d1c102610 0x7f9d1c1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9d080a2c30 0x7f9d080a50f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d1c101c50 0x7f9d1c19f070 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 --2- 172.21.15.16:0/2482232850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d1c100a60 0x7f9d1c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:34.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 >> 172.21.15.16:0/2482232850 conn(0x7f9d1c0fc1e0 msgr2=0x7f9d1c104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:34.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 shutdown_connections 2024-09-12T02:47:34.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:34.906+0000 7f9d220b7640 1 -- 172.21.15.16:0/2482232850 wait complete. 2024-09-12T02:47:34.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-12T02:47:35.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 -- 172.21.15.16:0/1648046337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 msgr2=0x7fa42c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 --2- 172.21.15.16:0/1648046337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c108ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa424009ca0 tx=0x7fa42402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 -- 172.21.15.16:0/1648046337 shutdown_connections 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 --2- 172.21.15.16:0/1648046337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c108ea0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 --2- 172.21.15.16:0/1648046337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 --2- 172.21.15.16:0/1648046337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 0x7fa42c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 -- 172.21.15.16:0/1648046337 >> 172.21.15.16:0/1648046337 conn(0x7fa42c0fc1e0 msgr2=0x7fa42c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 -- 172.21.15.16:0/1648046337 shutdown_connections 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.024+0000 7fa432253640 1 -- 172.21.15.16:0/1648046337 wait complete. 2024-09-12T02:47:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 Processor -- start 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 -- start start 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 0x7fa42c104980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c6c0 con 0x7fa42c102610 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c540 con 0x7fa42c100a60 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa432253640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c840 con 0x7fa42c101c50 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58972/0 (socket says 172.21.15.16:58972) 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 -- 172.21.15.16:0/2564983138 learned_addr learned my addr 172.21.15.16:0/2564983138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42b7fe640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 0x7fa42c104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa422ffd640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 -- 172.21.15.16:0/2564983138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 msgr2=0x7fa42c103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 -- 172.21.15.16:0/2564983138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 msgr2=0x7fa42c104980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 0x7fa42c104980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa42bfff640 1 -- 172.21.15.16:0/2564983138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c103d90 con 0x7fa42c102610 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.025+0000 7fa422ffd640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa42bfff640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa424005e30 tx=0x7fa424031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424003c30 con 0x7fa42c102610 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa424003dd0 con 0x7fa42c102610 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1a1e20 con 0x7fa42c102610 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42402fde0 con 0x7fa42c102610 2024-09-12T02:47:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.026+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1a2330 con 0x7fa42c102610 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.027+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c1a2590 con 0x7fa42c102610 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.028+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa424004050 con 0x7fa42c102610 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.029+0000 7fa4297fa640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 0x7fa4180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.029+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa4240f03d0 con 0x7fa42c102610 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.029+0000 7fa42b7fe640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 0x7fa4180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.029+0000 7fa42b7fe640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 0x7fa4180a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa42c101ab0 tx=0x7fa41c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.033+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4240bd340 con 0x7fa42c102610 2024-09-12T02:47:35.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:34 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-12T02:47:35.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.235+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fa42c0fe3a0 con 0x7fa4180a2cb0 2024-09-12T02:47:35.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:34 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-12T02:47:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.258+0000 7fa4297fa640 1 -- 172.21.15.16:0/2564983138 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fa42c0fe3a0 con 0x7fa4180a2cb0 2024-09-12T02:47:35.260 INFO:teuthology.orchestra.run.smithi016.stdout:{} 2024-09-12T02:47:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 msgr2=0x7fa4180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 0x7fa4180a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa42c101ab0 tx=0x7fa41c00b390 comp rx=0 tx=0).stop 2024-09-12T02:47:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 msgr2=0x7fa42c103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa424005e30 tx=0x7fa424031ee0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 shutdown_connections 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4180a2cb0 0x7fa4180a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa42c102610 0x7fa42c103570 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa42c101c50 0x7fa42c103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 --2- 172.21.15.16:0/2564983138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100a60 0x7fa42c104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 >> 172.21.15.16:0/2564983138 conn(0x7fa42c0fc1e0 msgr2=0x7fa42c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 shutdown_connections 2024-09-12T02:47:35.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.261+0000 7fa432253640 1 -- 172.21.15.16:0/2564983138 wait complete. 2024-09-12T02:47:35.273 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-12T02:47:35.285 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-12T02:47:35.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T02:47:35.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:34 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 -- 172.21.15.16:0/4100398383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c81078b0 msgr2=0x7f55c8109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 --2- 172.21.15.16:0/4100398383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c81078b0 0x7f55c8109ca0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009ca0 tx=0x7f55c002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 -- 172.21.15.16:0/4100398383 shutdown_connections 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 --2- 172.21.15.16:0/4100398383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c81078b0 0x7f55c8109ca0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 --2- 172.21.15.16:0/4100398383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8104f80 0x7f55c8107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 --2- 172.21.15.16:0/4100398383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c8102650 0x7f55c8104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.414+0000 7f55cfaba640 1 -- 172.21.15.16:0/4100398383 >> 172.21.15.16:0/4100398383 conn(0x7f55c80fc170 msgr2=0x7f55c80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- 172.21.15.16:0/4100398383 shutdown_connections 2024-09-12T02:47:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- 172.21.15.16:0/4100398383 wait complete. 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 Processor -- start 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- start start 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 0x7f55c8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 0x7f55c819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55c810d7c0 con 0x7f55c8104f80 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55c810d640 con 0x7f55c81078b0 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cfaba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55c810d940 con 0x7f55c8102650 2024-09-12T02:47:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cd055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.415+0000 7f55cd055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36536/0 (socket says 172.21.15.16:36536) 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 -- 172.21.15.16:0/2630717089 learned_addr learned my addr 172.21.15.16:0/2630717089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd856640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 0x7f55c8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55ce057640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 0x7f55c819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 -- 172.21.15.16:0/2630717089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 msgr2=0x7f55c8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 0x7f55c8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 -- 172.21.15.16:0/2630717089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 msgr2=0x7f55c819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 0x7f55c819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 -- 172.21.15.16:0/2630717089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c819fa70 con 0x7f55c8104f80 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd856640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 0x7f55c8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55ce057640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 0x7f55c819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cd055640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f55b800d740 tx=0x7f55b800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8017070 con 0x7f55c8104f80 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c819fd60 con 0x7f55c8104f80 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b800de50 con 0x7f55c8104f80 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8004e00 con 0x7f55c8104f80 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.416+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c81a0200 con 0x7f55c8104f80 2024-09-12T02:47:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.417+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c80fdc60 con 0x7f55c8104f80 2024-09-12T02:47:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.418+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f55b8004fa0 con 0x7f55c8104f80 2024-09-12T02:47:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.419+0000 7f55beffd640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 0x7f55b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.419+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f55b80cee70 con 0x7f55c8104f80 2024-09-12T02:47:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.419+0000 7f55cd856640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 0x7f55b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.420+0000 7f55cd856640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 0x7f55b40a5050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009c20 tx=0x7f55c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.422+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f55b809bde0 con 0x7f55c8104f80 2024-09-12T02:47:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.600+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55c802da10 con 0x7f55c8104f80 2024-09-12T02:47:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.601+0000 7f55beffd640 1 -- 172.21.15.16:0/2630717089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f55b809b6e0 con 0x7f55c8104f80 2024-09-12T02:47:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:35.603 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:47:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.603+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 msgr2=0x7f55b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.603+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 0x7f55b40a5050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009c20 tx=0x7f55c4005f90 comp rx=0 tx=0).stop 2024-09-12T02:47:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.603+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 msgr2=0x7f55c8198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.603+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f55b800d740 tx=0x7f55b800dc00 comp rx=0 tx=0).stop 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 shutdown_connections 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55c81078b0 0x7f55c819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55b40a2b90 0x7f55b40a5050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c8104f80 0x7f55c8198390 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 --2- 172.21.15.16:0/2630717089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c8102650 0x7f55c8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 >> 172.21.15.16:0/2630717089 conn(0x7f55c80fc170 msgr2=0x7f55c8102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 shutdown_connections 2024-09-12T02:47:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.604+0000 7f55cfaba640 1 -- 172.21.15.16:0/2630717089 wait complete. 2024-09-12T02:47:35.615 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:47:35.239+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:47:35.615 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:47:35.239+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:47:35.615 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:47:35.239+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:47:35.615 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:47:35.239+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:47:35.615 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x[31153]: 2024-09-12T02:47:35.239+0000 7f902263c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-12T02:47:35.617 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-12T02:41:16.432997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-12T02:45:43.869635+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-12T02:45:46.156619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-12T02:47:35.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 -- 172.21.15.16:0/2155750869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024104f80 msgr2=0x7f7024107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 --2- 172.21.15.16:0/2155750869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024104f80 0x7f7024107370 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7014009900 tx=0x7f701402a410 comp rx=0 tx=0).stop 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 -- 172.21.15.16:0/2155750869 shutdown_connections 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 --2- 172.21.15.16:0/2155750869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70241078b0 0x7f7024109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 --2- 172.21.15.16:0/2155750869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024104f80 0x7f7024107370 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 --2- 172.21.15.16:0/2155750869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024102650 0x7f7024104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 -- 172.21.15.16:0/2155750869 >> 172.21.15.16:0/2155750869 conn(0x7f70240fc1b0 msgr2=0x7f70240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.722+0000 7f702b2f8640 1 -- 172.21.15.16:0/2155750869 shutdown_connections 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 -- 172.21.15.16:0/2155750869 wait complete. 2024-09-12T02:47:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 Processor -- start 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 -- start start 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024104f80 0x7f7024100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70241078b0 0x7f7024100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d7c0 con 0x7f7024102650 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d640 con 0x7f7024104f80 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f702b2f8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d940 con 0x7f70241078b0 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f7029094640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f7029094640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36540/0 (socket says 172.21.15.16:36540) 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f7029094640 1 -- 172.21.15.16:0/1499675320 learned_addr learned my addr 172.21.15.16:0/1499675320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.723+0000 7f7028893640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024104f80 0x7f7024100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 -- 172.21.15.16:0/1499675320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70241078b0 msgr2=0x7f7024100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70241078b0 0x7f7024100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 -- 172.21.15.16:0/1499675320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024104f80 msgr2=0x7f7024100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024104f80 0x7f7024100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 -- 172.21.15.16:0/1499675320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70241013d0 con 0x7f7024102650 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f7029094640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f702000d950 tx=0x7f702000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020017030 con 0x7f7024102650 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7020004020 con 0x7f7024102650 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70241a2080 con 0x7f7024102650 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70200047a0 con 0x7f7024102650 2024-09-12T02:47:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.724+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241a25a0 con 0x7f7024102650 2024-09-12T02:47:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.725+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024105360 con 0x7f7024102650 2024-09-12T02:47:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.726+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f70200041d0 con 0x7f7024102650 2024-09-12T02:47:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.727+0000 7f701a7fc640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 0x7f70100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.727+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f70200cf1f0 con 0x7f7024102650 2024-09-12T02:47:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.727+0000 7f7028893640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 0x7f70100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.728+0000 7f7028893640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 0x7f70100a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7014009230 tx=0x7f70140315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.730+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f702009c100 con 0x7f7024102650 2024-09-12T02:47:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.905+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f702402d7c0 con 0x7f7024102650 2024-09-12T02:47:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.905+0000 7f701a7fc640 1 -- 172.21.15.16:0/1499675320 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f702009ba00 con 0x7f7024102650 2024-09-12T02:47:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-12T02:47:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 msgr2=0x7f70100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 0x7f70100a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7014009230 tx=0x7f70140315b0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 msgr2=0x7f7024101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f702000d950 tx=0x7f702000de10 comp rx=0 tx=0).stop 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 shutdown_connections 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70241078b0 0x7f7024100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7024104f80 0x7f7024100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70100a2b90 0x7f70100a5050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 --2- 172.21.15.16:0/1499675320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7024102650 0x7f7024101fc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 >> 172.21.15.16:0/1499675320 conn(0x7f70240fc1b0 msgr2=0x7f7024102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 shutdown_connections 2024-09-12T02:47:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:35.907+0000 7f702b2f8640 1 -- 172.21.15.16:0/1499675320 wait complete. 2024-09-12T02:47:35.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-12T02:47:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 -- 172.21.15.16:0/850396116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58101bf0 msgr2=0x7ffa58102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/850396116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58101bf0 0x7ffa58102070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009900 tx=0x7ffa4c02a410 comp rx=0 tx=0).stop 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 -- 172.21.15.16:0/850396116 shutdown_connections 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/850396116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa581025b0 0x7ffa58108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/850396116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58101bf0 0x7ffa58102070 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/850396116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa58100a00 0x7ffa58100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 -- 172.21.15.16:0/850396116 >> 172.21.15.16:0/850396116 conn(0x7ffa580fc140 msgr2=0x7ffa580fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 -- 172.21.15.16:0/850396116 shutdown_connections 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.026+0000 7ffa5d20e640 1 -- 172.21.15.16:0/850396116 wait complete. 2024-09-12T02:47:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 Processor -- start 2024-09-12T02:47:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 -- start start 2024-09-12T02:47:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa58101bf0 0x7ffa58198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa581025b0 0x7ffa5819f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810c6e0 con 0x7ffa58100a00 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810c560 con 0x7ffa581025b0 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5d20e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810c860 con 0x7ffa58101bf0 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36554/0 (socket says 172.21.15.16:36554) 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa56d9d640 1 -- 172.21.15.16:0/3646401338 learned_addr learned my addr 172.21.15.16:0/3646401338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5759e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa581025b0 0x7ffa5819f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa5659c640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa58101bf0 0x7ffa58198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa56d9d640 1 -- 172.21.15.16:0/3646401338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa58101bf0 msgr2=0x7ffa58198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.027+0000 7ffa56d9d640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa58101bf0 0x7ffa58198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa56d9d640 1 -- 172.21.15.16:0/3646401338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa581025b0 msgr2=0x7ffa5819f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa56d9d640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa581025b0 0x7ffa5819f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa56d9d640 1 -- 172.21.15.16:0/3646401338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5819fa50 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa56d9d640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c970 tx=0x7ffa4800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48007c10 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5819fd40 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa48007db0 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800f8a0 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.028+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa581a0270 con 0x7ffa58100a00 2024-09-12T02:47:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.029+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa581a0560 con 0x7ffa58100a00 2024-09-12T02:47:36.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.030+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ffa4800fa80 con 0x7ffa58100a00 2024-09-12T02:47:36.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.031+0000 7ffa3ffff640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 0x7ffa400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.031+0000 7ffa5659c640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 0x7ffa400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.031+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffa480cf680 con 0x7ffa58100a00 2024-09-12T02:47:36.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.032+0000 7ffa5659c640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 0x7ffa400a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.034+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa48026080 con 0x7ffa58100a00 2024-09-12T02:47:36.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-mon[30925]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-12T02:47:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-12T02:47:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:47:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2630717089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.212+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa581a0730 con 0x7ffa58100a00 2024-09-12T02:47:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.213+0000 7ffa3ffff640 1 -- 172.21.15.16:0/3646401338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7ffa4809c680 con 0x7ffa58100a00 2024-09-12T02:47:36.214 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-09-12T02:47:36.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 msgr2=0x7ffa400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 0x7ffa400a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c031400 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 msgr2=0x7ffa58197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c970 tx=0x7ffa4800ce30 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 shutdown_connections 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa581025b0 0x7ffa5819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa58101bf0 0x7ffa58198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa400a2b90 0x7ffa400a5050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 --2- 172.21.15.16:0/3646401338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa58100a00 0x7ffa58197e30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 >> 172.21.15.16:0/3646401338 conn(0x7ffa580fc140 msgr2=0x7ffa58069360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 shutdown_connections 2024-09-12T02:47:36.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.215+0000 7ffa5d20e640 1 -- 172.21.15.16:0/3646401338 wait complete. 2024-09-12T02:47:36.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-12T02:47:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:35 smithi033 ceph-mon[35092]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-12T02:47:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-12T02:47:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:35 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:47:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2630717089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:35 smithi184 ceph-mon[35321]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-12T02:47:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2482232850' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-12T02:47:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:35 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:47:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2630717089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.332+0000 7f70cfadd640 1 -- 172.21.15.16:0/881881138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c81025b0 msgr2=0x7f70c8108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.332+0000 7f70cfadd640 1 --2- 172.21.15.16:0/881881138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c81025b0 0x7f70c8108e40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009ca0 tx=0x7f70c002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- 172.21.15.16:0/881881138 shutdown_connections 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- 172.21.15.16:0/881881138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c81025b0 0x7f70c8108e40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- 172.21.15.16:0/881881138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 0x7f70c8102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- 172.21.15.16:0/881881138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c8100a00 0x7f70c8100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- 172.21.15.16:0/881881138 >> 172.21.15.16:0/881881138 conn(0x7f70c80fc140 msgr2=0x7f70c80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- 172.21.15.16:0/881881138 shutdown_connections 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- 172.21.15.16:0/881881138 wait complete. 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 Processor -- start 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- start start 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 0x7f70c81099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c81025b0 0x7f70c8109f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70c810d7d0 con 0x7f70c8100a00 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70c810d650 con 0x7f70c81025b0 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.333+0000 7f70cfadd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70c810d950 con 0x7f70c8101bf0 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36574/0 (socket says 172.21.15.16:36574) 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 -- 172.21.15.16:0/3566126891 learned_addr learned my addr 172.21.15.16:0/3566126891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd078640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 0x7f70c81099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70ce07a640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c81025b0 0x7f70c8109f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 -- 172.21.15.16:0/3566126891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 msgr2=0x7f70c81099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 0x7f70c81099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 -- 172.21.15.16:0/3566126891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c81025b0 msgr2=0x7f70c8109f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c81025b0 0x7f70c8109f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 -- 172.21.15.16:0/3566126891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c810a790 con 0x7f70c8100a00 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cd879640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c970 tx=0x7f70c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4007c10 con 0x7f70c8100a00 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c4007db0 con 0x7f70c8100a00 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c81a1e80 con 0x7f70c8100a00 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c400f940 con 0x7f70c8100a00 2024-09-12T02:47:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.334+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c81a22e0 con 0x7f70c8100a00 2024-09-12T02:47:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.335+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c8107da0 con 0x7f70c8100a00 2024-09-12T02:47:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.337+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f70c400fae0 con 0x7f70c8100a00 2024-09-12T02:47:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.337+0000 7f70beffd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 0x7f70b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.337+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f70c40cf530 con 0x7f70c8100a00 2024-09-12T02:47:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.337+0000 7f70cd078640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 0x7f70b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.338+0000 7f70cd078640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 0x7f70b40a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70c810b0f0 tx=0x7f70b800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.340+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f70c409c5c0 con 0x7f70c8100a00 2024-09-12T02:47:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.514+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f70c802da50 con 0x7f70c8100a00 2024-09-12T02:47:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.515+0000 7f70beffd640 1 -- 172.21.15.16:0/3566126891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f70c409bec0 con 0x7f70c8100a00 2024-09-12T02:47:36.516 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.516+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 msgr2=0x7f70b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 0x7f70b40a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70c810b0f0 tx=0x7f70b800b390 comp rx=0 tx=0).stop 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 msgr2=0x7f70c810b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c970 tx=0x7f70c400ce30 comp rx=0 tx=0).stop 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 shutdown_connections 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70c81025b0 0x7f70c8109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70c8101bf0 0x7f70c81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70b40a2c10 0x7f70b40a50d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 --2- 172.21.15.16:0/3566126891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70c8100a00 0x7f70c810b380 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 >> 172.21.15.16:0/3566126891 conn(0x7f70c80fc140 msgr2=0x7f70c8104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 shutdown_connections 2024-09-12T02:47:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:36.517+0000 7f70cfadd640 1 -- 172.21.15.16:0/3566126891 wait complete. 2024-09-12T02:47:36.529 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-12T02:47:36.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:37.210 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:36 smithi016 ceph-mon[30925]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:37.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:36 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1499675320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:36 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3646401338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:36 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3566126891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:36 smithi033 ceph-mon[35092]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:37.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:36 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1499675320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:36 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3646401338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:36 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3566126891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:36 smithi184 ceph-mon[35321]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-12T02:47:37.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:36 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1499675320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:36 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3646401338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:36 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3566126891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T02:47:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 -- 172.21.15.16:0/3299609806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0101430 msgr2=0x7f8aa010d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/3299609806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0101430 0x7f8aa010d960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009cd0 tx=0x7f8a9802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 -- 172.21.15.16:0/3299609806 shutdown_connections 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/3299609806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0101430 0x7f8aa010d960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/3299609806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0100a90 0x7f8aa0100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/3299609806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0106a90 0x7f8aa0106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 -- 172.21.15.16:0/3299609806 >> 172.21.15.16:0/3299609806 conn(0x7f8aa00fc7d0 msgr2=0x7f8aa00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 -- 172.21.15.16:0/3299609806 shutdown_connections 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.292+0000 7f8aa4c70640 1 -- 172.21.15.16:0/3299609806 wait complete. 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 Processor -- start 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 -- start start 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0101430 0x7f8aa01977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0106a90 0x7f8aa019bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:37.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0110e90 con 0x7f8aa0100a90 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0110d10 con 0x7f8aa0106a90 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.293+0000 7f8aa4c70640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0111010 con 0x7f8aa0101430 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36584/0 (socket says 172.21.15.16:36584) 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 -- 172.21.15.16:0/673676612 learned_addr learned my addr 172.21.15.16:0/673676612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9dd9b640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0101430 0x7f8aa01977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9ed9d640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0106a90 0x7f8aa019bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 -- 172.21.15.16:0/673676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0101430 msgr2=0x7f8aa01977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0101430 0x7f8aa01977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 -- 172.21.15.16:0/673676612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0106a90 msgr2=0x7f8aa019bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0106a90 0x7f8aa019bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 -- 172.21.15.16:0/673676612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa019c230 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a9e59c640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009860 tx=0x7f8a90009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90005020 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a900047a0 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90007b20 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa019c520 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.294+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa006a790 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.295+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa01021d0 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.297+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f8a90007cc0 con 0x7f8aa0100a90 2024-09-12T02:47:37.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.297+0000 7f8a8f7fe640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 0x7f8a880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:37.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.297+0000 7f8a9dd9b640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 0x7f8a880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:37.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.297+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8a900d0df0 con 0x7f8aa0100a90 2024-09-12T02:47:37.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.298+0000 7f8a9dd9b640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 0x7f8a880a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00fece0 tx=0x7f8a9400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:37.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.300+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8a9009dc00 con 0x7f8aa0100a90 2024-09-12T02:47:37.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.513+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8aa0100ef0 con 0x7f8aa0100a90 2024-09-12T02:47:37.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.514+0000 7f8a8f7fe640 1 -- 172.21.15.16:0/673676612 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7f8a9009d500 con 0x7f8aa0100a90 2024-09-12T02:47:37.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:37.515 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:47:37.516 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 msgr2=0x7f8a880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 0x7f8a880a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00fece0 tx=0x7f8a9400a2d0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 msgr2=0x7f8aa0197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009860 tx=0x7f8a90009c60 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 shutdown_connections 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8aa0106a90 0x7f8aa019bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8aa0101430 0x7f8aa01977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a880a2c10 0x7f8a880a50d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 --2- 172.21.15.16:0/673676612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa0100a90 0x7f8aa0197280 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 >> 172.21.15.16:0/673676612 conn(0x7f8aa00fc7d0 msgr2=0x7f8aa00fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 shutdown_connections 2024-09-12T02:47:37.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:37.516+0000 7f8aa4c70640 1 -- 172.21.15.16:0/673676612 wait complete. 2024-09-12T02:47:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:37 smithi033 ceph-mon[35092]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-12T02:47:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:37 smithi033 ceph-mon[35092]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-09-12T02:47:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:37 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/673676612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:37 smithi184 ceph-mon[35321]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-12T02:47:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:37 smithi184 ceph-mon[35321]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-09-12T02:47:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:37 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/673676612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:38.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:37 smithi016 ceph-mon[30925]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-12T02:47:38.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:37 smithi016 ceph-mon[30925]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-09-12T02:47:38.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:37 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/673676612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:38.360 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-12T02:47:38.360 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-12T02:47:38.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 -- 172.21.15.16:0/2025852326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c101c20 msgr2=0x7f8d1c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2025852326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1020a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009900 tx=0x7f8d1002a410 comp rx=0 tx=0).stop 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 -- 172.21.15.16:0/2025852326 shutdown_connections 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2025852326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2025852326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1020a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2025852326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c100a30 0x7f8d1c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 -- 172.21.15.16:0/2025852326 >> 172.21.15.16:0/2025852326 conn(0x7f8d1c0fc1b0 msgr2=0x7f8d1c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 -- 172.21.15.16:0/2025852326 shutdown_connections 2024-09-12T02:47:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.472+0000 7f8d20b59640 1 -- 172.21.15.16:0/2025852326 wait complete. 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 Processor -- start 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 -- start start 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10c6b0 con 0x7f8d1c100a30 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10c530 con 0x7f8d1c1025e0 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d20b59640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10c830 con 0x7f8d1c101c20 2024-09-12T02:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36598/0 (socket says 172.21.15.16:36598) 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.473+0000 7f8d1a59c640 1 -- 172.21.15.16:0/2614669228 learned_addr learned my addr 172.21.15.16:0/2614669228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d19d9b640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1ad9d640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 -- 172.21.15.16:0/2614669228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c101c20 msgr2=0x7f8d1c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 -- 172.21.15.16:0/2614669228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 msgr2=0x7f8d1c19f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c19f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 -- 172.21.15.16:0/2614669228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c19fa90 con 0x7f8d1c100a30 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1ad9d640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c19f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d1a59c640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e9c0 tx=0x7f8d0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c00cd80 con 0x7f8d1c100a30 2024-09-12T02:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0c0035f0 con 0x7f8d1c100a30 2024-09-12T02:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c19fd80 con 0x7f8d1c100a30 2024-09-12T02:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c010690 con 0x7f8d1c100a30 2024-09-12T02:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.474+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c1a01e0 con 0x7f8d1c100a30 2024-09-12T02:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.475+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c1a0470 con 0x7f8d1c100a30 2024-09-12T02:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.477+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f8d0c0037a0 con 0x7f8d1c100a30 2024-09-12T02:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.477+0000 7f8d0b7fe640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 0x7f8d040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.478+0000 7f8d19d9b640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 0x7f8d040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.478+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8d0c0cf6e0 con 0x7f8d1c100a30 2024-09-12T02:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.478+0000 7f8d19d9b640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 0x7f8d040a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d100314e0 tx=0x7f8d10000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.481+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d0c09c580 con 0x7f8d1c100a30 2024-09-12T02:47:38.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.653+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d1c062630 con 0x7f8d040a2c20 2024-09-12T02:47:38.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.660+0000 7f8d0b7fe640 1 -- 172.21.15.16:0/2614669228 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258966 (secure 0 0 0) 0x7f8d1c062630 con 0x7f8d040a2c20 2024-09-12T02:47:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:38.668 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 msgr2=0x7f8d040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 0x7f8d040a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d100314e0 tx=0x7f8d10000bc0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 msgr2=0x7f8d1c197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e9c0 tx=0x7f8d0c00ee80 comp rx=0 tx=0).stop 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 shutdown_connections 2024-09-12T02:47:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d1c1025e0 0x7f8d1c19f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d1c101c20 0x7f8d1c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8d040a2c20 0x7f8d040a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 --2- 172.21.15.16:0/2614669228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d1c100a30 0x7f8d1c197e70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 >> 172.21.15.16:0/2614669228 conn(0x7f8d1c0fc1b0 msgr2=0x7f8d1c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 shutdown_connections 2024-09-12T02:47:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.663+0000 7f8d20b59640 1 -- 172.21.15.16:0/2614669228 wait complete. 2024-09-12T02:47:38.676 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-12T02:47:38.023817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659733,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1136,"num_read_kb":688,"num_write":443,"num_write_kb":3087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382596,"kb_used_data":12548,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559228,"statfs":{"total":1151940427776,"available":1151548649472,"internally_reserved":0,"allocated":12849152,"data_stored":7131085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1766,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":16,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002985"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.070809+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:34.070809+0000","last_peered":"2024-09-12T02:47:34.070809+0000","last_clean":"2024-09-12T02:47:34.070809+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:34.070809+0000","last_undegraded":"2024-09-12T02:47:34.070809+0000","last_fullsized":"2024-09-12T02:47:34.070809+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'5","reported_seq":219,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.068640+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:33.068640+0000","last_peered":"2024-09-12T02:47:33.068640+0000","last_clean":"2024-09-12T02:47:33.068640+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:33.068640+0000","last_undegraded":"2024-09-12T02:47:33.068640+0000","last_fullsized":"2024-09-12T02:47:33.068640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'32","reported_seq":255,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.069157+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:33.069157+0000","last_peered":"2024-09-12T02:47:33.069157+0000","last_clean":"2024-09-12T02:47:33.069157+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:33.069157+0000","last_undegraded":"2024-09-12T02:47:33.069157+0000","last_fullsized":"2024-09-12T02:47:33.069157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60013,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":201,"num_read_kb":159,"num_write":34,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.068949+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:35.068949+0000","last_peered":"2024-09-12T02:47:35.068949+0000","last_clean":"2024-09-12T02:47:35.068949+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:35.068949+0000","last_undegraded":"2024-09-12T02:47:35.068949+0000","last_fullsized":"2024-09-12T02:47:35.068949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.072121+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:35.072121+0000","last_peered":"2024-09-12T02:47:35.072121+0000","last_clean":"2024-09-12T02:47:35.072121+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:35.072121+0000","last_undegraded":"2024-09-12T02:47:35.072121+0000","last_fullsized":"2024-09-12T02:47:35.072121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":126,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.066110+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:33.066110+0000","last_peered":"2024-09-12T02:47:33.066110+0000","last_clean":"2024-09-12T02:47:33.066110+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:33.066110+0000","last_undegraded":"2024-09-12T02:47:33.066110+0000","last_fullsized":"2024-09-12T02:47:33.066110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'7","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.062664+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:33.062664+0000","last_peered":"2024-09-12T02:47:33.062664+0000","last_clean":"2024-09-12T02:47:33.062664+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:33.062664+0000","last_undegraded":"2024-09-12T02:47:33.062664+0000","last_fullsized":"2024-09-12T02:47:33.062664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.201701+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:30.201701+0000","last_peered":"2024-09-12T02:47:30.201701+0000","last_clean":"2024-09-12T02:47:30.201701+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:30.201701+0000","last_undegraded":"2024-09-12T02:47:30.201701+0000","last_fullsized":"2024-09-12T02:47:30.201701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018296+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018296+0000","last_peered":"2024-09-12T02:47:30.018296+0000","last_clean":"2024-09-12T02:47:30.018296+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018296+0000","last_undegraded":"2024-09-12T02:47:30.018296+0000","last_fullsized":"2024-09-12T02:47:30.018296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":133,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.061989+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:33.061989+0000","last_peered":"2024-09-12T02:47:33.061989+0000","last_clean":"2024-09-12T02:47:33.061989+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:33.061989+0000","last_undegraded":"2024-09-12T02:47:33.061989+0000","last_fullsized":"2024-09-12T02:47:33.061989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69131,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1010,"num_read_kb":579,"num_write":260,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":218755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1081145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234034,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462967,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051653,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280587,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":497143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:38.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T02:47:38.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 -- 172.21.15.16:0/2201733150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 msgr2=0x7fb134109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 --2- 172.21.15.16:0/2201733150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134109da0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009ca0 tx=0x7fb12c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 -- 172.21.15.16:0/2201733150 shutdown_connections 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 --2- 172.21.15.16:0/2201733150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134109da0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 --2- 172.21.15.16:0/2201733150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134102b50 0x7fb134102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 --2- 172.21.15.16:0/2201733150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134101960 0x7fb134101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 -- 172.21.15.16:0/2201733150 >> 172.21.15.16:0/2201733150 conn(0x7fb1340fd0a0 msgr2=0x7fb1340ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 -- 172.21.15.16:0/2201733150 shutdown_connections 2024-09-12T02:47:38.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.819+0000 7fb139d9f640 1 -- 172.21.15.16:0/2201733150 wait complete. 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 Processor -- start 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 -- start start 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134101960 0x7fb134075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134102b50 0x7fb1340745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb13410d630 con 0x7fb134103510 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb13410d4b0 con 0x7fb134102b50 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb139d9f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb13410d7b0 con 0x7fb134101960 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb13959e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb13959e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36620/0 (socket says 172.21.15.16:36620) 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb13959e640 1 -- 172.21.15.16:0/4040655586 learned_addr learned my addr 172.21.15.16:0/4040655586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb133fff640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134102b50 0x7fb1340745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.820+0000 7fb138d9d640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134101960 0x7fb134075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 -- 172.21.15.16:0/4040655586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134101960 msgr2=0x7fb134075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134101960 0x7fb134075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 -- 172.21.15.16:0/4040655586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134102b50 msgr2=0x7fb1340745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134102b50 0x7fb1340745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 -- 172.21.15.16:0/4040655586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1340753b0 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb13959e640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0319f0 tx=0x7fb12c02f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c003c70 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1341a1c80 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c003e10 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02fd30 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.821+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1341a21d0 con 0x7fb134103510 2024-09-12T02:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.822+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb134108c20 con 0x7fb134103510 2024-09-12T02:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.823+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fb12c004040 con 0x7fb134103510 2024-09-12T02:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.824+0000 7fb131ffb640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 0x7fb1240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.824+0000 7fb138d9d640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 0x7fb1240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.824+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb12c0f0650 con 0x7fb134103510 2024-09-12T02:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.825+0000 7fb138d9d640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 0x7fb1240a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1280058c0 tx=0x7fb128005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:38.828+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb12c0bd570 con 0x7fb134103510 2024-09-12T02:47:39.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.002+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb134062630 con 0x7fb1240a2c20 2024-09-12T02:47:39.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.009+0000 7fb131ffb640 1 -- 172.21.15.16:0/4040655586 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258966 (secure 0 0 0) 0x7fb134062630 con 0x7fb1240a2c20 2024-09-12T02:47:39.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 msgr2=0x7fb1240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 0x7fb1240a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1280058c0 tx=0x7fb128005850 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 msgr2=0x7fb134074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0319f0 tx=0x7fb12c02f450 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 shutdown_connections 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1240a2c20 0x7fb1240a50e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb134103510 0x7fb134074b20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb134102b50 0x7fb1340745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 --2- 172.21.15.16:0/4040655586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb134101960 0x7fb134075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 >> 172.21.15.16:0/4040655586 conn(0x7fb1340fd0a0 msgr2=0x7fb134105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 shutdown_connections 2024-09-12T02:47:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.012+0000 7fb139d9f640 1 -- 172.21.15.16:0/4040655586 wait complete. 2024-09-12T02:47:39.024 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-12T02:47:38.023817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659733,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1136,"num_read_kb":688,"num_write":443,"num_write_kb":3087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382596,"kb_used_data":12548,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559228,"statfs":{"total":1151940427776,"available":1151548649472,"internally_reserved":0,"allocated":12849152,"data_stored":7131085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1766,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":16,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002985"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.070809+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:34.070809+0000","last_peered":"2024-09-12T02:47:34.070809+0000","last_clean":"2024-09-12T02:47:34.070809+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:34.070809+0000","last_undegraded":"2024-09-12T02:47:34.070809+0000","last_fullsized":"2024-09-12T02:47:34.070809+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'5","reported_seq":219,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.068640+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:33.068640+0000","last_peered":"2024-09-12T02:47:33.068640+0000","last_clean":"2024-09-12T02:47:33.068640+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:33.068640+0000","last_undegraded":"2024-09-12T02:47:33.068640+0000","last_fullsized":"2024-09-12T02:47:33.068640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'32","reported_seq":255,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.069157+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:33.069157+0000","last_peered":"2024-09-12T02:47:33.069157+0000","last_clean":"2024-09-12T02:47:33.069157+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:33.069157+0000","last_undegraded":"2024-09-12T02:47:33.069157+0000","last_fullsized":"2024-09-12T02:47:33.069157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60013,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":201,"num_read_kb":159,"num_write":34,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.068949+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:35.068949+0000","last_peered":"2024-09-12T02:47:35.068949+0000","last_clean":"2024-09-12T02:47:35.068949+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:35.068949+0000","last_undegraded":"2024-09-12T02:47:35.068949+0000","last_fullsized":"2024-09-12T02:47:35.068949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.072121+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:35.072121+0000","last_peered":"2024-09-12T02:47:35.072121+0000","last_clean":"2024-09-12T02:47:35.072121+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:35.072121+0000","last_undegraded":"2024-09-12T02:47:35.072121+0000","last_fullsized":"2024-09-12T02:47:35.072121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":126,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.066110+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:33.066110+0000","last_peered":"2024-09-12T02:47:33.066110+0000","last_clean":"2024-09-12T02:47:33.066110+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:33.066110+0000","last_undegraded":"2024-09-12T02:47:33.066110+0000","last_fullsized":"2024-09-12T02:47:33.066110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'7","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.062664+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:33.062664+0000","last_peered":"2024-09-12T02:47:33.062664+0000","last_clean":"2024-09-12T02:47:33.062664+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:33.062664+0000","last_undegraded":"2024-09-12T02:47:33.062664+0000","last_fullsized":"2024-09-12T02:47:33.062664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.201701+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:30.201701+0000","last_peered":"2024-09-12T02:47:30.201701+0000","last_clean":"2024-09-12T02:47:30.201701+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:30.201701+0000","last_undegraded":"2024-09-12T02:47:30.201701+0000","last_fullsized":"2024-09-12T02:47:30.201701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018296+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018296+0000","last_peered":"2024-09-12T02:47:30.018296+0000","last_clean":"2024-09-12T02:47:30.018296+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018296+0000","last_undegraded":"2024-09-12T02:47:30.018296+0000","last_fullsized":"2024-09-12T02:47:30.018296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":133,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.061989+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:33.061989+0000","last_peered":"2024-09-12T02:47:33.061989+0000","last_clean":"2024-09-12T02:47:33.061989+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:33.061989+0000","last_undegraded":"2024-09-12T02:47:33.061989+0000","last_fullsized":"2024-09-12T02:47:33.061989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69131,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1010,"num_read_kb":579,"num_write":260,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":218755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1081145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234034,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462967,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051653,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280587,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":497143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:39.028 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-12T02:47:39.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T02:47:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1729295473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 msgr2=0x7f9ef8109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1729295473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8109ca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009ca0 tx=0x7f9ef002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1729295473 shutdown_connections 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1729295473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8109ca0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1729295473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1729295473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 0x7f9ef8104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1729295473 >> 172.21.15.16:0/1729295473 conn(0x7f9ef80fc1b0 msgr2=0x7f9ef80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1729295473 shutdown_connections 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.167+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1729295473 wait complete. 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 Processor -- start 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 -- start start 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 0x7f9ef8101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d7d0 con 0x7f9ef81078b0 2024-09-12T02:47:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d650 con 0x7f9ef8102650 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9efd9a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d950 con 0x7f9ef8104f80 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36644/0 (socket says 172.21.15.16:36644) 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 -- 172.21.15.16:0/1619301449 learned_addr learned my addr 172.21.15.16:0/1619301449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef6ffd640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 0x7f9ef8101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9eee7fc640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 -- 172.21.15.16:0/1619301449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 msgr2=0x7f9ef8100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.168+0000 7f9ef77fe640 1 -- 172.21.15.16:0/1619301449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 msgr2=0x7f9ef8101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9ef77fe640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 0x7f9ef8101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9ef77fe640 1 -- 172.21.15.16:0/1619301449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef81013d0 con 0x7f9ef81078b0 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9eee7fc640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9ef77fe640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0031a40 tx=0x7f9ef0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0003c30 con 0x7f9ef81078b0 2024-09-12T02:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef0003dd0 con 0x7f9ef81078b0 2024-09-12T02:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef81a1f80 con 0x7f9ef81078b0 2024-09-12T02:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef002fdb0 con 0x7f9ef81078b0 2024-09-12T02:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.169+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef81a23e0 con 0x7f9ef81078b0 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.170+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef80fdca0 con 0x7f9ef81078b0 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.171+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f9ef0004050 con 0x7f9ef81078b0 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.172+0000 7f9efc9a4640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 0x7f9ee40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.172+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9ef00f0510 con 0x7f9ef81078b0 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.172+0000 7f9ef6ffd640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 0x7f9ee40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.173+0000 7f9ef6ffd640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 0x7f9ee40a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800bbe0 tx=0x7f9ee800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.175+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9ef00bd430 con 0x7f9ef81078b0 2024-09-12T02:47:39.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.350+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ef8062630 con 0x7f9ee40a2c20 2024-09-12T02:47:39.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.357+0000 7f9efc9a4640 1 -- 172.21.15.16:0/1619301449 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258966 (secure 0 0 0) 0x7f9ef8062630 con 0x7f9ee40a2c20 2024-09-12T02:47:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:39.362 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:39.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.359+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 msgr2=0x7f9ee40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.359+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 0x7f9ee40a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800bbe0 tx=0x7f9ee800b390 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.359+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 msgr2=0x7f9ef8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.359+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0031a40 tx=0x7f9ef0031ee0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 shutdown_connections 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9ee40a2c20 0x7f9ee40a50e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef81078b0 0x7f9ef8100b40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ef8104f80 0x7f9ef8100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 --2- 172.21.15.16:0/1619301449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ef8102650 0x7f9ef8101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 >> 172.21.15.16:0/1619301449 conn(0x7f9ef80fc1b0 msgr2=0x7f9ef8102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 shutdown_connections 2024-09-12T02:47:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:39.360+0000 7f9efd9a6640 1 -- 172.21.15.16:0/1619301449 wait complete. 2024-09-12T02:47:39.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-12T02:47:38.023817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659733,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1136,"num_read_kb":688,"num_write":443,"num_write_kb":3087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382596,"kb_used_data":12548,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559228,"statfs":{"total":1151940427776,"available":1151548649472,"internally_reserved":0,"allocated":12849152,"data_stored":7131085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1766,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":16,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002985"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.070809+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:34.070809+0000","last_peered":"2024-09-12T02:47:34.070809+0000","last_clean":"2024-09-12T02:47:34.070809+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:34.070809+0000","last_undegraded":"2024-09-12T02:47:34.070809+0000","last_fullsized":"2024-09-12T02:47:34.070809+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'5","reported_seq":219,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.068640+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:33.068640+0000","last_peered":"2024-09-12T02:47:33.068640+0000","last_clean":"2024-09-12T02:47:33.068640+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:33.068640+0000","last_undegraded":"2024-09-12T02:47:33.068640+0000","last_fullsized":"2024-09-12T02:47:33.068640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":192,"num_read_kb":192,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'32","reported_seq":255,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.069157+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:33.069157+0000","last_peered":"2024-09-12T02:47:33.069157+0000","last_clean":"2024-09-12T02:47:33.069157+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:33.069157+0000","last_undegraded":"2024-09-12T02:47:33.069157+0000","last_fullsized":"2024-09-12T02:47:33.069157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60013,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":201,"num_read_kb":159,"num_write":34,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.068949+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:35.068949+0000","last_peered":"2024-09-12T02:47:35.068949+0000","last_clean":"2024-09-12T02:47:35.068949+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:35.068949+0000","last_undegraded":"2024-09-12T02:47:35.068949+0000","last_fullsized":"2024-09-12T02:47:35.068949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:35.072121+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:35.072121+0000","last_peered":"2024-09-12T02:47:35.072121+0000","last_clean":"2024-09-12T02:47:35.072121+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:35.072121+0000","last_undegraded":"2024-09-12T02:47:35.072121+0000","last_fullsized":"2024-09-12T02:47:35.072121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":126,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.066110+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:33.066110+0000","last_peered":"2024-09-12T02:47:33.066110+0000","last_clean":"2024-09-12T02:47:33.066110+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:33.066110+0000","last_undegraded":"2024-09-12T02:47:33.066110+0000","last_fullsized":"2024-09-12T02:47:33.066110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'7","reported_seq":128,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.062664+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:33.062664+0000","last_peered":"2024-09-12T02:47:33.062664+0000","last_clean":"2024-09-12T02:47:33.062664+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:33.062664+0000","last_undegraded":"2024-09-12T02:47:33.062664+0000","last_fullsized":"2024-09-12T02:47:33.062664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.201701+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:30.201701+0000","last_peered":"2024-09-12T02:47:30.201701+0000","last_clean":"2024-09-12T02:47:30.201701+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:30.201701+0000","last_undegraded":"2024-09-12T02:47:30.201701+0000","last_fullsized":"2024-09-12T02:47:30.201701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253073+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253073+0000","last_peered":"2024-09-12T02:46:01.253073+0000","last_clean":"2024-09-12T02:46:01.253073+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253073+0000","last_undegraded":"2024-09-12T02:46:01.253073+0000","last_fullsized":"2024-09-12T02:46:01.253073+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018296+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018296+0000","last_peered":"2024-09-12T02:47:30.018296+0000","last_clean":"2024-09-12T02:47:30.018296+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018296+0000","last_undegraded":"2024-09-12T02:47:30.018296+0000","last_fullsized":"2024-09-12T02:47:30.018296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":133,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:33.061989+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:33.061989+0000","last_peered":"2024-09-12T02:47:33.061989+0000","last_clean":"2024-09-12T02:47:33.061989+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:33.061989+0000","last_undegraded":"2024-09-12T02:47:33.061989+0000","last_fullsized":"2024-09-12T02:47:33.061989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69131,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1010,"num_read_kb":579,"num_write":260,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":218755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645347,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":429703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1081145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234034,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462967,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1023914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051653,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280587,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":497143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:39.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:40.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:39 smithi016 ceph-mon[30925]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-12T02:47:40.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:39 smithi016 ceph-mon[30925]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 -- 172.21.15.16:0/525009692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102d40 msgr2=0x7ff070103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 --2- 172.21.15.16:0/525009692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102d40 0x7ff070103120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff060009900 tx=0x7ff06002a480 comp rx=0 tx=0).stop 2024-09-12T02:47:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 -- 172.21.15.16:0/525009692 shutdown_connections 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 --2- 172.21.15.16:0/525009692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 0x7ff07010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 --2- 172.21.15.16:0/525009692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070103660 0x7ff07010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 --2- 172.21.15.16:0/525009692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102d40 0x7ff070103120 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 -- 172.21.15.16:0/525009692 >> 172.21.15.16:0/525009692 conn(0x7ff0700fc7a0 msgr2=0x7ff0700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 -- 172.21.15.16:0/525009692 shutdown_connections 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.126+0000 7ff07688e640 1 -- 172.21.15.16:0/525009692 wait complete. 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 Processor -- start 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 -- start start 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070102d40 0x7ff070198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 0x7ff07019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff070112030 con 0x7ff070103660 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff070111eb0 con 0x7ff070102d40 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff07688e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0701121b0 con 0x7ff07010c170 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff06f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff06f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36662/0 (socket says 172.21.15.16:36662) 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff06f7fe640 1 -- 172.21.15.16:0/1919381087 learned_addr learned my addr 172.21.15.16:0/1919381087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.127+0000 7ff06ffff640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070102d40 0x7ff070198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff074e2b640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 0x7ff07019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 -- 172.21.15.16:0/1919381087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 msgr2=0x7ff07019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 0x7ff07019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 -- 172.21.15.16:0/1919381087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070102d40 msgr2=0x7ff070198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070102d40 0x7ff070198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 -- 172.21.15.16:0/1919381087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07019d5c0 con 0x7ff070103660 2024-09-12T02:47:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06f7fe640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff06400b3f0 tx=0x7ff06400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff064015020 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff064011d10 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07019d8b0 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06400f2e0 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.128+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0701a5190 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.129+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0701031a0 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.130+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7ff064018020 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.131+0000 7ff06d7fa640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 0x7ff0580a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.131+0000 7ff06ffff640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 0x7ff0580a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.131+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff0640ce860 con 0x7ff070103660 2024-09-12T02:47:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.132+0000 7ff06ffff640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 0x7ff0580a5150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff060000c00 tx=0x7ff0600058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.136+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff06401f020 con 0x7ff070103660 2024-09-12T02:47:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:39 smithi033 ceph-mon[35092]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-12T02:47:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:39 smithi033 ceph-mon[35092]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:39 smithi184 ceph-mon[35321]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-09-12T02:47:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:39 smithi184 ceph-mon[35321]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:40.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.351+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff070107270 con 0x7ff070103660 2024-09-12T02:47:40.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.352+0000 7ff06d7fa640 1 -- 172.21.15.16:0/1919381087 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7ff06409b780 con 0x7ff070103660 2024-09-12T02:47:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:40.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:47:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 msgr2=0x7ff0580a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 0x7ff0580a5150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff060000c00 tx=0x7ff0600058d0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 msgr2=0x7ff070198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff06400b3f0 tx=0x7ff06400b8b0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 shutdown_connections 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff07010c170 0x7ff07019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0580a2c90 0x7ff0580a5150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070103660 0x7ff070198b50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 --2- 172.21.15.16:0/1919381087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff070102d40 0x7ff070198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 >> 172.21.15.16:0/1919381087 conn(0x7ff0700fc7a0 msgr2=0x7ff0700fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 shutdown_connections 2024-09-12T02:47:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:40.354+0000 7ff07688e640 1 -- 172.21.15.16:0/1919381087 wait complete. 2024-09-12T02:47:41.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:40 smithi016 ceph-mon[30925]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:40 smithi016 ceph-mon[30925]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:40 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1919381087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:41.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:47:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:40 smithi033 ceph-mon[35092]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:40 smithi033 ceph-mon[35092]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:40 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1919381087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:40 smithi184 ceph-mon[35321]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:40 smithi184 ceph-mon[35321]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:41.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:40 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1919381087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:41.402 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/1206182189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f74102d40 msgr2=0x7f1f74103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/1206182189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f74102d40 0x7f1f74103120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009900 tx=0x7f1f6402a480 comp rx=0 tx=0).stop 2024-09-12T02:47:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/1206182189 shutdown_connections 2024-09-12T02:47:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/1206182189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f7410c170 0x7f1f7410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/1206182189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 0x7f1f7410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/1206182189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f74102d40 0x7f1f74103120 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/1206182189 >> 172.21.15.16:0/1206182189 conn(0x7f1f740fc7a0 msgr2=0x7f1f740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/1206182189 shutdown_connections 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.858+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/1206182189 wait complete. 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 Processor -- start 2024-09-12T02:47:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 -- start start 2024-09-12T02:47:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f74102d40 0x7f1f741009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 0x7f1f74100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74111f70 con 0x7f1f7410c170 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74111df0 con 0x7f1f74102d40 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f7aaa7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f741120f0 con 0x7f1f74103660 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f79044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f79044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36672/0 (socket says 172.21.15.16:36672) 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.859+0000 7f1f79044640 1 -- 172.21.15.16:0/501645576 learned_addr learned my addr 172.21.15.16:0/501645576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f73fff640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 0x7f1f74100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f78843640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f74102d40 0x7f1f741009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 -- 172.21.15.16:0/501645576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 msgr2=0x7f1f74100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 0x7f1f74100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 -- 172.21.15.16:0/501645576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f74102d40 msgr2=0x7f1f741009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f74102d40 0x7f1f741009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 -- 172.21.15.16:0/501645576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f741a68f0 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f79044640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e970 tx=0x7f1f6c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00ccc0 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c00ce60 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f741a6be0 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c010490 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.860+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f741a7120 con 0x7f1f7410c170 2024-09-12T02:47:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.861+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f74103120 con 0x7f1f7410c170 2024-09-12T02:47:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.863+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f1f6c005020 con 0x7f1f7410c170 2024-09-12T02:47:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.863+0000 7f1f71ffb640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 0x7f1f5c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.863+0000 7f1f78843640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 0x7f1f5c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.863+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f6c0cf540 con 0x7f1f7410c170 2024-09-12T02:47:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.864+0000 7f1f78843640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 0x7f1f5c0a5150 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009230 tx=0x7f1f640058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:41.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:41.867+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1f6c09c350 con 0x7f1f7410c170 2024-09-12T02:47:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.045+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f741071e0 con 0x7f1f7410c170 2024-09-12T02:47:42.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:41 smithi016 ceph-mon[30925]: pgmap v330: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-12T02:47:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.047+0000 7f1f71ffb640 1 -- 172.21.15.16:0/501645576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f1f6c09bc50 con 0x7f1f7410c170 2024-09-12T02:47:42.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:42.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:47:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 msgr2=0x7f1f5c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 0x7f1f5c0a5150 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009230 tx=0x7f1f640058d0 comp rx=0 tx=0).stop 2024-09-12T02:47:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 msgr2=0x7f1f74101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e970 tx=0x7f1f6c00ed70 comp rx=0 tx=0).stop 2024-09-12T02:47:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 shutdown_connections 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f5c0a2c90 0x7f1f5c0a5150 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7410c170 0x7f1f74101420 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74103660 0x7f1f74100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 --2- 172.21.15.16:0/501645576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f74102d40 0x7f1f741009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 >> 172.21.15.16:0/501645576 conn(0x7f1f740fc7a0 msgr2=0x7f1f740fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 shutdown_connections 2024-09-12T02:47:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:42.049+0000 7f1f7aaa7640 1 -- 172.21.15.16:0/501645576 wait complete. 2024-09-12T02:47:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:41 smithi033 ceph-mon[35092]: pgmap v330: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-12T02:47:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:41 smithi184 ceph-mon[35321]: pgmap v330: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-12T02:47:42.845 DEBUG:tasks.fwd_scrub:Yielding 2024-09-12T02:47:42.845 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-12T02:47:42.859 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-12T02:47:42.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:47:42.860 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-12T02:47:42.860 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:47:42.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:47:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:43.143 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:43.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:42 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/501645576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:43.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:42 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/501645576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:42 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/501645576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 -- 172.21.15.16:0/1863887775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 msgr2=0x7f578810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 --2- 172.21.15.16:0/1863887775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f578810ea90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5780009ca0 tx=0x7f578002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 -- 172.21.15.16:0/1863887775 shutdown_connections 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 --2- 172.21.15.16:0/1863887775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f578810ea90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 --2- 172.21.15.16:0/1863887775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788102790 0x7f57881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 --2- 172.21.15.16:0/1863887775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788101de0 0x7f57881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.731+0000 7f579008c640 1 -- 172.21.15.16:0/1863887775 >> 172.21.15.16:0/1863887775 conn(0x7f57880fd8c0 msgr2=0x7f57880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.732+0000 7f579008c640 1 -- 172.21.15.16:0/1863887775 shutdown_connections 2024-09-12T02:47:43.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.732+0000 7f579008c640 1 -- 172.21.15.16:0/1863887775 wait complete. 2024-09-12T02:47:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.732+0000 7f579008c640 1 Processor -- start 2024-09-12T02:47:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.732+0000 7f579008c640 1 -- start start 2024-09-12T02:47:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.732+0000 7f579008c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788101de0 0x7f57880744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f579008c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788102790 0x7f5788074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f579008c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f579008c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5788111f70 con 0x7f5788106f10 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f579008c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5788111df0 con 0x7f5788101de0 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f579008c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57881120f0 con 0x7f5788102790 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36702/0 (socket says 172.21.15.16:36702) 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 -- 172.21.15.16:0/2155416862 learned_addr learned my addr 172.21.15.16:0/2155416862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578d627640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788102790 0x7f5788074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578de28640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788101de0 0x7f57880744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 -- 172.21.15.16:0/2155416862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788102790 msgr2=0x7f5788074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788102790 0x7f5788074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 -- 172.21.15.16:0/2155416862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788101de0 msgr2=0x7f57880744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788101de0 0x7f57880744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 -- 172.21.15.16:0/2155416862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57881a6890 con 0x7f5788106f10 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f578e629640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5780031a20 tx=0x7f5780031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.733+0000 7f577effd640 1 -- 172.21.15.16:0/2155416862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780003c30 con 0x7f5788106f10 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.734+0000 7f577effd640 1 -- 172.21.15.16:0/2155416862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5780003dd0 con 0x7f5788106f10 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.734+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57881a6b20 con 0x7f5788106f10 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.734+0000 7f577effd640 1 -- 172.21.15.16:0/2155416862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578002fc40 con 0x7f5788106f10 2024-09-12T02:47:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.734+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57881a70b0 con 0x7f5788106f10 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.736+0000 7f577effd640 1 -- 172.21.15.16:0/2155416862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f5780004050 con 0x7f5788106f10 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.736+0000 7f5792465740 1 Processor -- start 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.736+0000 7f5792465740 1 -- start start 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a31f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1a1997280 con 0x55b1a1a32480 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b1a193eb10 con 0x55b1a1a31b60 2024-09-12T02:47:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f5792465740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b1a1993240 con 0x55b1a1a3cd40 2024-09-12T02:47:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f577effd640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 0x7f57740a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578e629640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 0x7f57740a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578d627640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36708/0 (socket says 172.21.15.16:36708) 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 -- 172.21.15.16:0/976891242 learned_addr learned my addr 172.21.15.16:0/976891242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578d627640 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a31f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f577effd640 1 -- 172.21.15.16:0/2155416862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f57800f0380 con 0x7f5788106f10 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 -- 172.21.15.16:0/976891242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 msgr2=0x55b1a1a3f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 -- 172.21.15.16:0/976891242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 msgr2=0x55b1a1a31f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a31f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578de28640 1 -- 172.21.15.16:0/976891242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a194b1f0 con 0x55b1a1a32480 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.737+0000 7f578e629640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 0x7f57740a5150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57800b9600 tx=0x7f578002a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f578de28640 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5784009d20 tx=0x7f5784011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923ef8777f5b5a99 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f5773fff640 1 -- 172.21.15.16:0/976891242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578401e070 con 0x55b1a1a32480 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f5773fff640 1 -- 172.21.15.16:0/976891242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5784019b10 con 0x55b1a1a32480 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f5773fff640 1 -- 172.21.15.16:0/976891242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578400eb20 con 0x55b1a1a32480 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f5792465740 1 -- 172.21.15.16:0/976891242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 msgr2=0x55b1a1a3c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.738+0000 7f5792465740 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5784009d20 tx=0x7f5784011810 comp rx=0 tx=0).stop 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 -- 172.21.15.16:0/976891242 shutdown_connections 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a32480 0x55b1a1a3c800 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 --2- 172.21.15.16:0/976891242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a31f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 -- 172.21.15.16:0/976891242 >> 172.21.15.16:0/976891242 conn(0x55b1a1a2e0f0 msgr2=0x55b1a1a2e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 -- 172.21.15.16:0/976891242 shutdown_connections 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 -- 172.21.15.16:0/976891242 wait complete. 2024-09-12T02:47:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 -- start start 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 Processor -- start 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a39bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.739+0000 7f5792465740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a3a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1a1a3f910 con 0x55b1a1a3cd40 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b1a1951f10 con 0x55b1a1a31b60 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b1a1a3fa90 con 0x55b1a1a32480 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 10 client.0 _ll_register_callbacks cb 0x55b1a191aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36710/0 (socket says 172.21.15.16:36710) 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 -- 172.21.15.16:0/2548952240 learned_addr learned my addr 172.21.15.16:0/2548952240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578de28640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a3a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a39bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 -- 172.21.15.16:0/2548952240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 msgr2=0x55b1a1a3a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a3a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 -- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 msgr2=0x55b1a1a39bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a39bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 -- 172.21.15.16:0/2548952240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a17086a0 con 0x55b1a1a3cd40 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f578d627640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5778012a20 tx=0x7f57780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab6e64c90ba8453 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f57717fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778010be0 con 0x55b1a1a3cd40 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5778010d80 con 0x55b1a1a3cd40 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.740+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5784019b10 con 0x55b1a1a3cd40 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5778004030 con 0x55b1a1a3cd40 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 10 client.15276 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1a1a4b4f0 con 0x55b1a1a3cd40 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5784019b10 con 0x55b1a1a3cd40 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5778020400 con 0x55b1a1a3cd40 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 10 client.15276 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 10 client.15276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 20 client.15276 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1a4c900 0x55b1a1a4ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 4 client.15276 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.741+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1a19a47f0 con 0x55b1a1a4c900 2024-09-12T02:47:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.742+0000 7f578de28640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1a4c900 0x55b1a1a4ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.742+0000 7f578de28640 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1a4c900 0x55b1a1a4ed10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b2795c0db278bc server_cookie=4bdbf4cc7bdabd06 in_seq=0 out_seq=0 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.742+0000 7f57717fa640 10 client.15276 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f57717fa640 1 -- 172.21.15.16:0/2548952240 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b1a19a47f0 con 0x55b1a1a4c900 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f57717fa640 10 client.15276 handle_command_reply: tid=0 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 client.15276 shutdown 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 msgr2=0x55b1a1a3a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5778012a20 tx=0x7f57780046c0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 shutdown_connections 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1a4c900 0x55b1a1a4ed10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a3a650 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a3a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 --2- 172.21.15.16:0/2548952240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a31b60 0x55b1a1a39bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 >> 172.21.15.16:0/2548952240 conn(0x55b1a1a2e0f0 msgr2=0x55b1a1a2d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f57717fa640 0 client.15276 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 shutdown_connections 2024-09-12T02:47:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 1 -- 172.21.15.16:0/2548952240 wait complete. 2024-09-12T02:47:43.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.743+0000 7f5792465740 20 client.15276 trim_cache size 0 max 16384 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.760+0000 7f5792465740 1 Processor -- start 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 -- start start 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 0x55b1a1a545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a54b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57881af6c0 con 0x55b1a1a31b60 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57881b1200 con 0x55b1a1a32480 2024-09-12T02:47:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f5792465740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57881afd90 con 0x55b1a1a3cd40 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36718/0 (socket says 172.21.15.16:36718) 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 -- 172.21.15.16:0/2262891609 learned_addr learned my addr 172.21.15.16:0/2262891609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578de28640 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 0x55b1a1a545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 -- 172.21.15.16:0/2262891609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 msgr2=0x55b1a1a54b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a54b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 -- 172.21.15.16:0/2262891609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 msgr2=0x55b1a1a545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 0x55b1a1a545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 -- 172.21.15.16:0/2262891609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a1a322b0 con 0x55b1a1a31b60 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578de28640 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 0x55b1a1a545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.761+0000 7f578d627640 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f57780129f0 tx=0x7f57780126e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc97df463b65a95 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f57737fe640 1 -- 172.21.15.16:0/2262891609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778005850 con 0x55b1a1a31b60 2024-09-12T02:47:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f57737fe640 1 -- 172.21.15.16:0/2262891609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f577802f070 con 0x55b1a1a31b60 2024-09-12T02:47:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f57737fe640 1 -- 172.21.15.16:0/2262891609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778010be0 con 0x55b1a1a31b60 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 -- 172.21.15.16:0/2262891609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 msgr2=0x55b1a1a3ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f57780129f0 tx=0x7f57780126e0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 -- 172.21.15.16:0/2262891609 shutdown_connections 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a54b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a32480 0x55b1a1a545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 --2- 172.21.15.16:0/2262891609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a3ba30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 -- 172.21.15.16:0/2262891609 >> 172.21.15.16:0/2262891609 conn(0x55b1a1a2e0f0 msgr2=0x55b1a1a3b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 -- 172.21.15.16:0/2262891609 shutdown_connections 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.762+0000 7f5792465740 1 -- 172.21.15.16:0/2262891609 wait complete. 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 -- start start 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 Processor -- start 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a41800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57881b0b30 con 0x55b1a1a31b60 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57881b0460 con 0x55b1a1a3cd40 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57881ad6a0 con 0x55b1a1a32480 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f5792465740 10 client.0 _ll_register_callbacks cb 0x7f5778012630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f578d627640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36720/0 (socket says 172.21.15.16:36720) 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f578d627640 1 -- 172.21.15.16:0/2193965709 learned_addr learned my addr 172.21.15.16:0/2193965709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.763+0000 7f578de28640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 -- 172.21.15.16:0/2193965709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 msgr2=0x55b1a1a412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 -- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a3cd40 msgr2=0x55b1a1a41800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a41800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 -- 172.21.15.16:0/2193965709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a1ab6880 con 0x55b1a1a31b60 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578de28640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f578d627640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5778012a20 tx=0x7f5778005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee8b2d0c2c66dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f57717fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778012070 con 0x55b1a1a31b60 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5778010be0 con 0x55b1a1a31b60 2024-09-12T02:47:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f577802f070 con 0x55b1a1a31b60 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5778012210 con 0x55b1a1a31b60 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f5792465740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1a1ab7b30 con 0x55b1a1a31b60 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.764+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5778038030 con 0x55b1a1a31b60 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5778034460 con 0x55b1a1a31b60 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 10 client.15282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 20 client.15282 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1ab81a0 0x55b1a1aba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 4 client.15282 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f577802f070 con 0x55b1a1ab81a0 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f578de28640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1ab81a0 0x55b1a1aba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f578de28640 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1ab81a0 0x55b1a1aba590 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2832c5ae99d61c79 server_cookie=7edc938a0b444fe9 in_seq=0 out_seq=0 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.765+0000 7f57717fa640 10 client.15282 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.766+0000 7f57717fa640 1 -- 172.21.15.16:0/2193965709 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f577802f070 con 0x55b1a1ab81a0 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.766+0000 7f57717fa640 10 client.15282 handle_command_reply: tid=0 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.766+0000 7f5792465740 1 client.15282 shutdown 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.766+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 msgr2=0x55b1a1a40d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.766+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5778012a20 tx=0x7f5778005850 comp rx=0 tx=0).stop 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 shutdown_connections 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b1a1ab81a0 0x55b1a1aba590 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b1a1a3cd40 0x55b1a1a41800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b1a1a32480 0x55b1a1a412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 --2- 172.21.15.16:0/2193965709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1a1a31b60 0x55b1a1a40d60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f57717fa640 0 client.15282 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 >> 172.21.15.16:0/2193965709 conn(0x55b1a1a2e0f0 msgr2=0x55b1a1a3a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 shutdown_connections 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 1 -- 172.21.15.16:0/2193965709 wait complete. 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f5792465740 20 client.15282 trim_cache size 0 max 16384 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f16383d4-abec-4f3b-b65b-13bfec4ad2f3", 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:47:43.773 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 msgr2=0x7f57740a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 0x7f57740a5150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57800b9600 tx=0x7f578002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 msgr2=0x7f5788074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5780031a20 tx=0x7f5780031a50 comp rx=0 tx=0).stop 2024-09-12T02:47:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 shutdown_connections 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57740a2c90 0x7f57740a5150 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5788106f10 0x7f5788074f70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5788102790 0x7f5788074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 --2- 172.21.15.16:0/2155416862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5788101de0 0x7f57880744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 >> 172.21.15.16:0/2155416862 conn(0x7f57880fd8c0 msgr2=0x7f57880feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 shutdown_connections 2024-09-12T02:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.767+0000 7f579008c640 1 -- 172.21.15.16:0/2155416862 wait complete. 2024-09-12T02:47:44.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:43 smithi016 ceph-mon[30925]: pgmap v331: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T02:47:44.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:43 smithi184 ceph-mon[35321]: pgmap v331: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T02:47:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:43 smithi033 ceph-mon[35092]: pgmap v331: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T02:47:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 -- 172.21.15.16:0/931495656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c106f10 msgr2=0x7f882c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 --2- 172.21.15.16:0/931495656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c106f10 0x7f882c10ea90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8824009ca0 tx=0x7f882402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 -- 172.21.15.16:0/931495656 shutdown_connections 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 --2- 172.21.15.16:0/931495656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c106f10 0x7f882c10ea90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 --2- 172.21.15.16:0/931495656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c102790 0x7f882c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 --2- 172.21.15.16:0/931495656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c101de0 0x7f882c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 -- 172.21.15.16:0/931495656 >> 172.21.15.16:0/931495656 conn(0x7f882c0fd8c0 msgr2=0x7f882c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 -- 172.21.15.16:0/931495656 shutdown_connections 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.937+0000 7f8831df3640 1 -- 172.21.15.16:0/931495656 wait complete. 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 Processor -- start 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 -- start start 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c101de0 0x7f882c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c106f10 0x7f882c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f882c111f70 con 0x7f882c102790 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f882c111df0 con 0x7f882c101de0 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f8831df3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f882c1120f0 con 0x7f882c106f10 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f882affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f882affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36738/0 (socket says 172.21.15.16:36738) 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f882affd640 1 -- 172.21.15.16:0/1019423237 learned_addr learned my addr 172.21.15.16:0/1019423237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.938+0000 7f882bfff640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c106f10 0x7f882c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882b7fe640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c101de0 0x7f882c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 -- 172.21.15.16:0/1019423237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c106f10 msgr2=0x7f882c19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c106f10 0x7f882c19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 -- 172.21.15.16:0/1019423237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c101de0 msgr2=0x7f882c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c101de0 0x7f882c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 -- 172.21.15.16:0/1019423237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c19d380 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f882affd640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f882000e9f0 tx=0x7f882000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882000cd80 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8820010910 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882c19d670 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820018990 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.939+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882c1a4f50 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.940+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882c1021c0 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.941+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f8820010ac0 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.942+0000 7f8828ff9640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 0x7f88140a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.942+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f88200cf450 con 0x7f882c102790 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.942+0000 7f882b7fe640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 0x7f88140a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.943+0000 7f882b7fe640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 0x7f88140a5150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f881c009b60 tx=0x7f881c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:43.946+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f882009c260 con 0x7f882c102790 2024-09-12T02:47:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.121+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f882c062630 con 0x7f88140a2c90 2024-09-12T02:47:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.127+0000 7f8828ff9640 1 -- 172.21.15.16:0/1019423237 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258896 (secure 0 0 0) 0x7f882c062630 con 0x7f88140a2c90 2024-09-12T02:47:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 msgr2=0x7f88140a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 0x7f88140a5150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f881c009b60 tx=0x7f881c005f90 comp rx=0 tx=0).stop 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 msgr2=0x7f882c198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f882000e9f0 tx=0x7f882000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 shutdown_connections 2024-09-12T02:47:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f882c106f10 0x7f882c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f88140a2c90 0x7f88140a5150 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f882c102790 0x7f882c198910 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 --2- 172.21.15.16:0/1019423237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f882c101de0 0x7f882c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 >> 172.21.15.16:0/1019423237 conn(0x7f882c0fd8c0 msgr2=0x7f882c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 shutdown_connections 2024-09-12T02:47:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:44.270+0000 7f8831df3640 1 -- 172.21.15.16:0/1019423237 wait complete. 2024-09-12T02:47:45.020 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-12T02:47:44.025242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668720,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1212,"num_read_kb":741,"num_write":452,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382612,"kb_used_data":12564,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559212,"statfs":{"total":1151940427776,"available":1151548633088,"internally_reserved":0,"allocated":12865536,"data_stored":7146684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":82,"num_write":23,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:38.076957+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:38.076957+0000","last_peered":"2024-09-12T02:47:38.076957+0000","last_clean":"2024-09-12T02:47:38.076957+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:38.076957+0000","last_undegraded":"2024-09-12T02:47:38.076957+0000","last_fullsized":"2024-09-12T02:47:38.076957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":240,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083940+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:43.083940+0000","last_peered":"2024-09-12T02:47:43.083940+0000","last_clean":"2024-09-12T02:47:43.083940+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:43.083940+0000","last_undegraded":"2024-09-12T02:47:43.083940+0000","last_fullsized":"2024-09-12T02:47:43.083940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":212,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'35","reported_seq":281,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083969+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:43.083969+0000","last_peered":"2024-09-12T02:47:43.083969+0000","last_clean":"2024-09-12T02:47:43.083969+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:43.083969+0000","last_undegraded":"2024-09-12T02:47:43.083969+0000","last_fullsized":"2024-09-12T02:47:43.083969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68607,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":179,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":138,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:39.074367+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:39.074367+0000","last_peered":"2024-09-12T02:47:39.074367+0000","last_clean":"2024-09-12T02:47:39.074367+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:39.074367+0000","last_undegraded":"2024-09-12T02:47:39.074367+0000","last_fullsized":"2024-09-12T02:47:39.074367+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":134,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:39.078475+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:39.078475+0000","last_peered":"2024-09-12T02:47:39.078475+0000","last_clean":"2024-09-12T02:47:39.078475+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:39.078475+0000","last_undegraded":"2024-09-12T02:47:39.078475+0000","last_fullsized":"2024-09-12T02:47:39.078475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250079+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250079+0000","last_peered":"2024-09-12T02:46:01.250079+0000","last_clean":"2024-09-12T02:46:01.250079+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250079+0000","last_undegraded":"2024-09-12T02:46:01.250079+0000","last_fullsized":"2024-09-12T02:46:01.250079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":131,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:38.072524+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:38.072524+0000","last_peered":"2024-09-12T02:47:38.072524+0000","last_clean":"2024-09-12T02:47:38.072524+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:38.072524+0000","last_undegraded":"2024-09-12T02:47:38.072524+0000","last_fullsized":"2024-09-12T02:47:38.072524+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'8","reported_seq":135,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:38.070459+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:38.070459+0000","last_peered":"2024-09-12T02:47:38.070459+0000","last_clean":"2024-09-12T02:47:38.070459+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:38.070459+0000","last_undegraded":"2024-09-12T02:47:38.070459+0000","last_fullsized":"2024-09-12T02:47:38.070459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'18","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.202422+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:34.202422+0000","last_peered":"2024-09-12T02:47:34.202422+0000","last_clean":"2024-09-12T02:47:34.202422+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:34.202422+0000","last_undegraded":"2024-09-12T02:47:34.202422+0000","last_fullsized":"2024-09-12T02:47:34.202422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253128+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253128+0000","last_peered":"2024-09-12T02:46:01.253128+0000","last_clean":"2024-09-12T02:46:01.253128+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253128+0000","last_undegraded":"2024-09-12T02:46:01.253128+0000","last_fullsized":"2024-09-12T02:46:01.253128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018296+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018296+0000","last_peered":"2024-09-12T02:47:30.018296+0000","last_clean":"2024-09-12T02:47:30.018296+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018296+0000","last_undegraded":"2024-09-12T02:47:30.018296+0000","last_fullsized":"2024-09-12T02:47:30.018296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":138,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:38.069603+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:38.069603+0000","last_peered":"2024-09-12T02:47:38.069603+0000","last_clean":"2024-09-12T02:47:38.069603+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:38.069603+0000","last_undegraded":"2024-09-12T02:47:38.069603+0000","last_fullsized":"2024-09-12T02:47:38.069603+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78118,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1086,"num_read_kb":632,"num_write":269,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":234354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":430096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874281,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1090132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005102,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234036,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462968,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1024700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659197,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888130,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051654,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280588,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":500191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607638,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":499663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:45.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:47:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:44 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:47:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:47:45.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:44 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:47:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:47:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:44 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:47:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:47:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:47:45.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:47:45.527 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 -- 172.21.15.16:0/3986020858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8105de0 msgr2=0x7f36f810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 --2- 172.21.15.16:0/3986020858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8105de0 0x7f36f810d960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009ca0 tx=0x7f36f002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 -- 172.21.15.16:0/3986020858 shutdown_connections 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 --2- 172.21.15.16:0/3986020858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8105de0 0x7f36f810d960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 --2- 172.21.15.16:0/3986020858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101e20 0x7f36f81056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 --2- 172.21.15.16:0/3986020858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8101470 0x7f36f8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 -- 172.21.15.16:0/3986020858 >> 172.21.15.16:0/3986020858 conn(0x7f36f8077230 msgr2=0x7f36f80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 -- 172.21.15.16:0/3986020858 shutdown_connections 2024-09-12T02:47:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.943+0000 7f36ffa39640 1 -- 172.21.15.16:0/3986020858 wait complete. 2024-09-12T02:47:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 Processor -- start 2024-09-12T02:47:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 -- start start 2024-09-12T02:47:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101470 0x7f36f80fc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8105de0 0x7f36f80fd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8110e40 con 0x7f36f8101e20 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8110cc0 con 0x7f36f8101470 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36ffa39640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8110fc0 con 0x7f36f8105de0 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57964/0 (socket says 172.21.15.16:57964) 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 -- 172.21.15.16:0/754052287 learned_addr learned my addr 172.21.15.16:0/754052287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fd7d5640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101470 0x7f36f80fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fdfd6640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8105de0 0x7f36f80fd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 -- 172.21.15.16:0/754052287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8105de0 msgr2=0x7f36f80fd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8105de0 0x7f36f80fd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 -- 172.21.15.16:0/754052287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101470 msgr2=0x7f36f80fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.944+0000 7f36fcfd4640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101470 0x7f36f80fc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36fcfd4640 1 -- 172.21.15.16:0/754052287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f81a6850 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36fcfd4640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36e800d950 tx=0x7f36e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8017030 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e8004020 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f81a6b40 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e80047a0 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.945+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f81a70d0 con 0x7f36f8101e20 2024-09-12T02:47:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.946+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f8101850 con 0x7f36f8101e20 2024-09-12T02:47:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.948+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f36e80041d0 con 0x7f36f8101e20 2024-09-12T02:47:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.948+0000 7f36ee7fc640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 0x7f36e40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.949+0000 7f36fd7d5640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 0x7f36e40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.949+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f36e80cea90 con 0x7f36f8101e20 2024-09-12T02:47:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.949+0000 7f36fd7d5640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 0x7f36e40a5150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009bd0 tx=0x7f36f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:45.953+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f36e809b8a0 con 0x7f36f8101e20 2024-09-12T02:47:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:46.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:45 smithi016 ceph-mon[30925]: pgmap v332: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-12T02:47:46.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:45 smithi016 ceph-mon[30925]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:46.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.132+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36f8062630 con 0x7f36e40a2c90 2024-09-12T02:47:46.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.138+0000 7f36ee7fc640 1 -- 172.21.15.16:0/754052287 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258896 (secure 0 0 0) 0x7f36f8062630 con 0x7f36e40a2c90 2024-09-12T02:47:46.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:46.143 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 msgr2=0x7f36e40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 0x7f36e40a5150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009bd0 tx=0x7f36f4005f90 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 msgr2=0x7f36f80fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36e800d950 tx=0x7f36e800de10 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 shutdown_connections 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36f8105de0 0x7f36f80fd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36e40a2c90 0x7f36e40a5150 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f8101e20 0x7f36f80fcc40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 --2- 172.21.15.16:0/754052287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36f8101470 0x7f36f80fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 >> 172.21.15.16:0/754052287 conn(0x7f36f8077230 msgr2=0x7f36f8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 shutdown_connections 2024-09-12T02:47:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.141+0000 7f36ffa39640 1 -- 172.21.15.16:0/754052287 wait complete. 2024-09-12T02:47:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:45 smithi033 ceph-mon[35092]: pgmap v332: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-12T02:47:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:45 smithi033 ceph-mon[35092]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:45 smithi184 ceph-mon[35321]: pgmap v332: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-12T02:47:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:45 smithi184 ceph-mon[35321]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 -- 172.21.15.16:0/425622689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c103660 msgr2=0x7f981c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 --2- 172.21.15.16:0/425622689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c103660 0x7f981c10bb50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9804009900 tx=0x7f980402a480 comp rx=0 tx=0).stop 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 -- 172.21.15.16:0/425622689 shutdown_connections 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 --2- 172.21.15.16:0/425622689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c10c170 0x7f981c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 --2- 172.21.15.16:0/425622689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c103660 0x7f981c10bb50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 --2- 172.21.15.16:0/425622689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c102d40 0x7f981c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 -- 172.21.15.16:0/425622689 >> 172.21.15.16:0/425622689 conn(0x7f981c0fc7a0 msgr2=0x7f981c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 -- 172.21.15.16:0/425622689 shutdown_connections 2024-09-12T02:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.684+0000 7f982255b640 1 -- 172.21.15.16:0/425622689 wait complete. 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 Processor -- start 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 -- start start 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 0x7f981c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 0x7f981c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f981c111fc0 con 0x7f981c102d40 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f981c111e40 con 0x7f981c10c170 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f982255b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f981c112140 con 0x7f981c103660 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57974/0 (socket says 172.21.15.16:57974) 2024-09-12T02:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 -- 172.21.15.16:0/1693388477 learned_addr learned my addr 172.21.15.16:0/1693388477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f9820af8640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 0x7f981c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f9813fff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 0x7f981c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 -- 172.21.15.16:0/1693388477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 msgr2=0x7f981c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 0x7f981c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 -- 172.21.15.16:0/1693388477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 msgr2=0x7f981c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 0x7f981c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f981bfff640 1 -- 172.21.15.16:0/1693388477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981c19d3c0 con 0x7f981c102d40 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f9820af8640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 0x7f981c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.685+0000 7f9813fff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 0x7f981c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f981bfff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f980c00c920 tx=0x7f980c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f9819ffb640 1 -- 172.21.15.16:0/1693388477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c007c70 con 0x7f981c102d40 2024-09-12T02:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981c19d6b0 con 0x7f981c102d40 2024-09-12T02:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f9819ffb640 1 -- 172.21.15.16:0/1693388477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f980c007e10 con 0x7f981c102d40 2024-09-12T02:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f9819ffb640 1 -- 172.21.15.16:0/1693388477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c018890 con 0x7f981c102d40 2024-09-12T02:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.686+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f981c1008d0 con 0x7f981c102d40 2024-09-12T02:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.688+0000 7f9824934740 1 Processor -- start 2024-09-12T02:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.688+0000 7f9819ffb640 1 -- 172.21.15.16:0/1693388477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f980c018a50 con 0x7f981c102d40 2024-09-12T02:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.688+0000 7f9824934740 1 -- start start 2024-09-12T02:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c04140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c06a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561973b5f2e0 con 0x561973bf94a0 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561973b06ce0 con 0x561973c04680 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9824934740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561973b5b2a0 con 0x561973bf9dc0 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57982/0 (socket says 172.21.15.16:57982) 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 -- 172.21.15.16:0/395728490 learned_addr learned my addr 172.21.15.16:0/395728490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f981bfff640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c04140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9813fff640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c06a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 -- 172.21.15.16:0/395728490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 msgr2=0x561973c04140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c04140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 -- 172.21.15.16:0/395728490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 msgr2=0x561973c06a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c06a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 -- 172.21.15.16:0/395728490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561973aeb440 con 0x561973bf94a0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9813fff640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c06a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9820af8640 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9814007290 tx=0x7f9814007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ded59801b6e220 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.689+0000 7f9819ffb640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f98127fc640 1 -- 172.21.15.16:0/395728490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981401c070 con 0x561973bf94a0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f981bfff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f98127fc640 1 -- 172.21.15.16:0/395728490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9814015bf0 con 0x561973bf94a0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f98127fc640 1 -- 172.21.15.16:0/395728490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814017d50 con 0x561973bf94a0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f9819ffb640 1 -- 172.21.15.16:0/1693388477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f980c0cf3b0 con 0x7f981c102d40 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f9824934740 1 -- 172.21.15.16:0/395728490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 msgr2=0x561973bf9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f981bfff640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 0x7f98080a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x561973bf6890 tx=0x7f980c0d5060 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.690+0000 7f9824934740 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9814007290 tx=0x7f9814007750 comp rx=0 tx=0).stop 2024-09-12T02:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 -- 172.21.15.16:0/395728490 shutdown_connections 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c06a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c04140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 --2- 172.21.15.16:0/395728490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973bf9880 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 -- 172.21.15.16:0/395728490 >> 172.21.15.16:0/395728490 conn(0x561973bf5a30 msgr2=0x561973bf5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 -- 172.21.15.16:0/395728490 shutdown_connections 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 -- 172.21.15.16:0/395728490 wait complete. 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 -- start start 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.691+0000 7f9824934740 1 Processor -- start 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c01270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561973c07250 con 0x561973bf94a0 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561973b1a050 con 0x561973c04680 2024-09-12T02:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561973c073d0 con 0x561973bf9dc0 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 10 client.0 _ll_register_callbacks cb 0x561973b5a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57998/0 (socket says 172.21.15.16:57998) 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 -- 172.21.15.16:0/1002320711 learned_addr learned my addr 172.21.15.16:0/1002320711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9813fff640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c01270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 -- 172.21.15.16:0/1002320711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 msgr2=0x561973c01270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c01270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 -- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 msgr2=0x561973c017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 -- 172.21.15.16:0/1002320711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619738d03c0 con 0x561973bf94a0 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9813fff640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c01270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9820af8640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f981401a040 tx=0x7f98140175c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad37dd8228824c72 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f97fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814025070 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9814015510 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.692+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9814015bf0 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9814030050 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561973c12c00 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9814015bf0 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9814021bd0 con 0x561973bf94a0 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 10 client.15306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 20 client.15306 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c14010 0x561973c16400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 4 client.15306 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561973b6c850 con 0x561973c14010 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.693+0000 7f9813fff640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c14010 0x561973c16400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.694+0000 7f9813fff640 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c14010 0x561973c16400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c100434cee5f0b server_cookie=b5c247d474ed488e in_seq=0 out_seq=0 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.694+0000 7f97fb7fe640 10 client.15306 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.694+0000 7f97fb7fe640 1 -- 172.21.15.16:0/1002320711 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561973b6c850 con 0x561973c14010 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.694+0000 7f97fb7fe640 10 client.15306 handle_command_reply: tid=0 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.694+0000 7f9824934740 1 client.15306 shutdown 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 msgr2=0x561973c03fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f981401a040 tx=0x7f98140175c0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 shutdown_connections 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c14010 0x561973c16400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973c04680 0x561973c017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973bf9dc0 0x561973c01270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 --2- 172.21.15.16:0/1002320711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c03fc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 >> 172.21.15.16:0/1002320711 conn(0x561973bf5a30 msgr2=0x561973bfb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f97fb7fe640 0 client.15306 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 shutdown_connections 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 1 -- 172.21.15.16:0/1002320711 wait complete. 2024-09-12T02:47:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.695+0000 7f9824934740 20 client.15306 trim_cache size 0 max 16384 2024-09-12T02:47:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 Processor -- start 2024-09-12T02:47:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 -- start start 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf9dc0 0x561973c1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c1bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1ab3b0 con 0x561973bf94a0 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1acef0 con 0x561973bf9dc0 2024-09-12T02:47:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9824934740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1aba80 con 0x561973c04680 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9820af8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58010/0 (socket says 172.21.15.16:58010) 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.712+0000 7f9820af8640 1 -- 172.21.15.16:0/870946612 learned_addr learned my addr 172.21.15.16:0/870946612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9813fff640 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf9dc0 0x561973c1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 -- 172.21.15.16:0/870946612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 msgr2=0x561973c1bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c1bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 -- 172.21.15.16:0/870946612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf9dc0 msgr2=0x561973c1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf9dc0 0x561973c1ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 -- 172.21.15.16:0/870946612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561973bf6ae0 con 0x561973bf94a0 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9820af8640 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9814000f80 tx=0x7f9814027f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255aabd4b224128a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9811ffb640 1 -- 172.21.15.16:0/870946612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814039070 con 0x561973bf94a0 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9811ffb640 1 -- 172.21.15.16:0/870946612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9814020d60 con 0x561973bf94a0 2024-09-12T02:47:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.713+0000 7f9811ffb640 1 -- 172.21.15.16:0/870946612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981402d030 con 0x561973bf94a0 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- 172.21.15.16:0/870946612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 msgr2=0x561973c104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9814000f80 tx=0x7f9814027f30 comp rx=0 tx=0).stop 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- 172.21.15.16:0/870946612 shutdown_connections 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c1bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf9dc0 0x561973c1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 --2- 172.21.15.16:0/870946612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf94a0 0x561973c104b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- 172.21.15.16:0/870946612 >> 172.21.15.16:0/870946612 conn(0x561973bf5a30 msgr2=0x561973c00360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- 172.21.15.16:0/870946612 shutdown_connections 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- 172.21.15.16:0/870946612 wait complete. 2024-09-12T02:47:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 -- start start 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.714+0000 7f9824934740 1 Processor -- start 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 0x561973c097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c7ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1ac820 con 0x561973bf9dc0 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1ac150 con 0x561973bf94a0 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f981c1a9390 con 0x561973c04680 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 10 client.0 _ll_register_callbacks cb 0x7f981401a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58022/0 (socket says 172.21.15.16:58022) 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 -- 172.21.15.16:0/3465659143 learned_addr learned my addr 172.21.15.16:0/3465659143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9820af8640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c7ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9820af8640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 0x561973c097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 -- 172.21.15.16:0/3465659143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 msgr2=0x561973c7ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c7ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 -- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 msgr2=0x561973c097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 0x561973c097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 -- 172.21.15.16:0/3465659143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561973c7cfa0 con 0x561973bf9dc0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9820af8640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 0x561973c097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9813fff640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f981c10d540 tx=0x7f9804005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f1c1be9c2304f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f97fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804035040 con 0x561973bf9dc0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9804005040 con 0x561973bf9dc0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.715+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9814020d60 con 0x561973bf9dc0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9804009520 con 0x561973bf9dc0 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 10 client.15312 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561973c7e200 con 0x561973bf9dc0 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9814020d60 con 0x561973bf9dc0 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f98040317b0 con 0x561973bf9dc0 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 10 client.15312 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 10 client.15312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 20 client.15312 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c7e960 0x561973c80d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 4 client.15312 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561973b66d60 con 0x561973c7e960 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.716+0000 7f9820af8640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c7e960 0x561973c80d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9820af8640 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c7e960 0x561973c80d80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c85841e4133e15e6 server_cookie=6f0e639877218dbe in_seq=0 out_seq=0 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f97fb7fe640 10 client.15312 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f97fb7fe640 1 -- 172.21.15.16:0/3465659143 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561973b66d60 con 0x561973c7e960 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f97fb7fe640 10 client.15312 handle_command_reply: tid=0 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 client.15312 shutdown 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 msgr2=0x561973c09ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f981c10d540 tx=0x7f9804005b50 comp rx=0 tx=0).stop 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 shutdown_connections 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561973c7e960 0x561973c80d80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561973c04680 0x561973c7ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561973bf9dc0 0x561973c09ce0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f97fb7fe640 0 client.15312 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 --2- 172.21.15.16:0/3465659143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561973bf94a0 0x561973c097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 >> 172.21.15.16:0/3465659143 conn(0x561973bf5a30 msgr2=0x561973bf4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 shutdown_connections 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.717+0000 7f9824934740 1 -- 172.21.15.16:0/3465659143 wait complete. 2024-09-12T02:47:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f9824934740 20 client.15312 trim_cache size 0 max 16384 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "f16383d4-abec-4f3b-b65b-13bfec4ad2f3": { 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f16383d4-abec-4f3b-b65b-13bfec4ad2f3", 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:47:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:47:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:47:46.728 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 msgr2=0x7f98080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 0x7f98080a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x561973bf6890 tx=0x7f980c0d5060 comp rx=0 tx=0).stop 2024-09-12T02:47:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 msgr2=0x7f981c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f980c00c920 tx=0x7f980c00cde0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 shutdown_connections 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f981c10c170 0x7f981c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f981c103660 0x7f981c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 --2- 172.21.15.16:0/1693388477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981c102d40 0x7f981c198410 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 >> 172.21.15.16:0/1693388477 conn(0x7f981c0fc7a0 msgr2=0x7f981c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 shutdown_connections 2024-09-12T02:47:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:46.718+0000 7f982255b640 1 -- 172.21.15.16:0/1693388477 wait complete. 2024-09-12T02:47:47.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-12T02:47:46.025771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668720,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1243,"num_read_kb":752,"num_write":452,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382612,"kb_used_data":12564,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559212,"statfs":{"total":1151940427776,"available":1151548633088,"internally_reserved":0,"allocated":12865536,"data_stored":7146684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10363,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":77,"num_write":19,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":142,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.084945+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:43.084945+0000","last_peered":"2024-09-12T02:47:43.084945+0000","last_clean":"2024-09-12T02:47:43.084945+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:43.084945+0000","last_undegraded":"2024-09-12T02:47:43.084945+0000","last_fullsized":"2024-09-12T02:47:43.084945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":240,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083940+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:43.083940+0000","last_peered":"2024-09-12T02:47:43.083940+0000","last_clean":"2024-09-12T02:47:43.083940+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:43.083940+0000","last_undegraded":"2024-09-12T02:47:43.083940+0000","last_fullsized":"2024-09-12T02:47:43.083940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":212,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'35","reported_seq":281,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083969+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:43.083969+0000","last_peered":"2024-09-12T02:47:43.083969+0000","last_clean":"2024-09-12T02:47:43.083969+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:43.083969+0000","last_undegraded":"2024-09-12T02:47:43.083969+0000","last_fullsized":"2024-09-12T02:47:43.083969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68607,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":179,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":143,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:44.080060+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:44.080060+0000","last_peered":"2024-09-12T02:47:44.080060+0000","last_clean":"2024-09-12T02:47:44.080060+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:44.080060+0000","last_undegraded":"2024-09-12T02:47:44.080060+0000","last_fullsized":"2024-09-12T02:47:44.080060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":139,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:44.085976+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:44.085976+0000","last_peered":"2024-09-12T02:47:44.085976+0000","last_clean":"2024-09-12T02:47:44.085976+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:44.085976+0000","last_undegraded":"2024-09-12T02:47:44.085976+0000","last_fullsized":"2024-09-12T02:47:44.085976+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250155+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250155+0000","last_peered":"2024-09-12T02:46:01.250155+0000","last_clean":"2024-09-12T02:46:01.250155+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250155+0000","last_undegraded":"2024-09-12T02:46:01.250155+0000","last_fullsized":"2024-09-12T02:46:01.250155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":136,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.078390+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:43.078390+0000","last_peered":"2024-09-12T02:47:43.078390+0000","last_clean":"2024-09-12T02:47:43.078390+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:43.078390+0000","last_undegraded":"2024-09-12T02:47:43.078390+0000","last_fullsized":"2024-09-12T02:47:43.078390+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'8","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.077768+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:43.077768+0000","last_peered":"2024-09-12T02:47:43.077768+0000","last_clean":"2024-09-12T02:47:43.077768+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:43.077768+0000","last_undegraded":"2024-09-12T02:47:43.077768+0000","last_fullsized":"2024-09-12T02:47:43.077768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'18","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.202422+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:34.202422+0000","last_peered":"2024-09-12T02:47:34.202422+0000","last_clean":"2024-09-12T02:47:34.202422+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:34.202422+0000","last_undegraded":"2024-09-12T02:47:34.202422+0000","last_fullsized":"2024-09-12T02:47:34.202422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253128+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253128+0000","last_peered":"2024-09-12T02:46:01.253128+0000","last_clean":"2024-09-12T02:46:01.253128+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253128+0000","last_undegraded":"2024-09-12T02:46:01.253128+0000","last_fullsized":"2024-09-12T02:46:01.253128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244705+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244705+0000","last_peered":"2024-09-12T02:46:01.244705+0000","last_clean":"2024-09-12T02:46:01.244705+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244705+0000","last_undegraded":"2024-09-12T02:46:01.244705+0000","last_fullsized":"2024-09-12T02:46:01.244705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018296+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018296+0000","last_peered":"2024-09-12T02:47:30.018296+0000","last_clean":"2024-09-12T02:47:30.018296+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018296+0000","last_undegraded":"2024-09-12T02:47:30.018296+0000","last_fullsized":"2024-09-12T02:47:30.018296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":144,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.767889+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:43.767889+0000","last_peered":"2024-09-12T02:47:43.767889+0000","last_clean":"2024-09-12T02:47:43.767889+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:43.767889+0000","last_undegraded":"2024-09-12T02:47:43.767889+0000","last_fullsized":"2024-09-12T02:47:43.767889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78118,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1117,"num_read_kb":643,"num_write":269,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":234354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":430096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874281,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1090132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005103,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234036,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462969,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1024700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659198,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888131,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051655,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280588,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":500191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607638,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":499663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:47.219 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-12T02:47:47.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-12T02:47:47.980 INFO:tasks.cephfs.filesystem:scrub status for tag:f16383d4-abec-4f3b-b65b-13bfec4ad2f3 - {'path': '/', 'tag': 'f16383d4-abec-4f3b-b65b-13bfec4ad2f3', 'options': 'recursive,force'} 2024-09-12T02:47:47.990 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:47 smithi033 ceph-mon[35092]: pgmap v333: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-12T02:47:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:47 smithi033 ceph-mon[35092]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:48.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:47 smithi016 ceph-mon[30925]: pgmap v333: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-12T02:47:48.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:47 smithi016 ceph-mon[30925]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:47 smithi184 ceph-mon[35321]: pgmap v333: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-12T02:47:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:47 smithi184 ceph-mon[35321]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- 172.21.15.16:0/1574596098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 msgr2=0x7f66a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 --2- 172.21.15.16:0/1574596098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a01029b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6690009900 tx=0x7f669002a480 comp rx=0 tx=0).stop 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- 172.21.15.16:0/1574596098 shutdown_connections 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 --2- 172.21.15.16:0/1574596098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0102ef0 0x7f66a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 --2- 172.21.15.16:0/1574596098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a01029b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 --2- 172.21.15.16:0/1574596098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0100e20 0x7f66a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- 172.21.15.16:0/1574596098 >> 172.21.15.16:0/1574596098 conn(0x7f66a00fc7d0 msgr2=0x7f66a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- 172.21.15.16:0/1574596098 shutdown_connections 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- 172.21.15.16:0/1574596098 wait complete. 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 Processor -- start 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.477+0000 7f66a7be9640 1 -- start start 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0100e20 0x7f66a0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0102ef0 0x7f66a019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66a0110fa0 con 0x7f66a0102570 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a0110e20 con 0x7f66a0102ef0 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a7be9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66a0111120 con 0x7f66a0100e20 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58032/0 (socket says 172.21.15.16:58032) 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 -- 172.21.15.16:0/3580058963 learned_addr learned my addr 172.21.15.16:0/3580058963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5985640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0100e20 0x7f66a0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a6186640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0102ef0 0x7f66a019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 -- 172.21.15.16:0/3580058963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0100e20 msgr2=0x7f66a0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0100e20 0x7f66a0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 -- 172.21.15.16:0/3580058963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0102ef0 msgr2=0x7f66a019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0102ef0 0x7f66a019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.478+0000 7f66a5184640 1 -- 172.21.15.16:0/3580058963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a019d4f0 con 0x7f66a0102570 2024-09-12T02:47:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f66a5184640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6690006010 tx=0x7f6690003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669003f070 con 0x7f66a0102570 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6690031c30 con 0x7f66a0102570 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a019d780 con 0x7f66a0102570 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690035030 con 0x7f66a0102570 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.479+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a01a50c0 con 0x7f66a0102570 2024-09-12T02:47:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.480+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a0101200 con 0x7f66a0102570 2024-09-12T02:47:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.481+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f6690042070 con 0x7f66a0102570 2024-09-12T02:47:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.482+0000 7f6696ffd640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 0x7f668c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.482+0000 7f66a5985640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 0x7f668c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.482+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f66900ef970 con 0x7f66a0102570 2024-09-12T02:47:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.482+0000 7f66a5985640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 0x7f668c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f669c00bbe0 tx=0x7f669c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.486+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6690031de0 con 0x7f66a0102570 2024-09-12T02:47:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.658+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 --> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66a0106f10 con 0x7f668c0a2c10 2024-09-12T02:47:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.665+0000 7f6696ffd640 1 -- 172.21.15.16:0/3580058963 <== mgr.14150 v2:172.21.15.16:6800/2564464416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258894 (secure 0 0 0) 0x7f66a0106f10 con 0x7f668c0a2c10 2024-09-12T02:47:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.667+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 msgr2=0x7f668c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.667+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 0x7f668c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f669c00bbe0 tx=0x7f669c00b390 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 msgr2=0x7f66a0198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6690006010 tx=0x7f6690003c10 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 shutdown_connections 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a0102ef0 0x7f66a019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f668c0a2c10 0x7f668c0a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a0102570 0x7f66a0198a80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 --2- 172.21.15.16:0/3580058963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a0100e20 0x7f66a0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 >> 172.21.15.16:0/3580058963 conn(0x7f66a00fc7d0 msgr2=0x7f66a0069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 shutdown_connections 2024-09-12T02:47:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:48.668+0000 7f66a7be9640 1 -- 172.21.15.16:0/3580058963 wait complete. 2024-09-12T02:47:49.553 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-12T02:47:48.026328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668720,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1243,"num_read_kb":752,"num_write":452,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180870,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382612,"kb_used_data":12564,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559212,"statfs":{"total":1151940427776,"available":1151548633088,"internally_reserved":0,"allocated":12865536,"data_stored":7146684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":378713405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":64,"num_write":9,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033458+0000","last_change":"2024-09-12T02:45:47.179804+0000","last_active":"2024-09-12T02:47:30.033458+0000","last_peered":"2024-09-12T02:47:30.033458+0000","last_clean":"2024-09-12T02:47:30.033458+0000","last_became_active":"2024-09-12T02:45:47.179474+0000","last_became_peered":"2024-09-12T02:45:47.179474+0000","last_unstale":"2024-09-12T02:47:30.033458+0000","last_undegraded":"2024-09-12T02:47:30.033458+0000","last_fullsized":"2024-09-12T02:47:30.033458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:48.814082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017003+0000","last_change":"2024-09-12T02:45:47.187736+0000","last_active":"2024-09-12T02:47:30.017003+0000","last_peered":"2024-09-12T02:47:30.017003+0000","last_clean":"2024-09-12T02:47:30.017003+0000","last_became_active":"2024-09-12T02:45:47.187447+0000","last_became_peered":"2024-09-12T02:45:47.187447+0000","last_unstale":"2024-09-12T02:47:30.017003+0000","last_undegraded":"2024-09-12T02:47:30.017003+0000","last_fullsized":"2024-09-12T02:47:30.017003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:35.206538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028861+0000","last_change":"2024-09-12T02:45:47.180213+0000","last_active":"2024-09-12T02:47:30.028861+0000","last_peered":"2024-09-12T02:47:30.028861+0000","last_clean":"2024-09-12T02:47:30.028861+0000","last_became_active":"2024-09-12T02:45:47.179880+0000","last_became_peered":"2024-09-12T02:45:47.179880+0000","last_unstale":"2024-09-12T02:47:30.028861+0000","last_undegraded":"2024-09-12T02:47:30.028861+0000","last_fullsized":"2024-09-12T02:47:30.028861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:37:10.948035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024434+0000","last_change":"2024-09-12T02:45:47.185211+0000","last_active":"2024-09-12T02:47:30.024434+0000","last_peered":"2024-09-12T02:47:30.024434+0000","last_clean":"2024-09-12T02:47:30.024434+0000","last_became_active":"2024-09-12T02:45:47.184865+0000","last_became_peered":"2024-09-12T02:45:47.184865+0000","last_unstale":"2024-09-12T02:47:30.024434+0000","last_undegraded":"2024-09-12T02:47:30.024434+0000","last_fullsized":"2024-09-12T02:47:30.024434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:49:59.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009742+0000","last_change":"2024-09-12T02:45:47.173117+0000","last_active":"2024-09-12T02:47:30.009742+0000","last_peered":"2024-09-12T02:47:30.009742+0000","last_clean":"2024-09-12T02:47:30.009742+0000","last_became_active":"2024-09-12T02:45:47.172777+0000","last_became_peered":"2024-09-12T02:45:47.172777+0000","last_unstale":"2024-09-12T02:47:30.009742+0000","last_undegraded":"2024-09-12T02:47:30.009742+0000","last_fullsized":"2024-09-12T02:47:30.009742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:47:49.038276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024809+0000","last_change":"2024-09-12T02:45:47.185747+0000","last_active":"2024-09-12T02:47:30.024809+0000","last_peered":"2024-09-12T02:47:30.024809+0000","last_clean":"2024-09-12T02:47:30.024809+0000","last_became_active":"2024-09-12T02:45:47.184619+0000","last_became_peered":"2024-09-12T02:45:47.184619+0000","last_unstale":"2024-09-12T02:47:30.024809+0000","last_undegraded":"2024-09-12T02:47:30.024809+0000","last_fullsized":"2024-09-12T02:47:30.024809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:30:50.754229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026321+0000","last_change":"2024-09-12T02:45:47.178822+0000","last_active":"2024-09-12T02:47:30.026321+0000","last_peered":"2024-09-12T02:47:30.026321+0000","last_clean":"2024-09-12T02:47:30.026321+0000","last_became_active":"2024-09-12T02:45:47.178509+0000","last_became_peered":"2024-09-12T02:45:47.178509+0000","last_unstale":"2024-09-12T02:47:30.026321+0000","last_undegraded":"2024-09-12T02:47:30.026321+0000","last_fullsized":"2024-09-12T02:47:30.026321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:33:41.272355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018593+0000","last_change":"2024-09-12T02:45:47.189878+0000","last_active":"2024-09-12T02:47:30.018593+0000","last_peered":"2024-09-12T02:47:30.018593+0000","last_clean":"2024-09-12T02:47:30.018593+0000","last_became_active":"2024-09-12T02:45:47.189340+0000","last_became_peered":"2024-09-12T02:45:47.189340+0000","last_unstale":"2024-09-12T02:47:30.018593+0000","last_undegraded":"2024-09-12T02:47:30.018593+0000","last_fullsized":"2024-09-12T02:47:30.018593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:55:17.906575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033102+0000","last_change":"2024-09-12T02:45:47.187271+0000","last_active":"2024-09-12T02:47:30.033102+0000","last_peered":"2024-09-12T02:47:30.033102+0000","last_clean":"2024-09-12T02:47:30.033102+0000","last_became_active":"2024-09-12T02:45:47.186978+0000","last_became_peered":"2024-09-12T02:45:47.186978+0000","last_unstale":"2024-09-12T02:47:30.033102+0000","last_undegraded":"2024-09-12T02:47:30.033102+0000","last_fullsized":"2024-09-12T02:47:30.033102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:42:28.530013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.023276+0000","last_change":"2024-09-12T02:46:01.255844+0000","last_active":"2024-09-12T02:47:30.023276+0000","last_peered":"2024-09-12T02:47:30.023276+0000","last_clean":"2024-09-12T02:47:30.023276+0000","last_became_active":"2024-09-12T02:46:01.255375+0000","last_became_peered":"2024-09-12T02:46:01.255375+0000","last_unstale":"2024-09-12T02:47:30.023276+0000","last_undegraded":"2024-09-12T02:47:30.023276+0000","last_fullsized":"2024-09-12T02:47:30.023276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:02:54.242451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031710+0000","last_change":"2024-09-12T02:45:47.187625+0000","last_active":"2024-09-12T02:47:30.031710+0000","last_peered":"2024-09-12T02:47:30.031710+0000","last_clean":"2024-09-12T02:47:30.031710+0000","last_became_active":"2024-09-12T02:45:47.187131+0000","last_became_peered":"2024-09-12T02:45:47.187131+0000","last_unstale":"2024-09-12T02:47:30.031710+0000","last_undegraded":"2024-09-12T02:47:30.031710+0000","last_fullsized":"2024-09-12T02:47:30.031710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:31:06.024060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024547+0000","last_change":"2024-09-12T02:45:47.190927+0000","last_active":"2024-09-12T02:47:30.024547+0000","last_peered":"2024-09-12T02:47:30.024547+0000","last_clean":"2024-09-12T02:47:30.024547+0000","last_became_active":"2024-09-12T02:45:47.190554+0000","last_became_peered":"2024-09-12T02:45:47.190554+0000","last_unstale":"2024-09-12T02:47:30.024547+0000","last_undegraded":"2024-09-12T02:47:30.024547+0000","last_fullsized":"2024-09-12T02:47:30.024547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:10:00.071985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020577+0000","last_change":"2024-09-12T02:46:01.251093+0000","last_active":"2024-09-12T02:47:30.020577+0000","last_peered":"2024-09-12T02:47:30.020577+0000","last_clean":"2024-09-12T02:47:30.020577+0000","last_became_active":"2024-09-12T02:46:01.250736+0000","last_became_peered":"2024-09-12T02:46:01.250736+0000","last_unstale":"2024-09-12T02:47:30.020577+0000","last_undegraded":"2024-09-12T02:47:30.020577+0000","last_fullsized":"2024-09-12T02:47:30.020577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:18:39.504068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.025132+0000","last_change":"2024-09-12T02:45:47.189870+0000","last_active":"2024-09-12T02:47:30.025132+0000","last_peered":"2024-09-12T02:47:30.025132+0000","last_clean":"2024-09-12T02:47:30.025132+0000","last_became_active":"2024-09-12T02:45:47.189548+0000","last_became_peered":"2024-09-12T02:45:47.189548+0000","last_unstale":"2024-09-12T02:47:30.025132+0000","last_undegraded":"2024-09-12T02:47:30.025132+0000","last_fullsized":"2024-09-12T02:47:30.025132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:55:16.910446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018978+0000","last_change":"2024-09-12T02:45:47.180881+0000","last_active":"2024-09-12T02:47:30.018978+0000","last_peered":"2024-09-12T02:47:30.018978+0000","last_clean":"2024-09-12T02:47:30.018978+0000","last_became_active":"2024-09-12T02:45:47.180555+0000","last_became_peered":"2024-09-12T02:45:47.180555+0000","last_unstale":"2024-09-12T02:47:30.018978+0000","last_undegraded":"2024-09-12T02:47:30.018978+0000","last_fullsized":"2024-09-12T02:47:30.018978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:06:31.193407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031531+0000","last_change":"2024-09-12T02:45:47.184430+0000","last_active":"2024-09-12T02:47:30.031531+0000","last_peered":"2024-09-12T02:47:30.031531+0000","last_clean":"2024-09-12T02:47:30.031531+0000","last_became_active":"2024-09-12T02:45:47.183179+0000","last_became_peered":"2024-09-12T02:45:47.183179+0000","last_unstale":"2024-09-12T02:47:30.031531+0000","last_undegraded":"2024-09-12T02:47:30.031531+0000","last_fullsized":"2024-09-12T02:47:30.031531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:28:25.580592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033624+0000","last_change":"2024-09-12T02:45:47.188832+0000","last_active":"2024-09-12T02:47:30.033624+0000","last_peered":"2024-09-12T02:47:30.033624+0000","last_clean":"2024-09-12T02:47:30.033624+0000","last_became_active":"2024-09-12T02:45:47.188455+0000","last_became_peered":"2024-09-12T02:45:47.188455+0000","last_unstale":"2024-09-12T02:47:30.033624+0000","last_undegraded":"2024-09-12T02:47:30.033624+0000","last_fullsized":"2024-09-12T02:47:30.033624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:04:42.253531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.032832+0000","last_change":"2024-09-12T02:45:47.190788+0000","last_active":"2024-09-12T02:47:30.032832+0000","last_peered":"2024-09-12T02:47:30.032832+0000","last_clean":"2024-09-12T02:47:30.032832+0000","last_became_active":"2024-09-12T02:45:47.190358+0000","last_became_peered":"2024-09-12T02:45:47.190358+0000","last_unstale":"2024-09-12T02:47:30.032832+0000","last_undegraded":"2024-09-12T02:47:30.032832+0000","last_fullsized":"2024-09-12T02:47:30.032832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:46:16.122266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015165+0000","last_change":"2024-09-12T02:45:47.180630+0000","last_active":"2024-09-12T02:47:30.015165+0000","last_peered":"2024-09-12T02:47:30.015165+0000","last_clean":"2024-09-12T02:47:30.015165+0000","last_became_active":"2024-09-12T02:45:47.180158+0000","last_became_peered":"2024-09-12T02:45:47.180158+0000","last_unstale":"2024-09-12T02:47:30.015165+0000","last_undegraded":"2024-09-12T02:47:30.015165+0000","last_fullsized":"2024-09-12T02:47:30.015165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:44:10.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024129+0000","last_change":"2024-09-12T02:45:47.179758+0000","last_active":"2024-09-12T02:47:30.024129+0000","last_peered":"2024-09-12T02:47:30.024129+0000","last_clean":"2024-09-12T02:47:30.024129+0000","last_became_active":"2024-09-12T02:45:47.179423+0000","last_became_peered":"2024-09-12T02:45:47.179423+0000","last_unstale":"2024-09-12T02:47:30.024129+0000","last_undegraded":"2024-09-12T02:47:30.024129+0000","last_fullsized":"2024-09-12T02:47:30.024129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:35:39.702659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018965+0000","last_change":"2024-09-12T02:45:47.181491+0000","last_active":"2024-09-12T02:47:30.018965+0000","last_peered":"2024-09-12T02:47:30.018965+0000","last_clean":"2024-09-12T02:47:30.018965+0000","last_became_active":"2024-09-12T02:45:47.181154+0000","last_became_peered":"2024-09-12T02:45:47.181154+0000","last_unstale":"2024-09-12T02:47:30.018965+0000","last_undegraded":"2024-09-12T02:47:30.018965+0000","last_fullsized":"2024-09-12T02:47:30.018965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:54:57.155747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031841+0000","last_change":"2024-09-12T02:45:47.183693+0000","last_active":"2024-09-12T02:47:30.031841+0000","last_peered":"2024-09-12T02:47:30.031841+0000","last_clean":"2024-09-12T02:47:30.031841+0000","last_became_active":"2024-09-12T02:45:47.183248+0000","last_became_peered":"2024-09-12T02:45:47.183248+0000","last_unstale":"2024-09-12T02:47:30.031841+0000","last_undegraded":"2024-09-12T02:47:30.031841+0000","last_fullsized":"2024-09-12T02:47:30.031841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:22:56.440538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017665+0000","last_change":"2024-09-12T02:45:47.183876+0000","last_active":"2024-09-12T02:47:30.017665+0000","last_peered":"2024-09-12T02:47:30.017665+0000","last_clean":"2024-09-12T02:47:30.017665+0000","last_became_active":"2024-09-12T02:45:47.183572+0000","last_became_peered":"2024-09-12T02:45:47.183572+0000","last_unstale":"2024-09-12T02:47:30.017665+0000","last_undegraded":"2024-09-12T02:47:30.017665+0000","last_fullsized":"2024-09-12T02:47:30.017665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:22:47.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.016358+0000","last_change":"2024-09-12T02:45:47.177993+0000","last_active":"2024-09-12T02:47:30.016358+0000","last_peered":"2024-09-12T02:47:30.016358+0000","last_clean":"2024-09-12T02:47:30.016358+0000","last_became_active":"2024-09-12T02:45:47.177599+0000","last_became_peered":"2024-09-12T02:45:47.177599+0000","last_unstale":"2024-09-12T02:47:30.016358+0000","last_undegraded":"2024-09-12T02:47:30.016358+0000","last_fullsized":"2024-09-12T02:47:30.016358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:18:21.581926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018615+0000","last_change":"2024-09-12T02:45:47.180434+0000","last_active":"2024-09-12T02:47:30.018615+0000","last_peered":"2024-09-12T02:47:30.018615+0000","last_clean":"2024-09-12T02:47:30.018615+0000","last_became_active":"2024-09-12T02:45:47.180129+0000","last_became_peered":"2024-09-12T02:45:47.180129+0000","last_unstale":"2024-09-12T02:47:30.018615+0000","last_undegraded":"2024-09-12T02:47:30.018615+0000","last_fullsized":"2024-09-12T02:47:30.018615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:36:42.673810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028480+0000","last_change":"2024-09-12T02:45:47.186526+0000","last_active":"2024-09-12T02:47:30.028480+0000","last_peered":"2024-09-12T02:47:30.028480+0000","last_clean":"2024-09-12T02:47:30.028480+0000","last_became_active":"2024-09-12T02:45:47.186228+0000","last_became_peered":"2024-09-12T02:45:47.186228+0000","last_unstale":"2024-09-12T02:47:30.028480+0000","last_undegraded":"2024-09-12T02:47:30.028480+0000","last_fullsized":"2024-09-12T02:47:30.028480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:36:51.287106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019127+0000","last_change":"2024-09-12T02:45:47.180167+0000","last_active":"2024-09-12T02:47:30.019127+0000","last_peered":"2024-09-12T02:47:30.019127+0000","last_clean":"2024-09-12T02:47:30.019127+0000","last_became_active":"2024-09-12T02:45:47.178664+0000","last_became_peered":"2024-09-12T02:45:47.178664+0000","last_unstale":"2024-09-12T02:47:30.019127+0000","last_undegraded":"2024-09-12T02:47:30.019127+0000","last_fullsized":"2024-09-12T02:47:30.019127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:32.244525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018463+0000","last_change":"2024-09-12T02:45:47.175913+0000","last_active":"2024-09-12T02:47:30.018463+0000","last_peered":"2024-09-12T02:47:30.018463+0000","last_clean":"2024-09-12T02:47:30.018463+0000","last_became_active":"2024-09-12T02:45:47.175581+0000","last_became_peered":"2024-09-12T02:45:47.175581+0000","last_unstale":"2024-09-12T02:47:30.018463+0000","last_undegraded":"2024-09-12T02:47:30.018463+0000","last_fullsized":"2024-09-12T02:47:30.018463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:31:49.753170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019232+0000","last_change":"2024-09-12T02:45:47.175942+0000","last_active":"2024-09-12T02:47:30.019232+0000","last_peered":"2024-09-12T02:47:30.019232+0000","last_clean":"2024-09-12T02:47:30.019232+0000","last_became_active":"2024-09-12T02:45:47.175602+0000","last_became_peered":"2024-09-12T02:45:47.175602+0000","last_unstale":"2024-09-12T02:47:30.019232+0000","last_undegraded":"2024-09-12T02:47:30.019232+0000","last_fullsized":"2024-09-12T02:47:30.019232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:06.836061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015599+0000","last_change":"2024-09-12T02:45:47.183178+0000","last_active":"2024-09-12T02:47:30.015599+0000","last_peered":"2024-09-12T02:47:30.015599+0000","last_clean":"2024-09-12T02:47:30.015599+0000","last_became_active":"2024-09-12T02:45:47.182802+0000","last_became_peered":"2024-09-12T02:45:47.182802+0000","last_unstale":"2024-09-12T02:47:30.015599+0000","last_undegraded":"2024-09-12T02:47:30.015599+0000","last_fullsized":"2024-09-12T02:47:30.015599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:51:41.926175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019930+0000","last_change":"2024-09-12T02:45:47.186417+0000","last_active":"2024-09-12T02:47:30.019930+0000","last_peered":"2024-09-12T02:47:30.019930+0000","last_clean":"2024-09-12T02:47:30.019930+0000","last_became_active":"2024-09-12T02:45:47.186040+0000","last_became_peered":"2024-09-12T02:45:47.186040+0000","last_unstale":"2024-09-12T02:47:30.019930+0000","last_undegraded":"2024-09-12T02:47:30.019930+0000","last_fullsized":"2024-09-12T02:47:30.019930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:41:13.028019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022153+0000","last_change":"2024-09-12T02:45:47.182436+0000","last_active":"2024-09-12T02:47:30.022153+0000","last_peered":"2024-09-12T02:47:30.022153+0000","last_clean":"2024-09-12T02:47:30.022153+0000","last_became_active":"2024-09-12T02:45:47.182057+0000","last_became_peered":"2024-09-12T02:45:47.182057+0000","last_unstale":"2024-09-12T02:47:30.022153+0000","last_undegraded":"2024-09-12T02:47:30.022153+0000","last_fullsized":"2024-09-12T02:47:30.022153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:52:50.437242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018079+0000","last_change":"2024-09-12T02:45:47.180855+0000","last_active":"2024-09-12T02:47:30.018079+0000","last_peered":"2024-09-12T02:47:30.018079+0000","last_clean":"2024-09-12T02:47:30.018079+0000","last_became_active":"2024-09-12T02:45:47.180500+0000","last_became_peered":"2024-09-12T02:45:47.180500+0000","last_unstale":"2024-09-12T02:47:30.018079+0000","last_undegraded":"2024-09-12T02:47:30.018079+0000","last_fullsized":"2024-09-12T02:47:30.018079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:34:26.032900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":142,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.084945+0000","last_change":"2024-09-12T02:45:45.133836+0000","last_active":"2024-09-12T02:47:43.084945+0000","last_peered":"2024-09-12T02:47:43.084945+0000","last_clean":"2024-09-12T02:47:43.084945+0000","last_became_active":"2024-09-12T02:45:45.133489+0000","last_became_peered":"2024-09-12T02:45:45.133489+0000","last_unstale":"2024-09-12T02:47:43.084945+0000","last_undegraded":"2024-09-12T02:47:43.084945+0000","last_fullsized":"2024-09-12T02:47:43.084945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:35:14.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024797+0000","last_change":"2024-09-12T02:45:47.177994+0000","last_active":"2024-09-12T02:47:30.024797+0000","last_peered":"2024-09-12T02:47:30.024797+0000","last_clean":"2024-09-12T02:47:30.024797+0000","last_became_active":"2024-09-12T02:45:47.177644+0000","last_became_peered":"2024-09-12T02:45:47.177644+0000","last_unstale":"2024-09-12T02:47:30.024797+0000","last_undegraded":"2024-09-12T02:47:30.024797+0000","last_fullsized":"2024-09-12T02:47:30.024797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:51:43.523305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.245826+0000","last_change":"2024-09-12T02:45:45.128602+0000","last_active":"2024-09-12T02:46:01.245826+0000","last_peered":"2024-09-12T02:46:01.245826+0000","last_clean":"2024-09-12T02:46:01.245826+0000","last_became_active":"2024-09-12T02:45:45.128306+0000","last_became_peered":"2024-09-12T02:45:45.128306+0000","last_unstale":"2024-09-12T02:46:01.245826+0000","last_undegraded":"2024-09-12T02:46:01.245826+0000","last_fullsized":"2024-09-12T02:46:01.245826+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:38:46.780026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"78'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247527+0000","last_change":"2024-09-12T02:45:45.130208+0000","last_active":"2024-09-12T02:46:01.247527+0000","last_peered":"2024-09-12T02:46:01.247527+0000","last_clean":"2024-09-12T02:46:01.247527+0000","last_became_active":"2024-09-12T02:45:45.129838+0000","last_became_peered":"2024-09-12T02:45:45.129838+0000","last_unstale":"2024-09-12T02:46:01.247527+0000","last_undegraded":"2024-09-12T02:46:01.247527+0000","last_fullsized":"2024-09-12T02:46:01.247527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:32:27.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.013528+0000","last_change":"2024-09-12T02:45:47.171811+0000","last_active":"2024-09-12T02:47:30.013528+0000","last_peered":"2024-09-12T02:47:30.013528+0000","last_clean":"2024-09-12T02:47:30.013528+0000","last_became_active":"2024-09-12T02:45:47.171492+0000","last_became_peered":"2024-09-12T02:45:47.171492+0000","last_unstale":"2024-09-12T02:47:30.013528+0000","last_undegraded":"2024-09-12T02:47:30.013528+0000","last_fullsized":"2024-09-12T02:47:30.013528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:11.014787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249168+0000","last_change":"2024-09-12T02:45:45.131257+0000","last_active":"2024-09-12T02:46:01.249168+0000","last_peered":"2024-09-12T02:46:01.249168+0000","last_clean":"2024-09-12T02:46:01.249168+0000","last_became_active":"2024-09-12T02:45:45.130962+0000","last_became_peered":"2024-09-12T02:45:45.130962+0000","last_unstale":"2024-09-12T02:46:01.249168+0000","last_undegraded":"2024-09-12T02:46:01.249168+0000","last_fullsized":"2024-09-12T02:46:01.249168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:53:03.409122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.012439+0000","last_change":"2024-09-12T02:45:47.190695+0000","last_active":"2024-09-12T02:47:30.012439+0000","last_peered":"2024-09-12T02:47:30.012439+0000","last_clean":"2024-09-12T02:47:30.012439+0000","last_became_active":"2024-09-12T02:45:47.190278+0000","last_became_peered":"2024-09-12T02:45:47.190278+0000","last_unstale":"2024-09-12T02:47:30.012439+0000","last_undegraded":"2024-09-12T02:47:30.012439+0000","last_fullsized":"2024-09-12T02:47:30.012439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:12:51.853347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":240,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083940+0000","last_change":"2024-09-12T02:45:45.128849+0000","last_active":"2024-09-12T02:47:43.083940+0000","last_peered":"2024-09-12T02:47:43.083940+0000","last_clean":"2024-09-12T02:47:43.083940+0000","last_became_active":"2024-09-12T02:45:45.128556+0000","last_became_peered":"2024-09-12T02:45:45.128556+0000","last_unstale":"2024-09-12T02:47:43.083940+0000","last_undegraded":"2024-09-12T02:47:43.083940+0000","last_fullsized":"2024-09-12T02:47:43.083940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:52:15.338899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":212,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033758+0000","last_change":"2024-09-12T02:45:47.189581+0000","last_active":"2024-09-12T02:47:30.033758+0000","last_peered":"2024-09-12T02:47:30.033758+0000","last_clean":"2024-09-12T02:47:30.033758+0000","last_became_active":"2024-09-12T02:45:47.189231+0000","last_became_peered":"2024-09-12T02:45:47.189231+0000","last_unstale":"2024-09-12T02:47:30.033758+0000","last_undegraded":"2024-09-12T02:47:30.033758+0000","last_fullsized":"2024-09-12T02:47:30.033758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:17:38.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"78'4","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.269751+0000","last_change":"2024-09-12T02:46:01.269751+0000","last_active":"2024-09-12T02:46:01.269751+0000","last_peered":"2024-09-12T02:46:01.269751+0000","last_clean":"2024-09-12T02:46:01.269751+0000","last_became_active":"2024-09-12T02:46:01.255484+0000","last_became_peered":"2024-09-12T02:46:01.255484+0000","last_unstale":"2024-09-12T02:46:01.269751+0000","last_undegraded":"2024-09-12T02:46:01.269751+0000","last_fullsized":"2024-09-12T02:46:01.269751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:13:47.912635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019336+0000","last_change":"2024-09-12T02:46:01.253009+0000","last_active":"2024-09-12T02:47:30.019336+0000","last_peered":"2024-09-12T02:47:30.019336+0000","last_clean":"2024-09-12T02:47:30.019336+0000","last_became_active":"2024-09-12T02:46:01.252555+0000","last_became_peered":"2024-09-12T02:46:01.252555+0000","last_unstale":"2024-09-12T02:47:30.019336+0000","last_undegraded":"2024-09-12T02:47:30.019336+0000","last_fullsized":"2024-09-12T02:47:30.019336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:41:35.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247971+0000","last_change":"2024-09-12T02:45:46.121015+0000","last_active":"2024-09-12T02:46:01.247971+0000","last_peered":"2024-09-12T02:46:01.247971+0000","last_clean":"2024-09-12T02:46:01.247971+0000","last_became_active":"2024-09-12T02:45:46.120668+0000","last_became_peered":"2024-09-12T02:45:46.120668+0000","last_unstale":"2024-09-12T02:46:01.247971+0000","last_undegraded":"2024-09-12T02:46:01.247971+0000","last_fullsized":"2024-09-12T02:46:01.247971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:11:25.370697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033068+0000","last_change":"2024-09-12T02:45:47.176732+0000","last_active":"2024-09-12T02:47:30.033068+0000","last_peered":"2024-09-12T02:47:30.033068+0000","last_clean":"2024-09-12T02:47:30.033068+0000","last_became_active":"2024-09-12T02:45:47.176393+0000","last_became_peered":"2024-09-12T02:45:47.176393+0000","last_unstale":"2024-09-12T02:47:30.033068+0000","last_undegraded":"2024-09-12T02:47:30.033068+0000","last_fullsized":"2024-09-12T02:47:30.033068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:34:03.503987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'35","reported_seq":281,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.083969+0000","last_change":"2024-09-12T02:45:45.133841+0000","last_active":"2024-09-12T02:47:43.083969+0000","last_peered":"2024-09-12T02:47:43.083969+0000","last_clean":"2024-09-12T02:47:43.083969+0000","last_became_active":"2024-09-12T02:45:45.133290+0000","last_became_peered":"2024-09-12T02:45:45.133290+0000","last_unstale":"2024-09-12T02:47:43.083969+0000","last_undegraded":"2024-09-12T02:47:43.083969+0000","last_fullsized":"2024-09-12T02:47:43.083969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:39:31.025645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68607,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":179,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.021540+0000","last_change":"2024-09-12T02:45:47.185865+0000","last_active":"2024-09-12T02:47:30.021540+0000","last_peered":"2024-09-12T02:47:30.021540+0000","last_clean":"2024-09-12T02:47:30.021540+0000","last_became_active":"2024-09-12T02:45:47.185468+0000","last_became_peered":"2024-09-12T02:45:47.185468+0000","last_unstale":"2024-09-12T02:47:30.021540+0000","last_undegraded":"2024-09-12T02:47:30.021540+0000","last_fullsized":"2024-09-12T02:47:30.021540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:25:23.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"78'9","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253273+0000","last_change":"2024-09-12T02:45:45.132797+0000","last_active":"2024-09-12T02:46:01.253273+0000","last_peered":"2024-09-12T02:46:01.253273+0000","last_clean":"2024-09-12T02:46:01.253273+0000","last_became_active":"2024-09-12T02:45:45.132448+0000","last_became_peered":"2024-09-12T02:45:45.132448+0000","last_unstale":"2024-09-12T02:46:01.253273+0000","last_undegraded":"2024-09-12T02:46:01.253273+0000","last_fullsized":"2024-09-12T02:46:01.253273+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:01:25.937542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.015298+0000","last_change":"2024-09-12T02:45:47.178045+0000","last_active":"2024-09-12T02:47:30.015298+0000","last_peered":"2024-09-12T02:47:30.015298+0000","last_clean":"2024-09-12T02:47:30.015298+0000","last_became_active":"2024-09-12T02:45:47.177474+0000","last_became_peered":"2024-09-12T02:45:47.177474+0000","last_unstale":"2024-09-12T02:47:30.015298+0000","last_undegraded":"2024-09-12T02:47:30.015298+0000","last_fullsized":"2024-09-12T02:47:30.015298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:13:42.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248106+0000","last_change":"2024-09-12T02:45:45.132958+0000","last_active":"2024-09-12T02:46:01.248106+0000","last_peered":"2024-09-12T02:46:01.248106+0000","last_clean":"2024-09-12T02:46:01.248106+0000","last_became_active":"2024-09-12T02:45:45.132660+0000","last_became_peered":"2024-09-12T02:45:45.132660+0000","last_unstale":"2024-09-12T02:46:01.248106+0000","last_undegraded":"2024-09-12T02:46:01.248106+0000","last_fullsized":"2024-09-12T02:46:01.248106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:06:40.140023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027688+0000","last_change":"2024-09-12T02:45:47.184854+0000","last_active":"2024-09-12T02:47:30.027688+0000","last_peered":"2024-09-12T02:47:30.027688+0000","last_clean":"2024-09-12T02:47:30.027688+0000","last_became_active":"2024-09-12T02:45:47.184542+0000","last_became_peered":"2024-09-12T02:45:47.184542+0000","last_unstale":"2024-09-12T02:47:30.027688+0000","last_undegraded":"2024-09-12T02:47:30.027688+0000","last_fullsized":"2024-09-12T02:47:30.027688+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:05:54.461862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":143,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:44.080060+0000","last_change":"2024-09-12T02:45:45.131318+0000","last_active":"2024-09-12T02:47:44.080060+0000","last_peered":"2024-09-12T02:47:44.080060+0000","last_clean":"2024-09-12T02:47:44.080060+0000","last_became_active":"2024-09-12T02:45:45.130998+0000","last_became_peered":"2024-09-12T02:45:45.130998+0000","last_unstale":"2024-09-12T02:47:44.080060+0000","last_undegraded":"2024-09-12T02:47:44.080060+0000","last_fullsized":"2024-09-12T02:47:44.080060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:03:35.255637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.026360+0000","last_change":"2024-09-12T02:45:47.190058+0000","last_active":"2024-09-12T02:47:30.026360+0000","last_peered":"2024-09-12T02:47:30.026360+0000","last_clean":"2024-09-12T02:47:30.026360+0000","last_became_active":"2024-09-12T02:45:47.189719+0000","last_became_peered":"2024-09-12T02:45:47.189719+0000","last_unstale":"2024-09-12T02:47:30.026360+0000","last_undegraded":"2024-09-12T02:47:30.026360+0000","last_fullsized":"2024-09-12T02:47:30.026360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:46:26.338622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"78'17","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247804+0000","last_change":"2024-09-12T02:45:46.119421+0000","last_active":"2024-09-12T02:46:01.247804+0000","last_peered":"2024-09-12T02:46:01.247804+0000","last_clean":"2024-09-12T02:46:01.247804+0000","last_became_active":"2024-09-12T02:45:46.119042+0000","last_became_peered":"2024-09-12T02:45:46.119042+0000","last_unstale":"2024-09-12T02:46:01.247804+0000","last_undegraded":"2024-09-12T02:46:01.247804+0000","last_fullsized":"2024-09-12T02:46:01.247804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:24:35.938287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028814+0000","last_change":"2024-09-12T02:45:47.179680+0000","last_active":"2024-09-12T02:47:30.028814+0000","last_peered":"2024-09-12T02:47:30.028814+0000","last_clean":"2024-09-12T02:47:30.028814+0000","last_became_active":"2024-09-12T02:45:47.178420+0000","last_became_peered":"2024-09-12T02:45:47.178420+0000","last_unstale":"2024-09-12T02:47:30.028814+0000","last_undegraded":"2024-09-12T02:47:30.028814+0000","last_fullsized":"2024-09-12T02:47:30.028814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:02:03.436809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.263010+0000","last_change":"2024-09-12T02:46:01.262532+0000","last_active":"2024-09-12T02:46:01.263010+0000","last_peered":"2024-09-12T02:46:01.263010+0000","last_clean":"2024-09-12T02:46:01.263010+0000","last_became_active":"2024-09-12T02:46:01.250602+0000","last_became_peered":"2024-09-12T02:46:01.250602+0000","last_unstale":"2024-09-12T02:46:01.263010+0000","last_undegraded":"2024-09-12T02:46:01.263010+0000","last_fullsized":"2024-09-12T02:46:01.263010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:28:28.526608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019202+0000","last_change":"2024-09-12T02:45:47.178603+0000","last_active":"2024-09-12T02:47:30.019202+0000","last_peered":"2024-09-12T02:47:30.019202+0000","last_clean":"2024-09-12T02:47:30.019202+0000","last_became_active":"2024-09-12T02:45:47.178267+0000","last_became_peered":"2024-09-12T02:45:47.178267+0000","last_unstale":"2024-09-12T02:47:30.019202+0000","last_undegraded":"2024-09-12T02:47:30.019202+0000","last_fullsized":"2024-09-12T02:47:30.019202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:32:11.291436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"78'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247716+0000","last_change":"2024-09-12T02:45:45.133829+0000","last_active":"2024-09-12T02:46:01.247716+0000","last_peered":"2024-09-12T02:46:01.247716+0000","last_clean":"2024-09-12T02:46:01.247716+0000","last_became_active":"2024-09-12T02:45:45.133331+0000","last_became_peered":"2024-09-12T02:45:45.133331+0000","last_unstale":"2024-09-12T02:46:01.247716+0000","last_undegraded":"2024-09-12T02:46:01.247716+0000","last_fullsized":"2024-09-12T02:46:01.247716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:52:19.861166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022025+0000","last_change":"2024-09-12T02:45:47.188251+0000","last_active":"2024-09-12T02:47:30.022025+0000","last_peered":"2024-09-12T02:47:30.022025+0000","last_clean":"2024-09-12T02:47:30.022025+0000","last_became_active":"2024-09-12T02:45:47.187928+0000","last_became_peered":"2024-09-12T02:45:47.187928+0000","last_unstale":"2024-09-12T02:47:30.022025+0000","last_undegraded":"2024-09-12T02:47:30.022025+0000","last_fullsized":"2024-09-12T02:47:30.022025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:53:07.587797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"78'3","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248124+0000","last_change":"2024-09-12T02:45:45.131371+0000","last_active":"2024-09-12T02:46:01.248124+0000","last_peered":"2024-09-12T02:46:01.248124+0000","last_clean":"2024-09-12T02:46:01.248124+0000","last_became_active":"2024-09-12T02:45:45.131089+0000","last_became_peered":"2024-09-12T02:45:45.131089+0000","last_unstale":"2024-09-12T02:46:01.248124+0000","last_undegraded":"2024-09-12T02:46:01.248124+0000","last_fullsized":"2024-09-12T02:46:01.248124+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:20:54.708021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.027062+0000","last_change":"2024-09-12T02:45:47.181355+0000","last_active":"2024-09-12T02:47:30.027062+0000","last_peered":"2024-09-12T02:47:30.027062+0000","last_clean":"2024-09-12T02:47:30.027062+0000","last_became_active":"2024-09-12T02:45:47.181046+0000","last_became_peered":"2024-09-12T02:45:47.181046+0000","last_unstale":"2024-09-12T02:47:30.027062+0000","last_undegraded":"2024-09-12T02:47:30.027062+0000","last_fullsized":"2024-09-12T02:47:30.027062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:27:14.767872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'6","reported_seq":139,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:44.085976+0000","last_change":"2024-09-12T02:45:46.120992+0000","last_active":"2024-09-12T02:47:44.085976+0000","last_peered":"2024-09-12T02:47:44.085976+0000","last_clean":"2024-09-12T02:47:44.085976+0000","last_became_active":"2024-09-12T02:45:46.120604+0000","last_became_peered":"2024-09-12T02:45:46.120604+0000","last_unstale":"2024-09-12T02:47:44.085976+0000","last_undegraded":"2024-09-12T02:47:44.085976+0000","last_fullsized":"2024-09-12T02:47:44.085976+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:44.389738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.031976+0000","last_change":"2024-09-12T02:45:47.182491+0000","last_active":"2024-09-12T02:47:30.031976+0000","last_peered":"2024-09-12T02:47:30.031976+0000","last_clean":"2024-09-12T02:47:30.031976+0000","last_became_active":"2024-09-12T02:45:47.182103+0000","last_became_peered":"2024-09-12T02:45:47.182103+0000","last_unstale":"2024-09-12T02:47:30.031976+0000","last_undegraded":"2024-09-12T02:47:30.031976+0000","last_fullsized":"2024-09-12T02:47:30.031976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:21:38.017115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'137","reported_seq":335,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250098+0000","last_change":"2024-09-12T02:44:36.787327+0000","last_active":"2024-09-12T02:46:01.250098+0000","last_peered":"2024-09-12T02:46:01.250098+0000","last_clean":"2024-09-12T02:46:01.250098+0000","last_became_active":"2024-09-12T02:44:36.786791+0000","last_became_peered":"2024-09-12T02:44:36.786791+0000","last_unstale":"2024-09-12T02:46:01.250098+0000","last_undegraded":"2024-09-12T02:46:01.250098+0000","last_fullsized":"2024-09-12T02:46:01.250098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:41:16.863463+0000","last_clean_scrub_stamp":"2024-09-12T02:41:16.863463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:18:55.746191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248188+0000","last_change":"2024-09-12T02:45:45.133556+0000","last_active":"2024-09-12T02:46:01.248188+0000","last_peered":"2024-09-12T02:46:01.248188+0000","last_clean":"2024-09-12T02:46:01.248188+0000","last_became_active":"2024-09-12T02:45:45.133274+0000","last_became_peered":"2024-09-12T02:45:45.133274+0000","last_unstale":"2024-09-12T02:46:01.248188+0000","last_undegraded":"2024-09-12T02:46:01.248188+0000","last_fullsized":"2024-09-12T02:46:01.248188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:35:19.089512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.028535+0000","last_change":"2024-09-12T02:45:47.181882+0000","last_active":"2024-09-12T02:47:30.028535+0000","last_peered":"2024-09-12T02:47:30.028535+0000","last_clean":"2024-09-12T02:47:30.028535+0000","last_became_active":"2024-09-12T02:45:47.181594+0000","last_became_peered":"2024-09-12T02:45:47.181594+0000","last_unstale":"2024-09-12T02:47:30.028535+0000","last_undegraded":"2024-09-12T02:47:30.028535+0000","last_fullsized":"2024-09-12T02:47:30.028535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:11.372358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.250155+0000","last_change":"2024-09-12T02:45:45.134751+0000","last_active":"2024-09-12T02:46:01.250155+0000","last_peered":"2024-09-12T02:46:01.250155+0000","last_clean":"2024-09-12T02:46:01.250155+0000","last_became_active":"2024-09-12T02:45:45.134464+0000","last_became_peered":"2024-09-12T02:45:45.134464+0000","last_unstale":"2024-09-12T02:46:01.250155+0000","last_undegraded":"2024-09-12T02:46:01.250155+0000","last_fullsized":"2024-09-12T02:46:01.250155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:37:40.022207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018321+0000","last_change":"2024-09-12T02:45:47.181565+0000","last_active":"2024-09-12T02:47:30.018321+0000","last_peered":"2024-09-12T02:47:30.018321+0000","last_clean":"2024-09-12T02:47:30.018321+0000","last_became_active":"2024-09-12T02:45:47.181217+0000","last_became_peered":"2024-09-12T02:45:47.181217+0000","last_unstale":"2024-09-12T02:47:30.018321+0000","last_undegraded":"2024-09-12T02:47:30.018321+0000","last_fullsized":"2024-09-12T02:47:30.018321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:09:53.289826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":136,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.078390+0000","last_change":"2024-09-12T02:45:46.119072+0000","last_active":"2024-09-12T02:47:43.078390+0000","last_peered":"2024-09-12T02:47:43.078390+0000","last_clean":"2024-09-12T02:47:43.078390+0000","last_became_active":"2024-09-12T02:45:46.118728+0000","last_became_peered":"2024-09-12T02:45:46.118728+0000","last_unstale":"2024-09-12T02:47:43.078390+0000","last_undegraded":"2024-09-12T02:47:43.078390+0000","last_fullsized":"2024-09-12T02:47:43.078390+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:26:57.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.024490+0000","last_change":"2024-09-12T02:45:47.182704+0000","last_active":"2024-09-12T02:47:30.024490+0000","last_peered":"2024-09-12T02:47:30.024490+0000","last_clean":"2024-09-12T02:47:30.024490+0000","last_became_active":"2024-09-12T02:45:47.182382+0000","last_became_peered":"2024-09-12T02:45:47.182382+0000","last_unstale":"2024-09-12T02:47:30.024490+0000","last_undegraded":"2024-09-12T02:47:30.024490+0000","last_fullsized":"2024-09-12T02:47:30.024490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:35:36.596277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"80'8","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.077768+0000","last_change":"2024-09-12T02:45:45.136171+0000","last_active":"2024-09-12T02:47:43.077768+0000","last_peered":"2024-09-12T02:47:43.077768+0000","last_clean":"2024-09-12T02:47:43.077768+0000","last_became_active":"2024-09-12T02:45:45.135910+0000","last_became_peered":"2024-09-12T02:45:45.135910+0000","last_unstale":"2024-09-12T02:47:43.077768+0000","last_undegraded":"2024-09-12T02:47:43.077768+0000","last_fullsized":"2024-09-12T02:47:43.077768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:34:31.955627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.022706+0000","last_change":"2024-09-12T02:46:01.251067+0000","last_active":"2024-09-12T02:47:30.022706+0000","last_peered":"2024-09-12T02:47:30.022706+0000","last_clean":"2024-09-12T02:47:30.022706+0000","last_became_active":"2024-09-12T02:46:01.250601+0000","last_became_peered":"2024-09-12T02:46:01.250601+0000","last_unstale":"2024-09-12T02:47:30.022706+0000","last_undegraded":"2024-09-12T02:47:30.022706+0000","last_fullsized":"2024-09-12T02:47:30.022706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:36:32.276977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.249735+0000","last_change":"2024-09-12T02:45:46.119041+0000","last_active":"2024-09-12T02:46:01.249735+0000","last_peered":"2024-09-12T02:46:01.249735+0000","last_clean":"2024-09-12T02:46:01.249735+0000","last_became_active":"2024-09-12T02:45:46.118674+0000","last_became_peered":"2024-09-12T02:45:46.118674+0000","last_unstale":"2024-09-12T02:46:01.249735+0000","last_undegraded":"2024-09-12T02:46:01.249735+0000","last_fullsized":"2024-09-12T02:46:01.249735+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:03:52.586452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020935+0000","last_change":"2024-09-12T02:45:47.186393+0000","last_active":"2024-09-12T02:47:30.020935+0000","last_peered":"2024-09-12T02:47:30.020935+0000","last_clean":"2024-09-12T02:47:30.020935+0000","last_became_active":"2024-09-12T02:45:47.185010+0000","last_became_peered":"2024-09-12T02:45:47.185010+0000","last_unstale":"2024-09-12T02:47:30.020935+0000","last_undegraded":"2024-09-12T02:47:30.020935+0000","last_fullsized":"2024-09-12T02:47:30.020935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:02:43.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"78'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.259483+0000","last_change":"2024-09-12T02:46:01.258982+0000","last_active":"2024-09-12T02:46:01.259483+0000","last_peered":"2024-09-12T02:46:01.259483+0000","last_clean":"2024-09-12T02:46:01.259483+0000","last_became_active":"2024-09-12T02:46:01.250399+0000","last_became_peered":"2024-09-12T02:46:01.250399+0000","last_unstale":"2024-09-12T02:46:01.259483+0000","last_undegraded":"2024-09-12T02:46:01.259483+0000","last_fullsized":"2024-09-12T02:46:01.259483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:26:02.602817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.009998+0000","last_change":"2024-09-12T02:45:47.180684+0000","last_active":"2024-09-12T02:47:30.009998+0000","last_peered":"2024-09-12T02:47:30.009998+0000","last_clean":"2024-09-12T02:47:30.009998+0000","last_became_active":"2024-09-12T02:45:47.180366+0000","last_became_peered":"2024-09-12T02:45:47.180366+0000","last_unstale":"2024-09-12T02:47:30.009998+0000","last_undegraded":"2024-09-12T02:47:30.009998+0000","last_fullsized":"2024-09-12T02:47:30.009998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:50:17.461536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'18","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:34.202422+0000","last_change":"2024-09-12T02:45:45.127334+0000","last_active":"2024-09-12T02:47:34.202422+0000","last_peered":"2024-09-12T02:47:34.202422+0000","last_clean":"2024-09-12T02:47:34.202422+0000","last_became_active":"2024-09-12T02:45:45.127023+0000","last_became_peered":"2024-09-12T02:45:45.127023+0000","last_unstale":"2024-09-12T02:47:34.202422+0000","last_undegraded":"2024-09-12T02:47:34.202422+0000","last_fullsized":"2024-09-12T02:47:34.202422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:59:40.957905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.020185+0000","last_change":"2024-09-12T02:45:47.184159+0000","last_active":"2024-09-12T02:47:30.020185+0000","last_peered":"2024-09-12T02:47:30.020185+0000","last_clean":"2024-09-12T02:47:30.020185+0000","last_became_active":"2024-09-12T02:45:47.183707+0000","last_became_peered":"2024-09-12T02:45:47.183707+0000","last_unstale":"2024-09-12T02:47:30.020185+0000","last_undegraded":"2024-09-12T02:47:30.020185+0000","last_fullsized":"2024-09-12T02:47:30.020185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:39:13.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"78'7","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248079+0000","last_change":"2024-09-12T02:45:45.247782+0000","last_active":"2024-09-12T02:46:01.248079+0000","last_peered":"2024-09-12T02:46:01.248079+0000","last_clean":"2024-09-12T02:46:01.248079+0000","last_became_active":"2024-09-12T02:45:45.247363+0000","last_became_peered":"2024-09-12T02:45:45.247363+0000","last_unstale":"2024-09-12T02:46:01.248079+0000","last_undegraded":"2024-09-12T02:46:01.248079+0000","last_fullsized":"2024-09-12T02:46:01.248079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:02:06.190137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.019262+0000","last_change":"2024-09-12T02:45:47.175575+0000","last_active":"2024-09-12T02:47:30.019262+0000","last_peered":"2024-09-12T02:47:30.019262+0000","last_clean":"2024-09-12T02:47:30.019262+0000","last_became_active":"2024-09-12T02:45:47.175262+0000","last_became_peered":"2024-09-12T02:45:47.175262+0000","last_unstale":"2024-09-12T02:47:30.019262+0000","last_undegraded":"2024-09-12T02:47:30.019262+0000","last_fullsized":"2024-09-12T02:47:30.019262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T05:09:21.822648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.254448+0000","last_change":"2024-09-12T02:45:46.119116+0000","last_active":"2024-09-12T02:46:01.254448+0000","last_peered":"2024-09-12T02:46:01.254448+0000","last_clean":"2024-09-12T02:46:01.254448+0000","last_became_active":"2024-09-12T02:45:46.118750+0000","last_became_peered":"2024-09-12T02:45:46.118750+0000","last_unstale":"2024-09-12T02:46:01.254448+0000","last_undegraded":"2024-09-12T02:46:01.254448+0000","last_fullsized":"2024-09-12T02:46:01.254448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:03:54.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:29.985307+0000","last_change":"2024-09-12T02:45:47.190499+0000","last_active":"2024-09-12T02:47:29.985307+0000","last_peered":"2024-09-12T02:47:29.985307+0000","last_clean":"2024-09-12T02:47:29.985307+0000","last_became_active":"2024-09-12T02:45:47.190121+0000","last_became_peered":"2024-09-12T02:45:47.190121+0000","last_unstale":"2024-09-12T02:47:29.985307+0000","last_undegraded":"2024-09-12T02:47:29.985307+0000","last_fullsized":"2024-09-12T02:47:29.985307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:20:11.541900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"78'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244270+0000","last_change":"2024-09-12T02:45:45.131335+0000","last_active":"2024-09-12T02:46:01.244270+0000","last_peered":"2024-09-12T02:46:01.244270+0000","last_clean":"2024-09-12T02:46:01.244270+0000","last_became_active":"2024-09-12T02:45:45.130979+0000","last_became_peered":"2024-09-12T02:45:45.130979+0000","last_unstale":"2024-09-12T02:46:01.244270+0000","last_undegraded":"2024-09-12T02:46:01.244270+0000","last_fullsized":"2024-09-12T02:46:01.244270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.075211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.014777+0000","last_change":"2024-09-12T02:45:47.180587+0000","last_active":"2024-09-12T02:47:30.014777+0000","last_peered":"2024-09-12T02:47:30.014777+0000","last_clean":"2024-09-12T02:47:30.014777+0000","last_became_active":"2024-09-12T02:45:47.180262+0000","last_became_peered":"2024-09-12T02:45:47.180262+0000","last_unstale":"2024-09-12T02:47:30.014777+0000","last_undegraded":"2024-09-12T02:47:30.014777+0000","last_fullsized":"2024-09-12T02:47:30.014777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:07:28.125972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.253128+0000","last_change":"2024-09-12T02:45:45.132219+0000","last_active":"2024-09-12T02:46:01.253128+0000","last_peered":"2024-09-12T02:46:01.253128+0000","last_clean":"2024-09-12T02:46:01.253128+0000","last_became_active":"2024-09-12T02:45:45.131931+0000","last_became_peered":"2024-09-12T02:45:45.131931+0000","last_unstale":"2024-09-12T02:46:01.253128+0000","last_undegraded":"2024-09-12T02:46:01.253128+0000","last_fullsized":"2024-09-12T02:46:01.253128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:07:30.010281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.017648+0000","last_change":"2024-09-12T02:45:47.189697+0000","last_active":"2024-09-12T02:47:30.017648+0000","last_peered":"2024-09-12T02:47:30.017648+0000","last_clean":"2024-09-12T02:47:30.017648+0000","last_became_active":"2024-09-12T02:45:47.189327+0000","last_became_peered":"2024-09-12T02:45:47.189327+0000","last_unstale":"2024-09-12T02:47:30.017648+0000","last_undegraded":"2024-09-12T02:47:30.017648+0000","last_fullsized":"2024-09-12T02:47:30.017648+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:31:45.316765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.247820+0000","last_change":"2024-09-12T02:45:45.130832+0000","last_active":"2024-09-12T02:46:01.247820+0000","last_peered":"2024-09-12T02:46:01.247820+0000","last_clean":"2024-09-12T02:46:01.247820+0000","last_became_active":"2024-09-12T02:45:45.130495+0000","last_became_peered":"2024-09-12T02:45:45.130495+0000","last_unstale":"2024-09-12T02:46:01.247820+0000","last_undegraded":"2024-09-12T02:46:01.247820+0000","last_fullsized":"2024-09-12T02:46:01.247820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:09:51.589058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029672+0000","last_change":"2024-09-12T02:45:47.183256+0000","last_active":"2024-09-12T02:47:30.029672+0000","last_peered":"2024-09-12T02:47:30.029672+0000","last_clean":"2024-09-12T02:47:30.029672+0000","last_became_active":"2024-09-12T02:45:47.182968+0000","last_became_peered":"2024-09-12T02:45:47.182968+0000","last_unstale":"2024-09-12T02:47:30.029672+0000","last_undegraded":"2024-09-12T02:47:30.029672+0000","last_fullsized":"2024-09-12T02:47:30.029672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T03:58:08.883823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029518+0000","last_change":"2024-09-12T02:45:47.183009+0000","last_active":"2024-09-12T02:47:30.029518+0000","last_peered":"2024-09-12T02:47:30.029518+0000","last_clean":"2024-09-12T02:47:30.029518+0000","last_became_active":"2024-09-12T02:45:47.182664+0000","last_became_peered":"2024-09-12T02:45:47.182664+0000","last_unstale":"2024-09-12T02:47:30.029518+0000","last_undegraded":"2024-09-12T02:47:30.029518+0000","last_fullsized":"2024-09-12T02:47:30.029518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T06:03:02.233828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'5","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.248801+0000","last_change":"2024-09-12T02:45:45.130000+0000","last_active":"2024-09-12T02:46:01.248801+0000","last_peered":"2024-09-12T02:46:01.248801+0000","last_clean":"2024-09-12T02:46:01.248801+0000","last_became_active":"2024-09-12T02:45:45.129686+0000","last_became_peered":"2024-09-12T02:45:45.129686+0000","last_unstale":"2024-09-12T02:46:01.248801+0000","last_undegraded":"2024-09-12T02:46:01.248801+0000","last_fullsized":"2024-09-12T02:46:01.248801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T04:23:34.535895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.033412+0000","last_change":"2024-09-12T02:45:47.190791+0000","last_active":"2024-09-12T02:47:30.033412+0000","last_peered":"2024-09-12T02:47:30.033412+0000","last_clean":"2024-09-12T02:47:30.033412+0000","last_became_active":"2024-09-12T02:45:47.190434+0000","last_became_peered":"2024-09-12T02:45:47.190434+0000","last_unstale":"2024-09-12T02:47:30.033412+0000","last_undegraded":"2024-09-12T02:47:30.033412+0000","last_fullsized":"2024-09-12T02:47:30.033412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T02:47:00.836100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"78'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.266576+0000","last_change":"2024-09-12T02:46:01.266576+0000","last_active":"2024-09-12T02:46:01.266576+0000","last_peered":"2024-09-12T02:46:01.266576+0000","last_clean":"2024-09-12T02:46:01.266576+0000","last_became_active":"2024-09-12T02:46:01.255767+0000","last_became_peered":"2024-09-12T02:46:01.255767+0000","last_unstale":"2024-09-12T02:46:01.266576+0000","last_undegraded":"2024-09-12T02:46:01.266576+0000","last_fullsized":"2024-09-12T02:46:01.266576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:26:47.751367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.029062+0000","last_change":"2024-09-12T02:45:47.177963+0000","last_active":"2024-09-12T02:47:30.029062+0000","last_peered":"2024-09-12T02:47:30.029062+0000","last_clean":"2024-09-12T02:47:30.029062+0000","last_became_active":"2024-09-12T02:45:47.177580+0000","last_became_peered":"2024-09-12T02:45:47.177580+0000","last_unstale":"2024-09-12T02:47:30.029062+0000","last_undegraded":"2024-09-12T02:47:30.029062+0000","last_fullsized":"2024-09-12T02:47:30.029062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:02.998186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:46:01.244741+0000","last_change":"2024-09-12T02:45:46.119129+0000","last_active":"2024-09-12T02:46:01.244741+0000","last_peered":"2024-09-12T02:46:01.244741+0000","last_clean":"2024-09-12T02:46:01.244741+0000","last_became_active":"2024-09-12T02:45:46.118769+0000","last_became_peered":"2024-09-12T02:45:46.118769+0000","last_unstale":"2024-09-12T02:46:01.244741+0000","last_undegraded":"2024-09-12T02:46:01.244741+0000","last_fullsized":"2024-09-12T02:46:01.244741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:59:45.599017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:30.018452+0000","last_change":"2024-09-12T02:45:47.182376+0000","last_active":"2024-09-12T02:47:30.018452+0000","last_peered":"2024-09-12T02:47:30.018452+0000","last_clean":"2024-09-12T02:47:30.018452+0000","last_became_active":"2024-09-12T02:45:47.181900+0000","last_became_peered":"2024-09-12T02:45:47.181900+0000","last_unstale":"2024-09-12T02:47:30.018452+0000","last_undegraded":"2024-09-12T02:47:30.018452+0000","last_fullsized":"2024-09-12T02:47:30.018452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:46.156763+0000","last_clean_scrub_stamp":"2024-09-12T02:45:46.156763+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:48.827168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"80'14","reported_seq":144,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-12T02:47:43.767889+0000","last_change":"2024-09-12T02:45:45.130669+0000","last_active":"2024-09-12T02:47:43.767889+0000","last_peered":"2024-09-12T02:47:43.767889+0000","last_clean":"2024-09-12T02:47:43.767889+0000","last_became_active":"2024-09-12T02:45:45.130263+0000","last_became_peered":"2024-09-12T02:45:45.130263+0000","last_unstale":"2024-09-12T02:47:43.767889+0000","last_undegraded":"2024-09-12T02:47:43.767889+0000","last_fullsized":"2024-09-12T02:47:43.767889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T02:45:44.114164+0000","last_clean_scrub_stamp":"2024-09-12T02:45:44.114164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:01:13.806496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78118,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1117,"num_read_kb":643,"num_write":269,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":134,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":234354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645349,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":897024,"data_stored":430096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874281,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1090132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005103,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40392,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704760,"statfs":{"total":95995035648,"available":95953674240,"internally_reserved":0,"allocated":909312,"data_stored":429499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234036,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462969,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32652,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712500,"statfs":{"total":95995035648,"available":95961600000,"internally_reserved":0,"allocated":1503232,"data_stored":1024700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":41,"seq":176093659198,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888131,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1515520,"data_stored":1022524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":29,"seq":124554051655,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":430004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280589,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":429041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":500191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607639,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":499663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40396,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704756,"statfs":{"total":95995035648,"available":95953670144,"internally_reserved":0,"allocated":913408,"data_stored":430948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T02:47:49.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:49.825 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:50.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:49 smithi016 ceph-mon[30925]: pgmap v334: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-12T02:47:50.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:49 smithi016 ceph-mon[30925]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:50.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 -- 172.21.15.16:0/2167082006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102790 msgr2=0x7f4b1c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 --2- 172.21.15.16:0/2167082006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102790 0x7f4b1c1067e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009900 tx=0x7f4b0c02a410 comp rx=0 tx=0).stop 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 -- 172.21.15.16:0/2167082006 shutdown_connections 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 --2- 172.21.15.16:0/2167082006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 0x7f4b1c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 --2- 172.21.15.16:0/2167082006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c102790 0x7f4b1c1067e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 --2- 172.21.15.16:0/2167082006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c101de0 0x7f4b1c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 -- 172.21.15.16:0/2167082006 >> 172.21.15.16:0/2167082006 conn(0x7f4b1c0fd8c0 msgr2=0x7f4b1c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 -- 172.21.15.16:0/2167082006 shutdown_connections 2024-09-12T02:47:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.236+0000 7f4b23320640 1 -- 172.21.15.16:0/2167082006 wait complete. 2024-09-12T02:47:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 Processor -- start 2024-09-12T02:47:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 -- start start 2024-09-12T02:47:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c102790 0x7f4b1c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 0x7f4b1c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c111f90 con 0x7f4b1c101de0 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c111e10 con 0x7f4b1c102790 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b23320640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c112110 con 0x7f4b1c106f10 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b210bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b210bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58058/0 (socket says 172.21.15.16:58058) 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.237+0000 7f4b210bc640 1 -- 172.21.15.16:0/1740826930 learned_addr learned my addr 172.21.15.16:0/1740826930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b208bb640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c102790 0x7f4b1c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b218bd640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 0x7f4b1c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 -- 172.21.15.16:0/1740826930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 msgr2=0x7f4b1c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 0x7f4b1c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 -- 172.21.15.16:0/1740826930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c102790 msgr2=0x7f4b1c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c102790 0x7f4b1c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 -- 172.21.15.16:0/1740826930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c19d520 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b210bc640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9f0 tx=0x7f4b1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800cd80 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1c19d810 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b18010910 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1c1a50f0 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.238+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18018990 con 0x7f4b1c101de0 2024-09-12T02:47:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.239+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1c1021c0 con 0x7f4b1c101de0 2024-09-12T02:47:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.240+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f4b18010ac0 con 0x7f4b1c101de0 2024-09-12T02:47:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.241+0000 7f4b127fc640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 0x7f4b080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.241+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b180cfa70 con 0x7f4b1c101de0 2024-09-12T02:47:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.241+0000 7f4b208bb640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 0x7f4b080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.242+0000 7f4b208bb640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 0x7f4b080a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c1087b0 tx=0x7f4b0c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.244+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4b1809c880 con 0x7f4b1c101de0 2024-09-12T02:47:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:49 smithi033 ceph-mon[35092]: pgmap v334: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-12T02:47:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:49 smithi033 ceph-mon[35092]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:49 smithi184 ceph-mon[35321]: pgmap v334: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-12T02:47:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:49 smithi184 ceph-mon[35321]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-12T02:47:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.460+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b1c10a150 con 0x7f4b1c101de0 2024-09-12T02:47:50.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.461+0000 7f4b127fc640 1 -- 172.21.15.16:0/1740826930 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7f4b1809c180 con 0x7f4b1c101de0 2024-09-12T02:47:50.463 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:50.463 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:47:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 msgr2=0x7f4b080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 0x7f4b080a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c1087b0 tx=0x7f4b0c005a20 comp rx=0 tx=0).stop 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 msgr2=0x7f4b1c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9f0 tx=0x7f4b1800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 shutdown_connections 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b1c106f10 0x7f4b1c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b1c102790 0x7f4b1c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b080a2c10 0x7f4b080a50d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 --2- 172.21.15.16:0/1740826930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c101de0 0x7f4b1c198570 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 >> 172.21.15.16:0/1740826930 conn(0x7f4b1c0fd8c0 msgr2=0x7f4b1c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 shutdown_connections 2024-09-12T02:47:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:50.463+0000 7f4b23320640 1 -- 172.21.15.16:0/1740826930 wait complete. 2024-09-12T02:47:51.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:51.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:50 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1740826930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:50 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:50 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1740826930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:50 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:50 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1740826930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:50 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:47:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:51 smithi033 ceph-mon[35092]: pgmap v335: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-12T02:47:52.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:51 smithi016 ceph-mon[30925]: pgmap v335: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-12T02:47:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:51 smithi184 ceph-mon[35321]: pgmap v335: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-12T02:47:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 -- 172.21.15.16:0/931334363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102570 msgr2=0x7f3ddc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 --2- 172.21.15.16:0/931334363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102570 0x7f3ddc1029b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009900 tx=0x7f3dd002a480 comp rx=0 tx=0).stop 2024-09-12T02:47:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 -- 172.21.15.16:0/931334363 shutdown_connections 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 --2- 172.21.15.16:0/931334363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 --2- 172.21.15.16:0/931334363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102570 0x7f3ddc1029b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 --2- 172.21.15.16:0/931334363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 -- 172.21.15.16:0/931334363 >> 172.21.15.16:0/931334363 conn(0x7f3ddc0fc7d0 msgr2=0x7f3ddc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 -- 172.21.15.16:0/931334363 shutdown_connections 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.348+0000 7f3de3f39640 1 -- 172.21.15.16:0/931334363 wait complete. 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 Processor -- start 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 -- start start 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102570 0x7f3ddc19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc110f80 con 0x7f3ddc102ef0 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc110e00 con 0x7f3ddc100e20 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de3f39640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc111100 con 0x7f3ddc102570 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de24d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de24d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58076/0 (socket says 172.21.15.16:58076) 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36126/0 (socket says 172.21.15.16:36126) 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de24d6640 1 -- 172.21.15.16:0/1295043988 learned_addr learned my addr 172.21.15.16:0/1295043988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de14d4640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102570 0x7f3ddc19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 -- 172.21.15.16:0/1295043988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102570 msgr2=0x7f3ddc19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102570 0x7f3ddc19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 -- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 msgr2=0x7f3ddc19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de1cd5640 1 -- 172.21.15.16:0/1295043988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc19f660 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.349+0000 7f3de24d6640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3de1cd5640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400e9f0 tx=0x7f3dd400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd400ccc0 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ddc19f950 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd4010910 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4018960 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.350+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ddc1a7230 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.351+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ddc101200 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.352+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f3dd4010ac0 con 0x7f3ddc100e20 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.353+0000 7f3ddaffd640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 0x7f3dc40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.353+0000 7f3de14d4640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 0x7f3dc40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.353+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3dd40cf900 con 0x7f3ddc100e20 2024-09-12T02:47:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.353+0000 7f3de14d4640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 0x7f3dc40a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009230 tx=0x7f3dd00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.360+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3dd409c060 con 0x7f3ddc100e20 2024-09-12T02:47:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.621+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ddc1a74c0 con 0x7f3ddc100e20 2024-09-12T02:47:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.622+0000 7f3ddaffd640 1 -- 172.21.15.16:0/1295043988 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7f3dd40a0360 con 0x7f3ddc100e20 2024-09-12T02:47:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:52.624 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:47:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:47:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 msgr2=0x7f3dc40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 0x7f3dc40a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009230 tx=0x7f3dd00058d0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 msgr2=0x7f3ddc19a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400e9f0 tx=0x7f3dd400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 shutdown_connections 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3dc40a2c10 0x7f3dc40a50d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ddc102ef0 0x7f3ddc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ddc102570 0x7f3ddc19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 --2- 172.21.15.16:0/1295043988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc100e20 0x7f3ddc19a6b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 >> 172.21.15.16:0/1295043988 conn(0x7f3ddc0fc7d0 msgr2=0x7f3ddc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 shutdown_connections 2024-09-12T02:47:52.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:52.625+0000 7f3de3f39640 1 -- 172.21.15.16:0/1295043988 wait complete. 2024-09-12T02:47:53.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T02:47:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: pgmap v336: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:47:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1295043988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:53 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: pgmap v336: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1295043988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:53 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: pgmap v336: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1295043988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:53 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 -- 172.21.15.16:0/3767908515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364106f10 msgr2=0x7f036410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 --2- 172.21.15.16:0/3767908515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364106f10 0x7f036410ea90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f035c009ca0 tx=0x7f035c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 -- 172.21.15.16:0/3767908515 shutdown_connections 2024-09-12T02:47:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 --2- 172.21.15.16:0/3767908515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364106f10 0x7f036410ea90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 --2- 172.21.15.16:0/3767908515 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 --2- 172.21.15.16:0/3767908515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364101de0 0x7f03641021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 -- 172.21.15.16:0/3767908515 >> 172.21.15.16:0/3767908515 conn(0x7f03640fd8c0 msgr2=0x7f03640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 -- 172.21.15.16:0/3767908515 shutdown_connections 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.190+0000 7f036ade7640 1 -- 172.21.15.16:0/3767908515 wait complete. 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 Processor -- start 2024-09-12T02:47:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 -- start start 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364106f10 0x7f036419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0364111f90 con 0x7f0364101de0 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0364111e10 con 0x7f0364102790 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f036ade7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0364112110 con 0x7f0364106f10 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.191+0000 7f0368b83640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58100/0 (socket says 172.21.15.16:58100) 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 -- 172.21.15.16:0/1979831675 learned_addr learned my addr 172.21.15.16:0/1979831675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0363fff640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0369384640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364106f10 0x7f036419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 -- 172.21.15.16:0/1979831675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364106f10 msgr2=0x7f036419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364106f10 0x7f036419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 -- 172.21.15.16:0/1979831675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 msgr2=0x7f03641989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 -- 172.21.15.16:0/1979831675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036419d420 con 0x7f0364101de0 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0363fff640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0368b83640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f035800b4a0 tx=0x7f035800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0358014070 con 0x7f0364101de0 2024-09-12T02:47:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0358010af0 con 0x7f0364101de0 2024-09-12T02:47:54.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036419d710 con 0x7f0364101de0 2024-09-12T02:47:54.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.192+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0358017d10 con 0x7f0364101de0 2024-09-12T02:47:54.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.193+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03641a4ff0 con 0x7f0364101de0 2024-09-12T02:47:54.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.194+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03641021c0 con 0x7f0364101de0 2024-09-12T02:47:54.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.195+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f0358017070 con 0x7f0364101de0 2024-09-12T02:47:54.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.195+0000 7f0361ffb640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 0x7f03540a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:54.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.196+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f03580ce4d0 con 0x7f0364101de0 2024-09-12T02:47:54.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.196+0000 7f0363fff640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 0x7f03540a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:54.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.196+0000 7f0363fff640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 0x7f03540a5150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f03500058c0 tx=0x7f0350005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:54.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.200+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f035809b3f0 con 0x7f0364101de0 2024-09-12T02:47:54.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.378+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0364109b60 con 0x7f0364101de0 2024-09-12T02:47:54.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:47:54.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:47:54.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.380+0000 7f0361ffb640 1 -- 172.21.15.16:0/1979831675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f035809acf0 con 0x7f0364101de0 2024-09-12T02:47:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:54.381 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T02:47:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 msgr2=0x7f03540a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 0x7f03540a5150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f03500058c0 tx=0x7f0350005850 comp rx=0 tx=0).stop 2024-09-12T02:47:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 msgr2=0x7f0364198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f035800b4a0 tx=0x7f035800b960 comp rx=0 tx=0).stop 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 shutdown_connections 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0364106f10 0x7f036419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0364102790 0x7f03641989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f03540a2c90 0x7f03540a5150 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 --2- 172.21.15.16:0/1979831675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0364101de0 0x7f0364198470 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 >> 172.21.15.16:0/1979831675 conn(0x7f03640fd8c0 msgr2=0x7f03640fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 shutdown_connections 2024-09-12T02:47:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:54.382+0000 7f036ade7640 1 -- 172.21.15.16:0/1979831675 wait complete. 2024-09-12T02:47:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:47:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:47:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:47:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:47:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:47:55.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-12T02:47:55.265 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:55.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:55 smithi016 ceph-mon[30925]: pgmap v337: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-12T02:47:55.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:55 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1979831675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:55 smithi184 ceph-mon[35321]: pgmap v337: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-12T02:47:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:55 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1979831675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:55.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:55 smithi033 ceph-mon[35092]: pgmap v337: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-12T02:47:55.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:55 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1979831675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T02:47:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:{"run_dir":"/var/run/ceph"} 2024-09-12T02:47:56.394 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-12T02:47:56.395 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 481 2024-09-12T02:47:56.395 DEBUG:tasks.quiescer:Yielding 2024-09-12T02:47:56.395 INFO:teuthology.run_tasks:Running task workunit... 2024-09-12T02:47:56.410 INFO:tasks.workunit:Pulling workunits from ref 086c737fa75c22d521a3932c59b31c5e1d1c9ef4 2024-09-12T02:47:56.410 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-12T02:47:56.410 INFO:tasks.workunit:timeout=3h 2024-09-12T02:47:56.410 INFO:tasks.workunit:cleanup=True 2024-09-12T02:47:56.411 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:56.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 02:46:05.334200299 +0000 2024-09-12T02:47:56.430 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 02:46:05.334200299 +0000 2024-09-12T02:47:56.431 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 02:47:33.974853546 +0000 2024-09-12T02:47:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 02:46:05.334200299 +0000 2024-09-12T02:47:56.431 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-12T02:47:56.431 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-12T02:47:56.506 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 086c737fa75c22d521a3932c59b31c5e1d1c9ef4 2024-09-12T02:47:56.564 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-12T02:47:56.685 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 -- 172.21.15.16:0/1983547414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c10c180 msgr2=0x7f732c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 --2- 172.21.15.16:0/1983547414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c10c180 0x7f732c10e620 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7324009ca0 tx=0x7f732402a950 comp rx=0 tx=0).stop 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 -- 172.21.15.16:0/1983547414 shutdown_connections 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 --2- 172.21.15.16:0/1983547414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c10c180 0x7f732c10e620 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 --2- 172.21.15.16:0/1983547414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c103670 0x7f732c10bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 --2- 172.21.15.16:0/1983547414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c102d50 0x7f732c103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 -- 172.21.15.16:0/1983547414 >> 172.21.15.16:0/1983547414 conn(0x7f732c0fc7d0 msgr2=0x7f732c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 -- 172.21.15.16:0/1983547414 shutdown_connections 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.196+0000 7f733249b640 1 -- 172.21.15.16:0/1983547414 wait complete. 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 Processor -- start 2024-09-12T02:47:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 -- start start 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 0x7f732c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c10c180 0x7f732c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f732c111f80 con 0x7f732c102d50 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f732c111e00 con 0x7f732c10c180 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f733249b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f732c112100 con 0x7f732c103670 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f732bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f732bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46380/0 (socket says 172.21.15.16:46380) 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.197+0000 7f732bfff640 1 -- 172.21.15.16:0/2863421019 learned_addr learned my addr 172.21.15.16:0/2863421019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f73237fe640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 0x7f732c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f7330a38640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c10c180 0x7f732c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 -- 172.21.15.16:0/2863421019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 msgr2=0x7f732c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 0x7f732c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 -- 172.21.15.16:0/2863421019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c10c180 msgr2=0x7f732c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c10c180 0x7f732c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 -- 172.21.15.16:0/2863421019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c19d3c0 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f73237fe640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 0x7f732c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f732bfff640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f731c00d770 tx=0x7f731c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c017070 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732c19d6b0 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f731c005500 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.198+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c004e00 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.199+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732c1a4f90 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.200+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f732c103130 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.201+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f731c004fa0 con 0x7f732c102d50 2024-09-12T02:47:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.202+0000 7f7329ffb640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 0x7f73180a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.202+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f731c0ce930 con 0x7f732c102d50 2024-09-12T02:47:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.202+0000 7f73237fe640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 0x7f73180a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.203+0000 7f73237fe640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 0x7f73180a5150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7314005fd0 tx=0x7f7314005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.211+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f731c09b740 con 0x7f732c102d50 2024-09-12T02:47:57.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:57 smithi016 ceph-mon[30925]: pgmap v338: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:47:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.426+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f732c106c90 con 0x7f732c102d50 2024-09-12T02:47:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.427+0000 7f7329ffb640 1 -- 172.21.15.16:0/2863421019 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7f731c09b040 con 0x7f732c102d50 2024-09-12T02:47:57.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:47:57.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:47:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:47:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 msgr2=0x7f73180a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 0x7f73180a5150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7314005fd0 tx=0x7f7314005850 comp rx=0 tx=0).stop 2024-09-12T02:47:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 msgr2=0x7f732c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f731c00d770 tx=0x7f731c00dc30 comp rx=0 tx=0).stop 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 shutdown_connections 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f732c10c180 0x7f732c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f732c103670 0x7f732c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73180a2c90 0x7f73180a5150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 --2- 172.21.15.16:0/2863421019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c102d50 0x7f732c198410 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 >> 172.21.15.16:0/2863421019 conn(0x7f732c0fc7d0 msgr2=0x7f732c0fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 shutdown_connections 2024-09-12T02:47:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:57.429+0000 7f733249b640 1 -- 172.21.15.16:0/2863421019 wait complete. 2024-09-12T02:47:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:57 smithi184 ceph-mon[35321]: pgmap v338: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:47:57.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:57 smithi033 ceph-mon[35092]: pgmap v338: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:47:58.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-12T02:45:47.690653+0000', 'modified': '2024-09-12T02:47:34.701840+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24467, 'mds_2': 24445, 'mds_3': 24443, 'mds_4': 14595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14583': {'gid': 14583, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/2216233994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 2216233994}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 2216233994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14595': {'gid': 14595, 'name': 'a', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.16:6841/1593859053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 1593859053}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 1593859053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6833/897342399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 897342399}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 897342399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6833/1900353998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6832', 'nonce': 1900353998}, {'type': 'v1', 'addr': '172.21.15.33:6833', 'nonce': 1900353998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6835/1041505607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 1041505607}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 1041505607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6837/2803727914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 2803727914}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 2803727914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.33:6835/2990713107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 2990713107}, {'type': 'v1', 'addr': '172.21.15.33:6835', 'nonce': 2990713107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.184:6839/598714982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6838', 'nonce': 598714982}, {'type': 'v1', 'addr': '172.21.15.184:6839', 'nonce': 598714982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.33:6837/4242490884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6836', 'nonce': 4242490884}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 4242490884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.33:6839/3603617308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 3603617308}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 3603617308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [14595, 24443, 24445, 24467, 24500]} 2024-09-12T02:47:58.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-12T02:47:58.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.a status 2024-09-12T02:47:58.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:58 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2863421019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:58.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:47:58.424 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:47:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:58 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2863421019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:47:58.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:58 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2863421019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:47:58.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:47:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 -- 172.21.15.16:0/3721516535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4100d00 msgr2=0x7f09d4106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/3721516535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4100d00 0x7f09d4106840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009930 tx=0x7f09c802a420 comp rx=0 tx=0).stop 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 -- 172.21.15.16:0/3721516535 shutdown_connections 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/3721516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4106f50 0x7f09d410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/3721516535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4100d00 0x7f09d4106840 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/3721516535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 0x7f09d4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 -- 172.21.15.16:0/3721516535 >> 172.21.15.16:0/3721516535 conn(0x7f09d4077210 msgr2=0x7f09d40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 -- 172.21.15.16:0/3721516535 shutdown_connections 2024-09-12T02:47:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.881+0000 7f09dc0c1640 1 -- 172.21.15.16:0/3721516535 wait complete. 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 Processor -- start 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 -- start start 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 0x7f09d4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4100d00 0x7f09d4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09d4111f60 con 0x7f09d4106f50 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09d4111de0 con 0x7f09d4100350 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09dc0c1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09d41120e0 con 0x7f09d4100d00 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09da65e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09da65e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46396/0 (socket says 172.21.15.16:46396) 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09da65e640 1 -- 172.21.15.16:0/2611345880 learned_addr learned my addr 172.21.15.16:0/2611345880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 0x7f09d4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.882+0000 7f09d965c640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4100d00 0x7f09d4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 -- 172.21.15.16:0/2611345880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4100d00 msgr2=0x7f09d4198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4100d00 0x7f09d4198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 -- 172.21.15.16:0/2611345880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 msgr2=0x7f09d4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 0x7f09d4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 -- 172.21.15.16:0/2611345880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d419d4b0 con 0x7f09d4106f50 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09da65e640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c970 tx=0x7f09cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09c6ffd640 1 -- 172.21.15.16:0/2611345880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc007bb0 con 0x7f09d4106f50 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09c6ffd640 1 -- 172.21.15.16:0/2611345880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09cc007d50 con 0x7f09d4106f50 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09c6ffd640 1 -- 172.21.15.16:0/2611345880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc00f450 con 0x7f09d4106f50 2024-09-12T02:47:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d419d7a0 con 0x7f09d4106f50 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.883+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d41a5080 con 0x7f09d4106f50 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.885+0000 7f09c6ffd640 1 -- 172.21.15.16:0/2611345880 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f09cc004f60 con 0x7f09d4106f50 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 Processor -- start 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 -- start start 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae0bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae16670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eabad72090 con 0x55eabae16bb0 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eabad19a70 con 0x55eabae0b9d0 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09de49a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55eabad6e050 con 0x55eabae0c2f0 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09d965c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09c6ffd640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 0x7f09c00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.886+0000 7f09d965c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46410/0 (socket says 172.21.15.16:46410) 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 -- 172.21.15.16:0/2153718072 learned_addr learned my addr 172.21.15.16:0/2153718072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae0bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae16670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09da65e640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 0x7f09c00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 -- 172.21.15.16:0/2153718072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 msgr2=0x55eabae16670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae16670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 -- 172.21.15.16:0/2153718072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 msgr2=0x55eabae0bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae0bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 -- 172.21.15.16:0/2153718072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eabad26130 con 0x55eabae16bb0 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09c6ffd640 1 -- 172.21.15.16:0/2611345880 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09cc0ceda0 con 0x7f09d4106f50 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09d965c640 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f09d406a0a0 tx=0x7f09c8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad436da996aabf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09da65e640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 0x7f09c00a5150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0a3b90 tx=0x7f09cc095f20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09bffff640 1 -- 172.21.15.16:0/2153718072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8035070 con 0x55eabae16bb0 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09bffff640 1 -- 172.21.15.16:0/2153718072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80319f0 con 0x55eabae16bb0 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.887+0000 7f09bffff640 1 -- 172.21.15.16:0/2153718072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8004a00 con 0x55eabae16bb0 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 -- 172.21.15.16:0/2153718072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 msgr2=0x55eabae18fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f09d406a0a0 tx=0x7f09c8005fd0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 -- 172.21.15.16:0/2153718072 shutdown_connections 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae16bb0 0x55eabae18fa0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae16670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 --2- 172.21.15.16:0/2153718072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae0bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 -- 172.21.15.16:0/2153718072 >> 172.21.15.16:0/2153718072 conn(0x55eabae07f60 msgr2=0x55eabae08340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 -- 172.21.15.16:0/2153718072 shutdown_connections 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.888+0000 7f09de49a740 1 -- 172.21.15.16:0/2153718072 wait complete. 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 -- start start 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 Processor -- start 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae137b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae13cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eabae19780 con 0x55eabae0b9d0 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eabad2cf20 con 0x55eabae16bb0 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55eabae19900 con 0x55eabae0c2f0 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09de49a740 10 client.0 _ll_register_callbacks cb 0x55eabacf6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:58.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09d965c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09d965c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46418/0 (socket says 172.21.15.16:46418) 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09d965c640 1 -- 172.21.15.16:0/857205091 learned_addr learned my addr 172.21.15.16:0/857205091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.889+0000 7f09d965c640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae13cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 -- 172.21.15.16:0/857205091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 msgr2=0x55eabae137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 -- 172.21.15.16:0/857205091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 msgr2=0x55eabae13cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae13cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 -- 172.21.15.16:0/857205091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eabaae32b0 con 0x55eabae0b9d0 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09d965c640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09c802a460 tx=0x7f09c8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928bc65b62ea8ad0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8035030 con 0x55eabae0b9d0 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09c80047a0 con 0x55eabae0b9d0 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c80319f0 con 0x55eabae0b9d0 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09c80049a0 con 0x55eabae0b9d0 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09de49a740 10 client.15354 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eabae250a0 con 0x55eabae0b9d0 2024-09-12T02:47:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80319f0 con 0x55eabae0b9d0 2024-09-12T02:47:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.890+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f09c803dc20 con 0x55eabae0b9d0 2024-09-12T02:47:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 10 client.15354 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 10 client.15354 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 20 client.15354 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:58.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae263f0 0x55eabae28530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 4 client.15354 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:47:58.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eabad7f600 con 0x55eabae263f0 2024-09-12T02:47:58.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae263f0 0x55eabae28530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09bd7fa640 10 client.15354 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.891+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae263f0 0x55eabae28530 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a783ea8d6d2aee6c server_cookie=72d3c5e38f89b3 in_seq=0 out_seq=0 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09bd7fa640 1 -- 172.21.15.16:0/857205091 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eabad7f600 con 0x55eabae263f0 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09bd7fa640 10 client.15354 handle_command_reply: tid=0 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 client.15354 shutdown 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 msgr2=0x55eabae16460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09c802a460 tx=0x7f09c8005c90 comp rx=0 tx=0).stop 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 shutdown_connections 2024-09-12T02:47:58.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae263f0 0x55eabae28530 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae13cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0c2f0 0x55eabae137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 --2- 172.21.15.16:0/857205091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0b9d0 0x55eabae16460 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09bd7fa640 0 client.15354 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 >> 172.21.15.16:0/857205091 conn(0x55eabae07f60 msgr2=0x55eabae0dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 shutdown_connections 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 1 -- 172.21.15.16:0/857205091 wait complete. 2024-09-12T02:47:58.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.892+0000 7f09de49a740 20 client.15354 trim_cache size 0 max 16384 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 Processor -- start 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 -- start start 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae28530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 0x55eabae269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eabae26870 con 0x55eabae0c2f0 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eabae26f40 con 0x55eabae0b9d0 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09de49a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55eabae27610 con 0x55eabae16bb0 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46426/0 (socket says 172.21.15.16:46426) 2024-09-12T02:47:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1492393514 learned_addr learned my addr 172.21.15.16:0/1492393514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d965c640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 0x55eabae269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d965c640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae28530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1492393514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 msgr2=0x55eabae269f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 0x55eabae269f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1492393514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 msgr2=0x55eabae28530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae28530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1492393514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eabae08bf0 con 0x55eabae0c2f0 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.911+0000 7f09d965c640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 0x55eabae269f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.912+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55eabae14df0 tx=0x7f09d00075b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5fc7d06af37c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.912+0000 7f09bf7fe640 1 -- 172.21.15.16:0/1492393514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d001c770 con 0x55eabae0c2f0 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.912+0000 7f09bf7fe640 1 -- 172.21.15.16:0/1492393514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d001cdf0 con 0x55eabae0c2f0 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.912+0000 7f09bf7fe640 1 -- 172.21.15.16:0/1492393514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d001b3f0 con 0x55eabae0c2f0 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- 172.21.15.16:0/1492393514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 msgr2=0x55eabae27790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55eabae14df0 tx=0x7f09d00075b0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- 172.21.15.16:0/1492393514 shutdown_connections 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae16bb0 0x55eabae269f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae27790 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 --2- 172.21.15.16:0/1492393514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae0b9d0 0x55eabae28530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- 172.21.15.16:0/1492393514 >> 172.21.15.16:0/1492393514 conn(0x55eabae07f60 msgr2=0x55eabae02ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- 172.21.15.16:0/1492393514 shutdown_connections 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- 172.21.15.16:0/1492393514 wait complete. 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 -- start start 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.913+0000 7f09de49a740 1 Processor -- start 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 0x55eabae2be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae89800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eabae283b0 con 0x55eabae0c2f0 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eabae27ce0 con 0x55eabae16bb0 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55eabae28a80 con 0x55eabae0b9d0 2024-09-12T02:47:58.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 10 client.0 _ll_register_callbacks cb 0x7f09c8032ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46428/0 (socket says 172.21.15.16:46428) 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1548189395 learned_addr learned my addr 172.21.15.16:0/1548189395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d965c640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae89800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d965c640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 0x55eabae2be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1548189395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 msgr2=0x55eabae2be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 0x55eabae2be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1548189395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 msgr2=0x55eabae89800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:47:58.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae89800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 -- 172.21.15.16:0/1548189395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eabae89ea0 con 0x55eabae0c2f0 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d965c640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 0x55eabae2be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09d9e5d640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09d0010540 tx=0x7f09d0007fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ba6f3e3cd23b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d000f420 con 0x55eabae0c2f0 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09d000f5c0 con 0x55eabae0c2f0 2024-09-12T02:47:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.914+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09d001cdf0 con 0x55eabae0c2f0 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09d000f7c0 con 0x55eabae0c2f0 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 10 client.15360 fetch_fsmap learned FSMap version 28 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eabae91260 con 0x55eabae0c2f0 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d001cdf0 con 0x55eabae0c2f0 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f09d0039900 con 0x55eabae0c2f0 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 10 client.15360 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 10 client.15360 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 20 client.15360 populate_metadata read hostname 'smithi016' 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae95b50 0x55eabae94860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:47:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 4 client.15360 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "status"}) -- 0x55eabad79b10 con 0x55eabae95b50 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.915+0000 7f09d965c640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae95b50 0x55eabae94860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09bd7fa640 10 client.15360 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09d965c640 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae95b50 0x55eabae94860 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=511f4c0976f6310e server_cookie=6a7f4e6cedf027fe in_seq=0 out_seq=0 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09bd7fa640 1 -- 172.21.15.16:0/1548189395 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55eabad79b10 con 0x55eabae95b50 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09bd7fa640 10 client.15360 handle_command_reply: tid=0 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 client.15360 shutdown 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 msgr2=0x55eabae2c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09d0010540 tx=0x7f09d0007fa0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 shutdown_connections 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55eabae95b50 0x55eabae94860 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eabae16bb0 0x55eabae89800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eabae0c2f0 0x55eabae2c380 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09bd7fa640 0 client.15360 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 --2- 172.21.15.16:0/1548189395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55eabae0b9d0 0x55eabae2be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 >> 172.21.15.16:0/1548189395 conn(0x55eabae07f60 msgr2=0x55eabae11fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 shutdown_connections 2024-09-12T02:47:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 1 -- 172.21.15.16:0/1548189395 wait complete. 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.916+0000 7f09de49a740 20 client.15360 trim_cache size 0 max 16384 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14595, 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:47:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 120.99406216200001, 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 144.03536056900001 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:47:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 msgr2=0x7f09c00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 0x7f09c00a5150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0a3b90 tx=0x7f09cc095f20 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 msgr2=0x7f09d419cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c970 tx=0x7f09cc00ce30 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 shutdown_connections 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09c00a2c90 0x7f09c00a5150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09d4106f50 0x7f09d419cdd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d4100d00 0x7f09d4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 --2- 172.21.15.16:0/2611345880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09d4100350 0x7f09d4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:47:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 >> 172.21.15.16:0/2611345880 conn(0x7f09d4077210 msgr2=0x7f09d40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:47:58.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 shutdown_connections 2024-09-12T02:47:58.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:47:58.917+0000 7f09dc0c1640 1 -- 172.21.15.16:0/2611345880 wait complete. 2024-09-12T02:47:59.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.e status 2024-09-12T02:47:59.896 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:47:59 smithi033 ceph-mon[35092]: pgmap v339: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:47:59 smithi184 ceph-mon[35321]: pgmap v339: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:00.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:47:59 smithi016 ceph-mon[30925]: pgmap v339: 97 pgs: 97 active+clean; 653 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 -- 172.21.15.16:0/1446125569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8106f10 msgr2=0x7f38c810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 --2- 172.21.15.16:0/1446125569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8106f10 0x7f38c810ea90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009ca0 tx=0x7f38c002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 -- 172.21.15.16:0/1446125569 shutdown_connections 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 --2- 172.21.15.16:0/1446125569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8106f10 0x7f38c810ea90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 --2- 172.21.15.16:0/1446125569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 0x7f38c81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 --2- 172.21.15.16:0/1446125569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8101de0 0x7f38c81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.406+0000 7f38cddf7640 1 -- 172.21.15.16:0/1446125569 >> 172.21.15.16:0/1446125569 conn(0x7f38c80fd8c0 msgr2=0x7f38c80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.407+0000 7f38cddf7640 1 -- 172.21.15.16:0/1446125569 shutdown_connections 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.407+0000 7f38cddf7640 1 -- 172.21.15.16:0/1446125569 wait complete. 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.407+0000 7f38cddf7640 1 Processor -- start 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.407+0000 7f38cddf7640 1 -- start start 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 0x7f38c81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8106f10 0x7f38c819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38c8111f70 con 0x7f38c8101de0 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38c8111df0 con 0x7f38c8102790 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38cddf7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38c81120f0 con 0x7f38c8106f10 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46440/0 (socket says 172.21.15.16:46440) 2024-09-12T02:48:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 -- 172.21.15.16:0/2072512476 learned_addr learned my addr 172.21.15.16:0/2072512476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 0x7f38c81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c7fff640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8106f10 0x7f38c819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 -- 172.21.15.16:0/2072512476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8106f10 msgr2=0x7f38c819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8106f10 0x7f38c819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 -- 172.21.15.16:0/2072512476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 msgr2=0x7f38c81988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 0x7f38c81988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 -- 172.21.15.16:0/2072512476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c819d360 con 0x7f38c8101de0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d770 tx=0x7f38b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.408+0000 7f38c4ff9640 1 -- 172.21.15.16:0/2072512476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8017070 con 0x7f38c8101de0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.409+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c819d650 con 0x7f38c8101de0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.409+0000 7f38c4ff9640 1 -- 172.21.15.16:0/2072512476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b8005500 con 0x7f38c8101de0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.409+0000 7f38c4ff9640 1 -- 172.21.15.16:0/2072512476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8004e00 con 0x7f38c8101de0 2024-09-12T02:48:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.409+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c81a4f30 con 0x7f38c8101de0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38c4ff9640 1 -- 172.21.15.16:0/2072512476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f38b8004fa0 con 0x7f38c8101de0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 Processor -- start 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 -- start start 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e417804cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e4178163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e41776b090 con 0x55e4178051f0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e417712a70 con 0x55e417813fc0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.411+0000 7f38d01d0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e417767050 con 0x55e4178048d0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c4ff9640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 0x7f38b00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46450/0 (socket says 172.21.15.16:46450) 2024-09-12T02:48:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 -- 172.21.15.16:0/2281371101 learned_addr learned my addr 172.21.15.16:0/2281371101 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c7fff640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 0x7f38b00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e4178163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c7fff640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e417804cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c4ff9640 1 -- 172.21.15.16:0/2072512476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38b80cec50 con 0x7f38c8101de0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 -- 172.21.15.16:0/2281371101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 msgr2=0x55e417804cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e417804cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 -- 172.21.15.16:0/2281371101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 msgr2=0x55e4178163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e4178163b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 -- 172.21.15.16:0/2281371101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e41771f130 con 0x55e4178051f0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c7fff640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e417804cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c77fe640 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f38b80b9f40 tx=0x7f38b80048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0644df831106cd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38adffb640 1 -- 172.21.15.16:0/2281371101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80a3f70 con 0x55e4178051f0 2024-09-12T02:48:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38c7fff640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 0x7f38b00a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f38c002f540 tx=0x7f38c00093a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.412+0000 7f38adffb640 1 -- 172.21.15.16:0/2281371101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b809b300 con 0x55e4178051f0 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38adffb640 1 -- 172.21.15.16:0/2281371101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80cc800 con 0x55e4178051f0 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 -- 172.21.15.16:0/2281371101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 msgr2=0x55e417813a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f38b80b9f40 tx=0x7f38b80048a0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 -- 172.21.15.16:0/2281371101 shutdown_connections 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e4178163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417813a80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 --2- 172.21.15.16:0/2281371101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e417804cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 -- 172.21.15.16:0/2281371101 >> 172.21.15.16:0/2281371101 conn(0x55e417800f60 msgr2=0x55e417801340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 -- 172.21.15.16:0/2281371101 shutdown_connections 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.413+0000 7f38d01d0740 1 -- 172.21.15.16:0/2281371101 wait complete. 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 -- start start 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 Processor -- start 2024-09-12T02:48:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e417810c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417811170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e417816b90 con 0x55e4178048d0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e417725f20 con 0x55e417813fc0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e417816d10 con 0x55e4178051f0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38d01d0740 10 client.0 _ll_register_callbacks cb 0x55e4176ef030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46466/0 (socket says 172.21.15.16:46466) 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.414+0000 7f38c77fe640 1 -- 172.21.15.16:0/3330160291 learned_addr learned my addr 172.21.15.16:0/3330160291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c7fff640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417811170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e417810c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 -- 172.21.15.16:0/3330160291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 msgr2=0x55e417810c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e417810c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 -- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 msgr2=0x55e417811170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417811170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 -- 172.21.15.16:0/3330160291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4174dc2b0 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c7fff640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417811170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38c77fe640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f38b80cef40 tx=0x7f38b8009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ff3d8846465f1c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80b4060 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38b80b4200 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38b809b300 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f38b809f6b0 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38d01d0740 10 client.15372 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e417822550 con 0x55e4178048d0 2024-09-12T02:48:00.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.415+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b809b300 con 0x55e4178048d0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f38b809a750 con 0x55e4178048d0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 10 client.15372 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 10 client.15372 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 20 client.15372 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e4178246e0 0x55e417825b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 4 client.15372 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e417778600 con 0x55e4178246e0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e4178246e0 0x55e417825b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38a6ffd640 10 client.15372 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.416+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e4178246e0 0x55e417825b50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b24960009558c5b6 server_cookie=94d352b9286cf8ad in_seq=0 out_seq=0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38a6ffd640 1 -- 172.21.15.16:0/3330160291 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e417778600 con 0x55e4178246e0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38a6ffd640 10 client.15372 handle_command_reply: tid=0 2024-09-12T02:48:00.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 client.15372 shutdown 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 msgr2=0x55e4178138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f38b80cef40 tx=0x7f38b8009ea0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 shutdown_connections 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e4178246e0 0x55e417825b50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417811170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e417810c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 --2- 172.21.15.16:0/3330160291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178048d0 0x55e4178138e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 >> 172.21.15.16:0/3330160291 conn(0x55e417800f60 msgr2=0x55e417806cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.417+0000 7f38a6ffd640 0 client.15372 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.418+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 shutdown_connections 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.418+0000 7f38d01d0740 1 -- 172.21.15.16:0/3330160291 wait complete. 2024-09-12T02:48:00.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.418+0000 7f38d01d0740 20 client.15372 trim_cache size 0 max 16384 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.435+0000 7f38d01d0740 1 Processor -- start 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 -- start start 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e4178048d0 0x55e417824010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e41780c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e417823e90 con 0x55e417813fc0 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e417824560 con 0x55e4178048d0 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38d01d0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e417824c30 con 0x55e4178051f0 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46482/0 (socket says 172.21.15.16:46482) 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 -- 172.21.15.16:0/709747793 learned_addr learned my addr 172.21.15.16:0/709747793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c77fe640 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e4178048d0 0x55e417824010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e41780c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 -- 172.21.15.16:0/709747793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 msgr2=0x55e41780c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e41780c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 -- 172.21.15.16:0/709747793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e4178048d0 msgr2=0x55e417824010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e4178048d0 0x55e417824010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 -- 172.21.15.16:0/709747793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e417805020 con 0x55e417813fc0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.436+0000 7f38c7fff640 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f38c0035000 tx=0x7f38c003f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e584e9275bee0d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38ad7fa640 1 -- 172.21.15.16:0/709747793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c002f990 con 0x55e417813fc0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38ad7fa640 1 -- 172.21.15.16:0/709747793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0036040 con 0x55e417813fc0 2024-09-12T02:48:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38ad7fa640 1 -- 172.21.15.16:0/709747793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c00496a0 con 0x55e417813fc0 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 -- 172.21.15.16:0/709747793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 msgr2=0x55e41780de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f38c0035000 tx=0x7f38c003f070 comp rx=0 tx=0).stop 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 -- 172.21.15.16:0/709747793 shutdown_connections 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e417813fc0 0x55e41780de90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178051f0 0x55e41780c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 --2- 172.21.15.16:0/709747793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e4178048d0 0x55e417824010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 -- 172.21.15.16:0/709747793 >> 172.21.15.16:0/709747793 conn(0x55e417800f60 msgr2=0x55e417801340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 -- 172.21.15.16:0/709747793 shutdown_connections 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.437+0000 7f38d01d0740 1 -- 172.21.15.16:0/709747793 wait complete. 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 -- start start 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 Processor -- start 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e41782e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417827ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e4178259d0 con 0x55e4178051f0 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e417825300 con 0x55e417813fc0 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e4178260a0 con 0x55e4178048d0 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38d01d0740 10 client.0 _ll_register_callbacks cb 0x7f38bc00a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46486/0 (socket says 172.21.15.16:46486) 2024-09-12T02:48:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.438+0000 7f38c6ffd640 1 -- 172.21.15.16:0/538858474 learned_addr learned my addr 172.21.15.16:0/538858474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c7fff640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417827ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c77fe640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e41782e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 -- 172.21.15.16:0/538858474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 msgr2=0x55e41782e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e41782e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 -- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 msgr2=0x55e417827ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417827ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 -- 172.21.15.16:0/538858474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e417828110 con 0x55e4178051f0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38c6ffd640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55e417806470 tx=0x7f38bc012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3b71aa205c9b8a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc029020 con 0x55e4178051f0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38bc012e30 con 0x55e4178051f0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38c0036040 con 0x55e4178051f0 2024-09-12T02:48:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f38bc004020 con 0x55e4178051f0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38d01d0740 10 client.15378 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.439+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e41788edf0 con 0x55e4178051f0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0036040 con 0x55e4178051f0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f38bc010430 con 0x55e4178051f0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 10 client.15378 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 10 client.15378 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 20 client.15378 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e417893870 0x55e417892580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 4 client.15378 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "status"}) -- 0x55e417772b10 con 0x55e417893870 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38c77fe640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e417893870 0x55e417892580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38c77fe640 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e417893870 0x55e417892580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1fd05527cdcc77d9 server_cookie=27eb54edf67d030b in_seq=0 out_seq=0 2024-09-12T02:48:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.440+0000 7f38a6ffd640 10 client.15378 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38a6ffd640 1 -- 172.21.15.16:0/538858474 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e417772b10 con 0x55e417893870 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38a6ffd640 10 client.15378 handle_command_reply: tid=0 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 client.15378 shutdown 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 msgr2=0x55e417827660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55e417806470 tx=0x7f38bc012510 comp rx=0 tx=0).stop 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 shutdown_connections 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e417893870 0x55e417892580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e417813fc0 0x55e417827ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4178051f0 0x55e417827660 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38a6ffd640 0 client.15378 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 --2- 172.21.15.16:0/538858474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e4178048d0 0x55e41782e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 >> 172.21.15.16:0/538858474 conn(0x55e417800f60 msgr2=0x55e41780c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 shutdown_connections 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 1 -- 172.21.15.16:0/538858474 wait complete. 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.441+0000 7f38d01d0740 20 client.15378 trim_cache size 0 max 16384 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24443, 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:48:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 123.203448685, 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 161.50493619599999 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 msgr2=0x7f38b00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 0x7f38b00a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f38c002f540 tx=0x7f38c00093a0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 msgr2=0x7f38c81983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d770 tx=0x7f38b800dc30 comp rx=0 tx=0).stop 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 shutdown_connections 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38c8106f10 0x7f38c819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c8102790 0x7f38c81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38b00a2c10 0x7f38b00a50d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 --2- 172.21.15.16:0/2072512476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8101de0 0x7f38c81983b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 >> 172.21.15.16:0/2072512476 conn(0x7f38c80fd8c0 msgr2=0x7f38c80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 shutdown_connections 2024-09-12T02:48:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:00.442+0000 7f38cddf7640 1 -- 172.21.15.16:0/2072512476 wait complete. 2024-09-12T02:48:01.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.i status 2024-09-12T02:48:01.393 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:01.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 -- 172.21.15.16:0/1819941234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 msgr2=0x7f08a410ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 --2- 172.21.15.16:0/1819941234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a410ba00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0898009900 tx=0x7f089802a480 comp rx=0 tx=0).stop 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 -- 172.21.15.16:0/1819941234 shutdown_connections 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 --2- 172.21.15.16:0/1819941234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a410e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 --2- 172.21.15.16:0/1819941234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a410ba00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 --2- 172.21.15.16:0/1819941234 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 0x7f08a4102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 -- 172.21.15.16:0/1819941234 >> 172.21.15.16:0/1819941234 conn(0x7f08a40fc610 msgr2=0x7f08a40fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 -- 172.21.15.16:0/1819941234 shutdown_connections 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.864+0000 7f08a959e640 1 -- 172.21.15.16:0/1819941234 wait complete. 2024-09-12T02:48:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 Processor -- start 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 -- start start 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 0x7f08a4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4111db0 con 0x7f08a4103510 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4111c30 con 0x7f08a410bfd0 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a959e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4111f30 con 0x7f08a4102bf0 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46516/0 (socket says 172.21.15.16:46516) 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 -- 172.21.15.16:0/340411660 learned_addr learned my addr 172.21.15.16:0/340411660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a3fff640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 0x7f08a4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 -- 172.21.15.16:0/340411660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 msgr2=0x7f08a4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 0x7f08a4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 -- 172.21.15.16:0/340411660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 msgr2=0x7f08a419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a37fe640 1 -- 172.21.15.16:0/340411660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a419d130 con 0x7f08a4103510 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.865+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a37fe640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0898005ab0 tx=0x7f08980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a17fa640 1 -- 172.21.15.16:0/340411660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089803f030 con 0x7f08a4103510 2024-09-12T02:48:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a419d3c0 con 0x7f08a4103510 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a17fa640 1 -- 172.21.15.16:0/340411660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898003ce0 con 0x7f08a4103510 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a17fa640 1 -- 172.21.15.16:0/340411660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898035070 con 0x7f08a4103510 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.866+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a41a4d00 con 0x7f08a4103510 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.868+0000 7f08a17fa640 1 -- 172.21.15.16:0/340411660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f0898042030 con 0x7f08a4103510 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.868+0000 7f08acbd9740 1 Processor -- start 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 -- start start 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4dfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4eec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560afc446090 con 0x560afc4ef1c0 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560afc3eda70 con 0x560afc4dfab0 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08acbd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560afc442050 con 0x560afc4e03d0 2024-09-12T02:48:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a17fa640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 0x7f088c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46530/0 (socket says 172.21.15.16:46530) 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 -- 172.21.15.16:0/1111179514 learned_addr learned my addr 172.21.15.16:0/1111179514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a3fff640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4dfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 0x7f088c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a17fa640 1 -- 172.21.15.16:0/340411660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f08980efb60 con 0x7f08a4103510 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 -- 172.21.15.16:0/1111179514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 msgr2=0x560afc4eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4eec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 -- 172.21.15.16:0/1111179514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 msgr2=0x560afc4dfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4dfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a37fe640 1 -- 172.21.15.16:0/1111179514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560afc3fa130 con 0x560afc4ef1c0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4dfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.869+0000 7f08a3fff640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f08a37fe640 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0898000c00 tx=0x7f08980058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e3b170eba08502 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 0x7f088c0a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f089c012770 tx=0x7f089c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f088a7fc640 1 -- 172.21.15.16:0/1111179514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08980bc970 con 0x560afc4ef1c0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f088a7fc640 1 -- 172.21.15.16:0/1111179514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08980bc270 con 0x560afc4ef1c0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f088a7fc640 1 -- 172.21.15.16:0/1111179514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089803b460 con 0x560afc4ef1c0 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f08acbd9740 1 -- 172.21.15.16:0/1111179514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 msgr2=0x560afc4f15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.870+0000 7f08acbd9740 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0898000c00 tx=0x7f08980058b0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 -- 172.21.15.16:0/1111179514 shutdown_connections 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4ef1c0 0x560afc4f15b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- 172.21.15.16:0/1111179514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4dfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 -- 172.21.15.16:0/1111179514 >> 172.21.15.16:0/1111179514 conn(0x560afc4dc160 msgr2=0x560afc4dc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 -- 172.21.15.16:0/1111179514 shutdown_connections 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 -- 172.21.15.16:0/1111179514 wait complete. 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 -- start start 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 Processor -- start 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4eea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.871+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc4ec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560afc4f1b50 con 0x560afc4e03d0 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560afc400f20 con 0x560afc4dfab0 2024-09-12T02:48:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560afc4f1cd0 con 0x560afc4ef1c0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 10 client.0 _ll_register_callbacks cb 0x560afc3ca030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46544/0 (socket says 172.21.15.16:46544) 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 -- 172.21.15.16:0/3061612608 learned_addr learned my addr 172.21.15.16:0/3061612608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc4ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 -- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 msgr2=0x560afc4ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc4ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 -- 172.21.15.16:0/3061612608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 msgr2=0x560afc4eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 -- 172.21.15.16:0/3061612608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560afc1b72b0 con 0x560afc4e03d0 2024-09-12T02:48:01.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08a37fe640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f08980c49b0 tx=0x7f089802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fcc91a7ffd1794 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08837fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898049040 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f08980b88e0 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08980bc270 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08980bc970 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 10 client.15390 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.872+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560afc4f1b50 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08980bc270 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08980bc4d0 con 0x560afc4e03d0 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 10 client.15390 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 10 client.15390 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 20 client.15390 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc4ff7b0 0x560afc500c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 4 client.15390 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560afc453600 con 0x560afc4ff7b0 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.873+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc4ff7b0 0x560afc500c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08837fe640 10 client.15390 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc4ff7b0 0x560afc500c20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79326661d2f782e4 server_cookie=29c50277e336ccd5 in_seq=0 out_seq=0 2024-09-12T02:48:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08837fe640 1 -- 172.21.15.16:0/3061612608 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560afc453600 con 0x560afc4ff7b0 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08837fe640 10 client.15390 handle_command_reply: tid=0 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08acbd9740 1 client.15390 shutdown 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 msgr2=0x560afc4ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.874+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f08980c49b0 tx=0x7f089802a480 comp rx=0 tx=0).stop 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 shutdown_connections 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc4ff7b0 0x560afc500c20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc4ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4e03d0 0x560afc4ebdd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08837fe640 0 client.15390 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3061612608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4dfab0 0x560afc4eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 >> 172.21.15.16:0/3061612608 conn(0x560afc4dc160 msgr2=0x560afc4e1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 shutdown_connections 2024-09-12T02:48:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 1 -- 172.21.15.16:0/3061612608 wait complete. 2024-09-12T02:48:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.875+0000 7f08acbd9740 20 client.15390 trim_cache size 0 max 16384 2024-09-12T02:48:01.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 Processor -- start 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 -- start start 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4e9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 0x560afc4e9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560afc4fef60 con 0x560afc4dfab0 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560afc4ff630 con 0x560afc4ef1c0 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08acbd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560afc4ffd00 con 0x560afc4e03d0 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46556/0 (socket says 172.21.15.16:46556) 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 -- 172.21.15.16:0/3420952880 learned_addr learned my addr 172.21.15.16:0/3420952880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a37fe640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 0x560afc4e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 -- 172.21.15.16:0/3420952880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 msgr2=0x560afc4e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 -- 172.21.15.16:0/3420952880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 msgr2=0x560afc4e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 0x560afc4e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a3fff640 1 -- 172.21.15.16:0/3420952880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560afc4e0200 con 0x560afc4dfab0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a37fe640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.893+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 0x560afc4e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08a3fff640 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f08940041b0 tx=0x7f089400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c85aa1344c5385 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f0889ffb640 1 -- 172.21.15.16:0/3420952880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894014070 con 0x560afc4dfab0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f0889ffb640 1 -- 172.21.15.16:0/3420952880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0894003e10 con 0x560afc4dfab0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f0889ffb640 1 -- 172.21.15.16:0/3420952880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089401c3b0 con 0x560afc4dfab0 2024-09-12T02:48:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 -- 172.21.15.16:0/3420952880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 msgr2=0x560afc500c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f08940041b0 tx=0x7f089400ce30 comp rx=0 tx=0).stop 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 -- 172.21.15.16:0/3420952880 shutdown_connections 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4ef1c0 0x560afc4e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4e03d0 0x560afc4e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 --2- 172.21.15.16:0/3420952880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc500c20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 -- 172.21.15.16:0/3420952880 >> 172.21.15.16:0/3420952880 conn(0x560afc4dc160 msgr2=0x560afc4e7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 -- 172.21.15.16:0/3420952880 shutdown_connections 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.894+0000 7f08acbd9740 1 -- 172.21.15.16:0/3420952880 wait complete. 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 -- start start 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 Processor -- start 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4e03d0 0x560afc5095f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc502410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560afc500aa0 con 0x560afc4dfab0 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560afc5003d0 con 0x560afc4e03d0 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560afc501170 con 0x560afc4ef1c0 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08acbd9740 10 client.0 _ll_register_callbacks cb 0x7f089400c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.895+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46560/0 (socket says 172.21.15.16:46560) 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 -- 172.21.15.16:0/2279734846 learned_addr learned my addr 172.21.15.16:0/2279734846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a37fe640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4e03d0 0x560afc5095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc502410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 -- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 msgr2=0x560afc502410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc502410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 -- 172.21.15.16:0/2279734846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4e03d0 msgr2=0x560afc5095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4e03d0 0x560afc5095f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 -- 172.21.15.16:0/2279734846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560afc502c20 con 0x560afc4dfab0 2024-09-12T02:48:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a8d9d640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc502410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08a3fff640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0894000c00 tx=0x7f0894004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd272e01b9e121cb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08837fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089402b020 con 0x560afc4dfab0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0894005200 con 0x560afc4dfab0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0894003e10 con 0x560afc4dfab0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0894014040 con 0x560afc4dfab0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08acbd9740 10 client.15396 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.896+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560afc5075c0 con 0x560afc4dfab0 2024-09-12T02:48:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0894003e10 con 0x560afc4dfab0 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0894010430 con 0x560afc4dfab0 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 10 client.15396 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 10 client.15396 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 20 client.15396 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc56e070 0x560afc56cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 4 client.15396 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:48:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "status"}) -- 0x560afc44db10 con 0x560afc56e070 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.897+0000 7f08a37fe640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc56e070 0x560afc56cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08837fe640 10 client.15396 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08a37fe640 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc56e070 0x560afc56cd80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14379ced4d613b8b server_cookie=89b716787f6e8123 in_seq=0 out_seq=0 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08837fe640 1 -- 172.21.15.16:0/2279734846 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560afc44db10 con 0x560afc56e070 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08837fe640 10 client.15396 handle_command_reply: tid=0 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 client.15396 shutdown 2024-09-12T02:48:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 msgr2=0x560afc5090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0894000c00 tx=0x7f0894004ee0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 shutdown_connections 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560afc56e070 0x560afc56cd80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08837fe640 0 client.15396 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560afc4ef1c0 0x560afc502410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560afc4e03d0 0x560afc5095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 --2- 172.21.15.16:0/2279734846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560afc4dfab0 0x560afc5090b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 >> 172.21.15.16:0/2279734846 conn(0x560afc4dc160 msgr2=0x560afc4ea6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 shutdown_connections 2024-09-12T02:48:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 1 -- 172.21.15.16:0/2279734846 wait complete. 2024-09-12T02:48:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.898+0000 7f08acbd9740 20 client.15396 trim_cache size 0 max 16384 2024-09-12T02:48:01.917 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:48:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24445, 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 125.668638214, 2024-09-12T02:48:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-12T02:48:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:48:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-09-12T02:48:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 153.189168419 2024-09-12T02:48:01.919 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 msgr2=0x7f088c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 0x7f088c0a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f089c012770 tx=0x7f089c005f90 comp rx=0 tx=0).stop 2024-09-12T02:48:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 msgr2=0x7f08a41986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0898005ab0 tx=0x7f08980042f0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 shutdown_connections 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08a410bfd0 0x7f08a419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f088c0a2c10 0x7f088c0a50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a4103510 0x7f08a41986c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 --2- 172.21.15.16:0/340411660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08a4102bf0 0x7f08a4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 >> 172.21.15.16:0/340411660 conn(0x7f08a40fc610 msgr2=0x7f08a40fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 shutdown_connections 2024-09-12T02:48:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:01.899+0000 7f08a959e640 1 -- 172.21.15.16:0/340411660 wait complete. 2024-09-12T02:48:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:01 smithi016 ceph-mon[30925]: pgmap v340: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:01 smithi033 ceph-mon[35092]: pgmap v340: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:01 smithi184 ceph-mon[35321]: pgmap v340: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:02.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.h status 2024-09-12T02:48:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.323+0000 7fb8e1671640 1 -- 172.21.15.16:0/2920824407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc101420 msgr2=0x7fb8dc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.323+0000 7fb8e1671640 1 --2- 172.21.15.16:0/2920824407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc101420 0x7fb8dc10d950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009cd0 tx=0x7fb8d402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 -- 172.21.15.16:0/2920824407 shutdown_connections 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 --2- 172.21.15.16:0/2920824407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc101420 0x7fb8dc10d950 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 --2- 172.21.15.16:0/2920824407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc100a80 0x7fb8dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 --2- 172.21.15.16:0/2920824407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 -- 172.21.15.16:0/2920824407 >> 172.21.15.16:0/2920824407 conn(0x7fb8dc0fc7a0 msgr2=0x7fb8dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 -- 172.21.15.16:0/2920824407 shutdown_connections 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 -- 172.21.15.16:0/2920824407 wait complete. 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 Processor -- start 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.324+0000 7fb8e1671640 1 -- start start 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 0x7fb8dc1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc110e80 con 0x7fb8dc100a80 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc110d00 con 0x7fb8dc101420 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc111000 con 0x7fb8dc106a80 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46578/0 (socket says 172.21.15.16:46578) 2024-09-12T02:48:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 -- 172.21.15.16:0/3468489364 learned_addr learned my addr 172.21.15.16:0/3468489364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 0x7fb8dc1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 -- 172.21.15.16:0/3468489364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 msgr2=0x7fb8dc19bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc19bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 -- 172.21.15.16:0/3468489364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 msgr2=0x7fb8dc1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 0x7fb8dc1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 -- 172.21.15.16:0/3468489364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc19c330 con 0x7fb8dc100a80 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc19bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 0x7fb8dc1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c890 tx=0x7fb8cc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8d8ff9640 1 -- 172.21.15.16:0/3468489364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc004220 con 0x7fb8dc100a80 2024-09-12T02:48:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8dc19c620 con 0x7fb8dc100a80 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.325+0000 7fb8d8ff9640 1 -- 172.21.15.16:0/3468489364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8cc0043c0 con 0x7fb8dc100a80 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.326+0000 7fb8d8ff9640 1 -- 172.21.15.16:0/3468489364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc004a00 con 0x7fb8dc100a80 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.326+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8dc1a3f00 con 0x7fb8dc100a80 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.328+0000 7fb8d8ff9640 1 -- 172.21.15.16:0/3468489364 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fb8cc0079b0 con 0x7fb8dc100a80 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.328+0000 7fb8e3a4a740 1 Processor -- start 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.328+0000 7fb8e3a4a740 1 -- start start 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.328+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284267c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.328+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612841ce090 con 0x5612842681b0 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561284175a70 con 0x561284274dc0 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612841ca050 con 0x561284267890 2024-09-12T02:48:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8d8ff9640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 0x7fb8c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46580/0 (socket says 172.21.15.16:46580) 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 -- 172.21.15.16:0/4065477867 learned_addr learned my addr 172.21.15.16:0/4065477867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284267c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 0x7fb8c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8d8ff9640 1 -- 172.21.15.16:0/3468489364 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb8cc0cf260 con 0x7fb8dc100a80 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 -- 172.21.15.16:0/4065477867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 msgr2=0x561284267c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284267c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 -- 172.21.15.16:0/4065477867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 msgr2=0x5612842771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 -- 172.21.15.16:0/4065477867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561284182130 con 0x5612842681b0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8daffd640 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc097dd0 tx=0x7fb8cc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d9235295d9d6b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 0x7fb8c80a50d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d402a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.329+0000 7fb8d17fa640 1 -- 172.21.15.16:0/4065477867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc01d070 con 0x5612842681b0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8d17fa640 1 -- 172.21.15.16:0/4065477867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8cc0a30a0 con 0x5612842681b0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8d17fa640 1 -- 172.21.15.16:0/4065477867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc097fe0 con 0x5612842681b0 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/4065477867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 msgr2=0x561284274880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc097dd0 tx=0x7fb8cc000bc0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/4065477867 shutdown_connections 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284274880 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/4065477867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284267c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/4065477867 >> 172.21.15.16:0/4065477867 conn(0x561284263f60 msgr2=0x561284264340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/4065477867 shutdown_connections 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.330+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/4065477867 wait complete. 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 -- start start 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 Processor -- start 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284274690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x561284271f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561284277990 con 0x5612842681b0 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561284188f20 con 0x561284274dc0 2024-09-12T02:48:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561284277b10 con 0x561284267890 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8e3a4a740 10 client.0 _ll_register_callbacks cb 0x561284152030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46596/0 (socket says 172.21.15.16:46596) 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8d27fc640 1 -- 172.21.15.16:0/2705084011 learned_addr learned my addr 172.21.15.16:0/2705084011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.331+0000 7fb8daffd640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284274690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x561284271f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 -- 172.21.15.16:0/2705084011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 msgr2=0x561284274690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284274690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 -- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 msgr2=0x561284271f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x561284271f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 -- 172.21.15.16:0/2705084011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561283f3f2b0 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x561284271f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4010840 tx=0x7fb8c4010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae8561dab3dafb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c400ed40 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb8c4005020 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612841db600 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8c4005220 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8e3a4a740 10 client.15408 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612842833e0 con 0x5612842681b0 2024-09-12T02:48:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.332+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5612841db600 con 0x5612842681b0 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb8c4011440 con 0x5612842681b0 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 10 client.15408 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 10 client.15408 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 20 client.15408 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561284285a30 0x561284286ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 4 client.15408 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612841d5b10 con 0x561284285a30 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8daffd640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561284285a30 0x561284286ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8daffd640 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561284285a30 0x561284286ea0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=376e940d9bba708f server_cookie=cef445455211fb5d in_seq=0 out_seq=0 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.333+0000 7fb8c27fc640 10 client.15408 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8c27fc640 1 -- 172.21.15.16:0/2705084011 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612841d5b10 con 0x561284285a30 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8c27fc640 10 client.15408 handle_command_reply: tid=0 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 client.15408 shutdown 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 msgr2=0x561284271a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4010840 tx=0x7fb8c4010d00 comp rx=0 tx=0).stop 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 shutdown_connections 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561284285a30 0x561284286ea0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x561284271f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612842681b0 0x561284271a30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/2705084011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x561284274690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 >> 172.21.15.16:0/2705084011 conn(0x561284263f60 msgr2=0x561284269cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8c27fc640 0 client.15408 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 shutdown_connections 2024-09-12T02:48:03.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/2705084011 wait complete. 2024-09-12T02:48:03.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.334+0000 7fb8e3a4a740 20 client.15408 trim_cache size 0 max 16384 2024-09-12T02:48:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 Processor -- start 2024-09-12T02:48:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 -- start start 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x5612842867d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612842681b0 0x56128425ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612842851e0 con 0x561284274dc0 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5612842858b0 con 0x5612842681b0 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561284285f80 con 0x561284267890 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46608/0 (socket says 172.21.15.16:46608) 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8db7fe640 1 -- 172.21.15.16:0/825985060 learned_addr learned my addr 172.21.15.16:0/825985060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612842681b0 0x56128425ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.353+0000 7fb8daffd640 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x5612842867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 -- 172.21.15.16:0/825985060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 msgr2=0x5612842867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x5612842867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 -- 172.21.15.16:0/825985060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612842681b0 msgr2=0x56128425ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612842681b0 0x56128425ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 -- 172.21.15.16:0/825985060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128426ff80 con 0x561284274dc0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4035000 tx=0x7fb8d403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae547f3ba1c29f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8d0ff9640 1 -- 172.21.15.16:0/825985060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003df0 con 0x561284274dc0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8d0ff9640 1 -- 172.21.15.16:0/825985060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d4036040 con 0x561284274dc0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.354+0000 7fb8d0ff9640 1 -- 172.21.15.16:0/825985060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4030920 con 0x561284274dc0 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/825985060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 msgr2=0x56128428f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4035000 tx=0x7fb8d403a000 comp rx=0 tx=0).stop 2024-09-12T02:48:03.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/825985060 shutdown_connections 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284274dc0 0x56128428f570 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612842681b0 0x56128425ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/825985060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561284267890 0x5612842867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/825985060 >> 172.21.15.16:0/825985060 conn(0x561284263f60 msgr2=0x561284264340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/825985060 shutdown_connections 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/825985060 wait complete. 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- start start 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 Processor -- start 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612842681b0 0x561284288580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842f0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561284286d20 con 0x561284267890 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561284286650 con 0x561284274dc0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612842873f0 con 0x5612842681b0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.355+0000 7fb8e3a4a740 10 client.0 _ll_register_callbacks cb 0x7fb8c400c410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46618/0 (socket says 172.21.15.16:46618) 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 -- 172.21.15.16:0/3790350957 learned_addr learned my addr 172.21.15.16:0/3790350957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612842681b0 0x561284288580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8db7fe640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 -- 172.21.15.16:0/3790350957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612842681b0 msgr2=0x561284288580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612842681b0 0x561284288580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 -- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 msgr2=0x5612842f0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842f0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 -- 172.21.15.16:0/3790350957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612842f0680 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc007590 tx=0x7fb8cc005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8a85276c2f1a77 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc01d040 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb8cc0cce70 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d4036040 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8cc09c070 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8e3a4a740 10 client.15414 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.356+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612842f1780 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d4036040 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb8cc007ca0 con 0x561284267890 2024-09-12T02:48:03.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 10 client.15414 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 10 client.15414 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 20 client.15414 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5612842f5e80 0x5612842f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 4 client.15414 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "status"}) -- 0x5612841520e0 con 0x5612842f5e80 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.357+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5612842f5e80 0x5612842f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8d27fc640 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5612842f5e80 0x5612842f4b90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4bbc24feab524159 server_cookie=a03e76ccc8dbae2c in_seq=0 out_seq=0 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8c27fc640 10 client.15414 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8c27fc640 1 -- 172.21.15.16:0/3790350957 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5612841520e0 con 0x5612842f5e80 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8c27fc640 10 client.15414 handle_command_reply: tid=0 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 client.15414 shutdown 2024-09-12T02:48:03.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 msgr2=0x561284288040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc007590 tx=0x7fb8cc005300 comp rx=0 tx=0).stop 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 shutdown_connections 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5612842f5e80 0x5612842f4b90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561284274dc0 0x5612842f0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612842681b0 0x561284288580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 --2- 172.21.15.16:0/3790350957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561284267890 0x561284288040 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 >> 172.21.15.16:0/3790350957 conn(0x561284263f60 msgr2=0x561284270300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8c27fc640 0 client.15414 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 shutdown_connections 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 1 -- 172.21.15.16:0/3790350957 wait complete. 2024-09-12T02:48:03.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.358+0000 7fb8e3a4a740 20 client.15414 trim_cache size 0 max 16384 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24467, 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 128.13881957199999, 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:48:03.371 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 157.14450907 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 msgr2=0x7fb8c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 0x7fb8c80a50d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 msgr2=0x7fb8dc197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00c890 tx=0x7fb8cc00cd50 comp rx=0 tx=0).stop 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 shutdown_connections 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8dc106a80 0x7fb8dc19bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8dc101420 0x7fb8dc1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8c80a2c10 0x7fb8c80a50d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 --2- 172.21.15.16:0/3468489364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8dc100a80 0x7fb8dc197380 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 >> 172.21.15.16:0/3468489364 conn(0x7fb8dc0fc7a0 msgr2=0x7fb8dc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 shutdown_connections 2024-09-12T02:48:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:03.359+0000 7fb8e1671640 1 -- 172.21.15.16:0/3468489364 wait complete. 2024-09-12T02:48:04.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.b status 2024-09-12T02:48:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:03 smithi033 ceph-mon[35092]: pgmap v341: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:03 smithi184 ceph-mon[35321]: pgmap v341: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:04.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:03 smithi016 ceph-mon[30925]: pgmap v341: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 -- 172.21.15.16:0/1051990447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0106a80 msgr2=0x7f87b0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 --2- 172.21.15.16:0/1051990447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0106a80 0x7f87b0106e60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009930 tx=0x7f87a402a420 comp rx=0 tx=0).stop 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 -- 172.21.15.16:0/1051990447 shutdown_connections 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 --2- 172.21.15.16:0/1051990447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 0x7f87b010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 --2- 172.21.15.16:0/1051990447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0100a80 0x7f87b0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 --2- 172.21.15.16:0/1051990447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0106a80 0x7f87b0106e60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 -- 172.21.15.16:0/1051990447 >> 172.21.15.16:0/1051990447 conn(0x7f87b00fc7a0 msgr2=0x7f87b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 -- 172.21.15.16:0/1051990447 shutdown_connections 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.678+0000 7f87b70e3640 1 -- 172.21.15.16:0/1051990447 wait complete. 2024-09-12T02:48:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.679+0000 7f87b70e3640 1 Processor -- start 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.679+0000 7f87b70e3640 1 -- start start 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.679+0000 7f87b70e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 0x7f87b01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0106a80 0x7f87b019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0110ef0 con 0x7f87b0100a80 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0110d70 con 0x7f87b0101420 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0111070 con 0x7f87b0106a80 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46624/0 (socket says 172.21.15.16:46624) 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2448821522 learned_addr learned my addr 172.21.15.16:0/2448821522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87affff640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 0x7f87b01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b5680640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0106a80 0x7f87b019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2448821522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0106a80 msgr2=0x7f87b019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0106a80 0x7f87b019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2448821522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 msgr2=0x7f87b01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 0x7f87b01989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2448821522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b019d410 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f87a402a940 tx=0x7f87a4003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87adffb640 1 -- 172.21.15.16:0/2448821522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a403f070 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b019d6a0 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87adffb640 1 -- 172.21.15.16:0/2448821522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a4031c90 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.680+0000 7f87adffb640 1 -- 172.21.15.16:0/2448821522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4035030 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.681+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b01a4fe0 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87adffb640 1 -- 172.21.15.16:0/2448821522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f87a4042070 con 0x7f87b0100a80 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 Processor -- start 2024-09-12T02:48:04.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 -- start start 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af85174670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af85174bb0 0x55af85176fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af850d0090 con 0x55af851699d0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af85077a70 con 0x55af85174bb0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b94bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af850cc050 con 0x55af8516a2f0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b5680640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.683+0000 7f87b5680640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46630/0 (socket says 172.21.15.16:46630) 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 -- 172.21.15.16:0/233400186 learned_addr learned my addr 172.21.15.16:0/233400186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87adffb640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 0x7f87a00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af85174670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87affff640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af85174bb0 0x55af85176fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 -- 172.21.15.16:0/233400186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 msgr2=0x55af85174670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af85174670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 -- 172.21.15.16:0/233400186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af85174bb0 msgr2=0x55af85176fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af85174bb0 0x55af85176fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 -- 172.21.15.16:0/233400186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af85084130 con 0x55af851699d0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af85174670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87adffb640 1 -- 172.21.15.16:0/2448821522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f87a40efa40 con 0x7f87b0100a80 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 0x7f87a00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007fd0 tx=0x7f87a800e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f21b4efdf55d105 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f879affd640 1 -- 172.21.15.16:0/233400186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8022070 con 0x55af851699d0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f879affd640 1 -- 172.21.15.16:0/233400186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8011c30 con 0x55af851699d0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f879affd640 1 -- 172.21.15.16:0/233400186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a801a400 con 0x55af851699d0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.684+0000 7f87b5680640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 0x7f87a00a50d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87a801f000 tx=0x7f87a8020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- 172.21.15.16:0/233400186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 msgr2=0x55af85169db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007fd0 tx=0x7f87a800e7e0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- 172.21.15.16:0/233400186 shutdown_connections 2024-09-12T02:48:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af85174bb0 0x55af85176fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af85174670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 --2- 172.21.15.16:0/233400186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af85169db0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- 172.21.15.16:0/233400186 >> 172.21.15.16:0/233400186 conn(0x55af85165f60 msgr2=0x55af85166340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- 172.21.15.16:0/233400186 shutdown_connections 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- 172.21.15.16:0/233400186 wait complete. 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.685+0000 7f87b94bc740 1 -- start start 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 Processor -- start 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 0x55af85171800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af85171d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af85177780 con 0x55af851699d0 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af8508af20 con 0x55af8516a2f0 2024-09-12T02:48:04.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af85177900 con 0x55af85174bb0 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b94bc740 10 client.0 _ll_register_callbacks cb 0x55af85054030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46634/0 (socket says 172.21.15.16:46634) 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2645055694 learned_addr learned my addr 172.21.15.16:0/2645055694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87affff640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 0x55af85171800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2645055694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 msgr2=0x55af85171d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b5680640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af85171d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af85171d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 msgr2=0x55af85171800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 0x55af85171800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87b4e7f640 1 -- 172.21.15.16:0/2645055694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af84e412b0 con 0x55af851699d0 2024-09-12T02:48:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.686+0000 7f87affff640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 0x55af85171800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f87a40b85b0 tx=0x7f87a408b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00e71afc85caea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4049070 con 0x55af851699d0 2024-09-12T02:48:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f87a40dc0b0 con 0x55af851699d0 2024-09-12T02:48:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a8011c30 con 0x55af851699d0 2024-09-12T02:48:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f87a40bbab0 con 0x55af851699d0 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 10 client.15426 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55af85183060 con 0x55af851699d0 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8011c30 con 0x55af851699d0 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f87a40c0150 con 0x55af851699d0 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 10 client.15426 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 10 client.15426 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 20 client.15426 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af85185750 0x55af85186bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 4 client.15426 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.687+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af850dd600 con 0x55af85185750 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.688+0000 7f87affff640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af85185750 0x55af85186bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.688+0000 7f878ffff640 10 client.15426 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.688+0000 7f87affff640 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af85185750 0x55af85186bc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8090a13cac52e068 server_cookie=75d5b99e6dc5846e in_seq=0 out_seq=0 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f878ffff640 1 -- 172.21.15.16:0/2645055694 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55af850dd600 con 0x55af85185750 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f878ffff640 10 client.15426 handle_command_reply: tid=0 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 client.15426 shutdown 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 msgr2=0x55af851744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f87a40b85b0 tx=0x7f87a408b0d0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 shutdown_connections 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af85185750 0x55af85186bc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af85171d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f878ffff640 0 client.15426 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af8516a2f0 0x55af85171800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 --2- 172.21.15.16:0/2645055694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af851699d0 0x55af851744b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 >> 172.21.15.16:0/2645055694 conn(0x55af85165f60 msgr2=0x55af8516bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 shutdown_connections 2024-09-12T02:48:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 1 -- 172.21.15.16:0/2645055694 wait complete. 2024-09-12T02:48:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.689+0000 7f87b94bc740 20 client.15426 trim_cache size 0 max 16384 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.707+0000 7f87b94bc740 1 Processor -- start 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.707+0000 7f87b94bc740 1 -- start start 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85185080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af851864f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af85184f00 con 0x55af85174bb0 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af851855d0 con 0x55af851699d0 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b94bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af85185ca0 con 0x55af8516a2f0 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46644/0 (socket says 172.21.15.16:46644) 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 -- 172.21.15.16:0/3925972804 learned_addr learned my addr 172.21.15.16:0/3925972804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87affff640 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af851864f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85185080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 -- 172.21.15.16:0/3925972804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 msgr2=0x55af851864f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af851864f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 -- 172.21.15.16:0/3925972804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 msgr2=0x55af85185080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85185080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 -- 172.21.15.16:0/3925972804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af851670d0 con 0x55af85174bb0 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.708+0000 7f87b5680640 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f87a8012480 tx=0x7f87a8013f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a627bff25ec037c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f879a7fc640 1 -- 172.21.15.16:0/3925972804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8035070 con 0x55af85174bb0 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f879a7fc640 1 -- 172.21.15.16:0/3925972804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8007ad0 con 0x55af85174bb0 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f879a7fc640 1 -- 172.21.15.16:0/3925972804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8022040 con 0x55af85174bb0 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 -- 172.21.15.16:0/3925972804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 msgr2=0x55af851843b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f87a8012480 tx=0x7f87a8013f60 comp rx=0 tx=0).stop 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 -- 172.21.15.16:0/3925972804 shutdown_connections 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85174bb0 0x55af851843b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af8516a2f0 0x55af851864f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.709+0000 7f87b94bc740 1 --2- 172.21.15.16:0/3925972804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85185080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- 172.21.15.16:0/3925972804 >> 172.21.15.16:0/3925972804 conn(0x55af85165f60 msgr2=0x55af85160ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- 172.21.15.16:0/3925972804 shutdown_connections 2024-09-12T02:48:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- 172.21.15.16:0/3925972804 wait complete. 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- start start 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 Processor -- start 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85189e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af8518a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af85186a40 con 0x55af8516a2f0 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af85186370 con 0x55af851699d0 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55af85187110 con 0x55af85174bb0 2024-09-12T02:48:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.710+0000 7f87b94bc740 10 client.0 _ll_register_callbacks cb 0x7f87a4031e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46646/0 (socket says 172.21.15.16:46646) 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 -- 172.21.15.16:0/967474581 learned_addr learned my addr 172.21.15.16:0/967474581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87b5680640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af8518a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 -- 172.21.15.16:0/967474581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 msgr2=0x55af8518a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af8518a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 -- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 msgr2=0x55af85189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85189e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 -- 172.21.15.16:0/967474581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af851ee120 con 0x55af8516a2f0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87b5680640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af8518a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87affff640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f879c009a40 tx=0x7f879c0157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26bcdd4dd8094d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f878ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c01c640 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f879c01c7e0 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a8007ad0 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f879c015e50 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.711+0000 7f87b94bc740 10 client.15432 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55af851ef4b0 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8007ad0 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f879c023070 con 0x55af8516a2f0 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 10 client.15432 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 10 client.15432 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 20 client.15432 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af851f4110 0x55af851f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 4 client.15432 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "status"}) -- 0x55af850d7b10 con 0x55af851f4110 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.712+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af851f4110 0x55af851f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b4e7f640 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af851f4110 0x55af851f2e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e0b311ac996828 server_cookie=a602e75b793d6a6d in_seq=0 out_seq=0 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f878ffff640 10 client.15432 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f878ffff640 1 -- 172.21.15.16:0/967474581 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55af850d7b10 con 0x55af851f4110 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f878ffff640 10 client.15432 handle_command_reply: tid=0 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 client.15432 shutdown 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 msgr2=0x55af8518a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f879c009a40 tx=0x7f879c0157f0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 shutdown_connections 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55af851f4110 0x55af851f2e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55af85174bb0 0x55af8518a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af8516a2f0 0x55af8518a3b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 --2- 172.21.15.16:0/967474581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af851699d0 0x55af85189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 >> 172.21.15.16:0/967474581 conn(0x55af85165f60 msgr2=0x55af851843b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f878ffff640 0 client.15432 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 shutdown_connections 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 1 -- 172.21.15.16:0/967474581 wait complete. 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.713+0000 7f87b94bc740 20 client.15432 trim_cache size 0 max 16384 2024-09-12T02:48:04.724 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "ef86fd2a-70af-11ef-bcea-c7b262605968", 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 136.53203853400001, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 80, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 80, 2024-09-12T02:48:04.725 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 147.155364206 2024-09-12T02:48:04.726 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 msgr2=0x7f87a00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 0x7f87a00a50d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87a801f000 tx=0x7f87a8020000 comp rx=0 tx=0).stop 2024-09-12T02:48:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 msgr2=0x7f87b0198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f87a402a940 tx=0x7f87a4003c00 comp rx=0 tx=0).stop 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 shutdown_connections 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87b0106a80 0x7f87b019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87b0101420 0x7f87b01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87a00a2c10 0x7f87a00a50d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 --2- 172.21.15.16:0/2448821522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87b0100a80 0x7f87b0198460 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 >> 172.21.15.16:0/2448821522 conn(0x7f87b00fc7a0 msgr2=0x7f87b006a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 shutdown_connections 2024-09-12T02:48:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:04.714+0000 7f87b70e3640 1 -- 172.21.15.16:0/2448821522 wait complete. 2024-09-12T02:48:05.340 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-12T02:48:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:05 smithi033 ceph-mon[35092]: pgmap v342: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:48:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:05 smithi184 ceph-mon[35321]: pgmap v342: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:48:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:05 smithi016 ceph-mon[30925]: pgmap v342: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-12T02:48:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:07 smithi033 ceph-mon[35092]: pgmap v343: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:48:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:07 smithi184 ceph-mon[35321]: pgmap v343: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:48:08.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:07 smithi016 ceph-mon[30925]: pgmap v343: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-12T02:48:10.220 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:09 smithi016 ceph-mon[30925]: pgmap v344: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:09 smithi033 ceph-mon[35092]: pgmap v344: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:09 smithi184 ceph-mon[35321]: pgmap v344: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-09-12T02:48:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:11 smithi033 ceph-mon[35092]: pgmap v345: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:11 smithi184 ceph-mon[35321]: pgmap v345: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:12.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:11 smithi016 ceph-mon[30925]: pgmap v345: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-12T02:48:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:13 smithi033 ceph-mon[35092]: pgmap v346: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:13 smithi184 ceph-mon[35321]: pgmap v346: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:13 smithi016 ceph-mon[30925]: pgmap v346: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:15 smithi033 ceph-mon[35092]: pgmap v347: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:15 smithi184 ceph-mon[35321]: pgmap v347: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:15 smithi016 ceph-mon[30925]: pgmap v347: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:17.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:48:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:17 smithi033 ceph-mon[35092]: pgmap v348: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:17 smithi184 ceph-mon[35321]: pgmap v348: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:18.385 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:17 smithi016 ceph-mon[30925]: pgmap v348: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:18.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 -- 172.21.15.16:0/784619214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 msgr2=0x7fd944101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 --2- 172.21.15.16:0/784619214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd944101200 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd940009900 tx=0x7fd94002a480 comp rx=0 tx=0).stop 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 -- 172.21.15.16:0/784619214 shutdown_connections 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 --2- 172.21.15.16:0/784619214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 0x7fd94410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 --2- 172.21.15.16:0/784619214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 0x7fd9441029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 --2- 172.21.15.16:0/784619214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd944101200 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 -- 172.21.15.16:0/784619214 >> 172.21.15.16:0/784619214 conn(0x7fd9440fc7d0 msgr2=0x7fd9440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 -- 172.21.15.16:0/784619214 shutdown_connections 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.759+0000 7fd94c095640 1 -- 172.21.15.16:0/784619214 wait complete. 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 Processor -- start 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 -- start start 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 0x7fd944197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 0x7fd944197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd944111020 con 0x7fd944100e20 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd944110ea0 con 0x7fd944102ef0 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94c095640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441111a0 con 0x7fd944102570 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd949e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd949e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33356/0 (socket says 172.21.15.16:33356) 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd949e31640 1 -- 172.21.15.16:0/2661330116 learned_addr learned my addr 172.21.15.16:0/2661330116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd949630640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 0x7fd944197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.760+0000 7fd94a632640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 0x7fd944197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 -- 172.21.15.16:0/2661330116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 msgr2=0x7fd944197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 0x7fd944197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 -- 172.21.15.16:0/2661330116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 msgr2=0x7fd944197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 0x7fd944197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 -- 172.21.15.16:0/2661330116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9441a6b50 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd949e31640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd940006010 tx=0x7fd940003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd93affd640 1 -- 172.21.15.16:0/2661330116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94003f070 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9441a6de0 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd93affd640 1 -- 172.21.15.16:0/2661330116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd940031e30 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441a72c0 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.761+0000 7fd93affd640 1 -- 172.21.15.16:0/2661330116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940035030 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd93affd640 1 -- 172.21.15.16:0/2661330116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fd940042070 con 0x7fd944100e20 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 Processor -- start 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 -- start start 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 0x55683dc99880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc99dc0 0x55683dca4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55683dbff2e0 con 0x55683dca4680 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55683dba6ce0 con 0x55683dc994a0 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.764+0000 7fd94e46e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55683dbfb2a0 con 0x55683dc99dc0 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd94a632640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 0x55683dc99880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33358/0 (socket says 172.21.15.16:33358) 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 -- 172.21.15.16:0/2558486293 learned_addr learned my addr 172.21.15.16:0/2558486293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949e31640 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc99dc0 0x55683dca4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd93affd640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 0x7fd9300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 -- 172.21.15.16:0/2558486293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc99dc0 msgr2=0x55683dca4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc99dc0 0x55683dca4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 -- 172.21.15.16:0/2558486293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 msgr2=0x55683dc99880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 0x55683dc99880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 -- 172.21.15.16:0/2558486293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683db8b440 con 0x55683dca4680 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd94a632640 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 0x55683dc99880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd94a632640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 0x7fd9300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd93affd640 1 -- 172.21.15.16:0/2661330116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd9400efbc0 con 0x7fd944100e20 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.765+0000 7fd949630640 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd93400b7f0 tx=0x7fd93400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651ef05c10036e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd92ffff640 1 -- 172.21.15.16:0/2558486293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93400d950 con 0x55683dca4680 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd94a632640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 0x7fd9300a50d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009fd0 tx=0x7fd93c00c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd92ffff640 1 -- 172.21.15.16:0/2558486293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934018070 con 0x55683dca4680 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd92ffff640 1 -- 172.21.15.16:0/2558486293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9340123e0 con 0x55683dca4680 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd94e46e740 1 -- 172.21.15.16:0/2558486293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 msgr2=0x55683dca6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.766+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd93400b7f0 tx=0x7fd93400bcb0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 -- 172.21.15.16:0/2558486293 shutdown_connections 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca6a70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc99dc0 0x55683dca4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2558486293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc994a0 0x55683dc99880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 -- 172.21.15.16:0/2558486293 >> 172.21.15.16:0/2558486293 conn(0x55683dc95a30 msgr2=0x55683dc95e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 -- 172.21.15.16:0/2558486293 shutdown_connections 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 -- 172.21.15.16:0/2558486293 wait complete. 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 -- start start 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.767+0000 7fd94e46e740 1 Processor -- start 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55683dca7250 con 0x55683dca4680 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55683dbba050 con 0x55683dc99dc0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55683dca73d0 con 0x55683dc994a0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94e46e740 10 client.0 _ll_register_callbacks cb 0x55683dbfa160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33370/0 (socket says 172.21.15.16:33370) 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 -- 172.21.15.16:0/3048914300 learned_addr learned my addr 172.21.15.16:0/3048914300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd949630640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd949e31640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 -- 172.21.15.16:0/3048914300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 msgr2=0x55683dca3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 -- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 msgr2=0x55683dca11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 -- 172.21.15.16:0/3048914300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683d9703c0 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd949630640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd94a632640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd93c007f20 tx=0x7fd93c00e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576a18618f2650f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.768+0000 7fd92d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c027060 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd93c0190d0 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd934018070 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd93c017420 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 10 client.15444 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55683dcb2bd0 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934018070 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd93c022670 con 0x55683dca4680 2024-09-12T02:48:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 10 client.15444 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 10 client.15444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 20 client.15444 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dcb4090 0x55683dcb64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 4 client.15444 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.769+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55683dc0c850 con 0x55683dcb4090 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.770+0000 7fd949e31640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dcb4090 0x55683dcb64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.770+0000 7fd949e31640 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dcb4090 0x55683dcb64c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3159a15521b5632 server_cookie=de9f2fc94f7b6197 in_seq=0 out_seq=0 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.770+0000 7fd92d7fa640 10 client.15444 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd92d7fa640 1 -- 172.21.15.16:0/3048914300 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55683dc0c850 con 0x55683dcb4090 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd92d7fa640 10 client.15444 handle_command_reply: tid=0 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 client.15444 shutdown 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 msgr2=0x55683dca1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd93c007f20 tx=0x7fd93c00e040 comp rx=0 tx=0).stop 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 shutdown_connections 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dcb4090 0x55683dcb64c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dca4680 0x55683dca1710 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd92d7fa640 0 client.15444 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 --2- 172.21.15.16:0/3048914300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 >> 172.21.15.16:0/3048914300 conn(0x55683dc95a30 msgr2=0x55683dc9b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 shutdown_connections 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 1 -- 172.21.15.16:0/3048914300 wait complete. 2024-09-12T02:48:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.771+0000 7fd94e46e740 20 client.15444 trim_cache size 0 max 16384 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 Processor -- start 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 -- start start 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc95a30 0x55683dcb04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dcbbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441af890 con 0x55683dc994a0 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441b13d0 con 0x55683dc99dc0 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94e46e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441aff60 con 0x55683dc95a30 2024-09-12T02:48:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33376/0 (socket says 172.21.15.16:33376) 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 -- 172.21.15.16:0/1422631387 learned_addr learned my addr 172.21.15.16:0/1422631387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949e31640 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc95a30 0x55683dcb04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd94a632640 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dcbbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 -- 172.21.15.16:0/1422631387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc95a30 msgr2=0x55683dcb04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc95a30 0x55683dcb04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 -- 172.21.15.16:0/1422631387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 msgr2=0x55683dcbbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dcbbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.805+0000 7fd949630640 1 -- 172.21.15.16:0/1422631387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683dc95860 con 0x55683dc994a0 2024-09-12T02:48:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.806+0000 7fd949630640 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x55683dc96db0 tx=0x7fd934013be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b371524edf6404 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.806+0000 7fd92f7fe640 1 -- 172.21.15.16:0/1422631387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934022070 con 0x55683dc994a0 2024-09-12T02:48:18.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.806+0000 7fd92f7fe640 1 -- 172.21.15.16:0/1422631387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934018040 con 0x55683dc994a0 2024-09-12T02:48:18.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.806+0000 7fd92f7fe640 1 -- 172.21.15.16:0/1422631387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93401c650 con 0x55683dc994a0 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- 172.21.15.16:0/1422631387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 msgr2=0x55683dc9c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x55683dc96db0 tx=0x7fd934013be0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- 172.21.15.16:0/1422631387 shutdown_connections 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dcbbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc994a0 0x55683dc9c7f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 --2- 172.21.15.16:0/1422631387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc95a30 0x55683dcb04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- 172.21.15.16:0/1422631387 >> 172.21.15.16:0/1422631387 conn(0x55683dca1870 msgr2=0x55683dc9c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- 172.21.15.16:0/1422631387 shutdown_connections 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- 172.21.15.16:0/1422631387 wait complete. 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 -- start start 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.807+0000 7fd94e46e740 1 Processor -- start 2024-09-12T02:48:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441b0d00 con 0x55683dc95a30 2024-09-12T02:48:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441b0630 con 0x55683dc99dc0 2024-09-12T02:48:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ad870 con 0x55683dc994a0 2024-09-12T02:48:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94e46e740 10 client.0 _ll_register_callbacks cb 0x7fd93c009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33378/0 (socket says 172.21.15.16:33378) 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 -- 172.21.15.16:0/2206950405 learned_addr learned my addr 172.21.15.16:0/2206950405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949630640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94a632640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 -- 172.21.15.16:0/2206950405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 msgr2=0x55683dca80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 -- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 msgr2=0x55683dca8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd949e31640 1 -- 172.21.15.16:0/2206950405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683dca8e20 con 0x55683dc95a30 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.808+0000 7fd94a632640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd949e31640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd9400bc260 tx=0x7fd9400dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b3a25f680424fe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:18.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940049040 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd9400b8940 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd934018040 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9400bc2d0 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55683dd1ee00 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934018040 con 0x55683dc95a30 2024-09-12T02:48:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd9400d50a0 con 0x55683dc95a30 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 10 client.15450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 20 client.15450 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dca4680 0x55683dd21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 4 client.15450 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:48:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.809+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55683dc06d60 con 0x55683dca4680 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd949630640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dca4680 0x55683dd21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd949630640 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dca4680 0x55683dd21510 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b1321909dd0317 server_cookie=ade29c173963a333 in_seq=0 out_seq=0 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd92d7fa640 10 client.15450 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd92d7fa640 1 -- 172.21.15.16:0/2206950405 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55683dc06d60 con 0x55683dca4680 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd92d7fa640 10 client.15450 handle_command_reply: tid=0 2024-09-12T02:48:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.810+0000 7fd94e46e740 1 client.15450 shutdown 2024-09-12T02:48:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 msgr2=0x55683dcacdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd9400bc260 tx=0x7fd9400dc040 comp rx=0 tx=0).stop 2024-09-12T02:48:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 shutdown_connections 2024-09-12T02:48:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55683dca4680 0x55683dd21510 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55683dc99dc0 0x55683dca8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55683dc994a0 0x55683dca80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd92d7fa640 0 client.15450 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 --2- 172.21.15.16:0/2206950405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55683dc95a30 0x55683dcacdb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 >> 172.21.15.16:0/2206950405 conn(0x55683dca1870 msgr2=0x55683dca2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 shutdown_connections 2024-09-12T02:48:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 1 -- 172.21.15.16:0/2206950405 wait complete. 2024-09-12T02:48:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94e46e740 20 client.15450 trim_cache size 0 max 16384 2024-09-12T02:48:18.826 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:18.826 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:48:18.826 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:48:18.826 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 msgr2=0x7fd9300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 0x7fd9300a50d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009fd0 tx=0x7fd93c00c040 comp rx=0 tx=0).stop 2024-09-12T02:48:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 msgr2=0x7fd9441972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.811+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd940006010 tx=0x7fd940003c80 comp rx=0 tx=0).stop 2024-09-12T02:48:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 shutdown_connections 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd944102ef0 0x7fd944197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd944102570 0x7fd944197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9300a2c10 0x7fd9300a50d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 --2- 172.21.15.16:0/2661330116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd944100e20 0x7fd9441972e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 >> 172.21.15.16:0/2661330116 conn(0x7fd9440fc7d0 msgr2=0x7fd9440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 shutdown_connections 2024-09-12T02:48:18.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:18.812+0000 7fd94c095640 1 -- 172.21.15.16:0/2661330116 wait complete. 2024-09-12T02:48:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:19 smithi033 ceph-mon[35092]: pgmap v349: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:19 smithi184 ceph-mon[35321]: pgmap v349: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:20.341 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-12T02:48:20.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:48:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:19 smithi016 ceph-mon[30925]: pgmap v349: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:21 smithi033 ceph-mon[35092]: pgmap v350: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:48:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:21 smithi184 ceph-mon[35321]: pgmap v350: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:48:22.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:21 smithi016 ceph-mon[30925]: pgmap v350: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-12T02:48:22.375 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:48:22.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:48:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 -- 172.21.15.16:0/140900836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101270 msgr2=0x7fdef010d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 --2- 172.21.15.16:0/140900836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101270 0x7fdef010d7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009cd0 tx=0x7fdee802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 -- 172.21.15.16:0/140900836 shutdown_connections 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 --2- 172.21.15.16:0/140900836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101270 0x7fdef010d7a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 --2- 172.21.15.16:0/140900836 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 0x7fdef0100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 --2- 172.21.15.16:0/140900836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef01068d0 0x7fdef0106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.018+0000 7fdef5659640 1 -- 172.21.15.16:0/140900836 >> 172.21.15.16:0/140900836 conn(0x7fdef00fc610 msgr2=0x7fdef00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.019+0000 7fdef5659640 1 -- 172.21.15.16:0/140900836 shutdown_connections 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.019+0000 7fdef5659640 1 -- 172.21.15.16:0/140900836 wait complete. 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.019+0000 7fdef5659640 1 Processor -- start 2024-09-12T02:48:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.019+0000 7fdef5659640 1 -- start start 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 0x7fdef0197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef0101270 0x7fdef01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0110cd0 con 0x7fdef01068d0 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0110b50 con 0x7fdef01008d0 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef5659640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0110e50 con 0x7fdef0101270 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33386/0 (socket says 172.21.15.16:33386) 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 -- 172.21.15.16:0/3828709168 learned_addr learned my addr 172.21.15.16:0/3828709168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdeeffff640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 0x7fdef0197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdeef7fe640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef0101270 0x7fdef01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 -- 172.21.15.16:0/3828709168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef0101270 msgr2=0x7fdef01976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef0101270 0x7fdef01976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 -- 172.21.15.16:0/3828709168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 msgr2=0x7fdef0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 0x7fdef0197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 -- 172.21.15.16:0/3828709168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef019c120 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.020+0000 7fdef4e58640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdee8005cc0 tx=0x7fdee8031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.021+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee804a070 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.021+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee8004040 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.021+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef019c3b0 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.021+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80308d0 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.021+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef0069f70 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.022+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef0102010 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.023+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fdee80041f0 con 0x7fdef01068d0 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.024+0000 7fdeed7fa640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 0x7fded80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.024+0000 7fdeeffff640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 0x7fded80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.024+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdee802e020 con 0x7fdef01068d0 2024-09-12T02:48:23.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.025+0000 7fdeeffff640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 0x7fded80a50d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdee00058c0 tx=0x7fdee000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:23.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.028+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdee80811b0 con 0x7fdef01068d0 2024-09-12T02:48:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:23.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.271+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdef006aa50 con 0x7fdef01068d0 2024-09-12T02:48:23.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.272+0000 7fdeed7fa640 1 -- 172.21.15.16:0/3828709168 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7fdee80bdd50 con 0x7fdef01068d0 2024-09-12T02:48:23.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:48:23.273 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:48:23.273 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 msgr2=0x7fded80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 0x7fded80a50d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdee00058c0 tx=0x7fdee000a250 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 msgr2=0x7fdef019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdee8005cc0 tx=0x7fdee8031e60 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 shutdown_connections 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fded80a2c10 0x7fded80a50d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01068d0 0x7fdef019ba40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdef0101270 0x7fdef01976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 --2- 172.21.15.16:0/3828709168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef01008d0 0x7fdef0197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.274+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 >> 172.21.15.16:0/3828709168 conn(0x7fdef00fc610 msgr2=0x7fdef00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.275+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 shutdown_connections 2024-09-12T02:48:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.275+0000 7fdef5659640 1 -- 172.21.15.16:0/3828709168 wait complete. 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 -- 172.21.15.16:0/2755515397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 msgr2=0x7f4b0810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2755515397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0810ec30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009930 tx=0x7f4b0002a420 comp rx=0 tx=0).stop 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 -- 172.21.15.16:0/2755515397 shutdown_connections 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2755515397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0810ec30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2755515397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 0x7f4b08107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2755515397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 0x7f4b08103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.833+0000 7f4b0f352640 1 -- 172.21.15.16:0/2755515397 >> 172.21.15.16:0/2755515397 conn(0x7f4b080fc7d0 msgr2=0x7f4b080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.834+0000 7f4b0f352640 1 -- 172.21.15.16:0/2755515397 shutdown_connections 2024-09-12T02:48:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.834+0000 7f4b0f352640 1 -- 172.21.15.16:0/2755515397 wait complete. 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.834+0000 7f4b0f352640 1 Processor -- start 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.834+0000 7f4b0f352640 1 -- start start 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 0x7f4b08198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 0x7f4b08198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081120a0 con 0x7f4b08108290 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08111f20 con 0x7f4b08103740 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0f352640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08112220 con 0x7f4b08102d90 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33406/0 (socket says 172.21.15.16:33406) 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 -- 172.21.15.16:0/2538563892 learned_addr learned my addr 172.21.15.16:0/2538563892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d0ee640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 0x7f4b08198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0c8ed640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 0x7f4b08198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 -- 172.21.15.16:0/2538563892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 msgr2=0x7f4b08198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 0x7f4b08198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 -- 172.21.15.16:0/2538563892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 msgr2=0x7f4b08198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 0x7f4b08198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.835+0000 7f4b0d8ef640 1 -- 172.21.15.16:0/2538563892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0819d530 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b0d8ef640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b00004770 tx=0x7f4b00003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0003f070 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b00031c90 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0819d7c0 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00035030 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.836+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b081a5100 con 0x7f4b08108290 2024-09-12T02:48:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.837+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b08103170 con 0x7f4b08108290 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.839+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f4b00042070 con 0x7f4b08108290 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.840+0000 7f4b067fc640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 0x7f4af00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.840+0000 7f4b0d0ee640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 0x7f4af00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.840+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b000ef4d0 con 0x7f4b08108290 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.840+0000 7f4b0d0ee640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 0x7f4af00a5150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005c40 tx=0x7f4af800a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:23.844+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4b000bbc90 con 0x7f4b08108290 2024-09-12T02:48:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.077+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b081998e0 con 0x7f4b08108290 2024-09-12T02:48:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.078+0000 7f4b067fc640 1 -- 172.21.15.16:0/2538563892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7f4b081998e0 con 0x7f4b08108290 2024-09-12T02:48:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:48:24.078 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:48:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:48:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 msgr2=0x7f4af00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 0x7f4af00a5150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005c40 tx=0x7f4af800a590 comp rx=0 tx=0).stop 2024-09-12T02:48:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 msgr2=0x7f4b0819ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b00004770 tx=0x7f4b00003b90 comp rx=0 tx=0).stop 2024-09-12T02:48:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 shutdown_connections 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4af00a2c90 0x7f4af00a5150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b08108290 0x7f4b0819ce50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b08103740 0x7f4b08198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 --2- 172.21.15.16:0/2538563892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b08102d90 0x7f4b08198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 >> 172.21.15.16:0/2538563892 conn(0x7f4b080fc7d0 msgr2=0x7f4b08106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 shutdown_connections 2024-09-12T02:48:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:24.080+0000 7f4b0f352640 1 -- 172.21.15.16:0/2538563892 wait complete. 2024-09-12T02:48:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:23 smithi033 ceph-mon[35092]: pgmap v351: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:23 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3828709168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:23 smithi184 ceph-mon[35321]: pgmap v351: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:23 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3828709168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:24.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:23 smithi016 ceph-mon[30925]: pgmap v351: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:24.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:23 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3828709168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:24.380 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-12T02:48:24.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-12T02:48:25.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:24 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2538563892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:25.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:48:25.155 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:25.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:24 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2538563892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:24 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2538563892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 -- 172.21.15.16:0/912711096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 msgr2=0x7fc0f8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 --2- 172.21.15.16:0/912711096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8107ba0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009930 tx=0x7fc0e802a420 comp rx=0 tx=0).stop 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 -- 172.21.15.16:0/912711096 shutdown_connections 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 --2- 172.21.15.16:0/912711096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f81082e0 0x7fc0f810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 --2- 172.21.15.16:0/912711096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8107ba0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 --2- 172.21.15.16:0/912711096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f8102db0 0x7fc0f8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.670+0000 7fc0ff853640 1 -- 172.21.15.16:0/912711096 >> 172.21.15.16:0/912711096 conn(0x7fc0f80fc7d0 msgr2=0x7fc0f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.671+0000 7fc0ff853640 1 -- 172.21.15.16:0/912711096 shutdown_connections 2024-09-12T02:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.671+0000 7fc0ff853640 1 -- 172.21.15.16:0/912711096 wait complete. 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.671+0000 7fc0ff853640 1 Processor -- start 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 -- start start 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 0x7fc0f81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f81082e0 0x7fc0f819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8112050 con 0x7fc0f8103760 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8111ed0 con 0x7fc0f8102db0 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0ff853640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81121d0 con 0x7fc0f81082e0 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43452/0 (socket says 172.21.15.16:43452) 2024-09-12T02:48:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 -- 172.21.15.16:0/2546029543 learned_addr learned my addr 172.21.15.16:0/2546029543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 0x7fc0f81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f81082e0 0x7fc0f819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 -- 172.21.15.16:0/2546029543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f81082e0 msgr2=0x7fc0f819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f81082e0 0x7fc0f819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 -- 172.21.15.16:0/2546029543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 msgr2=0x7fc0f81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 0x7fc0f81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fcdee640 1 -- 172.21.15.16:0/2546029543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f819d560 con 0x7fc0f8103760 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.672+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 0x7fc0f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0fcdee640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8005940 tx=0x7fc0e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0ee7fc640 1 -- 172.21.15.16:0/2546029543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e803f070 con 0x7fc0f8103760 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f819d7f0 con 0x7fc0f8103760 2024-09-12T02:48:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0ee7fc640 1 -- 172.21.15.16:0/2546029543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e8031c30 con 0x7fc0f8103760 2024-09-12T02:48:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0ee7fc640 1 -- 172.21.15.16:0/2546029543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8035030 con 0x7fc0f8103760 2024-09-12T02:48:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.673+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f8100940 con 0x7fc0f8103760 2024-09-12T02:48:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.677+0000 7fc0ee7fc640 1 -- 172.21.15.16:0/2546029543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fc0e8042070 con 0x7fc0f8103760 2024-09-12T02:48:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.677+0000 7fc101c2c740 1 Processor -- start 2024-09-12T02:48:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.677+0000 7fc101c2c740 1 -- start start 2024-09-12T02:48:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 0x5565e630b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0ee7fc640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 0x7fc0e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e630dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565e62662a0 con 0x5565e6300760 2024-09-12T02:48:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565e620db10 con 0x5565e6301080 2024-09-12T02:48:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc101c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5565e6262260 con 0x5565e630b940 2024-09-12T02:48:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0ee7fc640 1 -- 172.21.15.16:0/2546029543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc0e80ef9b0 con 0x7fc0f8103760 2024-09-12T02:48:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0fddf0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0fddf0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43454/0 (socket says 172.21.15.16:43454) 2024-09-12T02:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0fddf0640 1 -- 172.21.15.16:0/2228488410 learned_addr learned my addr 172.21.15.16:0/2228488410 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.678+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 0x5565e630b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fcdee640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 0x7fc0e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/2228488410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 msgr2=0x5565e630dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e630dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/2228488410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 msgr2=0x5565e6300b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/2228488410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e5fd7b90 con 0x5565e6301080 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 0x5565e630b400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009fd0 tx=0x7fc0f40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd91cac4e6ff847 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0e37fe640 1 -- 172.21.15.16:0/2228488410 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f401e070 con 0x5565e6301080 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0e37fe640 1 -- 172.21.15.16:0/2228488410 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f40035f0 con 0x5565e6301080 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.679+0000 7fc0e37fe640 1 -- 172.21.15.16:0/2228488410 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f400f800 con 0x5565e6301080 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc0fcdee640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 0x7fc0e40a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80b8520 tx=0x7fc0e8009230 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 -- 172.21.15.16:0/2228488410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 msgr2=0x5565e630b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 0x5565e630b400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009fd0 tx=0x7fc0f40118d0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 -- 172.21.15.16:0/2228488410 shutdown_connections 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e630dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6301080 0x5565e630b400 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 --2- 172.21.15.16:0/2228488410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6300760 0x5565e6300b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.680+0000 7fc101c2c740 1 -- 172.21.15.16:0/2228488410 >> 172.21.15.16:0/2228488410 conn(0x5565e62fccf0 msgr2=0x5565e62fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.681+0000 7fc101c2c740 1 -- 172.21.15.16:0/2228488410 shutdown_connections 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.681+0000 7fc101c2c740 1 -- 172.21.15.16:0/2228488410 wait complete. 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.681+0000 7fc101c2c740 1 -- start start 2024-09-12T02:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 Processor -- start 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6308890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6309310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565e630e510 con 0x5565e6301080 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565e61ee6e0 con 0x5565e6300760 2024-09-12T02:48:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5565e630e690 con 0x5565e630b940 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc101c2c740 10 client.0 _ll_register_callbacks cb 0x5565e6212a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43468/0 (socket says 172.21.15.16:43468) 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/3247672549 learned_addr learned my addr 172.21.15.16:0/3247672549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.682+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6308890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6309310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/3247672549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 msgr2=0x5565e6309310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6309310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 msgr2=0x5565e6308890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6308890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/3247672549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e6317a80 con 0x5565e6301080 2024-09-12T02:48:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6309310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.683+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009730 tx=0x7fc0f40041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1ea194838d23ad server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f401e040 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc0f4004310 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0f40035f0 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc0f4005040 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc101c2c740 10 client.15474 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.684+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5565e631a3f0 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f40035f0 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc0f4021030 con 0x5565e6301080 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 10 client.15474 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 10 client.15474 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 20 client.15474 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e631b7a0 0x5565e631db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 4 client.15474 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565e62739d0 con 0x5565e631b7a0 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.685+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e631b7a0 0x5565e631db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.686+0000 7fc0e0ff9640 10 client.15474 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.686+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e631b7a0 0x5565e631db90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7970003d6a814f16 server_cookie=60a8f0bfe409413f in_seq=0 out_seq=0 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/3247672549 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565e62739d0 con 0x5565e631b7a0 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc0e0ff9640 10 client.15474 handle_command_reply: tid=0 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 client.15474 shutdown 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 msgr2=0x5565e6308dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009730 tx=0x7fc0f40041a0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 shutdown_connections 2024-09-12T02:48:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e631b7a0 0x5565e631db90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6309310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.687+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e6308dd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc101c2c740 1 --2- 172.21.15.16:0/3247672549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6308890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 >> 172.21.15.16:0/3247672549 conn(0x5565e62fccf0 msgr2=0x5565e6302a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc0e0ff9640 0 client.15474 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 shutdown_connections 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc101c2c740 1 -- 172.21.15.16:0/3247672549 wait complete. 2024-09-12T02:48:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.688+0000 7fc101c2c740 20 client.15474 trim_cache size 0 max 16384 2024-09-12T02:48:25.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.717+0000 7fc101c2c740 1 Processor -- start 2024-09-12T02:48:25.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.717+0000 7fc101c2c740 1 -- start start 2024-09-12T02:48:25.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.717+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6303db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.717+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc0fddf0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6303db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc0fddf0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6303db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47614/0 (socket says 172.21.15.16:47614) 2024-09-12T02:48:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6323280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc0fddf0640 1 -- 172.21.15.16:0/1463348364 learned_addr learned my addr 172.21.15.16:0/1463348364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43478/0 (socket says 172.21.15.16:43478) 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81af890 con 0x5565e6301080 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81b13d0 con 0x5565e6300760 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.718+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81aff60 con 0x5565e630b940 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/1463348364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 msgr2=0x5565e6323280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6323280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/1463348364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 msgr2=0x5565e6303db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6303db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/1463348364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e62fde60 con 0x5565e6301080 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.719+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x5565e6309e40 tx=0x7fc0f4004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491e6639b11a4ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.720+0000 7fc0e2ffd640 1 -- 172.21.15.16:0/1463348364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f401e040 con 0x5565e6301080 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.720+0000 7fc0e2ffd640 1 -- 172.21.15.16:0/1463348364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f4016070 con 0x5565e6301080 2024-09-12T02:48:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.720+0000 7fc0e2ffd640 1 -- 172.21.15.16:0/1463348364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f40055e0 con 0x5565e6301080 2024-09-12T02:48:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 msgr2=0x5565e63077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x5565e6309e40 tx=0x7fc0f4004f20 comp rx=0 tx=0).stop 2024-09-12T02:48:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 shutdown_connections 2024-09-12T02:48:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6323280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e6301080 0x5565e63077d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 --2- 172.21.15.16:0/1463348364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6303db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 >> 172.21.15.16:0/1463348364 conn(0x5565e62fccf0 msgr2=0x5565e630a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 shutdown_connections 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.721+0000 7fc101c2c740 1 -- 172.21.15.16:0/1463348364 wait complete. 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.722+0000 7fc101c2c740 1 -- start start 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.722+0000 7fc101c2c740 1 Processor -- start 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.722+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6305720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.722+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.722+0000 7fc101c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 0x5565e6385250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc101c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81b0d00 con 0x5565e63061a0 2024-09-12T02:48:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc101c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81b0630 con 0x5565e6300760 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc101c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81ad870 con 0x5565e630b940 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fddf0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6305720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46540/0 (socket says 172.21.15.16:46540) 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/613682441 learned_addr learned my addr 172.21.15.16:0/613682441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 0x5565e6385250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc101c2c740 10 client.0 _ll_register_callbacks cb 0x7fc0f4009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 msgr2=0x5565e6305720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6305720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/613682441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 msgr2=0x5565e6385250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 0x5565e6385250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 -- 172.21.15.16:0/613682441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e6314550 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 0x5565e6385250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0fd5ef640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4011910 tx=0x7fc0f40057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba361a3a8312509b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.723+0000 7fc0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f401e040 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0f4016070 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc0f400fb60 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc0f4048050 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc101c2c740 10 client.25057 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5565e63867d0 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.724+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f4016070 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.725+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc0f4037020 con 0x5565e630b940 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 10 client.25057 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 10 client.25057 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:48:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 20 client.25057 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e6386e50 0x5565e6389240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 4 client.25057 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x5565e626dd20 con 0x5565e6386e50 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e6386e50 0x5565e6389240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc0fddf0640 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e6386e50 0x5565e6389240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eb87619a5c38dcf server_cookie=aa863ba5fb55ec17 in_seq=0 out_seq=0 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.726+0000 7fc0e0ff9640 10 client.25057 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:25.731 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.739+0000 7fc0e0ff9640 1 -- 172.21.15.16:0/613682441 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5565e626dd20 con 0x5565e6386e50 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.739+0000 7fc0e0ff9640 10 client.25057 handle_command_reply: tid=0 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.739+0000 7fc101c2c740 1 client.25057 shutdown 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.739+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 msgr2=0x5565e6305c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.739+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4011910 tx=0x7fc0f40057a0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 shutdown_connections 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5565e6386e50 0x5565e6389240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e63061a0 0x5565e6385250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5565e630b940 0x5565e6305c60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc0e0ff9640 0 client.25057 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 --2- 172.21.15.16:0/613682441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565e6300760 0x5565e6305720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 >> 172.21.15.16:0/613682441 conn(0x5565e62fccf0 msgr2=0x5565e62fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 shutdown_connections 2024-09-12T02:48:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 1 -- 172.21.15.16:0/613682441 wait complete. 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.740+0000 7fc101c2c740 20 client.25057 trim_cache size 0 max 16384 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 2, 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "5ce30e01": { 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 2, 2024-09-12T02:48:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:48:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 msgr2=0x7fc0e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 0x7fc0e40a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80b8520 tx=0x7fc0e8009230 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 msgr2=0x7fc0f8198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8005940 tx=0x7fc0e8003c10 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 shutdown_connections 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0f81082e0 0x7fc0f819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc0e40a2c10 0x7fc0e40a50d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f8103760 0x7fc0f8198af0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 --2- 172.21.15.16:0/2546029543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0f8102db0 0x7fc0f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.742+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 >> 172.21.15.16:0/2546029543 conn(0x7fc0f80fc7d0 msgr2=0x7fc0f8105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.743+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 shutdown_connections 2024-09-12T02:48:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:25.743+0000 7fc0ff853640 1 -- 172.21.15.16:0/2546029543 wait complete. 2024-09-12T02:48:26.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:25 smithi016 ceph-mon[30925]: pgmap v352: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:25 smithi033 ceph-mon[35092]: pgmap v352: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:25 smithi184 ceph-mon[35321]: pgmap v352: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-12T02:48:26.442 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5ce30e01', quiesce took 6.1 seconds. Will release after: 18.4 2024-09-12T02:48:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 -- 172.21.15.16:0/1391869360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 msgr2=0x7fe68810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 --2- 172.21.15.16:0/1391869360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68810ec30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe680009cd0 tx=0x7fe68002a8f0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 -- 172.21.15.16:0/1391869360 shutdown_connections 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 --2- 172.21.15.16:0/1391869360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68810ec30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 --2- 172.21.15.16:0/1391869360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688103740 0x7fe688107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 --2- 172.21.15.16:0/1391869360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688102d90 0x7fe688103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.989+0000 7fe68d494640 1 -- 172.21.15.16:0/1391869360 >> 172.21.15.16:0/1391869360 conn(0x7fe6880fc7d0 msgr2=0x7fe6880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.990+0000 7fe68d494640 1 -- 172.21.15.16:0/1391869360 shutdown_connections 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.990+0000 7fe68d494640 1 -- 172.21.15.16:0/1391869360 wait complete. 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.991+0000 7fe68d494640 1 Processor -- start 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 -- start start 2024-09-12T02:48:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 0x7fe68819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 0x7fe68819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe688114260 con 0x7fe688108290 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881140e0 con 0x7fe688102d90 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe68d494640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881143e0 con 0x7fe688103740 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43514/0 (socket says 172.21.15.16:43514) 2024-09-12T02:48:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.992+0000 7fe6877fe640 1 -- 172.21.15.16:0/3565126663 learned_addr learned my addr 172.21.15.16:0/3565126663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe6867fc640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 0x7fe68819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 0x7fe68819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 -- 172.21.15.16:0/3565126663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 msgr2=0x7fe68819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 0x7fe68819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 -- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 msgr2=0x7fe68819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.993+0000 7fe686ffd640 1 -- 172.21.15.16:0/3565126663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68819f6d0 con 0x7fe688102d90 2024-09-12T02:48:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe6877fe640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe6867fc640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 0x7fe68819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 0x7fe68819a720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe67800e990 tx=0x7fe67800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe66ffff640 1 -- 172.21.15.16:0/3565126663 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67800cbe0 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe66ffff640 1 -- 172.21.15.16:0/3565126663 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe67800cd80 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68819f9c0 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe66ffff640 1 -- 172.21.15.16:0/3565126663 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67800cbe0 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.994+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6881a72a0 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.997+0000 7fe66ffff640 1 -- 172.21.15.16:0/3565126663 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fe678011470 con 0x7fe688102d90 2024-09-12T02:48:27.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 Processor -- start 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 -- start start 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c6ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c7b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558453bd42f0 con 0x558453c79370 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558453b7bcf0 con 0x558453c6fb40 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe68f86d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558453bd02b0 con 0x558453c6e5d0 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47648/0 (socket says 172.21.15.16:47648) 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.998+0000 7fe6877fe640 1 -- 172.21.15.16:0/3325604727 learned_addr learned my addr 172.21.15.16:0/3325604727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c6ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 -- 172.21.15.16:0/3325604727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 msgr2=0x558453c6ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c6ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 -- 172.21.15.16:0/3325604727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 msgr2=0x558453c7b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c7b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 -- 172.21.15.16:0/3325604727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558453b603a0 con 0x558453c6fb40 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe66ffff640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 0x7fe6700a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe6877fe640 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe680030890 tx=0x7fe680009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54642a07a67759d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:26.999+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 0x7fe6700a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe66d7fa640 1 -- 172.21.15.16:0/3325604727 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680003970 con 0x558453c6fb40 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe66d7fa640 1 -- 172.21.15.16:0/3325604727 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe680036070 con 0x558453c6fb40 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe66d7fa640 1 -- 172.21.15.16:0/3325604727 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680031da0 con 0x558453c6fb40 2024-09-12T02:48:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe66ffff640 1 -- 172.21.15.16:0/3565126663 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe6780cefa0 con 0x7fe688102d90 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe68f86d740 1 -- 172.21.15.16:0/3325604727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 msgr2=0x558453c6e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe686ffd640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 0x7fe6700a5310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x558453c6bac0 tx=0x7fe6780d4040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.000+0000 7fe68f86d740 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe680030890 tx=0x7fe680009a20 comp rx=0 tx=0).stop 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 -- 172.21.15.16:0/3325604727 shutdown_connections 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c7b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c6ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 --2- 172.21.15.16:0/3325604727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c6e090 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 -- 172.21.15.16:0/3325604727 >> 172.21.15.16:0/3325604727 conn(0x558453c37310 msgr2=0x558453c6ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 -- 172.21.15.16:0/3325604727 shutdown_connections 2024-09-12T02:48:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 -- 172.21.15.16:0/3325604727 wait complete. 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 -- start start 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.001+0000 7fe68f86d740 1 Processor -- start 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c78d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c76020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c76610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c78d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c78d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46580/0 (socket says 172.21.15.16:46580) 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe6867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c76020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe6877fe640 1 -- 172.21.15.16:0/4088982138 learned_addr learned my addr 172.21.15.16:0/4088982138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558453c7c0b0 con 0x558453c79370 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558453b8f060 con 0x558453c6fb40 2024-09-12T02:48:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558453c7c230 con 0x558453c6e5d0 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.002+0000 7fe6877fe640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c76610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 -- 172.21.15.16:0/4088982138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 msgr2=0x558453c78d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c78d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 -- 172.21.15.16:0/4088982138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 msgr2=0x558453c76020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c76020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584539453c0 con 0x558453c79370 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe6877fe640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c76610 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe680030860 tx=0x7fe680000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31cd78968a41d1a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe66aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe68f86d740 10 client.0 _ll_register_callbacks cb 0x5584538e28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe66aff5640 1 -- 172.21.15.16:0/4088982138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680031d90 con 0x558453c79370 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe66aff5640 1 -- 172.21.15.16:0/4088982138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe680003970 con 0x558453c79370 2024-09-12T02:48:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.003+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe680036070 con 0x558453c79370 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe66aff5640 1 -- 172.21.15.16:0/4088982138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe680003b70 con 0x558453c79370 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x558453c7c140 con 0x558453c79370 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe66aff5640 1 -- 172.21.15.16:0/4088982138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe680051070 con 0x558453c79370 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 10 client.15489 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 20 client.15489 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453c90ff0 0x558453c933e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 4 client.15489 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.004+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558453be1860 con 0x558453c90ff0 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.005+0000 7fe6867fc640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453c90ff0 0x558453c933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.005+0000 7fe66aff5640 10 client.15489 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:27.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.005+0000 7fe6867fc640 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453c90ff0 0x558453c933e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=707372f09532069c server_cookie=c0c933232f57075b in_seq=0 out_seq=0 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.006+0000 7fe66aff5640 1 -- 172.21.15.16:0/4088982138 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558453be1860 con 0x558453c90ff0 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.006+0000 7fe66aff5640 10 client.15489 handle_command_reply: tid=0 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.006+0000 7fe68f86d740 1 client.15489 shutdown 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 msgr2=0x558453c76610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c76610 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe680030860 tx=0x7fe680000f40 comp rx=0 tx=0).stop 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 shutdown_connections 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c76610 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453c90ff0 0x558453c933e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6fb40 0x558453c76020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 --2- 172.21.15.16:0/4088982138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6e5d0 0x558453c78d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 >> 172.21.15.16:0/4088982138 conn(0x558453c37310 msgr2=0x558453c70960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe66aff5640 0 client.15489 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:27.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 shutdown_connections 2024-09-12T02:48:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 1 -- 172.21.15.16:0/4088982138 wait complete. 2024-09-12T02:48:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.007+0000 7fe68f86d740 20 client.15489 trim_cache size 0 max 16384 2024-09-12T02:48:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.031+0000 7fe68f86d740 1 Processor -- start 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.031+0000 7fe68f86d740 1 -- start start 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.031+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c8dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c98ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c8dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43540/0 (socket says 172.21.15.16:43540) 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 -- 172.21.15.16:0/1693040127 learned_addr learned my addr 172.21.15.16:0/1693040127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881ab390 con 0x558453c79370 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881aced0 con 0x558453c6e5d0 2024-09-12T02:48:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881aba60 con 0x558453c6fb40 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6867fc640 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c98ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 -- 172.21.15.16:0/1693040127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 msgr2=0x558453c98ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c98ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 -- 172.21.15.16:0/1693040127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 msgr2=0x558453c8dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c8dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 -- 172.21.15.16:0/1693040127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558453c78890 con 0x558453c79370 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.032+0000 7fe6877fe640 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe680009270 tx=0x7fe68003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b148335f31f0af server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.033+0000 7fe6867fc640 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c98ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.033+0000 7fe66cff9640 1 -- 172.21.15.16:0/1693040127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680030c40 con 0x558453c79370 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.033+0000 7fe66cff9640 1 -- 172.21.15.16:0/1693040127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe680036070 con 0x558453c79370 2024-09-12T02:48:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.033+0000 7fe66cff9640 1 -- 172.21.15.16:0/1693040127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800535c0 con 0x558453c79370 2024-09-12T02:48:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 msgr2=0x558453c991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe680009270 tx=0x7fe68003e070 comp rx=0 tx=0).stop 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 shutdown_connections 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c991e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c98ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 --2- 172.21.15.16:0/1693040127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c8dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 >> 172.21.15.16:0/1693040127 conn(0x558453c37310 msgr2=0x558453c71290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 shutdown_connections 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.034+0000 7fe68f86d740 1 -- 172.21.15.16:0/1693040127 wait complete. 2024-09-12T02:48:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.035+0000 7fe68f86d740 1 -- start start 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.035+0000 7fe68f86d740 1 Processor -- start 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.035+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c89d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c8a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881ac800 con 0x558453c79370 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43550/0 (socket says 172.21.15.16:43550) 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 -- 172.21.15.16:0/203799974 learned_addr learned my addr 172.21.15.16:0/203799974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6867fc640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c8a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881ac130 con 0x558453c6e5d0 2024-09-12T02:48:27.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6881a93b0 con 0x558453c6fb40 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe68f86d740 10 client.0 _ll_register_callbacks cb 0x7fe680031570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c89d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 -- 172.21.15.16:0/203799974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 msgr2=0x558453c8a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c8a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 -- 172.21.15.16:0/203799974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 msgr2=0x558453c89d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c89d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6877fe640 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558453c87260 con 0x558453c79370 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.036+0000 7fe6867fc640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c8a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe6877fe640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe680009620 tx=0x7fe680003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fec7502ba3ccb74 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe66aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680031920 con 0x558453c79370 2024-09-12T02:48:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe680003970 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe680036070 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe680053540 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe68f86d740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.037+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558453cfc3f0 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe680036070 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe680051040 con 0x558453c79370 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 10 client.15495 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 20 client.15495 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453cfc870 0x558453cfec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 4 client.15495 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558453bdbd70 con 0x558453cfc870 2024-09-12T02:48:27.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.038+0000 7fe6867fc640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453cfc870 0x558453cfec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe6867fc640 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453cfc870 0x558453cfec60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c8660294a0d21a79 server_cookie=fc2ec73bd57d2028 in_seq=0 out_seq=0 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe66aff5640 10 client.15495 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe66aff5640 1 -- 172.21.15.16:0/203799974 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558453bdbd70 con 0x558453cfc870 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe66aff5640 10 client.15495 handle_command_reply: tid=0 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe68f86d740 1 client.15495 shutdown 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 msgr2=0x558453c86d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.039+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe680009620 tx=0x7fe680003e30 comp rx=0 tx=0).stop 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 shutdown_connections 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558453c79370 0x558453c86d20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558453c6fb40 0x558453c8a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x558453cfc870 0x558453cfec60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe66aff5640 0 client.15495 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 --2- 172.21.15.16:0/203799974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558453c6e5d0 0x558453c89d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 >> 172.21.15.16:0/203799974 conn(0x558453c37310 msgr2=0x558453c780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 shutdown_connections 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 1 -- 172.21.15.16:0/203799974 wait complete. 2024-09-12T02:48:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.040+0000 7fe68f86d740 20 client.15495 trim_cache size 0 max 16384 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 msgr2=0x7fe6700a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 0x7fe6700a5310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x558453c6bac0 tx=0x7fe6780d4040 comp rx=0 tx=0).stop 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 msgr2=0x7fe68819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 0x7fe68819a720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe67800e990 tx=0x7fe67800ee50 comp rx=0 tx=0).stop 2024-09-12T02:48:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 shutdown_connections 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6700a2e50 0x7fe6700a5310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe688108290 0x7fe68819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe688103740 0x7fe68819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 --2- 172.21.15.16:0/3565126663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe688102d90 0x7fe68819a720 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 >> 172.21.15.16:0/3565126663 conn(0x7fe6880fc7d0 msgr2=0x7fe688105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 shutdown_connections 2024-09-12T02:48:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:27.041+0000 7fe68d494640 1 -- 172.21.15.16:0/3565126663 wait complete. 2024-09-12T02:48:27.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:48:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:27 smithi033 ceph-mon[35092]: pgmap v353: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:27 smithi184 ceph-mon[35321]: pgmap v353: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:28.321 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:27 smithi016 ceph-mon[30925]: pgmap v353: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T02:48:28.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 -- 172.21.15.16:0/626295725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4100b20 msgr2=0x7ff4f4100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/626295725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4100b20 0x7ff4f4100f00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009900 tx=0x7ff4e402a410 comp rx=0 tx=0).stop 2024-09-12T02:48:28.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 -- 172.21.15.16:0/626295725 shutdown_connections 2024-09-12T02:48:28.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/626295725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f410e500 0x7ff4f4110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/626295725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f4101440 0x7ff4f410df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/626295725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4100b20 0x7ff4f4100f00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 -- 172.21.15.16:0/626295725 >> 172.21.15.16:0/626295725 conn(0x7ff4f40fc7d0 msgr2=0x7ff4f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.710+0000 7ff4f9e98640 1 -- 172.21.15.16:0/626295725 shutdown_connections 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.711+0000 7ff4f9e98640 1 -- 172.21.15.16:0/626295725 wait complete. 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.711+0000 7ff4f9e98640 1 Processor -- start 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 -- start start 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f4100b20 0x7ff4f41a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f410e500 0x7ff4f419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4114280 con 0x7ff4f4101440 2024-09-12T02:48:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4114100 con 0x7ff4f4100b20 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f9e98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4114400 con 0x7ff4f410e500 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f4100b20 0x7ff4f41a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43578/0 (socket says 172.21.15.16:43578) 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 -- 172.21.15.16:0/1120865572 learned_addr learned my addr 172.21.15.16:0/1120865572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f410e500 0x7ff4f419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 -- 172.21.15.16:0/1120865572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f410e500 msgr2=0x7ff4f419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f410e500 0x7ff4f419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 -- 172.21.15.16:0/1120865572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f4100b20 msgr2=0x7ff4f41a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f4100b20 0x7ff4f41a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.712+0000 7ff4ebfff640 1 -- 172.21.15.16:0/1120865572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f419aea0 con 0x7ff4f4101440 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c970 tx=0x7ff4dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4f17fa640 1 -- 172.21.15.16:0/1120865572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc004220 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f419b190 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4f17fa640 1 -- 172.21.15.16:0/1120865572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4dc0043c0 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4f17fa640 1 -- 172.21.15.16:0/1120865572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc004a00 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.713+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f41a7500 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.716+0000 7ff4fc271740 1 Processor -- start 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.716+0000 7ff4fc271740 1 -- start start 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.717+0000 7ff4f17fa640 1 -- 172.21.15.16:0/1120865572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7ff4dc0079b0 con 0x7ff4f4101440 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.717+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.717+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7dba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.717+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4fc271740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56380e7412f0 con 0x56380e7eeb80 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4fc271740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56380e6e8cf0 con 0x56380e7db5d0 2024-09-12T02:48:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4fc271740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56380e73d2b0 con 0x56380e7dcb40 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7db090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46644/0 (socket says 172.21.15.16:46644) 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f3fff640 1 -- 172.21.15.16:0/1454694982 learned_addr learned my addr 172.21.15.16:0/1454694982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7dba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f17fa640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 0x7ff4e00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 -- 172.21.15.16:0/1454694982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 msgr2=0x56380e7db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7db090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 -- 172.21.15.16:0/1454694982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 msgr2=0x56380e7f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f37fe640 1 -- 172.21.15.16:0/1454694982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380e6cd3a0 con 0x56380e7db5d0 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.718+0000 7ff4f17fa640 1 -- 172.21.15.16:0/1120865572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff4dc0cee30 con 0x7ff4f4101440 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.719+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7dba10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4005940 tx=0x7ff4e4031630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39de4be77b1a2613 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.719+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 0x7ff4e00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.719+0000 7ff4e9ffb640 1 -- 172.21.15.16:0/1454694982 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4040070 con 0x56380e7db5d0 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.719+0000 7ff4e9ffb640 1 -- 172.21.15.16:0/1454694982 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4e4005020 con 0x56380e7db5d0 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.719+0000 7ff4e9ffb640 1 -- 172.21.15.16:0/1454694982 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4035040 con 0x56380e7db5d0 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- 172.21.15.16:0/1454694982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 msgr2=0x56380e7dba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 0x7ff4e00a5150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec003b10 tx=0x7ff4ec004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7dba10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4005940 tx=0x7ff4e4031630 comp rx=0 tx=0).stop 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- 172.21.15.16:0/1454694982 shutdown_connections 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7dba10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1454694982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- 172.21.15.16:0/1454694982 >> 172.21.15.16:0/1454694982 conn(0x56380e7a4310 msgr2=0x56380e7d7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- 172.21.15.16:0/1454694982 shutdown_connections 2024-09-12T02:48:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- 172.21.15.16:0/1454694982 wait complete. 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 -- start start 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.720+0000 7ff4fc271740 1 Processor -- start 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7eb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56380e7f1820 con 0x56380e7eeb80 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56380e6fc060 con 0x56380e7db5d0 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56380e7f19a0 con 0x56380e7dcb40 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4fc271740 10 client.0 _ll_register_callbacks cb 0x56380e44f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47740/0 (socket says 172.21.15.16:47740) 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f37fe640 1 -- 172.21.15.16:0/1651758883 learned_addr learned my addr 172.21.15.16:0/1651758883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 -- 172.21.15.16:0/1651758883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 msgr2=0x56380e7eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 -- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 msgr2=0x56380e7ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 -- 172.21.15.16:0/1651758883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380e4b23c0 con 0x56380e7eeb80 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7ebcb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0106e0 tx=0x7ff4ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c6bda0fc7ff472 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.721+0000 7ff4d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec011850 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff4ec0119f0 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4e4005020 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4ec011bf0 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56380e7fd2a0 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4ec025020 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4ec01c3e0 con 0x56380e7eeb80 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 10 client.15507 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 20 client.15507 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.722+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e7fe760 0x56380e800bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.723+0000 7ff4fc271740 4 client.15507 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.723+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff4e4005020 con 0x56380e7fe760 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.723+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e7fe760 0x56380e800bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.723+0000 7ff4d2ffd640 10 client.15507 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.723+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e7fe760 0x56380e800bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=647de464784521e0 server_cookie=a88e5aa091890dfb in_seq=0 out_seq=0 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/1651758883 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7ff4e4005020 con 0x56380e7fe760 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4d2ffd640 10 client.15507 handle_command_reply: tid=0 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 client.15507 shutdown 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 msgr2=0x56380e7ebcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7ebcb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0106e0 tx=0x7ff4ec004ab0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 shutdown_connections 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e7fe760 0x56380e800bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e7ebcb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e7eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.724+0000 7ff4fc271740 1 --2- 172.21.15.16:0/1651758883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.725+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 >> 172.21.15.16:0/1651758883 conn(0x56380e7a4310 msgr2=0x56380e7dd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.725+0000 7ff4d2ffd640 0 client.15507 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.725+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 shutdown_connections 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.725+0000 7ff4fc271740 1 -- 172.21.15.16:0/1651758883 wait complete. 2024-09-12T02:48:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.725+0000 7ff4fc271740 20 client.15507 trim_cache size 0 max 16384 2024-09-12T02:48:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 Processor -- start 2024-09-12T02:48:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 -- start start 2024-09-12T02:48:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7db5d0 0x56380e7faaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 0x56380e7ebc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41ab2e0 con 0x56380e7dcb40 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41ace20 con 0x56380e7eeb80 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4fc271740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41ab9b0 con 0x56380e7db5d0 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43594/0 (socket says 172.21.15.16:43594) 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4ebfff640 1 -- 172.21.15.16:0/3529697246 learned_addr learned my addr 172.21.15.16:0/3529697246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7db5d0 0x56380e7faaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.750+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 0x56380e7ebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 -- 172.21.15.16:0/3529697246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7db5d0 msgr2=0x56380e7faaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7db5d0 0x56380e7faaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 -- 172.21.15.16:0/3529697246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 msgr2=0x56380e7ebc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 0x56380e7ebc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 -- 172.21.15.16:0/3529697246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380e7de120 con 0x56380e7dcb40 2024-09-12T02:48:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 0x56380e7ebc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc097fc0 tx=0x7ff4dc0d2980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32994868105cdd43 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4e97fa640 1 -- 172.21.15.16:0/3529697246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc01d040 con 0x56380e7dcb40 2024-09-12T02:48:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4e97fa640 1 -- 172.21.15.16:0/3529697246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4dc0bb050 con 0x56380e7dcb40 2024-09-12T02:48:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.751+0000 7ff4e97fa640 1 -- 172.21.15.16:0/3529697246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0a41b0 con 0x56380e7dcb40 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 -- 172.21.15.16:0/3529697246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 msgr2=0x56380e7eb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc097fc0 tx=0x7ff4dc0d2980 comp rx=0 tx=0).stop 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 -- 172.21.15.16:0/3529697246 shutdown_connections 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7eeb80 0x56380e7ebc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7dcb40 0x56380e7eb730 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 --2- 172.21.15.16:0/3529697246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7db5d0 0x56380e7faaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 -- 172.21.15.16:0/3529697246 >> 172.21.15.16:0/3529697246 conn(0x56380e7a4310 msgr2=0x56380e7e6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 -- 172.21.15.16:0/3529697246 shutdown_connections 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.752+0000 7ff4fc271740 1 -- 172.21.15.16:0/3529697246 wait complete. 2024-09-12T02:48:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 -- start start 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 Processor -- start 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e805e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e806370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41ac750 con 0x56380e7eeb80 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41ac080 con 0x56380e7db5d0 2024-09-12T02:48:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41a92f0 con 0x56380e7dcb40 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.753+0000 7ff4fc271740 10 client.0 _ll_register_callbacks cb 0x7ff4ec009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47752/0 (socket says 172.21.15.16:47752) 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 -- 172.21.15.16:0/557390158 learned_addr learned my addr 172.21.15.16:0/557390158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e805e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f3fff640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e806370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 -- 172.21.15.16:0/557390158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 msgr2=0x56380e805e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e805e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 -- 172.21.15.16:0/557390158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 msgr2=0x56380e806370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e806370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.754+0000 7ff4f37fe640 1 -- 172.21.15.16:0/557390158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380e806b70 con 0x56380e7db5d0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4f37fe640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4005940 tx=0x7ff4e4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55ea31822d0975f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:28.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4040070 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4dc0bb050 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff4e4004240 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4e4060050 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4fc271740 10 client.25178 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.755+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56380e869500 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4dc0bb050 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4e40047e0 con 0x56380e7db5d0 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 10 client.25178 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 10 client.25178 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:48:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 20 client.25178 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e8697f0 0x56380e86bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 4 client.25178 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56380e800f40 con 0x56380e8697f0 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.756+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e8697f0 0x56380e86bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4d2ffd640 10 client.25178 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4ebfff640 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e8697f0 0x56380e86bbe0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85e80ffef546719c server_cookie=40857e8d765a0489 in_seq=0 out_seq=0 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4d2ffd640 1 -- 172.21.15.16:0/557390158 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56380e800f40 con 0x56380e8697f0 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4d2ffd640 10 client.25178 handle_command_reply: tid=0 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 client.25178 shutdown 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 msgr2=0x56380e7f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4005940 tx=0x7ff4e4005850 comp rx=0 tx=0).stop 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 shutdown_connections 2024-09-12T02:48:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56380e8697f0 0x56380e86bbe0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56380e7eeb80 0x56380e806370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56380e7dcb40 0x56380e805e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4d2ffd640 0 client.25178 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 --2- 172.21.15.16:0/557390158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56380e7db5d0 0x56380e7f7430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.757+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 >> 172.21.15.16:0/557390158 conn(0x56380e7a4310 msgr2=0x56380e7eb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 shutdown_connections 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4fc271740 1 -- 172.21.15.16:0/557390158 wait complete. 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4fc271740 20 client.25178 trim_cache size 0 max 16384 2024-09-12T02:48:28.764 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 msgr2=0x7ff4e00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 0x7ff4e00a5150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec003b10 tx=0x7ff4ec004a40 comp rx=0 tx=0).stop 2024-09-12T02:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 msgr2=0x7ff4f41a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.758+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c970 tx=0x7ff4dc00ce30 comp rx=0 tx=0).stop 2024-09-12T02:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 shutdown_connections 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4f410e500 0x7ff4f419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff4e00a2c90 0x7ff4e00a5150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f4101440 0x7ff4f41a0a50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 --2- 172.21.15.16:0/1120865572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4f4100b20 0x7ff4f41a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 >> 172.21.15.16:0/1120865572 conn(0x7ff4f40fc7d0 msgr2=0x7ff4f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 shutdown_connections 2024-09-12T02:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:28.759+0000 7ff4f9e98640 1 -- 172.21.15.16:0/1120865572 wait complete. 2024-09-12T02:48:29.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:48:29.755 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:29 smithi016 ceph-mon[30925]: pgmap v354: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:29 smithi033 ceph-mon[35092]: pgmap v354: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:30.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:29 smithi184 ceph-mon[35321]: pgmap v354: 97 pgs: 97 active+clean; 659 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 -- 172.21.15.16:0/1562785722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102570 msgr2=0x7f7a501029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 --2- 172.21.15.16:0/1562785722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102570 0x7f7a501029b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009890 tx=0x7f7a4402a3f0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 -- 172.21.15.16:0/1562785722 shutdown_connections 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 --2- 172.21.15.16:0/1562785722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50102ef0 0x7f7a5010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 --2- 172.21.15.16:0/1562785722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102570 0x7f7a501029b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 --2- 172.21.15.16:0/1562785722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50100e20 0x7f7a50101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 -- 172.21.15.16:0/1562785722 >> 172.21.15.16:0/1562785722 conn(0x7f7a500fc7d0 msgr2=0x7f7a500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 -- 172.21.15.16:0/1562785722 shutdown_connections 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.381+0000 7f7a56039640 1 -- 172.21.15.16:0/1562785722 wait complete. 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 Processor -- start 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 -- start start 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50100e20 0x7f7a50196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102ef0 0x7f7a5019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5010fe30 con 0x7f7a50102570 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5010fcb0 con 0x7f7a50102ef0 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a56039640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5010ffb0 con 0x7f7a50100e20 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.382+0000 7f7a4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43612/0 (socket says 172.21.15.16:43612) 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 -- 172.21.15.16:0/3611958157 learned_addr learned my addr 172.21.15.16:0/3611958157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102ef0 0x7f7a5019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50100e20 0x7f7a50196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 -- 172.21.15.16:0/3611958157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50100e20 msgr2=0x7f7a50196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50100e20 0x7f7a50196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 -- 172.21.15.16:0/3611958157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102ef0 msgr2=0x7f7a5019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102ef0 0x7f7a5019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 -- 172.21.15.16:0/3611958157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5019b2e0 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4effd640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000c00 tx=0x7f7a44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.383+0000 7f7a4cff9640 1 -- 172.21.15.16:0/3611958157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44037040 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.384+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5019b570 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.384+0000 7f7a4cff9640 1 -- 172.21.15.16:0/3611958157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a44040070 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.384+0000 7f7a4cff9640 1 -- 172.21.15.16:0/3611958157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44030ac0 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.384+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a501a2eb0 con 0x7f7a50102570 2024-09-12T02:48:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.388+0000 7f7a4cff9640 1 -- 172.21.15.16:0/3611958157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f7a44003c60 con 0x7f7a50102570 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.388+0000 7f7a58412740 1 Processor -- start 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.388+0000 7f7a58412740 1 -- start start 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.388+0000 7f7a58412740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43620/0 (socket says 172.21.15.16:43620) 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a58412740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f870a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4cff9640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 0x7f7a380a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a58412740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a58412740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56139f7d62f0 con 0x56139f871b40 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a58412740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56139f77dcf0 con 0x56139f87b370 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a58412740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56139f7d22b0 con 0x56139f8705d0 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4ffff640 1 -- 172.21.15.16:0/3806160755 learned_addr learned my addr 172.21.15.16:0/3806160755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f870a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4cff9640 1 -- 172.21.15.16:0/3611958157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7a440ee8f0 con 0x7f7a50102570 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.389+0000 7f7a4effd640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 0x7f7a380a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4ffff640 1 -- 172.21.15.16:0/3806160755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 msgr2=0x56139f870a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f870a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4ffff640 1 -- 172.21.15.16:0/3806160755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 msgr2=0x56139f87d760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4effd640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 0x7f7a380a5150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005e00 tx=0x7f7a4402a3f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.390+0000 7f7a4ffff640 1 -- 172.21.15.16:0/3806160755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56139f7623a0 con 0x56139f871b40 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.391+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a48005c40 tx=0x7f7a480118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88216682f11f020c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.391+0000 7f7a35ffb640 1 -- 172.21.15.16:0/3806160755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4801e070 con 0x56139f871b40 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.391+0000 7f7a35ffb640 1 -- 172.21.15.16:0/3806160755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4800f960 con 0x56139f871b40 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.391+0000 7f7a35ffb640 1 -- 172.21.15.16:0/3806160755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a48019c40 con 0x56139f871b40 2024-09-12T02:48:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 -- 172.21.15.16:0/3806160755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 msgr2=0x56139f870090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a48005c40 tx=0x7f7a480118d0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 -- 172.21.15.16:0/3806160755 shutdown_connections 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f870a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 --2- 172.21.15.16:0/3806160755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f870090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.392+0000 7f7a58412740 1 -- 172.21.15.16:0/3806160755 >> 172.21.15.16:0/3806160755 conn(0x56139f839310 msgr2=0x56139f86cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.393+0000 7f7a58412740 1 -- 172.21.15.16:0/3806160755 shutdown_connections 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.393+0000 7f7a58412740 1 -- 172.21.15.16:0/3806160755 wait complete. 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.393+0000 7f7a58412740 1 -- start start 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.393+0000 7f7a58412740 1 Processor -- start 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f8705d0 0x56139f87adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f87b370 0x56139f8787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56139f87e070 con 0x56139f871b40 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56139f791060 con 0x56139f8705d0 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56139f87e1f0 con 0x56139f87b370 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a58412740 10 client.0 _ll_register_callbacks cb 0x56139f4e48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43628/0 (socket says 172.21.15.16:43628) 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.394+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/2847194702 learned_addr learned my addr 172.21.15.16:0/2847194702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f87b370 msgr2=0x56139f8787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f87b370 0x56139f8787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/2847194702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f8705d0 msgr2=0x56139f87adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f8705d0 0x56139f87adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/2847194702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56139f5473c0 con 0x56139f871b40 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009a80 tx=0x7f7a40009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b7c822868d414b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a2effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40020670 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7a40020810 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.395+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4800f960 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.396+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7a40020a50 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.396+0000 7f7a58412740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.396+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56139f889c00 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.396+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4800f960 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7a4002a020 con 0x56139f871b40 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 10 client.15525 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 20 client.15525 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:30.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f88b460 0x56139f88d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 4 client.15525 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56139f7e3860 con 0x56139f88b460 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.397+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f88b460 0x56139f88d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.398+0000 7f7a2effd640 10 client.15525 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.398+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f88b460 0x56139f88d920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6342a74dcd1f380 server_cookie=d726d06a0de1e9d2 in_seq=0 out_seq=0 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.398+0000 7f7a2effd640 1 -- 172.21.15.16:0/2847194702 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56139f7e3860 con 0x56139f88b460 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.398+0000 7f7a2effd640 10 client.15525 handle_command_reply: tid=0 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 client.15525 shutdown 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 msgr2=0x56139f8780a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009a80 tx=0x7f7a40009f40 comp rx=0 tx=0).stop 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 shutdown_connections 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f88b460 0x56139f88d920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f87b370 0x56139f8787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f871b40 0x56139f8780a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 --2- 172.21.15.16:0/2847194702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f8705d0 0x56139f87adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 >> 172.21.15.16:0/2847194702 conn(0x56139f839310 msgr2=0x56139f872960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a2effd640 0 client.15525 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 shutdown_connections 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 1 -- 172.21.15.16:0/2847194702 wait complete. 2024-09-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.399+0000 7f7a58412740 20 client.15525 trim_cache size 0 max 16384 2024-09-12T02:48:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 Processor -- start 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 -- start start 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f887310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f893690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501ab1c0 con 0x56139f893230 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501acd00 con 0x56139f87b370 2024-09-12T02:48:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.424+0000 7f7a58412740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501ab890 con 0x56139f8705d0 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f893690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f893690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43630/0 (socket says 172.21.15.16:43630) 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4ffff640 1 -- 172.21.15.16:0/63852715 learned_addr learned my addr 172.21.15.16:0/63852715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f887310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/63852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 msgr2=0x56139f887310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f887310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/63852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 msgr2=0x56139f893690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f893690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/63852715 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56139f88dfa0 con 0x56139f87b370 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.425+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87a010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x56139f879160 tx=0x7f7a400058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77781b0b7e90769d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.426+0000 7f7a357fa640 1 -- 172.21.15.16:0/63852715 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40020df0 con 0x56139f87b370 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.426+0000 7f7a357fa640 1 -- 172.21.15.16:0/63852715 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a400039c0 con 0x56139f87b370 2024-09-12T02:48:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.426+0000 7f7a357fa640 1 -- 172.21.15.16:0/63852715 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40038480 con 0x56139f87b370 2024-09-12T02:48:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- 172.21.15.16:0/63852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 msgr2=0x56139f87a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87a010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x56139f879160 tx=0x7f7a400058d0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- 172.21.15.16:0/63852715 shutdown_connections 2024-09-12T02:48:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f893690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f87a010 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 --2- 172.21.15.16:0/63852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f887310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- 172.21.15.16:0/63852715 >> 172.21.15.16:0/63852715 conn(0x56139f839310 msgr2=0x56139f879900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- 172.21.15.16:0/63852715 shutdown_connections 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- 172.21.15.16:0/63852715 wait complete. 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.427+0000 7f7a58412740 1 -- start start 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 Processor -- start 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f880450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f880990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46738/0 (socket says 172.21.15.16:46738) 2024-09-12T02:48:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a4ffff640 1 -- 172.21.15.16:0/3815888279 learned_addr learned my addr 172.21.15.16:0/3815888279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501ac630 con 0x56139f893230 2024-09-12T02:48:30.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501abf60 con 0x56139f87b370 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501a9250 con 0x56139f8705d0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f880990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f880450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.428+0000 7f7a58412740 10 client.0 _ll_register_callbacks cb 0x7f7a40004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 msgr2=0x56139f883950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/3815888279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 msgr2=0x56139f880990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f880990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56139f873120 con 0x56139f87b370 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a4f7fe640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f880450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x56139f8767d0 tx=0x7f7a400206a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=162f1a43840e7a6b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a2effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40022860 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a400039c0 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7a40022a00 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7a40020e40 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a58412740 10 client.25190 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.429+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56139f8f6d90 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.430+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a400039c0 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7a4003f070 con 0x56139f87b370 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 10 client.25190 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 10 client.25190 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:48:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 20 client.25190 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f8f73f0 0x56139f8f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 4 client.25190 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56139f7ddd70 con 0x56139f8f73f0 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f8f73f0 0x56139f8f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a4ffff640 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f8f73f0 0x56139f8f97e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ce6681221dca0ae server_cookie=b5dc1a6cb328323e in_seq=0 out_seq=0 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.431+0000 7f7a2effd640 10 client.25190 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a2effd640 1 -- 172.21.15.16:0/3815888279 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56139f7ddd70 con 0x56139f8f73f0 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a2effd640 10 client.25190 handle_command_reply: tid=0 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 client.25190 shutdown 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 msgr2=0x56139f880450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f880450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x56139f8767d0 tx=0x7f7a400206a0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 shutdown_connections 2024-09-12T02:48:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56139f893230 0x56139f880990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56139f87b370 0x56139f880450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56139f8f73f0 0x56139f8f97e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 --2- 172.21.15.16:0/3815888279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56139f8705d0 0x56139f883950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 >> 172.21.15.16:0/3815888279 conn(0x56139f839310 msgr2=0x56139f872110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a2effd640 0 client.25190 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 shutdown_connections 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 1 -- 172.21.15.16:0/3815888279 wait complete. 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.432+0000 7f7a58412740 20 client.25190 trim_cache size 0 max 16384 2024-09-12T02:48:30.438 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 msgr2=0x7f7a380a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 0x7f7a380a5150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005e00 tx=0x7f7a4402a3f0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 msgr2=0x7f7a50196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000c00 tx=0x7f7a44004270 comp rx=0 tx=0).stop 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 shutdown_connections 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7a50102ef0 0x7f7a5019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7a380a2c90 0x7f7a380a5150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a50102570 0x7f7a50196870 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 --2- 172.21.15.16:0/3611958157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a50100e20 0x7f7a50196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.433+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 >> 172.21.15.16:0/3611958157 conn(0x7f7a500fc7d0 msgr2=0x7f7a501048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.434+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 shutdown_connections 2024-09-12T02:48:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:30.434+0000 7f7a56039640 1 -- 172.21.15.16:0/3611958157 wait complete. 2024-09-12T02:48:31.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:32.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:31 smithi016 ceph-mon[30925]: pgmap v355: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 -- 172.21.15.16:0/2231210977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b14102d30 msgr2=0x7f2b14103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/2231210977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b14102d30 0x7f2b14103110 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009930 tx=0x7f2b0402a420 comp rx=0 tx=0).stop 2024-09-12T02:48:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 -- 172.21.15.16:0/2231210977 shutdown_connections 2024-09-12T02:48:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/2231210977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141082a0 0x7f2b1410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/2231210977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b141036e0 0x7f2b14107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/2231210977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b14102d30 0x7f2b14103110 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 -- 172.21.15.16:0/2231210977 >> 172.21.15.16:0/2231210977 conn(0x7f2b140fc7d0 msgr2=0x7f2b140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 -- 172.21.15.16:0/2231210977 shutdown_connections 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.062+0000 7f2b19a7b640 1 -- 172.21.15.16:0/2231210977 wait complete. 2024-09-12T02:48:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.063+0000 7f2b19a7b640 1 Processor -- start 2024-09-12T02:48:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.063+0000 7f2b19a7b640 1 -- start start 2024-09-12T02:48:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b14102d30 0x7f2b14100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 0x7f2b14101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141120a0 con 0x7f2b141082a0 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b14111f20 con 0x7f2b141036e0 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b19a7b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b14112220 con 0x7f2b14102d30 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 0x7f2b14101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47836/0 (socket says 172.21.15.16:47836) 2024-09-12T02:48:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 -- 172.21.15.16:0/1872482512 learned_addr learned my addr 172.21.15.16:0/1872482512 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b137fe640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b14102d30 0x7f2b14100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 -- 172.21.15.16:0/1872482512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b14102d30 msgr2=0x7f2b14100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b14102d30 0x7f2b14100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 -- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 msgr2=0x7f2b14101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 0x7f2b14101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b12ffd640 1 -- 172.21.15.16:0/1872482512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b141a6a60 con 0x7f2b141036e0 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.064+0000 7f2b13fff640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 0x7f2b14101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800d950 tx=0x7f2b0800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b10ff9640 1 -- 172.21.15.16:0/1872482512 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08017070 con 0x7f2b141036e0 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b10ff9640 1 -- 172.21.15.16:0/1872482512 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b08009c30 con 0x7f2b141036e0 2024-09-12T02:48:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b10ff9640 1 -- 172.21.15.16:0/1872482512 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b080047a0 con 0x7f2b141036e0 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b141a6d50 con 0x7f2b141036e0 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.065+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b141a7180 con 0x7f2b141036e0 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.068+0000 7f2b10ff9640 1 -- 172.21.15.16:0/1872482512 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f2b08004020 con 0x7f2b141036e0 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 Processor -- start 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 -- start start 2024-09-12T02:48:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x56141314ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5614130b22f0 con 0x56141314db40 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561413059cf0 con 0x56141314c5d0 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b1be54740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5614130ae2b0 con 0x561413157370 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43660/0 (socket says 172.21.15.16:43660) 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.069+0000 7f2b13fff640 1 -- 172.21.15.16:0/3689882278 learned_addr learned my addr 172.21.15.16:0/3689882278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b137fe640 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x56141314ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b10ff9640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 0x7f2afc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b13fff640 1 -- 172.21.15.16:0/3689882278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 msgr2=0x561413159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b13fff640 1 -- 172.21.15.16:0/3689882278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 msgr2=0x56141314ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x56141314ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b13fff640 1 -- 172.21.15.16:0/3689882278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56141303e3a0 con 0x56141314db40 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.070+0000 7f2b10ff9640 1 -- 172.21.15.16:0/1872482512 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b080cee30 con 0x7f2b141036e0 2024-09-12T02:48:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.071+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c011790 tx=0x7f2b0c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec55ac7e280f707 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.071+0000 7f2b13fff640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 0x7f2afc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.072+0000 7f2b13fff640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 0x7f2afc0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009fd0 tx=0x7f2b0c012000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.072+0000 7f2af9ffb640 1 -- 172.21.15.16:0/3689882278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c003d10 con 0x56141314db40 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.072+0000 7f2af9ffb640 1 -- 172.21.15.16:0/3689882278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c004d20 con 0x56141314db40 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.072+0000 7f2af9ffb640 1 -- 172.21.15.16:0/3689882278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c005540 con 0x56141314db40 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- 172.21.15.16:0/3689882278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 msgr2=0x56141314c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c011790 tx=0x7f2b0c011c50 comp rx=0 tx=0).stop 2024-09-12T02:48:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- 172.21.15.16:0/3689882278 shutdown_connections 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x56141314ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3689882278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x56141314c090 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- 172.21.15.16:0/3689882278 >> 172.21.15.16:0/3689882278 conn(0x561413115310 msgr2=0x561413148c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- 172.21.15.16:0/3689882278 shutdown_connections 2024-09-12T02:48:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- 172.21.15.16:0/3689882278 wait complete. 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.073+0000 7f2b1be54740 1 -- start start 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.074+0000 7f2b1be54740 1 Processor -- start 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.074+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413156d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.074+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.074+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.074+0000 7f2b137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43664/0 (socket says 172.21.15.16:43664) 2024-09-12T02:48:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47866/0 (socket says 172.21.15.16:47866) 2024-09-12T02:48:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 -- 172.21.15.16:0/2616224832 learned_addr learned my addr 172.21.15.16:0/2616224832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56141315a070 con 0x56141314db40 2024-09-12T02:48:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56141306d060 con 0x56141314c5d0 2024-09-12T02:48:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56141315a1f0 con 0x561413157370 2024-09-12T02:48:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 -- 172.21.15.16:0/2616224832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 msgr2=0x561413154610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413154610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 -- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 msgr2=0x561413156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561412e233c0 con 0x56141314db40 2024-09-12T02:48:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2b08021000 tx=0x7f2b080045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4608f22cd3c1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2af2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2af2ffd640 1 -- 172.21.15.16:0/2616224832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08097780 con 0x56141314db40 2024-09-12T02:48:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.075+0000 7f2b1be54740 10 client.0 _ll_register_callbacks cb 0x561412dc08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2af2ffd640 1 -- 172.21.15.16:0/2616224832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b080cc9e0 con 0x56141314db40 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b0c004d20 con 0x56141314db40 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2af2ffd640 1 -- 172.21.15.16:0/2616224832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b0809bbe0 con 0x56141314db40 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2b1be54740 10 client.15543 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x561413165880 con 0x56141314db40 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.076+0000 7f2af2ffd640 1 -- 172.21.15.16:0/2616224832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2b0809a930 con 0x56141314db40 2024-09-12T02:48:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 10 client.15543 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 10 client.15543 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:48:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 20 client.15543 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 4 client.15543 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614130bf860 con 0x561413166d40 2024-09-12T02:48:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.077+0000 7f2b137fe640 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.078+0000 7f2b137fe640 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131691d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24c43cb28bc98c1c server_cookie=e4e717846bd91516 in_seq=0 out_seq=0 2024-09-12T02:48:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.078+0000 7f2af2ffd640 10 client.15543 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.078+0000 7f2af2ffd640 1 -- 172.21.15.16:0/2616224832 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614130bf860 con 0x561413166d40 2024-09-12T02:48:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.078+0000 7f2af2ffd640 10 client.15543 handle_command_reply: tid=0 2024-09-12T02:48:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.079+0000 7f2b1be54740 1 client.15543 shutdown 2024-09-12T02:48:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.079+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 msgr2=0x561413154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.079+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2b08021000 tx=0x7f2b080045c0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 shutdown_connections 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131691d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2af2ffd640 0 client.15543 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x561413154020 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 --2- 172.21.15.16:0/2616224832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 >> 172.21.15.16:0/2616224832 conn(0x561413115310 msgr2=0x56141314e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 shutdown_connections 2024-09-12T02:48:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.080+0000 7f2b1be54740 1 -- 172.21.15.16:0/2616224832 wait complete. 2024-09-12T02:48:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.081+0000 7f2b1be54740 20 client.15543 trim_cache size 0 max 16384 2024-09-12T02:48:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b1be54740 1 Processor -- start 2024-09-12T02:48:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b1be54740 1 -- start start 2024-09-12T02:48:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56141314c5d0 0x561413163310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314db40 0x56141316eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43670/0 (socket says 172.21.15.16:43670) 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.107+0000 7f2b13fff640 1 -- 172.21.15.16:0/3927252437 learned_addr learned my addr 172.21.15.16:0/3927252437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141af760 con 0x561413157370 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314db40 0x56141316eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141b12a0 con 0x56141314db40 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141afe30 con 0x56141314c5d0 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b137fe640 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56141314c5d0 0x561413163310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 -- 172.21.15.16:0/3927252437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56141314c5d0 msgr2=0x561413163310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56141314c5d0 0x561413163310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 -- 172.21.15.16:0/3927252437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314db40 msgr2=0x56141316eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314db40 0x56141316eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 -- 172.21.15.16:0/3927252437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561413152570 con 0x561413157370 2024-09-12T02:48:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.108+0000 7f2b13fff640 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00f790 tx=0x7f2b0c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeed7b5116c3dfa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.109+0000 7f2af97fa640 1 -- 172.21.15.16:0/3927252437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c026430 con 0x561413157370 2024-09-12T02:48:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.109+0000 7f2af97fa640 1 -- 172.21.15.16:0/3927252437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c026ab0 con 0x561413157370 2024-09-12T02:48:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.109+0000 7f2af97fa640 1 -- 172.21.15.16:0/3927252437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c01cb60 con 0x561413157370 2024-09-12T02:48:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 msgr2=0x56141316f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00f790 tx=0x7f2b0c005bd0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 shutdown_connections 2024-09-12T02:48:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561413157370 0x56141316f020 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314db40 0x56141316eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 --2- 172.21.15.16:0/3927252437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56141314c5d0 0x561413163310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 >> 172.21.15.16:0/3927252437 conn(0x561413115310 msgr2=0x5614131560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 shutdown_connections 2024-09-12T02:48:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- 172.21.15.16:0/3927252437 wait complete. 2024-09-12T02:48:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.110+0000 7f2b1be54740 1 -- start start 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.111+0000 7f2b1be54740 1 Processor -- start 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.111+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413151180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.111+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.111+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43682/0 (socket says 172.21.15.16:43682) 2024-09-12T02:48:32.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.111+0000 7f2b1be54740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413151c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 -- 172.21.15.16:0/4151838732 learned_addr learned my addr 172.21.15.16:0/4151838732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141b0bd0 con 0x56141314db40 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141b0500 con 0x56141314c5d0 2024-09-12T02:48:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141ad740 con 0x561413157370 2024-09-12T02:48:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b13fff640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b1be54740 10 client.0 _ll_register_callbacks cb 0x7f2b080046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 -- 172.21.15.16:0/4151838732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 msgr2=0x561413151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 -- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 msgr2=0x561413151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614131d0bb0 con 0x56141314db40 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b12ffd640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b08005bf0 tx=0x7f2b080d1280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de45d53efc5255b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.112+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b0c026ab0 con 0x56141314db40 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2af2ffd640 10 client.15549 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2af2ffd640 1 -- 172.21.15.16:0/4151838732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08097780 con 0x56141314db40 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2af2ffd640 1 -- 172.21.15.16:0/4151838732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b08097230 con 0x56141314db40 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2af2ffd640 1 -- 172.21.15.16:0/4151838732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b08097470 con 0x56141314db40 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 10 client.15549 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5614131d1d40 con 0x56141314db40 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2af2ffd640 1 -- 172.21.15.16:0/4151838732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2b0809a3a0 con 0x56141314db40 2024-09-12T02:48:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 10 client.15549 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 10 client.15549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 20 client.15549 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 4 client.15549 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.113+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2b0c026ab0 con 0x561413166d40 2024-09-12T02:48:32.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.114+0000 7f2b137fe640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.114+0000 7f2af2ffd640 10 client.15549 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.114+0000 7f2b137fe640 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131d4360 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b422c8a5ded75cd server_cookie=70726b738e10e579 in_seq=0 out_seq=0 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2af2ffd640 1 -- 172.21.15.16:0/4151838732 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2b0c026ab0 con 0x561413166d40 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2af2ffd640 10 client.15549 handle_command_reply: tid=0 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 client.15549 shutdown 2024-09-12T02:48:32.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 msgr2=0x5614131516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b08005bf0 tx=0x7f2b080d1280 comp rx=0 tx=0).stop 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 shutdown_connections 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561413166d40 0x5614131d4360 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561413157370 0x561413151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56141314db40 0x5614131516c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 --2- 172.21.15.16:0/4151838732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56141314c5d0 0x561413151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 >> 172.21.15.16:0/4151838732 conn(0x561413115310 msgr2=0x561413162300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2af2ffd640 0 client.15549 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 shutdown_connections 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 1 -- 172.21.15.16:0/4151838732 wait complete. 2024-09-12T02:48:32.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.115+0000 7f2b1be54740 20 client.15549 trim_cache size 0 max 16384 2024-09-12T02:48:32.134 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:32.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 msgr2=0x7f2afc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 0x7f2afc0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009fd0 tx=0x7f2b0c012000 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 msgr2=0x7f2b14100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800d950 tx=0x7f2b0800de10 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 shutdown_connections 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2afc0a2c10 0x7f2afc0a50d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b141082a0 0x7f2b14101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b141036e0 0x7f2b14100ed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 --2- 172.21.15.16:0/1872482512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b14102d30 0x7f2b14100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:32.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 >> 172.21.15.16:0/1872482512 conn(0x7f2b140fc7d0 msgr2=0x7f2b14105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 shutdown_connections 2024-09-12T02:48:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:32.116+0000 7f2b19a7b640 1 -- 172.21.15.16:0/1872482512 wait complete. 2024-09-12T02:48:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:31 smithi033 ceph-mon[35092]: pgmap v355: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:31 smithi184 ceph-mon[35321]: pgmap v355: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:32.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:48:33.033 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.689+0000 7f3ee418f640 1 -- 172.21.15.16:0/2448394957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc1082a0 msgr2=0x7f3edc10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.689+0000 7f3ee418f640 1 --2- 172.21.15.16:0/2448394957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc1082a0 0x7f3edc10ec40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009cd0 tx=0x7f3ed402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 -- 172.21.15.16:0/2448394957 shutdown_connections 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 --2- 172.21.15.16:0/2448394957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc1082a0 0x7f3edc10ec40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 --2- 172.21.15.16:0/2448394957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1036e0 0x7f3edc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 --2- 172.21.15.16:0/2448394957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc102d30 0x7f3edc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 -- 172.21.15.16:0/2448394957 >> 172.21.15.16:0/2448394957 conn(0x7f3edc0fc7d0 msgr2=0x7f3edc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 -- 172.21.15.16:0/2448394957 shutdown_connections 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.690+0000 7f3ee418f640 1 -- 172.21.15.16:0/2448394957 wait complete. 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.691+0000 7f3ee418f640 1 Processor -- start 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.691+0000 7f3ee418f640 1 -- start start 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.691+0000 7f3ee418f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.691+0000 7f3ee418f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 0x7f3edc198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.691+0000 7f3ee418f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1082a0 0x7f3edc19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee418f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1120a0 con 0x7f3edc102d30 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee418f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111f20 con 0x7f3edc1036e0 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee418f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc112220 con 0x7f3edc1082a0 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43698/0 (socket says 172.21.15.16:43698) 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/3889695399 learned_addr learned my addr 172.21.15.16:0/3889695399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee272c640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1082a0 0x7f3edc19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 0x7f3edc198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/3889695399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1082a0 msgr2=0x7f3edc19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1082a0 0x7f3edc19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.692+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/3889695399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 msgr2=0x7f3edc198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 0x7f3edc198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/3889695399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc19d510 con 0x7f3edc102d30 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 0x7f3edc198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800e9f0 tx=0x7f3ed800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ed2ffd640 1 -- 172.21.15.16:0/3889695399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed800cd80 con 0x7f3edc102d30 2024-09-12T02:48:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ed2ffd640 1 -- 172.21.15.16:0/3889695399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed8010910 con 0x7f3edc102d30 2024-09-12T02:48:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc19d800 con 0x7f3edc102d30 2024-09-12T02:48:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.693+0000 7f3ed2ffd640 1 -- 172.21.15.16:0/3889695399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8018980 con 0x7f3edc102d30 2024-09-12T02:48:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.694+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1a50e0 con 0x7f3edc102d30 2024-09-12T02:48:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.696+0000 7f3ed2ffd640 1 -- 172.21.15.16:0/3889695399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f3ed8010ac0 con 0x7f3edc102d30 2024-09-12T02:48:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.697+0000 7f3ed2ffd640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 0x7f3ec80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.697+0000 7f3ee6568740 1 Processor -- start 2024-09-12T02:48:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.697+0000 7f3ed2ffd640 1 -- 172.21.15.16:0/3889695399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ed80cf600 con 0x7f3edc102d30 2024-09-12T02:48:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.697+0000 7f3ee6568740 1 -- start start 2024-09-12T02:48:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 0x7f3ec80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d03090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d10760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca12c692f0 con 0x55ca12d035d0 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ca12c10cf0 con 0x55ca12d04b40 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee6568740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ca12c652b0 con 0x55ca12d0e370 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43714/0 (socket says 172.21.15.16:43714) 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 -- 172.21.15.16:0/3904960201 learned_addr learned my addr 172.21.15.16:0/3904960201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee272c640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d10760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d03090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 -- 172.21.15.16:0/3904960201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 msgr2=0x55ca12d10760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d10760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 -- 172.21.15.16:0/3904960201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 msgr2=0x55ca12d03090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d03090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee172a640 1 -- 172.21.15.16:0/3904960201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca12bf53a0 con 0x55ca12d035d0 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.698+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d03090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.699+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009ea0 tx=0x7f3ecc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87180f8adceab3a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.699+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 0x7f3ec80a5150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc015820 tx=0x7f3ecc009a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.699+0000 7f3ec7fff640 1 -- 172.21.15.16:0/3904960201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc0150d0 con 0x55ca12d035d0 2024-09-12T02:48:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.699+0000 7f3ec7fff640 1 -- 172.21.15.16:0/3904960201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc015270 con 0x55ca12d035d0 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 -- 172.21.15.16:0/3904960201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 msgr2=0x55ca12d03a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009ea0 tx=0x7f3ecc01c000 comp rx=0 tx=0).stop 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 -- 172.21.15.16:0/3904960201 shutdown_connections 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d10760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d03a10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3904960201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d03090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 -- 172.21.15.16:0/3904960201 >> 172.21.15.16:0/3904960201 conn(0x55ca12ccc310 msgr2=0x55ca12cffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.700+0000 7f3ee6568740 1 -- 172.21.15.16:0/3904960201 shutdown_connections 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.701+0000 7f3ee6568740 1 -- 172.21.15.16:0/3904960201 wait complete. 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.701+0000 7f3ee6568740 1 -- start start 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.701+0000 7f3ee6568740 1 Processor -- start 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.702+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 0x55ca12d0dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.702+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.702+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d0b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 0x55ca12d0dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 0x55ca12d0dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47908/0 (socket says 172.21.15.16:47908) 2024-09-12T02:48:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43716/0 (socket says 172.21.15.16:43716) 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/1073163032 learned_addr learned my addr 172.21.15.16:0/1073163032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca12d15470 con 0x55ca12d04b40 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ca12c24060 con 0x55ca12d035d0 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ca12d155f0 con 0x55ca12d0e370 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 -- 172.21.15.16:0/1073163032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 msgr2=0x55ca12d0b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d0b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 -- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 msgr2=0x55ca12d0dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 0x55ca12d0dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee172a640 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca129da3c0 con 0x55ca12d04b40 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.703+0000 7f3ee6568740 10 client.0 _ll_register_callbacks cb 0x55ca129778c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca12c76860 con 0x55ca12d04b40 2024-09-12T02:48:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ee172a640 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009740 tx=0x7f3ecc0155a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3925fce4f7afd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ec57fa640 10 client.15561 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ec57fa640 1 -- 172.21.15.16:0/1073163032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc025040 con 0x55ca12d04b40 2024-09-12T02:48:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ec57fa640 1 -- 172.21.15.16:0/1073163032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ecc004680 con 0x55ca12d04b40 2024-09-12T02:48:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ec57fa640 1 -- 172.21.15.16:0/1073163032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ecc0150d0 con 0x55ca12d04b40 2024-09-12T02:48:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ee6568740 10 client.15561 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.704+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55ca12d20e70 con 0x55ca12d04b40 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ec57fa640 1 -- 172.21.15.16:0/1073163032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ecc00f040 con 0x55ca12d04b40 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 10 client.15561 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 20 client.15561 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 4 client.15561 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca12c70d70 con 0x55ca12d221c0 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.705+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ec57fa640 10 client.15561 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d245b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=671d93e9f13f31a8 server_cookie=233e2a07ff7ae5c3 in_seq=0 out_seq=0 2024-09-12T02:48:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ec57fa640 1 -- 172.21.15.16:0/1073163032 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca12c70d70 con 0x55ca12d221c0 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ec57fa640 10 client.15561 handle_command_reply: tid=0 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ee6568740 1 client.15561 shutdown 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 msgr2=0x55ca12d0b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.706+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009740 tx=0x7f3ecc0155a0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 shutdown_connections 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d245b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ec57fa640 0 client.15561 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d0b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d04b40 0x55ca12d0b090 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 --2- 172.21.15.16:0/1073163032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d035d0 0x55ca12d0dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 >> 172.21.15.16:0/1073163032 conn(0x55ca12ccc310 msgr2=0x55ca12d05930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 shutdown_connections 2024-09-12T02:48:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 1 -- 172.21.15.16:0/1073163032 wait complete. 2024-09-12T02:48:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.707+0000 7f3ee6568740 20 client.15561 trim_cache size 0 max 16384 2024-09-12T02:48:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.733+0000 7f3ee6568740 1 Processor -- start 2024-09-12T02:48:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.733+0000 7f3ee6568740 1 -- start start 2024-09-12T02:48:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.733+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.733+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d29c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d14ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee6568740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1ab400 con 0x55ca12d035d0 2024-09-12T02:48:33.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee6568740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1acf40 con 0x55ca12d04b40 2024-09-12T02:48:33.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee6568740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1abad0 con 0x55ca12d0e370 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee1f2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43722/0 (socket says 172.21.15.16:43722) 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee1f2b640 1 -- 172.21.15.16:0/109410245 learned_addr learned my addr 172.21.15.16:0/109410245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d29c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee272c640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d14ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 -- 172.21.15.16:0/109410245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 msgr2=0x55ca12d14ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d14ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 -- 172.21.15.16:0/109410245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 msgr2=0x55ca12d05820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee172a640 1 -- 172.21.15.16:0/109410245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca12d09fd0 con 0x55ca12d04b40 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.734+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.735+0000 7f3ee172a640 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d29c10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc005940 tx=0x7f3ecc028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494901648ad22652 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.735+0000 7f3ec77fe640 1 -- 172.21.15.16:0/109410245 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc025040 con 0x55ca12d04b40 2024-09-12T02:48:33.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.735+0000 7f3ec77fe640 1 -- 172.21.15.16:0/109410245 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca12c76860 con 0x55ca12d04b40 2024-09-12T02:48:33.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.735+0000 7f3ec77fe640 1 -- 172.21.15.16:0/109410245 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc038060 con 0x55ca12d04b40 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 -- 172.21.15.16:0/109410245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 msgr2=0x55ca12d29c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d29c10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc005940 tx=0x7f3ecc028000 comp rx=0 tx=0).stop 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 -- 172.21.15.16:0/109410245 shutdown_connections 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d0e370 0x55ca12d14ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d04b40 0x55ca12d29c10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 --2- 172.21.15.16:0/109410245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d05820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 -- 172.21.15.16:0/109410245 >> 172.21.15.16:0/109410245 conn(0x55ca12ccc310 msgr2=0x55ca12d05110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 -- 172.21.15.16:0/109410245 shutdown_connections 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.736+0000 7f3ee6568740 1 -- 172.21.15.16:0/109410245 wait complete. 2024-09-12T02:48:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.737+0000 7f3ee6568740 1 -- start start 2024-09-12T02:48:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.737+0000 7f3ee6568740 1 Processor -- start 2024-09-12T02:48:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.737+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d1a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.737+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.737+0000 7f3ee6568740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d0e370 0x55ca12d11b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46854/0 (socket says 172.21.15.16:46854) 2024-09-12T02:48:33.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 -- 172.21.15.16:0/3964658569 learned_addr learned my addr 172.21.15.16:0/3964658569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee272c640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d0e370 0x55ca12d11b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1ac870 con 0x55ca12d035d0 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1ac1a0 con 0x55ca12d0e370 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1a93e0 con 0x55ca12d04b40 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d1a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 -- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d0e370 msgr2=0x55ca12d11b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d0e370 0x55ca12d11b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 -- 172.21.15.16:0/3964658569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 msgr2=0x55ca12d1a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee6568740 10 client.0 _ll_register_callbacks cb 0x7f3ed8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d1a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca12d12150 con 0x55ca12d04b40 2024-09-12T02:48:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca12c76860 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d1a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ee172a640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc005940 tx=0x7f3ecc0056a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aad460fa0f1301d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.738+0000 7f3ec57fa640 10 client.25129 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.739+0000 7f3ec57fa640 1 -- 172.21.15.16:0/3964658569 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc025050 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.739+0000 7f3ec57fa640 1 -- 172.21.15.16:0/3964658569 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ecc004230 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.739+0000 7f3ec57fa640 1 -- 172.21.15.16:0/3964658569 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ecc038040 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.739+0000 7f3ee6568740 10 client.25129 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.739+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55ca12d18250 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ec57fa640 1 -- 172.21.15.16:0/3964658569 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ecc00e730 con 0x55ca12d04b40 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 10 client.25129 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 10 client.25129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 20 client.25129 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:33.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d933e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:33.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 4 client.25129 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:33.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca12bed170 con 0x55ca12d221c0 2024-09-12T02:48:33.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:33.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ee1f2b640 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d933e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d6f69fde573fa0b server_cookie=181d7e39101e01dd in_seq=0 out_seq=0 2024-09-12T02:48:33.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.740+0000 7f3ec57fa640 10 client.25129 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ec57fa640 1 -- 172.21.15.16:0/3964658569 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca12bed170 con 0x55ca12d221c0 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ec57fa640 10 client.25129 handle_command_reply: tid=0 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 client.25129 shutdown 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 msgr2=0x55ca12d1aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc005940 tx=0x7f3ecc0056a0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 shutdown_connections 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ca12d221c0 0x55ca12d933e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca12d0e370 0x55ca12d11b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca12d04b40 0x55ca12d1aea0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ec57fa640 0 client.25129 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 --2- 172.21.15.16:0/3964658569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca12d035d0 0x55ca12d1a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 >> 172.21.15.16:0/3964658569 conn(0x55ca12ccc310 msgr2=0x55ca12d05930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 shutdown_connections 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.741+0000 7f3ee6568740 1 -- 172.21.15.16:0/3964658569 wait complete. 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee6568740 20 client.25129 trim_cache size 0 max 16384 2024-09-12T02:48:33.750 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 msgr2=0x7f3ec80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 0x7f3ec80a5150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc015820 tx=0x7f3ecc009a80 comp rx=0 tx=0).stop 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 msgr2=0x7f3edc198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800e9f0 tx=0x7f3ed800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 shutdown_connections 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3edc1082a0 0x7f3edc19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc1036e0 0x7f3edc198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ec80a2c90 0x7f3ec80a5150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 --2- 172.21.15.16:0/3889695399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3edc102d30 0x7f3edc198560 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:33.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 >> 172.21.15.16:0/3889695399 conn(0x7f3edc0fc7d0 msgr2=0x7f3edc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:33.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.742+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 shutdown_connections 2024-09-12T02:48:33.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:33.743+0000 7f3ee418f640 1 -- 172.21.15.16:0/3889695399 wait complete. 2024-09-12T02:48:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:33 smithi033 ceph-mon[35092]: pgmap v356: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T02:48:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:33 smithi184 ceph-mon[35321]: pgmap v356: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T02:48:34.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:33 smithi016 ceph-mon[30925]: pgmap v356: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T02:48:35.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:34 smithi184 ceph-mon[35321]: pgmap v357: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-12T02:48:35.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:34 smithi016 ceph-mon[30925]: pgmap v357: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-12T02:48:35.445 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:48:35.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:48:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:34 smithi033 ceph-mon[35092]: pgmap v357: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-12T02:48:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 -- 172.21.15.16:0/2702073093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102d50 msgr2=0x7f7c48103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/2702073093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102d50 0x7f7c48103130 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c4402a480 comp rx=0 tx=0).stop 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 -- 172.21.15.16:0/2702073093 shutdown_connections 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/2702073093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c4810c180 0x7f7c4810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/2702073093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48103670 0x7f7c4810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/2702073093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102d50 0x7f7c48103130 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.218+0000 7f7c4fb09640 1 -- 172.21.15.16:0/2702073093 >> 172.21.15.16:0/2702073093 conn(0x7f7c480fc7d0 msgr2=0x7f7c480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.219+0000 7f7c4fb09640 1 -- 172.21.15.16:0/2702073093 shutdown_connections 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.219+0000 7f7c4fb09640 1 -- 172.21.15.16:0/2702073093 wait complete. 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.219+0000 7f7c4fb09640 1 Processor -- start 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.219+0000 7f7c4fb09640 1 -- start start 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 0x7f7c481989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 0x7f7c4819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48111f80 con 0x7f7c4810c180 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48111e00 con 0x7f7c48102d50 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4fb09640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48112100 con 0x7f7c48103670 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 0x7f7c481989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54972/0 (socket says 172.21.15.16:54972) 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 0x7f7c481989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32852/0 (socket says 172.21.15.16:32852) 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 -- 172.21.15.16:0/1592483100 learned_addr learned my addr 172.21.15.16:0/1592483100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 0x7f7c4819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 -- 172.21.15.16:0/1592483100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 msgr2=0x7f7c481989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 0x7f7c481989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.220+0000 7f7c4d8a5640 1 -- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 msgr2=0x7f7c4819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4d8a5640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 0x7f7c4819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4d8a5640 1 -- 172.21.15.16:0/1592483100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4819d450 con 0x7f7c48102d50 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 0x7f7c4819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4d8a5640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006010 tx=0x7f7c440042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c3effd640 1 -- 172.21.15.16:0/1592483100 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4403f030 con 0x7f7c48102d50 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4819d6e0 con 0x7f7c48102d50 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481008e0 con 0x7f7c48102d50 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.221+0000 7f7c3effd640 1 -- 172.21.15.16:0/1592483100 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44003ce0 con 0x7f7c48102d50 2024-09-12T02:48:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.222+0000 7f7c3effd640 1 -- 172.21.15.16:0/1592483100 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44035070 con 0x7f7c48102d50 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.225+0000 7f7c51ee2740 1 Processor -- start 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.225+0000 7f7c51ee2740 1 -- start start 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.225+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.225+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab6827f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ab681d7280 con 0x55ab6827cd40 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ab6817eb10 con 0x55ab68272480 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ab681d3240 con 0x55ab68271b60 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32862/0 (socket says 172.21.15.16:32862) 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/2227654092 learned_addr learned my addr 172.21.15.16:0/2227654092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab6827f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/2227654092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 msgr2=0x55ab6827c800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/2227654092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 msgr2=0x55ab6827f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c3effd640 1 -- 172.21.15.16:0/1592483100 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f7c44042030 con 0x7f7c48102d50 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab6827f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/2227654092 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab6818b1f0 con 0x55ab68271b60 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.226+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab6827f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.227+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009ac0 tx=0x7f7c40009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fb968164c0ddce server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.227+0000 7f7c33fff640 1 -- 172.21.15.16:0/2227654092 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40025070 con 0x55ab68271b60 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.227+0000 7f7c33fff640 1 -- 172.21.15.16:0/2227654092 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c40003e50 con 0x55ab68271b60 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.227+0000 7f7c33fff640 1 -- 172.21.15.16:0/2227654092 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40020480 con 0x55ab68271b60 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.227+0000 7f7c3effd640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 0x7f7c340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c3effd640 1 -- 172.21.15.16:0/1592483100 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c44003800 con 0x7f7c48102d50 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2227654092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 msgr2=0x55ab68271f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009ac0 tx=0x7f7c40009500 comp rx=0 tx=0).stop 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2227654092 shutdown_connections 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab6827f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2227654092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab68271f40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2227654092 >> 172.21.15.16:0/2227654092 conn(0x55ab6826e0f0 msgr2=0x55ab6826e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.228+0000 7f7c4d8a5640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 0x7f7c340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.229+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2227654092 shutdown_connections 2024-09-12T02:48:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.229+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2227654092 wait complete. 2024-09-12T02:48:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.229+0000 7f7c4d8a5640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 0x7f7c340a52d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ab6826ef50 tx=0x7f7c44005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.229+0000 7f7c51ee2740 1 -- start start 2024-09-12T02:48:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 Processor -- start 2024-09-12T02:48:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab6827a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ab6827f960 con 0x55ab68271b60 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ab68191f10 con 0x55ab68272480 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ab6827fae0 con 0x55ab6827cd40 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c51ee2740 10 client.0 _ll_register_callbacks cb 0x55ab6815aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.230+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42682/0 (socket says 172.21.15.16:42682) 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/3560373760 learned_addr learned my addr 172.21.15.16:0/3560373760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/3560373760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 msgr2=0x55ab6827a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab6827a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/3560373760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 msgr2=0x55ab68279bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/3560373760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab67f486a0 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827a100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c4810d500 tx=0x7f7c3800e7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39cfe57818857b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.231+0000 7f7c317fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3801e060 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c380035f0 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c40003e50 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c380037f0 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c51ee2740 10 client.25232 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab6828b630 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.232+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c40003e50 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7c38019a10 con 0x55ab68272480 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 10 client.25232 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 10 client.25232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 20 client.25232 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab6828f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 4 client.25232 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.233+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab681e47f0 con 0x55ab6828cbc0 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.234+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab6828f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.234+0000 7f7c317fa640 10 client.25232 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.234+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab6828f000 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c93c5bbc1253d2 server_cookie=1f21da22d26958b4 in_seq=0 out_seq=0 2024-09-12T02:48:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.235+0000 7f7c317fa640 1 -- 172.21.15.16:0/3560373760 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ab681e47f0 con 0x55ab6828cbc0 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.235+0000 7f7c317fa640 10 client.25232 handle_command_reply: tid=0 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.235+0000 7f7c51ee2740 1 client.25232 shutdown 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.235+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 msgr2=0x55ab6827a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.235+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827a100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c4810d500 tx=0x7f7c3800e7e0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 shutdown_connections 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab6828f000 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab6827a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c317fa640 0 client.25232 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827a100 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/3560373760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68271b60 0x55ab68279bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 >> 172.21.15.16:0/3560373760 conn(0x55ab6826e0f0 msgr2=0x55ab6826d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 shutdown_connections 2024-09-12T02:48:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 1 -- 172.21.15.16:0/3560373760 wait complete. 2024-09-12T02:48:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.236+0000 7f7c51ee2740 20 client.25232 trim_cache size 0 max 16384 2024-09-12T02:48:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 Processor -- start 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 -- start start 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab6827a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab682948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c00e4c0 con 0x55ab6827cd40 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c010000 con 0x55ab68272480 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.254+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c00eb90 con 0x55ab68271b60 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54982/0 (socket says 172.21.15.16:54982) 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/1606067287 learned_addr learned my addr 172.21.15.16:0/1606067287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab682948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab6827a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/1606067287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 msgr2=0x55ab6827a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab6827a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/1606067287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 msgr2=0x55ab682948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab682948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/1606067287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab682722b0 con 0x55ab68272480 2024-09-12T02:48:36.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.255+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c380058c0 tx=0x7f7c38003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd52e342e4446de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.256+0000 7f7c337fe640 1 -- 172.21.15.16:0/1606067287 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38033070 con 0x55ab68272480 2024-09-12T02:48:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.256+0000 7f7c337fe640 1 -- 172.21.15.16:0/1606067287 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c38003e70 con 0x55ab68272480 2024-09-12T02:48:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.256+0000 7f7c337fe640 1 -- 172.21.15.16:0/1606067287 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38029040 con 0x55ab68272480 2024-09-12T02:48:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- 172.21.15.16:0/1606067287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 msgr2=0x55ab6827ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c380058c0 tx=0x7f7c38003780 comp rx=0 tx=0).stop 2024-09-12T02:48:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- 172.21.15.16:0/1606067287 shutdown_connections 2024-09-12T02:48:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab6827cd40 0x55ab682948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68272480 0x55ab6827ba20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/1606067287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab68271b60 0x55ab6827a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- 172.21.15.16:0/1606067287 >> 172.21.15.16:0/1606067287 conn(0x55ab6826e0f0 msgr2=0x55ab6827b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- 172.21.15.16:0/1606067287 shutdown_connections 2024-09-12T02:48:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- 172.21.15.16:0/1606067287 wait complete. 2024-09-12T02:48:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 -- start start 2024-09-12T02:48:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.257+0000 7f7c51ee2740 1 Processor -- start 2024-09-12T02:48:36.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 0x55ab68276c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 0x55ab68277150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab68277690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c00f930 con 0x55ab68272480 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c00f260 con 0x55ab68271b60 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c00c4a0 con 0x55ab6827cd40 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c51ee2740 10 client.0 _ll_register_callbacks cb 0x7f7c3800f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 0x55ab68276c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c4d0a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 0x55ab68277150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c4e0a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 0x55ab68276c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54996/0 (socket says 172.21.15.16:54996) 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c4e0a6640 1 -- 172.21.15.16:0/2372824657 learned_addr learned my addr 172.21.15.16:0/2372824657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.258+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab68277690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/2372824657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 msgr2=0x55ab68277690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab68277690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 msgr2=0x55ab68276c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 0x55ab68276c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 -- 172.21.15.16:0/2372824657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab682f68b0 con 0x55ab68272480 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c4d0a4640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 0x55ab68277150 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x55ab682694e0 tx=0x7f7c380055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e2c1b270acbc70 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38029070 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c38003990 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c38003e70 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.259+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c3803c050 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 10 client.15585 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab682f7bb0 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c38003e70 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7c3802c070 con 0x55ab68272480 2024-09-12T02:48:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 10 client.15585 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 20 client.15585 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab682fa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 4 client.15585 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ab681ded00 con 0x55ab6828cbc0 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.260+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab682fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.261+0000 7f7c4e0a6640 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab682fa250 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e72ebd15c09094 server_cookie=7928521ea7af5e7c in_seq=0 out_seq=0 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.261+0000 7f7c317fa640 10 client.15585 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c317fa640 1 -- 172.21.15.16:0/2372824657 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ab681ded00 con 0x55ab6828cbc0 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c317fa640 10 client.15585 handle_command_reply: tid=0 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 client.15585 shutdown 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 msgr2=0x55ab68277150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 0x55ab68277150 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x55ab682694e0 tx=0x7f7c380055d0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 shutdown_connections 2024-09-12T02:48:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ab6828cbc0 0x55ab682fa250 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ab6827cd40 0x55ab68277690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.273 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:48:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "7de48b5b-4567-47d2-b959-3fbdb8a817f8", 2024-09-12T02:48:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:48:36.274 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:36.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c317fa640 0 client.15585 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:36.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab68272480 0x55ab68277150 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 --2- 172.21.15.16:0/2372824657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab68271b60 0x55ab68276c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 >> 172.21.15.16:0/2372824657 conn(0x55ab6826e0f0 msgr2=0x55ab6826d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 shutdown_connections 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 1 -- 172.21.15.16:0/2372824657 wait complete. 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.262+0000 7f7c51ee2740 20 client.15585 trim_cache size 0 max 16384 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 msgr2=0x7f7c340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 0x7f7c340a52d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ab6826ef50 tx=0x7f7c44005e50 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 msgr2=0x7f7c481984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006010 tx=0x7f7c440042f0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 shutdown_connections 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2e10 0x7f7c340a52d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4810c180 0x7f7c4819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48103670 0x7f7c481989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 --2- 172.21.15.16:0/1592483100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102d50 0x7f7c481984a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 >> 172.21.15.16:0/1592483100 conn(0x7f7c480fc7d0 msgr2=0x7f7c480fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:36.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 shutdown_connections 2024-09-12T02:48:36.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:36.263+0000 7f7c4fb09640 1 -- 172.21.15.16:0/1592483100 wait complete. 2024-09-12T02:48:37.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:37 smithi016 ceph-mon[30925]: pgmap v358: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:37.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:37 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:48:37.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:37.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:48:37.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:37.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:48:37.438 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:37 smithi033 ceph-mon[35092]: pgmap v358: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:37 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:48:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:48:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:37 smithi184 ceph-mon[35321]: pgmap v358: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-12T02:48:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:37 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:48:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:48:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.102+0000 7f6fd6ffd640 1 -- 172.21.15.16:0/1304133454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00319b0 con 0x7f6fd8101290 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 -- 172.21.15.16:0/1304133454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8101290 msgr2=0x7f6fd8107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 --2- 172.21.15.16:0/1304133454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8101290 0x7f6fd8107a50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009930 tx=0x7f6fc002a440 comp rx=0 tx=0).stop 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 -- 172.21.15.16:0/1304133454 shutdown_connections 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 --2- 172.21.15.16:0/1304133454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8108160 0x7f6fd810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 --2- 172.21.15.16:0/1304133454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8101290 0x7f6fd8107a50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 --2- 172.21.15.16:0/1304133454 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 -- 172.21.15.16:0/1304133454 >> 172.21.15.16:0/1304133454 conn(0x7f6fd8077210 msgr2=0x7f6fd8077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 -- 172.21.15.16:0/1304133454 shutdown_connections 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.103+0000 7f6fddca2640 1 -- 172.21.15.16:0/1304133454 wait complete. 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 Processor -- start 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 -- start start 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd810e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 0x7f6fd810e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fddca2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8101b20 con 0x7f6fd8108160 2024-09-12T02:48:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.104+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd810e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd810e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55012/0 (socket says 172.21.15.16:55012) 2024-09-12T02:48:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42718/0 (socket says 172.21.15.16:42718) 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd77fe640 1 -- 172.21.15.16:0/394666490 learned_addr learned my addr 172.21.15.16:0/394666490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 0x7f6fd810e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81019a0 con 0x7f6fd81008e0 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8101ca0 con 0x7f6fd8101290 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 -- 172.21.15.16:0/394666490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 msgr2=0x7f6fd810e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 0x7f6fd810e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 -- 172.21.15.16:0/394666490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 msgr2=0x7f6fd810e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.105+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd810e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.106+0000 7f6fd7fff640 1 -- 172.21.15.16:0/394666490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd810abb0 con 0x7f6fd8108160 2024-09-12T02:48:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.106+0000 7f6fcffff640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 0x7f6fd810e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.106+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009fd0 tx=0x7f6fd000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.107+0000 7f6fd57fa640 1 -- 172.21.15.16:0/394666490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000ce20 con 0x7f6fd8108160 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.107+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd810aea0 con 0x7f6fd8108160 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.107+0000 7f6fd57fa640 1 -- 172.21.15.16:0/394666490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd0005020 con 0x7f6fd8108160 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.107+0000 7f6fd57fa640 1 -- 172.21.15.16:0/394666490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0010720 con 0x7f6fd8108160 2024-09-12T02:48:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.107+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd81ab7f0 con 0x7f6fd8108160 2024-09-12T02:48:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.110+0000 7f6fd57fa640 1 -- 172.21.15.16:0/394666490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f6fd0005540 con 0x7f6fd8108160 2024-09-12T02:48:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 Processor -- start 2024-09-12T02:48:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 -- start start 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35e7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35f2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647a354d2e0 con 0x5647a35f2680 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647a34f4ce0 con 0x5647a35e74a0 2024-09-12T02:48:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fe007b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647a35492a0 con 0x5647a35e7dc0 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42722/0 (socket says 172.21.15.16:42722) 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.111+0000 7f6fcffff640 1 -- 172.21.15.16:0/1728690330 learned_addr learned my addr 172.21.15.16:0/1728690330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 -- 172.21.15.16:0/1728690330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 msgr2=0x5647a35f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 -- 172.21.15.16:0/1728690330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 msgr2=0x5647a35e7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35e7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 -- 172.21.15.16:0/1728690330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a34d9440 con 0x5647a35f2680 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fd57fa640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 0x7f6fc40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.112+0000 7f6fcffff640 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8078760 tx=0x7f6fc0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fc589368487f12 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.113+0000 7f6fd57fa640 1 -- 172.21.15.16:0/394666490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fd00cf7e0 con 0x7f6fd8108160 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.113+0000 7f6fcdffb640 1 -- 172.21.15.16:0/1728690330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0035040 con 0x5647a35f2680 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.113+0000 7f6fcdffb640 1 -- 172.21.15.16:0/1728690330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc0005030 con 0x5647a35f2680 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.113+0000 7f6fcdffb640 1 -- 172.21.15.16:0/1728690330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0004920 con 0x5647a35f2680 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.113+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 0x7f6fc40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 -- 172.21.15.16:0/1728690330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 msgr2=0x5647a35f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8078760 tx=0x7f6fc0000bc0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 0x7f6fc40a5150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x5647a35e4890 tx=0x7f6fc8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 -- 172.21.15.16:0/1728690330 shutdown_connections 2024-09-12T02:48:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35f2680 0x5647a35f4a70 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 --2- 172.21.15.16:0/1728690330 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35e7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 -- 172.21.15.16:0/1728690330 >> 172.21.15.16:0/1728690330 conn(0x5647a35e3a30 msgr2=0x5647a35e3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 -- 172.21.15.16:0/1728690330 shutdown_connections 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.114+0000 7f6fe007b740 1 -- 172.21.15.16:0/1728690330 wait complete. 2024-09-12T02:48:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.115+0000 7f6fe007b740 1 -- start start 2024-09-12T02:48:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.115+0000 7f6fe007b740 1 Processor -- start 2024-09-12T02:48:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35efa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647a35f5250 con 0x5647a35eff70 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647a3508050 con 0x5647a35e74a0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647a35f53d0 con 0x5647a35e7dc0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42732/0 (socket says 172.21.15.16:42732) 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 -- 172.21.15.16:0/2569708585 learned_addr learned my addr 172.21.15.16:0/2569708585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fe007b740 10 client.0 _ll_register_callbacks cb 0x5647a3548160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fcffff640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 -- 172.21.15.16:0/2569708585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 msgr2=0x5647a35efa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35efa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 -- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 msgr2=0x5647a35ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35ef4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd7fff640 1 -- 172.21.15.16:0/2569708585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a32be3c0 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.116+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.117+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00a4630 tx=0x7f6fd009c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169373b0fad35cf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.117+0000 7f6fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00a46d0 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fd00a4870 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc0005030 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fd00bab30 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fe007b740 10 client.15597 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.118+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647a3601300 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd00bacd0 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fd009adb0 con 0x5647a35eff70 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 10 client.15597 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 20 client.15597 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a3602710 0x5647a3604b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 4 client.15597 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6fc0005030 con 0x5647a3602710 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.119+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a3602710 0x5647a3604b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.120+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a3602710 0x5647a3604b00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28456b1129db5285 server_cookie=2bdab93f1a05797e in_seq=0 out_seq=0 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.120+0000 7f6fb6ffd640 10 client.15597 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/2569708585 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f6fc0005030 con 0x5647a3602710 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fb6ffd640 10 client.15597 handle_command_reply: tid=0 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 client.15597 shutdown 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 msgr2=0x5647a35fe6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00a4630 tx=0x7f6fd009c730 comp rx=0 tx=0).stop 2024-09-12T02:48:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 shutdown_connections 2024-09-12T02:48:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a3602710 0x5647a3604b00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35eff70 0x5647a35fe6c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e7dc0 0x5647a35efa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 --2- 172.21.15.16:0/2569708585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e74a0 0x5647a35ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 >> 172.21.15.16:0/2569708585 conn(0x5647a35e3a30 msgr2=0x5647a35e9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fb6ffd640 0 client.15597 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 shutdown_connections 2024-09-12T02:48:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 1 -- 172.21.15.16:0/2569708585 wait complete. 2024-09-12T02:48:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.121+0000 7f6fe007b740 20 client.15597 trim_cache size 0 max 16384 2024-09-12T02:48:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.156+0000 7f6fe007b740 1 Processor -- start 2024-09-12T02:48:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.156+0000 7f6fe007b740 1 -- start start 2024-09-12T02:48:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e3a30 0x5647a35f1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81a3010 con 0x5647a35e74a0 2024-09-12T02:48:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81acf50 con 0x5647a35e7dc0 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fe007b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81abae0 con 0x5647a35e3a30 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e3a30 0x5647a35f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42748/0 (socket says 172.21.15.16:42748) 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fcffff640 1 -- 172.21.15.16:0/9898590 learned_addr learned my addr 172.21.15.16:0/9898590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.157+0000 7f6fcffff640 1 -- 172.21.15.16:0/9898590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e3a30 msgr2=0x5647a35f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcffff640 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e3a30 0x5647a35f1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcffff640 1 -- 172.21.15.16:0/9898590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 msgr2=0x5647a35fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcffff640 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcffff640 1 -- 172.21.15.16:0/9898590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a35e7bc0 con 0x5647a35e74a0 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fd7fff640 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcffff640 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0031550 tx=0x7f6fc0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91562c891bb55b39 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcd7fa640 1 -- 172.21.15.16:0/9898590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc004e070 con 0x5647a35e74a0 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcd7fa640 1 -- 172.21.15.16:0/9898590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc0032e80 con 0x5647a35e74a0 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.158+0000 7f6fcd7fa640 1 -- 172.21.15.16:0/9898590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0035040 con 0x5647a35e74a0 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 -- 172.21.15.16:0/9898590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 msgr2=0x5647a35fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0031550 tx=0x7f6fc0009260 comp rx=0 tx=0).stop 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 -- 172.21.15.16:0/9898590 shutdown_connections 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e74a0 0x5647a35fe4b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.159+0000 7f6fe007b740 1 --2- 172.21.15.16:0/9898590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e3a30 0x5647a35f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.160+0000 7f6fe007b740 1 -- 172.21.15.16:0/9898590 >> 172.21.15.16:0/9898590 conn(0x5647a35efb90 msgr2=0x5647a35f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.160+0000 7f6fe007b740 1 -- 172.21.15.16:0/9898590 shutdown_connections 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.160+0000 7f6fe007b740 1 -- 172.21.15.16:0/9898590 wait complete. 2024-09-12T02:48:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.160+0000 7f6fe007b740 1 -- start start 2024-09-12T02:48:38.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.160+0000 7f6fe007b740 1 Processor -- start 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42760/0 (socket says 172.21.15.16:42760) 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fd77fe640 1 -- 172.21.15.16:0/3269762134 learned_addr learned my addr 172.21.15.16:0/3269762134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e74a0 0x5647a35f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81ac880 con 0x5647a35e3a30 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81ac1b0 con 0x5647a35e7dc0 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8100cc0 con 0x5647a35e74a0 2024-09-12T02:48:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.161+0000 7f6fcffff640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e74a0 0x5647a35f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 -- 172.21.15.16:0/3269762134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e74a0 msgr2=0x5647a35f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e74a0 0x5647a35f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 -- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 msgr2=0x5647a35f63c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35f63c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a35f69b0 con 0x5647a35e3a30 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fe007b740 10 client.0 _ll_register_callbacks cb 0x7f6fd0098190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc0032e80 con 0x5647a35e3a30 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fd77fe640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009fd0 tx=0x7f6fc8004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a6518b27d1617e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.162+0000 7f6fb6ffd640 10 client.15603 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/3269762134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8023bf0 con 0x5647a35e3a30 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/3269762134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fc800f040 con 0x5647a35e3a30 2024-09-12T02:48:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/3269762134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fc8023d90 con 0x5647a35e3a30 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 10 client.15603 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5647a366cfa0 con 0x5647a35e3a30 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/3269762134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fc8005990 con 0x5647a35e3a30 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 10 client.15603 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 20 client.15603 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a35eff70 0x5647a366f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 4 client.15603 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.163+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5647a355a850 con 0x5647a35eff70 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.164+0000 7f6fcffff640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a35eff70 0x5647a366f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.164+0000 7f6fb6ffd640 10 client.15603 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.164+0000 7f6fcffff640 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a35eff70 0x5647a366f7c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce3ad7f23986a5fb server_cookie=2c1c0d831b06a169 in_seq=0 out_seq=0 2024-09-12T02:48:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fb6ffd640 1 -- 172.21.15.16:0/3269762134 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5647a355a850 con 0x5647a35eff70 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fb6ffd640 10 client.15603 handle_command_reply: tid=0 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 client.15603 shutdown 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 msgr2=0x5647a35fab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009fd0 tx=0x7f6fc8004700 comp rx=0 tx=0).stop 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 shutdown_connections 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647a35eff70 0x5647a366f7c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fb6ffd640 0 client.15603 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647a35e7dc0 0x5647a35f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:38.171 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:48:38.172 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:48:38.172 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647a35e74a0 0x5647a35f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 --2- 172.21.15.16:0/3269762134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647a35e3a30 0x5647a35fab50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 >> 172.21.15.16:0/3269762134 conn(0x5647a35efb90 msgr2=0x5647a35ea8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 shutdown_connections 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 1 -- 172.21.15.16:0/3269762134 wait complete. 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.165+0000 7f6fe007b740 20 client.15603 trim_cache size 0 max 16384 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 msgr2=0x7f6fc40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 0x7f6fc40a5150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x5647a35e4890 tx=0x7f6fc8009190 comp rx=0 tx=0).stop 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 msgr2=0x7f6fd810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009fd0 tx=0x7f6fd000ef10 comp rx=0 tx=0).stop 2024-09-12T02:48:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 shutdown_connections 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc40a2c90 0x7f6fc40a5150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8108160 0x7f6fd810a3e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd8101290 0x7f6fd810e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 --2- 172.21.15.16:0/394666490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd81008e0 0x7f6fd810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 >> 172.21.15.16:0/394666490 conn(0x7f6fd8077210 msgr2=0x7f6fd80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 shutdown_connections 2024-09-12T02:48:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:38.166+0000 7f6fddca2640 1 -- 172.21.15.16:0/394666490 wait complete. 2024-09-12T02:48:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:48:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:48:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:48:38.881 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:48:38.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:48:39.202 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:39.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:39 smithi016 ceph-mon[30925]: pgmap v359: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-12T02:48:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:39 smithi033 ceph-mon[35092]: pgmap v359: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-12T02:48:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:39 smithi184 ceph-mon[35321]: pgmap v359: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-09-12T02:48:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 -- 172.21.15.16:0/3927295634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c1082a0 msgr2=0x7fb07c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3927295634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c1082a0 0x7fb07c10ec40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb074009cd0 tx=0x7fb07402a910 comp rx=0 tx=0).stop 2024-09-12T02:48:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 -- 172.21.15.16:0/3927295634 shutdown_connections 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3927295634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c1082a0 0x7fb07c10ec40 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3927295634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3927295634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c102d30 0x7fb07c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 -- 172.21.15.16:0/3927295634 >> 172.21.15.16:0/3927295634 conn(0x7fb07c0fc7d0 msgr2=0x7fb07c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 -- 172.21.15.16:0/3927295634 shutdown_connections 2024-09-12T02:48:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.814+0000 7fb0808ee640 1 -- 172.21.15.16:0/3927295634 wait complete. 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.815+0000 7fb0808ee640 1 Processor -- start 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 -- start start 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 0x7fb07c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c1082a0 0x7fb07c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb07a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 0x7fb07c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb07a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 0x7fb07c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42778/0 (socket says 172.21.15.16:42778) 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb07a59c640 1 -- 172.21.15.16:0/3900804038 learned_addr learned my addr 172.21.15.16:0/3900804038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb079d9b640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c1120a0 con 0x7fb07c102d30 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c111f20 con 0x7fb07c1082a0 2024-09-12T02:48:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.816+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c112220 con 0x7fb07c1036e0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb07ad9d640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c1082a0 0x7fb07c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 -- 172.21.15.16:0/3900804038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c1082a0 msgr2=0x7fb07c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c1082a0 0x7fb07c19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 -- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 msgr2=0x7fb07c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 0x7fb07c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c19d4f0 con 0x7fb07c1036e0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb079d9b640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c198a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0697d0 tx=0x7fb07000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.817+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07000cc40 con 0x7fb07c1036e0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.818+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c19d7e0 con 0x7fb07c1036e0 2024-09-12T02:48:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.818+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb07000cde0 con 0x7fb07c1036e0 2024-09-12T02:48:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.818+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070018980 con 0x7fb07c1036e0 2024-09-12T02:48:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.818+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07c1008c0 con 0x7fb07c1036e0 2024-09-12T02:48:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.819+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07c06a850 con 0x7fb07c1036e0 2024-09-12T02:48:39.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.821+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fb070018b20 con 0x7fb07c1036e0 2024-09-12T02:48:39.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.822+0000 7fb06b7fe640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 0x7fb0640a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:39.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.822+0000 7fb07a59c640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 0x7fb0640a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:39.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.822+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0700cf330 con 0x7fb07c1036e0 2024-09-12T02:48:39.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.823+0000 7fb07a59c640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 0x7fb0640a5150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb06c005fd0 tx=0x7fb06c00b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:39.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:39.824+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb07009baf0 con 0x7fb07c1036e0 2024-09-12T02:48:40.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.036+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb07c103110 con 0x7fb07c1036e0 2024-09-12T02:48:40.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.038+0000 7fb06b7fe640 1 -- 172.21.15.16:0/3900804038 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7fb07c103110 con 0x7fb07c1036e0 2024-09-12T02:48:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:48:40.040 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:48:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.040+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 msgr2=0x7fb0640a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.040+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 0x7fb0640a5150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb06c005fd0 tx=0x7fb06c00b4a0 comp rx=0 tx=0).stop 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.040+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 msgr2=0x7fb07c198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.040+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c198a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0697d0 tx=0x7fb07000ee50 comp rx=0 tx=0).stop 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 shutdown_connections 2024-09-12T02:48:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb07c1082a0 0x7fb07c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb07c1036e0 0x7fb07c198a80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0640a2c90 0x7fb0640a5150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 --2- 172.21.15.16:0/3900804038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb07c102d30 0x7fb07c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 >> 172.21.15.16:0/3900804038 conn(0x7fb07c0fc7d0 msgr2=0x7fb07c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 shutdown_connections 2024-09-12T02:48:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:40.041+0000 7fb0808ee640 1 -- 172.21.15.16:0/3900804038 wait complete. 2024-09-12T02:48:40.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:40 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3900804038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:40.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:40 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3900804038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:40.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:40 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3900804038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:40.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:48:41.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:41 smithi016 ceph-mon[30925]: pgmap v360: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-12T02:48:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:41 smithi033 ceph-mon[35092]: pgmap v360: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-12T02:48:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:41 smithi184 ceph-mon[35321]: pgmap v360: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 -- 172.21.15.16:0/58801764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4101410 msgr2=0x7fc1b410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 --2- 172.21.15.16:0/58801764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4101410 0x7fc1b410d940 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009cd0 tx=0x7fc1ac02a8e0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 -- 172.21.15.16:0/58801764 shutdown_connections 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 --2- 172.21.15.16:0/58801764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4101410 0x7fc1b410d940 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 --2- 172.21.15.16:0/58801764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4100a70 0x7fc1b4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 --2- 172.21.15.16:0/58801764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1b4106a70 0x7fc1b4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 -- 172.21.15.16:0/58801764 >> 172.21.15.16:0/58801764 conn(0x7fc1b40fc7d0 msgr2=0x7fc1b40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 -- 172.21.15.16:0/58801764 shutdown_connections 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.721+0000 7fc1bc474640 1 -- 172.21.15.16:0/58801764 wait complete. 2024-09-12T02:48:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.722+0000 7fc1bc474640 1 Processor -- start 2024-09-12T02:48:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 -- start start 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1ba210640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1ba210640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42794/0 (socket says 172.21.15.16:42794) 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1b4101410 0x7fc1b4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1ba210640 1 -- 172.21.15.16:0/546280754 learned_addr learned my addr 172.21.15.16:0/546280754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 0x7fc1b419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4111fd0 con 0x7fc1b4100a70 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4111e50 con 0x7fc1b4101410 2024-09-12T02:48:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.723+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4112150 con 0x7fc1b4106a70 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1baa11640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 0x7fc1b419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 -- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 msgr2=0x7fc1b419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 0x7fc1b419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 -- 172.21.15.16:0/546280754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1b4101410 msgr2=0x7fc1b4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1b4101410 0x7fc1b4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b419d400 con 0x7fc1b4100a70 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1baa11640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 0x7fc1b419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ba210640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d950 tx=0x7fc1b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ab7fe640 1 -- 172.21.15.16:0/546280754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0017070 con 0x7fc1b4100a70 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ab7fe640 1 -- 172.21.15.16:0/546280754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1b0009c30 con 0x7fc1b4100a70 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1ab7fe640 1 -- 172.21.15.16:0/546280754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b000f6b0 con 0x7fc1b4100a70 2024-09-12T02:48:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.724+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b419d6f0 con 0x7fc1b4100a70 2024-09-12T02:48:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.725+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b41a4fd0 con 0x7fc1b4100a70 2024-09-12T02:48:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.728+0000 7fc1be84d740 1 Processor -- start 2024-09-12T02:48:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.728+0000 7fc1ab7fe640 1 -- 172.21.15.16:0/546280754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fc1b00109d0 con 0x7fc1b4100a70 2024-09-12T02:48:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.728+0000 7fc1be84d740 1 -- start start 2024-09-12T02:48:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e8905d0 0x55828e890a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55828e7f62f0 con 0x55828e891b40 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55828e79dcf0 con 0x55828e89f780 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1be84d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55828e7f22b0 con 0x55828e8905d0 2024-09-12T02:48:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42796/0 (socket says 172.21.15.16:42796) 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1baa11640 1 -- 172.21.15.16:0/1791324748 learned_addr learned my addr 172.21.15.16:0/1791324748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.729+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 -- 172.21.15.16:0/1791324748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e8905d0 msgr2=0x55828e890a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e8905d0 0x55828e890a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 -- 172.21.15.16:0/1791324748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 msgr2=0x55828e8a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 -- 172.21.15.16:0/1791324748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55828e7823a0 con 0x55828e891b40 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.730+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009df0 tx=0x7fc1ac031430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480d1e798a644c20 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1ab7fe640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 0x7fc1a00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1ba210640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 0x7fc1a00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1ab7fe640 1 -- 172.21.15.16:0/546280754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc1b00ced90 con 0x7fc1b4100a70 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1a8dea640 1 -- 172.21.15.16:0/1791324748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac0318c0 con 0x55828e891b40 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1ba210640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 0x7fc1a00a5150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55828e88dac0 tx=0x7fc1b0066510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.731+0000 7fc1a8dea640 1 -- 172.21.15.16:0/1791324748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1ac036040 con 0x55828e891b40 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.732+0000 7fc1a8dea640 1 -- 172.21.15.16:0/1791324748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac004670 con 0x55828e891b40 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.732+0000 7fc1be84d740 1 -- 172.21.15.16:0/1791324748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 msgr2=0x55828e890090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.732+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009df0 tx=0x7fc1ac031430 comp rx=0 tx=0).stop 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 -- 172.21.15.16:0/1791324748 shutdown_connections 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e8905d0 0x55828e890a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1791324748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e890090 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 -- 172.21.15.16:0/1791324748 >> 172.21.15.16:0/1791324748 conn(0x55828e859310 msgr2=0x55828e88cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 -- 172.21.15.16:0/1791324748 shutdown_connections 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 -- 172.21.15.16:0/1791324748 wait complete. 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 -- start start 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.733+0000 7fc1be84d740 1 Processor -- start 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e89c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e89ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55104/0 (socket says 172.21.15.16:55104) 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1baa11640 1 -- 172.21.15.16:0/170165431 learned_addr learned my addr 172.21.15.16:0/170165431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55828e8a2470 con 0x55828e891b40 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55828e7b1060 con 0x55828e8905d0 2024-09-12T02:48:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55828e8a25f0 con 0x55828e89f780 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1be84d740 10 client.0 _ll_register_callbacks cb 0x55828e5048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.734+0000 7fc1baa11640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e89ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 -- 172.21.15.16:0/170165431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 msgr2=0x55828e89ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e89ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 -- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 msgr2=0x55828e89c420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e89c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e89c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55828e5673c0 con 0x55828e8905d0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1ac036040 con 0x55828e8905d0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e89c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc1baa11640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac006010 tx=0x7fc1ac004210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee0b5e9aa0c1770 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.735+0000 7fc19dffb640 10 client.25262 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.736+0000 7fc19dffb640 1 -- 172.21.15.16:0/170165431 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac051070 con 0x55828e8905d0 2024-09-12T02:48:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.736+0000 7fc19dffb640 1 -- 172.21.15.16:0/170165431 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc1ac003970 con 0x55828e8905d0 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.736+0000 7fc19dffb640 1 -- 172.21.15.16:0/170165431 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1ac003bb0 con 0x55828e8905d0 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.736+0000 7fc1be84d740 10 client.25262 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.736+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55828e8addf0 con 0x55828e8905d0 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc19dffb640 1 -- 172.21.15.16:0/170165431 <== mon.2 v2:172.21.15.33:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc1ac030070 con 0x55828e8905d0 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 10 client.25262 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 10 client.25262 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 20 client.25262 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e8af140 0x55828e8b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 4 client.25262 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55828e803860 con 0x55828e8af140 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.737+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e8af140 0x55828e8b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.738+0000 7fc19dffb640 10 client.25262 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.738+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e8af140 0x55828e8b1530 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85e940d09199c2ac server_cookie=2fe65d623a60d8c2 in_seq=0 out_seq=0 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc19dffb640 1 -- 172.21.15.16:0/170165431 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55828e803860 con 0x55828e8af140 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc19dffb640 10 client.25262 handle_command_reply: tid=0 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 client.25262 shutdown 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 msgr2=0x55828e89f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac006010 tx=0x7fc1ac004210 comp rx=0 tx=0).stop 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 shutdown_connections 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e89ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e891b40 0x55828e89c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e8af140 0x55828e8b1530 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 --2- 172.21.15.16:0/170165431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e8905d0 0x55828e89f170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 >> 172.21.15.16:0/170165431 conn(0x55828e859310 msgr2=0x55828e892960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.739+0000 7fc19dffb640 0 client.25262 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.740+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 shutdown_connections 2024-09-12T02:48:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.740+0000 7fc1be84d740 1 -- 172.21.15.16:0/170165431 wait complete. 2024-09-12T02:48:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.740+0000 7fc1be84d740 20 client.25262 trim_cache size 0 max 16384 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 Processor -- start 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 -- start start 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 0x55828e8b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e899680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41ab4d0 con 0x55828e8905d0 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41ad010 con 0x55828e891b40 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.765+0000 7fc1be84d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41abba0 con 0x55828e89f780 2024-09-12T02:48:41.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1b9a0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 0x55828e8b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1b9a0f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 0x55828e8b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55116/0 (socket says 172.21.15.16:55116) 2024-09-12T02:48:41.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1b9a0f640 1 -- 172.21.15.16:0/1586426026 learned_addr learned my addr 172.21.15.16:0/1586426026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 -- 172.21.15.16:0/1586426026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 msgr2=0x55828e899680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e899680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 -- 172.21.15.16:0/1586426026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 msgr2=0x55828e8b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 0x55828e8b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.766+0000 7fc1baa11640 1 -- 172.21.15.16:0/1586426026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55828e88ce30 con 0x55828e8905d0 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.767+0000 7fc1baa11640 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4550 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009cd0 tx=0x7fc1ac009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbec35d0b2129ff server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.767+0000 7fc19ffff640 1 -- 172.21.15.16:0/1586426026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac051040 con 0x55828e8905d0 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.767+0000 7fc19ffff640 1 -- 172.21.15.16:0/1586426026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1ac036040 con 0x55828e8905d0 2024-09-12T02:48:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.767+0000 7fc19ffff640 1 -- 172.21.15.16:0/1586426026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac030a80 con 0x55828e8905d0 2024-09-12T02:48:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 -- 172.21.15.16:0/1586426026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 msgr2=0x55828e8b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4550 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009cd0 tx=0x7fc1ac009a20 comp rx=0 tx=0).stop 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 -- 172.21.15.16:0/1586426026 shutdown_connections 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e89f780 0x55828e899680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e891b40 0x55828e8b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 --2- 172.21.15.16:0/1586426026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4550 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 -- 172.21.15.16:0/1586426026 >> 172.21.15.16:0/1586426026 conn(0x55828e859310 msgr2=0x55828e897720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 -- 172.21.15.16:0/1586426026 shutdown_connections 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.768+0000 7fc1be84d740 1 -- 172.21.15.16:0/1586426026 wait complete. 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.769+0000 7fc1be84d740 1 -- start start 2024-09-12T02:48:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.769+0000 7fc1be84d740 1 Processor -- start 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 0x55828e8b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41ac940 con 0x55828e8905d0 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41ac270 con 0x55828e89f780 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41a94d0 con 0x55828e891b40 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2024-09-12T02:48:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 -- 172.21.15.16:0/2678986267 learned_addr learned my addr 172.21.15.16:0/2678986267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 0x55828e8b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1be84d740 10 client.0 _ll_register_callbacks cb 0x7fc1ac03f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 -- 172.21.15.16:0/2678986267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 msgr2=0x55828e8b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 0x55828e8b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 -- 172.21.15.16:0/2678986267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 msgr2=0x55828e8b29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8b29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1baa11640 1 -- 172.21.15.16:0/2678986267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55828e918a80 con 0x55828e8905d0 2024-09-12T02:48:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.770+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 0x55828e8b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc1baa11640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac004a70 tx=0x7fc1ac004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5121c109ba3485 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc19dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac051040 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc1ac003f50 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1ac036040 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.771+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1ac0305a0 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 10 client.15624 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55828e919c00 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1ac005040 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc1ac030050 con 0x55828e8905d0 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 10 client.15624 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 10 client.15624 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 20 client.15624 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e91a160 0x55828e91c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 4 client.15624 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc1ac036040 con 0x55828e91a160 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.772+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e91a160 0x55828e91c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc19dffb640 10 client.15624 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1b9a0f640 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e91a160 0x55828e91c580 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fcfc9f8ee6b021f9 server_cookie=66fa2efaca5d75c8 in_seq=0 out_seq=0 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc19dffb640 1 -- 172.21.15.16:0/2678986267 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc1ac036040 con 0x55828e91a160 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc19dffb640 10 client.15624 handle_command_reply: tid=0 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 client.15624 shutdown 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 msgr2=0x55828e8b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac004a70 tx=0x7fc1ac004aa0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 shutdown_connections 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55828e91a160 0x55828e91c580 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55828e89f780 0x55828e8b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55828e891b40 0x55828e8b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 --2- 172.21.15.16:0/2678986267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55828e8905d0 0x55828e8b4e70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.773+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 >> 172.21.15.16:0/2678986267 conn(0x55828e859310 msgr2=0x55828e88b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.778 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.774+0000 7fc19dffb640 0 client.15624 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.774+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 shutdown_connections 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.774+0000 7fc1be84d740 1 -- 172.21.15.16:0/2678986267 wait complete. 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.774+0000 7fc1be84d740 20 client.15624 trim_cache size 0 max 16384 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.774+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 msgr2=0x7fc1a00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 0x7fc1a00a5150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55828e88dac0 tx=0x7fc1b0066510 comp rx=0 tx=0).stop 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 msgr2=0x7fc1b4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d950 tx=0x7fc1b000de10 comp rx=0 tx=0).stop 2024-09-12T02:48:41.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 shutdown_connections 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b4106a70 0x7fc1b419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1b4101410 0x7fc1b4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc1a00a2c90 0x7fc1a00a5150 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 --2- 172.21.15.16:0/546280754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1b4100a70 0x7fc1b4198450 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 >> 172.21.15.16:0/546280754 conn(0x7fc1b40fc7d0 msgr2=0x7fc1b40fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 shutdown_connections 2024-09-12T02:48:41.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:41.775+0000 7fc1bc474640 1 -- 172.21.15.16:0/546280754 wait complete. 2024-09-12T02:48:42.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:48:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:43.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:43 smithi016 ceph-mon[30925]: pgmap v361: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:48:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:43 smithi033 ceph-mon[35092]: pgmap v361: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.510+0000 7f74174f5640 1 -- 172.21.15.16:0/2735020364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102ef0 msgr2=0x7f741010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.510+0000 7f74174f5640 1 --2- 172.21.15.16:0/2735020364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102ef0 0x7f741010da40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7408009ca0 tx=0x7f740802a950 comp rx=0 tx=0).stop 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 -- 172.21.15.16:0/2735020364 shutdown_connections 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 --2- 172.21.15.16:0/2735020364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102ef0 0x7f741010da40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 --2- 172.21.15.16:0/2735020364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102570 0x7f74101029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 --2- 172.21.15.16:0/2735020364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 -- 172.21.15.16:0/2735020364 >> 172.21.15.16:0/2735020364 conn(0x7f74100fc7d0 msgr2=0x7f74100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 -- 172.21.15.16:0/2735020364 shutdown_connections 2024-09-12T02:48:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.511+0000 7f74174f5640 1 -- 172.21.15.16:0/2735020364 wait complete. 2024-09-12T02:48:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.512+0000 7f74174f5640 1 Processor -- start 2024-09-12T02:48:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.512+0000 7f74174f5640 1 -- start start 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102570 0x7f7410197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102ef0 0x7f7410197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7410110fd0 con 0x7f7410102570 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7410110e50 con 0x7f7410102ef0 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f74174f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7410111150 con 0x7f7410100e20 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102570 0x7f7410197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32994/0 (socket says 172.21.15.16:32994) 2024-09-12T02:48:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 -- 172.21.15.16:0/1657261553 learned_addr learned my addr 172.21.15.16:0/1657261553 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415a92640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102ef0 0x7f7410197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 -- 172.21.15.16:0/1657261553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102ef0 msgr2=0x7f7410197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102ef0 0x7f7410197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 -- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102570 msgr2=0x7f7410197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102570 0x7f7410197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.513+0000 7f7415291640 1 -- 172.21.15.16:0/1657261553 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74101a69a0 con 0x7f7410100e20 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f7415291640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d950 tx=0x7f740c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f74067fc640 1 -- 172.21.15.16:0/1657261553 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c017030 con 0x7f7410100e20 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74101993a0 con 0x7f7410100e20 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f74067fc640 1 -- 172.21.15.16:0/1657261553 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c004020 con 0x7f7410100e20 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f74067fc640 1 -- 172.21.15.16:0/1657261553 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c0047a0 con 0x7f7410100e20 2024-09-12T02:48:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.514+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7410199930 con 0x7f7410100e20 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.517+0000 7f74067fc640 1 -- 172.21.15.16:0/1657261553 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f740c004940 con 0x7f7410100e20 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.517+0000 7f74198ce740 1 Processor -- start 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 -- start start 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac9686c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9686ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac967d22f0 con 0x55ac96877370 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ac96779cf0 con 0x55ac9686db40 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f74198ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ac967ce2b0 con 0x55ac9686c5d0 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42836/0 (socket says 172.21.15.16:42836) 2024-09-12T02:48:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.518+0000 7f7414a90640 1 -- 172.21.15.16:0/3468755781 learned_addr learned my addr 172.21.15.16:0/3468755781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:43.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7415a92640 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac9686c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7415291640 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9686ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 -- 172.21.15.16:0/3468755781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 msgr2=0x55ac9686ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9686ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 -- 172.21.15.16:0/3468755781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 msgr2=0x55ac9686c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac9686c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 -- 172.21.15.16:0/3468755781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac9675e3a0 con 0x55ac96877370 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.519+0000 7f7414a90640 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ba0 tx=0x7f74000117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64111d242edd874b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.520+0000 7f73fb7fe640 1 -- 172.21.15.16:0/3468755781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7400004020 con 0x55ac96877370 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.520+0000 7f73fb7fe640 1 -- 172.21.15.16:0/3468755781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74000041c0 con 0x55ac96877370 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.520+0000 7f73fb7fe640 1 -- 172.21.15.16:0/3468755781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7400012780 con 0x55ac96877370 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.520+0000 7f74067fc640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 0x7f73fc0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.520+0000 7f74067fc640 1 -- 172.21.15.16:0/1657261553 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f740c0cea60 con 0x7f7410100e20 2024-09-12T02:48:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f7415a92640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 0x7f73fc0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 -- 172.21.15.16:0/3468755781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 msgr2=0x55ac96879760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ba0 tx=0x7f74000117b0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 -- 172.21.15.16:0/3468755781 shutdown_connections 2024-09-12T02:48:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac96877370 0x55ac96879760 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9686ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 --2- 172.21.15.16:0/3468755781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac9686c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.521+0000 7f74198ce740 1 -- 172.21.15.16:0/3468755781 >> 172.21.15.16:0/3468755781 conn(0x55ac96835310 msgr2=0x55ac96868c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.522+0000 7f74198ce740 1 -- 172.21.15.16:0/3468755781 shutdown_connections 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.522+0000 7f74198ce740 1 -- 172.21.15.16:0/3468755781 wait complete. 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.522+0000 7f7415a92640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 0x7f73fc0a51d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7408006010 tx=0x7f7408042040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.522+0000 7f74198ce740 1 -- start start 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.522+0000 7f74198ce740 1 Processor -- start 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac96873fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac96874570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac9687a070 con 0x55ac9686db40 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ac9678d060 con 0x55ac9686c5d0 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ac9687a1f0 con 0x55ac96877370 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f74198ce740 10 client.0 _ll_register_callbacks cb 0x55ac964e08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac96873fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415291640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415291640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55154/0 (socket says 172.21.15.16:55154) 2024-09-12T02:48:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415291640 1 -- 172.21.15.16:0/443950533 learned_addr learned my addr 172.21.15.16:0/443950533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415a92640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac96874570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415291640 1 -- 172.21.15.16:0/443950533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 msgr2=0x55ac96874570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.523+0000 7f7415291640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac96874570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7415291640 1 -- 172.21.15.16:0/443950533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 msgr2=0x55ac96873fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7415291640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac96873fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7415291640 1 -- 172.21.15.16:0/443950533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac965433c0 con 0x55ac9686c5d0 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7415a92640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac96874570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7414a90640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac96873fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f7415291640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f740c097570 tx=0x7f740c09ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4295c38ae25bb06e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f73f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c09b110 con 0x55ac9686c5d0 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f740c0bb0c0 con 0x55ac9686c5d0 2024-09-12T02:48:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74000041c0 con 0x55ac9686c5d0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f740c09f4c0 con 0x55ac9686c5d0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f74198ce740 10 client.25277 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ac96885b50 con 0x55ac9686c5d0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.524+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74000041c0 con 0x55ac9686c5d0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f740c09b370 con 0x55ac9686c5d0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 10 client.25277 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 10 client.25277 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 20 client.25277 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac96886fb0 0x55ac96889430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 4 client.25277 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac967df860 con 0x55ac96886fb0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f7414a90640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac96886fb0 0x55ac96889430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.525+0000 7f73f8ff9640 10 client.25277 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.526+0000 7f7414a90640 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac96886fb0 0x55ac96889430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e439811165aebe26 server_cookie=2b2a101d00682ef5 in_seq=0 out_seq=0 2024-09-12T02:48:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f73f8ff9640 1 -- 172.21.15.16:0/443950533 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac967df860 con 0x55ac96886fb0 2024-09-12T02:48:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f73f8ff9640 10 client.25277 handle_command_reply: tid=0 2024-09-12T02:48:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 client.25277 shutdown 2024-09-12T02:48:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 msgr2=0x55ac96876d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f740c097570 tx=0x7f740c09ad10 comp rx=0 tx=0).stop 2024-09-12T02:48:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 shutdown_connections 2024-09-12T02:48:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac96886fb0 0x55ac96889430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac96874570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac96873fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f73f8ff9640 0 client.25277 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 --2- 172.21.15.16:0/443950533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686c5d0 0x55ac96876d20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 >> 172.21.15.16:0/443950533 conn(0x55ac96835310 msgr2=0x55ac9686e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 shutdown_connections 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 1 -- 172.21.15.16:0/443950533 wait complete. 2024-09-12T02:48:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.527+0000 7f74198ce740 20 client.25277 trim_cache size 0 max 16384 2024-09-12T02:48:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:43 smithi184 ceph-mon[35321]: pgmap v361: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-12T02:48:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.553+0000 7f74198ce740 1 Processor -- start 2024-09-12T02:48:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.553+0000 7f74198ce740 1 -- start start 2024-09-12T02:48:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 0x55ac96868e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac968833a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400e360 con 0x55ac9686c5d0 2024-09-12T02:48:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400fea0 con 0x55ac9686db40 2024-09-12T02:48:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f74198ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400ea30 con 0x55ac96877370 2024-09-12T02:48:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33024/0 (socket says 172.21.15.16:33024) 2024-09-12T02:48:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 -- 172.21.15.16:0/2083206271 learned_addr learned my addr 172.21.15.16:0/2083206271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7414a90640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac968833a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415291640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 0x55ac96868e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 -- 172.21.15.16:0/2083206271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 msgr2=0x55ac968833a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac968833a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 -- 172.21.15.16:0/2083206271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 msgr2=0x55ac96868e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 0x55ac96868e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415a92640 1 -- 172.21.15.16:0/2083206271 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac9686c3d0 con 0x55ac96877370 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.554+0000 7f7415291640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 0x55ac96868e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.555+0000 7f7415a92640 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7408009ca0 tx=0x7f74080095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38291bcd451ceecf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.555+0000 7f73faffd640 1 -- 172.21.15.16:0/2083206271 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408034040 con 0x55ac96877370 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.555+0000 7f73faffd640 1 -- 172.21.15.16:0/2083206271 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740802fe10 con 0x55ac96877370 2024-09-12T02:48:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.555+0000 7f73faffd640 1 -- 172.21.15.16:0/2083206271 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080306a0 con 0x55ac96877370 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.556+0000 7f74198ce740 1 -- 172.21.15.16:0/2083206271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 msgr2=0x55ac9688ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.556+0000 7f74198ce740 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7408009ca0 tx=0x7f74080095c0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.556+0000 7f74198ce740 1 -- 172.21.15.16:0/2083206271 shutdown_connections 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.556+0000 7f74198ce740 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac96877370 0x55ac9688ed00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac9686db40 0x55ac968833a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 --2- 172.21.15.16:0/2083206271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686c5d0 0x55ac96868e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 -- 172.21.15.16:0/2083206271 >> 172.21.15.16:0/2083206271 conn(0x55ac96835310 msgr2=0x55ac96868700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 -- 172.21.15.16:0/2083206271 shutdown_connections 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 -- 172.21.15.16:0/2083206271 wait complete. 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 -- start start 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.557+0000 7f74198ce740 1 Processor -- start 2024-09-12T02:48:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9687fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 0x55ac9687b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400f7d0 con 0x55ac9686db40 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400f100 con 0x55ac96877370 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400c340 con 0x55ac9686c5d0 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42864/0 (socket says 172.21.15.16:42864) 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 -- 172.21.15.16:0/614202154 learned_addr learned my addr 172.21.15.16:0/614202154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f74198ce740 10 client.0 _ll_register_callbacks cb 0x7f740c097480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7415a92640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 0x55ac9687b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 -- 172.21.15.16:0/614202154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 msgr2=0x55ac9687fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7415291640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9687fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9687fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 -- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 msgr2=0x55ac9687b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 0x55ac9687b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7414a90640 1 -- 172.21.15.16:0/614202154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac9687be90 con 0x55ac9686db40 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7415291640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9687fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.558+0000 7f7415a92640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 0x55ac9687b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f7414a90640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ba0 tx=0x7f7400003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f359cf924d31c39a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f73f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740001f040 con 0x55ac9686db40 2024-09-12T02:48:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7400003e70 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f740802fe10 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7400009e90 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f74198ce740 10 client.15642 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.559+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ac968f1da0 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7400003ad0 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7400004470 con 0x55ac9686db40 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 10 client.15642 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 10 client.15642 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 20 client.15642 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac968f2240 0x55ac968f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 4 client.15642 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f740802fe10 con 0x55ac968f2240 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.560+0000 7f7415291640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac968f2240 0x55ac968f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f73f8ff9640 10 client.15642 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f7415291640 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac968f2240 0x55ac968f4630 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc1486c45ee8506a server_cookie=3f8b4defa7271bdb in_seq=0 out_seq=0 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f73f8ff9640 1 -- 172.21.15.16:0/614202154 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f740802fe10 con 0x55ac968f2240 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f73f8ff9640 10 client.15642 handle_command_reply: tid=0 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 client.15642 shutdown 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 msgr2=0x55ac9687b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.571 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ba0 tx=0x7f7400003d80 comp rx=0 tx=0).stop 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 shutdown_connections 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ac968f2240 0x55ac968f4630 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ac96877370 0x55ac9687b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac9686db40 0x55ac9687b070 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f73f8ff9640 0 client.15642 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 --2- 172.21.15.16:0/614202154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac9686c5d0 0x55ac9687fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.561+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 >> 172.21.15.16:0/614202154 conn(0x55ac96835310 msgr2=0x55ac96872c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 shutdown_connections 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74198ce740 1 -- 172.21.15.16:0/614202154 wait complete. 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74198ce740 20 client.15642 trim_cache size 0 max 16384 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 msgr2=0x7f73fc0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 0x7f73fc0a51d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7408006010 tx=0x7f7408042040 comp rx=0 tx=0).stop 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 msgr2=0x7f7410197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.562+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d950 tx=0x7f740c00de10 comp rx=0 tx=0).stop 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 shutdown_connections 2024-09-12T02:48:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7410102ef0 0x7f7410197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f73fc0a2d10 0x7f73fc0a51d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7410102570 0x7f7410197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 --2- 172.21.15.16:0/1657261553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7410100e20 0x7f7410197130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.563+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 >> 172.21.15.16:0/1657261553 conn(0x7f74100fc7d0 msgr2=0x7f7410106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.564+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 shutdown_connections 2024-09-12T02:48:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:43.564+0000 7f74174f5640 1 -- 172.21.15.16:0/1657261553 wait complete. 2024-09-12T02:48:44.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:48:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:44.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 -- 172.21.15.16:0/2920557696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 msgr2=0x7f316010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 --2- 172.21.15.16:0/2920557696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316010ec40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3158009cd0 tx=0x7f315802a910 comp rx=0 tx=0).stop 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 -- 172.21.15.16:0/2920557696 shutdown_connections 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 --2- 172.21.15.16:0/2920557696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316010ec40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 --2- 172.21.15.16:0/2920557696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31601036e0 0x7f3160107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 --2- 172.21.15.16:0/2920557696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3160102d30 0x7f3160103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 -- 172.21.15.16:0/2920557696 >> 172.21.15.16:0/2920557696 conn(0x7f31600fc7d0 msgr2=0x7f31600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 -- 172.21.15.16:0/2920557696 shutdown_connections 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.073+0000 7f316784c640 1 -- 172.21.15.16:0/2920557696 wait complete. 2024-09-12T02:48:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.074+0000 7f316784c640 1 Processor -- start 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.074+0000 7f316784c640 1 -- start start 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.074+0000 7f316784c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3160102d30 0x7f3160198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f316784c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31601036e0 0x7f3160198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f316784c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f316784c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31601120a0 con 0x7f31601082a0 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f316784c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3160111f20 con 0x7f3160102d30 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f316784c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3160112220 con 0x7f31601036e0 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42890/0 (socket says 172.21.15.16:42890) 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 -- 172.21.15.16:0/3834501181 learned_addr learned my addr 172.21.15.16:0/3834501181 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3164de7640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31601036e0 0x7f3160198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f31655e8640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3160102d30 0x7f3160198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 -- 172.21.15.16:0/3834501181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31601036e0 msgr2=0x7f3160198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31601036e0 0x7f3160198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 -- 172.21.15.16:0/3834501181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3160102d30 msgr2=0x7f3160198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3160102d30 0x7f3160198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.075+0000 7f3165de9640 1 -- 172.21.15.16:0/3834501181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316019d4c0 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f3165de9640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3158005cc0 tx=0x7f3158031830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f31567fc640 1 -- 172.21.15.16:0/3834501181 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158003b70 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f31567fc640 1 -- 172.21.15.16:0/3834501181 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3158003d10 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316019d750 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f31567fc640 1 -- 172.21.15.16:0/3834501181 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315802fbb0 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.076+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31601a5090 con 0x7f31601082a0 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.080+0000 7f3169c25740 1 Processor -- start 2024-09-12T02:48:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.080+0000 7f31567fc640 1 -- 172.21.15.16:0/3834501181 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f315802fd50 con 0x7f31601082a0 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 -- start start 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7b5d0 0x563609e7ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e88760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563609de12f0 con 0x563609e7cb40 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563609d88cf0 con 0x563609e7b5d0 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3169c25740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563609ddd2b0 con 0x563609e86370 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f31567fc640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 0x7f314c0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3164de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3164de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42902/0 (socket says 172.21.15.16:42902) 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.081+0000 7f3164de7640 1 -- 172.21.15.16:0/600100393 learned_addr learned my addr 172.21.15.16:0/600100393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3165de9640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 0x7f314c0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f31567fc640 1 -- 172.21.15.16:0/3834501181 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31580f0320 con 0x7f31601082a0 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 -- 172.21.15.16:0/600100393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 msgr2=0x563609e88760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e88760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 -- 172.21.15.16:0/600100393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7b5d0 msgr2=0x563609e7ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7b5d0 0x563609e7ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 -- 172.21.15.16:0/600100393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563609d6d3a0 con 0x563609e7cb40 2024-09-12T02:48:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.082+0000 7f3164de7640 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f315000b6c0 tx=0x7f31500127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6324c282f82af0cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.083+0000 7f314b7fe640 1 -- 172.21.15.16:0/600100393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500120d0 con 0x563609e7cb40 2024-09-12T02:48:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.083+0000 7f3165de9640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 0x7f314c0a51d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f31580b94b0 tx=0x7f3158009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.083+0000 7f314b7fe640 1 -- 172.21.15.16:0/600100393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31500047a0 con 0x563609e7cb40 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.083+0000 7f314b7fe640 1 -- 172.21.15.16:0/600100393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150005280 con 0x563609e7cb40 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 -- 172.21.15.16:0/600100393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 msgr2=0x563609e7b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f315000b6c0 tx=0x7f31500127c0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 -- 172.21.15.16:0/600100393 shutdown_connections 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e88760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7b5d0 0x563609e7ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 --2- 172.21.15.16:0/600100393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7cb40 0x563609e7b090 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 -- 172.21.15.16:0/600100393 >> 172.21.15.16:0/600100393 conn(0x563609e44310 msgr2=0x563609e77c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 -- 172.21.15.16:0/600100393 shutdown_connections 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.084+0000 7f3169c25740 1 -- 172.21.15.16:0/600100393 wait complete. 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.085+0000 7f3169c25740 1 -- start start 2024-09-12T02:48:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.085+0000 7f3169c25740 1 Processor -- start 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.086+0000 7f3169c25740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.086+0000 7f3164de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.086+0000 7f3164de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42918/0 (socket says 172.21.15.16:42918) 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.086+0000 7f3169c25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.086+0000 7f3164de7640 1 -- 172.21.15.16:0/576597494 learned_addr learned my addr 172.21.15.16:0/576597494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e83850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563609e89070 con 0x563609e7b5d0 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f31655e8640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563609d9c060 con 0x563609e7cb40 2024-09-12T02:48:45.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563609e891f0 con 0x563609e86370 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3169c25740 10 client.0 _ll_register_callbacks cb 0x563609aef8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.087+0000 7f3164de7640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e83850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 -- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 msgr2=0x563609e83850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e83850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 -- 172.21.15.16:0/576597494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 msgr2=0x563609e85e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563609b523c0 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f31655e8640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e830c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f315c0116c0 tx=0x7f315c011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf61640e285ffb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f3148ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f3164de7640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c00fd90 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.088+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31500047a0 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.089+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f315c004020 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.089+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f315c004220 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.089+0000 7f3169c25740 10 client.25289 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.089+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563609e94c30 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.089+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c0043c0 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f315c0246f0 con 0x563609e7cb40 2024-09-12T02:48:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 10 client.25289 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 10 client.25289 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 20 client.25289 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609e98450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 4 client.25289 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f31500047a0 con 0x563609e96040 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.090+0000 7f3164de7640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609e98450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.091+0000 7f3148ff9640 10 client.25289 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.091+0000 7f3164de7640 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609e98450 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1340107f58f658d server_cookie=59875fdda731b29b in_seq=0 out_seq=0 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3148ff9640 1 -- 172.21.15.16:0/576597494 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f31500047a0 con 0x563609e96040 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3148ff9640 10 client.25289 handle_command_reply: tid=0 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 client.25289 shutdown 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 msgr2=0x563609e830c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e830c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f315c0116c0 tx=0x7f315c011b80 comp rx=0 tx=0).stop 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 shutdown_connections 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609e98450 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e83850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.092+0000 7f3148ff9640 0 client.25289 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e830c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 1 --2- 172.21.15.16:0/576597494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e85e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 >> 172.21.15.16:0/576597494 conn(0x563609e44310 msgr2=0x563609e7d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 shutdown_connections 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 1 -- 172.21.15.16:0/576597494 wait complete. 2024-09-12T02:48:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.093+0000 7f3169c25740 20 client.25289 trim_cache size 0 max 16384 2024-09-12T02:48:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.119+0000 7f3169c25740 1 Processor -- start 2024-09-12T02:48:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.119+0000 7f3169c25740 1 -- start start 2024-09-12T02:48:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e77d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e86370 0x563609e9dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31601ab350 con 0x563609e7b5d0 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31601ace90 con 0x563609e86370 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f3169c25740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31601aba20 con 0x563609e7cb40 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33086/0 (socket says 172.21.15.16:33086) 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 -- 172.21.15.16:0/3119632603 learned_addr learned my addr 172.21.15.16:0/3119632603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 -- 172.21.15.16:0/3119632603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e86370 msgr2=0x563609e9dd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e86370 0x563609e9dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 -- 172.21.15.16:0/3119632603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 msgr2=0x563609e77d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e77d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.120+0000 7f31655e8640 1 -- 172.21.15.16:0/3119632603 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563609e7e120 con 0x563609e7cb40 2024-09-12T02:48:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.121+0000 7f31655e8640 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f315c0041b0 tx=0x7f315c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76a8a0a096f251da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.121+0000 7f314affd640 1 -- 172.21.15.16:0/3119632603 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c00fd90 con 0x563609e7cb40 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.121+0000 7f314affd640 1 -- 172.21.15.16:0/3119632603 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c004480 con 0x563609e7cb40 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.121+0000 7f314affd640 1 -- 172.21.15.16:0/3119632603 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c0056e0 con 0x563609e7cb40 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 -- 172.21.15.16:0/3119632603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 msgr2=0x563609e820a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f315c0041b0 tx=0x7f315c00f790 comp rx=0 tx=0).stop 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 -- 172.21.15.16:0/3119632603 shutdown_connections 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e86370 0x563609e9dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e7cb40 0x563609e820a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 --2- 172.21.15.16:0/3119632603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e77d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 -- 172.21.15.16:0/3119632603 >> 172.21.15.16:0/3119632603 conn(0x563609e44310 msgr2=0x563609e81990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 -- 172.21.15.16:0/3119632603 shutdown_connections 2024-09-12T02:48:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.122+0000 7f3169c25740 1 -- 172.21.15.16:0/3119632603 wait complete. 2024-09-12T02:48:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.123+0000 7f3169c25740 1 -- start start 2024-09-12T02:48:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.123+0000 7f3169c25740 1 Processor -- start 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e8ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e89d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e8a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3164de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e8ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31601ac7c0 con 0x563609e7b5d0 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f31655e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e89d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f31655e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e89d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55230/0 (socket says 172.21.15.16:55230) 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f31655e8640 1 -- 172.21.15.16:0/64982919 learned_addr learned my addr 172.21.15.16:0/64982919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31601ac0f0 con 0x563609e7cb40 2024-09-12T02:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.124+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31601a9330 con 0x563609e86370 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e8a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 -- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 msgr2=0x563609e8a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e8a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 -- 172.21.15.16:0/64982919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 msgr2=0x563609e89d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e89d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3164de7640 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563609e8a8d0 con 0x563609e7b5d0 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3169c25740 10 client.0 _ll_register_callbacks cb 0x7f315c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.125+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563609dee860 con 0x563609e7b5d0 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.126+0000 7f3164de7640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e8ea20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x563609e98c90 tx=0x7f3150004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8835f6f07cf8ca48 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.126+0000 7f3148ff9640 10 client.15657 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.126+0000 7f3148ff9640 1 -- 172.21.15.16:0/64982919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150004c30 con 0x563609e7b5d0 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.126+0000 7f3148ff9640 1 -- 172.21.15.16:0/64982919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3150004f50 con 0x563609e7b5d0 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.126+0000 7f3148ff9640 1 -- 172.21.15.16:0/64982919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3150005150 con 0x563609e7b5d0 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x563609f009d0 con 0x563609e7b5d0 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3148ff9640 1 -- 172.21.15.16:0/64982919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f315003a070 con 0x563609e7b5d0 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 10 client.15657 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 20 client.15657 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609f02d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 4 client.15657 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.127+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563609de8d70 con 0x563609e96040 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.128+0000 7f31655e8640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609f02d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.128+0000 7f31655e8640 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609f02d90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e558b304af5d75a server_cookie=a4f0c53462855368 in_seq=0 out_seq=0 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.128+0000 7f3148ff9640 10 client.15657 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.128+0000 7f3148ff9640 1 -- 172.21.15.16:0/64982919 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563609de8d70 con 0x563609e96040 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.128+0000 7f3148ff9640 10 client.15657 handle_command_reply: tid=0 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 client.15657 shutdown 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 msgr2=0x563609e8ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e8ea20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x563609e98c90 tx=0x7f3150004530 comp rx=0 tx=0).stop 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 shutdown_connections 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563609e96040 0x563609f02d90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563609e86370 0x563609e8a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563609e7cb40 0x563609e89d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 --2- 172.21.15.16:0/64982919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563609e7b5d0 0x563609e8ea20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 >> 172.21.15.16:0/64982919 conn(0x563609e44310 msgr2=0x563609e76500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.129+0000 7f3148ff9640 0 client.15657 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.130+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 shutdown_connections 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.130+0000 7f3169c25740 1 -- 172.21.15.16:0/64982919 wait complete. 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.130+0000 7f3169c25740 20 client.15657 trim_cache size 0 max 16384 2024-09-12T02:48:45.133 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:45.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 msgr2=0x7f314c0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 0x7f314c0a51d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f31580b94b0 tx=0x7f3158009360 comp rx=0 tx=0).stop 2024-09-12T02:48:45.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 msgr2=0x7f316019cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:45.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3158005cc0 tx=0x7f3158031830 comp rx=0 tx=0).stop 2024-09-12T02:48:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 shutdown_connections 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f314c0a2d10 0x7f314c0a51d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31601082a0 0x7f316019cde0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31601036e0 0x7f3160198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 --2- 172.21.15.16:0/3834501181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3160102d30 0x7f3160198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.132+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 >> 172.21.15.16:0/3834501181 conn(0x7f31600fc7d0 msgr2=0x7f3160105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.133+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 shutdown_connections 2024-09-12T02:48:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:45.133+0000 7f316784c640 1 -- 172.21.15.16:0/3834501181 wait complete. 2024-09-12T02:48:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:45 smithi016 ceph-mon[30925]: pgmap v362: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-12T02:48:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:45 smithi033 ceph-mon[35092]: pgmap v362: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-12T02:48:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:45 smithi184 ceph-mon[35321]: pgmap v362: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-12T02:48:45.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:48:45.890 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.436+0000 7fb243cec640 1 -- 172.21.15.16:0/2852367170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 msgr2=0x7fb23c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.436+0000 7fb243cec640 1 --2- 172.21.15.16:0/2852367170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c10da40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb234009c10 tx=0x7fb23402a770 comp rx=0 tx=0).stop 2024-09-12T02:48:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.437+0000 7fb243cec640 1 -- 172.21.15.16:0/2852367170 shutdown_connections 2024-09-12T02:48:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.437+0000 7fb243cec640 1 --2- 172.21.15.16:0/2852367170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c10da40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.437+0000 7fb243cec640 1 --2- 172.21.15.16:0/2852367170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.437+0000 7fb243cec640 1 --2- 172.21.15.16:0/2852367170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.437+0000 7fb243cec640 1 -- 172.21.15.16:0/2852367170 >> 172.21.15.16:0/2852367170 conn(0x7fb23c0fc7d0 msgr2=0x7fb23c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.438+0000 7fb243cec640 1 -- 172.21.15.16:0/2852367170 shutdown_connections 2024-09-12T02:48:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.438+0000 7fb243cec640 1 -- 172.21.15.16:0/2852367170 wait complete. 2024-09-12T02:48:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.439+0000 7fb243cec640 1 Processor -- start 2024-09-12T02:48:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.439+0000 7fb243cec640 1 -- start start 2024-09-12T02:48:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.440+0000 7fb243cec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:46.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.440+0000 7fb241a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:46.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.440+0000 7fb241a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37492/0 (socket says 172.21.15.16:37492) 2024-09-12T02:48:46.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.440+0000 7fb241a88640 1 -- 172.21.15.16:0/466321491 learned_addr learned my addr 172.21.15.16:0/466321491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.440+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c110f30 con 0x7fb23c100e20 2024-09-12T02:48:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c110db0 con 0x7fb23c102570 2024-09-12T02:48:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1110b0 con 0x7fb23c102ef0 2024-09-12T02:48:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb242289640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 -- 172.21.15.16:0/466321491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 msgr2=0x7fb23c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 -- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 msgr2=0x7fb23c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241287640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c19d440 con 0x7fb23c102570 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.441+0000 7fb241a88640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.442+0000 7fb241287640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1989d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00d6d0 tx=0x7fb22c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.442+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c017070 con 0x7fb23c102570 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.442+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb22c009820 con 0x7fb23c102570 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.442+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c19d730 con 0x7fb23c102570 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.442+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c1a5010 con 0x7fb23c102570 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.443+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c004d00 con 0x7fb23c102570 2024-09-12T02:48:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.445+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23c1021b0 con 0x7fb23c102570 2024-09-12T02:48:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.446+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fb22c005220 con 0x7fb23c102570 2024-09-12T02:48:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.447+0000 7fb232ffd640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 0x7fb2280a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.447+0000 7fb241a88640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 0x7fb2280a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.447+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb22c0ced10 con 0x7fb23c102570 2024-09-12T02:48:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.447+0000 7fb241a88640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 0x7fb2280a4fb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb2380096c0 tx=0x7fb238009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.451+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb22c0d4020 con 0x7fb23c102570 2024-09-12T02:48:46.495 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:46.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.857+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb23c1997f0 con 0x7fb23c102570 2024-09-12T02:48:46.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.858+0000 7fb232ffd640 1 -- 172.21.15.16:0/466321491 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11969 (secure 0 0 0) 0x7fb22c09b470 con 0x7fb23c102570 2024-09-12T02:48:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:48:46.860 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-12T02:48:46.861 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-12T02:47:34:900440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:47:34.701840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:48:46.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 msgr2=0x7fb2280a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 0x7fb2280a4fb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb2380096c0 tx=0x7fb238009260 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 msgr2=0x7fb23c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1989d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00d6d0 tx=0x7fb22c00db90 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 shutdown_connections 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb23c102ef0 0x7fb23c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb23c102570 0x7fb23c1989d0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00d6d0 tx=0x7fb22c00db90 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb2280a2af0 0x7fb2280a4fb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 --2- 172.21.15.16:0/466321491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23c100e20 0x7fb23c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 >> 172.21.15.16:0/466321491 conn(0x7fb23c0fc7d0 msgr2=0x7fb23c06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 shutdown_connections 2024-09-12T02:48:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:46.861+0000 7fb243cec640 1 -- 172.21.15.16:0/466321491 wait complete. 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.020+0000 7f90ed3fb640 1 -- 172.21.15.16:0/2376655886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 msgr2=0x7f90e8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.020+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/2376655886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 0x7f90e8101200 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f90d8009900 tx=0x7f90d802a480 comp rx=0 tx=0).stop 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 -- 172.21.15.16:0/2376655886 shutdown_connections 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/2376655886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/2376655886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/2376655886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 0x7f90e8101200 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 -- 172.21.15.16:0/2376655886 >> 172.21.15.16:0/2376655886 conn(0x7f90e80fc7d0 msgr2=0x7f90e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 -- 172.21.15.16:0/2376655886 shutdown_connections 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.021+0000 7f90ed3fb640 1 -- 172.21.15.16:0/2376655886 wait complete. 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.022+0000 7f90ed3fb640 1 Processor -- start 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.022+0000 7f90ed3fb640 1 -- start start 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 0x7f90e81992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e81a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e8110f80 con 0x7f90e8100e20 2024-09-12T02:48:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90e8110e00 con 0x7f90e8102ef0 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90ed3fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90e8111100 con 0x7f90e8102570 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e67fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e67fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58812/0 (socket says 172.21.15.16:58812) 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e67fc640 1 -- 172.21.15.16:0/4196441245 learned_addr learned my addr 172.21.15.16:0/4196441245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e77fe640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e81a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e67fc640 1 -- 172.21.15.16:0/4196441245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 msgr2=0x7f90e81a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.023+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e81a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90e67fc640 1 -- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 msgr2=0x7f90e81992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 0x7f90e81992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90e67fc640 1 -- 172.21.15.16:0/4196441245 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e81a4890 con 0x7f90e8102570 2024-09-12T02:48:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90e77fe640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e81a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00e9f0 tx=0x7f90dc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90cffff640 1 -- 172.21.15.16:0/4196441245 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc00cd80 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90cffff640 1 -- 172.21.15.16:0/4196441245 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc005500 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e81970c0 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.024+0000 7f90cffff640 1 -- 172.21.15.16:0/4196441245 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc010690 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.025+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e81975a0 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.027+0000 7f90cffff640 1 -- 172.21.15.16:0/4196441245 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f90dc0046c0 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.027+0000 7f90cffff640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 0x7f90d00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90cffff640 1 -- 172.21.15.16:0/4196441245 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f90dc0cf570 con 0x7f90e8102570 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 0x7f90d00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 Processor -- start 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 -- start start 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef77f9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ef775f2f0 con 0x560ef77f95d0 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560ef7706cf0 con 0x560ef780cb80 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.028+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560ef775b2b0 con 0x560ef77fab40 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37520/0 (socket says 172.21.15.16:37520) 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 -- 172.21.15.16:0/4218496388 learned_addr learned my addr 172.21.15.16:0/4218496388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e77fe640 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 -- 172.21.15.16:0/4218496388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 msgr2=0x560ef77f9090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef77f9090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 -- 172.21.15.16:0/4218496388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 msgr2=0x560ef780ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 -- 172.21.15.16:0/4218496388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ef76eb3a0 con 0x560ef77f95d0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 0x7f90d00a5150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90d80096c0 tx=0x7f90d8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90e67fc640 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f90dc098650 tx=0x7f90dc09b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146ac7318acc3f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90cd7fa640 1 -- 172.21.15.16:0/4218496388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc0a0030 con 0x560ef77f95d0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90cd7fa640 1 -- 172.21.15.16:0/4218496388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc0a48f0 con 0x560ef77f95d0 2024-09-12T02:48:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.029+0000 7f90cd7fa640 1 -- 172.21.15.16:0/4218496388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc0cf0e0 con 0x560ef77f95d0 2024-09-12T02:48:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 -- 172.21.15.16:0/4218496388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 msgr2=0x560ef77f9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f90dc098650 tx=0x7f90dc09b6a0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 -- 172.21.15.16:0/4218496388 shutdown_connections 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77f95d0 0x560ef77f9a10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/4218496388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef77f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 -- 172.21.15.16:0/4218496388 >> 172.21.15.16:0/4218496388 conn(0x560ef77c2310 msgr2=0x560ef77f5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.030+0000 7f90ef7d4740 1 -- 172.21.15.16:0/4218496388 shutdown_connections 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.031+0000 7f90ef7d4740 1 -- 172.21.15.16:0/4218496388 wait complete. 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 -- start start 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 Processor -- start 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77f95d0 0x560ef780c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef78098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ef780f880 con 0x560ef77fab40 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560ef771a060 con 0x560ef780cb80 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560ef780fa00 con 0x560ef77f95d0 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90ef7d4740 10 client.0 _ll_register_callbacks cb 0x560ef746d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42778/0 (socket says 172.21.15.16:42778) 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.032+0000 7f90e77fe640 1 -- 172.21.15.16:0/882406649 learned_addr learned my addr 172.21.15.16:0/882406649 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef78098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77f95d0 0x560ef780c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 -- 172.21.15.16:0/882406649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77f95d0 msgr2=0x560ef780c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77f95d0 0x560ef780c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 -- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 msgr2=0x560ef780a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.033+0000 7f90e67fc640 1 -- 172.21.15.16:0/882406649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ef74d03c0 con 0x560ef77fab40 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90e67fc640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef78098c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f90dc098650 tx=0x7f90dc09b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b0a6856848cad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90caff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc07f060 con 0x560ef77fab40 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f90dc0982f0 con 0x560ef77fab40 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90dc0a48f0 con 0x560ef77fab40 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90dc0a0030 con 0x560ef77fab40 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.034+0000 7f90ef7d4740 10 client.15672 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ef780f880 con 0x560ef77fab40 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc0a4b40 con 0x560ef77fab40 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f90dc0b4030 con 0x560ef77fab40 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 10 client.15672 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 10 client.15672 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 20 client.15672 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef77ffb20 0x560ef7801f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 4 client.15672 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.035+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f90dc0a48f0 con 0x560ef77ffb20 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.036+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef77ffb20 0x560ef7801f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.036+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef77ffb20 0x560ef7801f10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eafde758dacb2de8 server_cookie=17818b0c9599f0e0 in_seq=0 out_seq=0 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.036+0000 7f90caff5640 10 client.15672 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90caff5640 1 -- 172.21.15.16:0/882406649 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f90dc0a48f0 con 0x560ef77ffb20 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90caff5640 10 client.15672 handle_command_reply: tid=0 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 client.15672 shutdown 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 msgr2=0x560ef78098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef78098c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f90dc098650 tx=0x7f90dc09b710 comp rx=0 tx=0).stop 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 shutdown_connections 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef77ffb20 0x560ef7801f10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef780cb80 0x560ef780a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef78098c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/882406649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77f95d0 0x560ef780c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 >> 172.21.15.16:0/882406649 conn(0x560ef77c2310 msgr2=0x560ef77fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.037+0000 7f90caff5640 0 client.15672 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.038+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 shutdown_connections 2024-09-12T02:48:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.038+0000 7f90ef7d4740 1 -- 172.21.15.16:0/882406649 wait complete. 2024-09-12T02:48:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.038+0000 7f90ef7d4740 20 client.15672 trim_cache size 0 max 16384 2024-09-12T02:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.063+0000 7f90ef7d4740 1 Processor -- start 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.063+0000 7f90ef7d4740 1 -- start start 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef7804c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef780cb80 0x560ef7805140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81ab2c0 con 0x560ef77fab40 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81ace00 con 0x560ef77f95d0 2024-09-12T02:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81ab990 con 0x560ef780cb80 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42790/0 (socket says 172.21.15.16:42790) 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1650623305 learned_addr learned my addr 172.21.15.16:0/1650623305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e67fc640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef7804c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e77fe640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef780cb80 0x560ef7805140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1650623305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef780cb80 msgr2=0x560ef7805140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef780cb80 0x560ef7805140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1650623305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 msgr2=0x560ef7804c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef7804c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.064+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1650623305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ef77fa790 con 0x560ef77f95d0 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.065+0000 7f90e67fc640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef7804c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.065+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f90d802a9a0 tx=0x7f90d803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab1d764a8a0f0dbf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.065+0000 7f90ccff9640 1 -- 172.21.15.16:0/1650623305 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8035040 con 0x560ef77f95d0 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.065+0000 7f90ccff9640 1 -- 172.21.15.16:0/1650623305 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d8055070 con 0x560ef77f95d0 2024-09-12T02:48:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.065+0000 7f90ccff9640 1 -- 172.21.15.16:0/1650623305 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d804f570 con 0x560ef77f95d0 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1650623305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 msgr2=0x560ef77fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f90d802a9a0 tx=0x7f90d803d000 comp rx=0 tx=0).stop 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1650623305 shutdown_connections 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef780cb80 0x560ef7805140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef77fab40 0x560ef7804c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1650623305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef77fc120 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1650623305 >> 172.21.15.16:0/1650623305 conn(0x560ef77c2310 msgr2=0x560ef78044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1650623305 shutdown_connections 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.066+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1650623305 wait complete. 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 -- start start 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 Processor -- start 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef7814c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef78151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81ac730 con 0x560ef780cb80 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81ac060 con 0x560ef77f95d0 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a92a0 con 0x560ef77fab40 2024-09-12T02:48:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.067+0000 7f90ef7d4740 10 client.0 _ll_register_callbacks cb 0x7f90dc09b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37538/0 (socket says 172.21.15.16:37538) 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef7814c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e77fe640 1 -- 172.21.15.16:0/1473055582 learned_addr learned my addr 172.21.15.16:0/1473055582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e67fc640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef78151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1473055582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 msgr2=0x560ef78151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef78151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1473055582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 msgr2=0x560ef7805d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 -- 172.21.15.16:0/1473055582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ef7806250 con 0x560ef77f95d0 2024-09-12T02:48:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e77fe640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90e6ffd640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef7814c90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005ec0 tx=0x7f90d8009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4711e20e31d9d18b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90caff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.068+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d805e060 con 0x560ef77f95d0 2024-09-12T02:48:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f90d8031540 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d8055070 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90d8035070 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 10 client.25316 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ef788f440 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d8040050 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f90d804fb30 con 0x560ef77f95d0 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 10 client.25316 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 10 client.25316 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:48:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 20 client.25316 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef788fab0 0x560ef7891ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 4 client.25316 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.069+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f90d8055070 con 0x560ef788fab0 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90e67fc640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef788fab0 0x560ef7891ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90caff5640 10 client.25316 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90e67fc640 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef788fab0 0x560ef7891ed0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=707853851586ef0e server_cookie=e497d413c56c818 in_seq=0 out_seq=0 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90caff5640 1 -- 172.21.15.16:0/1473055582 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f90d8055070 con 0x560ef788fab0 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90caff5640 10 client.25316 handle_command_reply: tid=0 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.070+0000 7f90ef7d4740 1 client.25316 shutdown 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 msgr2=0x560ef7814c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef7814c90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005ec0 tx=0x7f90d8009230 comp rx=0 tx=0).stop 2024-09-12T02:48:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 shutdown_connections 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x560ef788fab0 0x560ef7891ed0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ef780cb80 0x560ef7805d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560ef77fab40 0x560ef78151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 --2- 172.21.15.16:0/1473055582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560ef77f95d0 0x560ef7814c90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 >> 172.21.15.16:0/1473055582 conn(0x560ef77c2310 msgr2=0x560ef7818330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90caff5640 0 client.25316 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 shutdown_connections 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 1 -- 172.21.15.16:0/1473055582 wait complete. 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.071+0000 7f90ef7d4740 20 client.25316 trim_cache size 0 max 16384 2024-09-12T02:48:47.079 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:47.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 msgr2=0x7f90d00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 0x7f90d00a5150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90d80096c0 tx=0x7f90d8000bc0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 msgr2=0x7f90e8199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00e9f0 tx=0x7f90dc00edf0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 shutdown_connections 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e8102ef0 0x7f90e81a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90e8102570 0x7f90e8199810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f90d00a2c90 0x7f90d00a5150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 --2- 172.21.15.16:0/4196441245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e8100e20 0x7f90e81992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 >> 172.21.15.16:0/4196441245 conn(0x7f90e80fc7d0 msgr2=0x7f90e80fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 shutdown_connections 2024-09-12T02:48:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:47.072+0000 7f90ed3fb640 1 -- 172.21.15.16:0/4196441245 wait complete. 2024-09-12T02:48:47.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:47 smithi016 ceph-mon[30925]: pgmap v363: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:47.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:47 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/466321491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:47 smithi033 ceph-mon[35092]: pgmap v363: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:47 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/466321491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:47 smithi184 ceph-mon[35321]: pgmap v363: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:47 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/466321491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:48:47.966 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 5ce30e01 --release --await' 2024-09-12T02:48:47.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 5ce30e01 --release --await 2024-09-12T02:48:48.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:48:48.646 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 -- 172.21.15.16:0/1917485187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 msgr2=0x7f6538100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 --2- 172.21.15.16:0/1917485187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f6538100ed0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6524009930 tx=0x7f652402a420 comp rx=0 tx=0).stop 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 -- 172.21.15.16:0/1917485187 shutdown_connections 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 --2- 172.21.15.16:0/1917485187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538101410 0x7f653810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 --2- 172.21.15.16:0/1917485187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f6538100ed0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 --2- 172.21.15.16:0/1917485187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538106a70 0x7f6538106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.116+0000 7f653eb23640 1 -- 172.21.15.16:0/1917485187 >> 172.21.15.16:0/1917485187 conn(0x7f65380fc790 msgr2=0x7f65380febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.117+0000 7f653eb23640 1 -- 172.21.15.16:0/1917485187 shutdown_connections 2024-09-12T02:48:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.117+0000 7f653eb23640 1 -- 172.21.15.16:0/1917485187 wait complete. 2024-09-12T02:48:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.118+0000 7f653eb23640 1 Processor -- start 2024-09-12T02:48:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.118+0000 7f653eb23640 1 -- start start 2024-09-12T02:48:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 0x7f6538198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653c8bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538106a70 0x7f653819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653c8bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37550/0 (socket says 172.21.15.16:37550) 2024-09-12T02:48:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6538111f40 con 0x7f6538100a70 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6538111dc0 con 0x7f6538106a70 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653eb23640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65381120c0 con 0x7f6538101410 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653c8bf640 1 -- 172.21.15.16:0/2562503571 learned_addr learned my addr 172.21.15.16:0/2562503571 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f6537fff640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 0x7f6538198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.119+0000 7f653d0c0640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538106a70 0x7f653819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 -- 172.21.15.16:0/2562503571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538106a70 msgr2=0x7f653819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538106a70 0x7f653819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 -- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 msgr2=0x7f65381984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 -- 172.21.15.16:0/2562503571 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653819d470 con 0x7f6538101410 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f653c8bf640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.120+0000 7f6537fff640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 0x7f6538198a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6524005940 tx=0x7f6524031b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.122+0000 7f6535ffb640 1 -- 172.21.15.16:0/2562503571 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652403f070 con 0x7f6538101410 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.122+0000 7f6535ffb640 1 -- 172.21.15.16:0/2562503571 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6524003ad0 con 0x7f6538101410 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.122+0000 7f6535ffb640 1 -- 172.21.15.16:0/2562503571 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524035070 con 0x7f6538101410 2024-09-12T02:48:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.122+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653819d700 con 0x7f6538101410 2024-09-12T02:48:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.123+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653810e180 con 0x7f6538101410 2024-09-12T02:48:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.126+0000 7f6535ffb640 1 -- 172.21.15.16:0/2562503571 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f65240035f0 con 0x7f6538101410 2024-09-12T02:48:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.126+0000 7f6540efc740 1 Processor -- start 2024-09-12T02:48:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 -- start start 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6535ffb640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 0x7f65280a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20060 0x558549c20440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c2d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558549b85620 con 0x558549c20980 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558549b2ce50 con 0x558549c2b240 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6535ffb640 1 -- 172.21.15.16:0/2562503571 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f65240ef580 con 0x7f6538101410 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6540efc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558549b815e0 con 0x558549c20060 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6537fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6537fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37566/0 (socket says 172.21.15.16:37566) 2024-09-12T02:48:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.127+0000 7f6537fff640 1 -- 172.21.15.16:0/2091152268 learned_addr learned my addr 172.21.15.16:0/2091152268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f653c8bf640 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20060 0x558549c20440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 -- 172.21.15.16:0/2091152268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20060 msgr2=0x558549c20440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20060 0x558549c20440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 -- 172.21.15.16:0/2091152268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 msgr2=0x558549c2d630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c2d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 -- 172.21.15.16:0/2091152268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558549b1dd40 con 0x558549c20980 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f6537fff640 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6524032c00 tx=0x7f6524003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abff9bf384b36c81 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.128+0000 7f653d0c0640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 0x7f65280a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.129+0000 7f6522ffd640 1 -- 172.21.15.16:0/2091152268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524048040 con 0x558549c20980 2024-09-12T02:48:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.129+0000 7f6522ffd640 1 -- 172.21.15.16:0/2091152268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65240038e0 con 0x558549c20980 2024-09-12T02:48:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.129+0000 7f6522ffd640 1 -- 172.21.15.16:0/2091152268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240c4410 con 0x558549c20980 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 -- 172.21.15.16:0/2091152268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 msgr2=0x558549c2ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f653d0c0640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 0x7f65280a52a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x558549c1d970 tx=0x7f6530009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6524032c00 tx=0x7f6524003f60 comp rx=0 tx=0).stop 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 -- 172.21.15.16:0/2091152268 shutdown_connections 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c2d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c2ad00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 --2- 172.21.15.16:0/2091152268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20060 0x558549c20440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 -- 172.21.15.16:0/2091152268 >> 172.21.15.16:0/2091152268 conn(0x558549c1c5f0 msgr2=0x558549c1c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 -- 172.21.15.16:0/2091152268 shutdown_connections 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.130+0000 7f6540efc740 1 -- 172.21.15.16:0/2091152268 wait complete. 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 -- start start 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 Processor -- start 2024-09-12T02:48:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 0x558549c281f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c28730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c28c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558549c3ebe0 con 0x558549c20060 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558549b0df10 con 0x558549c2b240 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558549c3ed60 con 0x558549c20980 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.131+0000 7f6540efc740 10 client.0 _ll_register_callbacks cb 0x558549b31db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.132+0000 7f653d0c0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c28c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.132+0000 7f653d0c0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c28c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42836/0 (socket says 172.21.15.16:42836) 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.132+0000 7f653d0c0640 1 -- 172.21.15.16:0/3949315975 learned_addr learned my addr 172.21.15.16:0/3949315975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.132+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 0x558549c281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.133+0000 7f653c8bf640 1 -- 172.21.15.16:0/3949315975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 msgr2=0x558549c28730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:49.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.133+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c28730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.133+0000 7f653c8bf640 1 -- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 msgr2=0x558549c28c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.133+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c28c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.133+0000 7f653c8bf640 1 -- 172.21.15.16:0/3949315975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558549894580 con 0x558549c20060 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.134+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 0x558549c281f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f652c013ba0 tx=0x7f652c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8737b08895713733 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.134+0000 7f6517fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.134+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c004230 con 0x558549c20060 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.134+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f652c0043d0 con 0x558549c20060 2024-09-12T02:48:49.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.134+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65240038e0 con 0x558549c20060 2024-09-12T02:48:49.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.135+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f652c005020 con 0x558549c20060 2024-09-12T02:48:49.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.135+0000 7f6540efc740 10 client.15687 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:49.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.135+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55854995e960 con 0x558549c20060 2024-09-12T02:48:49.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.135+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65240038e0 con 0x558549c20060 2024-09-12T02:48:49.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.135+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f652c0035f0 con 0x558549c20060 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 10 client.15687 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 10 client.15687 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 20 client.15687 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549c4bbb0 0x558549c4dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 4 client.15687 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.136+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558549b92b90 con 0x558549c4bbb0 2024-09-12T02:48:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.137+0000 7f6537fff640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549c4bbb0 0x558549c4dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.137+0000 7f6537fff640 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549c4bbb0 0x558549c4dfe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1858837954dd883f server_cookie=71ac422d01a9e4da in_seq=0 out_seq=0 2024-09-12T02:48:49.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.137+0000 7f6517fff640 10 client.15687 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:49.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.138+0000 7f6517fff640 1 -- 172.21.15.16:0/3949315975 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558549b92b90 con 0x558549c4bbb0 2024-09-12T02:48:49.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.138+0000 7f6517fff640 10 client.15687 handle_command_reply: tid=0 2024-09-12T02:48:49.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.138+0000 7f6540efc740 1 client.15687 shutdown 2024-09-12T02:48:49.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.138+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 msgr2=0x558549c281f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.138+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 0x558549c281f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f652c013ba0 tx=0x7f652c0117b0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 shutdown_connections 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549c4bbb0 0x558549c4dfe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c2b240 0x558549c28c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c28730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 --2- 172.21.15.16:0/3949315975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20060 0x558549c281f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 >> 172.21.15.16:0/3949315975 conn(0x558549c1c5f0 msgr2=0x558549c1b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6517fff640 0 client.15687 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 shutdown_connections 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.139+0000 7f6540efc740 1 -- 172.21.15.16:0/3949315975 wait complete. 2024-09-12T02:48:49.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.140+0000 7f6540efc740 20 client.15687 trim_cache size 0 max 16384 2024-09-12T02:48:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.170+0000 7f6540efc740 1 Processor -- start 2024-09-12T02:48:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.170+0000 7f6540efc740 1 -- start start 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.170+0000 7f6540efc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c475a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.170+0000 7f6540efc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.170+0000 7f6537fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6537fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37586/0 (socket says 172.21.15.16:37586) 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6537fff640 1 -- 172.21.15.16:0/346587294 learned_addr learned my addr 172.21.15.16:0/346587294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6540efc740 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c2b240 0x558549c25990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f653c8bf640 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6538071f30 con 0x558549c20980 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6538073a70 con 0x558549c20060 2024-09-12T02:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6538072600 con 0x558549c2b240 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.171+0000 7f653d0c0640 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c2b240 0x558549c25990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 -- 172.21.15.16:0/346587294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c2b240 msgr2=0x558549c25990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c2b240 0x558549c25990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 -- 172.21.15.16:0/346587294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 msgr2=0x558549c475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 -- 172.21.15.16:0/346587294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558549c4a800 con 0x558549c20980 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f6537fff640 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f65240bbc70 tx=0x7f65240ef040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174d45d4c7128820 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f65227fc640 1 -- 172.21.15.16:0/346587294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524048040 con 0x558549c20980 2024-09-12T02:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f65227fc640 1 -- 172.21.15.16:0/346587294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65240038e0 con 0x558549c20980 2024-09-12T02:48:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.172+0000 7f65227fc640 1 -- 172.21.15.16:0/346587294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240c4410 con 0x558549c20980 2024-09-12T02:48:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.173+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 msgr2=0x558549c25450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.173+0000 7f6540efc740 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f65240bbc70 tx=0x7f65240ef040 comp rx=0 tx=0).stop 2024-09-12T02:48:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 shutdown_connections 2024-09-12T02:48:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c2b240 0x558549c25990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c20980 0x558549c25450 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 --2- 172.21.15.16:0/346587294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 >> 172.21.15.16:0/346587294 conn(0x558549c1c5f0 msgr2=0x558549c46e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 shutdown_connections 2024-09-12T02:48:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 -- 172.21.15.16:0/346587294 wait complete. 2024-09-12T02:48:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.174+0000 7f6540efc740 1 -- start start 2024-09-12T02:48:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.175+0000 7f6540efc740 1 Processor -- start 2024-09-12T02:48:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.175+0000 7f6540efc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c2b240 0x558549c524b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f653c8bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f653c8bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42858/0 (socket says 172.21.15.16:42858) 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58914/0 (socket says 172.21.15.16:58914) 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f653c8bf640 1 -- 172.21.15.16:0/3624518550 learned_addr learned my addr 172.21.15.16:0/3624518550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65380733a0 con 0x558549c2b240 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6538072cd0 con 0x558549c20060 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f653806ff10 con 0x558549c20980 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f653d0c0640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c2b240 0x558549c524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6540efc740 10 client.0 _ll_register_callbacks cb 0x7f652c0137b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 -- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 msgr2=0x558549c53bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 -- 172.21.15.16:0/3624518550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c2b240 msgr2=0x558549c524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c2b240 0x558549c524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.176+0000 7f6537fff640 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558549c52a20 con 0x558549c20980 2024-09-12T02:48:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.177+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.177+0000 7f6537fff640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558549c22be0 tx=0x7f65240bbbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3f1847fbaa0e4d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.177+0000 7f6517fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240c9060 con 0x558549c20980 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f65240b9fe0 con 0x558549c20980 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65240038e0 con 0x558549c20980 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6524048070 con 0x558549c20980 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6540efc740 10 client.25228 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.178+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558549c52a20 con 0x558549c20980 2024-09-12T02:48:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65240ed2a0 con 0x558549c20980 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f652404b060 con 0x558549c20980 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 10 client.25228 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 10 client.25228 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 20 client.25228 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549cb7540 0x558549cb9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 4 client.25228 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5ce30e01", "release": true, "await": true}] 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.179+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5ce30e01", "release": true, "await": true}) -- 0x7f65240038e0 con 0x558549cb7540 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.180+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549cb7540 0x558549cb9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.181+0000 7f6517fff640 10 client.25228 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.181+0000 7f653c8bf640 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549cb7540 0x558549cb9930 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6816d4cfc581657e server_cookie=8d54d3aa42d789a7 in_seq=0 out_seq=0 2024-09-12T02:48:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6517fff640 1 -- 172.21.15.16:0/3624518550 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f65240038e0 con 0x558549cb7540 2024-09-12T02:48:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6517fff640 10 client.25228 handle_command_reply: tid=0 2024-09-12T02:48:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 client.25228 shutdown 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 msgr2=0x558549c51f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558549c22be0 tx=0x7f65240bbbe0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 shutdown_connections 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558549cb7540 0x558549cb9930 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558549c2b240 0x558549c524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.183+0000 7f6517fff640 0 client.25228 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558549c20980 0x558549c51f70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 1 --2- 172.21.15.16:0/3624518550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558549c20060 0x558549c53bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 >> 172.21.15.16:0/3624518550 conn(0x558549c1c5f0 msgr2=0x558549c28830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 shutdown_connections 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 1 -- 172.21.15.16:0/3624518550 wait complete. 2024-09-12T02:48:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.184+0000 7f6540efc740 20 client.25228 trim_cache size 0 max 16384 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 4, 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "5ce30e01": { 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4, 2024-09-12T02:48:49.187 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:48:49.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 msgr2=0x7f65280a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 0x7f65280a52a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x558549c1d970 tx=0x7f6530009500 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 msgr2=0x7f6538198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 0x7f6538198a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6524005940 tx=0x7f6524031b50 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 shutdown_connections 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6538106a70 0x7f653819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6538101410 0x7f6538198a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f65280a2de0 0x7f65280a52a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 --2- 172.21.15.16:0/2562503571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6538100a70 0x7f65381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.185+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 >> 172.21.15.16:0/2562503571 conn(0x7f65380fc790 msgr2=0x7f65380fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.186+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 shutdown_connections 2024-09-12T02:48:49.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:49.186+0000 7f653eb23640 1 -- 172.21.15.16:0/2562503571 wait complete. 2024-09-12T02:48:49.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:49 smithi016 ceph-mon[30925]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T02:48:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:49 smithi033 ceph-mon[35092]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T02:48:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:49 smithi184 ceph-mon[35321]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T02:48:50.335 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5ce30e01', total seconds elapsed: 30.0 2024-09-12T02:48:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.539+0000 7f478af21640 1 -- 172.21.15.16:0/594791660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784100a60 msgr2=0x7f4784100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.539+0000 7f478af21640 1 --2- 172.21.15.16:0/594791660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784100a60 0x7f4784100ec0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4770009930 tx=0x7f477002a420 comp rx=0 tx=0).stop 2024-09-12T02:48:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 -- 172.21.15.16:0/594791660 shutdown_connections 2024-09-12T02:48:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 --2- 172.21.15.16:0/594791660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784101400 0x7f478410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 --2- 172.21.15.16:0/594791660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784100a60 0x7f4784100ec0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 --2- 172.21.15.16:0/594791660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4784106a60 0x7f4784106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 -- 172.21.15.16:0/594791660 >> 172.21.15.16:0/594791660 conn(0x7f47840fc780 msgr2=0x7f47840feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 -- 172.21.15.16:0/594791660 shutdown_connections 2024-09-12T02:48:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.540+0000 7f478af21640 1 -- 172.21.15.16:0/594791660 wait complete. 2024-09-12T02:48:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 Processor -- start 2024-09-12T02:48:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 -- start start 2024-09-12T02:48:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4784100a60 0x7f47841079d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784106a60 0x7f4784108450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4784110e80 con 0x7f4784101400 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4784110d00 con 0x7f4784106a60 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f478af21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4784111000 con 0x7f4784100a60 2024-09-12T02:48:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f4783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f4783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37620/0 (socket says 172.21.15.16:37620) 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.541+0000 7f4783fff640 1 -- 172.21.15.16:0/724057264 learned_addr learned my addr 172.21.15.16:0/724057264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f47894be640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784106a60 0x7f4784108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 -- 172.21.15.16:0/724057264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4784100a60 msgr2=0x7f47841079d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4784100a60 0x7f47841079d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 -- 172.21.15.16:0/724057264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784106a60 msgr2=0x7f4784108450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784106a60 0x7f4784108450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 -- 172.21.15.16:0/724057264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47841a6990 con 0x7f4784101400 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4783fff640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4770009230 tx=0x7f4770004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.542+0000 7f4781ffb640 1 -- 172.21.15.16:0/724057264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477003f030 con 0x7f4784101400 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.543+0000 7f4781ffb640 1 -- 172.21.15.16:0/724057264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4770003ad0 con 0x7f4784101400 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.543+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47841a6c20 con 0x7f4784101400 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.543+0000 7f4781ffb640 1 -- 172.21.15.16:0/724057264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770035070 con 0x7f4784101400 2024-09-12T02:48:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.543+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47841a7150 con 0x7f4784101400 2024-09-12T02:48:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.545+0000 7f4781ffb640 1 -- 172.21.15.16:0/724057264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f4770042030 con 0x7f4784101400 2024-09-12T02:48:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.546+0000 7f4781ffb640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 0x7f47740a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f4788cbd640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 0x7f47740a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f4781ffb640 1 -- 172.21.15.16:0/724057264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47700ef3f0 con 0x7f4784101400 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 Processor -- start 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 -- start start 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494b55d0 0x55ce494b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f4788cbd640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 0x7f47740a51d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4778009710 tx=0x7f4778006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 0x55ce494c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ce4941b2f0 con 0x55ce494c0370 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ce493c2cf0 con 0x55ce494b6b40 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f478d2fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce494172b0 con 0x55ce494b55d0 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.547+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42888/0 (socket says 172.21.15.16:42888) 2024-09-12T02:48:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 -- 172.21.15.16:0/1311593371 learned_addr learned my addr 172.21.15.16:0/1311593371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4783fff640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494b55d0 0x55ce494b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f47894be640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 0x55ce494c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 -- 172.21.15.16:0/1311593371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494b55d0 msgr2=0x55ce494b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494b55d0 0x55ce494b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 -- 172.21.15.16:0/1311593371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 msgr2=0x55ce494c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 0x55ce494c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 -- 172.21.15.16:0/1311593371 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce493a73a0 con 0x55ce494b6b40 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f47894be640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 0x55ce494c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.548+0000 7f4788cbd640 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f477800b880 tx=0x7f477800e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d716afe54894b5f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.549+0000 7f476effd640 1 -- 172.21.15.16:0/1311593371 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778023070 con 0x55ce494b6b40 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.549+0000 7f476effd640 1 -- 172.21.15.16:0/1311593371 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4778015e20 con 0x55ce494b6b40 2024-09-12T02:48:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.549+0000 7f476effd640 1 -- 172.21.15.16:0/1311593371 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778013cc0 con 0x55ce494b6b40 2024-09-12T02:48:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- 172.21.15.16:0/1311593371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 msgr2=0x55ce494b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f477800b880 tx=0x7f477800e000 comp rx=0 tx=0).stop 2024-09-12T02:48:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- 172.21.15.16:0/1311593371 shutdown_connections 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494c0370 0x55ce494c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494b55d0 0x55ce494b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 --2- 172.21.15.16:0/1311593371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b5090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- 172.21.15.16:0/1311593371 >> 172.21.15.16:0/1311593371 conn(0x55ce4947e310 msgr2=0x55ce494b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- 172.21.15.16:0/1311593371 shutdown_connections 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- 172.21.15.16:0/1311593371 wait complete. 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.550+0000 7f478d2fa740 1 -- start start 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.551+0000 7f478d2fa740 1 Processor -- start 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.551+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494bfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.551+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494bd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f478d2fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ce494c3070 con 0x55ce494b55d0 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f478d2fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ce493d6060 con 0x55ce494b6b40 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f478d2fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce494c31f0 con 0x55ce494c0370 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f4783fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58942/0 (socket says 172.21.15.16:58942) 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 -- 172.21.15.16:0/225164187 learned_addr learned my addr 172.21.15.16:0/225164187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f4788cbd640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494bfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 -- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 msgr2=0x55ce494bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 -- 172.21.15.16:0/225164187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 msgr2=0x55ce494bfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494bfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f47894be640 1 -- 172.21.15.16:0/225164187 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce4918c3c0 con 0x55ce494c0370 2024-09-12T02:48:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f4788cbd640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494bfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.552+0000 7f478d2fa740 10 client.0 _ll_register_callbacks cb 0x55ce491298c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f47894be640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f477c011710 tx=0x7f477c011bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5924c8741736af91 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4778015e20 con 0x55ce494c0370 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f4763fff640 10 client.25234 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f4763fff640 1 -- 172.21.15.16:0/225164187 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c004d20 con 0x55ce494c0370 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f4763fff640 1 -- 172.21.15.16:0/225164187 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f477c00f940 con 0x55ce494c0370 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f4763fff640 1 -- 172.21.15.16:0/225164187 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f477c00fb80 con 0x55ce494c0370 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f478d2fa740 10 client.25234 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.553+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55ce494cea30 con 0x55ce494c0370 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f4763fff640 1 -- 172.21.15.16:0/225164187 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f477c00f040 con 0x55ce494c0370 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 10 client.25234 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 10 client.25234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 20 client.25234 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce494cffc0 0x55ce494d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 4 client.25234 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4778015e20 con 0x55ce494cffc0 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.554+0000 7f4788cbd640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce494cffc0 0x55ce494d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.555+0000 7f4788cbd640 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce494cffc0 0x55ce494d2480 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e393d4936e866710 server_cookie=96f0ecefeae5b3bf in_seq=0 out_seq=0 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.555+0000 7f4763fff640 10 client.25234 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.555+0000 7f4763fff640 1 -- 172.21.15.16:0/225164187 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f4778015e20 con 0x55ce494cffc0 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.555+0000 7f4763fff640 10 client.25234 handle_command_reply: tid=0 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.555+0000 7f478d2fa740 1 client.25234 shutdown 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 msgr2=0x55ce494bd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f477c011710 tx=0x7f477c011bd0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 shutdown_connections 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce494cffc0 0x55ce494d2480 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494bd7e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 --2- 172.21.15.16:0/225164187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494bfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f4763fff640 0 client.25234 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 >> 172.21.15.16:0/225164187 conn(0x55ce4947e310 msgr2=0x55ce494b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 shutdown_connections 2024-09-12T02:48:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.556+0000 7f478d2fa740 1 -- 172.21.15.16:0/225164187 wait complete. 2024-09-12T02:48:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.557+0000 7f478d2fa740 20 client.25234 trim_cache size 0 max 16384 2024-09-12T02:48:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.582+0000 7f478d2fa740 1 Processor -- start 2024-09-12T02:48:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.582+0000 7f478d2fa740 1 -- start start 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494d7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47841af910 con 0x55ce494b55d0 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47841b1450 con 0x55ce494b6b40 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f478d2fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47841affe0 con 0x55ce494c0370 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37644/0 (socket says 172.21.15.16:37644) 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 -- 172.21.15.16:0/2667278896 learned_addr learned my addr 172.21.15.16:0/2667278896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4783fff640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f47894be640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494d7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 -- 172.21.15.16:0/2667278896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 msgr2=0x55ce494d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494d7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 -- 172.21.15.16:0/2667278896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 msgr2=0x55ce494b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f4788cbd640 1 -- 172.21.15.16:0/2667278896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce494cc7f0 con 0x55ce494b55d0 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.583+0000 7f47894be640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494d7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.584+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4778021040 tx=0x7f47780138c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ef7a886af0a495 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.584+0000 7f476e7fc640 1 -- 172.21.15.16:0/2667278896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778023040 con 0x55ce494b55d0 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.584+0000 7f476e7fc640 1 -- 172.21.15.16:0/2667278896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4778013420 con 0x55ce494b55d0 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.584+0000 7f476e7fc640 1 -- 172.21.15.16:0/2667278896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778015da0 con 0x55ce494b55d0 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 -- 172.21.15.16:0/2667278896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 msgr2=0x55ce494b8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4778021040 tx=0x7f47780138c0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 -- 172.21.15.16:0/2667278896 shutdown_connections 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494d7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2667278896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494b8120 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 -- 172.21.15.16:0/2667278896 >> 172.21.15.16:0/2667278896 conn(0x55ce4947e310 msgr2=0x55ce494bc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 -- 172.21.15.16:0/2667278896 shutdown_connections 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.585+0000 7f478d2fa740 1 -- 172.21.15.16:0/2667278896 wait complete. 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.586+0000 7f478d2fa740 1 -- start start 2024-09-12T02:48:50.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.586+0000 7f478d2fa740 1 Processor -- start 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.586+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f4788cbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37648/0 (socket says 172.21.15.16:37648) 2024-09-12T02:48:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47841b0d80 con 0x55ce494b55d0 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f4788cbd640 1 -- 172.21.15.16:0/2854599681 learned_addr learned my addr 172.21.15.16:0/2854599681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f4783fff640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47841b06b0 con 0x55ce494b6b40 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f47894be640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47841ad8f0 con 0x55ce494c0370 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.587+0000 7f478d2fa740 10 client.0 _ll_register_callbacks cb 0x7f477c0096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 -- 172.21.15.16:0/2854599681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 msgr2=0x55ce494c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 -- 172.21.15.16:0/2854599681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 msgr2=0x55ce494c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce495423f0 con 0x55ce494b6b40 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4783fff640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494c47c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47700ecd20 tx=0x7f4770005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3a64652a4478a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4763fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770049040 con 0x55ce494b6b40 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47700b8110 con 0x55ce494b6b40 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4778013420 con 0x55ce494b6b40 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f47700ecfa0 con 0x55ce494b6b40 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f478d2fa740 10 client.25349 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.588+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ce49543710 con 0x55ce494b6b40 2024-09-12T02:48:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47700ed140 con 0x55ce494b6b40 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f477004c060 con 0x55ce494b6b40 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 10 client.25349 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 10 client.25349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 20 client.25349 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce49543e30 0x55ce49546220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 4 client.25349 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:48:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4778013420 con 0x55ce49543e30 2024-09-12T02:48:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.589+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce49543e30 0x55ce49546220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f4788cbd640 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce49543e30 0x55ce49546220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6fc70b512a902cd server_cookie=758a3e61b7e3ef24 in_seq=0 out_seq=0 2024-09-12T02:48:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f4763fff640 10 client.25349 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f4763fff640 1 -- 172.21.15.16:0/2854599681 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4778013420 con 0x55ce49543e30 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f4763fff640 10 client.25349 handle_command_reply: tid=0 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 client.25349 shutdown 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 msgr2=0x55ce494c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494c47c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47700ecd20 tx=0x7f4770005b50 comp rx=0 tx=0).stop 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 shutdown_connections 2024-09-12T02:48:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ce49543e30 0x55ce49546220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f4763fff640 0 client.25349 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce494c0370 0x55ce494c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce494b6b40 0x55ce494c47c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 --2- 172.21.15.16:0/2854599681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce494b55d0 0x55ce494c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.590+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 >> 172.21.15.16:0/2854599681 conn(0x55ce4947e310 msgr2=0x55ce494be8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 shutdown_connections 2024-09-12T02:48:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478d2fa740 1 -- 172.21.15.16:0/2854599681 wait complete. 2024-09-12T02:48:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478d2fa740 20 client.25349 trim_cache size 0 max 16384 2024-09-12T02:48:50.600 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:48:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 msgr2=0x7f47740a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 0x7f47740a51d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4778009710 tx=0x7f4778006c90 comp rx=0 tx=0).stop 2024-09-12T02:48:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 msgr2=0x7f4784107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.591+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4770009230 tx=0x7f4770004270 comp rx=0 tx=0).stop 2024-09-12T02:48:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 shutdown_connections 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4784106a60 0x7f4784108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47740a2d10 0x7f47740a51d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4784101400 0x7f4784107f10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 --2- 172.21.15.16:0/724057264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4784100a60 0x7f47841079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 >> 172.21.15.16:0/724057264 conn(0x7f47840fc780 msgr2=0x7f4784104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 shutdown_connections 2024-09-12T02:48:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:50.592+0000 7f478af21640 1 -- 172.21.15.16:0/724057264 wait complete. 2024-09-12T02:48:51.337 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:51 smithi016 ceph-mon[30925]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:51 smithi033 ceph-mon[35092]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:51 smithi184 ceph-mon[35321]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-12T02:48:52.378 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:48:52.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:48:52.640 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.066+0000 7f4893a10640 1 -- 172.21.15.16:0/1017141455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c101360 msgr2=0x7f488c10df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.066+0000 7f4893a10640 1 --2- 172.21.15.16:0/1017141455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c101360 0x7f488c10df90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f487c009900 tx=0x7f487c02a410 comp rx=0 tx=0).stop 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 -- 172.21.15.16:0/1017141455 shutdown_connections 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 --2- 172.21.15.16:0/1017141455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f488c10e4d0 0x7f488c1108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 --2- 172.21.15.16:0/1017141455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c101360 0x7f488c10df90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 --2- 172.21.15.16:0/1017141455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f488c1009b0 0x7f488c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 -- 172.21.15.16:0/1017141455 >> 172.21.15.16:0/1017141455 conn(0x7f488c0fc7d0 msgr2=0x7f488c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 -- 172.21.15.16:0/1017141455 shutdown_connections 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 -- 172.21.15.16:0/1017141455 wait complete. 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.067+0000 7f4893a10640 1 Processor -- start 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 -- start start 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f488c1009b0 0x7f488c1a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f488c101360 0x7f488c1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f488c114250 con 0x7f488c10e4d0 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1140d0 con 0x7f488c1009b0 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4893a10640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1143d0 con 0x7f488c101360 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4891fad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4891fad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37672/0 (socket says 172.21.15.16:37672) 2024-09-12T02:48:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.068+0000 7f4891fad640 1 -- 172.21.15.16:0/3953819796 learned_addr learned my addr 172.21.15.16:0/3953819796 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 -- 172.21.15.16:0/3953819796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f488c101360 msgr2=0x7f488c1a0940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f488c101360 0x7f488c1a0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 -- 172.21.15.16:0/3953819796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f488c1009b0 msgr2=0x7f488c1a0400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f488c1009b0 0x7f488c1a0400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 -- 172.21.15.16:0/3953819796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c19acf0 con 0x7f488c10e4d0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f4891fad640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f488400eaa0 tx=0x7f488400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.069+0000 7f48827fc640 1 -- 172.21.15.16:0/3953819796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488400cdb0 con 0x7f488c10e4d0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.070+0000 7f48827fc640 1 -- 172.21.15.16:0/3953819796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4884005020 con 0x7f488c10e4d0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.070+0000 7f48827fc640 1 -- 172.21.15.16:0/3953819796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840116d0 con 0x7f488c10e4d0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.070+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c19afe0 con 0x7f488c10e4d0 2024-09-12T02:48:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.070+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c1a72d0 con 0x7f488c10e4d0 2024-09-12T02:48:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.074+0000 7f4895de9740 1 Processor -- start 2024-09-12T02:48:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.075+0000 7f48827fc640 1 -- 172.21.15.16:0/3953819796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f4884011870 con 0x7f488c10e4d0 2024-09-12T02:48:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.075+0000 7f4895de9740 1 -- start start 2024-09-12T02:48:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.075+0000 7f4895de9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa267e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.075+0000 7f4895de9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f48917ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f48917ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37682/0 (socket says 172.21.15.16:37682) 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f48917ac640 1 -- 172.21.15.16:0/2422213184 learned_addr learned my addr 172.21.15.16:0/2422213184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4895de9740 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa27d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563efa1cd280 con 0x563efa2683a0 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563efa174b10 con 0x563efa27b550 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563efa1c9240 con 0x563efa267a80 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa267e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 -- 172.21.15.16:0/2422213184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 msgr2=0x563efa27d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa27d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 -- 172.21.15.16:0/2422213184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 msgr2=0x563efa27b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 -- 172.21.15.16:0/2422213184 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563efa1811f0 con 0x563efa267a80 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.076+0000 7f4890fab640 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa267e60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f487c02a930 tx=0x7f487c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13114de7ddde9246 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48827fc640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 0x7f48780a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48917ac640 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48777fe640 1 -- 172.21.15.16:0/2422213184 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c03e070 con 0x563efa267a80 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48827fc640 1 -- 172.21.15.16:0/3953819796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48840cf2b0 con 0x7f488c10e4d0 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48917ac640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 0x7f48780a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48777fe640 1 -- 172.21.15.16:0/2422213184 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f487c0043f0 con 0x563efa267a80 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.077+0000 7f48777fe640 1 -- 172.21.15.16:0/2422213184 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c035040 con 0x563efa267a80 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f48917ac640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 0x7f48780a5220 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f488800bdb0 tx=0x7f488800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 msgr2=0x563efa267e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa267e60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f487c02a930 tx=0x7f487c004040 comp rx=0 tx=0).stop 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 shutdown_connections 2024-09-12T02:48:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa27d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa2683a0 0x563efa27b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 --2- 172.21.15.16:0/2422213184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa267e60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.078+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 >> 172.21.15.16:0/2422213184 conn(0x563efa2640f0 msgr2=0x563efa2644d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.079+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 shutdown_connections 2024-09-12T02:48:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.079+0000 7f4895de9740 1 -- 172.21.15.16:0/2422213184 wait complete. 2024-09-12T02:48:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.079+0000 7f4895de9740 1 -- start start 2024-09-12T02:48:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.079+0000 7f4895de9740 1 Processor -- start 2024-09-12T02:48:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa2784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa2789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563efa27e0b0 con 0x563efa27b550 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563efa187f10 con 0x563efa2683a0 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563efa27e230 con 0x563efa267a80 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4895de9740 10 client.0 _ll_register_callbacks cb 0x563efa150a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f48917ac640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa2784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37690/0 (socket says 172.21.15.16:37690) 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 -- 172.21.15.16:0/1643520961 learned_addr learned my addr 172.21.15.16:0/1643520961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 -- 172.21.15.16:0/1643520961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 msgr2=0x563efa2784a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4890fab640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa2789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa2784a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 -- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 msgr2=0x563efa2789e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa2789e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4891fad640 1 -- 172.21.15.16:0/1643520961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ef9f3e6a0 con 0x563efa27b550 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f48917ac640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa2784a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.080+0000 7f4890fab640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa2789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4891fad640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f48840a4100 tx=0x7f488409feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1eaa2187ec92edd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488401f040 con 0x563efa27b550 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f487c0043f0 con 0x563efa27b550 2024-09-12T02:48:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48840b4bc0 con 0x563efa27b550 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48840b4dc0 con 0x563efa27b550 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4895de9740 10 client.15723 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563efa289e40 con 0x563efa27b550 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.081+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f487c0043f0 con 0x563efa27b550 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f488400f040 con 0x563efa27b550 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 10 client.15723 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 10 client.15723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 20 client.15723 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa28d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 4 client.15723 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563efa1da7f0 con 0x563efa28b3a0 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.082+0000 7f48917ac640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa28d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.083+0000 7f4874ff9640 10 client.15723 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:53.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.083+0000 7f48917ac640 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa28d7d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5805f9948073485 server_cookie=a4a91f18a3cf3cac in_seq=0 out_seq=0 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.083+0000 7f4874ff9640 1 -- 172.21.15.16:0/1643520961 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563efa1da7f0 con 0x563efa28b3a0 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.083+0000 7f4874ff9640 10 client.15723 handle_command_reply: tid=0 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.083+0000 7f4895de9740 1 client.15723 shutdown 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 msgr2=0x563efa278f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f48840a4100 tx=0x7f488409feb0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 shutdown_connections 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa28d7d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4874ff9640 0 client.15723 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa27b550 0x563efa278f20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa2789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 --2- 172.21.15.16:0/1643520961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa267a80 0x563efa2784a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 >> 172.21.15.16:0/1643520961 conn(0x563efa2640f0 msgr2=0x563efa269d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 shutdown_connections 2024-09-12T02:48:53.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 1 -- 172.21.15.16:0/1643520961 wait complete. 2024-09-12T02:48:53.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.084+0000 7f4895de9740 20 client.15723 trim_cache size 0 max 16384 2024-09-12T02:48:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 Processor -- start 2024-09-12T02:48:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 -- start start 2024-09-12T02:48:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa2773e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 0x563efa2930e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa293620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f487000e400 con 0x563efa267a80 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f487000ff40 con 0x563efa27b550 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4895de9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f487000ead0 con 0x563efa2683a0 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4890fab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 0x563efa2930e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f48917ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa2773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4890fab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 0x563efa2930e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59022/0 (socket says 172.21.15.16:59022) 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4890fab640 1 -- 172.21.15.16:0/1486533563 learned_addr learned my addr 172.21.15.16:0/1486533563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.102+0000 7f4891fad640 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa293620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 -- 172.21.15.16:0/1486533563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 msgr2=0x563efa2930e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 0x563efa2930e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 -- 172.21.15.16:0/1486533563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 msgr2=0x563efa2773e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa2773e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 -- 172.21.15.16:0/1486533563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563efa268f50 con 0x563efa27b550 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f48917ac640 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa2773e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4891fad640 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa293620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f488409b6c0 tx=0x7f488409b450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57087b617f97ee8e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4876ffd640 1 -- 172.21.15.16:0/1486533563 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488401f070 con 0x563efa27b550 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4876ffd640 1 -- 172.21.15.16:0/1486533563 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48840b4070 con 0x563efa27b550 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.103+0000 7f4876ffd640 1 -- 172.21.15.16:0/1486533563 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488409a980 con 0x563efa27b550 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 -- 172.21.15.16:0/1486533563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 msgr2=0x563efa293620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa293620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f488409b6c0 tx=0x7f488409b450 comp rx=0 tx=0).stop 2024-09-12T02:48:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 -- 172.21.15.16:0/1486533563 shutdown_connections 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa27b550 0x563efa293620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa2683a0 0x563efa2930e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 --2- 172.21.15.16:0/1486533563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa2773e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 -- 172.21.15.16:0/1486533563 >> 172.21.15.16:0/1486533563 conn(0x563efa2640f0 msgr2=0x563efa27a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 -- 172.21.15.16:0/1486533563 shutdown_connections 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.104+0000 7f4895de9740 1 -- 172.21.15.16:0/1486533563 wait complete. 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.105+0000 7f4895de9740 1 -- start start 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.105+0000 7f4895de9740 1 Processor -- start 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.105+0000 7f4895de9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.105+0000 7f4895de9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.105+0000 7f4895de9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa27b550 0x563efa2f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f48917ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f48917ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37698/0 (socket says 172.21.15.16:37698) 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4895de9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f487000f870 con 0x563efa267a80 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4895de9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f487000f1a0 con 0x563efa2683a0 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4895de9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f487000c3e0 con 0x563efa27b550 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4895de9740 10 client.0 _ll_register_callbacks cb 0x7f488409b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42970/0 (socket says 172.21.15.16:42970) 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 -- 172.21.15.16:0/2723637419 learned_addr learned my addr 172.21.15.16:0/2723637419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 -- 172.21.15.16:0/2723637419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa27b550 msgr2=0x563efa2f50b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa27b550 0x563efa2f50b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 -- 172.21.15.16:0/2723637419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 msgr2=0x563efa27f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 -- 172.21.15.16:0/2723637419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563efa2f55f0 con 0x563efa2683a0 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f48917ac640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.106+0000 7f4890fab640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f487c02a450 tx=0x7f487c004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=867a6d918577760b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c035070 con 0x563efa2683a0 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48840b4070 con 0x563efa2683a0 2024-09-12T02:48:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f487c0319c0 con 0x563efa2683a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f487c03e040 con 0x563efa2683a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 10 client.25358 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563efa2f67f0 con 0x563efa2683a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48840b4070 con 0x563efa2683a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f487c042400 con 0x563efa2683a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 10 client.25358 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 10 client.25358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 20 client.25358 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa2f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 4 client.25358 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.107+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563efa1d4d00 con 0x563efa28b3a0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.108+0000 7f48917ac640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa2f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.108+0000 7f48917ac640 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa2f8b60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559767a237cd7ade server_cookie=e14d7948c578e1b2 in_seq=0 out_seq=0 2024-09-12T02:48:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.108+0000 7f4874ff9640 10 client.25358 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4874ff9640 1 -- 172.21.15.16:0/2723637419 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563efa1d4d00 con 0x563efa28b3a0 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4874ff9640 10 client.25358 handle_command_reply: tid=0 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4895de9740 1 client.25358 shutdown 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 msgr2=0x563efa27fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f487c02a450 tx=0x7f487c004420 comp rx=0 tx=0).stop 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 shutdown_connections 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.109+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563efa28b3a0 0x563efa2f8b60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563efa27b550 0x563efa2f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563efa2683a0 0x563efa27fba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 --2- 172.21.15.16:0/2723637419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563efa267a80 0x563efa27f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 >> 172.21.15.16:0/2723637419 conn(0x563efa2640f0 msgr2=0x563efa278460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4874ff9640 0 client.25358 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 shutdown_connections 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 1 -- 172.21.15.16:0/2723637419 wait complete. 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.110+0000 7f4895de9740 20 client.25358 trim_cache size 0 max 16384 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:48:53.120 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b0abb43d-6edc-471e-8d35-a9cf7b193b80", 2024-09-12T02:48:53.121 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:48:53.121 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 msgr2=0x7f48780a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 0x7f48780a5220 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f488800bdb0 tx=0x7f488800b390 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 msgr2=0x7f488c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f488400eaa0 tx=0x7f488400ef60 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 shutdown_connections 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48780a2d60 0x7f48780a5220 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c10e4d0 0x7f488c19a5c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f488c101360 0x7f488c1a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 --2- 172.21.15.16:0/3953819796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f488c1009b0 0x7f488c1a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 >> 172.21.15.16:0/3953819796 conn(0x7f488c0fc7d0 msgr2=0x7f488c10f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 shutdown_connections 2024-09-12T02:48:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:53.111+0000 7f4893a10640 1 -- 172.21.15.16:0/3953819796 wait complete. 2024-09-12T02:48:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:53 smithi016 ceph-mon[30925]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:53 smithi033 ceph-mon[35092]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:53 smithi184 ceph-mon[35321]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-12T02:48:54.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:54 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:48:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:54 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:54 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:48:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:54 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:54 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:48:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:54 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:48:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:54 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:54 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:48:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:54 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:54 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:48:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:54 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:48:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:54 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:54 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:48:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:54 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:48:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:54 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:48:55.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:55 smithi016 ceph-mon[30925]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-12T02:48:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:55 smithi033 ceph-mon[35092]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-12T02:48:55.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:48:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:55 smithi184 ceph-mon[35321]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-12T02:48:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 -- 172.21.15.16:0/1722666680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa328102d30 msgr2=0x7fa328103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 --2- 172.21.15.16:0/1722666680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa328102d30 0x7fa328103110 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa324009930 tx=0x7fa32402a420 comp rx=0 tx=0).stop 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 -- 172.21.15.16:0/1722666680 shutdown_connections 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 --2- 172.21.15.16:0/1722666680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 0x7fa32810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 --2- 172.21.15.16:0/1722666680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3281036e0 0x7fa328107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 --2- 172.21.15.16:0/1722666680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa328102d30 0x7fa328103110 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 -- 172.21.15.16:0/1722666680 >> 172.21.15.16:0/1722666680 conn(0x7fa3280fc7d0 msgr2=0x7fa3280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 -- 172.21.15.16:0/1722666680 shutdown_connections 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.351+0000 7fa32feb3640 1 -- 172.21.15.16:0/1722666680 wait complete. 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 Processor -- start 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 -- start start 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa328102d30 0x7fa328100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 0x7fa328101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa328112100 con 0x7fa3281036e0 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa328111f80 con 0x7fa3281082a0 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32feb3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa328112280 con 0x7fa328102d30 2024-09-12T02:48:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32d44e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32d44e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59008/0 (socket says 172.21.15.16:59008) 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32d44e640 1 -- 172.21.15.16:0/568734326 learned_addr learned my addr 172.21.15.16:0/568734326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.352+0000 7fa32e450640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 0x7fa328101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa328102d30 0x7fa328100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 -- 172.21.15.16:0/568734326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa328102d30 msgr2=0x7fa328100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa328102d30 0x7fa328100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 -- 172.21.15.16:0/568734326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 msgr2=0x7fa328101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 0x7fa328101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 -- 172.21.15.16:0/568734326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3281a6a80 con 0x7fa3281036e0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32d44e640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa31800d750 tx=0x7fa31800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa31effd640 1 -- 172.21.15.16:0/568734326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318017070 con 0x7fa3281036e0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa31effd640 1 -- 172.21.15.16:0/568734326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa318005500 con 0x7fa3281036e0 2024-09-12T02:48:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3281a6d70 con 0x7fa3281036e0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa31effd640 1 -- 172.21.15.16:0/568734326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318004e00 con 0x7fa3281036e0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.353+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3281a7250 con 0x7fa3281036e0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.356+0000 7fa31effd640 1 -- 172.21.15.16:0/568734326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fa318004fa0 con 0x7fa3281036e0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 Processor -- start 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 -- start start 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb8dc0 0x55847dcc3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 0x55847dcc5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55847dc1e2e0 con 0x55847dcb8dc0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55847dbc5ce0 con 0x55847dcc3680 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa33228c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55847dc1a2a0 con 0x55847dcb84a0 2024-09-12T02:48:56.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa32e450640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa32d44e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 0x55847dcc5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa32e450640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55546/0 (socket says 172.21.15.16:55546) 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa32e450640 1 -- 172.21.15.16:0/3743365828 learned_addr learned my addr 172.21.15.16:0/3743365828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.357+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb8dc0 0x55847dcc3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 -- 172.21.15.16:0/3743365828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 msgr2=0x55847dcb8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 -- 172.21.15.16:0/3743365828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb8dc0 msgr2=0x55847dcc3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb8dc0 0x55847dcc3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 -- 172.21.15.16:0/3743365828 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847dbaa440 con 0x55847dcc3680 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa31effd640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 0x7fa3140a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32e450640 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32d44e640 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 0x55847dcc5a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3180cc490 tx=0x7fa3180d7620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88c0d70735c914d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa31effd640 1 -- 172.21.15.16:0/568734326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3180ceb60 con 0x7fa3281036e0 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa32e450640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 0x7fa3140a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa313fff640 1 -- 172.21.15.16:0/3743365828 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318020070 con 0x55847dcc3680 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa313fff640 1 -- 172.21.15.16:0/3743365828 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3180b9e50 con 0x55847dcc3680 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.358+0000 7fa313fff640 1 -- 172.21.15.16:0/3743365828 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3180046c0 con 0x55847dcc3680 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa32e450640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 0x7fa3140a5150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa320005f40 tx=0x7fa3200096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 -- 172.21.15.16:0/3743365828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 msgr2=0x55847dcc5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 0x55847dcc5a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3180cc490 tx=0x7fa3180d7620 comp rx=0 tx=0).stop 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 -- 172.21.15.16:0/3743365828 shutdown_connections 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcc3680 0x55847dcc5a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb8dc0 0x55847dcc3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 --2- 172.21.15.16:0/3743365828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb84a0 0x55847dcb8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 -- 172.21.15.16:0/3743365828 >> 172.21.15.16:0/3743365828 conn(0x55847dcb4a30 msgr2=0x55847dcb4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 -- 172.21.15.16:0/3743365828 shutdown_connections 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.359+0000 7fa33228c740 1 -- 172.21.15.16:0/3743365828 wait complete. 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 -- start start 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 Processor -- start 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcc2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcc0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55847dcc6250 con 0x55847dcb84a0 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55847dbd9050 con 0x55847dcb8dc0 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55847dcc63d0 con 0x55847dcc3680 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.360+0000 7fa33228c740 10 client.0 _ll_register_callbacks cb 0x55847dc19160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59126/0 (socket says 172.21.15.16:59126) 2024-09-12T02:48:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 -- 172.21.15.16:0/4254901324 learned_addr learned my addr 172.21.15.16:0/4254901324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32e450640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcc0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 -- 172.21.15.16:0/4254901324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 msgr2=0x55847dcc0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcc0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 -- 172.21.15.16:0/4254901324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 msgr2=0x55847dcc2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcc2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 -- 172.21.15.16:0/4254901324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55847d98f3c0 con 0x55847dcb8dc0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa32d44e640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3180cc4c0 tx=0x7fa3180d2450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf77305b563b013 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa3117fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.361+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318020040 con 0x55847dcb8dc0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa318099510 con 0x55847dcb8dc0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3180b9e50 con 0x55847dcb8dc0 2024-09-12T02:48:56.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa31809afe0 con 0x55847dcb8dc0 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 10 client.25370 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55847dcd1bd0 con 0x55847dcb8dc0 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3180b9e50 con 0x55847dcb8dc0 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa3180d6070 con 0x55847dcb8dc0 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 10 client.25370 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 10 client.25370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 20 client.25370 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dcd2f20 0x55847dcd5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 4 client.25370 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.362+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55847dc2b850 con 0x55847dcd2f20 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.363+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dcd2f20 0x55847dcd5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.363+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dcd2f20 0x55847dcd5310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad0defb827e462 server_cookie=292ba31a3b14210b in_seq=0 out_seq=0 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.363+0000 7fa3117fa640 10 client.25370 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa3117fa640 1 -- 172.21.15.16:0/4254901324 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55847dc2b850 con 0x55847dcd2f20 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa3117fa640 10 client.25370 handle_command_reply: tid=0 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 client.25370 shutdown 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 msgr2=0x55847dcc0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3180cc4c0 tx=0x7fa3180d2450 comp rx=0 tx=0).stop 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 shutdown_connections 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dcd2f20 0x55847dcd5310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcc0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcc0250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa3117fa640 0 client.25370 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 --2- 172.21.15.16:0/4254901324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcc2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 >> 172.21.15.16:0/4254901324 conn(0x55847dcb4a30 msgr2=0x55847dcba760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 shutdown_connections 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 1 -- 172.21.15.16:0/4254901324 wait complete. 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.364+0000 7fa33228c740 20 client.25370 trim_cache size 0 max 16384 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 Processor -- start 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 -- start start 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb84a0 0x55847dcbd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb8dc0 0x55847dcbd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281af880 con 0x55847dcc3680 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281b13c0 con 0x55847dcb84a0 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa33228c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281aff50 con 0x55847dcb8dc0 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa32e450640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa32e450640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59042/0 (socket says 172.21.15.16:59042) 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa32e450640 1 -- 172.21.15.16:0/1636016614 learned_addr learned my addr 172.21.15.16:0/1636016614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa32d44e640 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb8dc0 0x55847dcbd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.382+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb84a0 0x55847dcbd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 -- 172.21.15.16:0/1636016614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb8dc0 msgr2=0x55847dcbd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb8dc0 0x55847dcbd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 -- 172.21.15.16:0/1636016614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb84a0 msgr2=0x55847dcbd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb84a0 0x55847dcbd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 -- 172.21.15.16:0/1636016614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847dcb8bf0 con 0x55847dcc3680 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa32e450640 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa320010aa0 tx=0x7fa320019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b9ad7fbf75b586 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa3137fe640 1 -- 172.21.15.16:0/1636016614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320004b20 con 0x55847dcc3680 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa3137fe640 1 -- 172.21.15.16:0/1636016614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa320004cc0 con 0x55847dcc3680 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.383+0000 7fa3137fe640 1 -- 172.21.15.16:0/1636016614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320005950 con 0x55847dcc3680 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 -- 172.21.15.16:0/1636016614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 msgr2=0x55847dcc1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa320010aa0 tx=0x7fa320019000 comp rx=0 tx=0).stop 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 -- 172.21.15.16:0/1636016614 shutdown_connections 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcc3680 0x55847dcc1ab0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcb8dc0 0x55847dcbd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 --2- 172.21.15.16:0/1636016614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb84a0 0x55847dcbd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 -- 172.21.15.16:0/1636016614 >> 172.21.15.16:0/1636016614 conn(0x55847dcb4a30 msgr2=0x55847dcbb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 -- 172.21.15.16:0/1636016614 shutdown_connections 2024-09-12T02:48:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.384+0000 7fa33228c740 1 -- 172.21.15.16:0/1636016614 wait complete. 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.385+0000 7fa33228c740 1 -- start start 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.385+0000 7fa33228c740 1 Processor -- start 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcd5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281b0cf0 con 0x55847dcb84a0 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281b0620 con 0x55847dcb8dc0 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32dc4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281ad860 con 0x55847dcc3680 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55580/0 (socket says 172.21.15.16:55580) 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 -- 172.21.15.16:0/390507214 learned_addr learned my addr 172.21.15.16:0/390507214 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:48:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32dc4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59046/0 (socket says 172.21.15.16:59046) 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa33228c740 10 client.0 _ll_register_callbacks cb 0x7fa3180cef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32d44e640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcd5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 -- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 msgr2=0x55847dcd5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcd5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 -- 172.21.15.16:0/390507214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 msgr2=0x55847dcd8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32e450640 1 -- 172.21.15.16:0/390507214 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55847dcd6b60 con 0x55847dcc3680 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.386+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa32e450640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa320005f10 tx=0x7fa320005340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f18d49659f69f1bf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320036060 con 0x55847dcc3680 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa320005080 con 0x55847dcc3680 2024-09-12T02:48:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa320004cc0 con 0x55847dcc3680 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa32001ea60 con 0x55847dcc3680 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa33228c740 10 client.25270 fetch_fsmap learned FSMap version 28 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55847dd3d880 con 0x55847dcc3680 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa320004cc0 con 0x55847dcc3680 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.387+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa320030030 con 0x55847dcc3680 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 10 client.25270 fetch_fsmap finished waiting for FSMap version 28 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 10 client.25270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 20 client.25270 populate_metadata read hostname 'smithi016' 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dd3de90 0x55847dd40280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 4 client.25270 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55847dc25d60 con 0x55847dd3de90 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dd3de90 0x55847dd40280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:48:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa32dc4f640 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dd3de90 0x55847dd40280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b77cc50be2fe6d6 server_cookie=cefc342c639d3f2b in_seq=0 out_seq=0 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.388+0000 7fa3117fa640 10 client.25270 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa3117fa640 1 -- 172.21.15.16:0/390507214 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55847dc25d60 con 0x55847dd3de90 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa3117fa640 10 client.25270 handle_command_reply: tid=0 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 client.25270 shutdown 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 msgr2=0x55847dcd6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa320005f10 tx=0x7fa320005340 comp rx=0 tx=0).stop 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 shutdown_connections 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55847dd3de90 0x55847dd40280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55847dcc3680 0x55847dcd6360 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55847dcb8dc0 0x55847dcd5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa3117fa640 0 client.25270 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 --2- 172.21.15.16:0/390507214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55847dcb84a0 0x55847dcd8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 >> 172.21.15.16:0/390507214 conn(0x55847dcb4a30 msgr2=0x55847dcb9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 shutdown_connections 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stdout: "b0abb43d-6edc-471e-8d35-a9cf7b193b80": { 2024-09-12T02:48:56.422 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:48:56.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b0abb43d-6edc-471e-8d35-a9cf7b193b80", 2024-09-12T02:48:56.423 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:48:56.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:56.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:48:56.423 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 1 -- 172.21.15.16:0/390507214 wait complete. 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.389+0000 7fa33228c740 20 client.25270 trim_cache size 0 max 16384 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 msgr2=0x7fa3140a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 0x7fa3140a5150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa320005f40 tx=0x7fa3200096a0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 msgr2=0x7fa328100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa31800d750 tx=0x7fa31800dc10 comp rx=0 tx=0).stop 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 shutdown_connections 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3281082a0 0x7fa328101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3140a2c90 0x7fa3140a5150 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3281036e0 0x7fa328100ed0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 --2- 172.21.15.16:0/568734326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa328102d30 0x7fa328100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:48:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 >> 172.21.15.16:0/568734326 conn(0x7fa3280fc7d0 msgr2=0x7fa328106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:48:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 shutdown_connections 2024-09-12T02:48:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:48:56.390+0000 7fa32feb3640 1 -- 172.21.15.16:0/568734326 wait complete. 2024-09-12T02:48:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:48:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:48:56.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:48:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:48:56.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:48:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:48:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:48:57.115 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 88% (11434/12903) Updating files: 89% (11484/12903) Updating files: 90% (11613/12903) Updating files: 91% (11742/12903) Updating files: 92% (11871/12903) Updating files: 93% (12000/12903) Updating files: 94% (12129/12903) Updating files: 95% (12258/12903) Updating files: 96% (12387/12903) Updating files: 97% (12516/12903) Updating files: 98% (12645/12903) Updating files: 99% (12774/12903) Updating files: 100% (12903/12903) Updating files: 100% (12903/12903), done. 2024-09-12T02:48:57.238 INFO:tasks.cephfs.filesystem:scrub status for tag:b0abb43d-6edc-471e-8d35-a9cf7b193b80 - {'path': '/', 'tag': 'b0abb43d-6edc-471e-8d35-a9cf7b193b80', 'options': 'recursive,force'} 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '086c737fa75c22d521a3932c59b31c5e1d1c9ef4'. 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-09-12T02:48:57.538 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-12T02:48:57.539 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 086c737fa75 Merge pull request #59733 from zdover23/wip-doc-2024-09-11-backport-59635-to-squid 2024-09-12T02:48:57.543 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-12T02:48:57.570 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-12T02:48:57.572 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-12T02:48:57.572 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-12T02:48:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:57 smithi016 ceph-mon[30925]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:48:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:57 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:48:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:57 smithi033 ceph-mon[35092]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:48:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:57 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:48:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:57 smithi184 ceph-mon[35321]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-12T02:48:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:57 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:48:58.289 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-12T02:48:58.381 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-12T02:48:58.423 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-12T02:48:58.424 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-12T02:48:58.425 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-12T02:48:58.465 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-12T02:48:58.468 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T02:48:58.468 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-12T02:48:58.525 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-12T02:48:58.526 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-12T02:48:58.526 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=086c737fa75c22d521a3932c59b31c5e1d1c9ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-12T02:48:58.588 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 1 2024-09-12T02:48:58.597 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-12T02:48:58.597 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T02:48:58.597 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-12T02:48:58.604 INFO:tasks.workunit.client.0.smithi016.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-12T02:48:58.704 INFO:tasks.workunit.client.0.smithi016.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-12T02:48:58.705 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-09-12T02:48:59.704 INFO:tasks.workunit.client.0.smithi016.stdout: 1 3501 38.34 MB/sec warmup 1 sec latency 6.933 ms 2024-09-12T02:48:59.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:48:59 smithi016 ceph-mon[30925]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-12T02:48:59.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:48:59 smithi033 ceph-mon[35092]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-12T02:49:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:48:59 smithi184 ceph-mon[35321]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-12T02:49:00.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 6537 26.32 MB/sec warmup 2 sec latency 10.353 ms 2024-09-12T02:49:01.704 INFO:tasks.workunit.client.0.smithi016.stdout: 1 9544 22.11 MB/sec warmup 3 sec latency 9.918 ms 2024-09-12T02:49:01.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:01 smithi016 ceph-mon[30925]: pgmap v370: 97 pgs: 97 active+clean; 956 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-09-12T02:49:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:01 smithi016 ceph-mon[30925]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-12T02:49:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:01 smithi033 ceph-mon[35092]: pgmap v370: 97 pgs: 97 active+clean; 956 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-09-12T02:49:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:01 smithi033 ceph-mon[35092]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-12T02:49:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:01 smithi184 ceph-mon[35321]: pgmap v370: 97 pgs: 97 active+clean; 956 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-09-12T02:49:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:01 smithi184 ceph-mon[35321]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-12T02:49:02.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 12046 20.84 MB/sec warmup 4 sec latency 31.350 ms 2024-09-12T02:49:03.704 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15185 19.68 MB/sec warmup 5 sec latency 10.759 ms 2024-09-12T02:49:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:03 smithi016 ceph-mon[30925]: pgmap v371: 97 pgs: 97 active+clean; 1.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 81 KiB/s wr, 13 op/s 2024-09-12T02:49:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:03 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:03 smithi033 ceph-mon[35092]: pgmap v371: 97 pgs: 97 active+clean; 1.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 81 KiB/s wr, 13 op/s 2024-09-12T02:49:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:03 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:03 smithi184 ceph-mon[35321]: pgmap v371: 97 pgs: 97 active+clean; 1.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 81 KiB/s wr, 13 op/s 2024-09-12T02:49:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:03 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:04.704 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15379 16.58 MB/sec warmup 6 sec latency 946.508 ms 2024-09-12T02:49:05.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15379 14.21 MB/sec warmup 7 sec latency 1946.628 ms 2024-09-12T02:49:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:05 smithi016 ceph-mon[30925]: pgmap v372: 97 pgs: 97 active+clean; 5.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 485 KiB/s wr, 75 op/s 2024-09-12T02:49:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:05 smithi033 ceph-mon[35092]: pgmap v372: 97 pgs: 97 active+clean; 5.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 485 KiB/s wr, 75 op/s 2024-09-12T02:49:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:05 smithi184 ceph-mon[35321]: pgmap v372: 97 pgs: 97 active+clean; 5.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 485 KiB/s wr, 75 op/s 2024-09-12T02:49:06.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15379 12.44 MB/sec warmup 8 sec latency 2946.722 ms 2024-09-12T02:49:07.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15379 11.05 MB/sec warmup 9 sec latency 3946.771 ms 2024-09-12T02:49:07.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:07 smithi033 ceph-mon[35092]: pgmap v373: 97 pgs: 97 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 736 KiB/s wr, 100 op/s 2024-09-12T02:49:08.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:07 smithi184 ceph-mon[35321]: pgmap v373: 97 pgs: 97 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 736 KiB/s wr, 100 op/s 2024-09-12T02:49:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:07 smithi016 ceph-mon[30925]: pgmap v373: 97 pgs: 97 active+clean; 7.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 736 KiB/s wr, 100 op/s 2024-09-12T02:49:08.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 15379 9.95 MB/sec warmup 10 sec latency 4946.860 ms 2024-09-12T02:49:08.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:08 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:09.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:08 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:08 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:49:09.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 18420 10.61 MB/sec warmup 11 sec latency 4964.861 ms 2024-09-12T02:49:09.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:09 smithi033 ceph-mon[35092]: pgmap v374: 97 pgs: 97 active+clean; 8.0 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 784 KiB/s wr, 100 op/s 2024-09-12T02:49:10.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:09 smithi184 ceph-mon[35321]: pgmap v374: 97 pgs: 97 active+clean; 8.0 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 784 KiB/s wr, 100 op/s 2024-09-12T02:49:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:09 smithi016 ceph-mon[30925]: pgmap v374: 97 pgs: 97 active+clean; 8.0 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 784 KiB/s wr, 100 op/s 2024-09-12T02:49:10.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21774 11.31 MB/sec warmup 12 sec latency 22.648 ms 2024-09-12T02:49:11.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22508 10.60 MB/sec warmup 13 sec latency 679.692 ms 2024-09-12T02:49:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:11 smithi033 ceph-mon[35092]: pgmap v375: 97 pgs: 97 active+clean; 8.1 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 809 KiB/s wr, 106 op/s 2024-09-12T02:49:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:11 smithi184 ceph-mon[35321]: pgmap v375: 97 pgs: 97 active+clean; 8.1 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 809 KiB/s wr, 106 op/s 2024-09-12T02:49:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:11 smithi016 ceph-mon[30925]: pgmap v375: 97 pgs: 97 active+clean; 8.1 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 809 KiB/s wr, 106 op/s 2024-09-12T02:49:12.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22508 9.84 MB/sec warmup 14 sec latency 1679.784 ms 2024-09-12T02:49:13.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22508 9.19 MB/sec warmup 15 sec latency 2679.907 ms 2024-09-12T02:49:13.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:13 smithi033 ceph-mon[35092]: pgmap v376: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-09-12T02:49:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:13 smithi184 ceph-mon[35321]: pgmap v376: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-09-12T02:49:14.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:13 smithi016 ceph-mon[30925]: pgmap v376: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-09-12T02:49:14.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25539 9.68 MB/sec warmup 16 sec latency 2694.662 ms 2024-09-12T02:49:15.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 28861 10.23 MB/sec warmup 17 sec latency 17.740 ms 2024-09-12T02:49:15.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:15 smithi033 ceph-mon[35092]: pgmap v377: 97 pgs: 97 active+clean; 11 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1016 KiB/s wr, 138 op/s 2024-09-12T02:49:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:15 smithi184 ceph-mon[35321]: pgmap v377: 97 pgs: 97 active+clean; 11 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1016 KiB/s wr, 138 op/s 2024-09-12T02:49:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:15 smithi016 ceph-mon[30925]: pgmap v377: 97 pgs: 97 active+clean; 11 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1016 KiB/s wr, 138 op/s 2024-09-12T02:49:16.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 32103 10.65 MB/sec warmup 18 sec latency 20.253 ms 2024-09-12T02:49:17.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 34969 10.82 MB/sec warmup 19 sec latency 11.286 ms 2024-09-12T02:49:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:17 smithi033 ceph-mon[35092]: pgmap v378: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 777 KiB/s wr, 101 op/s 2024-09-12T02:49:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:17 smithi184 ceph-mon[35321]: pgmap v378: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 777 KiB/s wr, 101 op/s 2024-09-12T02:49:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:17 smithi016 ceph-mon[30925]: pgmap v378: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 777 KiB/s wr, 101 op/s 2024-09-12T02:49:18.705 INFO:tasks.workunit.client.0.smithi016.stdout: 1 37998 11.00 MB/sec warmup 20 sec latency 10.652 ms 2024-09-12T02:49:19.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 40949 11.27 MB/sec warmup 21 sec latency 11.631 ms 2024-09-12T02:49:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:19 smithi033 ceph-mon[35092]: pgmap v379: 97 pgs: 97 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 715 KiB/s wr, 97 op/s 2024-09-12T02:49:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:19 smithi184 ceph-mon[35321]: pgmap v379: 97 pgs: 97 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 715 KiB/s wr, 97 op/s 2024-09-12T02:49:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:19 smithi016 ceph-mon[30925]: pgmap v379: 97 pgs: 97 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 715 KiB/s wr, 97 op/s 2024-09-12T02:49:20.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 44003 11.61 MB/sec warmup 22 sec latency 15.478 ms 2024-09-12T02:49:21.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 47126 11.76 MB/sec warmup 23 sec latency 11.443 ms 2024-09-12T02:49:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:21 smithi033 ceph-mon[35092]: pgmap v380: 97 pgs: 97 active+clean; 16 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 876 KiB/s wr, 113 op/s 2024-09-12T02:49:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:21 smithi184 ceph-mon[35321]: pgmap v380: 97 pgs: 97 active+clean; 16 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 876 KiB/s wr, 113 op/s 2024-09-12T02:49:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:21 smithi016 ceph-mon[30925]: pgmap v380: 97 pgs: 97 active+clean; 16 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 876 KiB/s wr, 113 op/s 2024-09-12T02:49:22.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 50478 12.05 MB/sec warmup 24 sec latency 9.987 ms 2024-09-12T02:49:23.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 53775 12.31 MB/sec warmup 25 sec latency 14.079 ms 2024-09-12T02:49:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:23 smithi033 ceph-mon[35092]: pgmap v381: 97 pgs: 97 active+clean; 20 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-09-12T02:49:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:23 smithi184 ceph-mon[35321]: pgmap v381: 97 pgs: 97 active+clean; 20 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-09-12T02:49:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:23 smithi016 ceph-mon[30925]: pgmap v381: 97 pgs: 97 active+clean; 20 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-09-12T02:49:24.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57104 12.56 MB/sec warmup 26 sec latency 13.924 ms 2024-09-12T02:49:25.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 60557 12.80 MB/sec warmup 27 sec latency 9.669 ms 2024-09-12T02:49:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:25 smithi033 ceph-mon[35092]: pgmap v382: 97 pgs: 97 active+clean; 20 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-09-12T02:49:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:25 smithi184 ceph-mon[35321]: pgmap v382: 97 pgs: 97 active+clean; 20 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-09-12T02:49:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:25 smithi016 ceph-mon[30925]: pgmap v382: 97 pgs: 97 active+clean; 20 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-09-12T02:49:26.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 63622 12.87 MB/sec warmup 28 sec latency 12.691 ms 2024-09-12T02:49:27.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:49:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:27.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 66630 12.97 MB/sec warmup 29 sec latency 13.382 ms 2024-09-12T02:49:27.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:27 smithi016 ceph-mon[30925]: pgmap v383: 97 pgs: 97 active+clean; 25 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-09-12T02:49:27.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:27 smithi033 ceph-mon[35092]: pgmap v383: 97 pgs: 97 active+clean; 25 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-09-12T02:49:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 -- 172.21.15.16:0/913471162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 msgr2=0x7f2e3c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/913471162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c1029b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009900 tx=0x7f2e2c02a480 comp rx=0 tx=0).stop 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 -- 172.21.15.16:0/913471162 shutdown_connections 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/913471162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c102ef0 0x7f2e3c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/913471162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c1029b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/913471162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c100e20 0x7f2e3c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 -- 172.21.15.16:0/913471162 >> 172.21.15.16:0/913471162 conn(0x7f2e3c0fc7d0 msgr2=0x7f2e3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 -- 172.21.15.16:0/913471162 shutdown_connections 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.996+0000 7f2e43ee3640 1 -- 172.21.15.16:0/913471162 wait complete. 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 Processor -- start 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 -- start start 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 0x7f2e3c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c102ef0 0x7f2e3c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c110f80 con 0x7f2e3c102570 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c110e00 con 0x7f2e3c100e20 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e43ee3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c111100 con 0x7f2e3c102ef0 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e4147e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 0x7f2e3c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e4147e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46696/0 (socket says 172.21.15.16:46696) 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e4147e640 1 -- 172.21.15.16:0/4082054563 learned_addr learned my addr 172.21.15.16:0/4082054563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.997+0000 7f2e42480640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c102ef0 0x7f2e3c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 -- 172.21.15.16:0/4082054563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c102ef0 msgr2=0x7f2e3c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c102ef0 0x7f2e3c19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 -- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 msgr2=0x7f2e3c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 -- 172.21.15.16:0/4082054563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3c19d480 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 0x7f2e3c1984d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800e9f0 tx=0x7f2e3800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e32ffd640 1 -- 172.21.15.16:0/4082054563 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800ccc0 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e32ffd640 1 -- 172.21.15.16:0/4082054563 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e38005020 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3c19d770 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e32ffd640 1 -- 172.21.15.16:0/4082054563 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380049e0 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:27.998+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3c1a5050 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.000+0000 7f2e32ffd640 1 -- 172.21.15.16:0/4082054563 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2e38005560 con 0x7f2e3c100e20 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 Processor -- start 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 -- start start 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212aba880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558212a202e0 con 0x558212ac5680 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5582129c7ce0 con 0x558212abadc0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e462bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558212a1c2a0 con 0x558212aba4a0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.001+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e4147e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:48084/0 (socket says 172.21.15.16:48084) 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3468470768 learned_addr learned my addr 172.21.15.16:0/3468470768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e32ffd640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 0x7f2e280a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e42480640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212aba880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e42480640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 0x7f2e280a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e32ffd640 1 -- 172.21.15.16:0/4082054563 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2e380cefa0 con 0x7f2e3c100e20 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3468470768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 msgr2=0x558212aba880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212aba880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3468470768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 msgr2=0x558212ac7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3468470768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582129ac440 con 0x558212abadc0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e4147e640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.002+0000 7f2e42480640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212aba880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.003+0000 7f2e42480640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 0x7f2e280a5040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e34005c40 tx=0x7f2e34004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.003+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e38097c70 tx=0x7f2e380054d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=391c15b0e138960b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.003+0000 7f2e27fff640 1 -- 172.21.15.16:0/3468470768 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3801c070 con 0x558212abadc0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.003+0000 7f2e27fff640 1 -- 172.21.15.16:0/3468470768 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380051d0 con 0x558212abadc0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.003+0000 7f2e27fff640 1 -- 172.21.15.16:0/3468470768 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380860e0 con 0x558212abadc0 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 -- 172.21.15.16:0/3468470768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 msgr2=0x558212ac5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e38097c70 tx=0x7f2e380054d0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 -- 172.21.15.16:0/3468470768 shutdown_connections 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac5140 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3468470768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212aba880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 -- 172.21.15.16:0/3468470768 >> 172.21.15.16:0/3468470768 conn(0x558212ab6a30 msgr2=0x558212ab6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 -- 172.21.15.16:0/3468470768 shutdown_connections 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.004+0000 7f2e462bc740 1 -- 172.21.15.16:0/3468470768 wait complete. 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 -- start start 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 Processor -- start 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558212acc660 con 0x558212ac5680 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5582129db050 con 0x558212abadc0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558212acc7e0 con 0x558212aba4a0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e462bc740 10 client.0 _ll_register_callbacks cb 0x558212a1b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e4147e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.005+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33946/0 (socket says 172.21.15.16:33946) 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e41c7f640 1 -- 172.21.15.16:0/1337111636 learned_addr learned my addr 172.21.15.16:0/1337111636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e42480640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 -- 172.21.15.16:0/1337111636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 msgr2=0x558212ac5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 -- 172.21.15.16:0/1337111636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 msgr2=0x558212ac2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 -- 172.21.15.16:0/1337111636 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582127913c0 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e42480640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e4147e640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac22e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009230 tx=0x7f2e2c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac09571402452570 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e257fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c035040 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2e2c0037f0 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.006+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e380051d0 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2e2c0044e0 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 10 client.25388 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558212ad8050 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380051d0 con 0x558212abadc0 2024-09-12T02:49:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2e2c045470 con 0x558212abadc0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 10 client.25388 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 10 client.25388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 20 client.25388 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212ad95b0 0x558212adb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 4 client.25388 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.007+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558212a2d850 con 0x558212ad95b0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.008+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212ad95b0 0x558212adb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.008+0000 7f2e257fa640 10 client.25388 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.008+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212ad95b0 0x558212adb9e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f17bab8816ac127 server_cookie=fc525777929385a5 in_seq=0 out_seq=0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e257fa640 1 -- 172.21.15.16:0/1337111636 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558212a2d850 con 0x558212ad95b0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e257fa640 10 client.25388 handle_command_reply: tid=0 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 client.25388 shutdown 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 msgr2=0x558212ac22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac22e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009230 tx=0x7f2e2c0042f0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 shutdown_connections 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212ad95b0 0x558212adb9e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ac2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212abadc0 0x558212ac22e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 --2- 172.21.15.16:0/1337111636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ac5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e257fa640 0 client.25388 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 >> 172.21.15.16:0/1337111636 conn(0x558212ab6a30 msgr2=0x558212abc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 shutdown_connections 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 1 -- 172.21.15.16:0/1337111636 wait complete. 2024-09-12T02:49:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.009+0000 7f2e462bc740 20 client.25388 trim_cache size 0 max 16384 2024-09-12T02:49:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.027+0000 7f2e462bc740 1 Processor -- start 2024-09-12T02:49:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.027+0000 7f2e462bc740 1 -- start start 2024-09-12T02:49:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212abadc0 0x558212ae1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ae17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1ab370 con 0x558212ac5680 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1acf70 con 0x558212aba4a0 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e462bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1aba40 con 0x558212abadc0 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e42480640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ae17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:48102/0 (socket says 172.21.15.16:48102) 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 -- 172.21.15.16:0/2092625684 learned_addr learned my addr 172.21.15.16:0/2092625684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e42480640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ae17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46724/0 (socket says 172.21.15.16:46724) 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e4147e640 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212abadc0 0x558212ae1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 -- 172.21.15.16:0/2092625684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212abadc0 msgr2=0x558212ae1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212abadc0 0x558212ae1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 -- 172.21.15.16:0/2092625684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 msgr2=0x558212ae17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ae17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 -- 172.21.15.16:0/2092625684 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558212ac53a0 con 0x558212aba4a0 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e3809b2e0 tx=0x7f2e3809b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4ad565b6d6fe5d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.028+0000 7f2e277fe640 1 -- 172.21.15.16:0/2092625684 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3801c040 con 0x558212aba4a0 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e277fe640 1 -- 172.21.15.16:0/2092625684 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380a4480 con 0x558212aba4a0 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e277fe640 1 -- 172.21.15.16:0/2092625684 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3809ba70 con 0x558212aba4a0 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 -- 172.21.15.16:0/2092625684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 msgr2=0x558212ad58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e3809b2e0 tx=0x7f2e3809b690 comp rx=0 tx=0).stop 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 -- 172.21.15.16:0/2092625684 shutdown_connections 2024-09-12T02:49:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212ac5680 0x558212ae17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212abadc0 0x558212ae1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 --2- 172.21.15.16:0/2092625684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212aba4a0 0x558212ad58c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 -- 172.21.15.16:0/2092625684 >> 172.21.15.16:0/2092625684 conn(0x558212ab6a30 msgr2=0x558212ac13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 -- 172.21.15.16:0/2092625684 shutdown_connections 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.029+0000 7f2e462bc740 1 -- 172.21.15.16:0/2092625684 wait complete. 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 -- start start 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 Processor -- start 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 0x558212acf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 0x558212acfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1ac7e0 con 0x558212abadc0 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1ac110 con 0x558212ac5680 2024-09-12T02:49:28.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.030+0000 7f2e462bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1a9350 con 0x558212aba4a0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e462bc740 10 client.0 _ll_register_callbacks cb 0x7f2e38005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e4147e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 0x558212acf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33960/0 (socket says 172.21.15.16:33960) 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3018991756 learned_addr learned my addr 172.21.15.16:0/3018991756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e42480640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 0x558212acfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 msgr2=0x558212acfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 0x558212acfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3018991756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 msgr2=0x558212acf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 0x558212acf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 -- 172.21.15.16:0/3018991756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558212ac2bf0 con 0x558212aba4a0 2024-09-12T02:49:28.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e42480640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 0x558212acfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e4147e640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 0x558212acf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e41c7f640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e38005480 tx=0x7f2e380096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8082f24027d564c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e257fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.031+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380b9060 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2e38086150 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e380a4480 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2e380b4070 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 10 client.25288 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558212b441d0 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380a4480 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2e3809ba70 con 0x558212aba4a0 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 10 client.25288 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:28.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 10 client.25288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 20 client.25288 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212b44400 0x558212b467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 4 client.25288 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558212a27d60 con 0x558212b44400 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.032+0000 7f2e4147e640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212b44400 0x558212b467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e4147e640 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212b44400 0x558212b467f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fe824f6840576c8 server_cookie=4df2cf71c858a382 in_seq=0 out_seq=0 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e257fa640 10 client.25288 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e257fa640 1 -- 172.21.15.16:0/3018991756 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558212a27d60 con 0x558212b44400 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e257fa640 10 client.25288 handle_command_reply: tid=0 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 client.25288 shutdown 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 msgr2=0x558212ad22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e38005480 tx=0x7f2e380096e0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 shutdown_connections 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558212b44400 0x558212b467f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558212ac5680 0x558212acfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e257fa640 0 client.25288 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558212abadc0 0x558212acf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 --2- 172.21.15.16:0/3018991756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558212aba4a0 0x558212ad22d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.033+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 >> 172.21.15.16:0/3018991756 conn(0x558212ab6a30 msgr2=0x558212ac3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 shutdown_connections 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e462bc740 1 -- 172.21.15.16:0/3018991756 wait complete. 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e462bc740 20 client.25288 trim_cache size 0 max 16384 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:49:28.040 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 msgr2=0x7f2e280a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 0x7f2e280a5040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e34005c40 tx=0x7f2e34004ab0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 msgr2=0x7f2e3c1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 0x7f2e3c1984d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800e9f0 tx=0x7f2e3800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 shutdown_connections 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e3c102ef0 0x7f2e3c19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2e280a2b80 0x7f2e280a5040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c102570 0x7f2e3c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 --2- 172.21.15.16:0/4082054563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e3c100e20 0x7f2e3c1984d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 >> 172.21.15.16:0/4082054563 conn(0x7f2e3c0fc7d0 msgr2=0x7f2e3c06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.034+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 shutdown_connections 2024-09-12T02:49:28.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:28.035+0000 7f2e43ee3640 1 -- 172.21.15.16:0/4082054563 wait complete. 2024-09-12T02:49:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:27 smithi184 ceph-mon[35321]: pgmap v383: 97 pgs: 97 active+clean; 25 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-09-12T02:49:28.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 69809 13.04 MB/sec warmup 30 sec latency 23.549 ms 2024-09-12T02:49:28.969 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:49:28.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:49:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:29.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 72113 13.08 MB/sec warmup 31 sec latency 22.447 ms 2024-09-12T02:49:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 -- 172.21.15.16:0/3593335311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5654103720 msgr2=0x7f565410bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 --2- 172.21.15.16:0/3593335311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5654103720 0x7f565410bc10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f563c009900 tx=0x7f563c02a480 comp rx=0 tx=0).stop 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 -- 172.21.15.16:0/3593335311 shutdown_connections 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 --2- 172.21.15.16:0/3593335311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410c1a0 0x7f565410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 --2- 172.21.15.16:0/3593335311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5654103720 0x7f565410bc10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 --2- 172.21.15.16:0/3593335311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654102e00 0x7f56541031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 -- 172.21.15.16:0/3593335311 >> 172.21.15.16:0/3593335311 conn(0x7f56540fc7a0 msgr2=0x7f56540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.761+0000 7f565a88f640 1 -- 172.21.15.16:0/3593335311 shutdown_connections 2024-09-12T02:49:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 -- 172.21.15.16:0/3593335311 wait complete. 2024-09-12T02:49:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 Processor -- start 2024-09-12T02:49:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 -- start start 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 0x7f56541983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654103720 0x7f5654198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5654111f70 con 0x7f565410c1a0 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5654111df0 con 0x7f5654102e00 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.762+0000 7f565a88f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56541120f0 con 0x7f5654103720 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46752/0 (socket says 172.21.15.16:46752) 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f564bfff640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654103720 0x7f5654198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 -- 172.21.15.16:0/2445481826 learned_addr learned my addr 172.21.15.16:0/2445481826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5653fff640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 0x7f56541983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 -- 172.21.15.16:0/2445481826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654103720 msgr2=0x7f5654198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654103720 0x7f5654198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 -- 172.21.15.16:0/2445481826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 msgr2=0x7f56541983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 0x7f56541983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 -- 172.21.15.16:0/2445481826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565419d390 con 0x7f565410c1a0 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5653fff640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 0x7f56541983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5658e2c640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f564c00eb10 tx=0x7f564c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c004b20 con 0x7f565410c1a0 2024-09-12T02:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f564c004cc0 con 0x7f565410c1a0 2024-09-12T02:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565419d680 con 0x7f565410c1a0 2024-09-12T02:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c010760 con 0x7f565410c1a0 2024-09-12T02:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.763+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56541a4f60 con 0x7f565410c1a0 2024-09-12T02:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.764+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56541031e0 con 0x7f565410c1a0 2024-09-12T02:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.765+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f564c005020 con 0x7f565410c1a0 2024-09-12T02:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.766+0000 7f5651ffb640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 0x7f56400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.767+0000 7f5653fff640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 0x7f56400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.767+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f564c0cf520 con 0x7f565410c1a0 2024-09-12T02:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.767+0000 7f5653fff640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 0x7f56400a50c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f564400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:29.771+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f564c09c430 con 0x7f565410c1a0 2024-09-12T02:49:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:29 smithi016 ceph-mon[30925]: pgmap v384: 97 pgs: 97 active+clean; 26 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-12T02:49:29.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:29 smithi033 ceph-mon[35092]: pgmap v384: 97 pgs: 97 active+clean; 26 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-12T02:49:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.025+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f565402d7c0 con 0x7f565410c1a0 2024-09-12T02:49:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.026+0000 7f5651ffb640 1 -- 172.21.15.16:0/2445481826 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7f564c09bd30 con 0x7f565410c1a0 2024-09-12T02:49:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:49:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:49:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:49:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 msgr2=0x7f56400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 0x7f56400a50c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f564400b390 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 msgr2=0x7f565419ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f564c00eb10 tx=0x7f564c00c2d0 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 shutdown_connections 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f56400a2c00 0x7f56400a50c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f565410c1a0 0x7f565419ccb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5654103720 0x7f5654198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 --2- 172.21.15.16:0/2445481826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102e00 0x7f56541983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 >> 172.21.15.16:0/2445481826 conn(0x7f56540fc7a0 msgr2=0x7f56540fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 shutdown_connections 2024-09-12T02:49:30.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:30.028+0000 7f565a88f640 1 -- 172.21.15.16:0/2445481826 wait complete. 2024-09-12T02:49:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:29 smithi184 ceph-mon[35321]: pgmap v384: 97 pgs: 97 active+clean; 26 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.5 MiB/s wr, 186 op/s 2024-09-12T02:49:30.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:49:30.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 75158 13.21 MB/sec warmup 32 sec latency 22.755 ms 2024-09-12T02:49:30.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2445481826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:30.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:30 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2445481826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:31.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:30 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2445481826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:31.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 78453 13.38 MB/sec warmup 33 sec latency 29.452 ms 2024-09-12T02:49:31.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:31 smithi033 ceph-mon[35092]: pgmap v385: 97 pgs: 97 active+clean; 34 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-09-12T02:49:32.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:31 smithi184 ceph-mon[35321]: pgmap v385: 97 pgs: 97 active+clean; 34 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-09-12T02:49:32.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:31 smithi016 ceph-mon[30925]: pgmap v385: 97 pgs: 97 active+clean; 34 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-09-12T02:49:32.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 81704 13.51 MB/sec warmup 34 sec latency 10.395 ms 2024-09-12T02:49:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 -- 172.21.15.16:0/2383570092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94106f10 msgr2=0x7f2a9410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/2383570092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94106f10 0x7f2a9410ea90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009ca0 tx=0x7f2a8c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 -- 172.21.15.16:0/2383570092 shutdown_connections 2024-09-12T02:49:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/2383570092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94106f10 0x7f2a9410ea90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/2383570092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94102790 0x7f2a941067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/2383570092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94101de0 0x7f2a941021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 -- 172.21.15.16:0/2383570092 >> 172.21.15.16:0/2383570092 conn(0x7f2a940fd8c0 msgr2=0x7f2a940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 -- 172.21.15.16:0/2383570092 shutdown_connections 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.106+0000 7f2a9a37d640 1 -- 172.21.15.16:0/2383570092 wait complete. 2024-09-12T02:49:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.107+0000 7f2a9a37d640 1 Processor -- start 2024-09-12T02:49:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.107+0000 7f2a9a37d640 1 -- start start 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94101de0 0x7f2a941983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94106f10 0x7f2a9419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94111f70 con 0x7f2a94102790 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94111df0 con 0x7f2a94106f10 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9a37d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941120f0 con 0x7f2a94101de0 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94101de0 0x7f2a941983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46762/0 (socket says 172.21.15.16:46762) 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 -- 172.21.15.16:0/3783072661 learned_addr learned my addr 172.21.15.16:0/3783072661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a9891a640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94106f10 0x7f2a9419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 -- 172.21.15.16:0/3783072661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94101de0 msgr2=0x7f2a941983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94101de0 0x7f2a941983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 -- 172.21.15.16:0/3783072661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94106f10 msgr2=0x7f2a9419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94106f10 0x7f2a9419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.108+0000 7f2a937fe640 1 -- 172.21.15.16:0/3783072661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9419d360 con 0x7f2a94102790 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e9f0 tx=0x7f2a8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a917fa640 1 -- 172.21.15.16:0/3783072661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800cd80 con 0x7f2a94102790 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a917fa640 1 -- 172.21.15.16:0/3783072661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a88005500 con 0x7f2a94102790 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9419d650 con 0x7f2a94102790 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a917fa640 1 -- 172.21.15.16:0/3783072661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88010690 con 0x7f2a94102790 2024-09-12T02:49:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.109+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a94074420 con 0x7f2a94102790 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 Processor -- start 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 -- start start 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a917fa640 1 -- 172.21.15.16:0/3783072661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2a88010830 con 0x7f2a94102790 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4bea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 0x55855f4cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55855f4242f0 con 0x55855f4c9370 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55855f3cbcf0 con 0x55855f4bfb40 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.112+0000 7f2a9c756740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55855f4202b0 con 0x55855f4be5d0 2024-09-12T02:49:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 0x55855f4cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:48146/0 (socket says 172.21.15.16:48146) 2024-09-12T02:49:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 -- 172.21.15.16:0/2591734085 learned_addr learned my addr 172.21.15.16:0/2591734085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 -- 172.21.15.16:0/2591734085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 msgr2=0x55855f4bea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4bea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 -- 172.21.15.16:0/2591734085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 msgr2=0x55855f4cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 0x55855f4cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 -- 172.21.15.16:0/2591734085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855f3b03a0 con 0x55855f4bfb40 2024-09-12T02:49:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 0x55855f4cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a93fff640 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4bea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a917fa640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 0x7f2a7c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a9891a640 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c005cc0 tx=0x7f2a8c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b06f8b6c14995363 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a93fff640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 0x7f2a7c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.113+0000 7f2a917fa640 1 -- 172.21.15.16:0/3783072661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2a880cf4d0 con 0x7f2a94102790 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a7a7fc640 1 -- 172.21.15.16:0/2591734085 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c031db0 con 0x55855f4bfb40 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a7a7fc640 1 -- 172.21.15.16:0/2591734085 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036050 con 0x55855f4bfb40 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a7a7fc640 1 -- 172.21.15.16:0/2591734085 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0042a0 con 0x55855f4bfb40 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 -- 172.21.15.16:0/2591734085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 msgr2=0x55855f4be090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a93fff640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 0x7f2a7c0a50c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55855f4bbac0 tx=0x7f2a84005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c005cc0 tx=0x7f2a8c005ef0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 -- 172.21.15.16:0/2591734085 shutdown_connections 2024-09-12T02:49:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4c9370 0x55855f4cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4bea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2591734085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4be090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.114+0000 7f2a9c756740 1 -- 172.21.15.16:0/2591734085 >> 172.21.15.16:0/2591734085 conn(0x55855f487310 msgr2=0x55855f4bac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- 172.21.15.16:0/2591734085 shutdown_connections 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- 172.21.15.16:0/2591734085 wait complete. 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- start start 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 Processor -- start 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4c6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55855f4cbe90 con 0x55855f4bfb40 2024-09-12T02:49:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55855f3df060 con 0x55855f4c9370 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.115+0000 7f2a9c756740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55855f4cc010 con 0x55855f4be5d0 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a9c756740 10 client.0 _ll_register_callbacks cb 0x55855f1328c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34016/0 (socket says 172.21.15.16:34016) 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 -- 172.21.15.16:0/3783436962 learned_addr learned my addr 172.21.15.16:0/3783436962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a9891a640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 -- 172.21.15.16:0/3783436962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 msgr2=0x55855f4c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 -- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 msgr2=0x55855f4c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 -- 172.21.15.16:0/3783436962 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855f1953c0 con 0x55855f4be5d0 2024-09-12T02:49:33.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a9891a640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a93fff640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a84020a10 tx=0x7f2a84004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924f3524d319ad51 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.116+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8402d060 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2a84018030 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a8c036050 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2a84009740 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 10 client.25306 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55855f4d79c0 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036050 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2a8401dc50 con 0x55855f4be5d0 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 10 client.25306 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 10 client.25306 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 20 client.25306 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f4db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 4 client.25306 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55855f431860 con 0x55855f4d8e80 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.117+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f4db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.118+0000 7f2a937fe640 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f4db270 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=34e864e6b6d01958 server_cookie=1bf265ab9e18c60a in_seq=0 out_seq=0 2024-09-12T02:49:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.118+0000 7f2a737fe640 10 client.25306 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:49:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.118+0000 7f2a737fe640 1 -- 172.21.15.16:0/3783436962 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55855f431860 con 0x55855f4d8e80 2024-09-12T02:49:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.118+0000 7f2a737fe640 10 client.25306 handle_command_reply: tid=0 2024-09-12T02:49:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 client.25306 shutdown 2024-09-12T02:49:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 msgr2=0x55855f4c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a84020a10 tx=0x7f2a84004730 comp rx=0 tx=0).stop 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 shutdown_connections 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f4db270 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a737fe640 0 client.25306 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 --2- 172.21.15.16:0/3783436962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4c8d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 >> 172.21.15.16:0/3783436962 conn(0x55855f487310 msgr2=0x55855f4c0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 shutdown_connections 2024-09-12T02:49:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 1 -- 172.21.15.16:0/3783436962 wait complete. 2024-09-12T02:49:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.119+0000 7f2a9c756740 20 client.25306 trim_cache size 0 max 16384 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.132+0000 7f2a9c756740 1 Processor -- start 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.132+0000 7f2a9c756740 1 -- start start 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4ba150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941af670 con 0x55855f4bfb40 2024-09-12T02:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941b1270 con 0x55855f4c9370 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9c756740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941afd40 con 0x55855f4be5d0 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4ba150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4b9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34026/0 (socket says 172.21.15.16:34026) 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a93fff640 1 -- 172.21.15.16:0/2931205965 learned_addr learned my addr 172.21.15.16:0/2931205965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a9891a640 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 -- 172.21.15.16:0/2931205965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 msgr2=0x55855f4b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4b9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 -- 172.21.15.16:0/2931205965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 msgr2=0x55855f4c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 -- 172.21.15.16:0/2931205965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855f4c69b0 con 0x55855f4bfb40 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.133+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4ba150 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2a880cd030 tx=0x7f2a880d5060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48210a5dc1dc69d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a79ffb640 1 -- 172.21.15.16:0/2931205965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88097fe0 con 0x55855f4bfb40 2024-09-12T02:49:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a79ffb640 1 -- 172.21.15.16:0/2931205965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8809c440 con 0x55855f4bfb40 2024-09-12T02:49:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a79ffb640 1 -- 172.21.15.16:0/2931205965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8809b6a0 con 0x55855f4bfb40 2024-09-12T02:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 -- 172.21.15.16:0/2931205965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 msgr2=0x55855f4ba150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4ba150 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2a880cd030 tx=0x7f2a880d5060 comp rx=0 tx=0).stop 2024-09-12T02:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 -- 172.21.15.16:0/2931205965 shutdown_connections 2024-09-12T02:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4c9370 0x55855f4c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4bfb40 0x55855f4ba150 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2931205965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4be5d0 0x55855f4b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.134+0000 7f2a9c756740 1 -- 172.21.15.16:0/2931205965 >> 172.21.15.16:0/2931205965 conn(0x55855f487310 msgr2=0x55855f4b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.135+0000 7f2a9c756740 1 -- 172.21.15.16:0/2931205965 shutdown_connections 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.135+0000 7f2a9c756740 1 -- 172.21.15.16:0/2931205965 wait complete. 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.135+0000 7f2a9c756740 1 -- start start 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.135+0000 7f2a9c756740 1 Processor -- start 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.135+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4c9370 0x55855f4cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941b0ae0 con 0x55855f4be5d0 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941b0410 con 0x55855f4bfb40 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941ad650 con 0x55855f4c9370 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9c756740 10 client.0 _ll_register_callbacks cb 0x7f2a84005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46796/0 (socket says 172.21.15.16:46796) 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 -- 172.21.15.16:0/2823062758 learned_addr learned my addr 172.21.15.16:0/2823062758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a9891a640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4c9370 0x55855f4cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 -- 172.21.15.16:0/2823062758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4c9370 msgr2=0x55855f4cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4c9370 0x55855f4cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 -- 172.21.15.16:0/2823062758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 msgr2=0x55855f4ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 -- 172.21.15.16:0/2823062758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855f4cda40 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a93fff640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2a840333d0 tx=0x7f2a84011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35865e6f3b55e84 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.136+0000 7f2a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84038070 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2a84033a50 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a8809c440 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2a840207d0 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a9c756740 10 client.15786 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55855f543b00 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8809c440 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.137+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2a84034ac0 con 0x55855f4be5d0 2024-09-12T02:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 10 client.15786 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 10 client.15786 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 20 client.15786 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f546260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 4 client.15786 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55855f42bd70 con 0x55855f4d8e80 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f546260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a737fe640 10 client.15786 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a937fe640 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f546260 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=54ab0f1f02309968 server_cookie=357bc46dcb5485 in_seq=0 out_seq=0 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a737fe640 1 -- 172.21.15.16:0/2823062758 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55855f42bd70 con 0x55855f4d8e80 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a737fe640 10 client.15786 handle_command_reply: tid=0 2024-09-12T02:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.138+0000 7f2a9c756740 1 client.15786 shutdown 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 msgr2=0x55855f4d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2a840333d0 tx=0x7f2a84011e00 comp rx=0 tx=0).stop 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 shutdown_connections 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55855f4c9370 0x55855f4cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55855f4d8e80 0x55855f546260 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55855f4bfb40 0x55855f4ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 --2- 172.21.15.16:0/2823062758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55855f4be5d0 0x55855f4d19f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 >> 172.21.15.16:0/2823062758 conn(0x55855f487310 msgr2=0x55855f4b9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a737fe640 0 client.15786 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 shutdown_connections 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 1 -- 172.21.15.16:0/2823062758 wait complete. 2024-09-12T02:49:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.139+0000 7f2a9c756740 20 client.15786 trim_cache size 0 max 16384 2024-09-12T02:49:33.147 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:49:33.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 msgr2=0x7f2a7c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 0x7f2a7c0a50c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55855f4bbac0 tx=0x7f2a84005f90 comp rx=0 tx=0).stop 2024-09-12T02:49:33.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 msgr2=0x7f2a941988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:33.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e9f0 tx=0x7f2a8800edf0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 shutdown_connections 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a94106f10 0x7f2a9419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2a7c0a2c00 0x7f2a7c0a50c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a94102790 0x7f2a941988f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 --2- 172.21.15.16:0/3783072661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94101de0 0x7f2a941983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 >> 172.21.15.16:0/3783072661 conn(0x7f2a940fd8c0 msgr2=0x7f2a940fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 shutdown_connections 2024-09-12T02:49:33.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:33.140+0000 7f2a9a37d640 1 -- 172.21.15.16:0/3783072661 wait complete. 2024-09-12T02:49:33.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 84554 13.52 MB/sec warmup 35 sec latency 13.289 ms 2024-09-12T02:49:33.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:33 smithi016 ceph-mon[30925]: pgmap v386: 97 pgs: 97 active+clean; 44 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 237 op/s 2024-09-12T02:49:33.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:49:33.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:33 smithi033 ceph-mon[35092]: pgmap v386: 97 pgs: 97 active+clean; 44 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 237 op/s 2024-09-12T02:49:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:33 smithi184 ceph-mon[35321]: pgmap v386: 97 pgs: 97 active+clean; 44 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 237 op/s 2024-09-12T02:49:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:34.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.634+0000 7fc55a7c8640 1 -- 172.21.15.16:0/1682707971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554106f10 msgr2=0x7fc55410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/1682707971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554106f10 0x7fc55410ea90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009ca0 tx=0x7fc54c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 -- 172.21.15.16:0/1682707971 shutdown_connections 2024-09-12T02:49:34.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/1682707971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554106f10 0x7fc55410ea90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/1682707971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554102790 0x7fc5541067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/1682707971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554101de0 0x7fc5541021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 -- 172.21.15.16:0/1682707971 >> 172.21.15.16:0/1682707971 conn(0x7fc5540fd8c0 msgr2=0x7fc5540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 -- 172.21.15.16:0/1682707971 shutdown_connections 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 -- 172.21.15.16:0/1682707971 wait complete. 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 Processor -- start 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.635+0000 7fc55a7c8640 1 -- start start 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 0x7fc554198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 0x7fc55419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc554111f70 con 0x7fc554101de0 2024-09-12T02:49:34.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc554111df0 con 0x7fc554106f10 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc55a7c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541120f0 con 0x7fc554102790 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46804/0 (socket says 172.21.15.16:46804) 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc553fff640 1 -- 172.21.15.16:0/544442504 learned_addr learned my addr 172.21.15.16:0/544442504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 0x7fc554198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc558d65640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 0x7fc55419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 -- 172.21.15.16:0/544442504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 msgr2=0x7fc55419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 0x7fc55419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 -- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 msgr2=0x7fc554198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 -- 172.21.15.16:0/544442504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55419d2f0 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc558d65640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 0x7fc55419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc553fff640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.636+0000 7fc54bfff640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 0x7fc554198880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c970 tx=0x7fc53c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.637+0000 7fc551ffb640 1 -- 172.21.15.16:0/544442504 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c007bb0 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.637+0000 7fc551ffb640 1 -- 172.21.15.16:0/544442504 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc53c007d50 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.637+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55419d5e0 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.637+0000 7fc551ffb640 1 -- 172.21.15.16:0/544442504 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c018820 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.637+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5541a4ec0 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.639+0000 7fc551ffb640 1 -- 172.21.15.16:0/544442504 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc53c0189c0 con 0x7fc554102790 2024-09-12T02:49:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.639+0000 7fc55cba1740 1 Processor -- start 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.639+0000 7fc55cba1740 1 -- start start 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e84aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e855370 0x55965e857760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55965e7b02f0 con 0x55965e84bb40 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55965e757cf0 con 0x55965e855370 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc55cba1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55965e7ac2b0 con 0x55965e84a5d0 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc54bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e855370 0x55965e857760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc553fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e84aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46816/0 (socket says 172.21.15.16:46816) 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc558d65640 1 -- 172.21.15.16:0/1922729544 learned_addr learned my addr 172.21.15.16:0/1922729544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.640+0000 7fc551ffb640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 0x7fc5400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 -- 172.21.15.16:0/1922729544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 msgr2=0x55965e84aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e84aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 -- 172.21.15.16:0/1922729544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e855370 msgr2=0x55965e857760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e855370 0x55965e857760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc551ffb640 1 -- 172.21.15.16:0/544442504 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc53c0ceff0 con 0x7fc554102790 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 -- 172.21.15.16:0/1922729544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965e73c3a0 con 0x55965e84bb40 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc553fff640 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e84aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 0x7fc5400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc558d65640 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc54c004af0 tx=0x7fc54c0098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cd223023cea5c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc54a7fc640 1 -- 172.21.15.16:0/1922729544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c031510 con 0x55965e84bb40 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc54a7fc640 1 -- 172.21.15.16:0/1922729544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54c036070 con 0x55965e84bb40 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.641+0000 7fc54a7fc640 1 -- 172.21.15.16:0/1922729544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c02f610 con 0x55965e84bb40 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 -- 172.21.15.16:0/1922729544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 msgr2=0x55965e84a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc558d65640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 0x7fc5400a50c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc54c04b000 tx=0x7fc54c048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc54c004af0 tx=0x7fc54c0098d0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 -- 172.21.15.16:0/1922729544 shutdown_connections 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e855370 0x55965e857760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e84aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1922729544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e84a090 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 -- 172.21.15.16:0/1922729544 >> 172.21.15.16:0/1922729544 conn(0x55965e813310 msgr2=0x55965e846c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 -- 172.21.15.16:0/1922729544 shutdown_connections 2024-09-12T02:49:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.642+0000 7fc55cba1740 1 -- 172.21.15.16:0/1922729544 wait complete. 2024-09-12T02:49:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 -- start start 2024-09-12T02:49:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 Processor -- start 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e854d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 0x55965e851fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55965e857ef0 con 0x55965e855370 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55965e76b060 con 0x55965e84bb40 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55965e858070 con 0x55965e84a5d0 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc55cba1740 10 client.0 _ll_register_callbacks cb 0x55965e4be8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc553fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e854d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46832/0 (socket says 172.21.15.16:46832) 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc558d65640 1 -- 172.21.15.16:0/641651078 learned_addr learned my addr 172.21.15.16:0/641651078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.643+0000 7fc54bfff640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 0x55965e851fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 -- 172.21.15.16:0/641651078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 msgr2=0x55965e854d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e854d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 -- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 msgr2=0x55965e851fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 0x55965e851fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 -- 172.21.15.16:0/641651078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55965e5213c0 con 0x55965e855370 2024-09-12T02:49:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc54bfff640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 0x55965e851fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc553fff640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e854d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc558d65640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc54c02a900 tx=0x7fc54c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb54aa7e99bb651f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc5337fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c02f610 con 0x55965e855370 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc54c04d050 con 0x55965e855370 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc54c036070 con 0x55965e855370 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc54c031720 con 0x55965e855370 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc55cba1740 10 client.15798 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.644+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55965e857ef0 con 0x55965e855370 2024-09-12T02:49:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54c036070 con 0x55965e855370 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc54c030760 con 0x55965e855370 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 10 client.15798 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 10 client.15798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 20 client.15798 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e864ad0 0x55965e866ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 4 client.15798 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55965e7bd860 con 0x55965e864ad0 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc553fff640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e864ad0 0x55965e866ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc553fff640 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e864ad0 0x55965e866ec0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6cf1a47ba5067631 server_cookie=a2dd9fb2fccb76fe in_seq=0 out_seq=0 2024-09-12T02:49:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.645+0000 7fc5337fe640 10 client.15798 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc5337fe640 1 -- 172.21.15.16:0/641651078 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55965e7bd860 con 0x55965e864ad0 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc5337fe640 10 client.15798 handle_command_reply: tid=0 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 client.15798 shutdown 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 msgr2=0x55965e8525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc54c02a900 tx=0x7fc54c004970 comp rx=0 tx=0).stop 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 shutdown_connections 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e864ad0 0x55965e866ec0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc5337fe640 0 client.15798 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e8525d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84bb40 0x55965e851fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 --2- 172.21.15.16:0/641651078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84a5d0 0x55965e854d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.646+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 >> 172.21.15.16:0/641651078 conn(0x55965e813310 msgr2=0x55965e84c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.647+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 shutdown_connections 2024-09-12T02:49:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.647+0000 7fc55cba1740 1 -- 172.21.15.16:0/641651078 wait complete. 2024-09-12T02:49:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.647+0000 7fc55cba1740 20 client.15798 trim_cache size 0 max 16384 2024-09-12T02:49:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 Processor -- start 2024-09-12T02:49:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 -- start start 2024-09-12T02:49:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e861190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84bb40 0x55965e86c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541ab300 con 0x55965e855370 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541acf00 con 0x55965e84a5d0 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc55cba1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541ab9d0 con 0x55965e84bb40 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46836/0 (socket says 172.21.15.16:46836) 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 -- 172.21.15.16:0/2041195852 learned_addr learned my addr 172.21.15.16:0/2041195852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc553fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e861190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc54bfff640 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84bb40 0x55965e86c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 -- 172.21.15.16:0/2041195852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84bb40 msgr2=0x55965e86c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84bb40 0x55965e86c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 -- 172.21.15.16:0/2041195852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 msgr2=0x55965e861190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e861190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc558d65640 1 -- 172.21.15.16:0/2041195852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965e855060 con 0x55965e855370 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.660+0000 7fc553fff640 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e861190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc558d65640 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc54c030b30 tx=0x7fc54c031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ec9961e8e2db10 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc549ffb640 1 -- 172.21.15.16:0/2041195852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c02f610 con 0x55965e855370 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc549ffb640 1 -- 172.21.15.16:0/2041195852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54c036040 con 0x55965e855370 2024-09-12T02:49:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc549ffb640 1 -- 172.21.15.16:0/2041195852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c02fa20 con 0x55965e855370 2024-09-12T02:49:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 -- 172.21.15.16:0/2041195852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 msgr2=0x55965e86cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc54c030b30 tx=0x7fc54c031bb0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 -- 172.21.15.16:0/2041195852 shutdown_connections 2024-09-12T02:49:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e855370 0x55965e86cbe0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e84bb40 0x55965e86c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 --2- 172.21.15.16:0/2041195852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e861190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 -- 172.21.15.16:0/2041195852 >> 172.21.15.16:0/2041195852 conn(0x55965e813310 msgr2=0x55965e8508b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.661+0000 7fc55cba1740 1 -- 172.21.15.16:0/2041195852 shutdown_connections 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 -- 172.21.15.16:0/2041195852 wait complete. 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 -- start start 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 Processor -- start 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e858f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e8594b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541ac770 con 0x55965e84bb40 2024-09-12T02:49:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541ac0a0 con 0x55965e84a5d0 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541a92e0 con 0x55965e855370 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc55cba1740 10 client.0 _ll_register_callbacks cb 0x7fc54c04a020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.662+0000 7fc558d65640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34092/0 (socket says 172.21.15.16:34092) 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc553fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e858f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 -- 172.21.15.16:0/1889771501 learned_addr learned my addr 172.21.15.16:0/1889771501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc54bfff640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e8594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 -- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 msgr2=0x55965e858f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e858f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 -- 172.21.15.16:0/1889771501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 msgr2=0x55965e8594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e8594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 -- 172.21.15.16:0/1889771501 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55965e8ce670 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc5337fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc558d65640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc54c031f10 tx=0x7fc54c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b69f599cd4ee06e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c02f610 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc54c036040 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.663+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc54c04d040 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc54c004b40 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 10 client.25330 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55965e8cf870 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54c036040 con 0x55965e855370 2024-09-12T02:49:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc54c051480 con 0x55965e855370 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 10 client.25330 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 10 client.25330 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 20 client.25330 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e8cfed0 0x55965e8d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 4 client.25330 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55965e7b7d70 con 0x55965e8cfed0 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.664+0000 7fc553fff640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e8cfed0 0x55965e8d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc553fff640 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e8cfed0 0x55965e8d22c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=da042474a3ddd8f1 server_cookie=7c0c42ada1ac17c9 in_seq=0 out_seq=0 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc5337fe640 10 client.25330 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc5337fe640 1 -- 172.21.15.16:0/1889771501 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55965e7b7d70 con 0x55965e8cfed0 2024-09-12T02:49:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc5337fe640 10 client.25330 handle_command_reply: tid=0 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 client.25330 shutdown 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 msgr2=0x55965e8599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc54c031f10 tx=0x7fc54c004ab0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 shutdown_connections 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55965e8cfed0 0x55965e8d22c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55965e855370 0x55965e8599f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55965e84bb40 0x55965e8594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 --2- 172.21.15.16:0/1889771501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55965e84a5d0 0x55965e858f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc5337fe640 0 client.25330 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 >> 172.21.15.16:0/1889771501 conn(0x55965e813310 msgr2=0x55965e853840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 shutdown_connections 2024-09-12T02:49:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 1 -- 172.21.15.16:0/1889771501 wait complete. 2024-09-12T02:49:34.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.665+0000 7fc55cba1740 20 client.25330 trim_cache size 0 max 16384 2024-09-12T02:49:34.673 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 msgr2=0x7fc5400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 0x7fc5400a50c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc54c04b000 tx=0x7fc54c048000 comp rx=0 tx=0).stop 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 msgr2=0x7fc554198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 0x7fc554198880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c970 tx=0x7fc53c00ce30 comp rx=0 tx=0).stop 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 shutdown_connections 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc554106f10 0x7fc55419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc554102790 0x7fc554198880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5400a2c00 0x7fc5400a50c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 --2- 172.21.15.16:0/544442504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc554101de0 0x7fc554198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 >> 172.21.15.16:0/544442504 conn(0x7fc5540fd8c0 msgr2=0x7fc5540febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 shutdown_connections 2024-09-12T02:49:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:34.666+0000 7fc55a7c8640 1 -- 172.21.15.16:0/544442504 wait complete. 2024-09-12T02:49:34.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 86766 13.48 MB/sec warmup 36 sec latency 57.179 ms 2024-09-12T02:49:35.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:49:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:35.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 89345 13.49 MB/sec warmup 37 sec latency 25.669 ms 2024-09-12T02:49:35.836 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.4 2024-09-12T02:49:35.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:49:35.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:35 smithi033 ceph-mon[35092]: pgmap v387: 97 pgs: 97 active+clean; 51 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-09-12T02:49:36.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:35 smithi016 ceph-mon[30925]: pgmap v387: 97 pgs: 97 active+clean; 51 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-09-12T02:49:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:35 smithi184 ceph-mon[35321]: pgmap v387: 97 pgs: 97 active+clean; 51 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-09-12T02:49:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 -- 172.21.15.16:0/365292098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c102790 msgr2=0x7f1f8c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 --2- 172.21.15.16:0/365292098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c102790 0x7f1f8c1067e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009900 tx=0x7f1f8002a3f0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 -- 172.21.15.16:0/365292098 shutdown_connections 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 --2- 172.21.15.16:0/365292098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c106f10 0x7f1f8c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 --2- 172.21.15.16:0/365292098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c102790 0x7f1f8c1067e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 --2- 172.21.15.16:0/365292098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c101de0 0x7f1f8c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 -- 172.21.15.16:0/365292098 >> 172.21.15.16:0/365292098 conn(0x7f1f8c0fd8c0 msgr2=0x7f1f8c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 -- 172.21.15.16:0/365292098 shutdown_connections 2024-09-12T02:49:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.124+0000 7f1f91152640 1 -- 172.21.15.16:0/365292098 wait complete. 2024-09-12T02:49:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 Processor -- start 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 -- start start 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c111f80 con 0x7f1f8c101de0 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c111e00 con 0x7f1f8c102790 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f91152640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c112100 con 0x7f1f8c106f10 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37532/0 (socket says 172.21.15.16:37532) 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8b59e640 1 -- 172.21.15.16:0/1791905262 learned_addr learned my addr 172.21.15.16:0/1791905262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.125+0000 7f1f8a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38380/0 (socket says 172.21.15.16:38380) 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 -- 172.21.15.16:0/1791905262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 msgr2=0x7f1f8c19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 -- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 msgr2=0x7f1f8c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 -- 172.21.15.16:0/1791905262 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c19f640 con 0x7f1f8c106f10 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400eb10 tx=0x7f1f8400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f73fff640 1 -- 172.21.15.16:0/1791905262 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f84004b20 con 0x7f1f8c106f10 2024-09-12T02:49:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f73fff640 1 -- 172.21.15.16:0/1791905262 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f84004cc0 con 0x7f1f8c106f10 2024-09-12T02:49:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c19f930 con 0x7f1f8c106f10 2024-09-12T02:49:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f73fff640 1 -- 172.21.15.16:0/1791905262 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f84010760 con 0x7f1f8c106f10 2024-09-12T02:49:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.126+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c1a7210 con 0x7f1f8c106f10 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 Processor -- start 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f73fff640 1 -- 172.21.15.16:0/1791905262 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1f84005020 con 0x7f1f8c106f10 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 -- start start 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a205090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a212760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56134a16b2f0 con 0x56134a2055d0 2024-09-12T02:49:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56134a112cf0 con 0x56134a206b40 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f9352b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56134a1672b0 con 0x56134a210370 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a205090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37540/0 (socket says 172.21.15.16:37540) 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a205090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38390/0 (socket says 172.21.15.16:38390) 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3852092066 learned_addr learned my addr 172.21.15.16:0/3852092066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a212760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3852092066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 msgr2=0x56134a212760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a212760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3852092066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 msgr2=0x56134a205090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a205090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.130+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3852092066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134a0f73a0 con 0x56134a2055d0 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c010480 tx=0x7f1f7c010940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5765e9c0ea14b509 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f73fff640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 0x7f1f740a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 0x7f1f740a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f717fa640 1 -- 172.21.15.16:0/3852092066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c011760 con 0x56134a2055d0 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f717fa640 1 -- 172.21.15.16:0/3852092066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f7c011de0 con 0x56134a2055d0 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f73fff640 1 -- 172.21.15.16:0/1791905262 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f840cf4c0 con 0x7f1f8c106f10 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.131+0000 7f1f717fa640 1 -- 172.21.15.16:0/3852092066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0193d0 con 0x56134a2055d0 2024-09-12T02:49:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- 172.21.15.16:0/3852092066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 msgr2=0x56134a205a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 0x7f1f740a50c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f8002a3f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c010480 tx=0x7f1f7c010940 comp rx=0 tx=0).stop 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- 172.21.15.16:0/3852092066 shutdown_connections 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a212760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a205a10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3852092066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a205090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- 172.21.15.16:0/3852092066 >> 172.21.15.16:0/3852092066 conn(0x56134a1ce310 msgr2=0x56134a201c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- 172.21.15.16:0/3852092066 shutdown_connections 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- 172.21.15.16:0/3852092066 wait complete. 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.132+0000 7f1f9352b740 1 -- start start 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 Processor -- start 2024-09-12T02:49:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a2055d0 0x56134a20fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a20cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56134a2130c0 con 0x56134a210370 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56134a126060 con 0x56134a206b40 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56134a213240 con 0x56134a2055d0 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f9352b740 10 client.0 _ll_register_callbacks cb 0x561349e798c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a2055d0 0x56134a20fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f8b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f8b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37552/0 (socket says 172.21.15.16:37552) 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f8b59e640 1 -- 172.21.15.16:0/833320796 learned_addr learned my addr 172.21.15.16:0/833320796 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.133+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a20cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 -- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a2055d0 msgr2=0x56134a20fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a2055d0 0x56134a20fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 -- 172.21.15.16:0/833320796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 msgr2=0x56134a20cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a20cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 -- 172.21.15.16:0/833320796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561349edc3c0 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f840a4470 tx=0x7f1f8409b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8dd6f9ac059460 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f840cf2e0 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f840a4510 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f7c011de0 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1f8409bd30 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f9352b740 10 client.15816 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56134a21ead0 con 0x56134a210370 2024-09-12T02:49:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f7c011de0 con 0x56134a210370 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.134+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1f840b2070 con 0x56134a210370 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 10 client.15816 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 10 client.15816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 20 client.15816 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a21ff90 0x56134a2223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 4 client.15816 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56134a178860 con 0x56134a21ff90 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a21ff90 0x56134a2223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f6eff5640 10 client.15816 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.135+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a21ff90 0x56134a2223e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e519a6a959b664fc server_cookie=4e6eeffef0e233c3 in_seq=0 out_seq=0 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f6eff5640 1 -- 172.21.15.16:0/833320796 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56134a178860 con 0x56134a21ff90 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f6eff5640 10 client.15816 handle_command_reply: tid=0 2024-09-12T02:49:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 client.15816 shutdown 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 msgr2=0x56134a20d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f840a4470 tx=0x7f1f8409b8f0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 shutdown_connections 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a210370 0x56134a20d460 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a20cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a21ff90 0x56134a2223e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 --2- 172.21.15.16:0/833320796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a2055d0 0x56134a20fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 >> 172.21.15.16:0/833320796 conn(0x56134a1ce310 msgr2=0x56134a207960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.136+0000 7f1f6eff5640 0 client.15816 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.137+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 shutdown_connections 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.137+0000 7f1f9352b740 1 -- 172.21.15.16:0/833320796 wait complete. 2024-09-12T02:49:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.137+0000 7f1f9352b740 20 client.15816 trim_cache size 0 max 16384 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 Processor -- start 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 -- start start 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a204c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a211760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1af7d0 con 0x56134a2055d0 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1b13d0 con 0x56134a206b40 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f9352b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1afea0 con 0x56134a210370 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37554/0 (socket says 172.21.15.16:37554) 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/1665105026 learned_addr learned my addr 172.21.15.16:0/1665105026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a211760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.152+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a204c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/1665105026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 msgr2=0x56134a211760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a211760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/1665105026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 msgr2=0x56134a204c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a204c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/1665105026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134a212f50 con 0x56134a2055d0 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c010480 tx=0x7f1f7c010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43bb28dc5a35f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f70ff9640 1 -- 172.21.15.16:0/1665105026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c00eb30 con 0x56134a2055d0 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f70ff9640 1 -- 172.21.15.16:0/1665105026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f7c029030 con 0x56134a2055d0 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.153+0000 7f1f70ff9640 1 -- 172.21.15.16:0/1665105026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c004660 con 0x56134a2055d0 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 -- 172.21.15.16:0/1665105026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 msgr2=0x56134a21c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c010480 tx=0x7f1f7c010ef0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 -- 172.21.15.16:0/1665105026 shutdown_connections 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a210370 0x56134a211760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a206b40 0x56134a204c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 --2- 172.21.15.16:0/1665105026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a21c360 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 -- 172.21.15.16:0/1665105026 >> 172.21.15.16:0/1665105026 conn(0x56134a1ce310 msgr2=0x56134a20cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 -- 172.21.15.16:0/1665105026 shutdown_connections 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.154+0000 7f1f9352b740 1 -- 172.21.15.16:0/1665105026 wait complete. 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 -- start start 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 Processor -- start 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a206b40 0x56134a20a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a210370 0x56134a20a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1b0c40 con 0x56134a2055d0 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1b0570 con 0x56134a210370 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1ad7b0 con 0x56134a206b40 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f9352b740 10 client.0 _ll_register_callbacks cb 0x7f1f8409bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37560/0 (socket says 172.21.15.16:37560) 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3957165098 learned_addr learned my addr 172.21.15.16:0/3957165098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:36.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a206b40 0x56134a20a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.155+0000 7f1f8b59e640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a210370 0x56134a20a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a206b40 msgr2=0x56134a20a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a206b40 0x56134a20a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3957165098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a210370 msgr2=0x56134a20a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a210370 0x56134a20a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 -- 172.21.15.16:0/3957165098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134a20ad30 con 0x56134a2055d0 2024-09-12T02:49:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f8ad9d640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c005d00 tx=0x7f1f7c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451ad4b6b49c2dbc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0053f0 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f7c005590 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f7c029030 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1f7c00e350 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f9352b740 10 client.15822 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.156+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56134a28ac80 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f7c037070 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1f7c004c10 con 0x56134a2055d0 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 10 client.15822 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 10 client.15822 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 20 client.15822 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a28b050 0x56134a28d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 4 client.15822 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1f7c029030 con 0x56134a28b050 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.157+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a28b050 0x56134a28d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f6eff5640 10 client.15822 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f8a59c640 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a28b050 0x56134a28d440 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=74cb623ad610561a server_cookie=bfe0fde718f9ad6e in_seq=0 out_seq=0 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f6eff5640 1 -- 172.21.15.16:0/3957165098 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1f7c029030 con 0x56134a28b050 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f6eff5640 10 client.15822 handle_command_reply: tid=0 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 client.15822 shutdown 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 msgr2=0x56134a2189f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c005d00 tx=0x7f1f7c00bdf0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 shutdown_connections 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56134a210370 0x56134a20a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56134a28b050 0x56134a28d440 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56134a206b40 0x56134a20a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 --2- 172.21.15.16:0/3957165098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56134a2055d0 0x56134a2189f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f6eff5640 0 client.15822 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.158+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 >> 172.21.15.16:0/3957165098 conn(0x56134a1ce310 msgr2=0x56134a207110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 shutdown_connections 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f9352b740 1 -- 172.21.15.16:0/3957165098 wait complete. 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f9352b740 20 client.15822 trim_cache size 0 max 16384 2024-09-12T02:49:36.164 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 msgr2=0x7f1f740a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 0x7f1f740a50c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f8002a3f0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 msgr2=0x7f1f8c19ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400eb10 tx=0x7f1f8400c2d0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 shutdown_connections 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f8c106f10 0x7f1f8c19ef60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f8c102790 0x7f1f8c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f740a2c00 0x7f1f740a50c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 --2- 172.21.15.16:0/1791905262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f8c101de0 0x7f1f8c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.159+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 >> 172.21.15.16:0/1791905262 conn(0x7f1f8c0fd8c0 msgr2=0x7f1f8c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.160+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 shutdown_connections 2024-09-12T02:49:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:36.160+0000 7f1f91152640 1 -- 172.21.15.16:0/1791905262 wait complete. 2024-09-12T02:49:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:36.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92548 13.63 MB/sec warmup 38 sec latency 40.686 ms 2024-09-12T02:49:36.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 -- 172.21.15.16:0/2102782735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f0105ec0 msgr2=0x7ff8f0106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 --2- 172.21.15.16:0/2102782735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f0105ec0 0x7ff8f0106340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009ca0 tx=0x7ff8e802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 -- 172.21.15.16:0/2102782735 shutdown_connections 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 --2- 172.21.15.16:0/2102782735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f0105ec0 0x7ff8f0106340 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 --2- 172.21.15.16:0/2102782735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f01011e0 0x7ff8f0105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 --2- 172.21.15.16:0/2102782735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01008c0 0x7ff8f0100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 -- 172.21.15.16:0/2102782735 >> 172.21.15.16:0/2102782735 conn(0x7ff8f00fc7d0 msgr2=0x7ff8f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 -- 172.21.15.16:0/2102782735 shutdown_connections 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.367+0000 7ff8f7109640 1 -- 172.21.15.16:0/2102782735 wait complete. 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 Processor -- start 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 -- start start 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01011e0 0x7ff8f01a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f0105ec0 0x7ff8f019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0118660 con 0x7ff8f01008c0 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01184e0 con 0x7ff8f0105ec0 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f7109640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01187e0 con 0x7ff8f01011e0 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f4ea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f4ea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37588/0 (socket says 172.21.15.16:37588) 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f4ea5640 1 -- 172.21.15.16:0/1495917605 learned_addr learned my addr 172.21.15.16:0/1495917605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8e7fff640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01011e0 0x7ff8f01a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.368+0000 7ff8f56a6640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f0105ec0 0x7ff8f019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 -- 172.21.15.16:0/1495917605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01011e0 msgr2=0x7ff8f01a4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01011e0 0x7ff8f01a4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 -- 172.21.15.16:0/1495917605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f0105ec0 msgr2=0x7ff8f019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f0105ec0 0x7ff8f019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 -- 172.21.15.16:0/1495917605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f019f020 con 0x7ff8f01008c0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f4ea5640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00e9f0 tx=0x7ff8ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec00cd80 con 0x7ff8f01008c0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f019f310 con 0x7ff8f01008c0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec010910 con 0x7ff8f01008c0 2024-09-12T02:49:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec018990 con 0x7ff8f01008c0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.369+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f01ab600 con 0x7ff8f01008c0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.370+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f0105980 con 0x7ff8f01008c0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.372+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff8ec010ac0 con 0x7ff8f01008c0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.372+0000 7ff8e5ffb640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 0x7ff8d80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.373+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8ec0cf300 con 0x7ff8f01008c0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.373+0000 7ff8e7fff640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 0x7ff8d80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.373+0000 7ff8e7fff640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 0x7ff8d80a50c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00ffdc0 tx=0x7ff8e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.378+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8ec09bbc0 con 0x7ff8f01008c0 2024-09-12T02:49:37.425 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:37.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.615+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8f019fe20 con 0x7ff8f01008c0 2024-09-12T02:49:37.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.616+0000 7ff8e5ffb640 1 -- 172.21.15.16:0/1495917605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7ff8ec09fec0 con 0x7ff8f01008c0 2024-09-12T02:49:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:49:37.618 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:49:37.618 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 msgr2=0x7ff8d80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 0x7ff8d80a50c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00ffdc0 tx=0x7ff8e0005f90 comp rx=0 tx=0).stop 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 msgr2=0x7ff8f01a4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00e9f0 tx=0x7ff8ec00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 shutdown_connections 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f0105ec0 0x7ff8f019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f01011e0 0x7ff8f01a4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8d80a2c00 0x7ff8d80a50c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 --2- 172.21.15.16:0/1495917605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f01008c0 0x7ff8f01a4730 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.618+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 >> 172.21.15.16:0/1495917605 conn(0x7ff8f00fc7d0 msgr2=0x7ff8f0103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.619+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 shutdown_connections 2024-09-12T02:49:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:37.619+0000 7ff8f7109640 1 -- 172.21.15.16:0/1495917605 wait complete. 2024-09-12T02:49:37.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 95608 13.66 MB/sec warmup 39 sec latency 18.246 ms 2024-09-12T02:49:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:37 smithi016 ceph-mon[30925]: pgmap v388: 97 pgs: 97 active+clean; 59 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 279 op/s 2024-09-12T02:49:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:37 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1495917605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:37.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:37 smithi033 ceph-mon[35092]: pgmap v388: 97 pgs: 97 active+clean; 59 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 279 op/s 2024-09-12T02:49:37.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:37 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1495917605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:37 smithi184 ceph-mon[35321]: pgmap v388: 97 pgs: 97 active+clean; 59 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.0 MiB/s wr, 279 op/s 2024-09-12T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:37 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1495917605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:38.338 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 133.4 --await-for 45.0' 2024-09-12T02:49:38.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 133.4 --await-for 45.0 2024-09-12T02:49:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.529+0000 7f681f0aa640 1 -- 172.21.15.16:0/902096151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818102d30 msgr2=0x7f6818103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 --2- 172.21.15.16:0/902096151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818102d30 0x7f6818103110 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f680c009930 tx=0x7f680c02a420 comp rx=0 tx=0).stop 2024-09-12T02:49:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 -- 172.21.15.16:0/902096151 shutdown_connections 2024-09-12T02:49:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 --2- 172.21.15.16:0/902096151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68181082a0 0x7f681810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 --2- 172.21.15.16:0/902096151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 0x7f6818107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 --2- 172.21.15.16:0/902096151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818102d30 0x7f6818103110 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 -- 172.21.15.16:0/902096151 >> 172.21.15.16:0/902096151 conn(0x7f68180fc7d0 msgr2=0x7f68180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 -- 172.21.15.16:0/902096151 shutdown_connections 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 -- 172.21.15.16:0/902096151 wait complete. 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 Processor -- start 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.530+0000 7f681f0aa640 1 -- start start 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 0x7f6818198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68181082a0 0x7f681819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181120a0 con 0x7f68181082a0 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6818111f20 con 0x7f6818102d30 2024-09-12T02:49:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681f0aa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6818112220 con 0x7f68181036e0 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38462/0 (socket says 172.21.15.16:38462) 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 -- 172.21.15.16:0/2681734280 learned_addr learned my addr 172.21.15.16:0/2681734280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681d647640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68181082a0 0x7f681819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f6817fff640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 0x7f6818198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 -- 172.21.15.16:0/2681734280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 msgr2=0x7f6818198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 0x7f6818198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 -- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68181082a0 msgr2=0x7f681819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68181082a0 0x7f681819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.531+0000 7f681ce46640 1 -- 172.21.15.16:0/2681734280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681819d590 con 0x7f6818102d30 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f681ce46640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f680c009230 tx=0x7f680c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f6815ffb640 1 -- 172.21.15.16:0/2681734280 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c049070 con 0x7f6818102d30 2024-09-12T02:49:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681819d820 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f6815ffb640 1 -- 172.21.15.16:0/2681734280 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c03cbd0 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f6815ffb640 1 -- 172.21.15.16:0/2681734280 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c035070 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.532+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68181008c0 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.534+0000 7f6815ffb640 1 -- 172.21.15.16:0/2681734280 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f680c04c070 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 Processor -- start 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6815ffb640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 0x7f68080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 -- start start 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6815ffb640 1 -- 172.21.15.16:0/2681734280 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f680c0f2ff0 con 0x7f6818102d30 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c244090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c244a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c251760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d3c1aa2f0 con 0x564d3c245b40 2024-09-12T02:49:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564d3c151cf0 con 0x564d3c24f370 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.535+0000 7f6821483740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564d3c1a62b0 con 0x564d3c2445d0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681d647640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c251760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681ce46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c244090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681d647640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 0x7f68080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c244a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681ce46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c244090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37608/0 (socket says 172.21.15.16:37608) 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681ce46640 1 -- 172.21.15.16:0/211732111 learned_addr learned my addr 172.21.15.16:0/211732111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 -- 172.21.15.16:0/211732111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 msgr2=0x564d3c251760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c251760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 -- 172.21.15.16:0/211732111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 msgr2=0x564d3c244090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c244090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 -- 172.21.15.16:0/211732111 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3c1363a0 con 0x564d3c2445d0 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f681d647640 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c251760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.536+0000 7f6817fff640 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c244a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f681806a370 tx=0x7f6804011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165d9defdee1cff8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.537+0000 7f681d647640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 0x7f68080a5040 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x564d3c245070 tx=0x7f6810009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.537+0000 7f6802ffd640 1 -- 172.21.15.16:0/211732111 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804004250 con 0x564d3c2445d0 2024-09-12T02:49:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.537+0000 7f6802ffd640 1 -- 172.21.15.16:0/211732111 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6804009e10 con 0x564d3c2445d0 2024-09-12T02:49:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.537+0000 7f6802ffd640 1 -- 172.21.15.16:0/211732111 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804004e00 con 0x564d3c2445d0 2024-09-12T02:49:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- 172.21.15.16:0/211732111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 msgr2=0x564d3c244a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c244a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f681806a370 tx=0x7f6804011810 comp rx=0 tx=0).stop 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- 172.21.15.16:0/211732111 shutdown_connections 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c251760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c244a10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 --2- 172.21.15.16:0/211732111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c244090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- 172.21.15.16:0/211732111 >> 172.21.15.16:0/211732111 conn(0x564d3c20d310 msgr2=0x564d3c240c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- 172.21.15.16:0/211732111 shutdown_connections 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- 172.21.15.16:0/211732111 wait complete. 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 -- start start 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.538+0000 7f6821483740 1 Processor -- start 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 0x564d3c24bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 0x564d3c24c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d3c252070 con 0x564d3c2445d0 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564d3c165060 con 0x564d3c245b40 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564d3c2521f0 con 0x564d3c24f370 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6821483740 10 client.0 _ll_register_callbacks cb 0x564d3beb88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37610/0 (socket says 172.21.15.16:37610) 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 -- 172.21.15.16:0/420013921 learned_addr learned my addr 172.21.15.16:0/420013921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6817fff640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 0x564d3c24bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681d647640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 0x564d3c24c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 -- 172.21.15.16:0/420013921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 msgr2=0x564d3c24c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 0x564d3c24c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 -- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 msgr2=0x564d3c24bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 0x564d3c24bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681ce46640 1 -- 172.21.15.16:0/420013921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3bf1b3c0 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f681d647640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 0x564d3c24c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.539+0000 7f6817fff640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 0x564d3c24bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f681ce46640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f680c009a50 tx=0x7f680c0bf250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1546c76e5e8388eb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0d8060 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6804009e10 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f680c0bfb50 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f680c0c3bf0 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f6821483740 10 client.15840 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564d3c252070 con 0x564d3c2445d0 2024-09-12T02:49:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6804009e10 con 0x564d3c2445d0 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.540+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f680c043030 con 0x564d3c2445d0 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 10 client.15840 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 10 client.15840 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 20 client.15840 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2612d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 4 client.15840 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d3c1b7860 con 0x564d3c25ee80 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6817fff640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f67f7fff640 10 client.15840 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.541+0000 7f6817fff640 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2612d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f82ade06ac0593b8 server_cookie=a645e4be544410aa in_seq=0 out_seq=0 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f67f7fff640 1 -- 172.21.15.16:0/420013921 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564d3c1b7860 con 0x564d3c25ee80 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f67f7fff640 10 client.15840 handle_command_reply: tid=0 2024-09-12T02:49:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 client.15840 shutdown 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 msgr2=0x564d3c24ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f680c009a50 tx=0x7f680c0bf250 comp rx=0 tx=0).stop 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 shutdown_connections 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2612d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f67f7fff640 0 client.15840 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c24f370 0x564d3c24c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c245b40 0x564d3c24bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 --2- 172.21.15.16:0/420013921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c2445d0 0x564d3c24ecb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 >> 172.21.15.16:0/420013921 conn(0x564d3c20d310 msgr2=0x564d3c246960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 shutdown_connections 2024-09-12T02:49:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.542+0000 7f6821483740 1 -- 172.21.15.16:0/420013921 wait complete. 2024-09-12T02:49:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.543+0000 7f6821483740 20 client.15840 trim_cache size 0 max 16384 2024-09-12T02:49:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.557+0000 7f6821483740 1 Processor -- start 2024-09-12T02:49:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.557+0000 7f6821483740 1 -- start start 2024-09-12T02:49:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.557+0000 7f6821483740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6821483740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c250760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6821483740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c250ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6821483740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ab300 con 0x564d3c245b40 2024-09-12T02:49:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6821483740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68181acf00 con 0x564d3c24f370 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6821483740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ab9d0 con 0x564d3c2445d0 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f6817fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c250760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47616/0 (socket says 172.21.15.16:47616) 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 -- 172.21.15.16:0/4095426916 learned_addr learned my addr 172.21.15.16:0/4095426916 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681d647640 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c250ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 -- 172.21.15.16:0/4095426916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 msgr2=0x564d3c250ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c250ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 -- 172.21.15.16:0/4095426916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 msgr2=0x564d3c250760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c250760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 -- 172.21.15.16:0/4095426916 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3c247b30 con 0x564d3c2445d0 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.558+0000 7f681ce46640 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f680c0bbd50 tx=0x7f680c0bfa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a0f6145873a63a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.559+0000 7f68027fc640 1 -- 172.21.15.16:0/4095426916 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0de170 con 0x564d3c2445d0 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.559+0000 7f68027fc640 1 -- 172.21.15.16:0/4095426916 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c0c8020 con 0x564d3c2445d0 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.559+0000 7f68027fc640 1 -- 172.21.15.16:0/4095426916 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0bfdb0 con 0x564d3c2445d0 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 -- 172.21.15.16:0/4095426916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 msgr2=0x564d3c25b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f680c0bbd50 tx=0x7f680c0bfa70 comp rx=0 tx=0).stop 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 -- 172.21.15.16:0/4095426916 shutdown_connections 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c24f370 0x564d3c250ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c245b40 0x564d3c250760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 --2- 172.21.15.16:0/4095426916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c2445d0 0x564d3c25b310 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 -- 172.21.15.16:0/4095426916 >> 172.21.15.16:0/4095426916 conn(0x564d3c20d310 msgr2=0x564d3c24bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 -- 172.21.15.16:0/4095426916 shutdown_connections 2024-09-12T02:49:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.560+0000 7f6821483740 1 -- 172.21.15.16:0/4095426916 wait complete. 2024-09-12T02:49:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.561+0000 7f6821483740 1 -- start start 2024-09-12T02:49:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.561+0000 7f6821483740 1 Processor -- start 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 0x564d3c265c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 0x564d3c2661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c24f370 0x564d3c256480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ac770 con 0x564d3c24f370 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ac0a0 con 0x564d3c2445d0 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68181a9300 con 0x564d3c245b40 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6821483740 10 client.0 _ll_register_callbacks cb 0x7f680c0a8e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f681ce46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 0x564d3c265c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 0x564d3c2661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f681ce46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 0x564d3c265c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38504/0 (socket says 172.21.15.16:38504) 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f681ce46640 1 -- 172.21.15.16:0/120927913 learned_addr learned my addr 172.21.15.16:0/120927913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 -- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 msgr2=0x564d3c265c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 0x564d3c265c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 -- 172.21.15.16:0/120927913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c24f370 msgr2=0x564d3c256480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c24f370 0x564d3c256480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.562+0000 7f6817fff640 1 -- 172.21.15.16:0/120927913 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3c266a60 con 0x564d3c245b40 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6817fff640 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 0x564d3c2661a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68040096e0 tx=0x7f6804009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=260a7ba45c4d53b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804034060 con 0x564d3c245b40 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f68040035f0 con 0x564d3c245b40 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c0c8020 con 0x564d3c245b40 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f680401e040 con 0x564d3c245b40 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 10 client.25372 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564d3c257a70 con 0x564d3c245b40 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c0c8020 con 0x564d3c245b40 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f680402e070 con 0x564d3c245b40 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 10 client.25372 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 10 client.25372 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.563+0000 7f6821483740 20 client.25372 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f6821483740 4 client.25372 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d3c1b1d70 con 0x564d3c25ee80 2024-09-12T02:49:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f681ce46640 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f67f7fff640 10 client.25372 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f681ce46640 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2cbb90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83162a943c345cc9 server_cookie=6b424475ea31e260 in_seq=0 out_seq=0 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f67f7fff640 1 -- 172.21.15.16:0/120927913 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d3c1b1d70 con 0x564d3c25ee80 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f67f7fff640 10 client.25372 handle_command_reply: tid=0 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.564+0000 7f6821483740 1 client.25372 shutdown 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 msgr2=0x564d3c2661a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 0x564d3c2661a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68040096e0 tx=0x7f6804009260 comp rx=0 tx=0).stop 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 shutdown_connections 2024-09-12T02:49:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x564d3c25ee80 0x564d3c2cbb90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d3c24f370 0x564d3c256480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d3c245b40 0x564d3c2661a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f67f7fff640 0 client.25372 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:49:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 --2- 172.21.15.16:0/120927913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564d3c2445d0 0x564d3c265c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 >> 172.21.15.16:0/120927913 conn(0x564d3c20d310 msgr2=0x564d3c23f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 shutdown_connections 2024-09-12T02:49:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 1 -- 172.21.15.16:0/120927913 wait complete. 2024-09-12T02:49:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.565+0000 7f6821483740 20 client.25372 trim_cache size 0 max 16384 2024-09-12T02:49:38.571 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:49:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 msgr2=0x7f68080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 0x7f68080a5040 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x564d3c245070 tx=0x7f6810009500 comp rx=0 tx=0).stop 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 msgr2=0x7f68181985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f680c009230 tx=0x7f680c004040 comp rx=0 tx=0).stop 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 shutdown_connections 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68080a2b80 0x7f68080a5040 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68181082a0 0x7f681819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68181036e0 0x7f6818198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 --2- 172.21.15.16:0/2681734280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6818102d30 0x7f68181985e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 >> 172.21.15.16:0/2681734280 conn(0x7f68180fc7d0 msgr2=0x7f6818105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 shutdown_connections 2024-09-12T02:49:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:38.566+0000 7f681f0aa640 1 -- 172.21.15.16:0/2681734280 wait complete. 2024-09-12T02:49:38.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 98186 13.61 MB/sec warmup 40 sec latency 11.268 ms 2024-09-12T02:49:39.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:49:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:39.706 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100364 13.60 MB/sec warmup 41 sec latency 37.315 ms 2024-09-12T02:49:39.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- 172.21.15.16:0/4288241271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 msgr2=0x7f2970101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 --2- 172.21.15.16:0/4288241271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970101200 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2958009900 tx=0x7f295802a480 comp rx=0 tx=0).stop 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- 172.21.15.16:0/4288241271 shutdown_connections 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 --2- 172.21.15.16:0/4288241271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102ef0 0x7f297010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 --2- 172.21.15.16:0/4288241271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102570 0x7f29701029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 --2- 172.21.15.16:0/4288241271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970101200 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- 172.21.15.16:0/4288241271 >> 172.21.15.16:0/4288241271 conn(0x7f29700fc7d0 msgr2=0x7f29700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- 172.21.15.16:0/4288241271 shutdown_connections 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- 172.21.15.16:0/4288241271 wait complete. 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 Processor -- start 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.759+0000 7f297705a640 1 -- start start 2024-09-12T02:49:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102570 0x7f2970197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102ef0 0x7f2970197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2970110f80 con 0x7f2970100e20 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2970110e00 con 0x7f2970102ef0 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f297705a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2970111100 con 0x7f2970102570 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37658/0 (socket says 172.21.15.16:37658) 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 -- 172.21.15.16:0/2484058614 learned_addr learned my addr 172.21.15.16:0/2484058614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f296ffff640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102570 0x7f2970197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f29755f7640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102ef0 0x7f2970197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 -- 172.21.15.16:0/2484058614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102570 msgr2=0x7f2970197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102570 0x7f2970197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 -- 172.21.15.16:0/2484058614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102ef0 msgr2=0x7f2970197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102ef0 0x7f2970197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 -- 172.21.15.16:0/2484058614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29701a6980 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.760+0000 7f2974df6640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2958006010 tx=0x7f2958003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.761+0000 7f296dffb640 1 -- 172.21.15.16:0/2484058614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295803f070 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.761+0000 7f296dffb640 1 -- 172.21.15.16:0/2484058614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958031c30 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.761+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29701a6c10 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.761+0000 7f296dffb640 1 -- 172.21.15.16:0/2484058614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958035030 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.761+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29701a70f0 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.763+0000 7f296dffb640 1 -- 172.21.15.16:0/2484058614 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2958042070 con 0x7f2970100e20 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.763+0000 7f2979433740 1 Processor -- start 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.763+0000 7f2979433740 1 -- start start 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7179b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 0x5586c7184400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7184940 0x5586c7186d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5586c70df2a0 con 0x5586c7184940 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5586c7086b10 con 0x5586c7179760 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2979433740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5586c70db260 con 0x5586c717a080 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f29755f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7179b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 0x5586c7184400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f296ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7184940 0x5586c7186d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f29755f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7179b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38540/0 (socket says 172.21.15.16:38540) 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f29755f7640 1 -- 172.21.15.16:0/1758538301 learned_addr learned my addr 172.21.15.16:0/1758538301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 -- 172.21.15.16:0/1758538301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 msgr2=0x5586c7179b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7179b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 -- 172.21.15.16:0/1758538301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7184940 msgr2=0x5586c7186d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7184940 0x5586c7186d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f2974df6640 1 -- 172.21.15.16:0/1758538301 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586c6e50b90 con 0x5586c717a080 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f296dffb640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 0x7f295c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.764+0000 7f29755f7640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 0x7f295c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.765+0000 7f2974df6640 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 0x5586c7184400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29580ed170 tx=0x7f29580f5cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e1bddb2568a7be server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.765+0000 7f296dffb640 1 -- 172.21.15.16:0/2484058614 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f29580ef690 con 0x7f2970100e20 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.765+0000 7f2966ffd640 1 -- 172.21.15.16:0/1758538301 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29580bbd80 con 0x5586c717a080 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.765+0000 7f2966ffd640 1 -- 172.21.15.16:0/1758538301 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958031de0 con 0x5586c717a080 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.765+0000 7f2966ffd640 1 -- 172.21.15.16:0/1758538301 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29580bb910 con 0x5586c717a080 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f29755f7640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 0x7f295c0a5040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2968009ee0 tx=0x7f2968005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- 172.21.15.16:0/1758538301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 msgr2=0x5586c7184400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 0x5586c7184400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29580ed170 tx=0x7f29580f5cd0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- 172.21.15.16:0/1758538301 shutdown_connections 2024-09-12T02:49:39.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7184940 0x5586c7186d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c717a080 0x5586c7184400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 --2- 172.21.15.16:0/1758538301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7179b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- 172.21.15.16:0/1758538301 >> 172.21.15.16:0/1758538301 conn(0x5586c7175cf0 msgr2=0x5586c71760d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- 172.21.15.16:0/1758538301 shutdown_connections 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- 172.21.15.16:0/1758538301 wait complete. 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 -- start start 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.766+0000 7f2979433740 1 Processor -- start 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7181810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c7182290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5586c71872d0 con 0x5586c717a080 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5586c70676e0 con 0x5586c7179760 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5586c7187450 con 0x5586c7184940 2024-09-12T02:49:39.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2979433740 10 client.0 _ll_register_callbacks cb 0x5586c708ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f29755f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c7182290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37684/0 (socket says 172.21.15.16:37684) 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 -- 172.21.15.16:0/2689928054 learned_addr learned my addr 172.21.15.16:0/2689928054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f2974df6640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7181810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 -- 172.21.15.16:0/2689928054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 msgr2=0x5586c7182290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c7182290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 -- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 msgr2=0x5586c7181810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7181810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 -- 172.21.15.16:0/2689928054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586c7190730 con 0x5586c717a080 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f296ffff640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2960011d40 tx=0x7f296000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04126984997f391 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f294ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:39.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.767+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296001f060 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f296001a3f0 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2958031de0 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f296001a5f0 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 10 client.15855 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5586c7193100 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958031de0 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2960019030 con 0x5586c717a080 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 10 client.15855 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 10 client.15855 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 20 client.15855 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 4 client.15855 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586c70ec9d0 con 0x5586c7194610 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.768+0000 7f2974df6640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.769+0000 7f2974df6640 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7196aa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79e5ab86432b2da9 server_cookie=bd05bf4c94908257 in_seq=0 out_seq=0 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.769+0000 7f294ffff640 10 client.15855 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.769+0000 7f294ffff640 1 -- 172.21.15.16:0/2689928054 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5586c70ec9d0 con 0x5586c7194610 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.769+0000 7f294ffff640 10 client.15855 handle_command_reply: tid=0 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.769+0000 7f2979433740 1 client.15855 shutdown 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 msgr2=0x5586c7181d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2960011d40 tx=0x7f296000f790 comp rx=0 tx=0).stop 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 shutdown_connections 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7196aa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c7182290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c7181d50 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 --2- 172.21.15.16:0/2689928054 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7181810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f294ffff640 0 client.15855 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 >> 172.21.15.16:0/2689928054 conn(0x5586c7175cf0 msgr2=0x5586c717b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 shutdown_connections 2024-09-12T02:49:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 1 -- 172.21.15.16:0/2689928054 wait complete. 2024-09-12T02:49:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.770+0000 7f2979433740 20 client.15855 trim_cache size 0 max 16384 2024-09-12T02:49:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.788+0000 7f2979433740 1 Processor -- start 2024-09-12T02:49:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.788+0000 7f2979433740 1 -- start start 2024-09-12T02:49:39.790 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:39 smithi016 ceph-mon[30925]: pgmap v389: 97 pgs: 97 active+clean; 62 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 218 op/s 2024-09-12T02:49:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7174d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c719c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c719c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29701af6a0 con 0x5586c717a080 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29701b12a0 con 0x5586c7179760 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2979433740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29701afd70 con 0x5586c7184940 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f296ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c719c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f296ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c719c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37692/0 (socket says 172.21.15.16:37692) 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f296ffff640 1 -- 172.21.15.16:0/1840103312 learned_addr learned my addr 172.21.15.16:0/1840103312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7174d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f29755f7640 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c719c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 -- 172.21.15.16:0/1840103312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 msgr2=0x5586c719c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c719c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 -- 172.21.15.16:0/1840103312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 msgr2=0x5586c719c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c719c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.789+0000 7f2974df6640 1 -- 172.21.15.16:0/1840103312 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586c7176980 con 0x5586c7179760 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.790+0000 7f2974df6640 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7174d60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2958000c00 tx=0x7f295802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b6771a87b11ca9f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.790+0000 7f29667fc640 1 -- 172.21.15.16:0/1840103312 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958049040 con 0x5586c7179760 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.790+0000 7f29667fc640 1 -- 172.21.15.16:0/1840103312 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29580f5320 con 0x5586c7179760 2024-09-12T02:49:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.790+0000 7f29667fc640 1 -- 172.21.15.16:0/1840103312 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29580bbaf0 con 0x5586c7179760 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 -- 172.21.15.16:0/1840103312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 msgr2=0x5586c7174d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7174d60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2958000c00 tx=0x7f295802a480 comp rx=0 tx=0).stop 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 -- 172.21.15.16:0/1840103312 shutdown_connections 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c719c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c717a080 0x5586c719c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 --2- 172.21.15.16:0/1840103312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c7179760 0x5586c7174d60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 -- 172.21.15.16:0/1840103312 >> 172.21.15.16:0/1840103312 conn(0x5586c7175cf0 msgr2=0x5586c7174650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 -- 172.21.15.16:0/1840103312 shutdown_connections 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.791+0000 7f2979433740 1 -- 172.21.15.16:0/1840103312 wait complete. 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.792+0000 7f2979433740 1 -- start start 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.792+0000 7f2979433740 1 Processor -- start 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.792+0000 7f2979433740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.792+0000 7f2979433740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c717a080 0x5586c7188d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.792+0000 7f2979433740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c71892b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2979433740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29701b0b10 con 0x5586c7179760 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2979433740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29701b0440 con 0x5586c717a080 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2979433740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29701ad680 con 0x5586c7184940 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37700/0 (socket says 172.21.15.16:37700) 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 -- 172.21.15.16:0/2915065688 learned_addr learned my addr 172.21.15.16:0/2915065688 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f29755f7640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c71892b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f296ffff640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c717a080 0x5586c7188d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 -- 172.21.15.16:0/2915065688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 msgr2=0x5586c71892b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c71892b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 -- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c717a080 msgr2=0x5586c7188d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c717a080 0x5586c7188d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 -- 172.21.15.16:0/2915065688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586c717c5d0 con 0x5586c7179760 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2979433740 10 client.0 _ll_register_callbacks cb 0x7f2960011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29580f5320 con 0x5586c7179760 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f2974df6640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f29580ba260 tx=0x7f2958005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b522ccddaae75784 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.793+0000 7f294ffff640 10 client.15861 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.794+0000 7f294ffff640 1 -- 172.21.15.16:0/2915065688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958049050 con 0x5586c7179760 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.794+0000 7f294ffff640 1 -- 172.21.15.16:0/2915065688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f29580f5730 con 0x5586c7179760 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.795+0000 7f294ffff640 1 -- 172.21.15.16:0/2915065688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f29580bbaf0 con 0x5586c7179760 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.795+0000 7f2979433740 10 client.15861 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.795+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5586c717c5d0 con 0x5586c7179760 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f294ffff640 1 -- 172.21.15.16:0/2915065688 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f29580b8620 con 0x5586c7179760 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 10 client.15861 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 10 client.15861 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 20 client.15861 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7201f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 4 client.15861 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.4, "await_for": 45.0}] 2024-09-12T02:49:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.4, "await_for": 45.0}) -- 0x5586c70e6d20 con 0x5586c7194610 2024-09-12T02:49:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.796+0000 7f296ffff640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7201f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.797+0000 7f296ffff640 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7201f80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df26e6fee976b819 server_cookie=4824249c592fb9b0 in_seq=0 out_seq=0 2024-09-12T02:49:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.797+0000 7f294ffff640 10 client.15861 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f294ffff640 1 -- 172.21.15.16:0/2915065688 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5586c70e6d20 con 0x5586c7194610 2024-09-12T02:49:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f294ffff640 10 client.15861 handle_command_reply: tid=0 2024-09-12T02:49:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 client.15861 shutdown 2024-09-12T02:49:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 msgr2=0x5586c718ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f29580ba260 tx=0x7f2958005b90 comp rx=0 tx=0).stop 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 shutdown_connections 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586c7194610 0x5586c7201f80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586c7184940 0x5586c71892b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586c717a080 0x5586c7188d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f294ffff640 0 client.15861 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 --2- 172.21.15.16:0/2915065688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586c7179760 0x5586c718ca30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 >> 172.21.15.16:0/2915065688 conn(0x5586c7175cf0 msgr2=0x5586c7174e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 shutdown_connections 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.857+0000 7f2979433740 1 -- 172.21.15.16:0/2915065688 wait complete. 2024-09-12T02:49:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f2979433740 20 client.15861 trim_cache size 0 max 16384 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 30, 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 6, 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "d5cee633": { 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 6, 2024-09-12T02:49:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 133.4, 2024-09-12T02:49:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:39.863 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 msgr2=0x7f295c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 0x7f295c0a5040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2968009ee0 tx=0x7f2968005f90 comp rx=0 tx=0).stop 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 msgr2=0x7f2970197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2958006010 tx=0x7f2958003c10 comp rx=0 tx=0).stop 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 shutdown_connections 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2970102ef0 0x7f2970197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2970102570 0x7f2970197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f295c0a2b80 0x7f295c0a5040 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 --2- 172.21.15.16:0/2484058614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2970100e20 0x7f2970197110 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 >> 172.21.15.16:0/2484058614 conn(0x7f29700fc7d0 msgr2=0x7f29700fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.858+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 shutdown_connections 2024-09-12T02:49:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:39.859+0000 7f297705a640 1 -- 172.21.15.16:0/2484058614 wait complete. 2024-09-12T02:49:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:39 smithi033 ceph-mon[35092]: pgmap v389: 97 pgs: 97 active+clean; 62 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 218 op/s 2024-09-12T02:49:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:39 smithi184 ceph-mon[35321]: pgmap v389: 97 pgs: 97 active+clean; 62 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 218 op/s 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 -- 172.21.15.16:0/75328956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 msgr2=0x7f0a9c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 --2- 172.21.15.16:0/75328956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c1021c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009900 tx=0x7f0a8c02a410 comp rx=0 tx=0).stop 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 -- 172.21.15.16:0/75328956 shutdown_connections 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 --2- 172.21.15.16:0/75328956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c106f10 0x7f0a9c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 --2- 172.21.15.16:0/75328956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c102790 0x7f0a9c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 --2- 172.21.15.16:0/75328956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c1021c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 -- 172.21.15.16:0/75328956 >> 172.21.15.16:0/75328956 conn(0x7f0a9c0fd8c0 msgr2=0x7f0a9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 -- 172.21.15.16:0/75328956 shutdown_connections 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.481+0000 7f0aa1359640 1 -- 172.21.15.16:0/75328956 wait complete. 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 Processor -- start 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 -- start start 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c102790 0x7f0a9c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c106f10 0x7f0a9c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c111f30 con 0x7f0a9c101de0 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c111db0 con 0x7f0a9c102790 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0aa1359640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1120b0 con 0x7f0a9c106f10 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37718/0 (socket says 172.21.15.16:37718) 2024-09-12T02:49:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0a9affd640 1 -- 172.21.15.16:0/1845308572 learned_addr learned my addr 172.21.15.16:0/1845308572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c106f10 0x7f0a9c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.482+0000 7f0a93fff640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c102790 0x7f0a9c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 -- 172.21.15.16:0/1845308572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c106f10 msgr2=0x7f0a9c19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c106f10 0x7f0a9c19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 -- 172.21.15.16:0/1845308572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c102790 msgr2=0x7f0a9c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c102790 0x7f0a9c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 -- 172.21.15.16:0/1845308572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c19d3f0 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c005ab0 tx=0x7f0a8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a98ff9640 1 -- 172.21.15.16:0/1845308572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c03f030 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a98ff9640 1 -- 172.21.15.16:0/1845308572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8c003ce0 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c19d680 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0a98ff9640 1 -- 172.21.15.16:0/1845308572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c035070 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.483+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c1a4fc0 con 0x7f0a9c101de0 2024-09-12T02:49:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.485+0000 7f0a98ff9640 1 -- 172.21.15.16:0/1845308572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0a8c042030 con 0x7f0a9c101de0 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 Processor -- start 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 -- start start 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a98ff9640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 0x7f0a880a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a98ff9640 1 -- 172.21.15.16:0/1845308572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0a8c0ef920 con 0x7f0a9c101de0 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 0x7f0a880a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51ad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629d51062f0 con 0x5629d51a1b40 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629d50adcf0 con 0x5629d51a05d0 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0aa3732740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5629d51022b0 con 0x5629d51ab370 2024-09-12T02:49:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37724/0 (socket says 172.21.15.16:37724) 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.486+0000 7f0a9affd640 1 -- 172.21.15.16:0/534742612 learned_addr learned my addr 172.21.15.16:0/534742612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a93fff640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 -- 172.21.15.16:0/534742612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 msgr2=0x5629d51ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 -- 172.21.15.16:0/534742612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 msgr2=0x5629d51a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 -- 172.21.15.16:0/534742612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d50923a0 con 0x5629d51a1b40 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 0x7f0a880a50c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400ad90 tx=0x7f0a9400a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a9affd640 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0bc0c0 tx=0x7f0a8c03b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42fe7bc99723e99 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a917fa640 1 -- 172.21.15.16:0/534742612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c049040 con 0x5629d51a1b40 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a917fa640 1 -- 172.21.15.16:0/534742612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8c0bc130 con 0x5629d51a1b40 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.487+0000 7f0a917fa640 1 -- 172.21.15.16:0/534742612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0b87a0 con 0x5629d51a1b40 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- 172.21.15.16:0/534742612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 msgr2=0x5629d51a0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0bc0c0 tx=0x7f0a8c03b800 comp rx=0 tx=0).stop 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- 172.21.15.16:0/534742612 shutdown_connections 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 --2- 172.21.15.16:0/534742612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51a0090 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- 172.21.15.16:0/534742612 >> 172.21.15.16:0/534742612 conn(0x5629d5169310 msgr2=0x5629d519cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- 172.21.15.16:0/534742612 shutdown_connections 2024-09-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- 172.21.15.16:0/534742612 wait complete. 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 -- start start 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.488+0000 7f0aa3732740 1 Processor -- start 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a1b40 0x5629d51a8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629d51ade30 con 0x5629d51a05d0 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629d50c1060 con 0x5629d51a1b40 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5629d51adfb0 con 0x5629d51ab370 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 10 client.0 _ll_register_callbacks cb 0x5629d4e148c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37740/0 (socket says 172.21.15.16:37740) 2024-09-12T02:49:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 -- 172.21.15.16:0/4134344254 learned_addr learned my addr 172.21.15.16:0/4134344254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a93fff640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a1b40 0x5629d51a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 -- 172.21.15.16:0/4134344254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 msgr2=0x5629d51a87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51a87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 -- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a1b40 msgr2=0x5629d51a8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a1b40 0x5629d51a8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 -- 172.21.15.16:0/4134344254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629d4e773c0 con 0x5629d51a05d0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a9affd640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c4870 tx=0x7f0a8c0ed340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8d2c65be87c8e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a827fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c049030 con 0x5629d51a05d0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0a8c0c0120 con 0x5629d51a05d0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.489+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a8c0bc130 con 0x5629d51a05d0 2024-09-12T02:49:40.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a8c0c4da0 con 0x5629d51a05d0 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 10 client.15873 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629d51ade30 con 0x5629d51a05d0 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8c0bc130 con 0x5629d51a05d0 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0a8c0bc390 con 0x5629d51a05d0 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 10 client.15873 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 20 client.15873 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d51ba9f0 0x5629d51bce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 4 client.15873 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629d5113860 con 0x5629d51ba9f0 2024-09-12T02:49:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.490+0000 7f0a93fff640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d51ba9f0 0x5629d51bce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.491+0000 7f0a827fc640 10 client.15873 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.491+0000 7f0a93fff640 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d51ba9f0 0x5629d51bce90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=737d005ad9526a03 server_cookie=92b1d277661c5f0f in_seq=0 out_seq=0 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.491+0000 7f0a827fc640 1 -- 172.21.15.16:0/4134344254 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5629d5113860 con 0x5629d51ba9f0 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.491+0000 7f0a827fc640 10 client.15873 handle_command_reply: tid=0 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.491+0000 7f0aa3732740 1 client.15873 shutdown 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 msgr2=0x5629d51aadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c4870 tx=0x7f0a8c0ed340 comp rx=0 tx=0).stop 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 shutdown_connections 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d51ba9f0 0x5629d51bce90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51ab370 0x5629d51a87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a1b40 0x5629d51a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0a827fc640 0 client.15873 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 --2- 172.21.15.16:0/4134344254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a05d0 0x5629d51aadb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 >> 172.21.15.16:0/4134344254 conn(0x5629d5169310 msgr2=0x5629d51a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 shutdown_connections 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 1 -- 172.21.15.16:0/4134344254 wait complete. 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.492+0000 7f0aa3732740 20 client.15873 trim_cache size 0 max 16384 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 Processor -- start 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 -- start start 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a1b40 0x5629d51c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1ad540 con 0x5629d51ab370 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1af140 con 0x5629d51a05d0 2024-09-12T02:49:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0aa3732740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1adc10 con 0x5629d51a1b40 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37748/0 (socket says 172.21.15.16:37748) 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a9b7fe640 1 -- 172.21.15.16:0/1760114193 learned_addr learned my addr 172.21.15.16:0/1760114193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a93fff640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a1b40 0x5629d51c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.505+0000 7f0a9b7fe640 1 -- 172.21.15.16:0/1760114193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a1b40 msgr2=0x5629d51c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a1b40 0x5629d51c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9b7fe640 1 -- 172.21.15.16:0/1760114193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 msgr2=0x5629d51a7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9b7fe640 1 -- 172.21.15.16:0/1760114193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d51a65b0 con 0x5629d51ab370 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0a9401c750 tx=0x7f0a94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f61c5f13aa3042 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a90ff9640 1 -- 172.21.15.16:0/1760114193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94016070 con 0x5629d51ab370 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a90ff9640 1 -- 172.21.15.16:0/1760114193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a94003e50 con 0x5629d51ab370 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.506+0000 7f0a90ff9640 1 -- 172.21.15.16:0/1760114193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940043c0 con 0x5629d51ab370 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- 172.21.15.16:0/1760114193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 msgr2=0x5629d51c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0a9401c750 tx=0x7f0a94017000 comp rx=0 tx=0).stop 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- 172.21.15.16:0/1760114193 shutdown_connections 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51ab370 0x5629d51c2ca0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a1b40 0x5629d51c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- 172.21.15.16:0/1760114193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51a05d0 0x5629d51a7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- 172.21.15.16:0/1760114193 >> 172.21.15.16:0/1760114193 conn(0x5629d5169310 msgr2=0x5629d51a6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- 172.21.15.16:0/1760114193 shutdown_connections 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- 172.21.15.16:0/1760114193 wait complete. 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- start start 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 Processor -- start 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a05d0 0x5629d51b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 0x5629d51af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1ae9b0 con 0x5629d51a1b40 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1ae2e0 con 0x5629d51ab370 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.507+0000 7f0aa3732740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1ab520 con 0x5629d51a05d0 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0aa3732740 10 client.0 _ll_register_callbacks cb 0x7f0a8400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37754/0 (socket says 172.21.15.16:37754) 2024-09-12T02:49:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 -- 172.21.15.16:0/694370354 learned_addr learned my addr 172.21.15.16:0/694370354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a9affd640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a05d0 0x5629d51b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 0x5629d51af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 -- 172.21.15.16:0/694370354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a05d0 msgr2=0x5629d51b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a05d0 0x5629d51b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 -- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 msgr2=0x5629d51af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 0x5629d51af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 -- 172.21.15.16:0/694370354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629d51afae0 con 0x5629d51a1b40 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a9b7fe640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 0x5629d51af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a93fff640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x5629d519dac0 tx=0x7f0a84010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81726f5dfa1359d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a827fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84004220 con 0x5629d51a1b40 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0a840043c0 con 0x5629d51a1b40 2024-09-12T02:49:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a94003e50 con 0x5629d51a1b40 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.508+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a8402b050 con 0x5629d51a1b40 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 10 client.15879 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629d5225860 con 0x5629d51a1b40 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a94003e50 con 0x5629d51a1b40 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0a840047a0 con 0x5629d51a1b40 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 10 client.15879 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 20 client.15879 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d5226080 0x5629d52284a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 4 client.15879 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:49:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629d510dd70 con 0x5629d5226080 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0a9affd640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d5226080 0x5629d52284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0a9affd640 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d5226080 0x5629d52284a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c68644da1b9706 server_cookie=56eaec6fd6e70821 in_seq=0 out_seq=0 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.509+0000 7f0a827fc640 10 client.15879 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0a827fc640 1 -- 172.21.15.16:0/694370354 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629d510dd70 con 0x5629d5226080 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0a827fc640 10 client.15879 handle_command_reply: tid=0 2024-09-12T02:49:40.569 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 client.15879 shutdown 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 msgr2=0x5629d51aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x5629d519dac0 tx=0x7f0a84010af0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 shutdown_connections 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5629d5226080 0x5629d52284a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629d51ab370 0x5629d51af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629d51a1b40 0x5629d51aeca0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 --2- 172.21.15.16:0/694370354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5629d51a05d0 0x5629d51b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 >> 172.21.15.16:0/694370354 conn(0x5629d5169310 msgr2=0x5629d51a7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0a827fc640 0 client.15879 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 shutdown_connections 2024-09-12T02:49:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 1 -- 172.21.15.16:0/694370354 wait complete. 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.510+0000 7f0aa3732740 20 client.15879 trim_cache size 0 max 16384 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 msgr2=0x7f0a880a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 0x7f0a880a50c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400ad90 tx=0x7f0a9400a4d0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 msgr2=0x7f0a9c198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c005ab0 tx=0x7f0a8c0042f0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 shutdown_connections 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a9c106f10 0x7f0a9c19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a9c102790 0x7f0a9c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a880a2c00 0x7f0a880a50c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 --2- 172.21.15.16:0/1845308572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a9c101de0 0x7f0a9c198440 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 >> 172.21.15.16:0/1845308572 conn(0x7f0a9c0fd8c0 msgr2=0x7f0a9c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 shutdown_connections 2024-09-12T02:49:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:40.511+0000 7f0aa1359640 1 -- 172.21.15.16:0/1845308572 wait complete. 2024-09-12T02:49:40.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 13.37 MB/sec warmup 42 sec latency 850.930 ms 2024-09-12T02:49:41.351 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd5cee633', quiesce took 5.5 seconds. Will release after: 7.9 2024-09-12T02:49:41.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 13.06 MB/sec warmup 43 sec latency 1851.046 ms 2024-09-12T02:49:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:41 smithi033 ceph-mon[35092]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-09-12T02:49:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:41 smithi184 ceph-mon[35321]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-09-12T02:49:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:41 smithi016 ceph-mon[30925]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 312 op/s 2024-09-12T02:49:42.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 12.76 MB/sec warmup 44 sec latency 2851.173 ms 2024-09-12T02:49:42.778 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:49:42.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:49:43.058 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 -- 172.21.15.16:0/3849854461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 msgr2=0x7f8af410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 --2- 172.21.15.16:0/3849854461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af410e610 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009ca0 tx=0x7f8aec02a950 comp rx=0 tx=0).stop 2024-09-12T02:49:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 -- 172.21.15.16:0/3849854461 shutdown_connections 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 --2- 172.21.15.16:0/3849854461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af410e610 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 --2- 172.21.15.16:0/3849854461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 0x7f8af410bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 --2- 172.21.15.16:0/3849854461 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 0x7f8af41031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 -- 172.21.15.16:0/3849854461 >> 172.21.15.16:0/3849854461 conn(0x7f8af40fc7a0 msgr2=0x7f8af40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 -- 172.21.15.16:0/3849854461 shutdown_connections 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.532+0000 7f8afa068640 1 -- 172.21.15.16:0/3849854461 wait complete. 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 Processor -- start 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 -- start start 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 0x7f8af4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 0x7f8af4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8af4111f50 con 0x7f8af410c180 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8af4111dd0 con 0x7f8af4102de0 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8afa068640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41120d0 con 0x7f8af4103700 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8af3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37778/0 (socket says 172.21.15.16:37778) 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8af3fff640 1 -- 172.21.15.16:0/302956294 learned_addr learned my addr 172.21.15.16:0/302956294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 0x7f8af4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.533+0000 7f8af37fe640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 0x7f8af4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 -- 172.21.15.16:0/302956294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 msgr2=0x7f8af4198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 0x7f8af4198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 -- 172.21.15.16:0/302956294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 msgr2=0x7f8af4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 0x7f8af4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 -- 172.21.15.16:0/302956294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af419d3c0 con 0x7f8af410c180 2024-09-12T02:49:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af3fff640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8aec02ae70 tx=0x7f8aec02f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af0ff9640 1 -- 172.21.15.16:0/302956294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec034040 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af0ff9640 1 -- 172.21.15.16:0/302956294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aec030920 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af419d650 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8af0ff9640 1 -- 172.21.15.16:0/302956294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec045480 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.534+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af41a4f90 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.536+0000 7f8af0ff9640 1 -- 172.21.15.16:0/302956294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8aec037020 con 0x7f8af410c180 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.536+0000 7f8afc441740 1 Processor -- start 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 -- start start 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c3915cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c39167800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c390c2280 con 0x558c39167d40 2024-09-12T02:49:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c39069b10 con 0x558c3915cb60 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8afc441740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c390be240 con 0x558c3915d480 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37790/0 (socket says 172.21.15.16:37790) 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 -- 172.21.15.16:0/2374991306 learned_addr learned my addr 172.21.15.16:0/2374991306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af0ff9640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 0x7f8adc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af37fe640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c39167800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c3915cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af0ff9640 1 -- 172.21.15.16:0/302956294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8aec0efe80 con 0x7f8af410c180 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af3fff640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 0x7f8adc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 -- 172.21.15.16:0/2374991306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 msgr2=0x558c39167800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c39167800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 -- 172.21.15.16:0/2374991306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 msgr2=0x558c3915cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c3915cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af2ffd640 1 -- 172.21.15.16:0/2374991306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c390761f0 con 0x558c39167d40 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.537+0000 7f8af37fe640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c39167800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8012720 tx=0x7f8ae800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b474f8fe068913 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8ad9ffb640 1 -- 172.21.15.16:0/2374991306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8012ca0 con 0x558c39167d40 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8af3fff640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 0x7f8adc0a5040 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8aec004af0 tx=0x7f8aec003f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8ad9ffb640 1 -- 172.21.15.16:0/2374991306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ae8012e40 con 0x558c39167d40 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8ad9ffb640 1 -- 172.21.15.16:0/2374991306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8004a90 con 0x558c39167d40 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 -- 172.21.15.16:0/2374991306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 msgr2=0x558c3916a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8012720 tx=0x7f8ae800b6e0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 -- 172.21.15.16:0/2374991306 shutdown_connections 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c39167d40 0x558c3916a130 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c39167800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 --2- 172.21.15.16:0/2374991306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c3915cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.538+0000 7f8afc441740 1 -- 172.21.15.16:0/2374991306 >> 172.21.15.16:0/2374991306 conn(0x558c391590f0 msgr2=0x558c391594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- 172.21.15.16:0/2374991306 shutdown_connections 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- 172.21.15.16:0/2374991306 wait complete. 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- start start 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 Processor -- start 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c391651a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c391656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c3916a910 con 0x558c3915cb60 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c3907cf10 con 0x558c39167d40 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c3916aa90 con 0x558c3915d480 2024-09-12T02:49:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8afc441740 10 client.0 _ll_register_callbacks cb 0x558c39045a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.539+0000 7f8af2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37806/0 (socket says 172.21.15.16:37806) 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 -- 172.21.15.16:0/1388247085 learned_addr learned my addr 172.21.15.16:0/1388247085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af37fe640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c391651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 -- 172.21.15.16:0/1388247085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 msgr2=0x558c391651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c391651a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 -- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 msgr2=0x558c391656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c391656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 -- 172.21.15.16:0/1388247085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c38e336a0 con 0x558c3915cb60 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8004770 tx=0x7f8ae8005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1be5743a7af771c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8012ca0 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c390cf7f0 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8ae8004460 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ae8005250 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8afc441740 10 client.15891 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.540+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c3917a990 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558c390cf7f0 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8ae8010ce0 con 0x558c3915cb60 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 10 client.15891 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 20 client.15891 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c3917bce0 0x558c3917e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 4 client.15891 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c390c9d00 con 0x558c3917bce0 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.541+0000 7f8af37fe640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c3917bce0 0x558c3917e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8af37fe640 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c3917bce0 0x558c3917e0d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23caa9d98d800d62 server_cookie=8b29ef30af874fcf in_seq=0 out_seq=0 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8ad2ffd640 10 client.15891 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/1388247085 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c390c9d00 con 0x558c3917bce0 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8ad2ffd640 10 client.15891 handle_command_reply: tid=0 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8afc441740 1 client.15891 shutdown 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 msgr2=0x558c39164c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.542+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8004770 tx=0x7f8ae8005fa0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 shutdown_connections 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c3917bce0 0x558c3917e0d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c391656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915d480 0x558c391651a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 --2- 172.21.15.16:0/1388247085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915cb60 0x558c39164c60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 >> 172.21.15.16:0/1388247085 conn(0x558c391590f0 msgr2=0x558c3915edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8ad2ffd640 0 client.15891 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 shutdown_connections 2024-09-12T02:49:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 1 -- 172.21.15.16:0/1388247085 wait complete. 2024-09-12T02:49:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.543+0000 7f8afc441740 20 client.15891 trim_cache size 0 max 16384 2024-09-12T02:49:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.560+0000 7f8afc441740 1 Processor -- start 2024-09-12T02:49:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.560+0000 7f8afc441740 1 -- start start 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c39166ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 0x558c39183e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41ab2d0 con 0x558c3915d480 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41aced0 con 0x558c3915cb60 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8afc441740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41ab9a0 con 0x558c39167d40 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37808/0 (socket says 172.21.15.16:37808) 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 -- 172.21.15.16:0/3465310282 learned_addr learned my addr 172.21.15.16:0/3465310282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c39166ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 0x558c39183e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 -- 172.21.15.16:0/3465310282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 msgr2=0x558c39183e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 0x558c39183e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 -- 172.21.15.16:0/3465310282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 msgr2=0x558c39166ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c39166ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 -- 172.21.15.16:0/3465310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c39163c10 con 0x558c3915d480 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 0x558c39183e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8af37fe640 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x558c391644f0 tx=0x7f8ae400a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ee03814e9c1448 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8ad97fa640 1 -- 172.21.15.16:0/3465310282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4004220 con 0x558c3915d480 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8ad97fa640 1 -- 172.21.15.16:0/3465310282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ae40043c0 con 0x558c3915d480 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.561+0000 7f8ad97fa640 1 -- 172.21.15.16:0/3465310282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae401ea30 con 0x558c3915d480 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- 172.21.15.16:0/3465310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 msgr2=0x558c391838e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x558c391644f0 tx=0x7f8ae400a2d0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- 172.21.15.16:0/3465310282 shutdown_connections 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c39167d40 0x558c39183e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c391838e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 --2- 172.21.15.16:0/3465310282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c3915cb60 0x558c39166ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- 172.21.15.16:0/3465310282 >> 172.21.15.16:0/3465310282 conn(0x558c391590f0 msgr2=0x558c391663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- 172.21.15.16:0/3465310282 shutdown_connections 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- 172.21.15.16:0/3465310282 wait complete. 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.562+0000 7f8afc441740 1 -- start start 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 Processor -- start 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915cb60 0x558c39174930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c39161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41ac740 con 0x558c3915d480 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41ac070 con 0x558c39167d40 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8af41a92b0 con 0x558c3915cb60 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8afc441740 10 client.0 _ll_register_callbacks cb 0x7f8ae800bdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37820/0 (socket says 172.21.15.16:37820) 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 -- 172.21.15.16:0/255171227 learned_addr learned my addr 172.21.15.16:0/255171227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c39161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 -- 172.21.15.16:0/255171227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915cb60 msgr2=0x558c39174930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915cb60 0x558c39174930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 -- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 msgr2=0x558c39161ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c39161ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af37fe640 1 -- 172.21.15.16:0/255171227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c39162820 con 0x558c3915d480 2024-09-12T02:49:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.563+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c39161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8af37fe640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4004250 tx=0x7f8ae4004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f86d851fdadd5e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4033020 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8ae40037f0 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ae40043c0 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ae402b040 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 10 client.15897 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c391e68d0 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ae40043c0 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8ae402e030 con 0x558c3915d480 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 10 client.15897 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 20 client.15897 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c391e70e0 0x558c391e9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 4 client.15897 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.564+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558c39046110 con 0x558c391e70e0 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.565+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c391e70e0 0x558c391e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.565+0000 7f8af2ffd640 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c391e70e0 0x558c391e9500 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb4f5f95a2dee538 server_cookie=a95d85bbd07a975 in_seq=0 out_seq=0 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.565+0000 7f8ad2ffd640 10 client.15897 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.566+0000 7f8ad2ffd640 1 -- 172.21.15.16:0/255171227 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558c39046110 con 0x558c391e70e0 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.566+0000 7f8ad2ffd640 10 client.15897 handle_command_reply: tid=0 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.566+0000 7f8afc441740 1 client.15897 shutdown 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 msgr2=0x558c39161a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4004250 tx=0x7f8ae4004280 comp rx=0 tx=0).stop 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 shutdown_connections 2024-09-12T02:49:43.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c391e70e0 0x558c391e9500 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c39167d40 0x558c39161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8ad2ffd640 0 client.15897 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c3915d480 0x558c39161a50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 --2- 172.21.15.16:0/255171227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c3915cb60 0x558c39174930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 >> 172.21.15.16:0/255171227 conn(0x558c391590f0 msgr2=0x558c39158270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 shutdown_connections 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 1 -- 172.21.15.16:0/255171227 wait complete. 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afc441740 20 client.15897 trim_cache size 0 max 16384 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:49:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f99a3b26-ce16-4b13-a1f7-d4bf3d6b5fef", 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 msgr2=0x7f8adc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.567+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 0x7f8adc0a5040 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8aec004af0 tx=0x7f8aec003f10 comp rx=0 tx=0).stop 2024-09-12T02:49:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 msgr2=0x7f8af419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8aec02ae70 tx=0x7f8aec02f5b0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 shutdown_connections 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8adc0a2b80 0x7f8adc0a5040 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8af410c180 0x7f8af419cce0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8af4103700 0x7f8af4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 --2- 172.21.15.16:0/302956294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8af4102de0 0x7f8af4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 >> 172.21.15.16:0/302956294 conn(0x7f8af40fc7a0 msgr2=0x7f8af40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 shutdown_connections 2024-09-12T02:49:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:43.568+0000 7f8afa068640 1 -- 172.21.15.16:0/302956294 wait complete. 2024-09-12T02:49:43.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 12.48 MB/sec warmup 45 sec latency 3851.290 ms 2024-09-12T02:49:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:43 smithi033 ceph-mon[35092]: pgmap v391: 97 pgs: 97 active+clean; 65 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 257 op/s 2024-09-12T02:49:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:43 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:49:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:43 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:43 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:49:43.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:43 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:43 smithi184 ceph-mon[35321]: pgmap v391: 97 pgs: 97 active+clean; 65 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 257 op/s 2024-09-12T02:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:43 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:43 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:43 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:43 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:43 smithi016 ceph-mon[30925]: pgmap v391: 97 pgs: 97 active+clean; 65 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 257 op/s 2024-09-12T02:49:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:43 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:49:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:43 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:43 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:49:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:43 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:49:44.465 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:44.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 12.21 MB/sec warmup 46 sec latency 4851.375 ms 2024-09-12T02:49:44.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:49:44.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.963+0000 7fe6eaa47640 1 -- 172.21.15.16:0/2395518153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 msgr2=0x7fe6e4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.963+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/2395518153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e4106e60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009930 tx=0x7fe6d402a420 comp rx=0 tx=0).stop 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 -- 172.21.15.16:0/2395518153 shutdown_connections 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/2395518153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4101420 0x7fe6e410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/2395518153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4100a80 0x7fe6e4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/2395518153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e4106e60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 -- 172.21.15.16:0/2395518153 >> 172.21.15.16:0/2395518153 conn(0x7fe6e40fc7a0 msgr2=0x7fe6e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 -- 172.21.15.16:0/2395518153 shutdown_connections 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 -- 172.21.15.16:0/2395518153 wait complete. 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 Processor -- start 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.964+0000 7fe6eaa47640 1 -- start start 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4100a80 0x7fe6e4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4101420 0x7fe6e41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e4110e80 con 0x7fe6e4106a80 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e4110d00 con 0x7fe6e4101420 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e4111000 con 0x7fe6e4100a80 2024-09-12T02:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37838/0 (socket says 172.21.15.16:37838) 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 -- 172.21.15.16:0/3194459386 learned_addr learned my addr 172.21.15.16:0/3194459386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4100a80 0x7fe6e4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4101420 0x7fe6e41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 -- 172.21.15.16:0/3194459386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4100a80 msgr2=0x7fe6e4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4100a80 0x7fe6e4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 -- 172.21.15.16:0/3194459386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4101420 msgr2=0x7fe6e41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4101420 0x7fe6e41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 -- 172.21.15.16:0/3194459386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e419d410 con 0x7fe6e4106a80 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e8fe4640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00eaa0 tx=0x7fe6dc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e17fa640 1 -- 172.21.15.16:0/3194459386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00cdb0 con 0x7fe6e4106a80 2024-09-12T02:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e419d700 con 0x7fe6e4106a80 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e17fa640 1 -- 172.21.15.16:0/3194459386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6dc005020 con 0x7fe6e4106a80 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.965+0000 7fe6e17fa640 1 -- 172.21.15.16:0/3194459386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0116b0 con 0x7fe6e4106a80 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.966+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e41a4fe0 con 0x7fe6e4106a80 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.968+0000 7fe6e17fa640 1 -- 172.21.15.16:0/3194459386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe6dc005560 con 0x7fe6e4106a80 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.968+0000 7fe6ece20740 1 Processor -- start 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.968+0000 7fe6ece20740 1 -- start start 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.968+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b54a0 0x557f025b5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.968+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025c0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6ece20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f0251b2e0 con 0x557f025c0680 2024-09-12T02:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6ece20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f024c2ce0 con 0x557f025b5dc0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6ece20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557f025172a0 con 0x557f025b54a0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e17fa640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 0x7fe6cc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37846/0 (socket says 172.21.15.16:37846) 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 -- 172.21.15.16:0/1441136439 learned_addr learned my addr 172.21.15.16:0/1441136439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e8fe4640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 0x7fe6cc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b54a0 0x557f025b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025c0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e17fa640 1 -- 172.21.15.16:0/3194459386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe6dc0cf270 con 0x7fe6e4106a80 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 -- 172.21.15.16:0/1441136439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b54a0 msgr2=0x557f025b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b54a0 0x557f025b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 -- 172.21.15.16:0/1441136439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 msgr2=0x557f025c0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025c0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 -- 172.21.15.16:0/1441136439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f024a7440 con 0x557f025c0680 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e8fe4640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 0x7fe6cc0a5040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc097ee0 tx=0x7fe6dc066700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.969+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80149b0 tx=0x7fe6d800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7bf373696b486a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ca7fc640 1 -- 172.21.15.16:0/1441136439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80142d0 con 0x557f025c0680 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ca7fc640 1 -- 172.21.15.16:0/1441136439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6d8005500 con 0x557f025c0680 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ca7fc640 1 -- 172.21.15.16:0/1441136439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d800f760 con 0x557f025c0680 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 -- 172.21.15.16:0/1441136439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 msgr2=0x557f025c2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80149b0 tx=0x7fe6d800c4b0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 -- 172.21.15.16:0/1441136439 shutdown_connections 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025c2a70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025c0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 --2- 172.21.15.16:0/1441136439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b54a0 0x557f025b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 -- 172.21.15.16:0/1441136439 >> 172.21.15.16:0/1441136439 conn(0x557f025b1a30 msgr2=0x557f025b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 -- 172.21.15.16:0/1441136439 shutdown_connections 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.970+0000 7fe6ece20740 1 -- 172.21.15.16:0/1441136439 wait complete. 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 -- start start 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 Processor -- start 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025bd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f025c3250 con 0x557f025b54a0 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f024d6050 con 0x557f025b5dc0 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557f025c33d0 con 0x557f025c0680 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6ece20740 10 client.0 _ll_register_callbacks cb 0x557f02516160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37858/0 (socket says 172.21.15.16:37858) 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.971+0000 7fe6e37fe640 1 -- 172.21.15.16:0/109053262 learned_addr learned my addr 172.21.15.16:0/109053262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 -- 172.21.15.16:0/109053262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 msgr2=0x557f025bd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025bd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 -- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 msgr2=0x557f025bd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025bd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 -- 172.21.15.16:0/109053262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f0228c3c0 con 0x557f025b54a0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8005bc0 tx=0x7fe6d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3c2d93c8b9b283 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8028030 con 0x557f025b54a0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe6d8004030 con 0x557f025b54a0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6d8005500 con 0x557f025b54a0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe6d8004230 con 0x557f025b54a0 2024-09-12T02:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6ece20740 10 client.15909 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.972+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557f025cec40 con 0x557f025b54a0 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6d8005500 con 0x557f025b54a0 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fe6d800fa00 con 0x557f025b54a0 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 10 client.15909 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 20 client.15909 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f025d0040 0x557f025d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 4 client.15909 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f02528850 con 0x557f025d0040 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.973+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f025d0040 0x557f025d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6c37fe640 10 client.15909 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f025d0040 0x557f025d2430 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66f17f595635c61 server_cookie=1825c8c6d0aa38f2 in_seq=0 out_seq=0 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6c37fe640 1 -- 172.21.15.16:0/109053262 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f02528850 con 0x557f025d0040 2024-09-12T02:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6c37fe640 10 client.15909 handle_command_reply: tid=0 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6ece20740 1 client.15909 shutdown 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 msgr2=0x557f025c0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.974+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8005bc0 tx=0x7fe6d800cc40 comp rx=0 tx=0).stop 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 shutdown_connections 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f025d0040 0x557f025d2430 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025bd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b5dc0 0x557f025bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 --2- 172.21.15.16:0/109053262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b54a0 0x557f025c0010 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6c37fe640 0 client.15909 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 >> 172.21.15.16:0/109053262 conn(0x557f025b1a30 msgr2=0x557f025b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 shutdown_connections 2024-09-12T02:49:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 1 -- 172.21.15.16:0/109053262 wait complete. 2024-09-12T02:49:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.975+0000 7fe6ece20740 20 client.15909 trim_cache size 0 max 16384 2024-09-12T02:49:45.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:49:45.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 Processor -- start 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 -- start start 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b5dc0 0x557f025d7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41ad680 con 0x557f025c0680 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41af280 con 0x557f025b54a0 2024-09-12T02:49:45.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.992+0000 7fe6ece20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41add50 con 0x557f025b5dc0 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37866/0 (socket says 172.21.15.16:37866) 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 -- 172.21.15.16:0/889754128 learned_addr learned my addr 172.21.15.16:0/889754128 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b5dc0 0x557f025d7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 -- 172.21.15.16:0/889754128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b5dc0 msgr2=0x557f025d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b5dc0 0x557f025d7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 -- 172.21.15.16:0/889754128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 msgr2=0x557f025cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 -- 172.21.15.16:0/889754128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f025b26c0 con 0x557f025c0680 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800cc10 tx=0x7fe6d8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba535f8e863a81 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6c9ffb640 1 -- 172.21.15.16:0/889754128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8028040 con 0x557f025c0680 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6c9ffb640 1 -- 172.21.15.16:0/889754128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6d800fc10 con 0x557f025c0680 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.993+0000 7fe6c9ffb640 1 -- 172.21.15.16:0/889754128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d802f480 con 0x557f025c0680 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- 172.21.15.16:0/889754128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 msgr2=0x557f025d7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800cc10 tx=0x7fe6d8001e50 comp rx=0 tx=0).stop 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- 172.21.15.16:0/889754128 shutdown_connections 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025c0680 0x557f025d7fd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025b5dc0 0x557f025d7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 --2- 172.21.15.16:0/889754128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- 172.21.15.16:0/889754128 >> 172.21.15.16:0/889754128 conn(0x557f025b1a30 msgr2=0x557f025b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- 172.21.15.16:0/889754128 shutdown_connections 2024-09-12T02:49:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- 172.21.15.16:0/889754128 wait complete. 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.994+0000 7fe6ece20740 1 -- start start 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 Processor -- start 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41aeaf0 con 0x557f025b5dc0 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41ae420 con 0x557f025b54a0 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e41ab660 con 0x557f025c0680 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6ece20740 10 client.0 _ll_register_callbacks cb 0x7fe6d800cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37880/0 (socket says 172.21.15.16:37880) 2024-09-12T02:49:45.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 -- 172.21.15.16:0/633475183 learned_addr learned my addr 172.21.15.16:0/633475183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 -- 172.21.15.16:0/633475183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 msgr2=0x557f025c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 -- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 msgr2=0x557f025c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e3fff640 1 -- 172.21.15.16:0/633475183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f025c4a50 con 0x557f025b5dc0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.995+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6e3fff640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4006010 tx=0x7fe6d40319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612231b113519c0b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4035040 con 0x557f025b5dc0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe6d4004230 con 0x557f025b5dc0 2024-09-12T02:49:45.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f02522d60 con 0x557f025b5dc0 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe6d4031a60 con 0x557f025b5dc0 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 10 client.15915 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557f0263a930 con 0x557f025b5dc0 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557f02522d60 con 0x557f025b5dc0 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fe6d403d610 con 0x557f025b5dc0 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 10 client.15915 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.996+0000 7fe6ece20740 20 client.15915 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f0263b150 0x557f0263d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 4 client.15915 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f0249f210 con 0x557f0263b150 2024-09-12T02:49:45.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f0263b150 0x557f0263d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6e37fe640 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f0263b150 0x557f0263d570 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30f9af35ecd5fd25 server_cookie=83c8b84dce57419f in_seq=0 out_seq=0 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6c37fe640 10 client.15915 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6c37fe640 1 -- 172.21.15.16:0/633475183 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557f0249f210 con 0x557f0263b150 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6c37fe640 10 client.15915 handle_command_reply: tid=0 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 1 client.15915 shutdown 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 msgr2=0x557f025c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.997+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4006010 tx=0x7fe6d40319f0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 shutdown_connections 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557f0263b150 0x557f0263d570 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557f025c0680 0x557f025c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f025b5dc0 0x557f025c3ed0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6c37fe640 0 client.15915 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 --2- 172.21.15.16:0/633475183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f025b54a0 0x557f025c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 >> 172.21.15.16:0/633475183 conn(0x557f025b1a30 msgr2=0x557f025bba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 shutdown_connections 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 1 -- 172.21.15.16:0/633475183 wait complete. 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6ece20740 20 client.15915 trim_cache size 0 max 16384 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "f99a3b26-ce16-4b13-a1f7-d4bf3d6b5fef": { 2024-09-12T02:49:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:49:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f99a3b26-ce16-4b13-a1f7-d4bf3d6b5fef", 2024-09-12T02:49:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:49:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:45.017 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 msgr2=0x7fe6cc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 0x7fe6cc0a5040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc097ee0 tx=0x7fe6dc066700 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 msgr2=0x7fe6e419cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00eaa0 tx=0x7fe6dc00ef60 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 shutdown_connections 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe6cc0a2b80 0x7fe6cc0a5040 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e4106a80 0x7fe6e419cd30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.998+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe6e4101420 0x7fe6e41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.999+0000 7fe6eaa47640 1 --2- 172.21.15.16:0/3194459386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6e4100a80 0x7fe6e4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.999+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 >> 172.21.15.16:0/3194459386 conn(0x7fe6e40fc7a0 msgr2=0x7fe6e406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.999+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 shutdown_connections 2024-09-12T02:49:45.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:44.999+0000 7fe6eaa47640 1 -- 172.21.15.16:0/3194459386 wait complete. 2024-09-12T02:49:45.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:49:45.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:49:45.657 INFO:tasks.cephfs.filesystem:scrub status for tag:f99a3b26-ce16-4b13-a1f7-d4bf3d6b5fef - {'path': '/', 'tag': 'f99a3b26-ce16-4b13-a1f7-d4bf3d6b5fef', 'options': 'recursive,force'} 2024-09-12T02:49:45.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 11.95 MB/sec warmup 47 sec latency 5851.456 ms 2024-09-12T02:49:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:45 smithi016 ceph-mon[30925]: pgmap v392: 97 pgs: 97 active+clean; 66 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T02:49:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:45 smithi033 ceph-mon[35092]: pgmap v392: 97 pgs: 97 active+clean; 66 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T02:49:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:45 smithi184 ceph-mon[35321]: pgmap v392: 97 pgs: 97 active+clean; 66 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T02:49:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 11.70 MB/sec warmup 48 sec latency 6851.577 ms 2024-09-12T02:49:47.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 11.46 MB/sec warmup 49 sec latency 7851.698 ms 2024-09-12T02:49:47.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:47 smithi033 ceph-mon[35092]: pgmap v393: 97 pgs: 97 active+clean; 66 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 237 op/s 2024-09-12T02:49:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:47 smithi184 ceph-mon[35321]: pgmap v393: 97 pgs: 97 active+clean; 66 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 237 op/s 2024-09-12T02:49:48.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:47 smithi016 ceph-mon[30925]: pgmap v393: 97 pgs: 97 active+clean; 66 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 237 op/s 2024-09-12T02:49:48.707 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 11.23 MB/sec warmup 50 sec latency 8851.833 ms 2024-09-12T02:49:49.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:49:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:49.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 11.01 MB/sec warmup 51 sec latency 9851.932 ms 2024-09-12T02:49:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:49 smithi016 ceph-mon[30925]: pgmap v394: 97 pgs: 97 active+clean; 66 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-09-12T02:49:49.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:49 smithi033 ceph-mon[35092]: pgmap v394: 97 pgs: 97 active+clean; 66 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-09-12T02:49:50.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:49 smithi184 ceph-mon[35321]: pgmap v394: 97 pgs: 97 active+clean; 66 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-09-12T02:49:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 -- 172.21.15.16:0/1410894124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8100a80 msgr2=0x7f5ff8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1410894124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8100a80 0x7f5ff8100ee0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009930 tx=0x7f5fec02a420 comp rx=0 tx=0).stop 2024-09-12T02:49:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 -- 172.21.15.16:0/1410894124 shutdown_connections 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1410894124 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8101420 0x7f5ff810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1410894124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8100a80 0x7f5ff8100ee0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1410894124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 -- 172.21.15.16:0/1410894124 >> 172.21.15.16:0/1410894124 conn(0x7f5ff80fc7a0 msgr2=0x7f5ff80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 -- 172.21.15.16:0/1410894124 shutdown_connections 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.086+0000 7f5ffe375640 1 -- 172.21.15.16:0/1410894124 wait complete. 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 Processor -- start 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 -- start start 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8100a80 0x7f5ff8102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff81030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8110e80 con 0x7f5ff8101420 2024-09-12T02:49:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8110d00 con 0x7f5ff8100a80 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ffe375640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8111000 con 0x7f5ff8106a80 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ff77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ff77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48672/0 (socket says 172.21.15.16:48672) 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ff77fe640 1 -- 172.21.15.16:0/1844391911 learned_addr learned my addr 172.21.15.16:0/1844391911 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.087+0000 7f5ff7fff640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8100a80 0x7f5ff8102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ffc912640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff81030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 -- 172.21.15.16:0/1844391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 msgr2=0x7f5ff81030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff81030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 -- 172.21.15.16:0/1844391911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8100a80 msgr2=0x7f5ff8102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8100a80 0x7f5ff8102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 -- 172.21.15.16:0/1844391911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff81a6930 con 0x7f5ff8101420 2024-09-12T02:49:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ffc912640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff77fe640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009230 tx=0x7f5fec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec03f030 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fec003ad0 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff81a6bc0 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec035070 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.088+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff81a70a0 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.089+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff8100ee0 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.090+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5fec042030 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.091+0000 7f5ff57fa640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 0x7f5fe00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.091+0000 7f5ff7fff640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 0x7f5fe00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.091+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5fec0ef8e0 con 0x7f5ff8101420 2024-09-12T02:49:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.092+0000 7f5ff7fff640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 0x7f5fe00a5040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8005fd0 tx=0x7f5fe800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.095+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5fec0bc850 con 0x7f5ff8101420 2024-09-12T02:49:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.333+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ff81a7270 con 0x7f5ff8101420 2024-09-12T02:49:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.334+0000 7f5ff57fa640 1 -- 172.21.15.16:0/1844391911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7f5fec0bc150 con 0x7f5ff8101420 2024-09-12T02:49:50.336 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:49:50.336 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:49:50.336 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.336+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 msgr2=0x7f5fe00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.336+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 0x7f5fe00a5040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8005fd0 tx=0x7f5fe800b390 comp rx=0 tx=0).stop 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.336+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 msgr2=0x7f5ff8102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.336+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009230 tx=0x7f5fec004270 comp rx=0 tx=0).stop 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 shutdown_connections 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ff8106a80 0x7f5ff81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fe00a2b80 0x7f5fe00a5040 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ff8101420 0x7f5ff8102b90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 --2- 172.21.15.16:0/1844391911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ff8100a80 0x7f5ff8102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:50.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 >> 172.21.15.16:0/1844391911 conn(0x7f5ff80fc7a0 msgr2=0x7f5ff81048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:50.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 shutdown_connections 2024-09-12T02:49:50.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:50.337+0000 7f5ffe375640 1 -- 172.21.15.16:0/1844391911 wait complete. 2024-09-12T02:49:50.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 10.80 MB/sec warmup 52 sec latency 10852.047 ms 2024-09-12T02:49:50.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:50 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1844391911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:51.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:50 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1844391911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:51.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:50 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1844391911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:49:51.114 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id d5cee633 --release --await' 2024-09-12T02:49:51.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id d5cee633 --release --await 2024-09-12T02:49:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:49:51.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100764 10.59 MB/sec warmup 53 sec latency 11852.154 ms 2024-09-12T02:49:51.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:51 smithi016 ceph-mon[30925]: pgmap v395: 97 pgs: 97 active+clean; 66 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-09-12T02:49:51.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- 172.21.15.16:0/29974458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044106f10 msgr2=0x7f904410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 --2- 172.21.15.16:0/29974458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044106f10 0x7f904410ea90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f903c009ca0 tx=0x7f903c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- 172.21.15.16:0/29974458 shutdown_connections 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 --2- 172.21.15.16:0/29974458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044106f10 0x7f904410ea90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 --2- 172.21.15.16:0/29974458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 0x7f90441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 --2- 172.21.15.16:0/29974458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044101de0 0x7f90441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- 172.21.15.16:0/29974458 >> 172.21.15.16:0/29974458 conn(0x7f90440fd8c0 msgr2=0x7f90440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- 172.21.15.16:0/29974458 shutdown_connections 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- 172.21.15.16:0/29974458 wait complete. 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 Processor -- start 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.843+0000 7f904b197640 1 -- start start 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 0x7f9044074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044106f10 0x7f9044074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9044112010 con 0x7f9044101de0 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9044111e90 con 0x7f9044106f10 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f904b197640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9044112190 con 0x7f9044102790 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48688/0 (socket says 172.21.15.16:48688) 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 -- 172.21.15.16:0/1038310172 learned_addr learned my addr 172.21.15.16:0/1038310172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9043fff640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 0x7f9044074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 -- 172.21.15.16:0/1038310172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 msgr2=0x7f9044074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9049734640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044106f10 0x7f9044074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 0x7f9044074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 -- 172.21.15.16:0/1038310172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044106f10 msgr2=0x7f9044074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044106f10 0x7f9044074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 -- 172.21.15.16:0/1038310172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90441a6a30 con 0x7f9044101de0 2024-09-12T02:49:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9048f33640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f903400b4a0 tx=0x7f903400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.844+0000 7f9041ffb640 1 -- 172.21.15.16:0/1038310172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034015020 con 0x7f9044101de0 2024-09-12T02:49:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.845+0000 7f9041ffb640 1 -- 172.21.15.16:0/1038310172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9034011d10 con 0x7f9044101de0 2024-09-12T02:49:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.845+0000 7f9041ffb640 1 -- 172.21.15.16:0/1038310172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903400f2e0 con 0x7f9044101de0 2024-09-12T02:49:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.845+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90441a6d20 con 0x7f9044101de0 2024-09-12T02:49:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.845+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90441a71b0 con 0x7f9044101de0 2024-09-12T02:49:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.847+0000 7f9041ffb640 1 -- 172.21.15.16:0/1038310172 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9034018020 con 0x7f9044101de0 2024-09-12T02:49:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 Processor -- start 2024-09-12T02:49:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 -- start start 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78060 0x56355dc78440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9041ffb640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 0x7f902c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 0x56355dc85630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56355dbdd620 con 0x56355dc78980 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56355db84e50 con 0x56355dc78060 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f904d570740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56355dbd95e0 con 0x56355dc83240 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9048f33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9041ffb640 1 -- 172.21.15.16:0/1038310172 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f90340ce980 con 0x7f9044101de0 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9048f33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48700/0 (socket says 172.21.15.16:48700) 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9048f33640 1 -- 172.21.15.16:0/3282532963 learned_addr learned my addr 172.21.15.16:0/3282532963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9048f33640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 0x7f902c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9043fff640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 0x56355dc85630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.848+0000 7f9049734640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78060 0x56355dc78440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 -- 172.21.15.16:0/3282532963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78060 msgr2=0x56355dc78440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78060 0x56355dc78440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 -- 172.21.15.16:0/3282532963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 msgr2=0x56355dc82d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 -- 172.21.15.16:0/3282532963 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56355db75d40 con 0x56355dc83240 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9048f33640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9043fff640 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 0x56355dc85630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9038009970 tx=0x7f9038009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a976fe486087557a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f9048f33640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 0x7f902c0a5040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9034096e70 tx=0x7f903409ae00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f902affd640 1 -- 172.21.15.16:0/3282532963 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038017070 con 0x56355dc83240 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f902affd640 1 -- 172.21.15.16:0/3282532963 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9038013470 con 0x56355dc83240 2024-09-12T02:49:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.849+0000 7f902affd640 1 -- 172.21.15.16:0/3282532963 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038012630 con 0x56355dc83240 2024-09-12T02:49:51.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- 172.21.15.16:0/3282532963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 msgr2=0x56355dc85630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 0x56355dc85630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9038009970 tx=0x7f9038009e30 comp rx=0 tx=0).stop 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- 172.21.15.16:0/3282532963 shutdown_connections 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc83240 0x56355dc85630 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc82d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 --2- 172.21.15.16:0/3282532963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78060 0x56355dc78440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- 172.21.15.16:0/3282532963 >> 172.21.15.16:0/3282532963 conn(0x56355dc745f0 msgr2=0x56355dc749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- 172.21.15.16:0/3282532963 shutdown_connections 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- 172.21.15.16:0/3282532963 wait complete. 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 -- start start 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.850+0000 7f904d570740 1 Processor -- start 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 0x56355dc800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 0x56355dc80600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc80b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56355dc85e10 con 0x56355dc78060 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56355db65f10 con 0x56355dc83240 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56355dc85f90 con 0x56355dc78980 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f904d570740 10 client.0 _ll_register_callbacks cb 0x56355db89db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9049734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 0x56355dc800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 0x56355dc80600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9049734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 0x56355dc800c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48712/0 (socket says 172.21.15.16:48712) 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9049734640 1 -- 172.21.15.16:0/1314735537 learned_addr learned my addr 172.21.15.16:0/1314735537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9049734640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc80b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 -- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 msgr2=0x56355dc80b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc80b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 -- 172.21.15.16:0/1314735537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 msgr2=0x56355dc800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 0x56355dc800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.851+0000 7f9043fff640 1 -- 172.21.15.16:0/1314735537 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56355d8ec580 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9043fff640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 0x56355dc80600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x56355dc75970 tx=0x7f903800ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=337aaf5fa9354835 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9023fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038017050 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9038013b30 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9038013470 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f903801c450 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f904d570740 10 client.25441 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56355d9b6960 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.852+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9038013470 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f90380267e0 con 0x56355dc78980 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 10 client.25441 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 10 client.25441 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 20 client.25441 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dc92d40 0x56355dc95130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 4 client.25441 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56355dbeab90 con 0x56355dc92d40 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f9049734640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dc92d40 0x56355dc95130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f9023fff640 10 client.25441 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.853+0000 7f9049734640 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dc92d40 0x56355dc95130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb5c0ae742326acb server_cookie=f96651cb73729948 in_seq=0 out_seq=0 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f9023fff640 1 -- 172.21.15.16:0/1314735537 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56355dbeab90 con 0x56355dc92d40 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f9023fff640 10 client.25441 handle_command_reply: tid=0 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 client.25441 shutdown 2024-09-12T02:49:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 msgr2=0x56355dc80600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 0x56355dc80600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x56355dc75970 tx=0x7f903800ef30 comp rx=0 tx=0).stop 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 shutdown_connections 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dc92d40 0x56355dc95130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc80b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78980 0x56355dc80600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f9023fff640 0 client.25441 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 --2- 172.21.15.16:0/1314735537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78060 0x56355dc800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.854+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 >> 172.21.15.16:0/1314735537 conn(0x56355dc745f0 msgr2=0x56355dc7a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.855+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 shutdown_connections 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.855+0000 7f904d570740 1 -- 172.21.15.16:0/1314735537 wait complete. 2024-09-12T02:49:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.855+0000 7f904d570740 20 client.25441 trim_cache size 0 max 16384 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.871+0000 7f904d570740 1 Processor -- start 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 -- start start 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc9a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc7d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90441af8a0 con 0x56355dc78980 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90441b14a0 con 0x56355dc83240 2024-09-12T02:49:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f904d570740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90441aff70 con 0x56355dc78060 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48720/0 (socket says 172.21.15.16:48720) 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 -- 172.21.15.16:0/2653639387 learned_addr learned my addr 172.21.15.16:0/2653639387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9049734640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc9a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 -- 172.21.15.16:0/2653639387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 msgr2=0x56355dc9a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc9a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9049734640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc7d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 -- 172.21.15.16:0/2653639387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 msgr2=0x56355dc7d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc7d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 -- 172.21.15.16:0/2653639387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56355dc787b0 con 0x56355dc78980 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9049734640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc7d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.872+0000 7f9043fff640 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f903800afd0 tx=0x7f9038006a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b9c877c34cd7d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f902a7fc640 1 -- 172.21.15.16:0/2653639387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038036070 con 0x56355dc78980 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f902a7fc640 1 -- 172.21.15.16:0/2653639387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90380313d0 con 0x56355dc78980 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f902a7fc640 1 -- 172.21.15.16:0/2653639387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038017070 con 0x56355dc78980 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 -- 172.21.15.16:0/2653639387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 msgr2=0x56355dc9ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f903800afd0 tx=0x7f9038006a30 comp rx=0 tx=0).stop 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 -- 172.21.15.16:0/2653639387 shutdown_connections 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc83240 0x56355dc7d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc78980 0x56355dc9ad60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 --2- 172.21.15.16:0/2653639387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc9a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 -- 172.21.15.16:0/2653639387 >> 172.21.15.16:0/2653639387 conn(0x56355dc745f0 msgr2=0x56355dc7e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 -- 172.21.15.16:0/2653639387 shutdown_connections 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.873+0000 7f904d570740 1 -- 172.21.15.16:0/2653639387 wait complete. 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 -- start start 2024-09-12T02:49:51.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 Processor -- start 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc8b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78980 0x56355dc881b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90441b0d10 con 0x56355dc83240 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90441b0640 con 0x56355dc78980 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90441ad880 con 0x56355dc78060 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f904d570740 10 client.0 _ll_register_callbacks cb 0x7f90380097f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f9049734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f9049734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48732/0 (socket says 172.21.15.16:48732) 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f9049734640 1 -- 172.21.15.16:0/3108508879 learned_addr learned my addr 172.21.15.16:0/3108508879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f9043fff640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78980 0x56355dc881b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.874+0000 7f9049734640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc8b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 -- 172.21.15.16:0/3108508879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 msgr2=0x56355dc8b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc8b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 -- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78980 msgr2=0x56355dc881b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78980 0x56355dc881b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 -- 172.21.15.16:0/3108508879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56355dc88cb0 con 0x56355dc83240 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9049734640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f903c02ade0 tx=0x7f903800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346d524098b18138 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9023fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c044040 con 0x56355dc83240 2024-09-12T02:49:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f903c04d460 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90380313d0 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f903c04d660 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f904d570740 10 client.15939 fetch_fsmap learned FSMap version 30 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56355dcfe580 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.875+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90380313d0 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f903c04a040 con 0x56355dc83240 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 10 client.15939 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 10 client.15939 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 20 client.15939 populate_metadata read hostname 'smithi016' 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dcfed70 0x56355dd01190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 4 client.15939 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d5cee633", "release": true, "await": true}] 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d5cee633", "release": true, "await": true}) -- 0x56355dbe50a0 con 0x56355dcfed70 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f9043fff640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dcfed70 0x56355dd01190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:49:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f9043fff640 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dcfed70 0x56355dd01190 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75212959d3c28e0 server_cookie=7f8d6494b9591d27 in_seq=0 out_seq=0 2024-09-12T02:49:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.876+0000 7f9023fff640 10 client.15939 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f9023fff640 1 -- 172.21.15.16:0/3108508879 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56355dbe50a0 con 0x56355dcfed70 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f9023fff640 10 client.15939 handle_command_reply: tid=0 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 client.15939 shutdown 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 msgr2=0x56355dc886f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f903c02ade0 tx=0x7f903800ed90 comp rx=0 tx=0).stop 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 shutdown_connections 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56355dcfed70 0x56355dd01190 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56355dc83240 0x56355dc886f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56355dc78980 0x56355dc881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 --2- 172.21.15.16:0/3108508879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56355dc78060 0x56355dc8b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f9023fff640 0 client.15939 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.888+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 >> 172.21.15.16:0/3108508879 conn(0x56355dc745f0 msgr2=0x56355dc72f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 shutdown_connections 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904d570740 1 -- 172.21.15.16:0/3108508879 wait complete. 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904d570740 20 client.15939 trim_cache size 0 max 16384 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 30, 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 8, 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "d5cee633": { 2024-09-12T02:49:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 8, 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 133.4, 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:49:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:51.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:51.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:49:51.897 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:49:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 msgr2=0x7f902c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 0x7f902c0a5040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9034096e70 tx=0x7f903409ae00 comp rx=0 tx=0).stop 2024-09-12T02:49:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 msgr2=0x7f90440744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:49:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.889+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f903400b4a0 tx=0x7f903400b8a0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 shutdown_connections 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9044106f10 0x7f9044074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9044102790 0x7f9044074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f902c0a2b80 0x7f902c0a5040 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 --2- 172.21.15.16:0/1038310172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9044101de0 0x7f90440744f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 >> 172.21.15.16:0/1038310172 conn(0x7f90440fd8c0 msgr2=0x7f90440fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 shutdown_connections 2024-09-12T02:49:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:49:51.890+0000 7f904b197640 1 -- 172.21.15.16:0/1038310172 wait complete. 2024-09-12T02:49:51.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:51 smithi033 ceph-mon[35092]: pgmap v395: 97 pgs: 97 active+clean; 66 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-09-12T02:49:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:51 smithi184 ceph-mon[35321]: pgmap v395: 97 pgs: 97 active+clean; 66 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-09-12T02:49:52.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 103268 10.66 MB/sec warmup 54 sec latency 12036.006 ms 2024-09-12T02:49:52.798 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd5cee633', total seconds elapsed: 17.0 2024-09-12T02:49:52.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:52 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:49:53.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:52 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:49:53.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:52 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:49:53.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 106525 10.79 MB/sec warmup 55 sec latency 10.132 ms 2024-09-12T02:49:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:53 smithi033 ceph-mon[35092]: pgmap v396: 97 pgs: 97 active+clean; 63 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-12T02:49:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:49:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:53 smithi184 ceph-mon[35321]: pgmap v396: 97 pgs: 97 active+clean; 63 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-12T02:49:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:49:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:53 smithi016 ceph-mon[30925]: pgmap v396: 97 pgs: 97 active+clean; 63 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-12T02:49:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:49:54.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 109381 10.84 MB/sec warmup 56 sec latency 13.446 ms 2024-09-12T02:49:55.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 111530 10.86 MB/sec warmup 57 sec latency 74.261 ms 2024-09-12T02:49:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:55 smithi033 ceph-mon[35092]: pgmap v397: 97 pgs: 97 active+clean; 61 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.4 MiB/s wr, 72 op/s 2024-09-12T02:49:56.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:55 smithi184 ceph-mon[35321]: pgmap v397: 97 pgs: 97 active+clean; 61 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.4 MiB/s wr, 72 op/s 2024-09-12T02:49:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:55 smithi016 ceph-mon[30925]: pgmap v397: 97 pgs: 97 active+clean; 61 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.4 MiB/s wr, 72 op/s 2024-09-12T02:49:56.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 114162 10.91 MB/sec warmup 58 sec latency 16.871 ms 2024-09-12T02:49:57.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:56 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:49:57.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:56 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:49:57.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:56 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:49:57.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 117367 11.05 MB/sec warmup 59 sec latency 15.872 ms 2024-09-12T02:49:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: pgmap v398: 97 pgs: 97 active+clean; 65 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-09-12T02:49:58.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:49:58.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:49:58.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:57 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: pgmap v398: 97 pgs: 97 active+clean; 65 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:49:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:49:58.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:57 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: pgmap v398: 97 pgs: 97 active+clean; 65 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-09-12T02:49:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:49:58.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:49:58.249 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:57 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:49:58.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 120599 11.16 MB/sec warmup 60 sec latency 14.545 ms 2024-09-12T02:49:59.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 123532 11.20 MB/sec warmup 61 sec latency 13.828 ms 2024-09-12T02:50:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:49:59 smithi184 ceph-mon[35321]: pgmap v399: 97 pgs: 97 active+clean; 66 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-09-12T02:50:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:49:59 smithi016 ceph-mon[30925]: pgmap v399: 97 pgs: 97 active+clean; 66 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-09-12T02:50:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:49:59 smithi033 ceph-mon[35092]: pgmap v399: 97 pgs: 97 active+clean; 66 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-09-12T02:50:00.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 125692 11.22 MB/sec warmup 62 sec latency 81.238 ms 2024-09-12T02:50:01.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:00 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T02:50:01.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:00 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T02:50:01.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:00 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T02:50:01.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 128459 11.27 MB/sec warmup 63 sec latency 15.048 ms 2024-09-12T02:50:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:01 smithi184 ceph-mon[35321]: pgmap v400: 97 pgs: 97 active+clean; 70 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-12T02:50:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:01 smithi016 ceph-mon[30925]: pgmap v400: 97 pgs: 97 active+clean; 70 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-12T02:50:02.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:01 smithi033 ceph-mon[35092]: pgmap v400: 97 pgs: 97 active+clean; 70 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 236 op/s 2024-09-12T02:50:02.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 131677 11.38 MB/sec warmup 64 sec latency 19.845 ms 2024-09-12T02:50:03.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134781 11.48 MB/sec warmup 65 sec latency 12.618 ms 2024-09-12T02:50:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:03 smithi184 ceph-mon[35321]: pgmap v401: 97 pgs: 97 active+clean; 72 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 242 op/s 2024-09-12T02:50:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:03 smithi016 ceph-mon[30925]: pgmap v401: 97 pgs: 97 active+clean; 72 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 242 op/s 2024-09-12T02:50:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:03 smithi033 ceph-mon[35092]: pgmap v401: 97 pgs: 97 active+clean; 72 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 242 op/s 2024-09-12T02:50:04.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 137189 11.43 MB/sec warmup 66 sec latency 18.706 ms 2024-09-12T02:50:05.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 139066 11.45 MB/sec warmup 67 sec latency 89.335 ms 2024-09-12T02:50:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:05 smithi184 ceph-mon[35321]: pgmap v402: 97 pgs: 97 active+clean; 76 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-09-12T02:50:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:05 smithi016 ceph-mon[30925]: pgmap v402: 97 pgs: 97 active+clean; 76 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-09-12T02:50:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:05 smithi033 ceph-mon[35092]: pgmap v402: 97 pgs: 97 active+clean; 76 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 291 op/s 2024-09-12T02:50:06.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 142199 11.55 MB/sec warmup 68 sec latency 10.451 ms 2024-09-12T02:50:07.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 145279 11.60 MB/sec warmup 69 sec latency 14.079 ms 2024-09-12T02:50:08.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:07 smithi184 ceph-mon[35321]: pgmap v403: 97 pgs: 97 active+clean; 78 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-12T02:50:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:07 smithi016 ceph-mon[30925]: pgmap v403: 97 pgs: 97 active+clean; 78 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-12T02:50:08.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:07 smithi033 ceph-mon[35092]: pgmap v403: 97 pgs: 97 active+clean; 78 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-09-12T02:50:08.708 INFO:tasks.workunit.client.0.smithi016.stdout: 1 147878 11.60 MB/sec warmup 70 sec latency 20.283 ms 2024-09-12T02:50:09.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 150179 11.64 MB/sec warmup 71 sec latency 19.055 ms 2024-09-12T02:50:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:09 smithi184 ceph-mon[35321]: pgmap v404: 97 pgs: 97 active+clean; 76 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 280 op/s 2024-09-12T02:50:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:09 smithi016 ceph-mon[30925]: pgmap v404: 97 pgs: 97 active+clean; 76 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 280 op/s 2024-09-12T02:50:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:09 smithi033 ceph-mon[35092]: pgmap v404: 97 pgs: 97 active+clean; 76 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 280 op/s 2024-09-12T02:50:10.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 153111 11.71 MB/sec warmup 72 sec latency 11.311 ms 2024-09-12T02:50:11.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156163 11.79 MB/sec warmup 73 sec latency 50.428 ms 2024-09-12T02:50:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:11 smithi184 ceph-mon[35321]: pgmap v405: 97 pgs: 97 active+clean; 81 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 374 op/s 2024-09-12T02:50:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:11 smithi016 ceph-mon[30925]: pgmap v405: 97 pgs: 97 active+clean; 81 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 374 op/s 2024-09-12T02:50:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:11 smithi033 ceph-mon[35092]: pgmap v405: 97 pgs: 97 active+clean; 81 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 374 op/s 2024-09-12T02:50:12.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 158655 11.76 MB/sec warmup 74 sec latency 51.128 ms 2024-09-12T02:50:13.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 161170 11.82 MB/sec warmup 75 sec latency 12.844 ms 2024-09-12T02:50:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:13 smithi033 ceph-mon[35092]: pgmap v406: 97 pgs: 97 active+clean; 80 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 300 op/s 2024-09-12T02:50:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:13 smithi184 ceph-mon[35321]: pgmap v406: 97 pgs: 97 active+clean; 80 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 300 op/s 2024-09-12T02:50:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:13 smithi016 ceph-mon[30925]: pgmap v406: 97 pgs: 97 active+clean; 80 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 300 op/s 2024-09-12T02:50:14.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 164097 11.86 MB/sec warmup 76 sec latency 13.109 ms 2024-09-12T02:50:15.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:50:15.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167194 11.94 MB/sec warmup 77 sec latency 10.629 ms 2024-09-12T02:50:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:16.226 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:15 smithi016 ceph-mon[30925]: pgmap v407: 97 pgs: 97 active+clean; 81 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 325 op/s 2024-09-12T02:50:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:15 smithi033 ceph-mon[35092]: pgmap v407: 97 pgs: 97 active+clean; 81 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 325 op/s 2024-09-12T02:50:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:15 smithi184 ceph-mon[35321]: pgmap v407: 97 pgs: 97 active+clean; 81 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 325 op/s 2024-09-12T02:50:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.408+0000 7ff8fac64640 1 -- 172.21.15.16:0/3905646644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41036e0 msgr2=0x7ff8f4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.408+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3905646644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41036e0 0x7ff8f4107b90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009cd0 tx=0x7ff8e402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.408+0000 7ff8fac64640 1 -- 172.21.15.16:0/3905646644 shutdown_connections 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3905646644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f41082a0 0x7ff8f410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3905646644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41036e0 0x7ff8f4107b90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3905646644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f4102d30 0x7ff8f4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 -- 172.21.15.16:0/3905646644 >> 172.21.15.16:0/3905646644 conn(0x7ff8f40fc7d0 msgr2=0x7ff8f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 -- 172.21.15.16:0/3905646644 shutdown_connections 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 -- 172.21.15.16:0/3905646644 wait complete. 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 Processor -- start 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.409+0000 7ff8fac64640 1 -- start start 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 0x7ff8f4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41082a0 0x7ff8f4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41120a0 con 0x7ff8f41082a0 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f4111f20 con 0x7ff8f4102d30 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8fac64640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f4112220 con 0x7ff8f41036e0 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f9201640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41082a0 0x7ff8f4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 0x7ff8f4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37888/0 (socket says 172.21.15.16:37888) 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 -- 172.21.15.16:0/3763049279 learned_addr learned my addr 172.21.15.16:0/3763049279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 0x7ff8f4100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41418/0 (socket says 172.21.15.16:41418) 2024-09-12T02:50:16.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 -- 172.21.15.16:0/3763049279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 msgr2=0x7ff8f4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 0x7ff8f4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 -- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41082a0 msgr2=0x7ff8f4101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41082a0 0x7ff8f4101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 -- 172.21.15.16:0/3763049279 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f41a6a60 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.410+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4000f80 tx=0x7ff8e4031e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.411+0000 7ff8f1ffb640 1 -- 172.21.15.16:0/3763049279 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4004040 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.411+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f41a6cf0 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.411+0000 7ff8f1ffb640 1 -- 172.21.15.16:0/3763049279 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e40046c0 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.411+0000 7ff8f1ffb640 1 -- 172.21.15.16:0/3763049279 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4030e40 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.411+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f41a7220 con 0x7ff8f41036e0 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.413+0000 7ff8fd03d740 1 Processor -- start 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.413+0000 7ff8fd03d740 1 -- start start 2024-09-12T02:50:16.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.413+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd09880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.413+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd16a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55916dc6f2e0 con 0x55916dd14680 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55916dc16ce0 con 0x55916dd094a0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55916dc6b2a0 con 0x55916dd09dc0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f1ffb640 1 -- 172.21.15.16:0/3763049279 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff8e40305a0 con 0x7ff8f41036e0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f9201640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd09880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37902/0 (socket says 172.21.15.16:37902) 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f9201640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd09880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41426/0 (socket says 172.21.15.16:41426) 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2982704447 learned_addr learned my addr 172.21.15.16:0/2982704447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd16a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2982704447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 msgr2=0x55916dd09880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd09880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2982704447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 msgr2=0x55916dd16a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd16a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2982704447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55916dbfb440 con 0x55916dd09dc0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009d20 tx=0x7ff8dc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f949a854c2fb06 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8f1ffb640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 0x7ff8e00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.414+0000 7ff8eaffd640 1 -- 172.21.15.16:0/2982704447 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc01e070 con 0x55916dd09dc0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8eaffd640 1 -- 172.21.15.16:0/2982704447 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8dc019b10 con 0x55916dd09dc0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 0x7ff8e00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8eaffd640 1 -- 172.21.15.16:0/2982704447 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00ebf0 con 0x55916dd09dc0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8f1ffb640 1 -- 172.21.15.16:0/3763049279 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8e402e030 con 0x7ff8f41036e0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2982704447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 msgr2=0x55916dd14140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 0x7ff8e00a5040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009d20 tx=0x7ff8dc011810 comp rx=0 tx=0).stop 2024-09-12T02:50:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2982704447 shutdown_connections 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd16a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd14140 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2982704447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd09880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2982704447 >> 172.21.15.16:0/2982704447 conn(0x55916dd05a30 msgr2=0x55916dd05e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2982704447 shutdown_connections 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.415+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2982704447 wait complete. 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.416+0000 7ff8fd03d740 1 -- start start 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.416+0000 7ff8fd03d740 1 Processor -- start 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.416+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.416+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd11840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55916dd17250 con 0x55916dd094a0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55916dc2a050 con 0x55916dd09dc0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55916dd173d0 con 0x55916dd14680 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8fd03d740 10 client.0 _ll_register_callbacks cb 0x55916dc6a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48332/0 (socket says 172.21.15.16:48332) 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2430823049 learned_addr learned my addr 172.21.15.16:0/2430823049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f9201640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd11840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2430823049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 msgr2=0x55916dd11840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd11840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 msgr2=0x55916dd112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f8a00640 1 -- 172.21.15.16:0/2430823049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55916d9e03c0 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f9201640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd11840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.417+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00e790 tx=0x7ff8dc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa157f8daa7f030e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc01e070 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8dc00eaf0 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8dc019b10 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc019960 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 10 client.15951 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55916dd22c70 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8dc019b10 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff8dc0046c0 con 0x55916dd094a0 2024-09-12T02:50:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 10 client.15951 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 20 client.15951 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd240d0 0x55916dd264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 4 client.15951 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.418+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55916dc7c850 con 0x55916dd240d0 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.419+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd240d0 0x55916dd264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.419+0000 7ff8d3fff640 10 client.15951 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.419+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd240d0 0x55916dd264c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed968d22b7e72d5 server_cookie=570e93669fa56f06 in_seq=0 out_seq=0 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8d3fff640 1 -- 172.21.15.16:0/2430823049 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55916dc7c850 con 0x55916dd240d0 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8d3fff640 10 client.15951 handle_command_reply: tid=0 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 client.15951 shutdown 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 msgr2=0x55916dd14010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00e790 tx=0x7ff8dc019660 comp rx=0 tx=0).stop 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 shutdown_connections 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd240d0 0x55916dd264c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd11840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/2430823049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd14010 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:50:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:50:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:50:16.469 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:50:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 >> 172.21.15.16:0/2430823049 conn(0x55916dd05a30 msgr2=0x55916dd0b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8d3fff640 0 client.15951 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 shutdown_connections 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 1 -- 172.21.15.16:0/2430823049 wait complete. 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.420+0000 7ff8fd03d740 20 client.15951 trim_cache size 0 max 16384 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 Processor -- start 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 -- start start 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd204e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd04a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41af680 con 0x55916dd14680 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41b1280 con 0x55916dd094a0 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41afd50 con 0x55916dd09dc0 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd204e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48342/0 (socket says 172.21.15.16:48342) 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 -- 172.21.15.16:0/3398832726 learned_addr learned my addr 172.21.15.16:0/3398832726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd04a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 -- 172.21.15.16:0/3398832726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 msgr2=0x55916dd04a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd04a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 -- 172.21.15.16:0/3398832726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 msgr2=0x55916dd204e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd204e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.441+0000 7ff8f9201640 1 -- 172.21.15.16:0/3398832726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55916dd09bf0 con 0x55916dd14680 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.442+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec003fa0 tx=0x7ff8ec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa0dbe25506d5d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.442+0000 7ff8ea7fc640 1 -- 172.21.15.16:0/3398832726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec003fd0 con 0x55916dd14680 2024-09-12T02:50:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.442+0000 7ff8ea7fc640 1 -- 172.21.15.16:0/3398832726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec005500 con 0x55916dd14680 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.442+0000 7ff8ea7fc640 1 -- 172.21.15.16:0/3398832726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0116a0 con 0x55916dd14680 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3398832726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 msgr2=0x55916dd2bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec003fa0 tx=0x7ff8ec019000 comp rx=0 tx=0).stop 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3398832726 shutdown_connections 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd14680 0x55916dd2bcc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd09dc0 0x55916dd04a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3398832726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd094a0 0x55916dd204e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3398832726 >> 172.21.15.16:0/3398832726 conn(0x55916dd05a30 msgr2=0x55916dd04350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3398832726 shutdown_connections 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3398832726 wait complete. 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 -- start start 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.443+0000 7ff8fd03d740 1 Processor -- start 2024-09-12T02:50:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd8cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41b0af0 con 0x55916dd094a0 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41b0420 con 0x55916dd09dc0 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f41ad660 con 0x55916dd14680 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8fd03d740 10 client.0 _ll_register_callbacks cb 0x7ff8dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48344/0 (socket says 172.21.15.16:48344) 2024-09-12T02:50:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 -- 172.21.15.16:0/3708738260 learned_addr learned my addr 172.21.15.16:0/3708738260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f9201640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd8cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 -- 172.21.15.16:0/3708738260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 msgr2=0x55916dd8cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd8cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 -- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 msgr2=0x55916dd180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd180f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 -- 172.21.15.16:0/3708738260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55916dd8d1c0 con 0x55916dd094a0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8f8a00640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009800 tx=0x7ff8dc019ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaeaf8742512f0a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.444+0000 7ff8d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00ec60 con 0x55916dd094a0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8dc00ee00 con 0x55916dd094a0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ec005500 con 0x55916dd094a0 2024-09-12T02:50:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc0098f0 con 0x55916dd094a0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 10 client.15957 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55916dd8e400 con 0x55916dd094a0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec005500 con 0x55916dd094a0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff8dc0035f0 con 0x55916dd094a0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 10 client.15957 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 20 client.15957 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd8ebd0 0x55916dd90ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 4 client.15957 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.445+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55916dc76d60 con 0x55916dd8ebd0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd8ebd0 0x55916dd90ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8d3fff640 10 client.15957 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:16.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8f3fff640 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd8ebd0 0x55916dd90ff0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502f97d40874b821 server_cookie=67e08485eab89388 in_seq=0 out_seq=0 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8d3fff640 1 -- 172.21.15.16:0/3708738260 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55916dc76d60 con 0x55916dd8ebd0 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8d3fff640 10 client.15957 handle_command_reply: tid=0 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 client.15957 shutdown 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 msgr2=0x55916dd17bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009800 tx=0x7ff8dc019ea0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 shutdown_connections 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55916dd8ebd0 0x55916dd90ff0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55916dd14680 0x55916dd8cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55916dd09dc0 0x55916dd180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 --2- 172.21.15.16:0/3708738260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55916dd094a0 0x55916dd17bb0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8d3fff640 0 client.15957 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 >> 172.21.15.16:0/3708738260 conn(0x55916dd05a30 msgr2=0x55916dd04350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 shutdown_connections 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.446+0000 7ff8fd03d740 1 -- 172.21.15.16:0/3708738260 wait complete. 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fd03d740 20 client.15957 trim_cache size 0 max 16384 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 msgr2=0x7ff8e00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 0x7ff8e00a5040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec005cc0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 msgr2=0x7ff8f4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4000f80 tx=0x7ff8e4031e60 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 shutdown_connections 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8e00a2b80 0x7ff8e00a5040 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f41082a0 0x7ff8f4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8f41036e0 0x7ff8f4100ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 --2- 172.21.15.16:0/3763049279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102d30 0x7ff8f4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 >> 172.21.15.16:0/3763049279 conn(0x7ff8f40fc7d0 msgr2=0x7ff8f4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:16.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 shutdown_connections 2024-09-12T02:50:16.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:16.447+0000 7ff8fac64640 1 -- 172.21.15.16:0/3763049279 wait complete. 2024-09-12T02:50:16.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 170282 12.02 MB/sec warmup 78 sec latency 34.948 ms 2024-09-12T02:50:17.386 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:50:17.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:50:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:17.690 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.7 2024-09-12T02:50:17.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:50:17.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172991 12.02 MB/sec warmup 79 sec latency 14.486 ms 2024-09-12T02:50:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 -- 172.21.15.16:0/543390597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101027c0 msgr2=0x7ff110106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 --2- 172.21.15.16:0/543390597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101027c0 0x7ff110106810 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009900 tx=0x7ff0f802a410 comp rx=0 tx=0).stop 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 -- 172.21.15.16:0/543390597 shutdown_connections 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 --2- 172.21.15.16:0/543390597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110106f40 0x7ff11010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 --2- 172.21.15.16:0/543390597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101027c0 0x7ff110106810 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 --2- 172.21.15.16:0/543390597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff110101e10 0x7ff1101021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.183+0000 7ff1156fe640 1 -- 172.21.15.16:0/543390597 >> 172.21.15.16:0/543390597 conn(0x7ff1100fd8d0 msgr2=0x7ff1100ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.184+0000 7ff1156fe640 1 -- 172.21.15.16:0/543390597 shutdown_connections 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.184+0000 7ff1156fe640 1 -- 172.21.15.16:0/543390597 wait complete. 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.184+0000 7ff1156fe640 1 Processor -- start 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 -- start start 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 0x7ff1101989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 0x7ff11019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff110111fa0 con 0x7ff110106f40 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff110111e20 con 0x7ff110101e10 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff1156fe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff110112120 con 0x7ff1101027c0 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 0x7ff1101989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 0x7ff11019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41480/0 (socket says 172.21.15.16:41480) 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 0x7ff1101989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37952/0 (socket says 172.21.15.16:37952) 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 -- 172.21.15.16:0/3415152972 learned_addr learned my addr 172.21.15.16:0/3415152972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 0x7ff11019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48358/0 (socket says 172.21.15.16:48358) 2024-09-12T02:50:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 -- 172.21.15.16:0/3415152972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 msgr2=0x7ff1101989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 0x7ff1101989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 -- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 msgr2=0x7ff11019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 0x7ff11019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.185+0000 7ff10effd640 1 -- 172.21.15.16:0/3415152972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11019d410 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff10effd640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff10800eb90 tx=0x7ff10800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108004d20 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11019d700 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1101a4fe0 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff10800ee90 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.186+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108010830 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.187+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1101032f0 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.188+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff108005000 con 0x7ff110101e10 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.189+0000 7ff107fff640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 0x7ff0fc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.189+0000 7ff10e7fc640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 0x7ff0fc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.189+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1080cf8e0 con 0x7ff110101e10 2024-09-12T02:50:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.190+0000 7ff10e7fc640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 0x7ff0fc0a5040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009230 tx=0x7ff0f80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.192+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff1080541b0 con 0x7ff110101e10 2024-09-12T02:50:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:17 smithi033 ceph-mon[35092]: pgmap v408: 97 pgs: 97 active+clean; 81 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-09-12T02:50:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:17 smithi184 ceph-mon[35321]: pgmap v408: 97 pgs: 97 active+clean; 81 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-09-12T02:50:18.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:17 smithi016 ceph-mon[30925]: pgmap v408: 97 pgs: 97 active+clean; 81 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-09-12T02:50:18.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.427+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1101021f0 con 0x7ff110101e10 2024-09-12T02:50:18.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.427+0000 7ff107fff640 1 -- 172.21.15.16:0/3415152972 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7ff10809c7f0 con 0x7ff110101e10 2024-09-12T02:50:18.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:50:18.428 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:50:18.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 msgr2=0x7ff0fc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 0x7ff0fc0a5040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009230 tx=0x7ff0f80058d0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 msgr2=0x7ff110198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff10800eb90 tx=0x7ff10800c6a0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 shutdown_connections 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0fc0a2b80 0x7ff0fc0a5040 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff110106f40 0x7ff11019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1101027c0 0x7ff1101989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 --2- 172.21.15.16:0/3415152972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff110101e10 0x7ff110198460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 >> 172.21.15.16:0/3415152972 conn(0x7ff1100fd8d0 msgr2=0x7ff1100fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 shutdown_connections 2024-09-12T02:50:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:18.430+0000 7ff1156fe640 1 -- 172.21.15.16:0/3415152972 wait complete. 2024-09-12T02:50:18.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174964 12.00 MB/sec warmup 80 sec latency 26.765 ms 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 -- 172.21.15.16:0/2876723239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec108f60 msgr2=0x7f75ec110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 --2- 172.21.15.16:0/2876723239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec108f60 0x7f75ec110ae0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009ca0 tx=0x7f75e402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 -- 172.21.15.16:0/2876723239 shutdown_connections 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 --2- 172.21.15.16:0/2876723239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec108f60 0x7f75ec110ae0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 --2- 172.21.15.16:0/2876723239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec1047e0 0x7f75ec108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 --2- 172.21.15.16:0/2876723239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 -- 172.21.15.16:0/2876723239 >> 172.21.15.16:0/2876723239 conn(0x7f75ec0ff910 msgr2=0x7f75ec101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 -- 172.21.15.16:0/2876723239 shutdown_connections 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.066+0000 7f75f179e640 1 -- 172.21.15.16:0/2876723239 wait complete. 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 Processor -- start 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 -- start start 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec108f60 0x7f75ec19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec113fc0 con 0x7f75ec1047e0 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec113e40 con 0x7f75ec103e30 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f179e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec114140 con 0x7f75ec108f60 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41490/0 (socket says 172.21.15.16:41490) 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48376/0 (socket says 172.21.15.16:48376) 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75ebfff640 1 -- 172.21.15.16:0/3347660798 learned_addr learned my addr 172.21.15.16:0/3347660798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.067+0000 7f75f0f9d640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec108f60 0x7f75ec19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 -- 172.21.15.16:0/3347660798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec108f60 msgr2=0x7f75ec19ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec108f60 0x7f75ec19ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 -- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 msgr2=0x7f75ec19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 -- 172.21.15.16:0/3347660798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec19f3d0 con 0x7f75ec103e30 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75eb7fe640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75ebfff640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00e9f0 tx=0x7f75dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00cd80 con 0x7f75ec103e30 2024-09-12T02:50:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75dc010910 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec19f6c0 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc018990 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.068+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec1a6fa0 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.069+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec105310 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.070+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f75dc010ac0 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.071+0000 7f75e97fa640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 0x7f75d40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.071+0000 7f75eb7fe640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 0x7f75d40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.071+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f75dc0cf8b0 con 0x7f75ec103e30 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.072+0000 7f75eb7fe640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 0x7f75d40a50c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0ffd30 tx=0x7f75e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.074+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75dc09c170 con 0x7f75ec103e30 2024-09-12T02:50:19.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3415152972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3415152972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:19.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.315+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75ec104210 con 0x7f75ec103e30 2024-09-12T02:50:19.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3415152972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:19.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.316+0000 7f75e97fa640 1 -- 172.21.15.16:0/3347660798 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7f75dc0a0470 con 0x7f75ec103e30 2024-09-12T02:50:19.317 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:50:19.317 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:50:19.317 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:50:19.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.318+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 msgr2=0x7f75d40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:19.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.318+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 0x7f75d40a50c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0ffd30 tx=0x7f75e0005f90 comp rx=0 tx=0).stop 2024-09-12T02:50:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 msgr2=0x7f75ec19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00e9f0 tx=0x7f75dc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 shutdown_connections 2024-09-12T02:50:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75ec108f60 0x7f75ec19ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75d40a2c00 0x7f75d40a50c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75ec1047e0 0x7f75ec19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 --2- 172.21.15.16:0/3347660798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75ec103e30 0x7f75ec19a420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:19.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 >> 172.21.15.16:0/3347660798 conn(0x7f75ec0ff910 msgr2=0x7f75ec100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:19.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 shutdown_connections 2024-09-12T02:50:19.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:19.319+0000 7f75f179e640 1 -- 172.21.15.16:0/3347660798 wait complete. 2024-09-12T02:50:19.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:50:19.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 177902 12.06 MB/sec warmup 81 sec latency 9.525 ms 2024-09-12T02:50:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:19 smithi016 ceph-mon[30925]: pgmap v409: 97 pgs: 97 active+clean; 81 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-09-12T02:50:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:19 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3347660798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:20.221 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 144.7 --await-for 45.0' 2024-09-12T02:50:20.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 144.7 --await-for 45.0 2024-09-12T02:50:20.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:19 smithi033 ceph-mon[35092]: pgmap v409: 97 pgs: 97 active+clean; 81 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-09-12T02:50:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:19 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3347660798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:19 smithi184 ceph-mon[35321]: pgmap v409: 97 pgs: 97 active+clean; 81 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-09-12T02:50:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:19 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3347660798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:20.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 180894 12.13 MB/sec warmup 82 sec latency 23.608 ms 2024-09-12T02:50:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 -- 172.21.15.16:0/1107545662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 msgr2=0x7fcbb410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 --2- 172.21.15.16:0/1107545662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb410df40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009900 tx=0x7fcba402a410 comp rx=0 tx=0).stop 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 -- 172.21.15.16:0/1107545662 shutdown_connections 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 --2- 172.21.15.16:0/1107545662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb410e480 0x7fcbb4110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 --2- 172.21.15.16:0/1107545662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb410df40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 --2- 172.21.15.16:0/1107545662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb41008d0 0x7fcbb4100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.759+0000 7fcbbc467640 1 -- 172.21.15.16:0/1107545662 >> 172.21.15.16:0/1107545662 conn(0x7fcbb40fc7a0 msgr2=0x7fcbb40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.760+0000 7fcbbc467640 1 -- 172.21.15.16:0/1107545662 shutdown_connections 2024-09-12T02:50:20.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.760+0000 7fcbbc467640 1 -- 172.21.15.16:0/1107545662 wait complete. 2024-09-12T02:50:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 Processor -- start 2024-09-12T02:50:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 -- start start 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb410e480 0x7fcbb419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4114200 con 0x7fcbb4101280 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4114080 con 0x7fcbb41008d0 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.761+0000 7fcbbc467640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4114380 con 0x7fcbb410e480 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbb9a02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbb9a02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb419ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48392/0 (socket says 172.21.15.16:48392) 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41502/0 (socket says 172.21.15.16:41502) 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbb9a02640 1 -- 172.21.15.16:0/2727327270 learned_addr learned my addr 172.21.15.16:0/2727327270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb410e480 0x7fcbb419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 -- 172.21.15.16:0/2727327270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb410e480 msgr2=0x7fcbb419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb410e480 0x7fcbb419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 -- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 msgr2=0x7fcbb419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 -- 172.21.15.16:0/2727327270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb419f690 con 0x7fcbb41008d0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbba203640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000e9f0 tx=0x7fcbb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbab7fe640 1 -- 172.21.15.16:0/2727327270 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000ccc0 con 0x7fcbb41008d0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbab7fe640 1 -- 172.21.15.16:0/2727327270 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0005020 con 0x7fcbb41008d0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.762+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb419f980 con 0x7fcbb41008d0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.763+0000 7fcbab7fe640 1 -- 172.21.15.16:0/2727327270 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00049e0 con 0x7fcbb41008d0 2024-09-12T02:50:20.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.763+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41a7260 con 0x7fcbb41008d0 2024-09-12T02:50:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.765+0000 7fcbab7fe640 1 -- 172.21.15.16:0/2727327270 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcbb0004cc0 con 0x7fcbb41008d0 2024-09-12T02:50:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.765+0000 7fcbbe840740 1 Processor -- start 2024-09-12T02:50:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 -- start start 2024-09-12T02:50:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbab7fe640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 0x7fcba00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b25090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b30370 0x55f9a0b32760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0a8b2f0 con 0x55f9a0b30370 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0a32cf0 con 0x55f9a0b26b40 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbbe840740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0a872b0 con 0x55f9a0b255d0 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbab7fe640 1 -- 172.21.15.16:0/2727327270 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbb00cefc0 con 0x7fcbb41008d0 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbb9a02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbb9a02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37984/0 (socket says 172.21.15.16:37984) 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbb9a02640 1 -- 172.21.15.16:0/45353820 learned_addr learned my addr 172.21.15.16:0/45353820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.766+0000 7fcbba203640 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b25090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 -- 172.21.15.16:0/45353820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 msgr2=0x55f9a0b25090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b25090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 -- 172.21.15.16:0/45353820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b30370 msgr2=0x55f9a0b32760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b30370 0x55f9a0b32760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 -- 172.21.15.16:0/45353820 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a0a173a0 con 0x55f9a0b255d0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 0x7fcba00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbb9a02640 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006010 tx=0x7fcba4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f4b09da51bd0098 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcba8dea640 1 -- 172.21.15.16:0/45353820 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4040070 con 0x55f9a0b255d0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 0x7fcba00a5040 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55f9a0b26070 tx=0x7fcbac009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcba8dea640 1 -- 172.21.15.16:0/45353820 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4004400 con 0x55f9a0b255d0 2024-09-12T02:50:20.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.767+0000 7fcba8dea640 1 -- 172.21.15.16:0/45353820 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4035040 con 0x55f9a0b255d0 2024-09-12T02:50:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 -- 172.21.15.16:0/45353820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 msgr2=0x55f9a0b25a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006010 tx=0x7fcba4005c00 comp rx=0 tx=0).stop 2024-09-12T02:50:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 -- 172.21.15.16:0/45353820 shutdown_connections 2024-09-12T02:50:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b30370 0x55f9a0b32760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b25a10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 --2- 172.21.15.16:0/45353820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b25090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.768+0000 7fcbbe840740 1 -- 172.21.15.16:0/45353820 >> 172.21.15.16:0/45353820 conn(0x55f9a0aee310 msgr2=0x55f9a0b21c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.769+0000 7fcbbe840740 1 -- 172.21.15.16:0/45353820 shutdown_connections 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.769+0000 7fcbbe840740 1 -- 172.21.15.16:0/45353820 wait complete. 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.769+0000 7fcbbe840740 1 -- start start 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.769+0000 7fcbbe840740 1 Processor -- start 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b2fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b2d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 0x55f9a0b2d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0b33010 con 0x55f9a0b26b40 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0a46060 con 0x55f9a0b30370 2024-09-12T02:50:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f9a0b33190 con 0x55f9a0b255d0 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbe840740 10 client.0 _ll_register_callbacks cb 0x55f9a07998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbba203640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b2fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 0x55f9a0b2d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbba203640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b2fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37988/0 (socket says 172.21.15.16:37988) 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbba203640 1 -- 172.21.15.16:0/515919230 learned_addr learned my addr 172.21.15.16:0/515919230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbb9a02640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b2d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 -- 172.21.15.16:0/515919230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 msgr2=0x55f9a0b2fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b2fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 -- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 msgr2=0x55f9a0b2d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b2d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.770+0000 7fcbbaa04640 1 -- 172.21.15.16:0/515919230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a07fc3c0 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 0x55f9a0b2d650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4102ab0 tx=0x7fcbac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf78f4de3e60d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcb9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac004d20 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbac01c950 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba4004400 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcbac004820 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcbbe840740 10 client.25586 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.771+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f9a0b3eb70 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4004400 con 0x55f9a0b30370 2024-09-12T02:50:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fcbac023070 con 0x55f9a0b30370 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 10 client.25586 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 10 client.25586 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 20 client.25586 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0b3ff80 0x55f9a0b42400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 4 client.25586 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9a0a98860 con 0x55f9a0b3ff80 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbba203640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0b3ff80 0x55f9a0b42400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcb9dffb640 10 client.25586 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.772+0000 7fcbba203640 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0b3ff80 0x55f9a0b42400 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=daf1a44c70f014af server_cookie=461dd9208abdb36a in_seq=0 out_seq=0 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcb9dffb640 1 -- 172.21.15.16:0/515919230 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f9a0a98860 con 0x55f9a0b3ff80 2024-09-12T02:50:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcb9dffb640 10 client.25586 handle_command_reply: tid=0 2024-09-12T02:50:20.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 client.25586 shutdown 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 msgr2=0x55f9a0b2d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 0x55f9a0b2d650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4102ab0 tx=0x7fcbac004ab0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 shutdown_connections 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0b3ff80 0x55f9a0b42400 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b30370 0x55f9a0b2d650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b2d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.773+0000 7fcbbe840740 1 --2- 172.21.15.16:0/515919230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b255d0 0x55f9a0b2fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.774+0000 7fcb9dffb640 0 client.25586 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.774+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 >> 172.21.15.16:0/515919230 conn(0x55f9a0aee310 msgr2=0x55f9a0b27960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.774+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 shutdown_connections 2024-09-12T02:50:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.774+0000 7fcbbe840740 1 -- 172.21.15.16:0/515919230 wait complete. 2024-09-12T02:50:20.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.774+0000 7fcbbe840740 20 client.25586 trim_cache size 0 max 16384 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 Processor -- start 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 -- start start 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b31ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41af7f0 con 0x55f9a0b26b40 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbba203640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbba203640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41528/0 (socket says 172.21.15.16:41528) 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbba203640 1 -- 172.21.15.16:0/2599132865 learned_addr learned my addr 172.21.15.16:0/2599132865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41b13f0 con 0x55f9a0b255d0 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41afec0 con 0x55f9a0b30370 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b31ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.801+0000 7fcbb9a02640 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 -- 172.21.15.16:0/2599132865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 msgr2=0x55f9a0b31ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b31ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 -- 172.21.15.16:0/2599132865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 msgr2=0x55f9a0b31760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b31760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 -- 172.21.15.16:0/2599132865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a0b2b590 con 0x55f9a0b255d0 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcbba203640 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c6a0 tx=0x7fcbb00cc9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed118624feb97b01 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcb9ffff640 1 -- 172.21.15.16:0/2599132865 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb001c040 con 0x55f9a0b255d0 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcb9ffff640 1 -- 172.21.15.16:0/2599132865 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb009fbe0 con 0x55f9a0b255d0 2024-09-12T02:50:20.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.802+0000 7fcb9ffff640 1 -- 172.21.15.16:0/2599132865 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0097ea0 con 0x55f9a0b255d0 2024-09-12T02:50:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 msgr2=0x55f9a0b3c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c6a0 tx=0x7fcbb00cc9a0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 shutdown_connections 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b31ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b26b40 0x55f9a0b31760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 --2- 172.21.15.16:0/2599132865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b255d0 0x55f9a0b3c2e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 >> 172.21.15.16:0/2599132865 conn(0x55f9a0aee310 msgr2=0x55f9a0b28310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.803+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 shutdown_connections 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.804+0000 7fcbbe840740 1 -- 172.21.15.16:0/2599132865 wait complete. 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 -- start start 2024-09-12T02:50:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 Processor -- start 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 0x55f9a0b38b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b47580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41b0c60 con 0x55f9a0b255d0 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41b0590 con 0x55f9a0b26b40 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41ad7d0 con 0x55f9a0b30370 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbe840740 10 client.0 _ll_register_callbacks cb 0x7fcbac003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbaa04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbaa04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38010/0 (socket says 172.21.15.16:38010) 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbbaa04640 1 -- 172.21.15.16:0/1806425083 learned_addr learned my addr 172.21.15.16:0/1806425083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.805+0000 7fcbba203640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 0x55f9a0b38b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 -- 172.21.15.16:0/1806425083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 msgr2=0x55f9a0b47580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b47580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 -- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 msgr2=0x55f9a0b38b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 0x55f9a0b38b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 -- 172.21.15.16:0/1806425083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a0b482c0 con 0x55f9a0b30370 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbba203640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 0x55f9a0b38b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbaa04640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005990 tx=0x7fcbac01cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92db3daf23f2c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcb9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac01a8c0 con 0x55f9a0b30370 2024-09-12T02:50:20.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbac01aa60 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.806+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb009fbe0 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcbac01ac60 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 10 client.25495 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f9a0baacf0 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb009fbe0 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fcbac023070 con 0x55f9a0b30370 2024-09-12T02:50:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 10 client.25495 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 10 client.25495 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 20 client.25495 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0bab460 0x55f9a0bad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 4 client.25495 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9a0a92d70 con 0x55f9a0bab460 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.807+0000 7fcbba203640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0bab460 0x55f9a0bad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbba203640 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0bab460 0x55f9a0bad880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=45821bd4abdb16d2 server_cookie=9055844dfb6e8824 in_seq=0 out_seq=0 2024-09-12T02:50:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcb9dffb640 10 client.25495 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcb9dffb640 1 -- 172.21.15.16:0/1806425083 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9a0a92d70 con 0x55f9a0bab460 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcb9dffb640 10 client.25495 handle_command_reply: tid=0 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 client.25495 shutdown 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 msgr2=0x55f9a0b47ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005990 tx=0x7fcbac01cd80 comp rx=0 tx=0).stop 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 shutdown_connections 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9a0b30370 0x55f9a0b47ac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f9a0bab460 0x55f9a0bad880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f9a0b26b40 0x55f9a0b47580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 --2- 172.21.15.16:0/1806425083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9a0b255d0 0x55f9a0b38b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 >> 172.21.15.16:0/1806425083 conn(0x55f9a0aee310 msgr2=0x55f9a0b20d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.808+0000 7fcb9dffb640 0 client.25495 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 shutdown_connections 2024-09-12T02:50:20.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbe840740 1 -- 172.21.15.16:0/1806425083 wait complete. 2024-09-12T02:50:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbe840740 20 client.25495 trim_cache size 0 max 16384 2024-09-12T02:50:20.815 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 msgr2=0x7fcba00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 0x7fcba00a5040 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55f9a0b26070 tx=0x7fcbac009500 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 msgr2=0x7fcbb419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000e9f0 tx=0x7fcbb000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 shutdown_connections 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbb410e480 0x7fcbb419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcba00a2b80 0x7fcba00a5040 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbb4101280 0x7fcbb419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 --2- 172.21.15.16:0/2727327270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbb41008d0 0x7fcbb419a6e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.809+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 >> 172.21.15.16:0/2727327270 conn(0x7fcbb40fc7a0 msgr2=0x7fcbb40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.810+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 shutdown_connections 2024-09-12T02:50:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:20.810+0000 7fcbbc467640 1 -- 172.21.15.16:0/2727327270 wait complete. 2024-09-12T02:50:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:21.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 183314 12.09 MB/sec warmup 83 sec latency 37.728 ms 2024-09-12T02:50:21.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:50:22.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:21 smithi016 ceph-mon[30925]: pgmap v410: 97 pgs: 97 active+clean; 87 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 -- 172.21.15.16:0/3681832895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 msgr2=0x7facf010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 --2- 172.21.15.16:0/3681832895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 0x7facf010da40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7face8009ca0 tx=0x7face802a950 comp rx=0 tx=0).stop 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 -- 172.21.15.16:0/3681832895 shutdown_connections 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 --2- 172.21.15.16:0/3681832895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0102ef0 0x7facf010da40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 --2- 172.21.15.16:0/3681832895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102570 0x7facf01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 --2- 172.21.15.16:0/3681832895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0100e20 0x7facf0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 -- 172.21.15.16:0/3681832895 >> 172.21.15.16:0/3681832895 conn(0x7facf00fc7d0 msgr2=0x7facf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 -- 172.21.15.16:0/3681832895 shutdown_connections 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.135+0000 7facf6676640 1 -- 172.21.15.16:0/3681832895 wait complete. 2024-09-12T02:50:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.136+0000 7facf6676640 1 Processor -- start 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.136+0000 7facf6676640 1 -- start start 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0100e20 0x7facf01993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf0199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facf0110f80 con 0x7facf0100e20 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf0110e00 con 0x7facf0102ef0 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf6676640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111100 con 0x7facf0102570 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facef7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facf4c13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf0199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facef7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38014/0 (socket says 172.21.15.16:38014) 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7facef7fe640 1 -- 172.21.15.16:0/3346044772 learned_addr learned my addr 172.21.15.16:0/3346044772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.137+0000 7faceffff640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0100e20 0x7facf01993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 -- 172.21.15.16:0/3346044772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 msgr2=0x7facf0199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf0199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 -- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0100e20 msgr2=0x7facf01993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0100e20 0x7facf01993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 -- 172.21.15.16:0/3346044772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf01a47d0 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facef7fe640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7face400e9f0 tx=0x7face400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7faced7fa640 1 -- 172.21.15.16:0/3346044772 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face400cd80 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf01a4ac0 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7faced7fa640 1 -- 172.21.15.16:0/3346044772 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face4010910 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a5050 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.138+0000 7faced7fa640 1 -- 172.21.15.16:0/3346044772 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4018980 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 Processor -- start 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7faced7fa640 1 -- 172.21.15.16:0/3346044772 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7face4010ac0 con 0x7facf0102570 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 -- start start 2024-09-12T02:50:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04aafa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 0x555f04abe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f04a152a0 con 0x555f04abed50 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f049bcb10 con 0x555f04aaf640 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf8a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555f04a11260 con 0x555f04aaff60 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facf4c13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04aafa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7faceffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 0x555f04abe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48470/0 (socket says 172.21.15.16:48470) 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.141+0000 7facef7fe640 1 -- 172.21.15.16:0/3623835347 learned_addr learned my addr 172.21.15.16:0/3623835347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 -- 172.21.15.16:0/3623835347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 msgr2=0x555f04aafa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04aafa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 -- 172.21.15.16:0/3623835347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 msgr2=0x555f04ac1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 -- 172.21.15.16:0/3623835347 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f04786b90 con 0x555f04aaff60 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7facef7fe640 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faced7fa640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 0x7facd80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faceffff640 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 0x555f04abe810 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7face0009c00 tx=0x7face0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b331659513456111 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7facd67fc640 1 -- 172.21.15.16:0/3623835347 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0004230 con 0x555f04aaff60 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7faced7fa640 1 -- 172.21.15.16:0/3346044772 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face4018720 con 0x7facf0102570 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7facf4c13640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 0x7facd80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7facd67fc640 1 -- 172.21.15.16:0/3623835347 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face00043d0 con 0x555f04aaff60 2024-09-12T02:50:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.142+0000 7facd67fc640 1 -- 172.21.15.16:0/3623835347 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0004dd0 con 0x555f04aaff60 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 -- 172.21.15.16:0/3623835347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 msgr2=0x555f04abe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 0x555f04abe810 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7face0009c00 tx=0x7face0011810 comp rx=0 tx=0).stop 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf4c13640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 0x7facd80a5240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7facf006b050 tx=0x7face8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 -- 172.21.15.16:0/3623835347 shutdown_connections 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ac1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaff60 0x555f04abe810 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 --2- 172.21.15.16:0/3623835347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04aafa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 -- 172.21.15.16:0/3623835347 >> 172.21.15.16:0/3623835347 conn(0x555f04aabcf0 msgr2=0x555f04aac0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 -- 172.21.15.16:0/3623835347 shutdown_connections 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.143+0000 7facf8a4f740 1 -- 172.21.15.16:0/3623835347 wait complete. 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 -- start start 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 Processor -- start 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaf640 0x555f04abbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 0x555f04abc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f04ac1920 con 0x555f04abed50 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f0499d6e0 con 0x555f04aaff60 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555f04ac1aa0 con 0x555f04aaf640 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.144+0000 7facf8a4f740 10 client.0 _ll_register_callbacks cb 0x555f049c1a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 0x555f04abc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facf4c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facf4c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48474/0 (socket says 172.21.15.16:48474) 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facf4c13640 1 -- 172.21.15.16:0/1422595784 learned_addr learned my addr 172.21.15.16:0/1422595784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7faceffff640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaf640 0x555f04abbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 -- 172.21.15.16:0/1422595784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaf640 msgr2=0x555f04abbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaf640 0x555f04abbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 -- 172.21.15.16:0/1422595784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 msgr2=0x555f04abc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facef7fe640 1 -- 172.21.15.16:0/1422595784 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f04acae80 con 0x555f04aaff60 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.145+0000 7facf4c13640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7facef7fe640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 0x555f04abc1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7face400e9c0 tx=0x7face409b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611a876824f707e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7faccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face409b9a0 con 0x555f04aaff60 2024-09-12T02:50:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face00043d0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face40860e0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7face409fca0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7facf8a4f740 10 client.25601 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f04acd8c0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.146+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face00043d0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7face40976f0 con 0x555f04aaff60 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 10 client.25601 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 10 client.25601 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 20 client.25601 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04aced20 0x555f04ad1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 4 client.25601 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f04a229d0 con 0x555f04aced20 2024-09-12T02:50:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7faceffff640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04aced20 0x555f04ad1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7faceffff640 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04aced20 0x555f04ad1160 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=476ce7b1600e6c9 server_cookie=226d005dbab29a23 in_seq=0 out_seq=0 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.147+0000 7faccf7fe640 10 client.25601 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7faccf7fe640 1 -- 172.21.15.16:0/1422595784 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555f04a229d0 con 0x555f04aced20 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7faccf7fe640 10 client.25601 handle_command_reply: tid=0 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 client.25601 shutdown 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 msgr2=0x555f04abc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 0x555f04abc1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7face400e9c0 tx=0x7face409b770 comp rx=0 tx=0).stop 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 shutdown_connections 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04aced20 0x555f04ad1160 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04abc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7faccf7fe640 0 client.25601 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaff60 0x555f04abc1c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1422595784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04aaf640 0x555f04abbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.148+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 >> 172.21.15.16:0/1422595784 conn(0x555f04aabcf0 msgr2=0x555f04ab1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.149+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 shutdown_connections 2024-09-12T02:50:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.149+0000 7facf8a4f740 1 -- 172.21.15.16:0/1422595784 wait complete. 2024-09-12T02:50:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.149+0000 7facf8a4f740 20 client.25601 trim_cache size 0 max 16384 2024-09-12T02:50:22.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 Processor -- start 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 -- start start 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04ababc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ad6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ab3c0 con 0x555f04abed50 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf01acfc0 con 0x555f04aaf640 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf8a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facf01aba90 con 0x555f04ad6a30 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48478/0 (socket says 172.21.15.16:48478) 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facef7fe640 1 -- 172.21.15.16:0/930673542 learned_addr learned my addr 172.21.15.16:0/930673542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7facf4c13640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ad6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.173+0000 7faceffff640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04ababc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 -- 172.21.15.16:0/930673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 msgr2=0x555f04ad6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ad6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 -- 172.21.15.16:0/930673542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 msgr2=0x555f04ababc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04ababc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 -- 172.21.15.16:0/930673542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f04aa7260 con 0x555f04abed50 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7faceffff640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04ababc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facf4c13640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ad6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facef7fe640 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7face409a880 tx=0x7face4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcc7d03ad18a11b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facd5ffb640 1 -- 172.21.15.16:0/930673542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face409b9a0 con 0x555f04abed50 2024-09-12T02:50:22.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facd5ffb640 1 -- 172.21.15.16:0/930673542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face40a4560 con 0x555f04abed50 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.174+0000 7facd5ffb640 1 -- 172.21.15.16:0/930673542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face409b100 con 0x555f04abed50 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- 172.21.15.16:0/930673542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 msgr2=0x555f04ab71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7face409a880 tx=0x7face4000bc0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- 172.21.15.16:0/930673542 shutdown_connections 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ad6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04abed50 0x555f04ab71c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 --2- 172.21.15.16:0/930673542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04aaf640 0x555f04ababc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- 172.21.15.16:0/930673542 >> 172.21.15.16:0/930673542 conn(0x555f04aabcf0 msgr2=0x555f04abc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- 172.21.15.16:0/930673542 shutdown_connections 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- 172.21.15.16:0/930673542 wait complete. 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.175+0000 7facf8a4f740 1 -- start start 2024-09-12T02:50:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 Processor -- start 2024-09-12T02:50:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 0x555f04ac6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ac7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ac830 con 0x555f04aaf640 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ac160 con 0x555f04abed50 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facf01a93a0 con 0x555f04ad6a30 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facf8a4f740 10 client.0 _ll_register_callbacks cb 0x7face4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7faceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7faceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48494/0 (socket says 172.21.15.16:48494) 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7faceffff640 1 -- 172.21.15.16:0/1730222139 learned_addr learned my addr 172.21.15.16:0/1730222139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.176+0000 7facef7fe640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 0x555f04ac6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 -- 172.21.15.16:0/1730222139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 msgr2=0x555f04ac7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7facf4c13640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ac7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ac7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 -- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 msgr2=0x555f04ac6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:22.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 0x555f04ac6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 -- 172.21.15.16:0/1730222139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f04abcb70 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7facf4c13640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ac7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7facef7fe640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 0x555f04ac6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faceffff640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7face0009990 tx=0x7face0004850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4384bbe4725800 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0032060 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face40a4560 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.177+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7face0004250 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7face0004450 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 10 client.16002 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f04ab3fe0 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face40a4560 con 0x555f04aaf640 2024-09-12T02:50:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7face0024040 con 0x555f04aaf640 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 10 client.16002 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 10 client.16002 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 20 client.16002 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.178+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04b3e650 0x555f04b40a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.179+0000 7facf8a4f740 4 client.16002 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.7, "await_for": 45.0}] 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.179+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.7, "await_for": 45.0}) -- 0x555f04a1cd20 con 0x555f04b3e650 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.179+0000 7facef7fe640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04b3e650 0x555f04b40a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.179+0000 7faccf7fe640 10 client.16002 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.179+0000 7facef7fe640 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04b3e650 0x555f04b40a70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbdbcd8c87bbc21f server_cookie=783574b19a2c535b in_seq=0 out_seq=0 2024-09-12T02:50:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:21 smithi033 ceph-mon[35092]: pgmap v410: 97 pgs: 97 active+clean; 87 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-09-12T02:50:22.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.281+0000 7faccf7fe640 1 -- 172.21.15.16:0/1730222139 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555f04a1cd20 con 0x555f04b3e650 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.281+0000 7faccf7fe640 10 client.16002 handle_command_reply: tid=0 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.281+0000 7facf8a4f740 1 client.16002 shutdown 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 msgr2=0x555f04ab5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7face0009990 tx=0x7face0004850 comp rx=0 tx=0).stop 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 shutdown_connections 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f04b3e650 0x555f04b40a70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f04ad6a30 0x555f04ac7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7faccf7fe640 0 client.16002 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f04abed50 0x555f04ac6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 --2- 172.21.15.16:0/1730222139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f04aaf640 0x555f04ab5e90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 >> 172.21.15.16:0/1730222139 conn(0x555f04aabcf0 msgr2=0x555f04abc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 shutdown_connections 2024-09-12T02:50:22.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 1 -- 172.21.15.16:0/1730222139 wait complete. 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.282+0000 7facf8a4f740 20 client.16002 trim_cache size 0 max 16384 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 30, 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 10, 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "e792f047": { 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 10, 2024-09-12T02:50:22.284 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.7, 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:50:22.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 msgr2=0x7facd80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 0x7facd80a5240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7facf006b050 tx=0x7face8005c50 comp rx=0 tx=0).stop 2024-09-12T02:50:22.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 msgr2=0x7facf0199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7face400e9f0 tx=0x7face400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 shutdown_connections 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf0102ef0 0x7facf0199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facf0102570 0x7facf0199910 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facd80a2d80 0x7facd80a5240 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 --2- 172.21.15.16:0/3346044772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facf0100e20 0x7facf01993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 >> 172.21.15.16:0/3346044772 conn(0x7facf00fc7d0 msgr2=0x7facf01068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 shutdown_connections 2024-09-12T02:50:22.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:22.283+0000 7facf6676640 1 -- 172.21.15.16:0/3346044772 wait complete. 2024-09-12T02:50:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:21 smithi184 ceph-mon[35321]: pgmap v410: 97 pgs: 97 active+clean; 87 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 318 op/s 2024-09-12T02:50:22.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 12.08 MB/sec warmup 84 sec latency 425.853 ms 2024-09-12T02:50:23.117 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e792f047', quiesce took 5.4 seconds. Will release after: 19.299999999999997 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 -- 172.21.15.16:0/509423679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 msgr2=0x7f2c4c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 --2- 172.21.15.16:0/509423679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c103120 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009900 tx=0x7f2c3c02a480 comp rx=0 tx=0).stop 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 -- 172.21.15.16:0/509423679 shutdown_connections 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 --2- 172.21.15.16:0/509423679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c10c170 0x7f2c4c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 --2- 172.21.15.16:0/509423679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c103660 0x7f2c4c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 --2- 172.21.15.16:0/509423679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c103120 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 -- 172.21.15.16:0/509423679 >> 172.21.15.16:0/509423679 conn(0x7f2c4c0fc7a0 msgr2=0x7f2c4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 -- 172.21.15.16:0/509423679 shutdown_connections 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.320+0000 7f2c52587640 1 -- 172.21.15.16:0/509423679 wait complete. 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 Processor -- start 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 -- start start 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 0x7f2c4c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 0x7f2c4c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c111f70 con 0x7f2c4c102d40 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c111df0 con 0x7f2c4c10c170 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c52587640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1120f0 con 0x7f2c4c103660 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48514/0 (socket says 172.21.15.16:48514) 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.321+0000 7f2c4bfff640 1 -- 172.21.15.16:0/585691975 learned_addr learned my addr 172.21.15.16:0/585691975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 0x7f2c4c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 -- 172.21.15.16:0/585691975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 msgr2=0x7f2c4c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c50b24640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 0x7f2c4c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 0x7f2c4c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 -- 172.21.15.16:0/585691975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 msgr2=0x7f2c4c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 0x7f2c4c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 -- 172.21.15.16:0/585691975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c1a68f0 con 0x7f2c4c102d40 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 0x7f2c4c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c50b24640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 0x7f2c4c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006010 tx=0x7f2c3c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c497fa640 1 -- 172.21.15.16:0/585691975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c03f030 con 0x7f2c4c102d40 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c497fa640 1 -- 172.21.15.16:0/585691975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c3c004440 con 0x7f2c4c102d40 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c1a6b80 con 0x7f2c4c102d40 2024-09-12T02:50:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c497fa640 1 -- 172.21.15.16:0/585691975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c035070 con 0x7f2c4c102d40 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.322+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c1a7060 con 0x7f2c4c102d40 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.324+0000 7f2c497fa640 1 -- 172.21.15.16:0/585691975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2c3c042030 con 0x7f2c4c102d40 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 Processor -- start 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 -- start start 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e295090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e295a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f30e1fb2f0 con 0x55f30e2a0370 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f30e1a2cf0 con 0x55f30e2955d0 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c54960740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f30e1f72b0 con 0x55f30e296b40 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48518/0 (socket says 172.21.15.16:48518) 2024-09-12T02:50:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/2412309540 learned_addr learned my addr 172.21.15.16:0/2412309540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e295a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c50b24640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e295090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.325+0000 7f2c497fa640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 0x7f2c340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/2412309540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 msgr2=0x55f30e295090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e295090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/2412309540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 msgr2=0x55f30e295a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e295a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/2412309540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f30e1873a0 con 0x55f30e2a0370 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c50b24640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e295090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e295a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c50b24640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 0x7f2c340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c497fa640 1 -- 172.21.15.16:0/585691975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2c3c0ef4d0 con 0x7f2c4c102d40 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009be0 tx=0x7f2c40010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66320e59a7d1a371 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c327fc640 1 -- 172.21.15.16:0/2412309540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40004030 con 0x55f30e2a0370 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c327fc640 1 -- 172.21.15.16:0/2412309540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c400041d0 con 0x55f30e2a0370 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.326+0000 7f2c327fc640 1 -- 172.21.15.16:0/2412309540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c400126b0 con 0x55f30e2a0370 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- 172.21.15.16:0/2412309540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 msgr2=0x55f30e2a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c50b24640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 0x7f2c340a5040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005fd0 tx=0x7f2c44004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009be0 tx=0x7f2c40010740 comp rx=0 tx=0).stop 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- 172.21.15.16:0/2412309540 shutdown_connections 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2a0370 0x55f30e2a2760 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e295a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 --2- 172.21.15.16:0/2412309540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e295090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- 172.21.15.16:0/2412309540 >> 172.21.15.16:0/2412309540 conn(0x55f30e25e310 msgr2=0x55f30e291c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- 172.21.15.16:0/2412309540 shutdown_connections 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- 172.21.15.16:0/2412309540 wait complete. 2024-09-12T02:50:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.327+0000 7f2c54960740 1 -- start start 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 Processor -- start 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e29fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e2a0370 0x55f30e29d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f30e2a3070 con 0x55f30e296b40 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f30e1b6060 con 0x55f30e2955d0 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f30e2a31f0 con 0x55f30e2a0370 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c54960740 10 client.0 _ll_register_callbacks cb 0x55f30df098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48534/0 (socket says 172.21.15.16:48534) 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/4123349813 learned_addr learned my addr 172.21.15.16:0/4123349813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e29fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c50b24640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e2a0370 0x55f30e29d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/4123349813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e2a0370 msgr2=0x55f30e29d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e2a0370 0x55f30e29d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 msgr2=0x55f30e29fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e29fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.328+0000 7f2c4b7fe640 1 -- 172.21.15.16:0/4123349813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f30df6c3c0 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2c40003d30 tx=0x7f2c40003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ddfc069623ec79 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40003ff0 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2c40004380 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c400041d0 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2c40005050 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 10 client.16014 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f30e2aea80 con 0x55f30e296b40 2024-09-12T02:50:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c400041d0 con 0x55f30e296b40 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2c4000f040 con 0x55f30e296b40 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 10 client.16014 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 10 client.16014 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 20 client.16014 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e2afdd0 0x55f30e2b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 4 client.16014 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.329+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f30e208860 con 0x55f30e2afdd0 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.330+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e2afdd0 0x55f30e2b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.330+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e2afdd0 0x55f30e2b21c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6331978e0ce5270a server_cookie=dd19d0a49f3e31fb in_seq=0 out_seq=0 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.330+0000 7f2c2b7fe640 10 client.16014 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/4123349813 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f30e208860 con 0x55f30e2afdd0 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c2b7fe640 10 client.16014 handle_command_reply: tid=0 2024-09-12T02:50:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 client.16014 shutdown 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 msgr2=0x55f30e29cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2c40003d30 tx=0x7f2c40003ad0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 shutdown_connections 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e2afdd0 0x55f30e2b21c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e2a0370 0x55f30e29d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e296b40 0x55f30e29cf90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c2b7fe640 0 client.16014 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 --2- 172.21.15.16:0/4123349813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2955d0 0x55f30e29fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 >> 172.21.15.16:0/4123349813 conn(0x55f30e25e310 msgr2=0x55f30e297960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 shutdown_connections 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.331+0000 7f2c54960740 1 -- 172.21.15.16:0/4123349813 wait complete. 2024-09-12T02:50:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.332+0000 7f2c54960740 20 client.16014 trim_cache size 0 max 16384 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 Processor -- start 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 -- start start 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e298120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e298660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1af750 con 0x55f30e2955d0 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b1350 con 0x55f30e2a0370 2024-09-12T02:50:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c54960740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1afe20 con 0x55f30e296b40 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48550/0 (socket says 172.21.15.16:48550) 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c4bfff640 1 -- 172.21.15.16:0/3571343615 learned_addr learned my addr 172.21.15.16:0/3571343615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.345+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e298120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c50b24640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e298660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 -- 172.21.15.16:0/3571343615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 msgr2=0x55f30e298120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e298120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 -- 172.21.15.16:0/3571343615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 msgr2=0x55f30e298660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e298660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 -- 172.21.15.16:0/3571343615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f30e2a0050 con 0x55f30e2955d0 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e298120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c000c00 tx=0x7f2c3c0b8110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaea9ee2f723d9b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c31ffb640 1 -- 172.21.15.16:0/3571343615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c049040 con 0x55f30e2955d0 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c31ffb640 1 -- 172.21.15.16:0/3571343615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c3c0bbd40 con 0x55f30e2955d0 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.346+0000 7f2c31ffb640 1 -- 172.21.15.16:0/3571343615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0b83b0 con 0x55f30e2955d0 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- 172.21.15.16:0/3571343615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 msgr2=0x55f30e2ac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c000c00 tx=0x7f2c3c0b8110 comp rx=0 tx=0).stop 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- 172.21.15.16:0/3571343615 shutdown_connections 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e298660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e298120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 --2- 172.21.15.16:0/3571343615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2ac310 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- 172.21.15.16:0/3571343615 >> 172.21.15.16:0/3571343615 conn(0x55f30e25e310 msgr2=0x55f30e29f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- 172.21.15.16:0/3571343615 shutdown_connections 2024-09-12T02:50:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- 172.21.15.16:0/3571343615 wait complete. 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 -- start start 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.347+0000 7f2c54960740 1 Processor -- start 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e299b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e29a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b0bc0 con 0x55f30e2955d0 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b04f0 con 0x55f30e2a0370 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1ad730 con 0x55f30e296b40 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c54960740 10 client.0 _ll_register_callbacks cb 0x7f2c3c0b8060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48560/0 (socket says 172.21.15.16:48560) 2024-09-12T02:50:23.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 -- 172.21.15.16:0/1897879505 learned_addr learned my addr 172.21.15.16:0/1897879505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e299b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c50b24640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e29a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 -- 172.21.15.16:0/1897879505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 msgr2=0x55f30e299b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e299b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 -- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 msgr2=0x55f30e29a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e29a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 -- 172.21.15.16:0/1897879505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f30e29a7c0 con 0x55f30e2955d0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c4bfff640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0b8710 tx=0x7f2c3c0f2ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8109e62f61543d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.348+0000 7f2c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0da060 con 0x55f30e2955d0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2c3c0b83b0 con 0x55f30e2955d0 2024-09-12T02:50:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c3c0bbd40 con 0x55f30e2955d0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2c3c049070 con 0x55f30e2955d0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 10 client.16020 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f30e31a850 con 0x55f30e2955d0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c3c0bbd40 con 0x55f30e2955d0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2c3c0bb190 con 0x55f30e2955d0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 10 client.16020 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 10 client.16020 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 20 client.16020 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e31ae60 0x55f30e31d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 4 client.16020 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.349+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f30e202d70 con 0x55f30e31ae60 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e31ae60 0x55f30e31d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c4b7fe640 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e31ae60 0x55f30e31d250 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ba154986aeed85a server_cookie=d3aaba0e0c925ba7 in_seq=0 out_seq=0 2024-09-12T02:50:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c2b7fe640 10 client.16020 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c2b7fe640 1 -- 172.21.15.16:0/1897879505 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f30e202d70 con 0x55f30e31ae60 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c2b7fe640 10 client.16020 handle_command_reply: tid=0 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 client.16020 shutdown 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 msgr2=0x55f30e2a8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0b8710 tx=0x7f2c3c0f2ca0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 shutdown_connections 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f30e31ae60 0x55f30e31d250 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f30e2a0370 0x55f30e29a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f30e296b40 0x55f30e299b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c2b7fe640 0 client.16020 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 --2- 172.21.15.16:0/1897879505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f30e2955d0 0x55f30e2a8ab0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 >> 172.21.15.16:0/1897879505 conn(0x55f30e25e310 msgr2=0x55f30e29b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.350+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 shutdown_connections 2024-09-12T02:50:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c54960740 1 -- 172.21.15.16:0/1897879505 wait complete. 2024-09-12T02:50:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c54960740 20 client.16020 trim_cache size 0 max 16384 2024-09-12T02:50:23.359 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 msgr2=0x7f2c340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 0x7f2c340a5040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005fd0 tx=0x7f2c44004a40 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 msgr2=0x7f2c4c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006010 tx=0x7f2c3c004040 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 shutdown_connections 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c4c10c170 0x7f2c4c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c4c103660 0x7f2c4c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2c340a2b80 0x7f2c340a5040 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 --2- 172.21.15.16:0/585691975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4c102d40 0x7f2c4c1009a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 >> 172.21.15.16:0/585691975 conn(0x7f2c4c0fc7a0 msgr2=0x7f2c4c0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 shutdown_connections 2024-09-12T02:50:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:23.351+0000 7f2c52587640 1 -- 172.21.15.16:0/585691975 wait complete. 2024-09-12T02:50:23.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.94 MB/sec warmup 85 sec latency 1425.884 ms 2024-09-12T02:50:23.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:50:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:23 smithi033 ceph-mon[35092]: pgmap v411: 97 pgs: 97 active+clean; 88 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-09-12T02:50:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:23 smithi184 ceph-mon[35321]: pgmap v411: 97 pgs: 97 active+clean; 88 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-09-12T02:50:24.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:23 smithi016 ceph-mon[30925]: pgmap v411: 97 pgs: 97 active+clean; 88 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-09-12T02:50:24.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.80 MB/sec warmup 86 sec latency 2425.914 ms 2024-09-12T02:50:24.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- 172.21.15.16:0/824688128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13481036e0 msgr2=0x7f1348107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 --2- 172.21.15.16:0/824688128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13481036e0 0x7f1348107b90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f133c009930 tx=0x7f133c02a420 comp rx=0 tx=0).stop 2024-09-12T02:50:24.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- 172.21.15.16:0/824688128 shutdown_connections 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 --2- 172.21.15.16:0/824688128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f134810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 --2- 172.21.15.16:0/824688128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13481036e0 0x7f1348107b90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 --2- 172.21.15.16:0/824688128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1348102d30 0x7f1348103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- 172.21.15.16:0/824688128 >> 172.21.15.16:0/824688128 conn(0x7f13480fc7d0 msgr2=0x7f13480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- 172.21.15.16:0/824688128 shutdown_connections 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- 172.21.15.16:0/824688128 wait complete. 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 Processor -- start 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.704+0000 7f134facf640 1 -- start start 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 0x7f1348100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f1348101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1348112060 con 0x7f1348102d30 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1348111ee0 con 0x7f13481082a0 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13481121e0 con 0x7f13481036e0 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48578/0 (socket says 172.21.15.16:48578) 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 -- 172.21.15.16:0/764934244 learned_addr learned my addr 172.21.15.16:0/764934244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d06a640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 0x7f1348100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134e06c640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f1348101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 -- 172.21.15.16:0/764934244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 msgr2=0x7f1348100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 0x7f1348100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 -- 172.21.15.16:0/764934244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 msgr2=0x7f1348101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f1348101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 -- 172.21.15.16:0/764934244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13481a6a60 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d06a640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 0x7f1348100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134e06c640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f1348101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134d86b640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f134000d950 tx=0x7f134000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f1346ffd640 1 -- 172.21.15.16:0/764934244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340017030 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f1346ffd640 1 -- 172.21.15.16:0/764934244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1340004020 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13481a6d50 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.705+0000 7f1346ffd640 1 -- 172.21.15.16:0/764934244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13400047a0 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.706+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13481a7200 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1346ffd640 1 -- 172.21.15.16:0/764934244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f13400041d0 con 0x7f1348102d30 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 Processor -- start 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 -- start start 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1bca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c31b1222f0 con 0x55c31b1c7370 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c31b0c9cf0 con 0x55c31b1bc5d0 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.708+0000 7f1351ea8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c31b11e2b0 con 0x55c31b1bdb40 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48586/0 (socket says 172.21.15.16:48586) 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 -- 172.21.15.16:0/710733318 learned_addr learned my addr 172.21.15.16:0/710733318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f1346ffd640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 0x7f13300a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134e06c640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d86b640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1bca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134e06c640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 0x7f13300a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 -- 172.21.15.16:0/710733318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 msgr2=0x55c31b1bc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1bc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f1346ffd640 1 -- 172.21.15.16:0/764934244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f13400cead0 con 0x7f1348102d30 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 -- 172.21.15.16:0/710733318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 msgr2=0x55c31b1bca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1bca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 -- 172.21.15.16:0/710733318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c31b0ae3a0 con 0x55c31b1c7370 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134e06c640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134d06a640 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f13480697d0 tx=0x7f133c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992959426ea33d5f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f134e06c640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 0x7f13300a5040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1338005fd0 tx=0x7f1338004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.709+0000 7f132ffff640 1 -- 172.21.15.16:0/710733318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c03f070 con 0x55c31b1c7370 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f132ffff640 1 -- 172.21.15.16:0/710733318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c032e30 con 0x55c31b1c7370 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f132ffff640 1 -- 172.21.15.16:0/710733318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c035070 con 0x55c31b1c7370 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 -- 172.21.15.16:0/710733318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 msgr2=0x55c31b1c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f13480697d0 tx=0x7f133c005e80 comp rx=0 tx=0).stop 2024-09-12T02:50:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 -- 172.21.15.16:0/710733318 shutdown_connections 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1c9760 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1bca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 --2- 172.21.15.16:0/710733318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 -- 172.21.15.16:0/710733318 >> 172.21.15.16:0/710733318 conn(0x55c31b185310 msgr2=0x55c31b1b8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 -- 172.21.15.16:0/710733318 shutdown_connections 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.710+0000 7f1351ea8740 1 -- 172.21.15.16:0/710733318 wait complete. 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 -- start start 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 Processor -- start 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bc5d0 0x55c31b1c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 0x55c31b1c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c31b1ca010 con 0x55c31b1bdb40 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c31b0dd060 con 0x55c31b1c7370 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c31b1ca190 con 0x55c31b1bc5d0 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f1351ea8740 10 client.0 _ll_register_callbacks cb 0x55c31ae308c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48600/0 (socket says 172.21.15.16:48600) 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f134d06a640 1 -- 172.21.15.16:0/3071924817 learned_addr learned my addr 172.21.15.16:0/3071924817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.711+0000 7f134d86b640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bc5d0 0x55c31b1c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134e06c640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 0x55c31b1c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 -- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bc5d0 msgr2=0x55c31b1c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bc5d0 0x55c31b1c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 -- 172.21.15.16:0/3071924817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 msgr2=0x55c31b1c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 0x55c31b1c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 -- 172.21.15.16:0/3071924817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c31ae933c0 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134e06c640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 0x55c31b1c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f134d06a640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f133c000c00 tx=0x7f133c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8099eda9f313d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f132d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c03f070 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f133c004450 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c032e30 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f133c035040 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f1351ea8740 10 client.16032 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.712+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c31b1d5a70 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c032e30 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f133c0526f0 con 0x55c31b1bdb40 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 10 client.16032 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 10 client.16032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 20 client.16032 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b1d6dc0 0x55c31b1d91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 4 client.16032 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c31b12f860 con 0x55c31b1d6dc0 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.713+0000 7f134d86b640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b1d6dc0 0x55c31b1d91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.714+0000 7f132d7fa640 10 client.16032 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.714+0000 7f134d86b640 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b1d6dc0 0x55c31b1d91b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa1010798116c242 server_cookie=f4a60bd0300ee8db in_seq=0 out_seq=0 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.714+0000 7f132d7fa640 1 -- 172.21.15.16:0/3071924817 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c31b12f860 con 0x55c31b1d6dc0 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.714+0000 7f132d7fa640 10 client.16032 handle_command_reply: tid=0 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.714+0000 7f1351ea8740 1 client.16032 shutdown 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 msgr2=0x55c31b1c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f133c000c00 tx=0x7f133c0317e0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 shutdown_connections 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1c7370 0x55c31b1c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1c3fa0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b1d6dc0 0x55c31b1d91b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3071924817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bc5d0 0x55c31b1c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f132d7fa640 0 client.16032 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 >> 172.21.15.16:0/3071924817 conn(0x55c31b185310 msgr2=0x55c31b1be960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 shutdown_connections 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 1 -- 172.21.15.16:0/3071924817 wait complete. 2024-09-12T02:50:24.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.715+0000 7f1351ea8740 20 client.16032 trim_cache size 0 max 16384 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 Processor -- start 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 -- start start 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1de7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13481af6b0 con 0x55c31b1c7370 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13481b12b0 con 0x55c31b1bc5d0 2024-09-12T02:50:24.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f1351ea8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13481afd80 con 0x55c31b1bdb40 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f134e06c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f134e06c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48610/0 (socket says 172.21.15.16:48610) 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f134e06c640 1 -- 172.21.15.16:0/484126597 learned_addr learned my addr 172.21.15.16:0/484126597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f134d86b640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.728+0000 7f134d06a640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1de7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 -- 172.21.15.16:0/484126597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 msgr2=0x55c31b1de7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1de7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 -- 172.21.15.16:0/484126597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 msgr2=0x55c31b1d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 -- 172.21.15.16:0/484126597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c31b1c68a0 con 0x55c31b1c7370 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134d06a640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1de7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:24.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f134e06c640 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f13380128e0 tx=0x7f1338009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f6fcf728d8e11f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f132f7fe640 1 -- 172.21.15.16:0/484126597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338004d20 con 0x55c31b1c7370 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f132f7fe640 1 -- 172.21.15.16:0/484126597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1338005020 con 0x55c31b1c7370 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f132f7fe640 1 -- 172.21.15.16:0/484126597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133801d640 con 0x55c31b1c7370 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f1351ea8740 1 -- 172.21.15.16:0/484126597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 msgr2=0x55c31b1decf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.729+0000 7f1351ea8740 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f13380128e0 tx=0x7f1338009660 comp rx=0 tx=0).stop 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- 172.21.15.16:0/484126597 shutdown_connections 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1c7370 0x55c31b1decf0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1bdb40 0x55c31b1de7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- 172.21.15.16:0/484126597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- 172.21.15.16:0/484126597 >> 172.21.15.16:0/484126597 conn(0x55c31b185310 msgr2=0x55c31b1bf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- 172.21.15.16:0/484126597 shutdown_connections 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- 172.21.15.16:0/484126597 wait complete. 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- start start 2024-09-12T02:50:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 Processor -- start 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1c7370 0x55c31b240780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13481b0b20 con 0x55c31b1bdb40 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13481b0450 con 0x55c31b1bc5d0 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13481ad6b0 con 0x55c31b1c7370 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.730+0000 7f1351ea8740 10 client.0 _ll_register_callbacks cb 0x7f134009ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48612/0 (socket says 172.21.15.16:48612) 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 -- 172.21.15.16:0/3813368476 learned_addr learned my addr 172.21.15.16:0/3813368476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134e06c640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1c7370 0x55c31b240780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d86b640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 -- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1c7370 msgr2=0x55c31b240780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1c7370 0x55c31b240780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 -- 172.21.15.16:0/3813368476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 msgr2=0x55c31b1cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 -- 172.21.15.16:0/3813368476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c31b240cc0 con 0x55c31b1bdb40 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d86b640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f134d06a640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a940 tx=0x7f133c032860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251d068514ba9ad0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f132d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c03f040 con 0x55c31b1bdb40 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f133c052480 con 0x55c31b1bdb40 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.731+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1338005020 con 0x55c31b1bdb40 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f133c062050 con 0x55c31b1bdb40 2024-09-12T02:50:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 10 client.16038 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c31b241e10 con 0x55c31b1bdb40 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1338005020 con 0x55c31b1bdb40 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f133c05b470 con 0x55c31b1bdb40 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 10 client.16038 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 10 client.16038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 20 client.16038 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b242550 0x55c31b244970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 4 client.16038 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c31b129d70 con 0x55c31b242550 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.732+0000 7f134d86b640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b242550 0x55c31b244970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f134d86b640 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b242550 0x55c31b244970 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea6f30a5aa0c82f2 server_cookie=5f896907ee78dce8 in_seq=0 out_seq=0 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f132d7fa640 10 client.16038 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f132d7fa640 1 -- 172.21.15.16:0/3813368476 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c31b129d70 con 0x55c31b242550 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f132d7fa640 10 client.16038 handle_command_reply: tid=0 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 client.16038 shutdown 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 msgr2=0x55c31b1cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a940 tx=0x7f133c032860 comp rx=0 tx=0).stop 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 shutdown_connections 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55c31b242550 0x55c31b244970 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c31b1c7370 0x55c31b240780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c31b1bdb40 0x55c31b1cbb30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f132d7fa640 0 client.16038 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:50:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 --2- 172.21.15.16:0/3813368476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c31b1bc5d0 0x55c31b1cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 >> 172.21.15.16:0/3813368476 conn(0x55c31b185310 msgr2=0x55c31b1c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 shutdown_connections 2024-09-12T02:50:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 1 -- 172.21.15.16:0/3813368476 wait complete. 2024-09-12T02:50:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.733+0000 7f1351ea8740 20 client.16038 trim_cache size 0 max 16384 2024-09-12T02:50:24.743 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 msgr2=0x7f13300a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 0x7f13300a5040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1338005fd0 tx=0x7f1338004ab0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 msgr2=0x7f1348100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f134000d950 tx=0x7f134000de10 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 shutdown_connections 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13481082a0 0x7f1348101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13481036e0 0x7f1348100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13300a2b80 0x7f13300a5040 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 --2- 172.21.15.16:0/764934244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1348102d30 0x7f1348100990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 >> 172.21.15.16:0/764934244 conn(0x7f13480fc7d0 msgr2=0x7f1348105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 shutdown_connections 2024-09-12T02:50:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:24.734+0000 7f134facf640 1 -- 172.21.15.16:0/764934244 wait complete. 2024-09-12T02:50:25.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:50:25.709 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.67 MB/sec warmup 87 sec latency 3426.323 ms 2024-09-12T02:50:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:25 smithi016 ceph-mon[30925]: pgmap v412: 97 pgs: 97 active+clean; 90 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 -- 172.21.15.16:0/744185218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720102570 msgr2=0x7fc7201029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 --2- 172.21.15.16:0/744185218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720102570 0x7fc7201029b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc710009900 tx=0x7fc71002a480 comp rx=0 tx=0).stop 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 -- 172.21.15.16:0/744185218 shutdown_connections 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 --2- 172.21.15.16:0/744185218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102ef0 0x7fc72010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 --2- 172.21.15.16:0/744185218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720102570 0x7fc7201029b0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 --2- 172.21.15.16:0/744185218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720100e20 0x7fc720101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 -- 172.21.15.16:0/744185218 >> 172.21.15.16:0/744185218 conn(0x7fc7200fc7d0 msgr2=0x7fc7200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 -- 172.21.15.16:0/744185218 shutdown_connections 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 -- 172.21.15.16:0/744185218 wait complete. 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.185+0000 7fc72597e640 1 Processor -- start 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 -- start start 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102570 0x7fc72019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 0x7fc72019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc720112060 con 0x7fc720100e20 2024-09-12T02:50:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc720111ee0 con 0x7fc720102ef0 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc72597e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201121e0 con 0x7fc720102570 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39760/0 (socket says 172.21.15.16:39760) 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 -- 172.21.15.16:0/1902754505 learned_addr learned my addr 172.21.15.16:0/1902754505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102570 0x7fc72019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 0x7fc72019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 -- 172.21.15.16:0/1902754505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102570 msgr2=0x7fc72019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102570 0x7fc72019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 -- 172.21.15.16:0/1902754505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 msgr2=0x7fc72019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 0x7fc72019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 -- 172.21.15.16:0/1902754505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72019f670 con 0x7fc720100e20 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 0x7fc72019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.186+0000 7fc71effd640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc70800d950 tx=0x7fc70800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.187+0000 7fc72497c640 1 -- 172.21.15.16:0/1902754505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708017070 con 0x7fc720100e20 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.187+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72019f960 con 0x7fc720100e20 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.187+0000 7fc72497c640 1 -- 172.21.15.16:0/1902754505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc708009c30 con 0x7fc720100e20 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.187+0000 7fc72497c640 1 -- 172.21.15.16:0/1902754505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70800f6b0 con 0x7fc720100e20 2024-09-12T02:50:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.187+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7201a7240 con 0x7fc720100e20 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.189+0000 7fc727d57740 1 Processor -- start 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.189+0000 7fc72497c640 1 -- 172.21.15.16:0/1902754505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc7080109d0 con 0x7fc720100e20 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.189+0000 7fc727d57740 1 -- start start 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4aaaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ed4a102f0 con 0x561ed4aabb40 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ed49b7cf0 con 0x561ed4ab5370 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc727d57740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ed4a0c2b0 con 0x561ed4aaa5d0 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4aaaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39766/0 (socket says 172.21.15.16:39766) 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 -- 172.21.15.16:0/141519938 learned_addr learned my addr 172.21.15.16:0/141519938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 -- 172.21.15.16:0/141519938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 msgr2=0x561ed4aaaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4aaaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 -- 172.21.15.16:0/141519938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 msgr2=0x561ed4ab7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 -- 172.21.15.16:0/141519938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed499c3a0 con 0x561ed4aabb40 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.190+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc718009fd0 tx=0x7fc7180108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8faa5579b2b6de server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc72497c640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 0x7fc70c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc715ffb640 1 -- 172.21.15.16:0/141519938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc718011760 con 0x561ed4aabb40 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc71effd640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 0x7fc70c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc715ffb640 1 -- 172.21.15.16:0/141519938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc718011de0 con 0x561ed4aabb40 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc715ffb640 1 -- 172.21.15.16:0/141519938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71800e3c0 con 0x561ed4aabb40 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc72497c640 1 -- 172.21.15.16:0/1902754505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc7080ce660 con 0x7fc720100e20 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc71effd640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 0x7fc70c0a5040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7080cc4e0 tx=0x7fc7080d4550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.191+0000 7fc727d57740 1 -- 172.21.15.16:0/141519938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 msgr2=0x561ed4aaa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc718009fd0 tx=0x7fc7180108e0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 -- 172.21.15.16:0/141519938 shutdown_connections 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4aaaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 --2- 172.21.15.16:0/141519938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4aaa090 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 -- 172.21.15.16:0/141519938 >> 172.21.15.16:0/141519938 conn(0x561ed4a73310 msgr2=0x561ed4aa6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 -- 172.21.15.16:0/141519938 shutdown_connections 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 -- 172.21.15.16:0/141519938 wait complete. 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 -- start start 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.192+0000 7fc727d57740 1 Processor -- start 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ab4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 0x561ed4ab2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ed4ab8070 con 0x561ed4ab5370 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ed49cb060 con 0x561ed4aabb40 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ed4ab81f0 con 0x561ed4aaa5d0 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc727d57740 10 client.0 _ll_register_callbacks cb 0x561ed471e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39774/0 (socket says 172.21.15.16:39774) 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3851421536 learned_addr learned my addr 172.21.15.16:0/3851421536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ab4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 0x561ed4ab2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3851421536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 msgr2=0x561ed4ab4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ab4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 msgr2=0x561ed4ab2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 0x561ed4ab2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3851421536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed47813c0 con 0x561ed4ab5370 2024-09-12T02:50:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 0x561ed4ab2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc718005cc0 tx=0x7fc71801bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d915db58efaaf99 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.193+0000 7fc6feffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71801f030 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc718024070 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc718011de0 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc718011a20 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 10 client.16050 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ed4acc350 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc718011de0 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc71802c4d0 con 0x561ed4ab5370 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 10 client.16050 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 10 client.16050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 20 client.16050 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4acd700 0x561ed4acfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 4 client.16050 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.194+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ed4a1d860 con 0x561ed4acd700 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.195+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4acd700 0x561ed4acfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.195+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4acd700 0x561ed4acfaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20183956ac587ab9 server_cookie=645c1d112581c91a in_seq=0 out_seq=0 2024-09-12T02:50:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.195+0000 7fc6feffd640 10 client.16050 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:50:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc6feffd640 1 -- 172.21.15.16:0/3851421536 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ed4a1d860 con 0x561ed4acd700 2024-09-12T02:50:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc6feffd640 10 client.16050 handle_command_reply: tid=0 2024-09-12T02:50:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 client.16050 shutdown 2024-09-12T02:50:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 msgr2=0x561ed4ab2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc718005cc0 tx=0x7fc71801bb30 comp rx=0 tx=0).stop 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 shutdown_connections 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4acd700 0x561ed4acfaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ab2650 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aabb40 0x561ed4ab2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc6feffd640 0 client.16050 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 --2- 172.21.15.16:0/3851421536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ab4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 >> 172.21.15.16:0/3851421536 conn(0x561ed4a73310 msgr2=0x561ed4aac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 shutdown_connections 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 1 -- 172.21.15.16:0/3851421536 wait complete. 2024-09-12T02:50:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.196+0000 7fc727d57740 20 client.16050 trim_cache size 0 max 16384 2024-09-12T02:50:26.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 Processor -- start 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 -- start start 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201ab3d0 con 0x561ed4aabb40 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201acfd0 con 0x561ed4ab5370 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.209+0000 7fc727d57740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201abaa0 con 0x561ed4aaa5d0 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39776/0 (socket says 172.21.15.16:39776) 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 -- 172.21.15.16:0/2757562134 learned_addr learned my addr 172.21.15.16:0/2757562134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:26.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 -- 172.21.15.16:0/2757562134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 msgr2=0x561ed4ac9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 -- 172.21.15.16:0/2757562134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 msgr2=0x561ed4ab1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 -- 172.21.15.16:0/2757562134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed4aa6510 con 0x561ed4aabb40 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc710033000 tx=0x7fc7100058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d553e04f989504 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc7157fa640 1 -- 172.21.15.16:0/2757562134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710035040 con 0x561ed4aabb40 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc7157fa640 1 -- 172.21.15.16:0/2757562134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc710049070 con 0x561ed4aabb40 2024-09-12T02:50:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.210+0000 7fc7157fa640 1 -- 172.21.15.16:0/2757562134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710003850 con 0x561ed4aabb40 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- 172.21.15.16:0/2757562134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 msgr2=0x561ed4acfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc710033000 tx=0x7fc7100058b0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- 172.21.15.16:0/2757562134 shutdown_connections 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4ab5370 0x561ed4ab1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4aabb40 0x561ed4acfb70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 --2- 172.21.15.16:0/2757562134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- 172.21.15.16:0/2757562134 >> 172.21.15.16:0/2757562134 conn(0x561ed4a73310 msgr2=0x561ed4ab0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- 172.21.15.16:0/2757562134 shutdown_connections 2024-09-12T02:50:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- 172.21.15.16:0/2757562134 wait complete. 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.211+0000 7fc727d57740 1 -- start start 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 Processor -- start 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aabb40 0x561ed4ac63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201ac840 con 0x561ed4ab5370 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201ac170 con 0x561ed4aaa5d0 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201a93b0 con 0x561ed4aabb40 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc727d57740 10 client.0 _ll_register_callbacks cb 0x7fc7180093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39788/0 (socket says 172.21.15.16:39788) 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3102375427 learned_addr learned my addr 172.21.15.16:0/3102375427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aabb40 0x561ed4ac63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3102375427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aabb40 msgr2=0x561ed4ac63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aabb40 0x561ed4ac63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aaa5d0 msgr2=0x561ed4ac5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.212+0000 7fc71f7fe640 1 -- 172.21.15.16:0/3102375427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed4ac1aa0 con 0x561ed4ab5370 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc71f7fe640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc71801e040 tx=0x7fc71801bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2799b60d1ad57e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71801f040 con 0x561ed4ab5370 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc71802cae0 con 0x561ed4ab5370 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc710049070 con 0x561ed4ab5370 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc718010e60 con 0x561ed4ab5370 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 10 client.16056 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ed4b37cc0 con 0x561ed4ab5370 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc710049070 con 0x561ed4ab5370 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc718005030 con 0x561ed4ab5370 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 10 client.16056 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 10 client.16056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 20 client.16056 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4b38320 0x561ed4b3a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 4 client.16056 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.213+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ed4a17d70 con 0x561ed4b38320 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4b38320 0x561ed4b3a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc71e7fc640 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4b38320 0x561ed4b3a710 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de76871575668165 server_cookie=da63588777085e2b in_seq=0 out_seq=0 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc6feffd640 10 client.16056 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc6feffd640 1 -- 172.21.15.16:0/3102375427 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ed4a17d70 con 0x561ed4b38320 2024-09-12T02:50:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc6feffd640 10 client.16056 handle_command_reply: tid=0 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc727d57740 1 client.16056 shutdown 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 msgr2=0x561ed4ac1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.214+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc71801e040 tx=0x7fc71801bb30 comp rx=0 tx=0).stop 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 shutdown_connections 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561ed4b38320 0x561ed4b3a710 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ed4ab5370 0x561ed4ac1270 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ed4aabb40 0x561ed4ac63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 --2- 172.21.15.16:0/3102375427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed4aaa5d0 0x561ed4ac5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 >> 172.21.15.16:0/3102375427 conn(0x561ed4a73310 msgr2=0x561ed4ab1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc6feffd640 0 client.16056 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 shutdown_connections 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 1 -- 172.21.15.16:0/3102375427 wait complete. 2024-09-12T02:50:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc727d57740 20 client.16056 trim_cache size 0 max 16384 2024-09-12T02:50:26.221 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:50:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 msgr2=0x7fc70c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 0x7fc70c0a5040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7080cc4e0 tx=0x7fc7080d4550 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 msgr2=0x7fc72019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc70800d950 tx=0x7fc70800de10 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 shutdown_connections 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc720102ef0 0x7fc72019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc720102570 0x7fc72019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc70c0a2b80 0x7fc70c0a5040 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 --2- 172.21.15.16:0/1902754505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc720100e20 0x7fc72019a6c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.215+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 >> 172.21.15.16:0/1902754505 conn(0x7fc7200fc7d0 msgr2=0x7fc720105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.216+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 shutdown_connections 2024-09-12T02:50:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:26.216+0000 7fc72597e640 1 -- 172.21.15.16:0/1902754505 wait complete. 2024-09-12T02:50:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:25 smithi033 ceph-mon[35092]: pgmap v412: 97 pgs: 97 active+clean; 90 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-09-12T02:50:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:25 smithi184 ceph-mon[35321]: pgmap v412: 97 pgs: 97 active+clean; 90 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-09-12T02:50:26.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.53 MB/sec warmup 88 sec latency 4426.390 ms 2024-09-12T02:50:26.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:50:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 -- 172.21.15.16:0/2755490789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 msgr2=0x7f52481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 --2- 172.21.15.16:0/2755490789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f52481067e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5230009900 tx=0x7f523002a410 comp rx=0 tx=0).stop 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 -- 172.21.15.16:0/2755490789 shutdown_connections 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 --2- 172.21.15.16:0/2755490789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 0x7f524810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 --2- 172.21.15.16:0/2755490789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f52481067e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 --2- 172.21.15.16:0/2755490789 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 0x7f52481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 -- 172.21.15.16:0/2755490789 >> 172.21.15.16:0/2755490789 conn(0x7f52480fd8c0 msgr2=0x7f52480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 -- 172.21.15.16:0/2755490789 shutdown_connections 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 -- 172.21.15.16:0/2755490789 wait complete. 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.648+0000 7f524c8f7640 1 Processor -- start 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 -- start start 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 0x7f5248198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 0x7f524819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5248111f30 con 0x7f5248102790 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5248111db0 con 0x7f5248101de0 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524c8f7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52481120b0 con 0x7f5248106f10 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39802/0 (socket says 172.21.15.16:39802) 2024-09-12T02:50:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 -- 172.21.15.16:0/297542588 learned_addr learned my addr 172.21.15.16:0/297542588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5246d9d640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 0x7f524819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f524659c640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 0x7f5248198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 -- 172.21.15.16:0/297542588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 msgr2=0x7f524819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 0x7f524819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 -- 172.21.15.16:0/297542588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 msgr2=0x7f5248198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 0x7f5248198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 -- 172.21.15.16:0/297542588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524819d400 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.649+0000 7f5245d9b640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f52300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.650+0000 7f523f7fe640 1 -- 172.21.15.16:0/297542588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523003f030 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.650+0000 7f523f7fe640 1 -- 172.21.15.16:0/297542588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5230003ce0 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.650+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524819d690 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.650+0000 7f523f7fe640 1 -- 172.21.15.16:0/297542588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230035070 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.650+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481a4fd0 con 0x7f5248102790 2024-09-12T02:50:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.652+0000 7f523f7fe640 1 -- 172.21.15.16:0/297542588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5230042030 con 0x7f5248102790 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.652+0000 7f524ecd0740 1 Processor -- start 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.652+0000 7f524ecd0740 1 -- start start 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 0x562469d6aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562469cd02f0 con 0x562469d6bb40 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562469c77cf0 con 0x562469d75370 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524ecd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562469ccc2b0 con 0x562469d6a5d0 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5246d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5246d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39808/0 (socket says 172.21.15.16:39808) 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5246d9d640 1 -- 172.21.15.16:0/3476325710 learned_addr learned my addr 172.21.15.16:0/3476325710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 0x562469d6aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5245d9b640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f523f7fe640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 0x7f52340a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 -- 172.21.15.16:0/3476325710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 msgr2=0x562469d77760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d77760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f523f7fe640 1 -- 172.21.15.16:0/297542588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52300ef510 con 0x7f5248102790 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 -- 172.21.15.16:0/3476325710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 msgr2=0x562469d6a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 -- 172.21.15.16:0/3476325710 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562469c5c3a0 con 0x562469d6a5d0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5246d9d640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f5246d9d640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 0x7f52340a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.653+0000 7f524659c640 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 0x562469d6aa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5240009fd0 tx=0x7f5240003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b9ed95d854d26f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.654+0000 7f5246d9d640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 0x7f52340a50c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5238004650 tx=0x7f52380046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.654+0000 7f523cff9640 1 -- 172.21.15.16:0/3476325710 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240011440 con 0x562469d6a5d0 2024-09-12T02:50:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.654+0000 7f523cff9640 1 -- 172.21.15.16:0/3476325710 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5240015030 con 0x562469d6a5d0 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.654+0000 7f523cff9640 1 -- 172.21.15.16:0/3476325710 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240010a50 con 0x562469d6a5d0 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- 172.21.15.16:0/3476325710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 msgr2=0x562469d6aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 0x562469d6aa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5240009fd0 tx=0x7f5240003e00 comp rx=0 tx=0).stop 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- 172.21.15.16:0/3476325710 shutdown_connections 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d77760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6a5d0 0x562469d6aa10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 --2- 172.21.15.16:0/3476325710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- 172.21.15.16:0/3476325710 >> 172.21.15.16:0/3476325710 conn(0x562469d33310 msgr2=0x562469d66c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- 172.21.15.16:0/3476325710 shutdown_connections 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- 172.21.15.16:0/3476325710 wait complete. 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 -- start start 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 Processor -- start 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.655+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6bb40 0x562469d71f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562469d78070 con 0x562469d6a5d0 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562469c8b060 con 0x562469d75370 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562469d781f0 con 0x562469d6bb40 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 10 client.0 _ll_register_callbacks cb 0x5624699de8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39822/0 (socket says 172.21.15.16:39822) 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 -- 172.21.15.16:0/4016203279 learned_addr learned my addr 172.21.15.16:0/4016203279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f5246d9d640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f5245d9b640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6bb40 0x562469d71f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 -- 172.21.15.16:0/4016203279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6bb40 msgr2=0x562469d71f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6bb40 0x562469d71f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 -- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 msgr2=0x562469d724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d724f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 -- 172.21.15.16:0/4016203279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562469a413c0 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524659c640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240010440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96811e3613cac388 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f522dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240029060 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5240010990 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.656+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5240015030 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5240010b90 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 10 client.16068 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562469d83a80 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5240015030 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f5240023070 con 0x562469d6a5d0 2024-09-12T02:50:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 10 client.16068 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 20 client.16068 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469d84dd0 0x562469d871c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 4 client.16068 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562469cdd860 con 0x562469d84dd0 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.657+0000 7f5245d9b640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469d84dd0 0x562469d871c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f5245d9b640 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469d84dd0 0x562469d871c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21dd81d523782c9b server_cookie=973972d79c61df74 in_seq=0 out_seq=0 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f522dffb640 10 client.16068 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f522dffb640 1 -- 172.21.15.16:0/4016203279 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562469cdd860 con 0x562469d84dd0 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f522dffb640 10 client.16068 handle_command_reply: tid=0 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f524ecd0740 1 client.16068 shutdown 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 msgr2=0x562469d74ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.658+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240010440 comp rx=0 tx=0).stop 2024-09-12T02:50:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 shutdown_connections 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469d84dd0 0x562469d871c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d75370 0x562469d724f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d6bb40 0x562469d71f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 --2- 172.21.15.16:0/4016203279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6a5d0 0x562469d74ce0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 >> 172.21.15.16:0/4016203279 conn(0x562469d33310 msgr2=0x562469d6c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f522dffb640 0 client.16068 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 shutdown_connections 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 1 -- 172.21.15.16:0/4016203279 wait complete. 2024-09-12T02:50:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.659+0000 7f524ecd0740 20 client.16068 trim_cache size 0 max 16384 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 Processor -- start 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 -- start start 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52481ad5b0 con 0x562469d6bb40 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52481af1b0 con 0x562469d6a5d0 2024-09-12T02:50:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524ecd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52481adc80 con 0x562469d75370 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39836/0 (socket says 172.21.15.16:39836) 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 -- 172.21.15.16:0/730803008 learned_addr learned my addr 172.21.15.16:0/730803008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f524659c640 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 -- 172.21.15.16:0/730803008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 msgr2=0x562469d6d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 -- 172.21.15.16:0/730803008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 msgr2=0x562469d81310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d81310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.672+0000 7f5245d9b640 1 -- 172.21.15.16:0/730803008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562469d6dba0 con 0x562469d6bb40 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f5245d9b640 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5230005ab0 tx=0x7f52300b8070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c66b903665ef65b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f522ffff640 1 -- 172.21.15.16:0/730803008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300be020 con 0x562469d6bb40 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f522ffff640 1 -- 172.21.15.16:0/730803008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523003b580 con 0x562469d6bb40 2024-09-12T02:50:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f522ffff640 1 -- 172.21.15.16:0/730803008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230049040 con 0x562469d6bb40 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 -- 172.21.15.16:0/730803008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 msgr2=0x562469d6d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5230005ab0 tx=0x7f52300b8070 comp rx=0 tx=0).stop 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 -- 172.21.15.16:0/730803008 shutdown_connections 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d6d120 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 --2- 172.21.15.16:0/730803008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d81310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 -- 172.21.15.16:0/730803008 >> 172.21.15.16:0/730803008 conn(0x562469d33310 msgr2=0x562469d65d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.673+0000 7f524ecd0740 1 -- 172.21.15.16:0/730803008 shutdown_connections 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 -- 172.21.15.16:0/730803008 wait complete. 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 -- start start 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 Processor -- start 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d7d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52481aea20 con 0x562469d6bb40 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52481ae350 con 0x562469d6a5d0 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52481ab590 con 0x562469d75370 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524ecd0740 10 client.0 _ll_register_callbacks cb 0x7f52380040e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f524659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d7d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f5245d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39848/0 (socket says 172.21.15.16:39848) 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.674+0000 7f5245d9b640 1 -- 172.21.15.16:0/2430728507 learned_addr learned my addr 172.21.15.16:0/2430728507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5246d9d640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 -- 172.21.15.16:0/2430728507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 msgr2=0x562469d6e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 -- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 msgr2=0x562469d7d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d7d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 -- 172.21.15.16:0/2430728507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562469d6ef70 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f524659c640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d7d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5246d9d640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f5245d9b640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f523003b480 tx=0x7f5230004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a239abb86acb56 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f522dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300be070 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52300b8560 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562469cd7d70 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5230049070 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f524ecd0740 10 client.16074 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562469def280 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.675+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562469cd7d70 con 0x562469d6bb40 2024-09-12T02:50:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f52300f2ce0 con 0x562469d6bb40 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 10 client.16074 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 20 client.16074 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469def920 0x562469df1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 4 client.16074 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562469c54170 con 0x562469def920 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524659c640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469def920 0x562469df1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524659c640 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469def920 0x562469df1d40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68e9e1d2022b7090 server_cookie=f5ae96185b49baba in_seq=0 out_seq=0 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f522dffb640 10 client.16074 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f522dffb640 1 -- 172.21.15.16:0/2430728507 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562469c54170 con 0x562469def920 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f522dffb640 10 client.16074 handle_command_reply: tid=0 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 client.16074 shutdown 2024-09-12T02:50:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 msgr2=0x562469d7dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f523003b480 tx=0x7f5230004500 comp rx=0 tx=0).stop 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 shutdown_connections 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562469def920 0x562469df1d40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.676+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562469d75370 0x562469d6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562469d6bb40 0x562469d7dba0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f522dffb640 0 client.16074 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 1 --2- 172.21.15.16:0/2430728507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562469d6a5d0 0x562469d7d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 >> 172.21.15.16:0/2430728507 conn(0x562469d33310 msgr2=0x562469d70730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 shutdown_connections 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 1 -- 172.21.15.16:0/2430728507 wait complete. 2024-09-12T02:50:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524ecd0740 20 client.16074 trim_cache size 0 max 16384 2024-09-12T02:50:27.686 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:50:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 msgr2=0x7f52340a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 0x7f52340a50c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5238004650 tx=0x7f52380046c0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 msgr2=0x7f5248198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f52300042f0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 shutdown_connections 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5248106f10 0x7f524819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52340a2c00 0x7f52340a50c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5248102790 0x7f5248198990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 --2- 172.21.15.16:0/297542588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5248101de0 0x7f5248198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 >> 172.21.15.16:0/297542588 conn(0x7f52480fd8c0 msgr2=0x7f52480fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 shutdown_connections 2024-09-12T02:50:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:27.677+0000 7f524c8f7640 1 -- 172.21.15.16:0/297542588 wait complete. 2024-09-12T02:50:27.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.40 MB/sec warmup 89 sec latency 5426.490 ms 2024-09-12T02:50:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:27 smithi184 ceph-mon[35321]: pgmap v413: 97 pgs: 97 active+clean; 91 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 212 op/s 2024-09-12T02:50:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:27 smithi016 ceph-mon[30925]: pgmap v413: 97 pgs: 97 active+clean; 91 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 212 op/s 2024-09-12T02:50:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:27 smithi033 ceph-mon[35092]: pgmap v413: 97 pgs: 97 active+clean; 91 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 212 op/s 2024-09-12T02:50:28.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.28 MB/sec warmup 90 sec latency 6426.604 ms 2024-09-12T02:50:29.528 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:50:29.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:50:29.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.15 MB/sec warmup 91 sec latency 7426.686 ms 2024-09-12T02:50:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:30.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:30 smithi016 ceph-mon[30925]: pgmap v414: 97 pgs: 97 active+clean; 94 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-09-12T02:50:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 -- 172.21.15.16:0/3124957159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 msgr2=0x7f0138101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3124957159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f0138101200 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0128009900 tx=0x7f012802a480 comp rx=0 tx=0).stop 2024-09-12T02:50:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 -- 172.21.15.16:0/3124957159 shutdown_connections 2024-09-12T02:50:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3124957159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102ef0 0x7f013810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3124957159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102570 0x7f01381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3124957159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f0138101200 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 -- 172.21.15.16:0/3124957159 >> 172.21.15.16:0/3124957159 conn(0x7f01380fc7d0 msgr2=0x7f01380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 -- 172.21.15.16:0/3124957159 shutdown_connections 2024-09-12T02:50:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.251+0000 7f013cf2b640 1 -- 172.21.15.16:0/3124957159 wait complete. 2024-09-12T02:50:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 Processor -- start 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 -- start start 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102570 0x7f01380fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102ef0 0x7f01380fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0138110fa0 con 0x7f0138100e20 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0138110e20 con 0x7f0138102ef0 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013cf2b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0138111120 con 0x7f0138102570 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39864/0 (socket says 172.21.15.16:39864) 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f013659c640 1 -- 172.21.15.16:0/3517129067 learned_addr learned my addr 172.21.15.16:0/3517129067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f012dd9b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102570 0x7f01380fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.252+0000 7f0136d9d640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102ef0 0x7f01380fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 -- 172.21.15.16:0/3517129067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102570 msgr2=0x7f01380fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102570 0x7f01380fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 -- 172.21.15.16:0/3517129067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102ef0 msgr2=0x7f01380fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102ef0 0x7f01380fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 -- 172.21.15.16:0/3517129067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01381a69e0 con 0x7f0138100e20 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013659c640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0128006010 tx=0x7f0128003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f012ffff640 1 -- 172.21.15.16:0/3517129067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012803f070 con 0x7f0138100e20 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01381a6c70 con 0x7f0138100e20 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f012ffff640 1 -- 172.21.15.16:0/3517129067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0128031c30 con 0x7f0138100e20 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f012ffff640 1 -- 172.21.15.16:0/3517129067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128035030 con 0x7f0138100e20 2024-09-12T02:50:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.253+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01381a7150 con 0x7f0138100e20 2024-09-12T02:50:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f012ffff640 1 -- 172.21.15.16:0/3517129067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0128042070 con 0x7f0138100e20 2024-09-12T02:50:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 Processor -- start 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 -- start start 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f539690f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f53969bd40 0x55f53969e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f5395f6280 con 0x55f539691480 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f53959db10 con 0x55f53969bd40 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013f304740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f5395f2240 con 0x55f539690b60 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.256+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39866/0 (socket says 172.21.15.16:39866) 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 -- 172.21.15.16:0/1997966134 learned_addr learned my addr 172.21.15.16:0/1997966134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012ffff640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 0x7f01240a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f539690f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012dd9b640 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f53969bd40 0x55f53969e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 -- 172.21.15.16:0/1997966134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 msgr2=0x55f539690f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f0136d9d640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 0x7f01240a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f539690f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 -- 172.21.15.16:0/1997966134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f53969bd40 msgr2=0x55f53969e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f53969bd40 0x55f53969e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 -- 172.21.15.16:0/1997966134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5395aa1f0 con 0x55f539691480 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012ffff640 1 -- 172.21.15.16:0/3517129067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01280ef7e0 con 0x7f0138100e20 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f013659c640 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f012808d180 tx=0x7f012808e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a62c8334d19867 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012cd99640 1 -- 172.21.15.16:0/1997966134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128049070 con 0x55f539691480 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f0136d9d640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 0x7f01240a5040 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f013000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012cd99640 1 -- 172.21.15.16:0/1997966134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f012808d6b0 con 0x55f539691480 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.257+0000 7f012cd99640 1 -- 172.21.15.16:0/1997966134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012808e240 con 0x55f539691480 2024-09-12T02:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 -- 172.21.15.16:0/1997966134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 msgr2=0x55f53969b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f012808d180 tx=0x7f012808e1d0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 -- 172.21.15.16:0/1997966134 shutdown_connections 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f53969bd40 0x55f53969e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539691480 0x55f53969b800 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 --2- 172.21.15.16:0/1997966134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f539690f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 -- 172.21.15.16:0/1997966134 >> 172.21.15.16:0/1997966134 conn(0x55f53968d0f0 msgr2=0x55f53968d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 -- 172.21.15.16:0/1997966134 shutdown_connections 2024-09-12T02:50:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.258+0000 7f013f304740 1 -- 172.21.15.16:0/1997966134 wait complete. 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 -- start start 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 Processor -- start 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 0x55f539698c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539691480 0x55f539699160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f53969e6d0 con 0x55f53969bd40 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5395b0f10 con 0x55f539690b60 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f53969e850 con 0x55f539691480 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013f304740 10 client.0 _ll_register_callbacks cb 0x55f539579a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39868/0 (socket says 172.21.15.16:39868) 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f013659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 0x55f539698c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 -- 172.21.15.16:0/2371200737 learned_addr learned my addr 172.21.15.16:0/2371200737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f012dd9b640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539691480 0x55f539699160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 -- 172.21.15.16:0/2371200737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539691480 msgr2=0x55f539699160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539691480 0x55f539699160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 -- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 msgr2=0x55f539698c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.259+0000 7f0136d9d640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 0x55f539698c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f0136d9d640 1 -- 172.21.15.16:0/2371200737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5393676a0 con 0x55f53969bd40 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f013659c640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 0x55f539698c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f0136d9d640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0130018c30 tx=0x7f0130004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac978d5d713d395 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130014040 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01300225f0 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f5396037f0 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f01300227f0 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f013f304740 10 client.16086 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f5396aa2f0 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.260+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f5396037f0 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f013002b870 con 0x55f53969bd40 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 10 client.16086 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 20 client.16086 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f5396ab8c0 0x55f5396add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 4 client.16086 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5395fdd00 con 0x55f5396ab8c0 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013659c640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f5396ab8c0 0x55f5396add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f011dffb640 10 client.16086 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.261+0000 7f013659c640 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f5396ab8c0 0x55f5396add40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af5750415d51787b server_cookie=56eee360df69bccf in_seq=0 out_seq=0 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f011dffb640 1 -- 172.21.15.16:0/2371200737 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f5395fdd00 con 0x55f5396ab8c0 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f011dffb640 10 client.16086 handle_command_reply: tid=0 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 client.16086 shutdown 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 msgr2=0x55f5396996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0130018c30 tx=0x7f0130004b50 comp rx=0 tx=0).stop 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 shutdown_connections 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f5396ab8c0 0x55f5396add40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396996a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539691480 0x55f539699160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 --2- 172.21.15.16:0/2371200737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539690b60 0x55f539698c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f011dffb640 0 client.16086 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 >> 172.21.15.16:0/2371200737 conn(0x55f53968d0f0 msgr2=0x55f539692e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 shutdown_connections 2024-09-12T02:50:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.262+0000 7f013f304740 1 -- 172.21.15.16:0/2371200737 wait complete. 2024-09-12T02:50:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.263+0000 7f013f304740 20 client.16086 trim_cache size 0 max 16384 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 Processor -- start 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 -- start start 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f53969aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f5396b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01381af710 con 0x55f53969bd40 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01381b1310 con 0x55f539691480 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013f304740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01381afde0 con 0x55f539690b60 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39874/0 (socket says 172.21.15.16:39874) 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 -- 172.21.15.16:0/1295296167 learned_addr learned my addr 172.21.15.16:0/1295296167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f013659c640 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f53969aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 -- 172.21.15.16:0/1295296167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 msgr2=0x55f53969aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f53969aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 -- 172.21.15.16:0/1295296167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 msgr2=0x55f5396b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f5396b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 -- 172.21.15.16:0/1295296167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f53968cf20 con 0x55f53969bd40 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.280+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f013000af80 tx=0x7f0130009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473f47cb7a093301 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f011ffff640 1 -- 172.21.15.16:0/1295296167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130018810 con 0x55f53969bd40 2024-09-12T02:50:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f011ffff640 1 -- 172.21.15.16:0/1295296167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0130027070 con 0x55f53969bd40 2024-09-12T02:50:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f011ffff640 1 -- 172.21.15.16:0/1295296167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013002b9b0 con 0x55f53969bd40 2024-09-12T02:50:30.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 -- 172.21.15.16:0/1295296167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 msgr2=0x55f5396b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f013000af80 tx=0x7f0130009f60 comp rx=0 tx=0).stop 2024-09-12T02:50:30.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 -- 172.21.15.16:0/1295296167 shutdown_connections 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f53969bd40 0x55f5396b3b50 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f5396b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 --2- 172.21.15.16:0/1295296167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f539690b60 0x55f53969aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 -- 172.21.15.16:0/1295296167 >> 172.21.15.16:0/1295296167 conn(0x55f53968d0f0 msgr2=0x55f53969a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 -- 172.21.15.16:0/1295296167 shutdown_connections 2024-09-12T02:50:30.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.281+0000 7f013f304740 1 -- 172.21.15.16:0/1295296167 wait complete. 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 -- start start 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 Processor -- start 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f53969f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 0x55f53969fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01381b0b80 con 0x55f539690b60 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01381b04b0 con 0x55f539691480 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01381ad6f0 con 0x55f53969bd40 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013f304740 10 client.0 _ll_register_callbacks cb 0x7f0130009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39882/0 (socket says 172.21.15.16:39882) 2024-09-12T02:50:30.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 -- 172.21.15.16:0/1986121909 learned_addr learned my addr 172.21.15.16:0/1986121909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f012dd9b640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f53969f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 0x55f53969fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 -- 172.21.15.16:0/1986121909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 msgr2=0x55f53969fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 0x55f53969fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 -- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 msgr2=0x55f53969f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f53969f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f013659c640 1 -- 172.21.15.16:0/1986121909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5396a0390 con 0x55f539690b60 2024-09-12T02:50:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.282+0000 7f0136d9d640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 0x55f53969fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013659c640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f01280ed660 tx=0x7f012803b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89762a0533d6e6e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128049040 con 0x55f539690b60 2024-09-12T02:50:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01280ba1a0 con 0x55f539690b60 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0130027070 con 0x55f539690b60 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f012808d230 con 0x55f539690b60 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 10 client.16092 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f539716730 con 0x55f539690b60 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0130027070 con 0x55f539690b60 2024-09-12T02:50:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f01280c0070 con 0x55f539690b60 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 10 client.16092 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 20 client.16092 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f539716d50 0x55f539719170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 4 client.16092 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:50:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.283+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f53957a110 con 0x55f539716d50 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.284+0000 7f012dd9b640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f539716d50 0x55f539719170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.284+0000 7f012dd9b640 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f539716d50 0x55f539719170 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=854c539716d8c2ec server_cookie=1eba2c6226879d25 in_seq=0 out_seq=0 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.284+0000 7f011dffb640 10 client.16092 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f011dffb640 1 -- 172.21.15.16:0/1986121909 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f53957a110 con 0x55f539716d50 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f011dffb640 10 client.16092 handle_command_reply: tid=0 2024-09-12T02:50:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 client.16092 shutdown 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 msgr2=0x55f5396a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f01280ed660 tx=0x7f012803b870 comp rx=0 tx=0).stop 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 shutdown_connections 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f539716d50 0x55f539719170 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f53969bd40 0x55f53969fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f539691480 0x55f53969f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 --2- 172.21.15.16:0/1986121909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f539690b60 0x55f5396a4240 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.285+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 >> 172.21.15.16:0/1986121909 conn(0x55f53968d0f0 msgr2=0x55f539699400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f011dffb640 0 client.16092 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 shutdown_connections 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013f304740 1 -- 172.21.15.16:0/1986121909 wait complete. 2024-09-12T02:50:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013f304740 20 client.16092 trim_cache size 0 max 16384 2024-09-12T02:50:30.297 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:50:30.297 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:50:30.298 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "e3072b4f-2154-481c-92f3-a28d9b89975c", 2024-09-12T02:50:30.298 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:50:30.298 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:50:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 msgr2=0x7f01240a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 0x7f01240a5040 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f013000d040 comp rx=0 tx=0).stop 2024-09-12T02:50:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 msgr2=0x7f01380fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0128006010 tx=0x7f0128003c10 comp rx=0 tx=0).stop 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 shutdown_connections 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0138102ef0 0x7f01380fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0138102570 0x7f01380fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01240a2b80 0x7f01240a5040 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 --2- 172.21.15.16:0/3517129067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138100e20 0x7f01380fcc80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.286+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 >> 172.21.15.16:0/3517129067 conn(0x7f01380fc7d0 msgr2=0x7f013806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.287+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 shutdown_connections 2024-09-12T02:50:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:30.287+0000 7f013cf2b640 1 -- 172.21.15.16:0/3517129067 wait complete. 2024-09-12T02:50:30.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:30 smithi184 ceph-mon[35321]: pgmap v414: 97 pgs: 97 active+clean; 94 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-09-12T02:50:30.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:30 smithi033 ceph-mon[35092]: pgmap v414: 97 pgs: 97 active+clean; 94 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 175 op/s 2024-09-12T02:50:30.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 11.03 MB/sec warmup 92 sec latency 8426.736 ms 2024-09-12T02:50:31.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:31.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:50:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:50:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:50:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:50:31.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.91 MB/sec warmup 93 sec latency 9426.768 ms 2024-09-12T02:50:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 -- 172.21.15.16:0/2564679233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 msgr2=0x7f8bb0103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 --2- 172.21.15.16:0/2564679233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb0103120 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009900 tx=0x7f8bac02a480 comp rx=0 tx=0).stop 2024-09-12T02:50:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 -- 172.21.15.16:0/2564679233 shutdown_connections 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 --2- 172.21.15.16:0/2564679233 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb010c170 0x7f8bb010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 --2- 172.21.15.16:0/2564679233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb0103660 0x7f8bb010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 --2- 172.21.15.16:0/2564679233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb0103120 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 -- 172.21.15.16:0/2564679233 >> 172.21.15.16:0/2564679233 conn(0x7f8bb00fc7a0 msgr2=0x7f8bb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 -- 172.21.15.16:0/2564679233 shutdown_connections 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.758+0000 7f8bb8508640 1 -- 172.21.15.16:0/2564679233 wait complete. 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 Processor -- start 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 -- start start 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 0x7f8bb0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb010c170 0x7f8bb0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0111f70 con 0x7f8bb0102d40 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0111df0 con 0x7f8bb0103660 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb8508640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01120f0 con 0x7f8bb010c170 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb62a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb62a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39902/0 (socket says 172.21.15.16:39902) 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.759+0000 7f8bb62a4640 1 -- 172.21.15.16:0/1226477334 learned_addr learned my addr 172.21.15.16:0/1226477334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb010c170 0x7f8bb0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 0x7f8bb0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 -- 172.21.15.16:0/1226477334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb010c170 msgr2=0x7f8bb0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb010c170 0x7f8bb0101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 -- 172.21.15.16:0/1226477334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 msgr2=0x7f8bb0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 0x7f8bb0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 -- 172.21.15.16:0/1226477334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb01a69b0 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 0x7f8bb0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8ba77fe640 1 -- 172.21.15.16:0/1226477334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac03f030 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb01a6c40 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8ba77fe640 1 -- 172.21.15.16:0/1226477334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bac003ce0 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8ba77fe640 1 -- 172.21.15.16:0/1226477334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac035070 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.760+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb01a7170 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.763+0000 7f8ba77fe640 1 -- 172.21.15.16:0/1226477334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8bac042030 con 0x7f8bb0102d40 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.763+0000 7f8bba8e1740 1 Processor -- start 2024-09-12T02:50:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.763+0000 7f8bba8e1740 1 -- start start 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e046394a0 0x564e04639880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 0x564e04644140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e0459f2e0 con 0x564e04644680 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e04546ce0 con 0x564e04639dc0 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e0459b2a0 con 0x564e046394a0 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8ba77fe640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 0x7f8b9c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39904/0 (socket says 172.21.15.16:39904) 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/3370591602 learned_addr learned my addr 172.21.15.16:0/3370591602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e046394a0 0x564e04639880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 0x7f8b9c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8ba77fe640 1 -- 172.21.15.16:0/1226477334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8bac0ef610 con 0x7f8bb0102d40 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 0x564e04644140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/3370591602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e046394a0 msgr2=0x564e04639880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e046394a0 0x564e04639880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/3370591602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 msgr2=0x564e04644140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 0x564e04644140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/3370591602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0452b440 con 0x564e04644680 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 0x564e04644140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.764+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009ce0 tx=0x7f8ba00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6664070fc888aa40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 0x7f8b9c0a5040 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba80096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8ba4ff9640 1 -- 172.21.15.16:0/3370591602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0012450 con 0x564e04644680 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8ba4ff9640 1 -- 172.21.15.16:0/3370591602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0012ad0 con 0x564e04644680 2024-09-12T02:50:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8ba4ff9640 1 -- 172.21.15.16:0/3370591602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000fbd0 con 0x564e04644680 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 -- 172.21.15.16:0/3370591602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 msgr2=0x564e04646a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009ce0 tx=0x7f8ba00117b0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 -- 172.21.15.16:0/3370591602 shutdown_connections 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04644680 0x564e04646a70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04639dc0 0x564e04644140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/3370591602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e046394a0 0x564e04639880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 -- 172.21.15.16:0/3370591602 >> 172.21.15.16:0/3370591602 conn(0x564e04635a30 msgr2=0x564e04635e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.765+0000 7f8bba8e1740 1 -- 172.21.15.16:0/3370591602 shutdown_connections 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 -- 172.21.15.16:0/3370591602 wait complete. 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 -- start start 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 Processor -- start 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e04643ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e046416e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e04647250 con 0x564e04639dc0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e0455a050 con 0x564e046394a0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e046473d0 con 0x564e04644680 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bba8e1740 10 client.0 _ll_register_callbacks cb 0x564e0459a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39912/0 (socket says 172.21.15.16:39912) 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/226397742 learned_addr learned my addr 172.21.15.16:0/226397742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.766+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e046416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e04643ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/226397742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 msgr2=0x564e046416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e046416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 msgr2=0x564e04643ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e04643ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/226397742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e043103c0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e04643ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000f7b0 tx=0x7f8ba0009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6502374bd4f14e2d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0029020 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8ba00126e0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ba0012ad0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ba00128e0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bba8e1740 10 client.16104 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.767+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564e04652ab0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0012ad0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8ba00046c0 con 0x564e04639dc0 2024-09-12T02:50:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 10 client.16104 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 20 client.16104 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e04656250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 4 client.16104 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e045ac850 con 0x564e04653e60 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.768+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e04656250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8b99ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e04656250 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cc58c19e7cd3346 server_cookie=d39edcd876f209a4 in_seq=0 out_seq=0 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8b99ffb640 1 -- 172.21.15.16:0/226397742 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e045ac850 con 0x564e04653e60 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8b99ffb640 10 client.16104 handle_command_reply: tid=0 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8bba8e1740 1 client.16104 shutdown 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 msgr2=0x564e046411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000f7b0 tx=0x7f8ba0009dd0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.769+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 shutdown_connections 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e04656250 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8b99ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e046416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e046411a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/226397742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e04643ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 >> 172.21.15.16:0/226397742 conn(0x564e04635a30 msgr2=0x564e0463b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 shutdown_connections 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 1 -- 172.21.15.16:0/226397742 wait complete. 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.770+0000 7f8bba8e1740 20 client.16104 trim_cache size 0 max 16384 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 Processor -- start 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 -- start start 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04639dc0 0x564e0463c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 0x564e0465b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01af6e0 con 0x564e046394a0 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b12e0 con 0x564e04644680 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01afdb0 con 0x564e04639dc0 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bb62a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bb62a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39928/0 (socket says 172.21.15.16:39928) 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.787+0000 7f8bb62a4640 1 -- 172.21.15.16:0/450248933 learned_addr learned my addr 172.21.15.16:0/450248933 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 -- 172.21.15.16:0/450248933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04639dc0 msgr2=0x564e0463c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04639dc0 0x564e0463c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 -- 172.21.15.16:0/450248933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 msgr2=0x564e0465b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 0x564e0465b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 0x564e0465b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 -- 172.21.15.16:0/450248933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e04635860 con 0x564e046394a0 2024-09-12T02:50:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 0x564e0465b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0bbc40 tx=0x7f8bac031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209d7c590428217b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8b9bfff640 1 -- 172.21.15.16:0/450248933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac049040 con 0x564e046394a0 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8b9bfff640 1 -- 172.21.15.16:0/450248933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bac0bc640 con 0x564e046394a0 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.788+0000 7f8b9bfff640 1 -- 172.21.15.16:0/450248933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0ba240 con 0x564e046394a0 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- 172.21.15.16:0/450248933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 msgr2=0x564e04650b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0bbc40 tx=0x7f8bac031a60 comp rx=0 tx=0).stop 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- 172.21.15.16:0/450248933 shutdown_connections 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e04644680 0x564e0465b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04639dc0 0x564e0463c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/450248933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e046394a0 0x564e04650b30 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- 172.21.15.16:0/450248933 >> 172.21.15.16:0/450248933 conn(0x564e04635a30 msgr2=0x564e0463c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- 172.21.15.16:0/450248933 shutdown_connections 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- 172.21.15.16:0/450248933 wait complete. 2024-09-12T02:50:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- start start 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 Processor -- start 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e0463e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e0463ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b0b50 con 0x564e04639dc0 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b0480 con 0x564e046394a0 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01ad6c0 con 0x564e04644680 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bba8e1740 10 client.0 _ll_register_callbacks cb 0x7f8ba8009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.789+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39942/0 (socket says 172.21.15.16:39942) 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/2811932964 learned_addr learned my addr 172.21.15.16:0/2811932964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e0463e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/2811932964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 msgr2=0x564e0463ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e0463ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e0463ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 msgr2=0x564e0463e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e0463e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 -- 172.21.15.16:0/2811932964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e046bdaa0 con 0x564e04639dc0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e0463e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb6aa5640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e0463ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bb5aa3640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0004b10 tx=0x7f8ba0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c585c3e88ebf53 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba003d060 con 0x564e04639dc0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8ba0012a80 con 0x564e04639dc0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bac0bc640 con 0x564e04639dc0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ba0029040 con 0x564e04639dc0 2024-09-12T02:50:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bba8e1740 10 client.16110 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.790+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564e046bed40 con 0x564e04639dc0 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bac0bc640 con 0x564e04639dc0 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8ba0037030 con 0x564e04639dc0 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 10 client.16110 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 20 client.16110 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e046c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 4 client.16110 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e045a6d60 con 0x564e04653e60 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e046c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8bb62a4640 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e046c15f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa77d0ca967cb615 server_cookie=304b5a4b84fd7129 in_seq=0 out_seq=0 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.791+0000 7f8b99ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2811932964 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564e045a6d60 con 0x564e04653e60 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8b99ffb640 10 client.16110 handle_command_reply: tid=0 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 client.16110 shutdown 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 msgr2=0x564e0463e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0004b10 tx=0x7f8ba0009b30 comp rx=0 tx=0).stop 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 shutdown_connections 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e04653e60 0x564e046c15f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e04644680 0x564e0463ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e04639dc0 0x564e0463e850 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 --2- 172.21.15.16:0/2811932964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e046394a0 0x564e0463e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 >> 172.21.15.16:0/2811932964 conn(0x564e04635a30 msgr2=0x564e0463b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.920 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "e3072b4f-2154-481c-92f3-a28d9b89975c": { 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "e3072b4f-2154-481c-92f3-a28d9b89975c", 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:50:31.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8b99ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 shutdown_connections 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 1 -- 172.21.15.16:0/2811932964 wait complete. 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bba8e1740 20 client.16110 trim_cache size 0 max 16384 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 msgr2=0x7f8b9c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.792+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 0x7f8b9c0a5040 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba80096a0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 msgr2=0x7f8bb01009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac0042f0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 shutdown_connections 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bb010c170 0x7f8bb0101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bb0103660 0x7f8bb0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b9c0a2b80 0x7f8b9c0a5040 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 --2- 172.21.15.16:0/1226477334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bb0102d40 0x7f8bb01009a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 >> 172.21.15.16:0/1226477334 conn(0x7f8bb00fc7a0 msgr2=0x7f8bb00fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:31.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 shutdown_connections 2024-09-12T02:50:31.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:31.793+0000 7f8bb8508640 1 -- 172.21.15.16:0/1226477334 wait complete. 2024-09-12T02:50:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:32 smithi016 ceph-mon[30925]: pgmap v415: 97 pgs: 97 active+clean; 99 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 183 op/s 2024-09-12T02:50:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:32 smithi184 ceph-mon[35321]: pgmap v415: 97 pgs: 97 active+clean; 99 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 183 op/s 2024-09-12T02:50:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:32 smithi033 ceph-mon[35092]: pgmap v415: 97 pgs: 97 active+clean; 99 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 183 op/s 2024-09-12T02:50:32.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.80 MB/sec warmup 94 sec latency 10427.090 ms 2024-09-12T02:50:32.724 INFO:tasks.cephfs.filesystem:scrub status for tag:e3072b4f-2154-481c-92f3-a28d9b89975c - {'path': '/', 'tag': 'e3072b4f-2154-481c-92f3-a28d9b89975c', 'options': 'recursive,force'} 2024-09-12T02:50:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:33 smithi016 ceph-mon[30925]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 108 op/s 2024-09-12T02:50:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:33 smithi033 ceph-mon[35092]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 108 op/s 2024-09-12T02:50:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:33 smithi184 ceph-mon[35321]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 108 op/s 2024-09-12T02:50:33.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.68 MB/sec warmup 95 sec latency 11427.210 ms 2024-09-12T02:50:34.710 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.57 MB/sec warmup 96 sec latency 12427.311 ms 2024-09-12T02:50:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:35 smithi016 ceph-mon[30925]: pgmap v417: 97 pgs: 97 active+clean; 97 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-09-12T02:50:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:35 smithi033 ceph-mon[35092]: pgmap v417: 97 pgs: 97 active+clean; 97 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-09-12T02:50:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:35 smithi184 ceph-mon[35321]: pgmap v417: 97 pgs: 97 active+clean; 97 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-09-12T02:50:35.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.46 MB/sec warmup 97 sec latency 13427.430 ms 2024-09-12T02:50:36.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.36 MB/sec warmup 98 sec latency 14427.547 ms 2024-09-12T02:50:37.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:37 smithi016 ceph-mon[30925]: pgmap v418: 97 pgs: 97 active+clean; 97 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T02:50:37.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:37 smithi033 ceph-mon[35092]: pgmap v418: 97 pgs: 97 active+clean; 97 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T02:50:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:37 smithi184 ceph-mon[35321]: pgmap v418: 97 pgs: 97 active+clean; 97 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T02:50:37.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.25 MB/sec warmup 99 sec latency 15427.668 ms 2024-09-12T02:50:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:50:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:50:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:50:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:50:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:50:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:50:38.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.15 MB/sec warmup 100 sec latency 16427.791 ms 2024-09-12T02:50:39.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:39 smithi016 ceph-mon[30925]: pgmap v419: 97 pgs: 97 active+clean; 94 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-09-12T02:50:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:39 smithi033 ceph-mon[35092]: pgmap v419: 97 pgs: 97 active+clean; 94 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-09-12T02:50:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:39 smithi184 ceph-mon[35321]: pgmap v419: 97 pgs: 97 active+clean; 94 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-09-12T02:50:39.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 10.05 MB/sec warmup 101 sec latency 17427.896 ms 2024-09-12T02:50:40.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 9.95 MB/sec warmup 102 sec latency 18428.014 ms 2024-09-12T02:50:41.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:41 smithi016 ceph-mon[30925]: pgmap v420: 97 pgs: 97 active+clean; 94 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-12T02:50:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:41 smithi033 ceph-mon[35092]: pgmap v420: 97 pgs: 97 active+clean; 94 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-12T02:50:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:41 smithi184 ceph-mon[35321]: pgmap v420: 97 pgs: 97 active+clean; 94 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.4 MiB/s wr, 95 op/s 2024-09-12T02:50:41.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 9.85 MB/sec warmup 103 sec latency 19428.065 ms 2024-09-12T02:50:42.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:50:42.689 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:42.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 9.76 MB/sec warmup 104 sec latency 20428.107 ms 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 -- 172.21.15.16:0/3270310954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 msgr2=0x7f66d8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 --2- 172.21.15.16:0/3270310954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8101200 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009900 tx=0x7f66c802a480 comp rx=0 tx=0).stop 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 -- 172.21.15.16:0/3270310954 shutdown_connections 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 --2- 172.21.15.16:0/3270310954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102ef0 0x7f66d810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 --2- 172.21.15.16:0/3270310954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102570 0x7f66d81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 --2- 172.21.15.16:0/3270310954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8101200 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 -- 172.21.15.16:0/3270310954 >> 172.21.15.16:0/3270310954 conn(0x7f66d80fc7d0 msgr2=0x7f66d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:43.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 -- 172.21.15.16:0/3270310954 shutdown_connections 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.117+0000 7f66de563640 1 -- 172.21.15.16:0/3270310954 wait complete. 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 Processor -- start 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 -- start start 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102570 0x7f66d8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102ef0 0x7f66d819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66d8110f80 con 0x7f66d8100e20 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66d8110e00 con 0x7f66d8102570 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66de563640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66d8111100 con 0x7f66d8102ef0 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53036/0 (socket says 172.21.15.16:53036) 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 -- 172.21.15.16:0/3738847755 learned_addr learned my addr 172.21.15.16:0/3738847755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66cffff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102570 0x7f66d8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66dcb00640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102ef0 0x7f66d819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 -- 172.21.15.16:0/3738847755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102ef0 msgr2=0x7f66d819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102ef0 0x7f66d819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 -- 172.21.15.16:0/3738847755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102570 msgr2=0x7f66d8198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102570 0x7f66d8198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 -- 172.21.15.16:0/3738847755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d819d4d0 con 0x7f66d8100e20 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.118+0000 7f66d7fff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66c8006010 tx=0x7f66c8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.119+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c803f070 con 0x7f66d8100e20 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.119+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d819d760 con 0x7f66d8100e20 2024-09-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.119+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66c8031c30 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.119+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8035030 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.119+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d81a50a0 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.120+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d8101200 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.121+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f66c8042070 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.122+0000 7f66d5ffb640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 0x7f66c40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.122+0000 7f66cffff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 0x7f66c40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.122+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f66c80ef8a0 con 0x7f66d8100e20 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.122+0000 7f66cffff640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 0x7f66c40a5040 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f66d80fff40 tx=0x7f66c000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.126+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f66c8031de0 con 0x7f66d8100e20 2024-09-12T02:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.340+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66d81a5270 con 0x7f66d8100e20 2024-09-12T02:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.341+0000 7f66d5ffb640 1 -- 172.21.15.16:0/3738847755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11976 (secure 0 0 0) 0x7f66c80bc810 con 0x7f66d8100e20 2024-09-12T02:50:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:50:43.343 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"btime":"2024-09-12T02:49:07:659771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:49:07.565981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:50:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 msgr2=0x7f66c40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 0x7f66c40a5040 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f66d80fff40 tx=0x7f66c000b390 comp rx=0 tx=0).stop 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 msgr2=0x7f66d8198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66c8006010 tx=0x7f66c8003c10 comp rx=0 tx=0).stop 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 shutdown_connections 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d8102ef0 0x7f66d819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66d8102570 0x7f66d8198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66c40a2b80 0x7f66c40a5040 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 --2- 172.21.15.16:0/3738847755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8100e20 0x7f66d8198520 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 >> 172.21.15.16:0/3738847755 conn(0x7f66d80fc7d0 msgr2=0x7f66d806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 shutdown_connections 2024-09-12T02:50:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:43.343+0000 7f66de563640 1 -- 172.21.15.16:0/3738847755 wait complete. 2024-09-12T02:50:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:43 smithi016 ceph-mon[30925]: pgmap v421: 97 pgs: 97 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:50:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:43 smithi033 ceph-mon[35092]: pgmap v421: 97 pgs: 97 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:50:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:43 smithi184 ceph-mon[35321]: pgmap v421: 97 pgs: 97 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:50:43.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 9.67 MB/sec warmup 105 sec latency 21428.181 ms 2024-09-12T02:50:44.099 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id e792f047 --release --await' 2024-09-12T02:50:44.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id e792f047 --release --await 2024-09-12T02:50:44.362 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:50:44.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3738847755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:44.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3738847755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:44.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3738847755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:50:44.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184707 9.58 MB/sec warmup 106 sec latency 22428.226 ms 2024-09-12T02:50:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 -- 172.21.15.16:0/3927270563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944102790 msgr2=0x7ff9441067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 --2- 172.21.15.16:0/3927270563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944102790 0x7ff9441067e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff938009900 tx=0x7ff93802a410 comp rx=0 tx=0).stop 2024-09-12T02:50:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 -- 172.21.15.16:0/3927270563 shutdown_connections 2024-09-12T02:50:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 --2- 172.21.15.16:0/3927270563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 --2- 172.21.15.16:0/3927270563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944102790 0x7ff9441067e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 --2- 172.21.15.16:0/3927270563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944101de0 0x7ff9441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 -- 172.21.15.16:0/3927270563 >> 172.21.15.16:0/3927270563 conn(0x7ff9440fd8c0 msgr2=0x7ff9440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 -- 172.21.15.16:0/3927270563 shutdown_connections 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.890+0000 7ff948ff2640 1 -- 172.21.15.16:0/3927270563 wait complete. 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 Processor -- start 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 -- start start 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 0x7ff944198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff944111f90 con 0x7ff944101de0 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff944111e10 con 0x7ff944102790 2024-09-12T02:50:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff948ff2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff944112110 con 0x7ff944106f10 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff94259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff94259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53056/0 (socket says 172.21.15.16:53056) 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff94259c640 1 -- 172.21.15.16:0/1222031166 learned_addr learned my addr 172.21.15.16:0/1222031166 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.891+0000 7ff941d9b640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 0x7ff944198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff942d9d640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 -- 172.21.15.16:0/1222031166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 msgr2=0x7ff94419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 -- 172.21.15.16:0/1222031166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 msgr2=0x7ff944198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 0x7ff944198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 -- 172.21.15.16:0/1222031166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94419d3a0 con 0x7ff944101de0 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff941d9b640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 0x7ff944198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff942d9d640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff94259c640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c920 tx=0x7ff93400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff9337fe640 1 -- 172.21.15.16:0/1222031166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934007c70 con 0x7ff944101de0 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94419d690 con 0x7ff944101de0 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff9337fe640 1 -- 172.21.15.16:0/1222031166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff934007e10 con 0x7ff944101de0 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff9337fe640 1 -- 172.21.15.16:0/1222031166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934018890 con 0x7ff944101de0 2024-09-12T02:50:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.892+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9441a4f70 con 0x7ff944101de0 2024-09-12T02:50:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.895+0000 7ff9337fe640 1 -- 172.21.15.16:0/1222031166 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff934018a70 con 0x7ff944101de0 2024-09-12T02:50:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.895+0000 7ff94b3cb740 1 Processor -- start 2024-09-12T02:50:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 -- start start 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b0913440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913980 0x55d3b091dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3b0878620 con 0x55d3b091e240 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3b081fe50 con 0x55d3b0913060 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d3b08745e0 con 0x55d3b0913980 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b0913440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53064/0 (socket says 172.21.15.16:53064) 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff941d9b640 1 -- 172.21.15.16:0/2190652053 learned_addr learned my addr 172.21.15.16:0/2190652053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff94259c640 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913980 0x55d3b091dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff9337fe640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 0x7ff92c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 -- 172.21.15.16:0/2190652053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913980 msgr2=0x55d3b091dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913980 0x55d3b091dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 -- 172.21.15.16:0/2190652053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 msgr2=0x55d3b0920630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.896+0000 7ff942d9d640 1 -- 172.21.15.16:0/2190652053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3b0810d40 con 0x55d3b0913060 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff941d9b640 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff9337fe640 1 -- 172.21.15.16:0/1222031166 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff9340cf290 con 0x7ff944101de0 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff942d9d640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 0x7ff92c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff942d9d640 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b0913440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00c2f0 tx=0x7ff93c00c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc4c6fd60b7ad34 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff930ff9640 1 -- 172.21.15.16:0/2190652053 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c016070 con 0x55d3b0913060 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff930ff9640 1 -- 172.21.15.16:0/2190652053 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93c01cc90 con 0x55d3b0913060 2024-09-12T02:50:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.897+0000 7ff930ff9640 1 -- 172.21.15.16:0/2190652053 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c00fb60 con 0x55d3b0913060 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2190652053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 msgr2=0x55d3b0913440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff942d9d640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 0x7ff92c0a5040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff93c01d000 tx=0x7ff93c01e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b0913440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00c2f0 tx=0x7ff93c00c7b0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2190652053 shutdown_connections 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b0920630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913980 0x55d3b091dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2190652053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b0913440 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2190652053 >> 172.21.15.16:0/2190652053 conn(0x55d3b090f5f0 msgr2=0x55d3b090f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2190652053 shutdown_connections 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2190652053 wait complete. 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 -- start start 2024-09-12T02:50:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.898+0000 7ff94b3cb740 1 Processor -- start 2024-09-12T02:50:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b091b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b091bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3b0920e10 con 0x55d3b091e240 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3b0800f10 con 0x55d3b0913980 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d3b0920f90 con 0x55d3b0913060 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94b3cb740 10 client.0 _ll_register_callbacks cb 0x55d3b0824db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff942d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b091bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff94259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b091b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:56186/0 (socket says 172.21.15.16:56186) 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 -- 172.21.15.16:0/123394508 learned_addr learned my addr 172.21.15.16:0/123394508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 -- 172.21.15.16:0/123394508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 msgr2=0x55d3b091b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b091b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 -- 172.21.15.16:0/123394508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 msgr2=0x55d3b091bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b091bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff941d9b640 1 -- 172.21.15.16:0/123394508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3b0587580 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.899+0000 7ff942d9d640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b091bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff941d9b640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff938009230 tx=0x7ff938031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6feca312721443e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938040070 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff93c01cc90 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff938004450 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff938035040 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff94b3cb740 10 client.25712 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d3b0651960 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93c01cc90 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff938004c90 con 0x55d3b0913980 2024-09-12T02:50:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff94b3cb740 10 client.25712 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.900+0000 7ff94b3cb740 10 client.25712 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94b3cb740 20 client.25712 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b0932270 0x55d3b09346b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94b3cb740 4 client.25712 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3b0885b90 con 0x55d3b0932270 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94259c640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b0932270 0x55d3b09346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff929ffb640 10 client.25712 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.901+0000 7ff94259c640 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b0932270 0x55d3b09346b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=165053bc0e11c0e2 server_cookie=71b8418128c9af2e in_seq=0 out_seq=0 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff929ffb640 1 -- 172.21.15.16:0/123394508 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d3b0885b90 con 0x55d3b0932270 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff929ffb640 10 client.25712 handle_command_reply: tid=0 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 client.25712 shutdown 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 msgr2=0x55d3b091b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff938009230 tx=0x7ff938031920 comp rx=0 tx=0).stop 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 shutdown_connections 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b0932270 0x55d3b09346b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b091e240 0x55d3b091bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913980 0x55d3b091b690 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/123394508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b091b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 >> 172.21.15.16:0/123394508 conn(0x55d3b090f5f0 msgr2=0x55d3b09152d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff929ffb640 0 client.25712 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 shutdown_connections 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 1 -- 172.21.15.16:0/123394508 wait complete. 2024-09-12T02:50:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.902+0000 7ff94b3cb740 20 client.25712 trim_cache size 0 max 16384 2024-09-12T02:50:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 Processor -- start 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 -- start start 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b091d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b091e240 0x55d3b093a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441ab350 con 0x55d3b0913980 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441acf50 con 0x55d3b0913060 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.919+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441aba20 con 0x55d3b091e240 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53084/0 (socket says 172.21.15.16:53084) 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 -- 172.21.15.16:0/3464343633 learned_addr learned my addr 172.21.15.16:0/3464343633 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff94259c640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b091d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff942d9d640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b091e240 0x55d3b093a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 -- 172.21.15.16:0/3464343633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b091e240 msgr2=0x55d3b093a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b091e240 0x55d3b093a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 -- 172.21.15.16:0/3464343633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 msgr2=0x55d3b091d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b091d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 -- 172.21.15.16:0/3464343633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3b091a100 con 0x55d3b0913980 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff94259c640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b091d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff941d9b640 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff938009850 tx=0x7ff93802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3479924d6102ac07 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff92bfff640 1 -- 172.21.15.16:0/3464343633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938040070 con 0x55d3b0913980 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff92bfff640 1 -- 172.21.15.16:0/3464343633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff938003a00 con 0x55d3b0913980 2024-09-12T02:50:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.920+0000 7ff92bfff640 1 -- 172.21.15.16:0/3464343633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93803cb20 con 0x55d3b0913980 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- 172.21.15.16:0/3464343633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 msgr2=0x55d3b0939f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff938009850 tx=0x7ff93802a410 comp rx=0 tx=0).stop 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- 172.21.15.16:0/3464343633 shutdown_connections 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b091e240 0x55d3b093a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0939f80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/3464343633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b0913060 0x55d3b091d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- 172.21.15.16:0/3464343633 >> 172.21.15.16:0/3464343633 conn(0x55d3b090f5f0 msgr2=0x55d3b091c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- 172.21.15.16:0/3464343633 shutdown_connections 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- 172.21.15.16:0/3464343633 wait complete. 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 -- start start 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.921+0000 7ff94b3cb740 1 Processor -- start 2024-09-12T02:50:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b092af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b091e240 0x55d3b09269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441ac7c0 con 0x55d3b0913980 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441ac0f0 con 0x55d3b091e240 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441a9330 con 0x55d3b0913060 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 10 client.0 _ll_register_callbacks cb 0x7ff9380315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53100/0 (socket says 172.21.15.16:53100) 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 -- 172.21.15.16:0/2018662716 learned_addr learned my addr 172.21.15.16:0/2018662716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94259c640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b092af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 -- 172.21.15.16:0/2018662716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 msgr2=0x55d3b092af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b092af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 -- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b091e240 msgr2=0x55d3b09269f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b091e240 0x55d3b09269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 -- 172.21.15.16:0/2018662716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3b099bf50 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff941d9b640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff938004460 tx=0x7ff938005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e58db881e0b7eb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff929ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93805c070 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff938003a00 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.922+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff938005250 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff938068050 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 10 client.16134 fetch_fsmap learned FSMap version 30 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d3b099d110 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff938003a00 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff938058550 con 0x55d3b0913980 2024-09-12T02:50:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 10 client.16134 fetch_fsmap finished waiting for FSMap version 30 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 10 client.16134 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 20 client.16134 populate_metadata read hostname 'smithi016' 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b099d880 0x55d3b099fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 4 client.16134 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e792f047", "release": true, "await": true}] 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e792f047", "release": true, "await": true}) -- 0x55d3b08800a0 con 0x55d3b099d880 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.923+0000 7ff94259c640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b099d880 0x55d3b099fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.924+0000 7ff94259c640 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b099d880 0x55d3b099fca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a59e1c6dc842f739 server_cookie=6c21c295885676e0 in_seq=0 out_seq=0 2024-09-12T02:50:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.924+0000 7ff929ffb640 10 client.16134 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff929ffb640 1 -- 172.21.15.16:0/2018662716 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d3b08800a0 con 0x55d3b099d880 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff929ffb640 10 client.16134 handle_command_reply: tid=0 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 client.16134 shutdown 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 msgr2=0x55d3b0926260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff938004460 tx=0x7ff938005fd0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 shutdown_connections 2024-09-12T02:50:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3b099d880 0x55d3b099fca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3b091e240 0x55d3b09269f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff929ffb640 0 client.16134 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.934+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3b0913980 0x55d3b0926260 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff94b3cb740 1 --2- 172.21.15.16:0/2018662716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3b0913060 0x55d3b092af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 >> 172.21.15.16:0/2018662716 conn(0x55d3b090f5f0 msgr2=0x55d3b09152d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 shutdown_connections 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff94b3cb740 1 -- 172.21.15.16:0/2018662716 wait complete. 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff94b3cb740 20 client.16134 trim_cache size 0 max 16384 2024-09-12T02:50:44.937 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 30, 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 12, 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "e792f047": { 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 12, 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:50:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.7, 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:50:44.940 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:50:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 msgr2=0x7ff92c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 0x7ff92c0a5040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff93c01d000 tx=0x7ff93c01e000 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 msgr2=0x7ff9441983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c920 tx=0x7ff93400cde0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 shutdown_connections 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff944106f10 0x7ff94419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff944102790 0x7ff944198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff92c0a2b80 0x7ff92c0a5040 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 --2- 172.21.15.16:0/1222031166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff944101de0 0x7ff9441983f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.935+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 >> 172.21.15.16:0/1222031166 conn(0x7ff9440fd8c0 msgr2=0x7ff9440febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.936+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 shutdown_connections 2024-09-12T02:50:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:50:44.936+0000 7ff948ff2640 1 -- 172.21.15.16:0/1222031166 wait complete. 2024-09-12T02:50:45.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:45 smithi016 ceph-mon[30925]: pgmap v422: 97 pgs: 97 active+clean; 94 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-12T02:50:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:45 smithi033 ceph-mon[35092]: pgmap v422: 97 pgs: 97 active+clean; 94 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-12T02:50:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:45 smithi184 ceph-mon[35321]: pgmap v422: 97 pgs: 97 active+clean; 94 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-12T02:50:45.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 186412 9.54 MB/sec warmup 107 sec latency 22654.381 ms 2024-09-12T02:50:45.763 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e792f047', total seconds elapsed: 28.1 2024-09-12T02:50:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188625 9.58 MB/sec warmup 108 sec latency 93.978 ms 2024-09-12T02:50:47.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:47 smithi016 ceph-mon[30925]: pgmap v423: 97 pgs: 97 active+clean; 94 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:50:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:47 smithi033 ceph-mon[35092]: pgmap v423: 97 pgs: 97 active+clean; 94 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:50:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:47 smithi184 ceph-mon[35321]: pgmap v423: 97 pgs: 97 active+clean; 94 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:50:47.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 191771 9.66 MB/sec warmup 109 sec latency 28.834 ms 2024-09-12T02:50:48.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 194534 9.69 MB/sec warmup 110 sec latency 11.463 ms 2024-09-12T02:50:49.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:49 smithi016 ceph-mon[30925]: pgmap v424: 97 pgs: 97 active+clean; 92 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.0 MiB/s wr, 88 op/s 2024-09-12T02:50:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:49 smithi033 ceph-mon[35092]: pgmap v424: 97 pgs: 97 active+clean; 92 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.0 MiB/s wr, 88 op/s 2024-09-12T02:50:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:49 smithi184 ceph-mon[35321]: pgmap v424: 97 pgs: 97 active+clean; 92 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 2.0 MiB/s wr, 88 op/s 2024-09-12T02:50:49.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 196782 9.71 MB/sec warmup 111 sec latency 19.074 ms 2024-09-12T02:50:50.711 INFO:tasks.workunit.client.0.smithi016.stdout: 1 199446 9.75 MB/sec warmup 112 sec latency 16.017 ms 2024-09-12T02:50:51.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:51 smithi033 ceph-mon[35092]: pgmap v425: 97 pgs: 97 active+clean; 88 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 142 op/s 2024-09-12T02:50:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:51 smithi184 ceph-mon[35321]: pgmap v425: 97 pgs: 97 active+clean; 88 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 142 op/s 2024-09-12T02:50:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:51 smithi016 ceph-mon[30925]: pgmap v425: 97 pgs: 97 active+clean; 88 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 142 op/s 2024-09-12T02:50:51.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202498 9.83 MB/sec warmup 113 sec latency 29.718 ms 2024-09-12T02:50:52.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 205181 9.86 MB/sec warmup 114 sec latency 59.898 ms 2024-09-12T02:50:53.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:53 smithi033 ceph-mon[35092]: pgmap v426: 97 pgs: 97 active+clean; 90 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 196 op/s 2024-09-12T02:50:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:53 smithi184 ceph-mon[35321]: pgmap v426: 97 pgs: 97 active+clean; 90 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 196 op/s 2024-09-12T02:50:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:53 smithi016 ceph-mon[30925]: pgmap v426: 97 pgs: 97 active+clean; 90 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 196 op/s 2024-09-12T02:50:53.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 207590 9.87 MB/sec warmup 115 sec latency 14.177 ms 2024-09-12T02:50:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:54 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:50:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:54 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:50:54.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:54 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:50:54.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210165 9.91 MB/sec warmup 116 sec latency 19.507 ms 2024-09-12T02:50:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:55 smithi033 ceph-mon[35092]: pgmap v427: 97 pgs: 97 active+clean; 91 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-09-12T02:50:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:55 smithi184 ceph-mon[35321]: pgmap v427: 97 pgs: 97 active+clean; 91 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-09-12T02:50:55.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:55 smithi016 ceph-mon[30925]: pgmap v427: 97 pgs: 97 active+clean; 91 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-09-12T02:50:55.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213239 9.99 MB/sec warmup 117 sec latency 11.115 ms 2024-09-12T02:50:56.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 216528 10.05 MB/sec warmup 118 sec latency 11.662 ms 2024-09-12T02:50:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:57 smithi033 ceph-mon[35092]: pgmap v428: 97 pgs: 97 active+clean; 95 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-09-12T02:50:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:57 smithi184 ceph-mon[35321]: pgmap v428: 97 pgs: 97 active+clean; 95 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-09-12T02:50:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:57 smithi016 ceph-mon[30925]: pgmap v428: 97 pgs: 97 active+clean; 95 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-09-12T02:50:57.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 219005 10.05 MB/sec warmup 119 sec latency 18.981 ms 2024-09-12T02:50:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:50:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:50:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:50:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:50:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:50:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:50:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:50:58.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:50:58.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:50:58.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:50:58.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:50:58.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:50:59.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 224333 14.55 MB/sec execute 1 sec latency 15.304 ms 2024-09-12T02:51:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:50:59 smithi184 ceph-mon[35321]: pgmap v429: 97 pgs: 97 active+clean; 94 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-12T02:51:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:50:59 smithi016 ceph-mon[30925]: pgmap v429: 97 pgs: 97 active+clean; 94 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-12T02:51:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:50:59 smithi033 ceph-mon[35092]: pgmap v429: 97 pgs: 97 active+clean; 94 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 271 op/s 2024-09-12T02:51:00.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227491 16.57 MB/sec execute 2 sec latency 10.968 ms 2024-09-12T02:51:01.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 230690 16.94 MB/sec execute 3 sec latency 11.866 ms 2024-09-12T02:51:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:01 smithi184 ceph-mon[35321]: pgmap v430: 97 pgs: 97 active+clean; 99 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 349 op/s 2024-09-12T02:51:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:01 smithi016 ceph-mon[30925]: pgmap v430: 97 pgs: 97 active+clean; 99 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 349 op/s 2024-09-12T02:51:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:01 smithi033 ceph-mon[35092]: pgmap v430: 97 pgs: 97 active+clean; 99 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 349 op/s 2024-09-12T02:51:02.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 233213 15.08 MB/sec execute 4 sec latency 16.781 ms 2024-09-12T02:51:02.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:51:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:03.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 -- 172.21.15.16:0/911811869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0102570 msgr2=0x7fbcb01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/911811869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0102570 0x7fbcb01029b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009900 tx=0x7fbca402a480 comp rx=0 tx=0).stop 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 -- 172.21.15.16:0/911811869 shutdown_connections 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/911811869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102ef0 0x7fbcb010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/911811869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0102570 0x7fbcb01029b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/911811869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0100e20 0x7fbcb0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 -- 172.21.15.16:0/911811869 >> 172.21.15.16:0/911811869 conn(0x7fbcb00fc7d0 msgr2=0x7fbcb00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 -- 172.21.15.16:0/911811869 shutdown_connections 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.524+0000 7fbcb63f6640 1 -- 172.21.15.16:0/911811869 wait complete. 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 Processor -- start 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 -- start start 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 0x7fbcb01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 0x7fbcb01a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0110f80 con 0x7fbcb0100e20 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0110e00 con 0x7fbcb0102570 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb63f6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0111100 con 0x7fbcb0102ef0 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52254/0 (socket says 172.21.15.16:52254) 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcaffff640 1 -- 172.21.15.16:0/3223259591 learned_addr learned my addr 172.21.15.16:0/3223259591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 0x7fbcb01a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.525+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 0x7fbcb01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 -- 172.21.15.16:0/3223259591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 msgr2=0x7fbcb01a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 0x7fbcb01a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 -- 172.21.15.16:0/3223259591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 msgr2=0x7fbcb01998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 0x7fbcb01998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 -- 172.21.15.16:0/3223259591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb01a4930 con 0x7fbcb0100e20 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 0x7fbcb01a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 0x7fbcb01998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9f0 tx=0x7fbca000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcad7fa640 1 -- 172.21.15.16:0/3223259591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca000cd80 con 0x7fbcb0100e20 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcad7fa640 1 -- 172.21.15.16:0/3223259591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca0005500 con 0x7fbcb0100e20 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb01a4b00 con 0x7fbcb0100e20 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcad7fa640 1 -- 172.21.15.16:0/3223259591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0010690 con 0x7fbcb0100e20 2024-09-12T02:51:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.526+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb01a4fc0 con 0x7fbcb0100e20 2024-09-12T02:51:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.529+0000 7fbcad7fa640 1 -- 172.21.15.16:0/3223259591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbca00046c0 con 0x7fbcb0100e20 2024-09-12T02:51:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 Processor -- start 2024-09-12T02:51:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 -- start start 2024-09-12T02:51:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 0x5645cee18140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee18680 0x5645cee1aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5645ced732e0 con 0x5645cee0d4a0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5645ced1ace0 con 0x5645cee18680 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5645ced6f2a0 con 0x5645cee0ddc0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcad7fa640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 0x7fbc980a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 0x5645cee18140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52266/0 (socket says 172.21.15.16:52266) 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb4993640 1 -- 172.21.15.16:0/3676208602 learned_addr learned my addr 172.21.15.16:0/3676208602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee18680 0x5645cee1aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 0x7fbc980a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.530+0000 7fbcad7fa640 1 -- 172.21.15.16:0/3223259591 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbca00cf520 con 0x7fbcb0100e20 2024-09-12T02:51:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 -- 172.21.15.16:0/3676208602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee18680 msgr2=0x5645cee1aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee18680 0x5645cee1aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 -- 172.21.15.16:0/3676208602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 msgr2=0x5645cee0d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 -- 172.21.15.16:0/3676208602 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645cecff440 con 0x5645cee0ddc0 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 0x5645cee18140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbca009ba30 tx=0x7fbca001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1210c0efcbce266 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 0x7fbc980a5040 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbca8005c40 tx=0x7fbca8004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbc967fc640 1 -- 172.21.15.16:0/3676208602 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca009b690 con 0x5645cee0ddc0 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbc967fc640 1 -- 172.21.15.16:0/3676208602 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca009c430 con 0x5645cee0ddc0 2024-09-12T02:51:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.531+0000 7fbc967fc640 1 -- 172.21.15.16:0/3676208602 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00cf2c0 con 0x5645cee0ddc0 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3676208602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 msgr2=0x5645cee18140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 0x5645cee18140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbca009ba30 tx=0x7fbca001d000 comp rx=0 tx=0).stop 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3676208602 shutdown_connections 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee18680 0x5645cee1aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0ddc0 0x5645cee18140 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3676208602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0d4a0 0x5645cee0d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3676208602 >> 172.21.15.16:0/3676208602 conn(0x5645cee09a30 msgr2=0x5645cee09e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3676208602 shutdown_connections 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.532+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3676208602 wait complete. 2024-09-12T02:51:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 -- start start 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 Processor -- start 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 0x5645cee17f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 0x5645cee15200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 0x5645cee15740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5645cee1b250 con 0x5645cee0ddc0 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5645ced2e050 con 0x5645cee0d4a0 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5645cee1b3d0 con 0x5645cee18680 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb87cf740 10 client.0 _ll_register_callbacks cb 0x5645ced6e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 0x5645cee15200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 0x5645cee15740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 0x5645cee15200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52282/0 (socket says 172.21.15.16:52282) 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcaf7fe640 1 -- 172.21.15.16:0/173253653 learned_addr learned my addr 172.21.15.16:0/173253653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.533+0000 7fbcaffff640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 0x5645cee17f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 -- 172.21.15.16:0/173253653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 msgr2=0x5645cee15740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 0x5645cee15740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 -- 172.21.15.16:0/173253653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 msgr2=0x5645cee15200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 0x5645cee15200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 -- 172.21.15.16:0/173253653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ceae43c0 con 0x5645cee0d4a0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcb4993640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 0x5645cee15740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcaffff640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 0x5645cee17f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbca009bef0 tx=0x7fbca009b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde12df99da20ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbc8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00a00e0 con 0x5645cee0d4a0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbca00a4720 con 0x5645cee0d4a0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbca009c430 con 0x5645cee0d4a0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbca00a49a0 con 0x5645cee0d4a0 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcb87cf740 10 client.25727 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.534+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5645cee26ba0 con 0x5645cee0d4a0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca009c430 con 0x5645cee0d4a0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fbca00b2070 con 0x5645cee0d4a0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 10 client.25727 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 20 client.25727 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee27ef0 0x5645cee2a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 4 client.25727 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645ced80850 con 0x5645cee27ef0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee27ef0 0x5645cee2a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee27ef0 0x5645cee2a2e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d9c5b5e11f424d8 server_cookie=c3b88f0d04905a02 in_seq=0 out_seq=0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.535+0000 7fbc8f7fe640 10 client.25727 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/173253653 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5645ced80850 con 0x5645cee27ef0 2024-09-12T02:51:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbc8f7fe640 10 client.25727 handle_command_reply: tid=0 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 client.25727 shutdown 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 msgr2=0x5645cee17f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 0x5645cee17f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbca009bef0 tx=0x7fbca009b760 comp rx=0 tx=0).stop 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 shutdown_connections 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee27ef0 0x5645cee2a2e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee18680 0x5645cee15740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee0ddc0 0x5645cee15200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/173253653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0d4a0 0x5645cee17f50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 >> 172.21.15.16:0/173253653 conn(0x5645cee09a30 msgr2=0x5645cee0f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbc8f7fe640 0 client.25727 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 shutdown_connections 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.536+0000 7fbcb87cf740 1 -- 172.21.15.16:0/173253653 wait complete. 2024-09-12T02:51:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.537+0000 7fbcb87cf740 20 client.25727 trim_cache size 0 max 16384 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 Processor -- start 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 -- start start 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee12460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee129a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01ab2e0 con 0x5645cee18680 2024-09-12T02:51:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01ace50 con 0x5645cee0ddc0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01ab9b0 con 0x5645cee0d4a0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52298/0 (socket says 172.21.15.16:52298) 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee12460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.556+0000 7fbcb4993640 1 -- 172.21.15.16:0/1638314327 learned_addr learned my addr 172.21.15.16:0/1638314327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 -- 172.21.15.16:0/1638314327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 msgr2=0x5645cee129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 -- 172.21.15.16:0/1638314327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 msgr2=0x5645cee16a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 -- 172.21.15.16:0/1638314327 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645cee182c0 con 0x5645cee0d4a0 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcb4993640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:03.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:03.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbcaffff640 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee12460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbca0099fe0 tx=0x7fbca0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56ab5d27be35c510 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbc95ffb640 1 -- 172.21.15.16:0/1638314327 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00a1360 con 0x5645cee0d4a0 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbc95ffb640 1 -- 172.21.15.16:0/1638314327 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca009bd30 con 0x5645cee0d4a0 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.557+0000 7fbc95ffb640 1 -- 172.21.15.16:0/1638314327 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00a4720 con 0x5645cee0d4a0 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- 172.21.15.16:0/1638314327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 msgr2=0x5645cee12460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee12460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbca0099fe0 tx=0x7fbca0004a40 comp rx=0 tx=0).stop 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- 172.21.15.16:0/1638314327 shutdown_connections 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee16a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/1638314327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee12460 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- 172.21.15.16:0/1638314327 >> 172.21.15.16:0/1638314327 conn(0x5645cee09a30 msgr2=0x5645cee142f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- 172.21.15.16:0/1638314327 shutdown_connections 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- 172.21.15.16:0/1638314327 wait complete. 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.558+0000 7fbcb87cf740 1 -- start start 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 Processor -- start 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee2c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee2c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01ac750 con 0x5645cee18680 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01ac080 con 0x5645cee0ddc0 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01a92e0 con 0x5645cee0d4a0 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb87cf740 10 client.0 _ll_register_callbacks cb 0x7fbca8009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb4993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52304/0 (socket says 172.21.15.16:52304) 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.559+0000 7fbcb4993640 1 -- 172.21.15.16:0/3383348026 learned_addr learned my addr 172.21.15.16:0/3383348026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee2c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 -- 172.21.15.16:0/3383348026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 msgr2=0x5645cee2c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee2c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 -- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 msgr2=0x5645cee2c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee2c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 -- 172.21.15.16:0/3383348026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645cee90d70 con 0x5645cee18680 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcaf7fe640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee2c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb4993640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbca8012040 tx=0x7fbca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5af1488fc3e3f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbc8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8004d20 con 0x5645cee18680 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbca801f9d0 con 0x5645cee18680 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.560+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5645ced7ad60 con 0x5645cee18680 2024-09-12T02:51:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbca801fbd0 con 0x5645cee18680 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 10 client.16152 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5645cee92050 con 0x5645cee18680 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5645ced7ad60 con 0x5645cee18680 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fbca802b400 con 0x5645cee18680 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 10 client.16152 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 20 client.16152 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee92850 0x5645cee94c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 4 client.16152 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645cecf7210 con 0x5645cee92850 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.561+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee92850 0x5645cee94c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbc8f7fe640 10 client.16152 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcaffff640 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee92850 0x5645cee94c70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd508ac43053807 server_cookie=8152a3b4abaaffdf in_seq=0 out_seq=0 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbc8f7fe640 1 -- 172.21.15.16:0/3383348026 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5645cecf7210 con 0x5645cee92850 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbc8f7fe640 10 client.16152 handle_command_reply: tid=0 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 client.16152 shutdown 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 msgr2=0x5645cee90830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbca8012040 tx=0x7fbca8004a40 comp rx=0 tx=0).stop 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 shutdown_connections 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5645cee92850 0x5645cee94c70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645cee18680 0x5645cee90830 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5645cee0ddc0 0x5645cee2c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 --2- 172.21.15.16:0/3383348026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5645cee0d4a0 0x5645cee2c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 >> 172.21.15.16:0/3383348026 conn(0x5645cee09a30 msgr2=0x5645cee08350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbc8f7fe640 0 client.16152 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 shutdown_connections 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 1 -- 172.21.15.16:0/3383348026 wait complete. 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.562+0000 7fbcb87cf740 20 client.16152 trim_cache size 0 max 16384 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:51:03.569 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 msgr2=0x7fbc980a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 0x7fbc980a5040 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbca8005c40 tx=0x7fbca8004ab0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 msgr2=0x7fbcb0199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9f0 tx=0x7fbca000edf0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 shutdown_connections 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcb0102ef0 0x7fbcb01a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcb0102570 0x7fbcb01998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc980a2b80 0x7fbc980a5040 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 --2- 172.21.15.16:0/3223259591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb0100e20 0x7fbcb0199370 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 >> 172.21.15.16:0/3223259591 conn(0x7fbcb00fc7d0 msgr2=0x7fbcb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 shutdown_connections 2024-09-12T02:51:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:03.563+0000 7fbcb63f6640 1 -- 172.21.15.16:0/3223259591 wait complete. 2024-09-12T02:51:03.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 235557 15.29 MB/sec execute 5 sec latency 28.806 ms 2024-09-12T02:51:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:03 smithi184 ceph-mon[35321]: pgmap v431: 97 pgs: 97 active+clean; 100 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-09-12T02:51:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:03 smithi016 ceph-mon[30925]: pgmap v431: 97 pgs: 97 active+clean; 100 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-09-12T02:51:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:03 smithi033 ceph-mon[35092]: pgmap v431: 97 pgs: 97 active+clean; 100 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-09-12T02:51:04.403 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:51:04.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:51:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:04.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238358 15.10 MB/sec execute 6 sec latency 12.452 ms 2024-09-12T02:51:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/3893898592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 msgr2=0x7f7bf41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/3893898592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf41067e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009900 tx=0x7f7be402a410 comp rx=0 tx=0).stop 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/3893898592 shutdown_connections 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/3893898592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4106f10 0x7f7bf410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/3893898592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf41067e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/3893898592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4101de0 0x7f7bf41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/3893898592 >> 172.21.15.16:0/3893898592 conn(0x7f7bf40fd8e0 msgr2=0x7f7bf40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/3893898592 shutdown_connections 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/3893898592 wait complete. 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 Processor -- start 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.155+0000 7f7bfb9c4640 1 -- start start 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 0x7f7bf40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4106f10 0x7f7bf4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4111f70 con 0x7f7bf4102790 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4111df0 con 0x7f7bf4106f10 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf41120f0 con 0x7f7bf4101de0 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf9760640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 0x7f7bf40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf9760640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 0x7f7bf40744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41774/0 (socket says 172.21.15.16:41774) 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf9760640 1 -- 172.21.15.16:0/2535158018 learned_addr learned my addr 172.21.15.16:0/2535158018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf9f61640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4106f10 0x7f7bf4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 -- 172.21.15.16:0/2535158018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 msgr2=0x7f7bf40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 0x7f7bf40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 -- 172.21.15.16:0/2535158018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4106f10 msgr2=0x7f7bf4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4106f10 0x7f7bf4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 -- 172.21.15.16:0/2535158018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf41a68f0 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bf8f5f640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf4074a30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7be4005940 tx=0x7f7be4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be403f030 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf41a6b80 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be4003ad0 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.156+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4035070 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.157+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf41a70b0 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.157+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf41021c0 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.159+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7be4042030 con 0x7f7bf4102790 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.160+0000 7f7bea7fc640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 0x7f7be00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.160+0000 7f7bf9760640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 0x7f7be00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.160+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7be40ef760 con 0x7f7bf4102790 2024-09-12T02:51:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.160+0000 7f7bf9760640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 0x7f7be00a50c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00058c0 tx=0x7f7bf000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.163+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7be40bc080 con 0x7f7bf4102790 2024-09-12T02:51:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.383+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bf4075a20 con 0x7f7bf4102790 2024-09-12T02:51:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.384+0000 7f7bea7fc640 1 -- 172.21.15.16:0/2535158018 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11970 (secure 0 0 0) 0x7f7be40c0380 con 0x7f7bf4102790 2024-09-12T02:51:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:51:05.386 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"btime":"2024-09-12T02:50:53:160662+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:50:52.566870+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T02:51:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 msgr2=0x7f7be00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 0x7f7be00a50c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00058c0 tx=0x7f7bf000a380 comp rx=0 tx=0).stop 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 msgr2=0x7f7bf4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf4074a30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7be4005940 tx=0x7f7be4004270 comp rx=0 tx=0).stop 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 shutdown_connections 2024-09-12T02:51:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf4106f10 0x7f7bf4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7be00a2c00 0x7f7be00a50c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf4102790 0x7f7bf4074a30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 --2- 172.21.15.16:0/2535158018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bf4101de0 0x7f7bf40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.386+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 >> 172.21.15.16:0/2535158018 conn(0x7f7bf40fd8e0 msgr2=0x7f7bf40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.387+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 shutdown_connections 2024-09-12T02:51:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:05.387+0000 7f7bfb9c4640 1 -- 172.21.15.16:0/2535158018 wait complete. 2024-09-12T02:51:05.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241465 15.62 MB/sec execute 7 sec latency 40.502 ms 2024-09-12T02:51:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:05 smithi184 ceph-mon[35321]: pgmap v432: 97 pgs: 97 active+clean; 101 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-09-12T02:51:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:05 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2535158018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:06.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:05 smithi016 ceph-mon[30925]: pgmap v432: 97 pgs: 97 active+clean; 101 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-09-12T02:51:06.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:05 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2535158018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:06.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:51:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:05 smithi033 ceph-mon[35092]: pgmap v432: 97 pgs: 97 active+clean; 101 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-09-12T02:51:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:05 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2535158018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:06.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 244243 15.28 MB/sec execute 8 sec latency 10.638 ms 2024-09-12T02:51:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 -- 172.21.15.16:0/2618284370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901082a0 msgr2=0x7fde9010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 --2- 172.21.15.16:0/2618284370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901082a0 0x7fde9010ec40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fde88009930 tx=0x7fde8802a420 comp rx=0 tx=0).stop 2024-09-12T02:51:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 -- 172.21.15.16:0/2618284370 shutdown_connections 2024-09-12T02:51:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 --2- 172.21.15.16:0/2618284370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901082a0 0x7fde9010ec40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 --2- 172.21.15.16:0/2618284370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde901036e0 0x7fde90107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 --2- 172.21.15.16:0/2618284370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde90102d30 0x7fde90103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 -- 172.21.15.16:0/2618284370 >> 172.21.15.16:0/2618284370 conn(0x7fde900fc7d0 msgr2=0x7fde900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 -- 172.21.15.16:0/2618284370 shutdown_connections 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.928+0000 7fde95d1f640 1 -- 172.21.15.16:0/2618284370 wait complete. 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 Processor -- start 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 -- start start 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde90102d30 0x7fde90198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde901082a0 0x7fde9019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde901120a0 con 0x7fde901036e0 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde90111f20 con 0x7fde901082a0 2024-09-12T02:51:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde95d1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde90112220 con 0x7fde90102d30 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51220/0 (socket says 172.21.15.16:51220) 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 -- 172.21.15.16:0/3035672327 learned_addr learned my addr 172.21.15.16:0/3035672327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde901082a0 0x7fde9019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde90102d30 0x7fde90198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 -- 172.21.15.16:0/3035672327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde90102d30 msgr2=0x7fde90198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde90102d30 0x7fde90198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 -- 172.21.15.16:0/3035672327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde901082a0 msgr2=0x7fde9019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde901082a0 0x7fde9019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.929+0000 7fde8effd640 1 -- 172.21.15.16:0/3035672327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9019d510 con 0x7fde901036e0 2024-09-12T02:51:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde8effd640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e9c0 tx=0x7fde8400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde8cff9640 1 -- 172.21.15.16:0/3035672327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8400ccc0 con 0x7fde901036e0 2024-09-12T02:51:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde8cff9640 1 -- 172.21.15.16:0/3035672327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde84005500 con 0x7fde901036e0 2024-09-12T02:51:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde8cff9640 1 -- 172.21.15.16:0/3035672327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84010690 con 0x7fde901036e0 2024-09-12T02:51:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9019d800 con 0x7fde901036e0 2024-09-12T02:51:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.930+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde901a50e0 con 0x7fde901036e0 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8cff9640 1 -- 172.21.15.16:0/3035672327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fde84004e80 con 0x7fde901036e0 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 Processor -- start 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 -- start start 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071b3370 0x5576071b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55760710e2f0 con 0x5576071a9b40 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576070b5cf0 con 0x5576071a85d0 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde980f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55760710a2b0 con 0x5576071b3370 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41862/0 (socket says 172.21.15.16:41862) 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2557259258 learned_addr learned my addr 172.21.15.16:0/2557259258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.933+0000 7fde8effd640 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071b3370 0x5576071b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2557259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071b3370 msgr2=0x5576071b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071b3370 0x5576071b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2557259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 msgr2=0x5576071a8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071a8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8cff9640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 0x7fde780a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2557259258 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760709a3a0 con 0x5576071a85d0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8effd640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 0x7fde780a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde800114f0 tx=0x7fde800119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17041c044186c010 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde75ffb640 1 -- 172.21.15.16:0/2557259258 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8001e070 con 0x5576071a85d0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde8cff9640 1 -- 172.21.15.16:0/3035672327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fde840cf300 con 0x7fde901036e0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde75ffb640 1 -- 172.21.15.16:0/2557259258 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde8000fb00 con 0x5576071a85d0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.934+0000 7fde75ffb640 1 -- 172.21.15.16:0/2557259258 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde80019bf0 con 0x5576071a85d0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- 172.21.15.16:0/2557259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 msgr2=0x5576071a8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde800114f0 tx=0x7fde800119b0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde8effd640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 0x7fde780a50c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde840cd180 tx=0x7fde840d2de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- 172.21.15.16:0/2557259258 shutdown_connections 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071b3370 0x5576071b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071a8a10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 --2- 172.21.15.16:0/2557259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071a8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- 172.21.15.16:0/2557259258 >> 172.21.15.16:0/2557259258 conn(0x557607171310 msgr2=0x5576071a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- 172.21.15.16:0/2557259258 shutdown_connections 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- 172.21.15.16:0/2557259258 wait complete. 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.935+0000 7fde980f8740 1 -- start start 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 Processor -- start 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5576071b6070 con 0x5576071b3370 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576070c9060 con 0x5576071a9b40 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5576071b61f0 con 0x5576071a85d0 2024-09-12T02:51:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde980f8740 10 client.0 _ll_register_callbacks cb 0x557606e1c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41866/0 (socket says 172.21.15.16:41866) 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2766140074 learned_addr learned my addr 172.21.15.16:0/2766140074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2766140074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 msgr2=0x5576071b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 msgr2=0x5576071b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.936+0000 7fde8f7fe640 1 -- 172.21.15.16:0/2766140074 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557606e7f3c0 con 0x5576071a9b40 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde80005c10 tx=0x7fde80019410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43a478beb781887 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde6effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8001e070 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fde80011240 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde8000fb00 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde800194c0 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde980f8740 10 client.25748 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5576071c1b10 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.937+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde8000fb00 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fde80005040 con 0x5576071a9b40 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 10 client.25748 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 10 client.25748 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 20 client.25748 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576071c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 4 client.25748 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55760711b860 con 0x5576071c2e60 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde8ffff640 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576071c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde8ffff640 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576071c5250 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=864714fd304b9bd server_cookie=20b5ab7d6b452dfb in_seq=0 out_seq=0 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.938+0000 7fde6effd640 10 client.25748 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde6effd640 1 -- 172.21.15.16:0/2766140074 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55760711b860 con 0x5576071c2e60 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde6effd640 10 client.25748 handle_command_reply: tid=0 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 client.25748 shutdown 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 msgr2=0x5576071b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde80005c10 tx=0x7fde80019410 comp rx=0 tx=0).stop 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 shutdown_connections 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576071c5250 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a9b40 0x5576071b0020 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde6effd640 0 client.25748 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 --2- 172.21.15.16:0/2766140074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 >> 172.21.15.16:0/2766140074 conn(0x557607171310 msgr2=0x5576071aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 shutdown_connections 2024-09-12T02:51:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 1 -- 172.21.15.16:0/2766140074 wait complete. 2024-09-12T02:51:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.939+0000 7fde980f8740 20 client.25748 trim_cache size 0 max 16384 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 Processor -- start 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 -- start start 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071bf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071b3370 0x5576071ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde901ad5e0 con 0x5576071a9b40 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde901af150 con 0x5576071b3370 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde980f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde901adcb0 con 0x5576071a85d0 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51240/0 (socket says 172.21.15.16:51240) 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3927622535 learned_addr learned my addr 172.21.15.16:0/3927622535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071bf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071b3370 0x5576071ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3927622535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 msgr2=0x5576071bf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071bf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3927622535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071b3370 msgr2=0x5576071ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.953+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071b3370 0x5576071ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.954+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3927622535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576071b0980 con 0x5576071a9b40 2024-09-12T02:51:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.954+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fde80005580 tx=0x7fde80005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eeeefd915e4c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.954+0000 7fde757fa640 1 -- 172.21.15.16:0/3927622535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8001e070 con 0x5576071a9b40 2024-09-12T02:51:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.954+0000 7fde757fa640 1 -- 172.21.15.16:0/3927622535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde80033070 con 0x5576071a9b40 2024-09-12T02:51:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.954+0000 7fde757fa640 1 -- 172.21.15.16:0/3927622535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde80009c40 con 0x5576071a9b40 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- 172.21.15.16:0/3927622535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 msgr2=0x5576071af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fde80005580 tx=0x7fde80005cf0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- 172.21.15.16:0/3927622535 shutdown_connections 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071b3370 0x5576071ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071a9b40 0x5576071af000 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 --2- 172.21.15.16:0/3927622535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a85d0 0x5576071bf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- 172.21.15.16:0/3927622535 >> 172.21.15.16:0/3927622535 conn(0x557607171310 msgr2=0x5576071ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- 172.21.15.16:0/3927622535 shutdown_connections 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- 172.21.15.16:0/3927622535 wait complete. 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 -- start start 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.955+0000 7fde980f8740 1 Processor -- start 2024-09-12T02:51:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071bbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde901aea50 con 0x5576071b3370 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde901ae380 con 0x5576071a85d0 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde901ab5e0 con 0x5576071a9b40 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde980f8740 10 client.0 _ll_register_callbacks cb 0x7fde8001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58994/0 (socket says 172.21.15.16:58994) 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 -- 172.21.15.16:0/942019401 learned_addr learned my addr 172.21.15.16:0/942019401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8ffff640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8ffff640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071bbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 -- 172.21.15.16:0/942019401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 msgr2=0x5576071bbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071bbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 -- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 msgr2=0x5576071ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 -- 172.21.15.16:0/942019401 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760722c9b0 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8ffff640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071bbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fde80009370 tx=0x7fde80005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada6761d6091fc57 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.956+0000 7fde6effd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8001e070 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde80033070 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fde80005030 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde80005230 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 10 client.25648 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55760722db40 con 0x5576071a9b40 2024-09-12T02:51:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde80033070 con 0x5576071a9b40 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fde80037410 con 0x5576071a9b40 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 10 client.25648 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 10 client.25648 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 20 client.25648 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576072301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 4 client.25648 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557607115d70 con 0x5576071c2e60 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.957+0000 7fde8ffff640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576072301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde8ffff640 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576072301f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db044f3903cc829f server_cookie=6d6885f01cc23fd8 in_seq=0 out_seq=0 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde6effd640 10 client.25648 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde6effd640 1 -- 172.21.15.16:0/942019401 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557607115d70 con 0x5576071c2e60 2024-09-12T02:51:06.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde6effd640 10 client.25648 handle_command_reply: tid=0 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 client.25648 shutdown 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 msgr2=0x5576071acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fde80009370 tx=0x7fde80005780 comp rx=0 tx=0).stop 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 shutdown_connections 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576071b3370 0x5576071ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576071a9b40 0x5576071acec0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5576071c2e60 0x5576072301f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 --2- 172.21.15.16:0/942019401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576071a85d0 0x5576071bbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 >> 172.21.15.16:0/942019401 conn(0x557607171310 msgr2=0x5576071af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde6effd640 0 client.25648 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 shutdown_connections 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 1 -- 172.21.15.16:0/942019401 wait complete. 2024-09-12T02:51:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.958+0000 7fde980f8740 20 client.25648 trim_cache size 0 max 16384 2024-09-12T02:51:06.966 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 msgr2=0x7fde780a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 0x7fde780a50c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde840cd180 tx=0x7fde840d2de0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 msgr2=0x7fde90198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e9c0 tx=0x7fde8400ee80 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 shutdown_connections 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde901082a0 0x7fde9019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde780a2c00 0x7fde780a50c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901036e0 0x7fde90198aa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 --2- 172.21.15.16:0/3035672327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde90102d30 0x7fde90198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 >> 172.21.15.16:0/3035672327 conn(0x7fde900fc7d0 msgr2=0x7fde90105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 shutdown_connections 2024-09-12T02:51:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:06.959+0000 7fde95d1f640 1 -- 172.21.15.16:0/3035672327 wait complete. 2024-09-12T02:51:07.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 246490 14.89 MB/sec execute 9 sec latency 28.448 ms 2024-09-12T02:51:07.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:51:07.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:07 smithi016 ceph-mon[30925]: pgmap v433: 97 pgs: 97 active+clean; 105 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 291 op/s 2024-09-12T02:51:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:08.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:07 smithi184 ceph-mon[35321]: pgmap v433: 97 pgs: 97 active+clean; 105 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 291 op/s 2024-09-12T02:51:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:07 smithi033 ceph-mon[35092]: pgmap v433: 97 pgs: 97 active+clean; 105 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.5 MiB/s wr, 291 op/s 2024-09-12T02:51:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 -- 172.21.15.16:0/3680975024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 msgr2=0x7fed54101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 --2- 172.21.15.16:0/3680975024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54101200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fed44009930 tx=0x7fed4402a4d0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 -- 172.21.15.16:0/3680975024 shutdown_connections 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 --2- 172.21.15.16:0/3680975024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102ef0 0x7fed5410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 --2- 172.21.15.16:0/3680975024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102570 0x7fed541029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 --2- 172.21.15.16:0/3680975024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54101200 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.681+0000 7fed58f02640 1 -- 172.21.15.16:0/3680975024 >> 172.21.15.16:0/3680975024 conn(0x7fed540fc7d0 msgr2=0x7fed540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.682+0000 7fed58f02640 1 -- 172.21.15.16:0/3680975024 shutdown_connections 2024-09-12T02:51:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.682+0000 7fed58f02640 1 -- 172.21.15.16:0/3680975024 wait complete. 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.682+0000 7fed58f02640 1 Processor -- start 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.682+0000 7fed58f02640 1 -- start start 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 0x7fed5419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed5410fec0 con 0x7fed54102570 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed5410fd40 con 0x7fed54102ef0 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed58f02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed54110040 con 0x7fed54100e20 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51264/0 (socket says 172.21.15.16:51264) 2024-09-12T02:51:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 -- 172.21.15.16:0/3724969734 learned_addr learned my addr 172.21.15.16:0/3724969734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed5259c640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed52d9d640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 0x7fed5419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 -- 172.21.15.16:0/3724969734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 msgr2=0x7fed54196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 -- 172.21.15.16:0/3724969734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 msgr2=0x7fed5419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 0x7fed5419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed51d9b640 1 -- 172.21.15.16:0/3724969734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5419b2f0 con 0x7fed54102570 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.683+0000 7fed52d9d640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 0x7fed5419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed51d9b640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fed4800ea60 tx=0x7fed4800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed437fe640 1 -- 172.21.15.16:0/3724969734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4800cd20 con 0x7fed54102570 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed437fe640 1 -- 172.21.15.16:0/3724969734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed480035f0 con 0x7fed54102570 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5419b5e0 con 0x7fed54102570 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed437fe640 1 -- 172.21.15.16:0/3724969734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48010690 con 0x7fed54102570 2024-09-12T02:51:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.684+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed541a2ec0 con 0x7fed54102570 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.686+0000 7fed437fe640 1 -- 172.21.15.16:0/3724969734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fed480037a0 con 0x7fed54102570 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 Processor -- start 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 -- start start 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 0x55831f96ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f978760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55831f8d12f0 con 0x55831f96cb40 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55831f878cf0 con 0x55831f976370 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed5b2db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55831f8cd2b0 con 0x55831f96b5d0 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51272/0 (socket says 172.21.15.16:51272) 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f978760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.687+0000 7fed52d9d640 1 -- 172.21.15.16:0/2016824423 learned_addr learned my addr 172.21.15.16:0/2016824423 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed437fe640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 0x7fed3c0a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed5259c640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 0x55831f96ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed52d9d640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 0x7fed3c0a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed437fe640 1 -- 172.21.15.16:0/3724969734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fed480cf310 con 0x7fed54102570 2024-09-12T02:51:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 -- 172.21.15.16:0/2016824423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 msgr2=0x55831f96ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 0x55831f96ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 -- 172.21.15.16:0/2016824423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 msgr2=0x55831f96b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 -- 172.21.15.16:0/2016824423 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55831f85d3a0 con 0x55831f976370 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed5259c640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 0x55831f96ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed52d9d640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed51d9b640 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f978760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed480cce70 tx=0x7fed48003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e9d81d172f8294d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed52d9d640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 0x7fed3c0a7250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009b80 tx=0x7fed4c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.688+0000 7fed40ff9640 1 -- 172.21.15.16:0/2016824423 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4809b6c0 con 0x55831f976370 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed40ff9640 1 -- 172.21.15.16:0/2016824423 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4809c280 con 0x55831f976370 2024-09-12T02:51:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed40ff9640 1 -- 172.21.15.16:0/2016824423 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4809b4e0 con 0x55831f976370 2024-09-12T02:51:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 -- 172.21.15.16:0/2016824423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 msgr2=0x55831f978760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f978760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed480cce70 tx=0x7fed48003f30 comp rx=0 tx=0).stop 2024-09-12T02:51:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 -- 172.21.15.16:0/2016824423 shutdown_connections 2024-09-12T02:51:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f978760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96b5d0 0x55831f96ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 --2- 172.21.15.16:0/2016824423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96cb40 0x55831f96b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 -- 172.21.15.16:0/2016824423 >> 172.21.15.16:0/2016824423 conn(0x55831f934310 msgr2=0x55831f967c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 -- 172.21.15.16:0/2016824423 shutdown_connections 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.689+0000 7fed5b2db740 1 -- 172.21.15.16:0/2016824423 wait complete. 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 -- start start 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 Processor -- start 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f973080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 0x55831f973810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55831f978fb0 con 0x55831f976370 2024-09-12T02:51:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55831f88c060 con 0x55831f96b5d0 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55831f979130 con 0x55831f96cb40 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5b2db740 10 client.0 _ll_register_callbacks cb 0x55831f5df8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 0x55831f973810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f973080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41920/0 (socket says 172.21.15.16:41920) 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.690+0000 7fed5259c640 1 -- 172.21.15.16:0/1470835215 learned_addr learned my addr 172.21.15.16:0/1470835215 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 -- 172.21.15.16:0/1470835215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 msgr2=0x55831f973080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f973080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 -- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 msgr2=0x55831f975dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 -- 172.21.15.16:0/1470835215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55831f6423c0 con 0x55831f976370 2024-09-12T02:51:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed5259c640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed52d9d640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 0x55831f973810 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed4c005fd0 tx=0x7fed4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999bd3b32462757f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c026070 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fed4c004d20 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4809c280 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed4c031050 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed5b2db740 10 client.16188 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.691+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55831f984b30 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4809c280 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fed4c01f050 con 0x55831f976370 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 10 client.16188 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 10 client.16188 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:51:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 20 client.16188 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f985f40 0x55831f988350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 4 client.16188 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55831f8de860 con 0x55831f985f40 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.692+0000 7fed5259c640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f985f40 0x55831f988350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed5259c640 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f985f40 0x55831f988350 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b17821afc6122e6 server_cookie=24414593e1793f9f in_seq=0 out_seq=0 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed39ffb640 10 client.16188 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed39ffb640 1 -- 172.21.15.16:0/1470835215 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55831f8de860 con 0x55831f985f40 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed39ffb640 10 client.16188 handle_command_reply: tid=0 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed5b2db740 1 client.16188 shutdown 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 msgr2=0x55831f973810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.693+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 0x55831f973810 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed4c005fd0 tx=0x7fed4c004a40 comp rx=0 tx=0).stop 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 shutdown_connections 2024-09-12T02:51:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f985f40 0x55831f988350 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f976370 0x55831f973810 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f973080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1470835215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96b5d0 0x55831f975dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 >> 172.21.15.16:0/1470835215 conn(0x55831f934310 msgr2=0x55831f96d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed39ffb640 0 client.16188 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 shutdown_connections 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 1 -- 172.21.15.16:0/1470835215 wait complete. 2024-09-12T02:51:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.694+0000 7fed5b2db740 20 client.16188 trim_cache size 0 max 16384 2024-09-12T02:51:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 Processor -- start 2024-09-12T02:51:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 -- start start 2024-09-12T02:51:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f96b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f96e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f96e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed541ab3a0 con 0x55831f96b5d0 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed541acf10 con 0x55831f976370 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5b2db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed541aba70 con 0x55831f96cb40 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f96b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f96b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51288/0 (socket says 172.21.15.16:51288) 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed5259c640 1 -- 172.21.15.16:0/4230162147 learned_addr learned my addr 172.21.15.16:0/4230162147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed52d9d640 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f96e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.708+0000 7fed51d9b640 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f96e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 -- 172.21.15.16:0/4230162147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 msgr2=0x55831f96e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f96e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 -- 172.21.15.16:0/4230162147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 msgr2=0x55831f96b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f96b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 -- 172.21.15.16:0/4230162147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55831f982250 con 0x55831f976370 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed52d9d640 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f96e660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fed4c012d50 tx=0x7fed4c00fbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ff38d213da00725 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed3bfff640 1 -- 172.21.15.16:0/4230162147 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c03a070 con 0x55831f976370 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed3bfff640 1 -- 172.21.15.16:0/4230162147 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4c00f9d0 con 0x55831f976370 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.709+0000 7fed3bfff640 1 -- 172.21.15.16:0/4230162147 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c026040 con 0x55831f976370 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- 172.21.15.16:0/4230162147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 msgr2=0x55831f96e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f96e660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fed4c012d50 tx=0x7fed4c00fbe0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- 172.21.15.16:0/4230162147 shutdown_connections 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f976370 0x55831f96e660 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f96cb40 0x55831f96e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 --2- 172.21.15.16:0/4230162147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f96b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- 172.21.15.16:0/4230162147 >> 172.21.15.16:0/4230162147 conn(0x55831f934310 msgr2=0x55831f962a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- 172.21.15.16:0/4230162147 shutdown_connections 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- 172.21.15.16:0/4230162147 wait complete. 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 -- start start 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.710+0000 7fed5b2db740 1 Processor -- start 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 0x55831f96fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f976370 0x55831f970380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed541ac810 con 0x55831f96b5d0 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed541ac140 con 0x55831f96cb40 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed541a9380 con 0x55831f976370 2024-09-12T02:51:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5b2db740 10 client.0 _ll_register_callbacks cb 0x7fed4c00f790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:08.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 249110 14.83 MB/sec execute 10 sec latency 15.887 ms 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51294/0 (socket says 172.21.15.16:51294) 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 0x55831f96fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5259c640 1 -- 172.21.15.16:0/1523661058 learned_addr learned my addr 172.21.15.16:0/1523661058 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed52d9d640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f976370 0x55831f970380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 -- 172.21.15.16:0/1523661058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f976370 msgr2=0x55831f970380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f976370 0x55831f970380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 -- 172.21.15.16:0/1523661058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 msgr2=0x55831f97eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 -- 172.21.15.16:0/1523661058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55831f970bb0 con 0x55831f96cb40 2024-09-12T02:51:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed5259c640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed51d9b640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 0x55831f96fe40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fed480a00f0 tx=0x7fed4809b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b44931ff11c0b6d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.711+0000 7fed39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4809c280 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fed480981f0 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55831f8d8d70 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed4809bb80 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 10 client.25772 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55831f9f5060 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55831f8d8d70 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fed4809afd0 con 0x55831f96cb40 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 10 client.25772 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 10 client.25772 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 20 client.25772 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f9f52f0 0x55831f9f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 4 client.25772 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55831f855170 con 0x55831f9f52f0 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.712+0000 7fed5259c640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f9f52f0 0x55831f9f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed39ffb640 10 client.25772 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5259c640 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f9f52f0 0x55831f9f76e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cbca39c9e40a73eb server_cookie=9ea03b51d8ae89f5 in_seq=0 out_seq=0 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed39ffb640 1 -- 172.21.15.16:0/1523661058 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55831f855170 con 0x55831f9f52f0 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed39ffb640 10 client.25772 handle_command_reply: tid=0 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 client.25772 shutdown 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 msgr2=0x55831f96fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 0x55831f96fe40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fed480a00f0 tx=0x7fed4809b740 comp rx=0 tx=0).stop 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 shutdown_connections 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55831f9f52f0 0x55831f9f76e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed39ffb640 0 client.25772 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55831f976370 0x55831f970380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55831f96cb40 0x55831f96fe40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 --2- 172.21.15.16:0/1523661058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55831f96b5d0 0x55831f97eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.713+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 >> 172.21.15.16:0/1523661058 conn(0x55831f934310 msgr2=0x55831f9748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 shutdown_connections 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed5b2db740 1 -- 172.21.15.16:0/1523661058 wait complete. 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed5b2db740 20 client.25772 trim_cache size 0 max 16384 2024-09-12T02:51:08.721 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 msgr2=0x7fed3c0a7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 0x7fed3c0a7250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009b80 tx=0x7fed4c004ab0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 msgr2=0x7fed54196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fed4800ea60 tx=0x7fed4800ee60 comp rx=0 tx=0).stop 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 shutdown_connections 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed54102ef0 0x7fed5419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fed3c0a4d90 0x7fed3c0a7250 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed54102570 0x7fed54196880 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 --2- 172.21.15.16:0/3724969734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed54100e20 0x7fed54196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.714+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 >> 172.21.15.16:0/3724969734 conn(0x7fed540fc7d0 msgr2=0x7fed540fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.715+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 shutdown_connections 2024-09-12T02:51:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:08.715+0000 7fed58f02640 1 -- 172.21.15.16:0/3724969734 wait complete. 2024-09-12T02:51:09.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:51:09.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 252119 15.15 MB/sec execute 11 sec latency 86.276 ms 2024-09-12T02:51:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:09 smithi184 ceph-mon[35321]: pgmap v434: 97 pgs: 97 active+clean; 106 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-09-12T02:51:10.058 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:09 smithi016 ceph-mon[30925]: pgmap v434: 97 pgs: 97 active+clean; 106 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-09-12T02:51:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:09 smithi033 ceph-mon[35092]: pgmap v434: 97 pgs: 97 active+clean; 106 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.308+0000 7f412795a640 1 -- 172.21.15.16:0/2938482800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 msgr2=0x7f41201031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 --2- 172.21.15.16:0/2938482800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f41201031e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f411c009900 tx=0x7f411c02a480 comp rx=0 tx=0).stop 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 -- 172.21.15.16:0/2938482800 shutdown_connections 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 --2- 172.21.15.16:0/2938482800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f412010c1a0 0x7f412010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 --2- 172.21.15.16:0/2938482800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4120103720 0x7f412010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 --2- 172.21.15.16:0/2938482800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f41201031e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 -- 172.21.15.16:0/2938482800 >> 172.21.15.16:0/2938482800 conn(0x7f41200fc7a0 msgr2=0x7f41200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 -- 172.21.15.16:0/2938482800 shutdown_connections 2024-09-12T02:51:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 -- 172.21.15.16:0/2938482800 wait complete. 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 Processor -- start 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.309+0000 7f412795a640 1 -- start start 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 0x7f4120100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f412010c1a0 0x7f41201014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4120111f70 con 0x7f4120102e00 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4120111df0 con 0x7f412010c1a0 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f412795a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41201120f0 con 0x7f4120103720 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51322/0 (socket says 172.21.15.16:51322) 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 -- 172.21.15.16:0/1651153424 learned_addr learned my addr 172.21.15.16:0/1651153424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f4124ef5640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 0x7f4120100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f412010c1a0 0x7f41201014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 -- 172.21.15.16:0/1651153424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 msgr2=0x7f4120100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 0x7f4120100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 -- 172.21.15.16:0/1651153424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f412010c1a0 msgr2=0x7f41201014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f412010c1a0 0x7f41201014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 -- 172.21.15.16:0/1651153424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41201a69b0 con 0x7f4120102e00 2024-09-12T02:51:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f4124ef5640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 0x7f4120100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41256f6640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f411c006010 tx=0x7f411c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.310+0000 7f41167fc640 1 -- 172.21.15.16:0/1651153424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c03f030 con 0x7f4120102e00 2024-09-12T02:51:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.311+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41201a6c40 con 0x7f4120102e00 2024-09-12T02:51:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.311+0000 7f41167fc640 1 -- 172.21.15.16:0/1651153424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f411c004440 con 0x7f4120102e00 2024-09-12T02:51:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.311+0000 7f41167fc640 1 -- 172.21.15.16:0/1651153424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c035070 con 0x7f4120102e00 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.311+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201a7170 con 0x7f4120102e00 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.313+0000 7f41167fc640 1 -- 172.21.15.16:0/1651153424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f411c042030 con 0x7f4120102e00 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 Processor -- start 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 -- start start 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56410e4312f0 con 0x56410e4cb5d0 2024-09-12T02:51:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56410e3d8cf0 con 0x56410e4d6370 2024-09-12T02:51:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4129d33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56410e42d2b0 con 0x56410e4ccb40 2024-09-12T02:51:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f41256f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4125ef7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4124ef5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f41256f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51326/0 (socket says 172.21.15.16:51326) 2024-09-12T02:51:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4125ef7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59100/0 (socket says 172.21.15.16:59100) 2024-09-12T02:51:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f41256f6640 1 -- 172.21.15.16:0/1190338348 learned_addr learned my addr 172.21.15.16:0/1190338348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f41167fc640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 0x7f410c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.314+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 0x7f410c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 -- 172.21.15.16:0/1190338348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 msgr2=0x56410e4cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f41167fc640 1 -- 172.21.15.16:0/1651153424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f411c0ef3d0 con 0x7f4120102e00 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 -- 172.21.15.16:0/1190338348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 msgr2=0x56410e4cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 -- 172.21.15.16:0/1190338348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410e3bd3a0 con 0x56410e4d6370 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f41256f6640 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4124ef5640 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d8760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4110009fa0 tx=0x7f41100118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82624a52fc744647 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 0x7f410c0a5040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4118009fd0 tx=0x7f4118004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f410b7fe640 1 -- 172.21.15.16:0/1190338348 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110004030 con 0x56410e4d6370 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f410b7fe640 1 -- 172.21.15.16:0/1190338348 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4110005500 con 0x56410e4d6370 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.315+0000 7f410b7fe640 1 -- 172.21.15.16:0/1190338348 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110004db0 con 0x56410e4d6370 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- 172.21.15.16:0/1190338348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 msgr2=0x56410e4d8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d8760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4110009fa0 tx=0x7f41100118d0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- 172.21.15.16:0/1190338348 shutdown_connections 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d8760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4cb5d0 0x56410e4cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 --2- 172.21.15.16:0/1190338348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- 172.21.15.16:0/1190338348 >> 172.21.15.16:0/1190338348 conn(0x56410e494310 msgr2=0x56410e4c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- 172.21.15.16:0/1190338348 shutdown_connections 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- 172.21.15.16:0/1190338348 wait complete. 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 -- start start 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.316+0000 7f4129d33740 1 Processor -- start 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 0x56410e4d5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56410e4d9070 con 0x56410e4ccb40 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56410e3ec060 con 0x56410e4d6370 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56410e4d91f0 con 0x56410e4cb5d0 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4129d33740 10 client.0 _ll_register_callbacks cb 0x56410e13f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51342/0 (socket says 172.21.15.16:51342) 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 -- 172.21.15.16:0/442463810 learned_addr learned my addr 172.21.15.16:0/442463810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4125ef7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f41256f6640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 0x56410e4d5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 -- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 msgr2=0x56410e4d5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 0x56410e4d5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 -- 172.21.15.16:0/442463810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 msgr2=0x56410e4d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4124ef5640 1 -- 172.21.15.16:0/442463810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410e1a23c0 con 0x56410e4ccb40 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f41256f6640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 0x56410e4d5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.317+0000 7f4125ef7640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4124ef5640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4110009560 tx=0x7f41100048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ab38468f4e1078 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411001d050 con 0x56410e4ccb40 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4110005500 con 0x56410e4ccb40 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4110003800 con 0x56410e4ccb40 2024-09-12T02:51:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41100046c0 con 0x56410e4ccb40 2024-09-12T02:51:10.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 10 client.16206 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:10.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56410e4e4ae0 con 0x56410e4ccb40 2024-09-12T02:51:10.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4110005500 con 0x56410e4ccb40 2024-09-12T02:51:10.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f411000f5a0 con 0x56410e4ccb40 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 10 client.16206 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 10 client.16206 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.318+0000 7f4129d33740 20 client.16206 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e4e5e90 0x56410e4e8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f4129d33740 4 client.16206 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56410e43e860 con 0x56410e4e5e90 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f41256f6640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e4e5e90 0x56410e4e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f41256f6640 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e4e5e90 0x56410e4e8280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3094f36a43a916 server_cookie=6c62e7b5712a9263 in_seq=0 out_seq=0 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.319+0000 7f4108ff9640 10 client.16206 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4108ff9640 1 -- 172.21.15.16:0/442463810 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56410e43e860 con 0x56410e4e5e90 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4108ff9640 10 client.16206 handle_command_reply: tid=0 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 client.16206 shutdown 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 msgr2=0x56410e4d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4110009560 tx=0x7f41100048b0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 shutdown_connections 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4d6370 0x56410e4d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4d2f90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e4e5e90 0x56410e4e8280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 --2- 172.21.15.16:0/442463810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4cb5d0 0x56410e4d5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 >> 172.21.15.16:0/442463810 conn(0x56410e494310 msgr2=0x56410e4cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4108ff9640 0 client.16206 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 shutdown_connections 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 1 -- 172.21.15.16:0/442463810 wait complete. 2024-09-12T02:51:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.321+0000 7f4129d33740 20 client.16206 trim_cache size 0 max 16384 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 Processor -- start 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 -- start start 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4e2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41201af710 con 0x56410e4d6370 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41201b1280 con 0x56410e4cb5d0 2024-09-12T02:51:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4129d33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41201afde0 con 0x56410e4ccb40 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.335+0000 7f4125ef7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51350/0 (socket says 172.21.15.16:51350) 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 -- 172.21.15.16:0/1962065192 learned_addr learned my addr 172.21.15.16:0/1962065192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4124ef5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f41256f6640 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 -- 172.21.15.16:0/1962065192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 msgr2=0x56410e4e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4e2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 -- 172.21.15.16:0/1962065192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 msgr2=0x56410e4c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 -- 172.21.15.16:0/1962065192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410e4d6050 con 0x56410e4d6370 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f4125ef7640 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4118013430 tx=0x7f4118009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5957a3c7ae6ed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f410affd640 1 -- 172.21.15.16:0/1962065192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118022b50 con 0x56410e4d6370 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f410affd640 1 -- 172.21.15.16:0/1962065192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f411800fe40 con 0x56410e4d6370 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.336+0000 7f410affd640 1 -- 172.21.15.16:0/1962065192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411802a600 con 0x56410e4d6370 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- 172.21.15.16:0/1962065192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 msgr2=0x56410e4eda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4118013430 tx=0x7f4118009c80 comp rx=0 tx=0).stop 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- 172.21.15.16:0/1962065192 shutdown_connections 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4d6370 0x56410e4eda90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4ccb40 0x56410e4e2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 --2- 172.21.15.16:0/1962065192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- 172.21.15.16:0/1962065192 >> 172.21.15.16:0/1962065192 conn(0x56410e494310 msgr2=0x56410e4c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- 172.21.15.16:0/1962065192 shutdown_connections 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- 172.21.15.16:0/1962065192 wait complete. 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.337+0000 7f4129d33740 1 -- start start 2024-09-12T02:51:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 Processor -- start 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4d6370 0x56410e4dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41201b0b80 con 0x56410e4ccb40 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41201b04b0 con 0x56410e4cb5d0 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41201ad6f0 con 0x56410e4d6370 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4129d33740 10 client.0 _ll_register_callbacks cb 0x7f411c0b7d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4124ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f41256f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f41256f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42000/0 (socket says 172.21.15.16:42000) 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f41256f6640 1 -- 172.21.15.16:0/2126750170 learned_addr learned my addr 172.21.15.16:0/2126750170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:10.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.338+0000 7f4125ef7640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4d6370 0x56410e4dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 -- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4d6370 msgr2=0x56410e4dea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4d6370 0x56410e4dea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 -- 172.21.15.16:0/2126750170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 msgr2=0x56410e4de4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4de4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 -- 172.21.15.16:0/2126750170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410e4cf240 con 0x56410e4cb5d0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4124ef5640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f41256f6640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f411c000c00 tx=0x7f411c0b7e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248b5c6a60939d6c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c049040 con 0x56410e4cb5d0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f411c0ecf00 con 0x56410e4cb5d0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56410e438d70 con 0x56410e4cb5d0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f411c0bb7d0 con 0x56410e4cb5d0 2024-09-12T02:51:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4129d33740 10 client.25790 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56410e54fc20 con 0x56410e4cb5d0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56410e438d70 con 0x56410e4cb5d0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.339+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f411c0b79e0 con 0x56410e4cb5d0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 10 client.25790 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 10 client.25790 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 20 client.25790 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e5502f0 0x56410e5526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 4 client.25790 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56410e3b5170 con 0x56410e5502f0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4124ef5640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e5502f0 0x56410e5526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4124ef5640 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e5502f0 0x56410e5526e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de5492458b521f5 server_cookie=b1d4d7c4d1f2f18c in_seq=0 out_seq=0 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.340+0000 7f4108ff9640 10 client.25790 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4108ff9640 1 -- 172.21.15.16:0/2126750170 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56410e3b5170 con 0x56410e5502f0 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4108ff9640 10 client.25790 handle_command_reply: tid=0 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 client.25790 shutdown 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 msgr2=0x56410e4d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f411c000c00 tx=0x7f411c0b7e10 comp rx=0 tx=0).stop 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 shutdown_connections 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56410e5502f0 0x56410e5526e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56410e4d6370 0x56410e4dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4108ff9640 0 client.25790 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56410e4ccb40 0x56410e4de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 --2- 172.21.15.16:0/2126750170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56410e4cb5d0 0x56410e4d0a70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 >> 172.21.15.16:0/2126750170 conn(0x56410e494310 msgr2=0x56410e4c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 shutdown_connections 2024-09-12T02:51:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 1 -- 172.21.15.16:0/2126750170 wait complete. 2024-09-12T02:51:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.341+0000 7f4129d33740 20 client.25790 trim_cache size 0 max 16384 2024-09-12T02:51:10.353 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 msgr2=0x7f410c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 0x7f410c0a5040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4118009fd0 tx=0x7f4118004ab0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 msgr2=0x7f4120100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f411c006010 tx=0x7f411c004040 comp rx=0 tx=0).stop 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 shutdown_connections 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f412010c1a0 0x7f41201014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4120103720 0x7f4120100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f410c0a2b80 0x7f410c0a5040 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 --2- 172.21.15.16:0/1651153424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4120102e00 0x7f4120100a60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 >> 172.21.15.16:0/1651153424 conn(0x7f41200fc7a0 msgr2=0x7f41200fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 shutdown_connections 2024-09-12T02:51:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:10.342+0000 7f412795a640 1 -- 172.21.15.16:0/1651153424 wait complete. 2024-09-12T02:51:10.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 254888 14.96 MB/sec execute 12 sec latency 19.434 ms 2024-09-12T02:51:11.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:51:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:11.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 257605 14.76 MB/sec execute 13 sec latency 14.244 ms 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.918+0000 7f8081879640 1 -- 172.21.15.16:0/2455280857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c100a80 msgr2=0x7f807c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.918+0000 7f8081879640 1 --2- 172.21.15.16:0/2455280857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c100a80 0x7f807c100ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8070009900 tx=0x7f807002a410 comp rx=0 tx=0).stop 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 -- 172.21.15.16:0/2455280857 shutdown_connections 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 --2- 172.21.15.16:0/2455280857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c101420 0x7f807c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 --2- 172.21.15.16:0/2455280857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c100a80 0x7f807c100ee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 --2- 172.21.15.16:0/2455280857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c106a80 0x7f807c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 -- 172.21.15.16:0/2455280857 >> 172.21.15.16:0/2455280857 conn(0x7f807c0fc7a0 msgr2=0x7f807c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 -- 172.21.15.16:0/2455280857 shutdown_connections 2024-09-12T02:51:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 -- 172.21.15.16:0/2455280857 wait complete. 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.919+0000 7f8081879640 1 Processor -- start 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 -- start start 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c101420 0x7f807c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c106a80 0x7f807c1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f807c110ed0 con 0x7f807c100a80 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f807c110d50 con 0x7f807c106a80 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f8081879640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f807c111050 con 0x7f807c101420 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51376/0 (socket says 172.21.15.16:51376) 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 -- 172.21.15.16:0/3744234489 learned_addr learned my addr 172.21.15.16:0/3744234489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807a7fc640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c101420 0x7f807c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c106a80 0x7f807c1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 -- 172.21.15.16:0/3744234489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c101420 msgr2=0x7f807c1997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c101420 0x7f807c1997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 -- 172.21.15.16:0/3744234489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c106a80 msgr2=0x7f807c1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c106a80 0x7f807c1a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 -- 172.21.15.16:0/3744234489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c1a4840 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.920+0000 7f807affd640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9f0 tx=0x7f806c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.921+0000 7f8080877640 1 -- 172.21.15.16:0/3744234489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c00cd80 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.921+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c1a4a10 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.921+0000 7f8080877640 1 -- 172.21.15.16:0/3744234489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f806c010910 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.921+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c1a4f20 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.921+0000 7f8080877640 1 -- 172.21.15.16:0/3744234489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c0188f0 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.923+0000 7f8080877640 1 -- 172.21.15.16:0/3744234489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f806c018bd0 con 0x7f807c100a80 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 Processor -- start 2024-09-12T02:51:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 -- start start 2024-09-12T02:51:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x555628048a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628059b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555627fae2f0 con 0x555628057780 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555627f55cf0 con 0x5556280485d0 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8083c52740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555627faa2b0 con 0x555628049b40 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x555628048a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59152/0 (socket says 172.21.15.16:59152) 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 -- 172.21.15.16:0/590507503 learned_addr learned my addr 172.21.15.16:0/590507503 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:11.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628059b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f8080877640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 0x7f80640a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 -- 172.21.15.16:0/590507503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 msgr2=0x555628048a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x555628048a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 -- 172.21.15.16:0/590507503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 msgr2=0x555628059b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628059b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.924+0000 7f807b7fe640 1 -- 172.21.15.16:0/590507503 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555627f3a3a0 con 0x555628049b40 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f8080877640 1 -- 172.21.15.16:0/3744234489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f806c0cf180 con 0x7f807c100a80 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 0x7f80640a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f807b7fe640 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8074011590 tx=0x7f8074011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cef5e54ba996f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f8061ffb640 1 -- 172.21.15.16:0/590507503 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074017050 con 0x555628049b40 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 0x7f80640a50c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8074022000 tx=0x7f8074023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f8061ffb640 1 -- 172.21.15.16:0/590507503 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f807400fd20 con 0x555628049b40 2024-09-12T02:51:11.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.925+0000 7f8061ffb640 1 -- 172.21.15.16:0/590507503 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807401bba0 con 0x555628049b40 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- 172.21.15.16:0/590507503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 msgr2=0x555628048090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8074011590 tx=0x7f8074011a50 comp rx=0 tx=0).stop 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- 172.21.15.16:0/590507503 shutdown_connections 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628059b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x555628048a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 --2- 172.21.15.16:0/590507503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x555628048090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- 172.21.15.16:0/590507503 >> 172.21.15.16:0/590507503 conn(0x555628011310 msgr2=0x555628044c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- 172.21.15.16:0/590507503 shutdown_connections 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- 172.21.15.16:0/590507503 wait complete. 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.926+0000 7f8083c52740 1 -- start start 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 Processor -- start 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x5556280543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628054900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55562805a480 con 0x555628057780 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555627f69060 con 0x5556280485d0 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55562805a600 con 0x555628049b40 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f8083c52740 10 client.0 _ll_register_callbacks cb 0x555627cbc8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628054900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x5556280543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42050/0 (socket says 172.21.15.16:42050) 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 -- 172.21.15.16:0/3235927859 learned_addr learned my addr 172.21.15.16:0/3235927859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 -- 172.21.15.16:0/3235927859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 msgr2=0x5556280543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x5556280543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 -- 172.21.15.16:0/3235927859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 msgr2=0x555628054900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628054900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.927+0000 7f807affd640 1 -- 172.21.15.16:0/3235927859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555627d1f3c0 con 0x5556280485d0 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f807affd640 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9c0 tx=0x7f806c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1dd96c9ce3abe9f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c09b5e0 con 0x5556280485d0 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f806c09b0f0 con 0x5556280485d0 2024-09-12T02:51:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f807400fd20 con 0x5556280485d0 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f806c09b400 con 0x5556280485d0 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 10 client.25802 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555628065ef0 con 0x5556280485d0 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f807400fd20 con 0x5556280485d0 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f806c097980 con 0x5556280485d0 2024-09-12T02:51:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 10 client.25802 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 10 client.25802 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 20 client.25802 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280673b0 0x555628069800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 4 client.25802 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.928+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555627fbb860 con 0x5556280673b0 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.929+0000 7f807a7fc640 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280673b0 0x555628069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.929+0000 7f807a7fc640 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280673b0 0x555628069800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8c516e9e3b5bc12 server_cookie=3f27d871a547ab09 in_seq=0 out_seq=0 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.929+0000 7f805affd640 10 client.25802 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f805affd640 1 -- 172.21.15.16:0/3235927859 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555627fbb860 con 0x5556280673b0 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f805affd640 10 client.25802 handle_command_reply: tid=0 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 client.25802 shutdown 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 msgr2=0x5556280570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9c0 tx=0x7f806c009310 comp rx=0 tx=0).stop 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 shutdown_connections 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280673b0 0x555628069800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628054900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628049b40 0x5556280543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 --2- 172.21.15.16:0/3235927859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556280485d0 0x5556280570f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 >> 172.21.15.16:0/3235927859 conn(0x555628011310 msgr2=0x55562804a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f805affd640 0 client.25802 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 shutdown_connections 2024-09-12T02:51:11.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 1 -- 172.21.15.16:0/3235927859 wait complete. 2024-09-12T02:51:11.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.930+0000 7f8083c52740 20 client.25802 trim_cache size 0 max 16384 2024-09-12T02:51:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 Processor -- start 2024-09-12T02:51:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 -- start start 2024-09-12T02:51:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 0x555628063780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628057780 0x55562806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f806800e360 con 0x5556280485d0 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f806800fed0 con 0x555628049b40 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.943+0000 7f8083c52740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f806800ea30 con 0x555628057780 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 0x555628063780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628057780 0x55562806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42066/0 (socket says 172.21.15.16:42066) 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807a7fc640 1 -- 172.21.15.16:0/403556226 learned_addr learned my addr 172.21.15.16:0/403556226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 -- 172.21.15.16:0/403556226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628057780 msgr2=0x55562806f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628057780 0x55562806f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 -- 172.21.15.16:0/403556226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 msgr2=0x555628053c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 -- 172.21.15.16:0/403556226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555628054c70 con 0x5556280485d0 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807a7fc640 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f807affd640 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 0x555628063780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f806c09ba30 tx=0x7f806c0d2cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f81804e4028c24 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f80617fa640 1 -- 172.21.15.16:0/403556226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c09b5e0 con 0x5556280485d0 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f80617fa640 1 -- 172.21.15.16:0/403556226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f806c09fda0 con 0x5556280485d0 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.944+0000 7f80617fa640 1 -- 172.21.15.16:0/403556226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c09b110 con 0x5556280485d0 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- 172.21.15.16:0/403556226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 msgr2=0x555628063780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 0x555628063780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f806c09ba30 tx=0x7f806c0d2cd0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- 172.21.15.16:0/403556226 shutdown_connections 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555628057780 0x55562806f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x555628053c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 --2- 172.21.15.16:0/403556226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556280485d0 0x555628063780 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- 172.21.15.16:0/403556226 >> 172.21.15.16:0/403556226 conn(0x555628011310 msgr2=0x555628052c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- 172.21.15.16:0/403556226 shutdown_connections 2024-09-12T02:51:11.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- 172.21.15.16:0/403556226 wait complete. 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.945+0000 7f8083c52740 1 -- start start 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 Processor -- start 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628051d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f806800f7d0 con 0x555628057780 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f806800f100 con 0x555628049b40 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f806800c340 con 0x5556280485d0 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f8083c52740 10 client.0 _ll_register_callbacks cb 0x7f806c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628051d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42080/0 (socket says 172.21.15.16:42080) 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807a7fc640 1 -- 172.21.15.16:0/3467087499 learned_addr learned my addr 172.21.15.16:0/3467087499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.946+0000 7f807affd640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59172/0 (socket says 172.21.15.16:59172) 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 -- 172.21.15.16:0/3467087499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 msgr2=0x555628051280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 -- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 msgr2=0x5556280517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 -- 172.21.15.16:0/3467087499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556280d10a0 con 0x555628057780 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807affd640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807a7fc640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f807b7fe640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628051d00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8074005fd0 tx=0x7f807401b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1742e8837652da06 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f805affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074036060 con 0x555628057780 2024-09-12T02:51:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f806c09fda0 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8074004df0 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8074017020 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f8083c52740 10 client.16230 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.947+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5556280d2280 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f806c09fda0 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f807401a070 con 0x555628057780 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 10 client.16230 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 10 client.16230 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 20 client.16230 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280d2860 0x5556280d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 4 client.16230 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555627fb5d70 con 0x5556280d2860 2024-09-12T02:51:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f807affd640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280d2860 0x5556280d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f807affd640 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280d2860 0x5556280d4c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b8944d573e8ed2d server_cookie=9821400cfd659259 in_seq=0 out_seq=0 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.948+0000 7f805affd640 10 client.16230 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f805affd640 1 -- 172.21.15.16:0/3467087499 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555627fb5d70 con 0x5556280d2860 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f805affd640 10 client.16230 handle_command_reply: tid=0 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 client.16230 shutdown 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 msgr2=0x555628051d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628051d00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8074005fd0 tx=0x7f807401b6e0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 shutdown_connections 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556280d2860 0x5556280d4c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555628057780 0x555628051d00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555628049b40 0x5556280517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 --2- 172.21.15.16:0/3467087499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556280485d0 0x555628051280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f805affd640 0 client.16230 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 >> 172.21.15.16:0/3467087499 conn(0x555628011310 msgr2=0x555628052bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 shutdown_connections 2024-09-12T02:51:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 1 -- 172.21.15.16:0/3467087499 wait complete. 2024-09-12T02:51:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.949+0000 7f8083c52740 20 client.16230 trim_cache size 0 max 16384 2024-09-12T02:51:11.957 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 msgr2=0x7f80640a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 0x7f80640a50c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8074022000 tx=0x7f8074023000 comp rx=0 tx=0).stop 2024-09-12T02:51:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 msgr2=0x7f807c199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9f0 tx=0x7f806c00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 shutdown_connections 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f807c106a80 0x7f807c1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f807c101420 0x7f807c1997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80640a2c00 0x7f80640a50c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 --2- 172.21.15.16:0/3744234489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c100a80 0x7f807c199280 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 >> 172.21.15.16:0/3744234489 conn(0x7f807c0fc7a0 msgr2=0x7f807c0fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 shutdown_connections 2024-09-12T02:51:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:11.950+0000 7f8081879640 1 -- 172.21.15.16:0/3744234489 wait complete. 2024-09-12T02:51:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:11 smithi016 ceph-mon[30925]: pgmap v435: 97 pgs: 97 active+clean; 111 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2024-09-12T02:51:12.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:11 smithi033 ceph-mon[35092]: pgmap v435: 97 pgs: 97 active+clean; 111 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2024-09-12T02:51:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:11 smithi184 ceph-mon[35321]: pgmap v435: 97 pgs: 97 active+clean; 111 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2024-09-12T02:51:12.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 259913 14.72 MB/sec execute 14 sec latency 16.439 ms 2024-09-12T02:51:12.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:51:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.539+0000 7f9bae3ab640 1 -- 172.21.15.16:0/3737068870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba8102d30 msgr2=0x7f9ba8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.539+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/3737068870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba8102d30 0x7f9ba8103110 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009930 tx=0x7f9b9802a420 comp rx=0 tx=0).stop 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 -- 172.21.15.16:0/3737068870 shutdown_connections 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/3737068870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 0x7f9ba810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/3737068870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba81036e0 0x7f9ba8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/3737068870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba8102d30 0x7f9ba8103110 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 -- 172.21.15.16:0/3737068870 >> 172.21.15.16:0/3737068870 conn(0x7f9ba80fc7d0 msgr2=0x7f9ba80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 -- 172.21.15.16:0/3737068870 shutdown_connections 2024-09-12T02:51:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 -- 172.21.15.16:0/3737068870 wait complete. 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 Processor -- start 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.540+0000 7f9bae3ab640 1 -- start start 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 0x7f9ba8100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 0x7f9ba8101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81120a0 con 0x7f9ba81036e0 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8111f20 con 0x7f9ba8102d30 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bae3ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8112220 con 0x7f9ba81082a0 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51424/0 (socket says 172.21.15.16:51424) 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 -- 172.21.15.16:0/2774852715 learned_addr learned my addr 172.21.15.16:0/2774852715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 0x7f9ba8100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9bac948640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 0x7f9ba8101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 -- 172.21.15.16:0/2774852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 msgr2=0x7f9ba8101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 0x7f9ba8101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 -- 172.21.15.16:0/2774852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 msgr2=0x7f9ba8100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 0x7f9ba8100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 -- 172.21.15.16:0/2774852715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba81a6a80 con 0x7f9ba81036e0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 0x7f9ba8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.541+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000e970 tx=0x7f9b9000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.542+0000 7f9ba5ffb640 1 -- 172.21.15.16:0/2774852715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9000cd80 con 0x7f9ba81036e0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.542+0000 7f9ba5ffb640 1 -- 172.21.15.16:0/2774852715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b900035f0 con 0x7f9ba81036e0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.542+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba81a6d70 con 0x7f9ba81036e0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.542+0000 7f9ba5ffb640 1 -- 172.21.15.16:0/2774852715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90010690 con 0x7f9ba81036e0 2024-09-12T02:51:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.542+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a7250 con 0x7f9ba81036e0 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.544+0000 7f9ba5ffb640 1 -- 172.21.15.16:0/2774852715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9b900037a0 con 0x7f9ba81036e0 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.544+0000 7f9bb0784740 1 Processor -- start 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.544+0000 7f9bb0784740 1 -- start start 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92aea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e92a432f0 con 0x559e92adeb40 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e929eacf0 con 0x559e92add5d0 2024-09-12T02:51:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bb0784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559e92a3f2b0 con 0x559e92ae8370 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba5ffb640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 0x7f9b940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bac948640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9b9ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92aea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42102/0 (socket says 172.21.15.16:42102) 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 -- 172.21.15.16:0/2986076257 learned_addr learned my addr 172.21.15.16:0/2986076257 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bac948640 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51432/0 (socket says 172.21.15.16:51432) 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bac948640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 0x7f9b940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba5ffb640 1 -- 172.21.15.16:0/2774852715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b900cf270 con 0x7f9ba81036e0 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 -- 172.21.15.16:0/2986076257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 msgr2=0x559e92aea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92aea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 -- 172.21.15.16:0/2986076257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 msgr2=0x559e92add090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9ba7fff640 1 -- 172.21.15.16:0/2986076257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e929cf3a0 con 0x559e92add5d0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.545+0000 7f9bac948640 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.546+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000c00 tx=0x7f9b98005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7599fbe4ca6ac012 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.546+0000 7f9bac948640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 0x7f9b940a5040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0005fd0 tx=0x7f9ba0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.546+0000 7f9b9e7fc640 1 -- 172.21.15.16:0/2986076257 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98035040 con 0x559e92add5d0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.546+0000 7f9b9e7fc640 1 -- 172.21.15.16:0/2986076257 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98031830 con 0x559e92add5d0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.546+0000 7f9b9e7fc640 1 -- 172.21.15.16:0/2986076257 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98003d40 con 0x559e92add5d0 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- 172.21.15.16:0/2986076257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 msgr2=0x559e92adda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000c00 tx=0x7f9b98005d50 comp rx=0 tx=0).stop 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- 172.21.15.16:0/2986076257 shutdown_connections 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92aea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92adda10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- 172.21.15.16:0/2986076257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92add090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- 172.21.15.16:0/2986076257 >> 172.21.15.16:0/2986076257 conn(0x559e92aa6310 msgr2=0x559e92ad9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- 172.21.15.16:0/2986076257 shutdown_connections 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- 172.21.15.16:0/2986076257 wait complete. 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- start start 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 Processor -- start 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92ae4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92ae5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e92aeb010 con 0x559e92adeb40 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e929fe060 con 0x559e92add5d0 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.547+0000 7f9bb0784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559e92aeb190 con 0x559e92ae8370 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9bb0784740 10 client.0 _ll_register_callbacks cb 0x559e927518c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9b9ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92ae4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42114/0 (socket says 172.21.15.16:42114) 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 -- 172.21.15.16:0/3631286042 learned_addr learned my addr 172.21.15.16:0/3631286042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9bac948640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92ae5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 -- 172.21.15.16:0/3631286042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 msgr2=0x559e92ae5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92ae5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 -- 172.21.15.16:0/3631286042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 msgr2=0x559e92ae4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92ae4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 -- 172.21.15.16:0/3631286042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e927b43c0 con 0x559e92add5d0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92ae4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9b9802a940 tx=0x7f9b980038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38e771bee21c5bf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9b877fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9804e070 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b9803c650 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.548+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b98031830 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b9805a050 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 10 client.25820 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559e92af6a40 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98031830 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f9b98003fe0 con 0x559e92add5d0 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 10 client.25820 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 10 client.25820 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 20 client.25820 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92af7d90 0x559e92afa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 4 client.25820 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e92a50860 con 0x559e92af7d90 2024-09-12T02:51:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.549+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92af7d90 0x559e92afa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.550+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92af7d90 0x559e92afa180 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a96131bdb67773 server_cookie=8f4bf4864a17e0e9 in_seq=0 out_seq=0 2024-09-12T02:51:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.550+0000 7f9b877fe640 10 client.25820 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.550+0000 7f9b877fe640 1 -- 172.21.15.16:0/3631286042 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e92a50860 con 0x559e92af7d90 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.550+0000 7f9b877fe640 10 client.25820 handle_command_reply: tid=0 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.550+0000 7f9bb0784740 1 client.25820 shutdown 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 msgr2=0x559e92ae7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9b9802a940 tx=0x7f9b980038a0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 shutdown_connections 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92af7d90 0x559e92afa180 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92ae5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92adeb40 0x559e92ae4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3631286042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92add5d0 0x559e92ae7d00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9b877fe640 0 client.25820 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 >> 172.21.15.16:0/3631286042 conn(0x559e92aa6310 msgr2=0x559e92adf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 shutdown_connections 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 1 -- 172.21.15.16:0/3631286042 wait complete. 2024-09-12T02:51:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.551+0000 7f9bb0784740 20 client.25820 trim_cache size 0 max 16384 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.564+0000 7f9bb0784740 1 Processor -- start 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.564+0000 7f9bb0784740 1 -- start start 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92adeb40 0x559e92ae6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92ae8370 0x559e92ae7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81af860 con 0x559e92add5d0 2024-09-12T02:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b13d0 con 0x559e92ae8370 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bb0784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81aff30 con 0x559e92adeb40 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51436/0 (socket says 172.21.15.16:51436) 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 -- 172.21.15.16:0/1032769973 learned_addr learned my addr 172.21.15.16:0/1032769973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9bac948640 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92ae8370 0x559e92ae7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92adeb40 0x559e92ae6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 -- 172.21.15.16:0/1032769973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92adeb40 msgr2=0x559e92ae6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92adeb40 0x559e92ae6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 -- 172.21.15.16:0/1032769973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92ae8370 msgr2=0x559e92ae7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92ae8370 0x559e92ae7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 -- 172.21.15.16:0/1032769973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e92ae58a0 con 0x559e92add5d0 2024-09-12T02:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9b980057c0 tx=0x7f9b98005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42a40aa3350b6c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.565+0000 7f9b9dffb640 1 -- 172.21.15.16:0/1032769973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98057020 con 0x559e92add5d0 2024-09-12T02:51:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9b9dffb640 1 -- 172.21.15.16:0/1032769973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98031870 con 0x559e92add5d0 2024-09-12T02:51:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9b9dffb640 1 -- 172.21.15.16:0/1032769973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9804e070 con 0x559e92add5d0 2024-09-12T02:51:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 -- 172.21.15.16:0/1032769973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 msgr2=0x559e92aea760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9b980057c0 tx=0x7f9b98005d50 comp rx=0 tx=0).stop 2024-09-12T02:51:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 -- 172.21.15.16:0/1032769973 shutdown_connections 2024-09-12T02:51:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92ae8370 0x559e92ae7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92adeb40 0x559e92ae6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 --2- 172.21.15.16:0/1032769973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92aea760 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 -- 172.21.15.16:0/1032769973 >> 172.21.15.16:0/1032769973 conn(0x559e92aa6310 msgr2=0x559e92ae6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 -- 172.21.15.16:0/1032769973 shutdown_connections 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.566+0000 7f9bb0784740 1 -- 172.21.15.16:0/1032769973 wait complete. 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 -- start start 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 Processor -- start 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92afa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 0x559e92afad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92b5a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b0cd0 con 0x559e92add5d0 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b0600 con 0x559e92adeb40 2024-09-12T02:51:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81ad860 con 0x559e92ae8370 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bb0784740 10 client.0 _ll_register_callbacks cb 0x7f9b98040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92afa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9bac948640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92b5a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9b9ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 0x559e92afad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92afa800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51446/0 (socket says 172.21.15.16:51446) 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.567+0000 7f9ba7fff640 1 -- 172.21.15.16:0/3058004170 learned_addr learned my addr 172.21.15.16:0/3058004170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 -- 172.21.15.16:0/3058004170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 msgr2=0x559e92b5a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92b5a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 -- 172.21.15.16:0/3058004170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 msgr2=0x559e92afa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92afa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 -- 172.21.15.16:0/3058004170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e92b5acd0 con 0x559e92adeb40 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9bac948640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92b5a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b9ffff640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 0x559e92afad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9b900ccd50 tx=0x7f9b9009bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b2cc6dae512dd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b877fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90003b10 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b900a42c0 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b98031870 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9b900a44c0 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9bb0784740 10 client.25826 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.568+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559e92b62060 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98031870 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f9b900b4ce0 con 0x559e92adeb40 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 10 client.25826 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 10 client.25826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 20 client.25826 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92b62730 0x559e92b64b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 4 client.25826 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e92a4ad70 con 0x559e92b62730 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92b62730 0x559e92b64b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9b877fe640 10 client.25826 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9ba7fff640 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92b62730 0x559e92b64b50 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da10563364f425f server_cookie=532aa8c55eb3357f in_seq=0 out_seq=0 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9b877fe640 1 -- 172.21.15.16:0/3058004170 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e92a4ad70 con 0x559e92b62730 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.569+0000 7f9b877fe640 10 client.25826 handle_command_reply: tid=0 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 client.25826 shutdown 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 msgr2=0x559e92afad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 0x559e92afad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9b900ccd50 tx=0x7f9b9009bbb0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 shutdown_connections 2024-09-12T02:51:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559e92b62730 0x559e92b64b50 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559e92ae8370 0x559e92b5a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e92adeb40 0x559e92afad40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9b877fe640 0 client.25826 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 --2- 172.21.15.16:0/3058004170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e92add5d0 0x559e92afa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 >> 172.21.15.16:0/3058004170 conn(0x559e92aa6310 msgr2=0x559e92ae7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 shutdown_connections 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 1 -- 172.21.15.16:0/3058004170 wait complete. 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bb0784740 20 client.25826 trim_cache size 0 max 16384 2024-09-12T02:51:13.576 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 msgr2=0x7f9b940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 0x7f9b940a5040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0005fd0 tx=0x7f9ba0009500 comp rx=0 tx=0).stop 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 msgr2=0x7f9ba8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000e970 tx=0x7f9b9000ee30 comp rx=0 tx=0).stop 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 shutdown_connections 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba81082a0 0x7f9ba8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b940a2b80 0x7f9b940a5040 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba81036e0 0x7f9ba8100ed0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.570+0000 7f9bae3ab640 1 --2- 172.21.15.16:0/2774852715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8102d30 0x7f9ba8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.571+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 >> 172.21.15.16:0/2774852715 conn(0x7f9ba80fc7d0 msgr2=0x7f9ba8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.571+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 shutdown_connections 2024-09-12T02:51:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:13.571+0000 7f9bae3ab640 1 -- 172.21.15.16:0/2774852715 wait complete. 2024-09-12T02:51:13.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262903 14.96 MB/sec execute 15 sec latency 27.223 ms 2024-09-12T02:51:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:13 smithi016 ceph-mon[30925]: pgmap v436: 97 pgs: 97 active+clean; 112 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.5 MiB/s wr, 202 op/s 2024-09-12T02:51:14.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:13 smithi033 ceph-mon[35092]: pgmap v436: 97 pgs: 97 active+clean; 112 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.5 MiB/s wr, 202 op/s 2024-09-12T02:51:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:13 smithi184 ceph-mon[35321]: pgmap v436: 97 pgs: 97 active+clean; 112 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.5 MiB/s wr, 202 op/s 2024-09-12T02:51:14.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266089 15.11 MB/sec execute 16 sec latency 11.946 ms 2024-09-12T02:51:15.370 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:51:15.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:51:15.659 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:15.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 268934 14.98 MB/sec execute 17 sec latency 11.788 ms 2024-09-12T02:51:16.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:15 smithi016 ceph-mon[30925]: pgmap v437: 97 pgs: 97 active+clean; 116 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 -- 172.21.15.16:0/2480649868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 msgr2=0x7f22581029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 --2- 172.21.15.16:0/2480649868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f22581029b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2248009900 tx=0x7f224802a480 comp rx=0 tx=0).stop 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 -- 172.21.15.16:0/2480649868 shutdown_connections 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 --2- 172.21.15.16:0/2480649868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258102ef0 0x7f225810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 --2- 172.21.15.16:0/2480649868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f22581029b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 --2- 172.21.15.16:0/2480649868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258100e20 0x7f2258101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 -- 172.21.15.16:0/2480649868 >> 172.21.15.16:0/2480649868 conn(0x7f22580fc7d0 msgr2=0x7f22580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 -- 172.21.15.16:0/2480649868 shutdown_connections 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.172+0000 7f225fe50640 1 -- 172.21.15.16:0/2480649868 wait complete. 2024-09-12T02:51:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 Processor -- start 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 -- start start 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 0x7f22581984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f2258198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258102ef0 0x7f225819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2258110fa0 con 0x7f2258102570 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2258110e20 con 0x7f2258100e20 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225fe50640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2258111120 con 0x7f2258102ef0 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f2258198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225dbec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 0x7f22581984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f2258198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42044/0 (socket says 172.21.15.16:42044) 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225d3eb640 1 -- 172.21.15.16:0/1065990875 learned_addr learned my addr 172.21.15.16:0/1065990875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.173+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258102ef0 0x7f225819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 -- 172.21.15.16:0/1065990875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258102ef0 msgr2=0x7f225819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258102ef0 0x7f225819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 -- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 msgr2=0x7f2258198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f2258198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 -- 172.21.15.16:0/1065990875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225819d490 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225dbec640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 0x7f22581984e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f225400e9f0 tx=0x7f225400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f224effd640 1 -- 172.21.15.16:0/1065990875 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225400ccc0 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f224effd640 1 -- 172.21.15.16:0/1065990875 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2254010910 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225819d780 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f224effd640 1 -- 172.21.15.16:0/1065990875 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254018960 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.174+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22581a5060 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.177+0000 7f224effd640 1 -- 172.21.15.16:0/1065990875 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2254010ac0 con 0x7f2258100e20 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.177+0000 7f2262229740 1 Processor -- start 2024-09-12T02:51:16.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.177+0000 7f2262229740 1 -- start start 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 0x5652e15f7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15f7d40 0x5652e15fa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5652e1552280 con 0x5652e15ed480 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f224effd640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 0x7f22440a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5652e14f9b10 con 0x5652e15ecb60 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f2262229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5652e154e240 con 0x5652e15f7d40 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40370/0 (socket says 172.21.15.16:40370) 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 -- 172.21.15.16:0/3692775547 learned_addr learned my addr 172.21.15.16:0/3692775547 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 0x7f22440a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225dbec640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 0x5652e15f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15f7d40 0x5652e15fa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f224effd640 1 -- 172.21.15.16:0/1065990875 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22540cf320 con 0x7f2258100e20 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 -- 172.21.15.16:0/3692775547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15f7d40 msgr2=0x5652e15fa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15f7d40 0x5652e15fa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 -- 172.21.15.16:0/3692775547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 msgr2=0x5652e15f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 0x5652e15f7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225e3ed640 1 -- 172.21.15.16:0/3692775547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e15061f0 con 0x5652e15ecb60 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.178+0000 7f225dbec640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 0x5652e15f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.179+0000 7f225e3ed640 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2250011e00 tx=0x7f225000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68232c8c1a1ff71 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.179+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 0x7f22440a50c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f225000fc90 tx=0x7f2250018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.179+0000 7f2243fff640 1 -- 172.21.15.16:0/3692775547 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250022070 con 0x5652e15ecb60 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.179+0000 7f2243fff640 1 -- 172.21.15.16:0/3692775547 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f225000ed10 con 0x5652e15ecb60 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.179+0000 7f2243fff640 1 -- 172.21.15.16:0/3692775547 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22500058b0 con 0x5652e15ecb60 2024-09-12T02:51:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- 172.21.15.16:0/3692775547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 msgr2=0x5652e15ecf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2250011e00 tx=0x7f225000f790 comp rx=0 tx=0).stop 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- 172.21.15.16:0/3692775547 shutdown_connections 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15f7d40 0x5652e15fa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ed480 0x5652e15f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 --2- 172.21.15.16:0/3692775547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15ecf40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- 172.21.15.16:0/3692775547 >> 172.21.15.16:0/3692775547 conn(0x5652e15e90f0 msgr2=0x5652e15e94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- 172.21.15.16:0/3692775547 shutdown_connections 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- 172.21.15.16:0/3692775547 wait complete. 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 -- start start 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.180+0000 7f2262229740 1 Processor -- start 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5652e15fa910 con 0x5652e15f7d40 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5652e150cf10 con 0x5652e15ecb60 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5652e15faa90 con 0x5652e15ed480 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f2262229740 10 client.0 _ll_register_callbacks cb 0x5652e14d5a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225dbec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225dbec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40374/0 (socket says 172.21.15.16:40374) 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54964/0 (socket says 172.21.15.16:54964) 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225dbec640 1 -- 172.21.15.16:0/1970555060 learned_addr learned my addr 172.21.15.16:0/1970555060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42050/0 (socket says 172.21.15.16:42050) 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 -- 172.21.15.16:0/1970555060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 msgr2=0x5652e15f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 -- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 msgr2=0x5652e15f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 -- 172.21.15.16:0/1970555060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e12c36a0 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225d3eb640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225dbec640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f225e3ed640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2250011e00 tx=0x7f2250011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283a3a93eccf5758 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.181+0000 7f22417fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225002e060 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2250003b70 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f225000ed10 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2250022040 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 10 client.16260 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5652e1606570 con 0x5652e15f7d40 2024-09-12T02:51:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f225000ed10 con 0x5652e15f7d40 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f225001e8e0 con 0x5652e15f7d40 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 10 client.16260 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 10 client.16260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 20 client.16260 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1609dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 4 client.16260 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.182+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652e155f7f0 con 0x5652e16079d0 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.183+0000 7f225dbec640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1609dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.183+0000 7f225dbec640 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1609dc0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=335b7dc529cee1ec server_cookie=5e02f0fa2e2b3ef1 in_seq=0 out_seq=0 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.183+0000 7f22417fa640 10 client.16260 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f22417fa640 1 -- 172.21.15.16:0/1970555060 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652e155f7f0 con 0x5652e16079d0 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f22417fa640 10 client.16260 handle_command_reply: tid=0 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 client.16260 shutdown 2024-09-12T02:51:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 msgr2=0x5652e15f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2250011e00 tx=0x7f2250011f50 comp rx=0 tx=0).stop 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 shutdown_connections 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1609dc0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f22417fa640 0 client.16260 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e15f5700 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 --2- 172.21.15.16:0/1970555060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ecb60 0x5652e15f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 >> 172.21.15.16:0/1970555060 conn(0x5652e15e90f0 msgr2=0x5652e15e8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 shutdown_connections 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 1 -- 172.21.15.16:0/1970555060 wait complete. 2024-09-12T02:51:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.184+0000 7f2262229740 20 client.16260 trim_cache size 0 max 16384 2024-09-12T02:51:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 Processor -- start 2024-09-12T02:51:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 -- start start 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 0x5652e15ef5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 0x5652e15f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e160f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22581ab3c0 con 0x5652e15f7d40 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22581acf30 con 0x5652e15ed480 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.204+0000 7f2262229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22581aba90 con 0x5652e15ecb60 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225dbec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 0x5652e15ef5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 0x5652e15f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225dbec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 0x5652e15ef5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54978/0 (socket says 172.21.15.16:54978) 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225dbec640 1 -- 172.21.15.16:0/3083525647 learned_addr learned my addr 172.21.15.16:0/3083525647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 -- 172.21.15.16:0/3083525647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 msgr2=0x5652e15ef5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 0x5652e15ef5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 -- 172.21.15.16:0/3083525647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 msgr2=0x5652e160f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e160f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 -- 172.21.15.16:0/3083525647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e15f3c30 con 0x5652e15ed480 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 0x5652e15f6ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22580fff50 tx=0x7f2248004180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e313d3bbc917e1c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f22437fe640 1 -- 172.21.15.16:0/3083525647 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248035040 con 0x5652e15ed480 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f22437fe640 1 -- 172.21.15.16:0/3083525647 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2248005500 con 0x5652e15ed480 2024-09-12T02:51:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.205+0000 7f22437fe640 1 -- 172.21.15.16:0/3083525647 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248003850 con 0x5652e15ed480 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 -- 172.21.15.16:0/3083525647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 msgr2=0x5652e15f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 0x5652e15f6ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22580fff50 tx=0x7f2248004180 comp rx=0 tx=0).stop 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 -- 172.21.15.16:0/3083525647 shutdown_connections 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15f7d40 0x5652e160f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15ed480 0x5652e15f6ae0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 --2- 172.21.15.16:0/3083525647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ecb60 0x5652e15ef5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 -- 172.21.15.16:0/3083525647 >> 172.21.15.16:0/3083525647 conn(0x5652e15e90f0 msgr2=0x5652e15f63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 -- 172.21.15.16:0/3083525647 shutdown_connections 2024-09-12T02:51:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.206+0000 7f2262229740 1 -- 172.21.15.16:0/3083525647 wait complete. 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 -- start start 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 Processor -- start 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 0x5652e16002a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15f7d40 0x5652e15fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22581ac830 con 0x5652e15ecb60 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22581ac160 con 0x5652e15f7d40 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22581a93c0 con 0x5652e15ed480 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f2262229740 10 client.0 _ll_register_callbacks cb 0x7f225400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f225dbec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 0x5652e16002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f225d3eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54990/0 (socket says 172.21.15.16:54990) 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.207+0000 7f225d3eb640 1 -- 172.21.15.16:0/3193034711 learned_addr learned my addr 172.21.15.16:0/3193034711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225e3ed640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15f7d40 0x5652e15fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 -- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15f7d40 msgr2=0x5652e15fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15f7d40 0x5652e15fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 -- 172.21.15.16:0/3193034711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 msgr2=0x5652e16002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 0x5652e16002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 -- 172.21.15.16:0/3193034711 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e15e4830 con 0x5652e15ed480 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225dbec640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 0x5652e16002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f225d3eb640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2248000c00 tx=0x7f2248004f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bea87f6955f10b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f22417fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248035040 con 0x5652e15ed480 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2248005500 con 0x5652e15ed480 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22480035f0 con 0x5652e15ed480 2024-09-12T02:51:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2248009520 con 0x5652e15ed480 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f2262229740 10 client.25738 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.208+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5652e1672240 con 0x5652e15ed480 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2248005500 con 0x5652e15ed480 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f2248045030 con 0x5652e15ed480 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 10 client.25738 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 20 client.25738 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:16.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1674960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 4 client.25738 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5652e1559d00 con 0x5652e16079d0 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.209+0000 7f225dbec640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1674960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.210+0000 7f22417fa640 10 client.25738 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.210+0000 7f225dbec640 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1674960 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46de4ff83f3054fa server_cookie=e5f087e70d163bc5 in_seq=0 out_seq=0 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.210+0000 7f22417fa640 1 -- 172.21.15.16:0/3193034711 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5652e1559d00 con 0x5652e16079d0 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.210+0000 7f22417fa640 10 client.25738 handle_command_reply: tid=0 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.210+0000 7f2262229740 1 client.25738 shutdown 2024-09-12T02:51:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 msgr2=0x5652e15fb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2248000c00 tx=0x7f2248004f90 comp rx=0 tx=0).stop 2024-09-12T02:51:16.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 shutdown_connections 2024-09-12T02:51:16.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652e16079d0 0x5652e1674960 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652e15f7d40 0x5652e15fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f22417fa640 0 client.25738 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652e15ed480 0x5652e15fb5d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 --2- 172.21.15.16:0/3193034711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652e15ecb60 0x5652e16002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 >> 172.21.15.16:0/3193034711 conn(0x5652e15e90f0 msgr2=0x5652e15e8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 shutdown_connections 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 1 -- 172.21.15.16:0/3193034711 wait complete. 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.211+0000 7f2262229740 20 client.25738 trim_cache size 0 max 16384 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "788816f2-0e53-42b8-89ba-4a0d4b48cb0b", 2024-09-12T02:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 msgr2=0x7f22440a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 0x7f22440a50c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f225000fc90 tx=0x7f2250018000 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 msgr2=0x7f22581984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 0x7f22581984e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f225400e9f0 tx=0x7f225400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 shutdown_connections 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2258102ef0 0x7f225819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22440a2c00 0x7f22440a50c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258102570 0x7f2258198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 --2- 172.21.15.16:0/1065990875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2258100e20 0x7f22581984e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 >> 172.21.15.16:0/1065990875 conn(0x7f22580fc7d0 msgr2=0x7f225806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 shutdown_connections 2024-09-12T02:51:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:16.212+0000 7f225fe50640 1 -- 172.21.15.16:0/1065990875 wait complete. 2024-09-12T02:51:16.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:15 smithi033 ceph-mon[35092]: pgmap v437: 97 pgs: 97 active+clean; 116 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-09-12T02:51:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:15 smithi184 ceph-mon[35321]: pgmap v437: 97 pgs: 97 active+clean; 116 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-09-12T02:51:16.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 271261 14.88 MB/sec execute 18 sec latency 16.309 ms 2024-09-12T02:51:17.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.148 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:16 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:51:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:51:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:51:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:16 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:51:17.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:51:17.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:51:17.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:16 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:51:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:17.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273748 14.83 MB/sec execute 19 sec latency 51.225 ms 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 -- 172.21.15.16:0/3907468220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 msgr2=0x7f8e30110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3907468220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e30110900 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009ca0 tx=0x7f8e2802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 -- 172.21.15.16:0/3907468220 shutdown_connections 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3907468220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e30110900 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3907468220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e30101380 0x7f8e3010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3907468220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e301009d0 0x7f8e30100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 -- 172.21.15.16:0/3907468220 >> 172.21.15.16:0/3907468220 conn(0x7f8e300fc7d0 msgr2=0x7f8e300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 -- 172.21.15.16:0/3907468220 shutdown_connections 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 -- 172.21.15.16:0/3907468220 wait complete. 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.762+0000 7f8e353a5640 1 Processor -- start 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 -- start start 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101380 0x7f8e3019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e3019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e30114250 con 0x7f8e3010e4f0 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301140d0 con 0x7f8e301009d0 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e353a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301143d0 con 0x7f8e30101380 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e3019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40400/0 (socket says 172.21.15.16:40400) 2024-09-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 -- 172.21.15.16:0/3782746204 learned_addr learned my addr 172.21.15.16:0/3782746204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101380 0x7f8e3019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 -- 172.21.15.16:0/3782746204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101380 msgr2=0x7f8e3019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101380 0x7f8e3019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 -- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 msgr2=0x7f8e3019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e3019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2effd640 1 -- 172.21.15.16:0/3782746204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3019f690 con 0x7f8e301009d0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.763+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e3019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e2effd640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000e9f0 tx=0x7f8e2000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e17fff640 1 -- 172.21.15.16:0/3782746204 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2000cd80 con 0x7f8e301009d0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3019f980 con 0x7f8e301009d0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e17fff640 1 -- 172.21.15.16:0/3782746204 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e20005500 con 0x7f8e301009d0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e301a7260 con 0x7f8e301009d0 2024-09-12T02:51:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.764+0000 7f8e17fff640 1 -- 172.21.15.16:0/3782746204 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20010690 con 0x7f8e301009d0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.766+0000 7f8e17fff640 1 -- 172.21.15.16:0/3782746204 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8e200046c0 con 0x7f8e301009d0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 Processor -- start 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 -- start start 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e17fff640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 0x7f8e180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b006b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b006da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5564affc62e0 con 0x5564b00604a0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564aff6dce0 con 0x5564b0060dc0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 0x7f8e180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e3777e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564affc22a0 con 0x5564b006b680 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e17fff640 1 -- 172.21.15.16:0/3782746204 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e200cf470 con 0x7f8e301009d0 2024-09-12T02:51:17.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42090/0 (socket says 172.21.15.16:42090) 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 -- 172.21.15.16:0/2448449922 learned_addr learned my addr 172.21.15.16:0/2448449922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b006da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 -- 172.21.15.16:0/2448449922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 msgr2=0x5564b006da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b006da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 -- 172.21.15.16:0/2448449922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 msgr2=0x5564b006b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b006b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2effd640 1 -- 172.21.15.16:0/2448449922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564aff52440 con 0x5564b00604a0 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b006da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.767+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b006b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:17.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.768+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8e20098650 tx=0x7f8e2009b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7aec34e06792ebe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.768+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 0x7f8e180a5040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005e30 tx=0x7f8e28005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.768+0000 7f8e157fa640 1 -- 172.21.15.16:0/2448449922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2009bc80 con 0x5564b00604a0 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.768+0000 7f8e157fa640 1 -- 172.21.15.16:0/2448449922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e200a0030 con 0x5564b00604a0 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.768+0000 7f8e157fa640 1 -- 172.21.15.16:0/2448449922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20098480 con 0x5564b00604a0 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- 172.21.15.16:0/2448449922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 msgr2=0x5564b0060880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8e20098650 tx=0x7f8e2009b5e0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- 172.21.15.16:0/2448449922 shutdown_connections 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b006da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b006b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2448449922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b00604a0 0x5564b0060880 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- 172.21.15.16:0/2448449922 >> 172.21.15.16:0/2448449922 conn(0x5564b005ca30 msgr2=0x5564b005ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- 172.21.15.16:0/2448449922 shutdown_connections 2024-09-12T02:51:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- 172.21.15.16:0/2448449922 wait complete. 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 -- start start 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.769+0000 7f8e3777e740 1 Processor -- start 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 0x5564b006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b00687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5564b006e250 con 0x5564b0060dc0 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5564aff81050 con 0x5564b00604a0 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564b006e3d0 con 0x5564b006b680 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e3777e740 10 client.0 _ll_register_callbacks cb 0x5564affc1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42106/0 (socket says 172.21.15.16:42106) 2024-09-12T02:51:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/4132613263 learned_addr learned my addr 172.21.15.16:0/4132613263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b00687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2effd640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 0x5564b006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/4132613263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 msgr2=0x5564b00687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b00687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 msgr2=0x5564b006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 0x5564b006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/4132613263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564afd373c0 con 0x5564b0060dc0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b00687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2effd640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 0x5564b006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400b9f0 tx=0x7f8e2400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11501eaa00da2e8d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.770+0000 7f8e12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24003e20 con 0x5564b0060dc0 2024-09-12T02:51:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e200a0030 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e24005020 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e24005220 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 10 client.16275 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5564b0079cb0 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e200a0030 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f8e2401b570 con 0x5564b0060dc0 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 10 client.16275 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 20 client.16275 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b007b110 0x5564b007d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 4 client.16275 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.771+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564affd3850 con 0x5564b007b110 2024-09-12T02:51:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.772+0000 7f8e2effd640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b007b110 0x5564b007d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.772+0000 7f8e12ff5640 10 client.16275 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.772+0000 7f8e2effd640 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b007b110 0x5564b007d500 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a36a894ec135363 server_cookie=4614ad88d6c85170 in_seq=0 out_seq=0 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e12ff5640 1 -- 172.21.15.16:0/4132613263 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5564affd3850 con 0x5564b007b110 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e12ff5640 10 client.16275 handle_command_reply: tid=0 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 client.16275 shutdown 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 msgr2=0x5564b0068270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400b9f0 tx=0x7f8e2400beb0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 shutdown_connections 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b007b110 0x5564b007d500 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b006b680 0x5564b00687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0068270 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e12ff5640 0 client.16275 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 --2- 172.21.15.16:0/4132613263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b00604a0 0x5564b006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 >> 172.21.15.16:0/4132613263 conn(0x5564b005ca30 msgr2=0x5564b005bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 shutdown_connections 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 1 -- 172.21.15.16:0/4132613263 wait complete. 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.773+0000 7f8e3777e740 20 client.16275 trim_cache size 0 max 16384 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 Processor -- start 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 -- start start 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0082e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b0083350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301af800 con 0x5564b005ca30 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301b1370 con 0x5564b0060dc0 2024-09-12T02:51:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e3777e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301afed0 con 0x5564b00604a0 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42114/0 (socket says 172.21.15.16:42114) 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e2effd640 1 -- 172.21.15.16:0/2210445131 learned_addr learned my addr 172.21.15.16:0/2210445131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.795+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b0083350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 -- 172.21.15.16:0/2210445131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 msgr2=0x5564b0082e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0082e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 -- 172.21.15.16:0/2210445131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 msgr2=0x5564b0083350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b0083350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 -- 172.21.15.16:0/2210445131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564b006b330 con 0x5564b005ca30 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b0083350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e2effd640 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8e20099f30 tx=0x7f8e20097f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461befd00fcc10a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e14ff9640 1 -- 172.21.15.16:0/2210445131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e200982f0 con 0x5564b005ca30 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e14ff9640 1 -- 172.21.15.16:0/2210445131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e200cd030 con 0x5564b005ca30 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.796+0000 7f8e14ff9640 1 -- 172.21.15.16:0/2210445131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2009b0d0 con 0x5564b005ca30 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- 172.21.15.16:0/2210445131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 msgr2=0x5564b006a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8e20099f30 tx=0x7f8e20097f20 comp rx=0 tx=0).stop 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- 172.21.15.16:0/2210445131 shutdown_connections 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b0060dc0 0x5564b0083350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0082e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 --2- 172.21.15.16:0/2210445131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b005ca30 0x5564b006a1e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- 172.21.15.16:0/2210445131 >> 172.21.15.16:0/2210445131 conn(0x5564b0068910 msgr2=0x5564b0069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- 172.21.15.16:0/2210445131 shutdown_connections 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- 172.21.15.16:0/2210445131 wait complete. 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 -- start start 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.797+0000 7f8e3777e740 1 Processor -- start 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 0x5564b00739b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301b0c70 con 0x5564b0060dc0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301b05a0 con 0x5564b005ca30 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e301ad7e0 con 0x5564b00604a0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e3777e740 10 client.0 _ll_register_callbacks cb 0x7f8e24004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 0x5564b00739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 0x5564b00739b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40436/0 (socket says 172.21.15.16:40436) 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2effd640 1 -- 172.21.15.16:0/1820302088 learned_addr learned my addr 172.21.15.16:0/1820302088 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 msgr2=0x5564b00739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 0x5564b00739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/1820302088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 msgr2=0x5564b0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 -- 172.21.15.16:0/1820302088 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564b0071780 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2f7fe640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e2e7fc640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0073ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8e2401b9e0 tx=0x7f8e24004010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5922b5b2873d6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.798+0000 7f8e12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e240055d0 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e200cd030 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e2401b470 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e2401b690 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 10 client.25756 fetch_fsmap learned FSMap version 31 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5564b00e59d0 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e200cd030 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f8e24011560 con 0x5564b00604a0 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 10 client.25756 fetch_fsmap finished waiting for FSMap version 31 2024-09-12T02:51:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 20 client.25756 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b006b680 0x5564b00e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 4 client.25756 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.799+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564affcdd60 con 0x5564b006b680 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.800+0000 7f8e2effd640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b006b680 0x5564b00e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.800+0000 7f8e2effd640 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b006b680 0x5564b00e81c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d988cf0fe386043 server_cookie=ff15bab9dad8ff8e in_seq=0 out_seq=0 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.800+0000 7f8e12ff5640 10 client.25756 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e12ff5640 1 -- 172.21.15.16:0/1820302088 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5564affcdd60 con 0x5564b006b680 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e12ff5640 10 client.25756 handle_command_reply: tid=0 2024-09-12T02:51:17.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 client.25756 shutdown 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "788816f2-0e53-42b8-89ba-4a0d4b48cb0b": { 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "788816f2-0e53-42b8-89ba-4a0d4b48cb0b", 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 msgr2=0x5564b0073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0073ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8e2401b9e0 tx=0x7f8e24004010 comp rx=0 tx=0).stop 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 shutdown_connections 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564b006b680 0x5564b00e81c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564b0060dc0 0x5564b0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564b00604a0 0x5564b0073ef0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 --2- 172.21.15.16:0/1820302088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5564b005ca30 0x5564b00739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e12ff5640 0 client.25756 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 >> 172.21.15.16:0/1820302088 conn(0x5564b0068910 msgr2=0x5564b0066710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 shutdown_connections 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 1 -- 172.21.15.16:0/1820302088 wait complete. 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.801+0000 7f8e3777e740 20 client.25756 trim_cache size 0 max 16384 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 msgr2=0x7f8e180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 0x7f8e180a5040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005e30 tx=0x7f8e28005f70 comp rx=0 tx=0).stop 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 msgr2=0x7f8e3019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000e9f0 tx=0x7f8e2000edf0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 shutdown_connections 2024-09-12T02:51:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e180a2b80 0x7f8e180a5040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e3010e4f0 0x7f8e3019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101380 0x7f8e3019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 --2- 172.21.15.16:0/3782746204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e301009d0 0x7f8e3019a6e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 >> 172.21.15.16:0/3782746204 conn(0x7f8e300fc7d0 msgr2=0x7f8e300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 shutdown_connections 2024-09-12T02:51:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:17.802+0000 7f8e353a5640 1 -- 172.21.15.16:0/3782746204 wait complete. 2024-09-12T02:51:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:17 smithi016 ceph-mon[30925]: pgmap v438: 97 pgs: 97 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-09-12T02:51:18.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:17 smithi033 ceph-mon[35092]: pgmap v438: 97 pgs: 97 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-09-12T02:51:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:17 smithi184 ceph-mon[35321]: pgmap v438: 97 pgs: 97 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-09-12T02:51:18.712 INFO:tasks.workunit.client.0.smithi016.stdout: 1 276688 14.96 MB/sec execute 20 sec latency 28.158 ms 2024-09-12T02:51:18.803 INFO:tasks.cephfs.filesystem:scrub status for tag:788816f2-0e53-42b8-89ba-4a0d4b48cb0b - {'path': '/', 'tag': '788816f2-0e53-42b8-89ba-4a0d4b48cb0b', 'options': 'recursive,force'} 2024-09-12T02:51:19.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 279191 14.70 MB/sec execute 21 sec latency 12.401 ms 2024-09-12T02:51:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:19 smithi016 ceph-mon[30925]: pgmap v439: 97 pgs: 97 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-09-12T02:51:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:19 smithi033 ceph-mon[35092]: pgmap v439: 97 pgs: 97 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-09-12T02:51:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:19 smithi184 ceph-mon[35321]: pgmap v439: 97 pgs: 97 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-09-12T02:51:20.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 281504 14.64 MB/sec execute 22 sec latency 15.588 ms 2024-09-12T02:51:21.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 284574 14.76 MB/sec execute 23 sec latency 17.533 ms 2024-09-12T02:51:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:21 smithi016 ceph-mon[30925]: pgmap v440: 97 pgs: 97 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-12T02:51:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:21 smithi033 ceph-mon[35092]: pgmap v440: 97 pgs: 97 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-12T02:51:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:21 smithi184 ceph-mon[35321]: pgmap v440: 97 pgs: 97 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-12T02:51:22.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287549 14.91 MB/sec execute 24 sec latency 44.952 ms 2024-09-12T02:51:23.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:22 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:51:23.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:22 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:51:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:22 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:51:23.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:22 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:51:23.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:22 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:51:23.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:22 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:51:23.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 290092 14.77 MB/sec execute 25 sec latency 13.566 ms 2024-09-12T02:51:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:23 smithi016 ceph-mon[30925]: pgmap v441: 97 pgs: 97 active+clean; 125 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-09-12T02:51:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:23 smithi033 ceph-mon[35092]: pgmap v441: 97 pgs: 97 active+clean; 125 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-09-12T02:51:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:23 smithi184 ceph-mon[35321]: pgmap v441: 97 pgs: 97 active+clean; 125 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-09-12T02:51:24.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 292524 14.71 MB/sec execute 26 sec latency 14.418 ms 2024-09-12T02:51:25.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 295337 14.71 MB/sec execute 27 sec latency 17.276 ms 2024-09-12T02:51:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:25 smithi016 ceph-mon[30925]: pgmap v442: 97 pgs: 97 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-09-12T02:51:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:25 smithi033 ceph-mon[35092]: pgmap v442: 97 pgs: 97 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-09-12T02:51:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:25 smithi184 ceph-mon[35321]: pgmap v442: 97 pgs: 97 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-09-12T02:51:26.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 298392 14.84 MB/sec execute 28 sec latency 10.749 ms 2024-09-12T02:51:27.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 300962 14.76 MB/sec execute 29 sec latency 88.503 ms 2024-09-12T02:51:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:27 smithi016 ceph-mon[30925]: pgmap v443: 97 pgs: 97 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-12T02:51:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:27 smithi033 ceph-mon[35092]: pgmap v443: 97 pgs: 97 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-12T02:51:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:27 smithi184 ceph-mon[35321]: pgmap v443: 97 pgs: 97 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-09-12T02:51:28.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303217 14.66 MB/sec execute 30 sec latency 12.712 ms 2024-09-12T02:51:29.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:28 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:51:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:28 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:51:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:28 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T02:51:29.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 305749 14.65 MB/sec execute 31 sec latency 16.419 ms 2024-09-12T02:51:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:29 smithi016 ceph-mon[30925]: pgmap v444: 97 pgs: 97 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-09-12T02:51:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:29 smithi033 ceph-mon[35092]: pgmap v444: 97 pgs: 97 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-09-12T02:51:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:29 smithi184 ceph-mon[35321]: pgmap v444: 97 pgs: 97 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 251 op/s 2024-09-12T02:51:30.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 308880 14.77 MB/sec execute 32 sec latency 10.966 ms 2024-09-12T02:51:31.635 INFO:tasks.quiescer.fs.[cephfs]:Total duration (40.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T02:51:31.636 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.15 2024-09-12T02:51:31.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:51:31.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 311951 14.77 MB/sec execute 33 sec latency 11.847 ms 2024-09-12T02:51:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:32.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:31 smithi016 ceph-mon[30925]: pgmap v445: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-09-12T02:51:32.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:31 smithi033 ceph-mon[35092]: pgmap v445: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-09-12T02:51:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:31 smithi184 ceph-mon[35321]: pgmap v445: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-09-12T02:51:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 -- 172.21.15.16:0/2100444089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964100e20 msgr2=0x7fb964101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 --2- 172.21.15.16:0/2100444089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964100e20 0x7fb964101200 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb954009900 tx=0x7fb95402a480 comp rx=0 tx=0).stop 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 -- 172.21.15.16:0/2100444089 shutdown_connections 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 --2- 172.21.15.16:0/2100444089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964102ef0 0x7fb96410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 --2- 172.21.15.16:0/2100444089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb9641029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 --2- 172.21.15.16:0/2100444089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964100e20 0x7fb964101200 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 -- 172.21.15.16:0/2100444089 >> 172.21.15.16:0/2100444089 conn(0x7fb9640fc7d0 msgr2=0x7fb9640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 -- 172.21.15.16:0/2100444089 shutdown_connections 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.424+0000 7fb96967f640 1 -- 172.21.15.16:0/2100444089 wait complete. 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.425+0000 7fb96967f640 1 Processor -- start 2024-09-12T02:51:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.425+0000 7fb96967f640 1 -- start start 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 0x7fb9641985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb964198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9641120a0 con 0x7fb964102ef0 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb964111f20 con 0x7fb964100e20 2024-09-12T02:51:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb96967f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb964112220 con 0x7fb964102570 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33932/0 (socket says 172.21.15.16:33932) 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb964198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9637fe640 1 -- 172.21.15.16:0/2336174692 learned_addr learned my addr 172.21.15.16:0/2336174692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb962ffd640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 0x7fb9641985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 -- 172.21.15.16:0/2336174692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 msgr2=0x7fb9641985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 0x7fb9641985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 -- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 msgr2=0x7fb96419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 -- 172.21.15.16:0/2336174692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96419d550 con 0x7fb964102570 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9637fe640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb962ffd640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 0x7fb9641985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.426+0000 7fb9627fc640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb964198ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb95800e9f0 tx=0x7fb95800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.427+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800ccc0 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.427+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb958005020 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.427+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96419d840 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.427+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9580049e0 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.427+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9641a5120 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.428+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb964101200 con 0x7fb964102570 2024-09-12T02:51:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.429+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb958005560 con 0x7fb964102570 2024-09-12T02:51:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.430+0000 7fb94bfff640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 0x7fb94c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.430+0000 7fb962ffd640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 0x7fb94c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.430+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb9580cef90 con 0x7fb964102570 2024-09-12T02:51:32.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.431+0000 7fb962ffd640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 0x7fb94c0a50c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb954009230 tx=0x7fb9540058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.434+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb9580051d0 con 0x7fb964102570 2024-09-12T02:51:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.685+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9641090c0 con 0x7fb964102570 2024-09-12T02:51:32.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.686+0000 7fb94bfff640 1 -- 172.21.15.16:0/2336174692 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fb95809bf90 con 0x7fb964102570 2024-09-12T02:51:32.687 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:51:32.687 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:51:32.687 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 msgr2=0x7fb94c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 0x7fb94c0a50c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb954009230 tx=0x7fb9540058d0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 msgr2=0x7fb964198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb964198ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb95800e9f0 tx=0x7fb95800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 shutdown_connections 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.688+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb94c0a2c00 0x7fb94c0a50c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb964102ef0 0x7fb96419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb964102570 0x7fb964198ae0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 --2- 172.21.15.16:0/2336174692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb964100e20 0x7fb9641985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 >> 172.21.15.16:0/2336174692 conn(0x7fb9640fc7d0 msgr2=0x7fb9640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 shutdown_connections 2024-09-12T02:51:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:32.689+0000 7fb96967f640 1 -- 172.21.15.16:0/2336174692 wait complete. 2024-09-12T02:51:32.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313581 14.62 MB/sec execute 34 sec latency 71.125 ms 2024-09-12T02:51:33.078 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2336174692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:33.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2336174692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:33.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2336174692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:33.457 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0' 2024-09-12T02:51:33.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0 2024-09-12T02:51:33.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 316324 14.60 MB/sec execute 35 sec latency 73.969 ms 2024-09-12T02:51:33.728 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:34.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:33 smithi016 ceph-mon[30925]: pgmap v446: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 305 op/s 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 -- 172.21.15.16:0/1706954810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908102570 msgr2=0x7f39081029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 --2- 172.21.15.16:0/1706954810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908102570 0x7f39081029b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009900 tx=0x7f38fc02a480 comp rx=0 tx=0).stop 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 -- 172.21.15.16:0/1706954810 shutdown_connections 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 --2- 172.21.15.16:0/1706954810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 0x7f390810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 --2- 172.21.15.16:0/1706954810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908102570 0x7f39081029b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 --2- 172.21.15.16:0/1706954810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908100e20 0x7f3908101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 -- 172.21.15.16:0/1706954810 >> 172.21.15.16:0/1706954810 conn(0x7f39080fc7d0 msgr2=0x7f39080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 -- 172.21.15.16:0/1706954810 shutdown_connections 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.236+0000 7f390e67c640 1 -- 172.21.15.16:0/1706954810 wait complete. 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 Processor -- start 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 -- start start 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 0x7f3908198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 0x7f390819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3908110f80 con 0x7f3908100e20 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3908110e00 con 0x7f3908102570 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390e67c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3908111100 con 0x7f3908102ef0 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f3907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f3907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33952/0 (socket says 172.21.15.16:33952) 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f3907fff640 1 -- 172.21.15.16:0/1869928219 learned_addr learned my addr 172.21.15.16:0/1869928219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f39077fe640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 0x7f3908198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.237+0000 7f390cc19640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 0x7f390819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 -- 172.21.15.16:0/1869928219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 msgr2=0x7f390819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 0x7f390819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 -- 172.21.15.16:0/1869928219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 msgr2=0x7f3908198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 0x7f3908198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 -- 172.21.15.16:0/1869928219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390819d480 con 0x7f3908100e20 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f39077fe640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 0x7f3908198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f3907fff640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f38f800e9f0 tx=0x7f38f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f39057fa640 1 -- 172.21.15.16:0/1869928219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f800cd80 con 0x7f3908100e20 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390819d770 con 0x7f3908100e20 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f39057fa640 1 -- 172.21.15.16:0/1869928219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38f8010910 con 0x7f3908100e20 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f39057fa640 1 -- 172.21.15.16:0/1869928219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f8018990 con 0x7f3908100e20 2024-09-12T02:51:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.238+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39081a5050 con 0x7f3908100e20 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f39057fa640 1 -- 172.21.15.16:0/1869928219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f38f8010ac0 con 0x7f3908100e20 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 Processor -- start 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 -- start start 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb2ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fbfa84160 con 0x563fbfb2aa30 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fbfa2b720 con 0x563fbfb1f850 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3910a55740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fbfa80120 con 0x563fbfb20170 2024-09-12T02:51:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f39057fa640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 0x7f38f00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3907fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f390cc19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3907fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34650/0 (socket says 172.21.15.16:34650) 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f390cc19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43736/0 (socket says 172.21.15.16:43736) 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.242+0000 7f3907fff640 1 -- 172.21.15.16:0/142635507 learned_addr learned my addr 172.21.15.16:0/142635507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f39077fe640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb2ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 0x7f38f00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f39057fa640 1 -- 172.21.15.16:0/1869928219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38f80cf2a0 con 0x7f3908100e20 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 -- 172.21.15.16:0/142635507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 msgr2=0x563fbfb2a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 -- 172.21.15.16:0/142635507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 msgr2=0x563fbfb2ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb2ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 -- 172.21.15.16:0/142635507 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fbf7f5ba0 con 0x563fbfb1f850 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f3907fff640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f39077fe640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb2ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3900005fd0 tx=0x7f390000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ae61e1d88258e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f390cc19640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 0x7f38f00a50c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f390000fc90 tx=0x7f390001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.243+0000 7f38ee7fc640 1 -- 172.21.15.16:0/142635507 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900011e30 con 0x563fbfb1f850 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f38ee7fc640 1 -- 172.21.15.16:0/142635507 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f390001dbf0 con 0x563fbfb1f850 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f38ee7fc640 1 -- 172.21.15.16:0/142635507 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000042b0 con 0x563fbfb1f850 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 -- 172.21.15.16:0/142635507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 msgr2=0x563fbfb1fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3900005fd0 tx=0x7f390000f790 comp rx=0 tx=0).stop 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 -- 172.21.15.16:0/142635507 shutdown_connections 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb2ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb2a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 --2- 172.21.15.16:0/142635507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb1fc30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 -- 172.21.15.16:0/142635507 >> 172.21.15.16:0/142635507 conn(0x563fbfb1bde0 msgr2=0x563fbfb1c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 -- 172.21.15.16:0/142635507 shutdown_connections 2024-09-12T02:51:34.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.244+0000 7f3910a55740 1 -- 172.21.15.16:0/142635507 wait complete. 2024-09-12T02:51:34.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 -- start start 2024-09-12T02:51:34.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:33 smithi033 ceph-mon[35092]: pgmap v446: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 305 op/s 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 Processor -- start 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb283a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fbfb2d600 con 0x563fbfb1f850 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fbfa0c900 con 0x563fbfb2aa30 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fbfb2d780 con 0x563fbfb20170 2024-09-12T02:51:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3910a55740 10 client.0 _ll_register_callbacks cb 0x563fbfa0a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f39077fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33972/0 (socket says 172.21.15.16:33972) 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f3907fff640 1 -- 172.21.15.16:0/1432658712 learned_addr learned my addr 172.21.15.16:0/1432658712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.245+0000 7f39077fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34666/0 (socket says 172.21.15.16:34666) 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f390cc19640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb283a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 -- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 msgr2=0x563fbfb283a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb283a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 -- 172.21.15.16:0/1432658712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 msgr2=0x563fbfb27920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 -- 172.21.15.16:0/1432658712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fbfa4cc50 con 0x563fbfb20170 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f3907fff640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f39077fe640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005940 tx=0x7f38fc0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39311fc8ab77c911 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f38e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc035070 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38fc003a30 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.246+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390001dbf0 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38fc003c30 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 10 client.25774 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563fbfb39300 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f390001dbf0 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38fc004b60 con 0x563fbfb20170 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 10 client.25774 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 10 client.25774 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 20 client.25774 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfb3a7c0 0x563fbfb3cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 4 client.25774 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fbfa91890 con 0x563fbfb3a7c0 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.247+0000 7f3907fff640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfb3a7c0 0x563fbfb3cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.248+0000 7f3907fff640 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfb3a7c0 0x563fbfb3cbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4e1c95313585e6 server_cookie=2286cbea7550565a in_seq=0 out_seq=0 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.248+0000 7f38e77fe640 10 client.25774 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.248+0000 7f38e77fe640 1 -- 172.21.15.16:0/1432658712 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563fbfa91890 con 0x563fbfb3a7c0 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.248+0000 7f38e77fe640 10 client.25774 handle_command_reply: tid=0 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.248+0000 7f3910a55740 1 client.25774 shutdown 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 msgr2=0x563fbfb27e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005940 tx=0x7f38fc0035f0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 shutdown_connections 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfb3a7c0 0x563fbfb3cbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb283a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb27e60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 --2- 172.21.15.16:0/1432658712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb1f850 0x563fbfb27920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 >> 172.21.15.16:0/1432658712 conn(0x563fbfb1bde0 msgr2=0x563fbfb21b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f38e77fe640 0 client.25774 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 shutdown_connections 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 1 -- 172.21.15.16:0/1432658712 wait complete. 2024-09-12T02:51:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.249+0000 7f3910a55740 20 client.25774 trim_cache size 0 max 16384 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 Processor -- start 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 -- start start 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 0x563fbfb22300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb42a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39081ad470 con 0x563fbfb20170 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39081aefb0 con 0x563fbfb2aa30 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3910a55740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f39081adb40 con 0x563fbfb1f850 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f39077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3907fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 0x563fbfb22300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f390cc19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb42a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f39077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33986/0 (socket says 172.21.15.16:33986) 2024-09-12T02:51:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 -- 172.21.15.16:0/2930505093 learned_addr learned my addr 172.21.15.16:0/2930505093 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.267+0000 7f3907fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 0x563fbfb22300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34674/0 (socket says 172.21.15.16:34674) 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 -- 172.21.15.16:0/2930505093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 msgr2=0x563fbfb22300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 0x563fbfb22300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 -- 172.21.15.16:0/2930505093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 msgr2=0x563fbfb42a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb42a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 -- 172.21.15.16:0/2930505093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fbfb1ce90 con 0x563fbfb20170 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f390cc19640 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb42a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f39077fe640 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009900 tx=0x7f38fc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b4a8d1a29cac0f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f38edffb640 1 -- 172.21.15.16:0/2930505093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc054070 con 0x563fbfb20170 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f38edffb640 1 -- 172.21.15.16:0/2930505093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc003dd0 con 0x563fbfb20170 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.268+0000 7f38edffb640 1 -- 172.21.15.16:0/2930505093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc035070 con 0x563fbfb20170 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- 172.21.15.16:0/2930505093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 msgr2=0x563fbfb424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009900 tx=0x7f38fc02a480 comp rx=0 tx=0).stop 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- 172.21.15.16:0/2930505093 shutdown_connections 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb2aa30 0x563fbfb42a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb20170 0x563fbfb424c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 --2- 172.21.15.16:0/2930505093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb1f850 0x563fbfb22300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- 172.21.15.16:0/2930505093 >> 172.21.15.16:0/2930505093 conn(0x563fbfb1bde0 msgr2=0x563fbfb1aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- 172.21.15.16:0/2930505093 shutdown_connections 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- 172.21.15.16:0/2930505093 wait complete. 2024-09-12T02:51:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.269+0000 7f3910a55740 1 -- start start 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 Processor -- start 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb31f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb32450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39081ae8e0 con 0x563fbfb2aa30 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39081ae210 con 0x563fbfb1f850 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f39081ab470 con 0x563fbfb20170 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3910a55740 10 client.0 _ll_register_callbacks cb 0x7f38fc005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3907fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f390cc19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb32450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb31f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3907fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43770/0 (socket says 172.21.15.16:43770) 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f3907fff640 1 -- 172.21.15.16:0/981011080 learned_addr learned my addr 172.21.15.16:0/981011080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f390cc19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb32450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34000/0 (socket says 172.21.15.16:34000) 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 -- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 msgr2=0x563fbfb319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 -- 172.21.15.16:0/981011080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 msgr2=0x563fbfb32450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb32450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.270+0000 7f39077fe640 1 -- 172.21.15.16:0/981011080 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fbfb32c50 con 0x563fbfb20170 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f39077fe640 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb31f10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009850 tx=0x7f38fc009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=984fbcfd54c4c18e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f38e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc054040 con 0x563fbfb20170 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38fc003a30 con 0x563fbfb20170 2024-09-12T02:51:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38fc003dd0 con 0x563fbfb20170 2024-09-12T02:51:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38fc060050 con 0x563fbfb20170 2024-09-12T02:51:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f3910a55740 10 client.25780 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563fbfba9ec0 con 0x563fbfb20170 2024-09-12T02:51:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.271+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc003dd0 con 0x563fbfb20170 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38fc03c440 con 0x563fbfb20170 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 10 client.25780 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 10 client.25780 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 20 client.25780 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfbaa4e0 0x563fbfbac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 4 client.25780 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}] 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}) -- 0x563fbfa8bbe0 con 0x563fbfbaa4e0 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3907fff640 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfbaa4e0 0x563fbfbac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f38e77fe640 10 client.25780 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.272+0000 7f3907fff640 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfbaa4e0 0x563fbfbac900 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24c0ec95590db5c server_cookie=260f39a1cae4f785 in_seq=0 out_seq=0 2024-09-12T02:51:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:33 smithi184 ceph-mon[35321]: pgmap v446: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 305 op/s 2024-09-12T02:51:34.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.346+0000 7f38e77fe640 1 -- 172.21.15.16:0/981011080 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563fbfa8bbe0 con 0x563fbfbaa4e0 2024-09-12T02:51:34.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.346+0000 7f38e77fe640 10 client.25780 handle_command_reply: tid=0 2024-09-12T02:51:34.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.346+0000 7f3910a55740 1 client.25780 shutdown 2024-09-12T02:51:34.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.346+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 msgr2=0x563fbfb31f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.346+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb31f10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009850 tx=0x7f38fc009670 comp rx=0 tx=0).stop 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 shutdown_connections 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563fbfbaa4e0 0x563fbfbac900 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fbfb2aa30 0x563fbfb32450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fbfb20170 0x563fbfb31f10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f38e77fe640 0 client.25780 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 --2- 172.21.15.16:0/981011080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fbfb1f850 0x563fbfb319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 >> 172.21.15.16:0/981011080 conn(0x563fbfb1bde0 msgr2=0x563fbfb35850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 shutdown_connections 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 1 -- 172.21.15.16:0/981011080 wait complete. 2024-09-12T02:51:34.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f3910a55740 20 client.25780 trim_cache size 0 max 16384 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 14, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "6810785f": { 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 14, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:51:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.2, 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:34.350 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:34.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 msgr2=0x7f38f00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 0x7f38f00a50c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f390000fc90 tx=0x7f390001e000 comp rx=0 tx=0).stop 2024-09-12T02:51:34.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 msgr2=0x7f3908198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:34.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.347+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f38f800e9f0 tx=0x7f38f800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 shutdown_connections 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3908102ef0 0x7f390819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3908102570 0x7f3908198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38f00a2c00 0x7f38f00a50c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 --2- 172.21.15.16:0/1869928219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3908100e20 0x7f3908198570 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 >> 172.21.15.16:0/1869928219 conn(0x7f39080fc7d0 msgr2=0x7f390806a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 shutdown_connections 2024-09-12T02:51:34.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:34.348+0000 7f390e67c640 1 -- 172.21.15.16:0/1869928219 wait complete. 2024-09-12T02:51:34.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 14.39 MB/sec execute 36 sec latency 439.243 ms 2024-09-12T02:51:35.218 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6810785f', quiesce took 3.6 seconds. Will release after: 16.549999999999997 2024-09-12T02:51:35.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 14.00 MB/sec execute 37 sec latency 1439.284 ms 2024-09-12T02:51:36.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:35 smithi016 ceph-mon[30925]: pgmap v447: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-12T02:51:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:35 smithi033 ceph-mon[35092]: pgmap v447: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-12T02:51:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:35 smithi184 ceph-mon[35321]: pgmap v447: 97 pgs: 97 active+clean; 133 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 254 op/s 2024-09-12T02:51:36.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 13.64 MB/sec execute 38 sec latency 2439.394 ms 2024-09-12T02:51:37.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 13.29 MB/sec execute 39 sec latency 3439.506 ms 2024-09-12T02:51:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:37 smithi016 ceph-mon[30925]: pgmap v448: 97 pgs: 97 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-12T02:51:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:37 smithi033 ceph-mon[35092]: pgmap v448: 97 pgs: 97 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-12T02:51:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:37 smithi184 ceph-mon[35321]: pgmap v448: 97 pgs: 97 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 325 op/s 2024-09-12T02:51:38.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 12.95 MB/sec execute 40 sec latency 4439.613 ms 2024-09-12T02:51:39.713 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 12.64 MB/sec execute 41 sec latency 5439.679 ms 2024-09-12T02:51:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:39 smithi016 ceph-mon[30925]: pgmap v449: 97 pgs: 97 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:51:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:39 smithi033 ceph-mon[35092]: pgmap v449: 97 pgs: 97 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:51:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:39 smithi184 ceph-mon[35321]: pgmap v449: 97 pgs: 97 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:51:40.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 12.34 MB/sec execute 42 sec latency 6439.799 ms 2024-09-12T02:51:41.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 12.05 MB/sec execute 43 sec latency 7439.914 ms 2024-09-12T02:51:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:41 smithi016 ceph-mon[30925]: pgmap v450: 97 pgs: 97 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-12T02:51:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:41 smithi033 ceph-mon[35092]: pgmap v450: 97 pgs: 97 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-12T02:51:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:41 smithi184 ceph-mon[35321]: pgmap v450: 97 pgs: 97 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-09-12T02:51:42.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 11.78 MB/sec execute 44 sec latency 8440.028 ms 2024-09-12T02:51:43.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 11.51 MB/sec execute 45 sec latency 9440.144 ms 2024-09-12T02:51:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:43 smithi016 ceph-mon[30925]: pgmap v451: 97 pgs: 97 active+clean; 140 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-09-12T02:51:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:43 smithi033 ceph-mon[35092]: pgmap v451: 97 pgs: 97 active+clean; 140 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-09-12T02:51:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:43 smithi184 ceph-mon[35321]: pgmap v451: 97 pgs: 97 active+clean; 140 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-09-12T02:51:44.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 11.26 MB/sec execute 46 sec latency 10440.200 ms 2024-09-12T02:51:45.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 11.02 MB/sec execute 47 sec latency 11440.272 ms 2024-09-12T02:51:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:45 smithi016 ceph-mon[30925]: pgmap v452: 97 pgs: 97 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 166 op/s 2024-09-12T02:51:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:45 smithi033 ceph-mon[35092]: pgmap v452: 97 pgs: 97 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 166 op/s 2024-09-12T02:51:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:45 smithi184 ceph-mon[35321]: pgmap v452: 97 pgs: 97 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 166 op/s 2024-09-12T02:51:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 10.79 MB/sec execute 48 sec latency 12440.365 ms 2024-09-12T02:51:47.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 10.57 MB/sec execute 49 sec latency 13440.480 ms 2024-09-12T02:51:48.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:47 smithi016 ceph-mon[30925]: pgmap v453: 97 pgs: 97 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 165 op/s 2024-09-12T02:51:48.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:47 smithi033 ceph-mon[35092]: pgmap v453: 97 pgs: 97 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 165 op/s 2024-09-12T02:51:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:47 smithi184 ceph-mon[35321]: pgmap v453: 97 pgs: 97 active+clean; 138 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 165 op/s 2024-09-12T02:51:48.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 10.36 MB/sec execute 50 sec latency 14440.533 ms 2024-09-12T02:51:48.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:51:49.076 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 -- 172.21.15.16:0/2076664325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270102de0 msgr2=0x7fe2701031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 --2- 172.21.15.16:0/2076664325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270102de0 0x7fe2701031c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe264009900 tx=0x7fe26402a480 comp rx=0 tx=0).stop 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 -- 172.21.15.16:0/2076664325 shutdown_connections 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 --2- 172.21.15.16:0/2076664325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 --2- 172.21.15.16:0/2076664325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270103700 0x7fe27010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 --2- 172.21.15.16:0/2076664325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270102de0 0x7fe2701031c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 -- 172.21.15.16:0/2076664325 >> 172.21.15.16:0/2076664325 conn(0x7fe2700fc780 msgr2=0x7fe2700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 -- 172.21.15.16:0/2076664325 shutdown_connections 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.557+0000 7fe276f73640 1 -- 172.21.15.16:0/2076664325 wait complete. 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 Processor -- start 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 -- start start 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 0x7fe270198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe270111f50 con 0x7fe270103700 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe270111dd0 con 0x7fe270102de0 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe276f73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701120d0 con 0x7fe27010c180 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44884/0 (socket says 172.21.15.16:44884) 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 -- 172.21.15.16:0/3874235918 learned_addr learned my addr 172.21.15.16:0/3874235918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe274d0f640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 0x7fe270198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe275510640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 -- 172.21.15.16:0/3874235918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 msgr2=0x7fe27019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 -- 172.21.15.16:0/3874235918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 msgr2=0x7fe270198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 0x7fe270198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe26ffff640 1 -- 172.21.15.16:0/3874235918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27019d430 con 0x7fe270103700 2024-09-12T02:51:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe274d0f640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 0x7fe270198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.558+0000 7fe275510640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe26ffff640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d950 tx=0x7fe25c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe26dffb640 1 -- 172.21.15.16:0/3874235918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c017070 con 0x7fe270103700 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27019d720 con 0x7fe270103700 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe26dffb640 1 -- 172.21.15.16:0/3874235918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25c009c30 con 0x7fe270103700 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe26dffb640 1 -- 172.21.15.16:0/3874235918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00f6b0 con 0x7fe270103700 2024-09-12T02:51:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.559+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2701a5000 con 0x7fe270103700 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.561+0000 7fe26dffb640 1 -- 172.21.15.16:0/3874235918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe25c0109d0 con 0x7fe270103700 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.561+0000 7fe27934c740 1 Processor -- start 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 -- start start 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42efda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b42e562e0 con 0x563b42ef0dc0 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563b42dfdce0 con 0x563b42ef04a0 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe27934c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563b42e522a0 con 0x563b42efb680 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44898/0 (socket says 172.21.15.16:44898) 2024-09-12T02:51:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 -- 172.21.15.16:0/1979472148 learned_addr learned my addr 172.21.15.16:0/1979472148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe26dffb640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42efda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe275510640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe275510640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 -- 172.21.15.16:0/1979472148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 msgr2=0x563b42efda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42efda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 -- 172.21.15.16:0/1979472148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 msgr2=0x563b42ef0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe26dffb640 1 -- 172.21.15.16:0/3874235918 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe25c0ceca0 con 0x7fe270103700 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe274d0f640 1 -- 172.21.15.16:0/1979472148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b42de2440 con 0x563b42ef0dc0 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.562+0000 7fe275510640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.563+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe26402a9a0 tx=0x7fe264031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac2f4693f6f6fc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.563+0000 7fe275510640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe268007fd0 tx=0x7fe2680072a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.563+0000 7fe25affd640 1 -- 172.21.15.16:0/1979472148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264035040 con 0x563b42ef0dc0 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.563+0000 7fe25affd640 1 -- 172.21.15.16:0/1979472148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264031a20 con 0x563b42ef0dc0 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.563+0000 7fe25affd640 1 -- 172.21.15.16:0/1979472148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264004a00 con 0x563b42ef0dc0 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- 172.21.15.16:0/1979472148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 msgr2=0x563b42efb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe26402a9a0 tx=0x7fe264031630 comp rx=0 tx=0).stop 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- 172.21.15.16:0/1979472148 shutdown_connections 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42efda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42efb140 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 --2- 172.21.15.16:0/1979472148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- 172.21.15.16:0/1979472148 >> 172.21.15.16:0/1979472148 conn(0x563b42eeca30 msgr2=0x563b42eece10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- 172.21.15.16:0/1979472148 shutdown_connections 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- 172.21.15.16:0/1979472148 wait complete. 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 -- start start 2024-09-12T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.564+0000 7fe27934c740 1 Processor -- start 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42efaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b42efe250 con 0x563b42ef0dc0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563b42e11050 con 0x563b42ef04a0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563b42efe3d0 con 0x563b42efb680 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 10 client.0 _ll_register_callbacks cb 0x563b42e51160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44902/0 (socket says 172.21.15.16:44902) 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 -- 172.21.15.16:0/1309065821 learned_addr learned my addr 172.21.15.16:0/1309065821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42efaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe275510640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 -- 172.21.15.16:0/1309065821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 msgr2=0x563b42ef8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 -- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 msgr2=0x563b42efaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42efaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 -- 172.21.15.16:0/1309065821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b42bc73c0 con 0x563b42ef0dc0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42efaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cc7b0 tx=0x7fe25c06a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb6a2c2e2189a50 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c020070 con 0x563b42ef0dc0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe25c099810 con 0x563b42ef0dc0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.565+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe264031a20 con 0x563b42ef0dc0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe25c09bc10 con 0x563b42ef0dc0 2024-09-12T02:51:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 10 client.16317 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b42f09b00 con 0x563b42ef0dc0 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264031a20 con 0x563b42ef0dc0 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe25c097b80 con 0x563b42ef0dc0 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 10 client.16317 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 20 client.16317 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f0af60 0x563b42f0d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 4 client.16317 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b42e63850 con 0x563b42f0af60 2024-09-12T02:51:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.566+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f0af60 0x563b42f0d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.567+0000 7fe274d0f640 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f0af60 0x563b42f0d3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22df790fef8c940c server_cookie=bd8c911c38e6575c in_seq=0 out_seq=0 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.567+0000 7fe24ffff640 10 client.16317 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe24ffff640 1 -- 172.21.15.16:0/1309065821 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b42e63850 con 0x563b42f0af60 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe24ffff640 10 client.16317 handle_command_reply: tid=0 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 client.16317 shutdown 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 msgr2=0x563b42ef8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cc7b0 tx=0x7fe25c06a490 comp rx=0 tx=0).stop 2024-09-12T02:51:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 shutdown_connections 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f0af60 0x563b42f0d3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef8150 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 --2- 172.21.15.16:0/1309065821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42efaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 >> 172.21.15.16:0/1309065821 conn(0x563b42eeca30 msgr2=0x563b42ef2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe24ffff640 0 client.16317 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 shutdown_connections 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 1 -- 172.21.15.16:0/1309065821 wait complete. 2024-09-12T02:51:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.568+0000 7fe27934c740 20 client.16317 trim_cache size 0 max 16384 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.585+0000 7fe27934c740 1 Processor -- start 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.585+0000 7fe27934c740 1 -- start start 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42f12c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42ef0dc0 0x563b42f131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ad400 con 0x563b42efb680 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701aef40 con 0x563b42ef04a0 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe27934c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701adad0 con 0x563b42ef0dc0 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44910/0 (socket says 172.21.15.16:44910) 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 -- 172.21.15.16:0/2365241885 learned_addr learned my addr 172.21.15.16:0/2365241885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe274d0f640 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42f12c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42ef0dc0 0x563b42f131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 -- 172.21.15.16:0/2365241885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42ef0dc0 msgr2=0x563b42f131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42ef0dc0 0x563b42f131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 -- 172.21.15.16:0/2365241885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 msgr2=0x563b42f12c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42f12c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 -- 172.21.15.16:0/2365241885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b42efe130 con 0x563b42efb680 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe275510640 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe268010850 tx=0x7fe268007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e674a8e050494a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.586+0000 7fe25a7fc640 1 -- 172.21.15.16:0/2365241885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26801c040 con 0x563b42efb680 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe25a7fc640 1 -- 172.21.15.16:0/2365241885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe268022470 con 0x563b42efb680 2024-09-12T02:51:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe25a7fc640 1 -- 172.21.15.16:0/2365241885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268005280 con 0x563b42efb680 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 -- 172.21.15.16:0/2365241885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 msgr2=0x563b42ef53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe268010850 tx=0x7fe268007620 comp rx=0 tx=0).stop 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 -- 172.21.15.16:0/2365241885 shutdown_connections 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42efb680 0x563b42ef53b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42ef0dc0 0x563b42f131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 --2- 172.21.15.16:0/2365241885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42f12c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 -- 172.21.15.16:0/2365241885 >> 172.21.15.16:0/2365241885 conn(0x563b42eeca30 msgr2=0x563b42ef6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 -- 172.21.15.16:0/2365241885 shutdown_connections 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.587+0000 7fe27934c740 1 -- 172.21.15.16:0/2365241885 wait complete. 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 -- start start 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 Processor -- start 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ae870 con 0x563b42ef0dc0 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ae1a0 con 0x563b42ef04a0 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ab3e0 con 0x563b42efb680 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe27934c740 10 client.0 _ll_register_callbacks cb 0x7fe25c097130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44918/0 (socket says 172.21.15.16:44918) 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe26ffff640 1 -- 172.21.15.16:0/2733648308 learned_addr learned my addr 172.21.15.16:0/2733648308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe274d0f640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.588+0000 7fe275510640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 -- 172.21.15.16:0/2733648308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 msgr2=0x563b42ef5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 -- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 msgr2=0x563b42ef52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 -- 172.21.15.16:0/2733648308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b42eed6c0 con 0x563b42ef0dc0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe274d0f640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009610 tx=0x7fe25c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a637b09a8dd097 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c020070 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe25c09ab60 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe268022470 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe25c009de0 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe27934c740 10 client.16323 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b42eec860 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe268022470 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe25c09a4f0 con 0x563b42ef0dc0 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe27934c740 10 client.16323 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.589+0000 7fe27934c740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe27934c740 20 client.16323 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f763f0 0x563b42f78810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe27934c740 4 client.16323 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:51:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b42ef8a20 con 0x563b42f763f0 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe274d0f640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f763f0 0x563b42f78810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe24ffff640 10 client.16323 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe274d0f640 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f763f0 0x563b42f78810 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a36abeb8dc763fa5 server_cookie=c8a7cc385e055025 in_seq=0 out_seq=0 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe24ffff640 1 -- 172.21.15.16:0/2733648308 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563b42ef8a20 con 0x563b42f763f0 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe24ffff640 10 client.16323 handle_command_reply: tid=0 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.590+0000 7fe27934c740 1 client.16323 shutdown 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 msgr2=0x563b42ef57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009610 tx=0x7fe25c000bc0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 shutdown_connections 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563b42f763f0 0x563b42f78810 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b42efb680 0x563b42ef5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b42ef0dc0 0x563b42ef57e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 --2- 172.21.15.16:0/2733648308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b42ef04a0 0x563b42ef52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe24ffff640 0 client.16323 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 >> 172.21.15.16:0/2733648308 conn(0x563b42eeca30 msgr2=0x563b42ef65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 shutdown_connections 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 1 -- 172.21.15.16:0/2733648308 wait complete. 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe27934c740 20 client.16323 trim_cache size 0 max 16384 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:51:49.598 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 msgr2=0x7fe2600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe268007fd0 tx=0x7fe2680072a0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 msgr2=0x7fe2701989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.591+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d950 tx=0x7fe25c00de10 comp rx=0 tx=0).stop 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 shutdown_connections 2024-09-12T02:51:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe27010c180 0x7fe27019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270103700 0x7fe2701989c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 --2- 172.21.15.16:0/3874235918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270102de0 0x7fe270198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 >> 172.21.15.16:0/3874235918 conn(0x7fe2700fc780 msgr2=0x7fe2700fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 shutdown_connections 2024-09-12T02:51:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:49.592+0000 7fe276f73640 1 -- 172.21.15.16:0/3874235918 wait complete. 2024-09-12T02:51:49.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 10.16 MB/sec execute 51 sec latency 15440.637 ms 2024-09-12T02:51:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:49 smithi016 ceph-mon[30925]: pgmap v454: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:51:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:49 smithi033 ceph-mon[35092]: pgmap v454: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:51:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:49 smithi184 ceph-mon[35321]: pgmap v454: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:51:50.342 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:51:50.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:51:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:50.714 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 9.96 MB/sec execute 52 sec latency 16440.693 ms 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4037959024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268101dd0 msgr2=0x7ff2681021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4037959024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268101dd0 0x7ff2681021b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff264009900 tx=0x7ff26402a410 comp rx=0 tx=0).stop 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4037959024 shutdown_connections 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4037959024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268106f00 0x7ff26810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4037959024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4037959024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268101dd0 0x7ff2681021b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4037959024 >> 172.21.15.16:0/4037959024 conn(0x7ff2680fd890 msgr2=0x7ff2680ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4037959024 shutdown_connections 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.033+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4037959024 wait complete. 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 Processor -- start 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 -- start start 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 0x7ff268198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff268111f80 con 0x7ff268106f00 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff268111e00 con 0x7ff268102780 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26f6ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff268112100 con 0x7ff268101dd0 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26dc6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26dc6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44936/0 (socket says 172.21.15.16:44936) 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26dc6b640 1 -- 172.21.15.16:0/4013539913 learned_addr learned my addr 172.21.15.16:0/4013539913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26d46a640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 0x7ff268198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.034+0000 7ff26cc69640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 -- 172.21.15.16:0/4013539913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 msgr2=0x7ff268198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 0x7ff268198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 -- 172.21.15.16:0/4013539913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 msgr2=0x7ff2681989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 -- 172.21.15.16:0/4013539913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26819d410 con 0x7ff268106f00 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26d46a640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 0x7ff268198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26cc69640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26dc6b640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff26000b970 tx=0x7ff26000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260019070 con 0x7ff268106f00 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff26000cb40 con 0x7ff268106f00 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26819d700 con 0x7ff268106f00 2024-09-12T02:51:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260007650 con 0x7ff268106f00 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.035+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2681a4fe0 con 0x7ff268106f00 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.036+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2681021b0 con 0x7ff268106f00 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.037+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff26001c070 con 0x7ff268106f00 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.038+0000 7ff25e7fc640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 0x7ff2540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.038+0000 7ff26d46a640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 0x7ff2540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.038+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff2600ce6d0 con 0x7ff268106f00 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.039+0000 7ff26d46a640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 0x7ff2540a5040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff264005ab0 tx=0x7ff264005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.042+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff26009b640 con 0x7ff268106f00 2024-09-12T02:51:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.254+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff26810a140 con 0x7ff268106f00 2024-09-12T02:51:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.255+0000 7ff25e7fc640 1 -- 172.21.15.16:0/4013539913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7ff26009af40 con 0x7ff268106f00 2024-09-12T02:51:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:51:51.257 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:51:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:51:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 msgr2=0x7ff2540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 0x7ff2540a5040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff264005ab0 tx=0x7ff264005a20 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 msgr2=0x7ff26819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff26000b970 tx=0x7ff26000be30 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 shutdown_connections 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff2540a2b80 0x7ff2540a5040 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff268106f00 0x7ff26819cd30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff268102780 0x7ff2681989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 --2- 172.21.15.16:0/4013539913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff268101dd0 0x7ff268198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 >> 172.21.15.16:0/4013539913 conn(0x7ff2680fd890 msgr2=0x7ff2680fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 shutdown_connections 2024-09-12T02:51:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:51.257+0000 7ff26f6ce640 1 -- 172.21.15.16:0/4013539913 wait complete. 2024-09-12T02:51:51.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 9.78 MB/sec execute 53 sec latency 17440.722 ms 2024-09-12T02:51:51.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:51:51.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:51:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:52.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:51 smithi016 ceph-mon[30925]: pgmap v455: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:51:52.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:51 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4013539913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:52.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:51 smithi033 ceph-mon[35092]: pgmap v455: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:51:52.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:51 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4013539913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:52.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:51 smithi184 ceph-mon[35321]: pgmap v455: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:51:52.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:51 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4013539913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 -- 172.21.15.16:0/4022493707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 msgr2=0x7f5f00100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 --2- 172.21.15.16:0/4022493707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f00100ee0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009930 tx=0x7f5ee802a420 comp rx=0 tx=0).stop 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 -- 172.21.15.16:0/4022493707 shutdown_connections 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 --2- 172.21.15.16:0/4022493707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00101420 0x7f5f0010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 --2- 172.21.15.16:0/4022493707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f00100ee0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 --2- 172.21.15.16:0/4022493707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00106a80 0x7f5f00106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 -- 172.21.15.16:0/4022493707 >> 172.21.15.16:0/4022493707 conn(0x7f5f000fc7a0 msgr2=0x7f5f000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 -- 172.21.15.16:0/4022493707 shutdown_connections 2024-09-12T02:51:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.559+0000 7f5f06378640 1 -- 172.21.15.16:0/4022493707 wait complete. 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 Processor -- start 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 -- start start 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 0x7f5f00198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 0x7f5f0019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f00110e40 con 0x7f5f00100a80 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5f00110cc0 con 0x7f5f00101420 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5f06378640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f00110fc0 con 0x7f5f00106a80 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5efffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5efffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44946/0 (socket says 172.21.15.16:44946) 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5efffff640 1 -- 172.21.15.16:0/1036820779 learned_addr learned my addr 172.21.15.16:0/1036820779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.560+0000 7f5ef7fff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 0x7f5f00198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 -- 172.21.15.16:0/1036820779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 msgr2=0x7f5f0019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5f04915640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 0x7f5f0019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 0x7f5f0019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 -- 172.21.15.16:0/1036820779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 msgr2=0x7f5f00198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 0x7f5f00198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 -- 172.21.15.16:0/1036820779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0019d3a0 con 0x7f5f00100a80 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5ef7fff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 0x7f5f00198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5f04915640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 0x7f5f0019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efffff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000e9f0 tx=0x7f5ef000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000cd80 con 0x7f5f00100a80 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ef0010910 con 0x7f5f00100a80 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0019d690 con 0x7f5f00100a80 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0018990 con 0x7f5f00100a80 2024-09-12T02:51:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.561+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f001a4f70 con 0x7f5f00100a80 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.562+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f00100ee0 con 0x7f5f00100a80 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.563+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5ef0010ac0 con 0x7f5f00100a80 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.564+0000 7f5efdffb640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 0x7f5eec0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.564+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5ef00cf7b0 con 0x7f5f00100a80 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.564+0000 7f5ef7fff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 0x7f5eec0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.565+0000 7f5ef7fff640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 0x7f5eec0a50c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8000c00 tx=0x7f5ee8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.567+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5ef009c720 con 0x7f5f00100a80 2024-09-12T02:51:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:52.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 9.60 MB/sec execute 54 sec latency 18440.766 ms 2024-09-12T02:51:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.786+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f00199750 con 0x7f5f00100a80 2024-09-12T02:51:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.786+0000 7f5efdffb640 1 -- 172.21.15.16:0/1036820779 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f5ef00bc0e0 con 0x7f5f00100a80 2024-09-12T02:51:52.789 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:51:52.789 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:51:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:51:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 msgr2=0x7f5eec0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 0x7f5eec0a50c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8000c00 tx=0x7f5ee8005a20 comp rx=0 tx=0).stop 2024-09-12T02:51:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 msgr2=0x7f5f001983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000e9f0 tx=0x7f5ef000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 shutdown_connections 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f00106a80 0x7f5f0019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5f00101420 0x7f5f00198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5eec0a2c00 0x7f5eec0a50c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 --2- 172.21.15.16:0/1036820779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100a80 0x7f5f001983f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 >> 172.21.15.16:0/1036820779 conn(0x7f5f000fc7a0 msgr2=0x7f5f0006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 shutdown_connections 2024-09-12T02:51:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:52.789+0000 7f5f06378640 1 -- 172.21.15.16:0/1036820779 wait complete. 2024-09-12T02:51:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1036820779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:53.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1036820779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1036820779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:51:53.623 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 6810785f --release --await' 2024-09-12T02:51:53.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 6810785f --release --await 2024-09-12T02:51:53.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 9.42 MB/sec execute 55 sec latency 19440.809 ms 2024-09-12T02:51:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 -- 172.21.15.16:0/1912626482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518074620 msgr2=0x7fa518074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 --2- 172.21.15.16:0/1912626482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518074620 0x7fa518074a00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa514009900 tx=0x7fa51402a480 comp rx=0 tx=0).stop 2024-09-12T02:51:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 -- 172.21.15.16:0/1912626482 shutdown_connections 2024-09-12T02:51:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 --2- 172.21.15.16:0/1912626482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa51810e3c0 0x7fa518110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 --2- 172.21.15.16:0/1912626482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074f40 0x7fa51810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 --2- 172.21.15.16:0/1912626482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518074620 0x7fa518074a00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 -- 172.21.15.16:0/1912626482 >> 172.21.15.16:0/1912626482 conn(0x7fa5180fd8c0 msgr2=0x7fa5180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 -- 172.21.15.16:0/1912626482 shutdown_connections 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.805+0000 7fa51f277640 1 -- 172.21.15.16:0/1912626482 wait complete. 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 Processor -- start 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 -- start start 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074620 0x7fa51819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 0x7fa51819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa518114170 con 0x7fa51810e3c0 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa518113ff0 con 0x7fa518074620 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51f277640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181142f0 con 0x7fa518074f40 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51d814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51d013640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074620 0x7fa51819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51d814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44950/0 (socket says 172.21.15.16:44950) 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51d814640 1 -- 172.21.15.16:0/3514853038 learned_addr learned my addr 172.21.15.16:0/3514853038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.806+0000 7fa51c812640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 0x7fa51819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 -- 172.21.15.16:0/3514853038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 msgr2=0x7fa51819ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 0x7fa51819ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 -- 172.21.15.16:0/3514853038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074620 msgr2=0x7fa51819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074620 0x7fa51819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 -- 172.21.15.16:0/3514853038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51819f680 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51c812640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 0x7fa51819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51d814640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa51000eaa0 tx=0x7fa51000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa50e7fc640 1 -- 172.21.15.16:0/3514853038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51000cdb0 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa50e7fc640 1 -- 172.21.15.16:0/3514853038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa510005020 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51819f970 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa50e7fc640 1 -- 172.21.15.16:0/3514853038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100116b0 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.807+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5181a7250 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.810+0000 7fa50e7fc640 1 -- 172.21.15.16:0/3514853038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa510005560 con 0x7fa51810e3c0 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.810+0000 7fa521650740 1 Processor -- start 2024-09-12T02:51:53.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.810+0000 7fa521650740 1 -- start start 2024-09-12T02:51:53.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71b9fa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba07760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e71b9602f0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e71b907cf0 con 0x55e71ba05370 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa521650740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e71b95c2b0 con 0x55e71b9fbb40 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44956/0 (socket says 172.21.15.16:44956) 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 -- 172.21.15.16:0/919367477 learned_addr learned my addr 172.21.15.16:0/919367477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51c812640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71b9fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51c812640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba07760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 -- 172.21.15.16:0/919367477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 msgr2=0x55e71b9fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71b9fa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 -- 172.21.15.16:0/919367477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 msgr2=0x55e71ba07760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba07760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51d013640 1 -- 172.21.15.16:0/919367477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e71b8ec3a0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa51c812640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71b9fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.811+0000 7fa50e7fc640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 0x7fa5040a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa51d013640 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa514009900 tx=0x7fa514005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cd724db8e35ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa51c812640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 0x7fa5040a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa50e7fc640 1 -- 172.21.15.16:0/3514853038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa5100cf160 con 0x7fa51810e3c0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa5037fe640 1 -- 172.21.15.16:0/919367477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514040070 con 0x55e71b9fa5d0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa5037fe640 1 -- 172.21.15.16:0/919367477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5140319f0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.812+0000 7fa5037fe640 1 -- 172.21.15.16:0/919367477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514035040 con 0x55e71b9fa5d0 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 -- 172.21.15.16:0/919367477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 msgr2=0x55e71b9faa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa51c812640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 0x7fa5040a5040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e71b9fb070 tx=0x7fa50800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa514009900 tx=0x7fa514005e80 comp rx=0 tx=0).stop 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 -- 172.21.15.16:0/919367477 shutdown_connections 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba07760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9faa10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 --2- 172.21.15.16:0/919367477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71b9fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 -- 172.21.15.16:0/919367477 >> 172.21.15.16:0/919367477 conn(0x55e71b9c3310 msgr2=0x55e71b9f6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.813+0000 7fa521650740 1 -- 172.21.15.16:0/919367477 shutdown_connections 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.814+0000 7fa521650740 1 -- 172.21.15.16:0/919367477 wait complete. 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.814+0000 7fa521650740 1 -- start start 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.814+0000 7fa521650740 1 Processor -- start 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 0x55e71ba04cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fbb40 0x55e71ba01fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 0x55e71ba02520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e71ba08060 con 0x55e71ba05370 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e71b91b060 con 0x55e71b9fbb40 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e71ba081e0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa521650740 10 client.0 _ll_register_callbacks cb 0x55e71b66e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d013640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 0x55e71ba04cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d013640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 0x55e71ba04cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44176/0 (socket says 172.21.15.16:44176) 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d013640 1 -- 172.21.15.16:0/3069064891 learned_addr learned my addr 172.21.15.16:0/3069064891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 0x55e71ba02520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51c812640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fbb40 0x55e71ba01fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 -- 172.21.15.16:0/3069064891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 msgr2=0x55e71ba04cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 0x55e71ba04cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 -- 172.21.15.16:0/3069064891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fbb40 msgr2=0x55e71ba01fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fbb40 0x55e71ba01fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.815+0000 7fa51d814640 1 -- 172.21.15.16:0/3069064891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e71b6d13c0 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa51d814640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 0x55e71ba02520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa51000efd0 tx=0x7fa51009bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df616cf76d123a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa500ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51001f070 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa5100b4dd0 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5140319f0 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5100ba1e0 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa521650740 10 client.16347 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.816+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e71ba13af0 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5100ba380 con 0x55e71ba05370 2024-09-12T02:51:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa51000f040 con 0x55e71ba05370 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 10 client.16347 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 10 client.16347 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 20 client.16347 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba17450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 4 client.16347 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa5140319f0 con 0x55e71ba15000 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.817+0000 7fa51d013640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba17450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.818+0000 7fa51d013640 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba17450 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a323aa7be88f2eba server_cookie=ba9d5c2cf7aa789e in_seq=0 out_seq=0 2024-09-12T02:51:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.818+0000 7fa500ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.818+0000 7fa500ff9640 1 -- 172.21.15.16:0/3069064891 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fa5140319f0 con 0x55e71ba15000 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.818+0000 7fa500ff9640 10 client.16347 handle_command_reply: tid=0 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.818+0000 7fa521650740 1 client.16347 shutdown 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 msgr2=0x55e71ba02520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 0x55e71ba02520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa51000efd0 tx=0x7fa51009bbc0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 shutdown_connections 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71ba05370 0x55e71ba02520 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba17450 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa500ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fbb40 0x55e71ba01fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 --2- 172.21.15.16:0/3069064891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fa5d0 0x55e71ba04cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 >> 172.21.15.16:0/3069064891 conn(0x55e71b9c3310 msgr2=0x55e71b9fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 shutdown_connections 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 1 -- 172.21.15.16:0/3069064891 wait complete. 2024-09-12T02:51:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.819+0000 7fa521650740 20 client.16347 trim_cache size 0 max 16384 2024-09-12T02:51:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 Processor -- start 2024-09-12T02:51:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 -- start start 2024-09-12T02:51:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71ba11300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba1cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181af7d0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181b1310 con 0x55e71ba05370 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa521650740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181afea0 con 0x55e71b9fbb40 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa51d013640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa51d013640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44972/0 (socket says 172.21.15.16:44972) 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa51d013640 1 -- 172.21.15.16:0/3151302763 learned_addr learned my addr 172.21.15.16:0/3151302763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.834+0000 7fa51d814640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba1cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51c812640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71ba11300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 -- 172.21.15.16:0/3151302763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 msgr2=0x55e71ba11300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71ba11300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 -- 172.21.15.16:0/3151302763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 msgr2=0x55e71ba1cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba1cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 -- 172.21.15.16:0/3151302763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e71ba081d0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51c812640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71ba11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d814640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba1cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa51d013640 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa514009900 tx=0x7fa514047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262f07199a3ac456 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa502ffd640 1 -- 172.21.15.16:0/3151302763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514040040 con 0x55e71b9fa5d0 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa502ffd640 1 -- 172.21.15.16:0/3151302763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa51403cba0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.835+0000 7fa502ffd640 1 -- 172.21.15.16:0/3151302763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514035040 con 0x55e71b9fa5d0 2024-09-12T02:51:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- 172.21.15.16:0/3151302763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 msgr2=0x55e71b9f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa514009900 tx=0x7fa514047040 comp rx=0 tx=0).stop 2024-09-12T02:51:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- 172.21.15.16:0/3151302763 shutdown_connections 2024-09-12T02:51:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71ba05370 0x55e71ba1cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71b9fbb40 0x55e71ba11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 --2- 172.21.15.16:0/3151302763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fa5d0 0x55e71b9f6e10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- 172.21.15.16:0/3151302763 >> 172.21.15.16:0/3151302763 conn(0x55e71b9c3310 msgr2=0x55e71b9f6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- 172.21.15.16:0/3151302763 shutdown_connections 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- 172.21.15.16:0/3151302763 wait complete. 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 -- start start 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.836+0000 7fa521650740 1 Processor -- start 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fbb40 0x55e71b9ff680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71ba05370 0x55e71b9ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181b0c40 con 0x55e71b9fbb40 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181b0570 con 0x55e71b9fa5d0 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181ad7b0 con 0x55e71ba05370 2024-09-12T02:51:53.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa521650740 10 client.0 _ll_register_callbacks cb 0x7fa510097d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa51d013640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa51d013640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34628/0 (socket says 172.21.15.16:34628) 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa51d013640 1 -- 172.21.15.16:0/2227115569 learned_addr learned my addr 172.21.15.16:0/2227115569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa51c812640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fbb40 0x55e71b9ff680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.837+0000 7fa51d814640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71ba05370 0x55e71b9ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 -- 172.21.15.16:0/2227115569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71ba05370 msgr2=0x55e71b9ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71ba05370 0x55e71b9ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 -- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fbb40 msgr2=0x55e71b9ff680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fbb40 0x55e71b9ff680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 -- 172.21.15.16:0/2227115569 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e71ba7ee90 con 0x55e71b9fa5d0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa51d013640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa514031670 tx=0x7fa514005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658966aba9432078 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa500ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51405d070 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa51403c8d0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.838+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51403cba0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa514040070 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 10 client.25925 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e71ba800c0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa514005040 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa5140046c0 con 0x55e71b9fa5d0 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 10 client.25925 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 10 client.25925 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 20 client.25925 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba823b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 4 client.25925 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:53.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa51403cba0 con 0x55e71ba15000 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.839+0000 7fa51c812640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba823b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.840+0000 7fa500ff9640 10 client.25925 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.840+0000 7fa51c812640 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba823b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bb058a756576f37e server_cookie=7d5cdfa4c34bdd88 in_seq=0 out_seq=0 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.840+0000 7fa500ff9640 1 -- 172.21.15.16:0/2227115569 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa51403cba0 con 0x55e71ba15000 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.840+0000 7fa500ff9640 10 client.25925 handle_command_reply: tid=0 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.840+0000 7fa521650740 1 client.25925 shutdown 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 msgr2=0x55e71b9ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa514031670 tx=0x7fa514005b50 comp rx=0 tx=0).stop 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 shutdown_connections 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e71ba05370 0x55e71b9ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e71b9fbb40 0x55e71b9ff680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e71ba15000 0x55e71ba823b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 --2- 172.21.15.16:0/2227115569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e71b9fa5d0 0x55e71b9ff140 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 >> 172.21.15.16:0/2227115569 conn(0x55e71b9c3310 msgr2=0x55e71ba03800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa500ff9640 0 client.25925 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 shutdown_connections 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 1 -- 172.21.15.16:0/2227115569 wait complete. 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.841+0000 7fa521650740 20 client.25925 trim_cache size 0 max 16384 2024-09-12T02:51:53.848 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 msgr2=0x7fa5040a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 0x7fa5040a5040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e71b9fb070 tx=0x7fa50800b410 comp rx=0 tx=0).stop 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 msgr2=0x7fa51819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa51000eaa0 tx=0x7fa51000ef60 comp rx=0 tx=0).stop 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 shutdown_connections 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5040a2b80 0x7fa5040a5040 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa51810e3c0 0x7fa51819efa0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa518074f40 0x7fa51819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 --2- 172.21.15.16:0/3514853038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa518074620 0x7fa51819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.842+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 >> 172.21.15.16:0/3514853038 conn(0x7fa5180fd8c0 msgr2=0x7fa5180fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.843+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 shutdown_connections 2024-09-12T02:51:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:53.843+0000 7fa51f277640 1 -- 172.21.15.16:0/3514853038 wait complete. 2024-09-12T02:51:54.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:53 smithi016 ceph-mon[30925]: pgmap v456: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-12T02:51:54.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:53 smithi033 ceph-mon[35092]: pgmap v456: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-12T02:51:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:53 smithi184 ceph-mon[35321]: pgmap v456: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-12T02:51:54.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:51:54.621 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:54.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318094 9.25 MB/sec execute 56 sec latency 20440.858 ms 2024-09-12T02:51:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 -- 172.21.15.16:0/1079196838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc102570 msgr2=0x7f26dc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1079196838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc102570 0x7f26dc1029b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009900 tx=0x7f26cc02a480 comp rx=0 tx=0).stop 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 -- 172.21.15.16:0/1079196838 shutdown_connections 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1079196838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102ef0 0x7f26dc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1079196838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc102570 0x7f26dc1029b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1079196838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc100e20 0x7f26dc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 -- 172.21.15.16:0/1079196838 >> 172.21.15.16:0/1079196838 conn(0x7f26dc0fc7d0 msgr2=0x7f26dc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 -- 172.21.15.16:0/1079196838 shutdown_connections 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.051+0000 7f26e3f44640 1 -- 172.21.15.16:0/1079196838 wait complete. 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 Processor -- start 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 -- start start 2024-09-12T02:51:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102570 0x7f26dc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc102ef0 0x7f26dc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc110f80 con 0x7f26dc100e20 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc110e00 con 0x7f26dc102570 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.052+0000 7f26e3f44640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc111100 con 0x7f26dc102ef0 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44990/0 (socket says 172.21.15.16:44990) 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 -- 172.21.15.16:0/1014559517 learned_addr learned my addr 172.21.15.16:0/1014559517 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e14df640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102570 0x7f26dc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e24e1640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc102ef0 0x7f26dc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 -- 172.21.15.16:0/1014559517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc102ef0 msgr2=0x7f26dc0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc102ef0 0x7f26dc0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 -- 172.21.15.16:0/1014559517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102570 msgr2=0x7f26dc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102570 0x7f26dc0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 -- 172.21.15.16:0/1014559517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc1a6970 con 0x7f26dc100e20 2024-09-12T02:51:55.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f26d800d950 tx=0x7f26d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.053+0000 7f26d2ffd640 1 -- 172.21.15.16:0/1014559517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8017070 con 0x7f26dc100e20 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.054+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc1a6c60 con 0x7f26dc100e20 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.054+0000 7f26d2ffd640 1 -- 172.21.15.16:0/1014559517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d8009c30 con 0x7f26dc100e20 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.054+0000 7f26d2ffd640 1 -- 172.21.15.16:0/1014559517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d800f6b0 con 0x7f26dc100e20 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.054+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc1a71c0 con 0x7f26dc100e20 2024-09-12T02:51:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.056+0000 7f26e631d740 1 Processor -- start 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.056+0000 7f26e631d740 1 -- start start 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee31440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee3bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632fed96620 con 0x5632fee3c240 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632fed3de50 con 0x5632fee31060 2024-09-12T02:51:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e631d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5632fed925e0 con 0x5632fee31980 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45006/0 (socket says 172.21.15.16:45006) 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 -- 172.21.15.16:0/4222468687 learned_addr learned my addr 172.21.15.16:0/4222468687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e24e1640 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee31440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26d2ffd640 1 -- 172.21.15.16:0/1014559517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f26d8010460 con 0x7f26dc100e20 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 -- 172.21.15.16:0/4222468687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 msgr2=0x5632fee3bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee3bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 -- 172.21.15.16:0/4222468687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 msgr2=0x5632fee31440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee31440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.057+0000 7f26e14df640 1 -- 172.21.15.16:0/4222468687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632fed2ed40 con 0x5632fee3c240 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26e24e1640 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee31440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26e14df640 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009900 tx=0x7f26cc004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae5439e57e27437 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26c7fff640 1 -- 172.21.15.16:0/4222468687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc035030 con 0x5632fee3c240 2024-09-12T02:51:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26c7fff640 1 -- 172.21.15.16:0/4222468687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc040070 con 0x5632fee3c240 2024-09-12T02:51:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26c7fff640 1 -- 172.21.15.16:0/4222468687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc031b70 con 0x5632fee3c240 2024-09-12T02:51:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26d2ffd640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 0x7f26c80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26e24e1640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 0x7f26c80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.058+0000 7f26d2ffd640 1 -- 172.21.15.16:0/1014559517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26d80cebf0 con 0x7f26dc100e20 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 -- 172.21.15.16:0/4222468687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 msgr2=0x5632fee3e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e24e1640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 0x7f26c80a5040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f26d4010540 tx=0x7f26d400a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009900 tx=0x7f26cc004170 comp rx=0 tx=0).stop 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 -- 172.21.15.16:0/4222468687 shutdown_connections 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee3c240 0x5632fee3e630 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee3bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 --2- 172.21.15.16:0/4222468687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee31440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 -- 172.21.15.16:0/4222468687 >> 172.21.15.16:0/4222468687 conn(0x5632fee2d5f0 msgr2=0x5632fee2d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 -- 172.21.15.16:0/4222468687 shutdown_connections 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.059+0000 7f26e631d740 1 -- 172.21.15.16:0/4222468687 wait complete. 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 -- start start 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 Processor -- start 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee396f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee3c240 0x5632fee39c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632fee3ee10 con 0x5632fee31060 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632fed1ef10 con 0x5632fee3c240 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5632fee3ef90 con 0x5632fee31980 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.060+0000 7f26e631d740 10 client.0 _ll_register_callbacks cb 0x5632fed42db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45008/0 (socket says 172.21.15.16:45008) 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e1ce0640 1 -- 172.21.15.16:0/1228231450 learned_addr learned my addr 172.21.15.16:0/1228231450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e24e1640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee3c240 0x5632fee39c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 -- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee3c240 msgr2=0x5632fee39c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee3c240 0x5632fee39c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 -- 172.21.15.16:0/1228231450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 msgr2=0x5632fee391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 -- 172.21.15.16:0/1228231450 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632feaa5580 con 0x5632fee31980 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26e14df640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee396f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f26cc005940 tx=0x7f26cc031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37060f100179dd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.061+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc04d070 con 0x5632fee31980 2024-09-12T02:51:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26cc040070 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26cc005020 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26cc058050 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 10 client.25825 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632feb6f960 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc040070 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26cc0035f0 con 0x5632fee31980 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 10 client.25825 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 10 client.25825 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 20 client.25825 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee4bcf0 0x5632fee4e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 4 client.25825 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.062+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632feda3b90 con 0x5632fee4bcf0 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.063+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee4bcf0 0x5632fee4e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.063+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee4bcf0 0x5632fee4e0e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdcbeccad7428a5 server_cookie=7b09ba290b14914d in_seq=0 out_seq=0 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.063+0000 7f26c57fa640 10 client.25825 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26c57fa640 1 -- 172.21.15.16:0/1228231450 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632feda3b90 con 0x5632fee4bcf0 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26c57fa640 10 client.25825 handle_command_reply: tid=0 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 client.25825 shutdown 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 msgr2=0x5632fee396f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee396f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f26cc005940 tx=0x7f26cc031630 comp rx=0 tx=0).stop 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 shutdown_connections 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee4bcf0 0x5632fee4e0e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee3c240 0x5632fee39c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee396f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26c57fa640 0 client.25825 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 --2- 172.21.15.16:0/1228231450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31060 0x5632fee391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 >> 172.21.15.16:0/1228231450 conn(0x5632fee2d5f0 msgr2=0x5632fee332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 shutdown_connections 2024-09-12T02:51:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 1 -- 172.21.15.16:0/1228231450 wait complete. 2024-09-12T02:51:55.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.064+0000 7f26e631d740 20 client.25825 trim_cache size 0 max 16384 2024-09-12T02:51:55.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 Processor -- start 2024-09-12T02:51:55.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 -- start start 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee2d5f0 0x5632fee344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 0x5632fee47920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1af770 con 0x5632fee31980 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b12b0 con 0x5632fee2d5f0 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.083+0000 7f26e631d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1afe40 con 0x5632fee31060 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e24e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e24e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45018/0 (socket says 172.21.15.16:45018) 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e24e1640 1 -- 172.21.15.16:0/2615818896 learned_addr learned my addr 172.21.15.16:0/2615818896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 0x5632fee47920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee2d5f0 0x5632fee344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 -- 172.21.15.16:0/2615818896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee2d5f0 msgr2=0x5632fee344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee2d5f0 0x5632fee344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 -- 172.21.15.16:0/2615818896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 msgr2=0x5632fee36410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 -- 172.21.15.16:0/2615818896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632fee3b0f0 con 0x5632fee31060 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e24e1640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.084+0000 7f26e14df640 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 0x5632fee47920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0314e0 tx=0x7f26cc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c3280549e4b6fd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.085+0000 7f26c77fe640 1 -- 172.21.15.16:0/2615818896 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc04d070 con 0x5632fee31060 2024-09-12T02:51:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.085+0000 7f26c77fe640 1 -- 172.21.15.16:0/2615818896 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc040040 con 0x5632fee31060 2024-09-12T02:51:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.085+0000 7f26c77fe640 1 -- 172.21.15.16:0/2615818896 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc035050 con 0x5632fee31060 2024-09-12T02:51:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.085+0000 7f26e631d740 1 -- 172.21.15.16:0/2615818896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 msgr2=0x5632fee47920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.085+0000 7f26e631d740 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 0x5632fee47920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0314e0 tx=0x7f26cc000bc0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 -- 172.21.15.16:0/2615818896 shutdown_connections 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee31980 0x5632fee36410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31060 0x5632fee47920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 --2- 172.21.15.16:0/2615818896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee2d5f0 0x5632fee344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 -- 172.21.15.16:0/2615818896 >> 172.21.15.16:0/2615818896 conn(0x5632fee34ba0 msgr2=0x5632fee47210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 -- 172.21.15.16:0/2615818896 shutdown_connections 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 -- 172.21.15.16:0/2615818896 wait complete. 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 -- start start 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.086+0000 7f26e631d740 1 Processor -- start 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee51670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee4f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b0be0 con 0x5632fee2d5f0 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b0510 con 0x5632fee31060 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1ad750 con 0x5632fee31980 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e631d740 10 client.0 _ll_register_callbacks cb 0x7f26cc02ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e24e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee4f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45022/0 (socket says 172.21.15.16:45022) 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 -- 172.21.15.16:0/2967513771 learned_addr learned my addr 172.21.15.16:0/2967513771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e14df640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee51670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 -- 172.21.15.16:0/2967513771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 msgr2=0x5632fee4f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee4f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 -- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 msgr2=0x5632fee51670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee51670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e1ce0640 1 -- 172.21.15.16:0/2967513771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632fee4fda0 con 0x5632fee2d5f0 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e14df640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee51670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.087+0000 7f26e24e1640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee4f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26e1ce0640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26d809b210 tx=0x7f26d8086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c0348d9a755967 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d80b4060 con 0x5632fee2d5f0 2024-09-12T02:51:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26d8099810 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26cc040040 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26d8020040 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26e631d740 10 client.16371 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632feeb7160 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc040040 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.088+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26d80109d0 con 0x5632fee2d5f0 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 10 client.16371 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 10 client.16371 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 20 client.16371 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee3c240 0x5632feeb9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 4 client.16371 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6810785f", "release": true, "await": true}] 2024-09-12T02:51:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6810785f", "release": true, "await": true}) -- 0x5632fed9e0a0 con 0x5632fee3c240 2024-09-12T02:51:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e14df640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee3c240 0x5632feeb9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26e14df640 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee3c240 0x5632feeb9850 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18249d73b72839f3 server_cookie=d6646287d7737fb7 in_seq=0 out_seq=0 2024-09-12T02:51:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.089+0000 7f26c57fa640 10 client.16371 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26c57fa640 1 -- 172.21.15.16:0/2967513771 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5632fed9e0a0 con 0x5632fee3c240 2024-09-12T02:51:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26c57fa640 10 client.16371 handle_command_reply: tid=0 2024-09-12T02:51:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 client.16371 shutdown 2024-09-12T02:51:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 msgr2=0x5632fee51130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26d809b210 tx=0x7f26d8086040 comp rx=0 tx=0).stop 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 shutdown_connections 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632fee3c240 0x5632feeb9850 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26c57fa640 0 client.16371 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632fee31980 0x5632fee4f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632fee31060 0x5632fee51670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 --2- 172.21.15.16:0/2967513771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632fee2d5f0 0x5632fee51130 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 >> 172.21.15.16:0/2967513771 conn(0x5632fee34ba0 msgr2=0x5632fee332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 shutdown_connections 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.100+0000 7f26e631d740 1 -- 172.21.15.16:0/2967513771 wait complete. 2024-09-12T02:51:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e631d740 20 client.16371 trim_cache size 0 max 16384 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 16, 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "6810785f": { 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 16, 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:51:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.2, 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:51:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:51:55.106 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 msgr2=0x7f26c80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 0x7f26c80a5040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f26d4010540 tx=0x7f26d400a4d0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 msgr2=0x7f26dc0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f26d800d950 tx=0x7f26d800de10 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 shutdown_connections 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26dc102ef0 0x7f26dc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26dc102570 0x7f26dc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26c80a2b80 0x7f26c80a5040 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 --2- 172.21.15.16:0/1014559517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26dc100e20 0x7f26dc0fcc80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.101+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 >> 172.21.15.16:0/1014559517 conn(0x7f26dc0fc7d0 msgr2=0x7f26dc06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.102+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 shutdown_connections 2024-09-12T02:51:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.102+0000 7f26e3f44640 1 -- 172.21.15.16:0/1014559517 wait complete. 2024-09-12T02:51:55.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 319542 9.30 MB/sec execute 57 sec latency 20828.883 ms 2024-09-12T02:51:55.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 -- 172.21.15.16:0/3596474286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 msgr2=0x7f6d481043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 --2- 172.21.15.16:0/3596474286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d481043d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009900 tx=0x7f6d4402a410 comp rx=0 tx=0).stop 2024-09-12T02:51:55.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 -- 172.21.15.16:0/3596474286 shutdown_connections 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 --2- 172.21.15.16:0/3596474286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d48110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 --2- 172.21.15.16:0/3596474286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 0x7f6d481089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 --2- 172.21.15.16:0/3596474286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d481043d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.773+0000 7f6d50090640 1 -- 172.21.15.16:0/3596474286 >> 172.21.15.16:0/3596474286 conn(0x7f6d480ffab0 msgr2=0x7f6d48101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.774+0000 7f6d50090640 1 -- 172.21.15.16:0/3596474286 shutdown_connections 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.774+0000 7f6d50090640 1 -- 172.21.15.16:0/3596474286 wait complete. 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 Processor -- start 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 -- start start 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 0x7f6d4819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d4819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48114190 con 0x7f6d48103ff0 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48114010 con 0x7f6d48109120 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d50090640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48114310 con 0x7f6d481049a0 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47546/0 (socket says 172.21.15.16:47546) 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 -- 172.21.15.16:0/1324483712 learned_addr learned my addr 172.21.15.16:0/1324483712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 0x7f6d4819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d4819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 -- 172.21.15.16:0/1324483712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 msgr2=0x7f6d4819abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 0x7f6d4819abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 -- 172.21.15.16:0/1324483712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 msgr2=0x7f6d4819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d4819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4de2c640 1 -- 172.21.15.16:0/1324483712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4819f640 con 0x7f6d48103ff0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.775+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d4819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004770 tx=0x7f6d44003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d3effd640 1 -- 172.21.15.16:0/1324483712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403f070 con 0x7f6d48103ff0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4819f8d0 con 0x7f6d48103ff0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d3effd640 1 -- 172.21.15.16:0/1324483712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44031ca0 con 0x7f6d48103ff0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d3effd640 1 -- 172.21.15.16:0/1324483712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44035030 con 0x7f6d48103ff0 2024-09-12T02:51:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.776+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d481a7210 con 0x7f6d48103ff0 2024-09-12T02:51:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d3effd640 1 -- 172.21.15.16:0/1324483712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6d44042070 con 0x7f6d48103ff0 2024-09-12T02:51:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 Processor -- start 2024-09-12T02:51:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 -- start start 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cb53472f0 con 0x558cb53ec370 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558cb52eecf0 con 0x558cb53e2b40 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.779+0000 7f6d52469740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558cb53432b0 con 0x558cb53e15d0 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d3effd640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 0x7f6d340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32964/0 (socket says 172.21.15.16:32964) 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3137693972 learned_addr learned my addr 172.21.15.16:0/3137693972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 0x7f6d340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 -- 172.21.15.16:0/3137693972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 msgr2=0x558cb53e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d3effd640 1 -- 172.21.15.16:0/1324483712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6d440ef5f0 con 0x7f6d48103ff0 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 -- 172.21.15.16:0/3137693972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 msgr2=0x558cb53e1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 -- 172.21.15.16:0/3137693972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb52d33a0 con 0x558cb53ec370 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53ee760 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009bd0 tx=0x7f6d38011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5aecc4ad735bdd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.780+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 0x7f6d340a5040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009fd0 tx=0x7f6d40009f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.781+0000 7f6d33fff640 1 -- 172.21.15.16:0/3137693972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38004230 con 0x558cb53ec370 2024-09-12T02:51:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.781+0000 7f6d33fff640 1 -- 172.21.15.16:0/3137693972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d38009e40 con 0x558cb53ec370 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.781+0000 7f6d33fff640 1 -- 172.21.15.16:0/3137693972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38004d50 con 0x558cb53ec370 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.781+0000 7f6d52469740 1 -- 172.21.15.16:0/3137693972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 msgr2=0x558cb53ee760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.781+0000 7f6d52469740 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53ee760 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009bd0 tx=0x7f6d38011810 comp rx=0 tx=0).stop 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 -- 172.21.15.16:0/3137693972 shutdown_connections 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53ee760 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 --2- 172.21.15.16:0/3137693972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 -- 172.21.15.16:0/3137693972 >> 172.21.15.16:0/3137693972 conn(0x558cb53aa310 msgr2=0x558cb53ddc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 -- 172.21.15.16:0/3137693972 shutdown_connections 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 -- 172.21.15.16:0/3137693972 wait complete. 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 -- start start 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.782+0000 7f6d52469740 1 Processor -- start 2024-09-12T02:51:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cb53ef070 con 0x558cb53ec370 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558cb5302060 con 0x558cb53e2b40 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558cb53ef1f0 con 0x558cb53e15d0 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d52469740 10 client.0 _ll_register_callbacks cb 0x558cb50558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32974/0 (socket says 172.21.15.16:32974) 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58860/0 (socket says 172.21.15.16:58860) 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3425686745 learned_addr learned my addr 172.21.15.16:0/3425686745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 msgr2=0x558cb53e8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3425686745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 msgr2=0x558cb53e94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53e94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3425686745 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb50b83c0 con 0x558cb53e15d0 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6d440b89a0 tx=0x7f6d440b6970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5396e0025e78943 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.783+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44049070 con 0x558cb53e15d0 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6d4403b4f0 con 0x558cb53e15d0 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558cb5354860 con 0x558cb53e15d0 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6d440bbb10 con 0x558cb53e15d0 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 10 client.25843 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558cb53ef0d0 con 0x558cb53e15d0 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558cb5354860 con 0x558cb53e15d0 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6d440a6ce0 con 0x558cb53e15d0 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 10 client.25843 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 10 client.25843 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 20 client.25843 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb53fbe70 0x558cb53fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 4 client.25843 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cb534ed70 con 0x558cb53fbe70 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.784+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb53fbe70 0x558cb53fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.785+0000 7f6d317fa640 10 client.25843 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.785+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb53fbe70 0x558cb53fe2e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3086cf9f03371609 server_cookie=cfaa093d09e44822 in_seq=0 out_seq=0 2024-09-12T02:51:55.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d317fa640 1 -- 172.21.15.16:0/3425686745 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558cb534ed70 con 0x558cb53fbe70 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d317fa640 10 client.25843 handle_command_reply: tid=0 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 client.25843 shutdown 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 msgr2=0x558cb53ebcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6d440b89a0 tx=0x7f6d440b6970 comp rx=0 tx=0).stop 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 shutdown_connections 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb53fbe70 0x558cb53fe2e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53ec370 0x558cb53e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53e8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 --2- 172.21.15.16:0/3425686745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53e15d0 0x558cb53ebcd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 >> 172.21.15.16:0/3425686745 conn(0x558cb53aa310 msgr2=0x558cb53e3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d317fa640 0 client.25843 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 shutdown_connections 2024-09-12T02:51:55.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 1 -- 172.21.15.16:0/3425686745 wait complete. 2024-09-12T02:51:55.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.786+0000 7f6d52469740 20 client.25843 trim_cache size 0 max 16384 2024-09-12T02:51:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.799+0000 7f6d52469740 1 Processor -- start 2024-09-12T02:51:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.799+0000 7f6d52469740 1 -- start start 2024-09-12T02:51:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e15d0 0x558cb53dde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53e8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481af6d0 con 0x558cb53e2b40 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481b1210 con 0x558cb53e15d0 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d52469740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481afda0 con 0x558cb53ec370 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47566/0 (socket says 172.21.15.16:47566) 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e15d0 0x558cb53dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 -- 172.21.15.16:0/919474482 learned_addr learned my addr 172.21.15.16:0/919474482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 -- 172.21.15.16:0/919474482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 msgr2=0x558cb53e8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53e8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 -- 172.21.15.16:0/919474482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e15d0 msgr2=0x558cb53dde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e15d0 0x558cb53dde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 -- 172.21.15.16:0/919474482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb53e8d50 con 0x558cb53e2b40 2024-09-12T02:51:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.800+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x558cb53e2070 tx=0x7f6d38005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e808ba918a289a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d337fe640 1 -- 172.21.15.16:0/919474482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3801e040 con 0x558cb53e2b40 2024-09-12T02:51:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d337fe640 1 -- 172.21.15.16:0/919474482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d38011d40 con 0x558cb53e2b40 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d337fe640 1 -- 172.21.15.16:0/919474482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3802a920 con 0x558cb53e2b40 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 -- 172.21.15.16:0/919474482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 msgr2=0x558cb53f8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x558cb53e2070 tx=0x7f6d38005f90 comp rx=0 tx=0).stop 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 -- 172.21.15.16:0/919474482 shutdown_connections 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53e8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e2b40 0x558cb53f8310 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 --2- 172.21.15.16:0/919474482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e15d0 0x558cb53dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 -- 172.21.15.16:0/919474482 >> 172.21.15.16:0/919474482 conn(0x558cb53aa310 msgr2=0x558cb53dd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 -- 172.21.15.16:0/919474482 shutdown_connections 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.801+0000 7f6d52469740 1 -- 172.21.15.16:0/919474482 wait complete. 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 -- start start 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 Processor -- start 2024-09-12T02:51:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481b0b40 con 0x558cb53e15d0 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481b0470 con 0x558cb53e2b40 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481ad6b0 con 0x558cb53ec370 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d52469740 10 client.0 _ll_register_callbacks cb 0x7f6d440b88a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4de2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4d62b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58876/0 (socket says 172.21.15.16:58876) 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4d62b640 1 -- 172.21.15.16:0/3528845576 learned_addr learned my addr 172.21.15.16:0/3528845576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47568/0 (socket says 172.21.15.16:47568) 2024-09-12T02:51:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.802+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3528845576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 msgr2=0x558cb53ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 msgr2=0x558cb53f4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 -- 172.21.15.16:0/3528845576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb53f01e0 con 0x558cb53e15d0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4e62d640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d4de2c640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6d440bbf60 tx=0x7f6d4403b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a24305e3336c3b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44049070 con 0x558cb53e15d0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6d4403b4f0 con 0x558cb53e15d0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d38011d40 con 0x558cb53e15d0 2024-09-12T02:51:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6d44005d50 con 0x558cb53e15d0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d52469740 10 client.16389 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558cb54665d0 con 0x558cb53e15d0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.803+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d38011d40 con 0x558cb53e15d0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6d440ad020 con 0x558cb53e15d0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 10 client.16389 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 10 client.16389 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 20 client.16389 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb5466ae0 0x558cb5468f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 4 client.16389 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558cb52cb170 con 0x558cb5466ae0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb5466ae0 0x558cb5468f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d317fa640 10 client.16389 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.804+0000 7f6d4d62b640 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb5466ae0 0x558cb5468f00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83cea6d4591d0806 server_cookie=fd3bdf37d68c0414 in_seq=0 out_seq=0 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d317fa640 1 -- 172.21.15.16:0/3528845576 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558cb52cb170 con 0x558cb5466ae0 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d317fa640 10 client.16389 handle_command_reply: tid=0 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 client.16389 shutdown 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 msgr2=0x558cb53f4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6d440bbf60 tx=0x7f6d4403b460 comp rx=0 tx=0).stop 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 shutdown_connections 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558cb5466ae0 0x558cb5468f00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558cb53ec370 0x558cb53ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558cb53e2b40 0x558cb53f4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d317fa640 0 client.16389 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 --2- 172.21.15.16:0/3528845576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cb53e15d0 0x558cb53f4540 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 >> 172.21.15.16:0/3528845576 conn(0x558cb53aa310 msgr2=0x558cb53dc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 shutdown_connections 2024-09-12T02:51:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 1 -- 172.21.15.16:0/3528845576 wait complete. 2024-09-12T02:51:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d52469740 20 client.16389 trim_cache size 0 max 16384 2024-09-12T02:51:55.812 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.805+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 msgr2=0x7f6d340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 0x7f6d340a5040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009fd0 tx=0x7f6d40009f60 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 msgr2=0x7f6d4819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004770 tx=0x7f6d44003c10 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 shutdown_connections 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d48109120 0x7f6d4819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d481049a0 0x7f6d4819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6d340a2b80 0x7f6d340a5040 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 --2- 172.21.15.16:0/1324483712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d48103ff0 0x7f6d4819a690 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 >> 172.21.15.16:0/1324483712 conn(0x7f6d480ffab0 msgr2=0x7f6d48100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 shutdown_connections 2024-09-12T02:51:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:55.806+0000 7f6d50090640 1 -- 172.21.15.16:0/1324483712 wait complete. 2024-09-12T02:51:56.124 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:55 smithi016 ceph-mon[30925]: pgmap v457: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.3 MiB/s wr, 85 op/s 2024-09-12T02:51:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:55 smithi033 ceph-mon[35092]: pgmap v457: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.3 MiB/s wr, 85 op/s 2024-09-12T02:51:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:55 smithi184 ceph-mon[35321]: pgmap v457: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.3 MiB/s wr, 85 op/s 2024-09-12T02:51:56.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:51:56.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 322623 9.39 MB/sec execute 58 sec latency 19.274 ms 2024-09-12T02:51:57.109 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6810785f', total seconds elapsed: 25.5 2024-09-12T02:51:57.109 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.15 2024-09-12T02:51:57.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:51:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 -- 172.21.15.16:0/1295970885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0102570 msgr2=0x7f2bf01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/1295970885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0102570 0x7f2bf01029b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009900 tx=0x7f2be002a480 comp rx=0 tx=0).stop 2024-09-12T02:51:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 -- 172.21.15.16:0/1295970885 shutdown_connections 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/1295970885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102ef0 0x7f2bf010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/1295970885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0102570 0x7f2bf01029b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/1295970885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0100e20 0x7f2bf0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 -- 172.21.15.16:0/1295970885 >> 172.21.15.16:0/1295970885 conn(0x7f2bf00fc7d0 msgr2=0x7f2bf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 -- 172.21.15.16:0/1295970885 shutdown_connections 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.638+0000 7f2bf83fa640 1 -- 172.21.15.16:0/1295970885 wait complete. 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 Processor -- start 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 -- start start 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102570 0x7f2bf00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0102ef0 0x7f2bf00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf0110f80 con 0x7f2bf0100e20 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf0110e00 con 0x7f2bf0102ef0 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf83fa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf0111100 con 0x7f2bf0102570 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47590/0 (socket says 172.21.15.16:47590) 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 -- 172.21.15.16:0/3397695855 learned_addr learned my addr 172.21.15.16:0/3397695855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6997640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0102ef0 0x7f2bf00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf5995640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102570 0x7f2bf00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 -- 172.21.15.16:0/3397695855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102570 msgr2=0x7f2bf00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102570 0x7f2bf00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 -- 172.21.15.16:0/3397695855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0102ef0 msgr2=0x7f2bf00fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0102ef0 0x7f2bf00fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.639+0000 7f2bf6196640 1 -- 172.21.15.16:0/3397695855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf01a6990 con 0x7f2bf0100e20 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2bf6196640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00d950 tx=0x7f2bec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2be77fe640 1 -- 172.21.15.16:0/3397695855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec017030 con 0x7f2bf0100e20 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2be77fe640 1 -- 172.21.15.16:0/3397695855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bec004020 con 0x7f2bf0100e20 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf01a6c80 con 0x7f2bf0100e20 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2be77fe640 1 -- 172.21.15.16:0/3397695855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0047a0 con 0x7f2bf0100e20 2024-09-12T02:51:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.640+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf01a7190 con 0x7f2bf0100e20 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2be77fe640 1 -- 172.21.15.16:0/3397695855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2bec0041d0 con 0x7f2bf0100e20 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 Processor -- start 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 -- start start 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bbda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bc8370 0x55da11bca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55da11b232f0 con 0x55da11bbd5d0 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55da11acacf0 con 0x55da11bbeb40 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55da11b1f2b0 con 0x55da11bc8370 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bf6997640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.643+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bbda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58904/0 (socket says 172.21.15.16:58904) 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 -- 172.21.15.16:0/2067493830 learned_addr learned my addr 172.21.15.16:0/2067493830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf5995640 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bc8370 0x55da11bca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 -- 172.21.15.16:0/2067493830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bc8370 msgr2=0x55da11bca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bc8370 0x55da11bca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 -- 172.21.15.16:0/2067493830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 msgr2=0x55da11bbda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bbda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2be77fe640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 0x7f2bdc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 -- 172.21.15.16:0/2067493830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da11aaf3a0 con 0x55da11bbeb40 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2be77fe640 1 -- 172.21.15.16:0/3397695855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2bec0ce8e0 con 0x7f2bf0100e20 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf5995640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 0x7f2bdc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.644+0000 7f2bf6997640 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009f50 tx=0x7f2be80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd99ae5df2a950d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2be4dea640 1 -- 172.21.15.16:0/2067493830 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8004d20 con 0x55da11bbeb40 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2be4dea640 1 -- 172.21.15.16:0/2067493830 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be8004050 con 0x55da11bbeb40 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2be4dea640 1 -- 172.21.15.16:0/2067493830 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8005280 con 0x55da11bbeb40 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2067493830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 msgr2=0x55da11bbd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bf5995640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 0x7f2bdc0a50c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009230 tx=0x7f2be00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009f50 tx=0x7f2be80118d0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2067493830 shutdown_connections 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bc8370 0x55da11bca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bbda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2067493830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bbd090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.645+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2067493830 >> 172.21.15.16:0/2067493830 conn(0x55da11b86310 msgr2=0x55da11bb9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.646+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2067493830 shutdown_connections 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.646+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2067493830 wait complete. 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.646+0000 7f2bfa7d3740 1 -- start start 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.646+0000 7f2bfa7d3740 1 Processor -- start 2024-09-12T02:51:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbeb40 0x55da11bc4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bc55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55da11bcb070 con 0x55da11bbeb40 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33030/0 (socket says 172.21.15.16:33030) 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 -- 172.21.15.16:0/2543257308 learned_addr learned my addr 172.21.15.16:0/2543257308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf5995640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbeb40 0x55da11bc4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6997640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bc55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55da11ade060 con 0x55da11bc8370 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55da11bcb1f0 con 0x55da11bbd5d0 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bfa7d3740 10 client.0 _ll_register_callbacks cb 0x55da118318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 -- 172.21.15.16:0/2543257308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 msgr2=0x55da11bc55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bc55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 -- 172.21.15.16:0/2543257308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbeb40 msgr2=0x55da11bc4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbeb40 0x55da11bc4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.647+0000 7f2bf6196640 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da118943c0 con 0x55da11bbd5d0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bf6196640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0cc3f0 tx=0x7f2bec005130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4de7216bcb4275 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec09af20 con 0x55da11bbd5d0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be8004050 con 0x55da11bbd5d0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2bec0977c0 con 0x55da11bbd5d0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2bec09aab0 con 0x55da11bbd5d0 2024-09-12T02:51:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 10 client.25861 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55da11bd6b80 con 0x55da11bbd5d0 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be8004050 con 0x55da11bbd5d0 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2bec09f140 con 0x55da11bbd5d0 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 10 client.25861 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 10 client.25861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 20 client.25861 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11bd8040 0x55da11bda430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 4 client.25861 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.648+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da11b30860 con 0x55da11bd8040 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.649+0000 7f2bf5995640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11bd8040 0x55da11bda430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.649+0000 7f2bd9ffb640 10 client.25861 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.649+0000 7f2bf5995640 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11bd8040 0x55da11bda430 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3cbb6783746348b server_cookie=e573c5a1919c82ae in_seq=0 out_seq=0 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/2543257308 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da11b30860 con 0x55da11bd8040 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bd9ffb640 10 client.25861 handle_command_reply: tid=0 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 client.25861 shutdown 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 msgr2=0x55da11bc7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0cc3f0 tx=0x7f2bec005130 comp rx=0 tx=0).stop 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 shutdown_connections 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bc55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbeb40 0x55da11bc4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11bd8040 0x55da11bda430 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/2543257308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc7d30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bd9ffb640 0 client.25861 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.650+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 >> 172.21.15.16:0/2543257308 conn(0x55da11b86310 msgr2=0x55da11bbf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.651+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 shutdown_connections 2024-09-12T02:51:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.651+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/2543257308 wait complete. 2024-09-12T02:51:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.651+0000 7f2bfa7d3740 20 client.25861 trim_cache size 0 max 16384 2024-09-12T02:51:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.671+0000 7f2bfa7d3740 1 Processor -- start 2024-09-12T02:51:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.671+0000 7f2bfa7d3740 1 -- start start 2024-09-12T02:51:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 0x55da11bdfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01af760 con 0x55da11bc8370 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01b12a0 con 0x55da11bbeb40 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01afe30 con 0x55da11bbd5d0 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58910/0 (socket says 172.21.15.16:58910) 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 -- 172.21.15.16:0/745124430 learned_addr learned my addr 172.21.15.16:0/745124430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf6196640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf6997640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 0x55da11bdfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 -- 172.21.15.16:0/745124430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 msgr2=0x55da11bc0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 -- 172.21.15.16:0/745124430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 msgr2=0x55da11bdfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 0x55da11bdfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf5995640 1 -- 172.21.15.16:0/745124430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da11bc5940 con 0x55da11bbeb40 2024-09-12T02:51:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.672+0000 7f2bf6997640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 0x55da11bdfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.673+0000 7f2bf5995640 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2be002a9a0 tx=0x7f2be0040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c09b1e390d7839 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.673+0000 7f2bdbfff640 1 -- 172.21.15.16:0/745124430 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0035040 con 0x55da11bbeb40 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.673+0000 7f2bdbfff640 1 -- 172.21.15.16:0/745124430 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be0003d30 con 0x55da11bbeb40 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.673+0000 7f2bdbfff640 1 -- 172.21.15.16:0/745124430 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be003d6d0 con 0x55da11bbeb40 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/745124430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 msgr2=0x55da11bc0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2be002a9a0 tx=0x7f2be0040000 comp rx=0 tx=0).stop 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/745124430 shutdown_connections 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bc8370 0x55da11bdfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bbeb40 0x55da11bc0660 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/745124430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbd5d0 0x55da11bc0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/745124430 >> 172.21.15.16:0/745124430 conn(0x55da11b86310 msgr2=0x55da11bc7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/745124430 shutdown_connections 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.674+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/745124430 wait complete. 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.675+0000 7f2bfa7d3740 1 -- start start 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.675+0000 7f2bfa7d3740 1 Processor -- start 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.675+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bcc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 0x55da11bcc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bcc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01b0bd0 con 0x55da11bbd5d0 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf6196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bcc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47626/0 (socket says 172.21.15.16:47626) 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf6196640 1 -- 172.21.15.16:0/536731678 learned_addr learned my addr 172.21.15.16:0/536731678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01b0500 con 0x55da11bc8370 2024-09-12T02:51:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01ad740 con 0x55da11bbeb40 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 0x55da11bcc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf6997640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bfa7d3740 10 client.0 _ll_register_callbacks cb 0x7f2bec0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 -- 172.21.15.16:0/536731678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 msgr2=0x55da11bccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 -- 172.21.15.16:0/536731678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 msgr2=0x55da11bcc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bcc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.676+0000 7f2bf5995640 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da11c41ef0 con 0x55da11bbeb40 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.677+0000 7f2bf5995640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 0x55da11bcc9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55da11bc5bd0 tx=0x7f2be002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8a99976cacc179 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.677+0000 7f2bd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.677+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0035040 con 0x55da11bbeb40 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.677+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2be0031b80 con 0x55da11bbeb40 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.677+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be0003d30 con 0x55da11bbeb40 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2be006b480 con 0x55da11bbeb40 2024-09-12T02:51:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 10 client.25867 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55da11c43130 con 0x55da11bbeb40 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be0064070 con 0x55da11bbeb40 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2be0044070 con 0x55da11bbeb40 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 10 client.25867 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 10 client.25867 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 20 client.25867 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11c43740 0x55da11c45b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 4 client.25867 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2be0003d30 con 0x55da11c43740 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.678+0000 7f2bf6196640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11c43740 0x55da11c45b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.679+0000 7f2bf6196640 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11c43740 0x55da11c45b60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73216ab66145fa98 server_cookie=1fafd429a265344c in_seq=0 out_seq=0 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.679+0000 7f2bd9ffb640 10 client.25867 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bd9ffb640 1 -- 172.21.15.16:0/536731678 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2be0003d30 con 0x55da11c43740 2024-09-12T02:51:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bd9ffb640 10 client.25867 handle_command_reply: tid=0 2024-09-12T02:51:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 client.25867 shutdown 2024-09-12T02:51:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 msgr2=0x55da11bcc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.684 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 0x55da11bcc9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55da11bc5bd0 tx=0x7f2be002a480 comp rx=0 tx=0).stop 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 shutdown_connections 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55da11bc8370 0x55da11bccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55da11c43740 0x55da11c45b60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55da11bbeb40 0x55da11bcc9f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 --2- 172.21.15.16:0/536731678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55da11bbd5d0 0x55da11bcc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 >> 172.21.15.16:0/536731678 conn(0x55da11b86310 msgr2=0x55da11bb84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bd9ffb640 0 client.25867 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 shutdown_connections 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 1 -- 172.21.15.16:0/536731678 wait complete. 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.680+0000 7f2bfa7d3740 20 client.25867 trim_cache size 0 max 16384 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 msgr2=0x7f2bdc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 0x7f2bdc0a50c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009230 tx=0x7f2be00058d0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 msgr2=0x7f2bf00fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00d950 tx=0x7f2bec00de10 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 shutdown_connections 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bf0102ef0 0x7f2bf00fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf0102570 0x7f2bf00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2bdc0a2c00 0x7f2bdc0a50c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 --2- 172.21.15.16:0/3397695855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bf0100e20 0x7f2bf00fcc80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 >> 172.21.15.16:0/3397695855 conn(0x7f2bf00fc7d0 msgr2=0x7f2bf006a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 shutdown_connections 2024-09-12T02:51:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:57.681+0000 7f2bf83fa640 1 -- 172.21.15.16:0/3397695855 wait complete. 2024-09-12T02:51:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:57.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 325352 9.45 MB/sec execute 59 sec latency 31.407 ms 2024-09-12T02:51:58.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:57 smithi016 ceph-mon[30925]: pgmap v458: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.4 MiB/s wr, 93 op/s 2024-09-12T02:51:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:57 smithi033 ceph-mon[35092]: pgmap v458: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.4 MiB/s wr, 93 op/s 2024-09-12T02:51:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:57 smithi184 ceph-mon[35321]: pgmap v458: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.4 MiB/s wr, 93 op/s 2024-09-12T02:51:58.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:51:58.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 327251 9.49 MB/sec execute 60 sec latency 99.424 ms 2024-09-12T02:51:59.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:51:59.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:51:59.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:51:59.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- 172.21.15.16:0/3726341793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 msgr2=0x7f7f40103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 --2- 172.21.15.16:0/3726341793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f40103170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009900 tx=0x7f7f3002a410 comp rx=0 tx=0).stop 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- 172.21.15.16:0/3726341793 shutdown_connections 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 --2- 172.21.15.16:0/3726341793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40108290 0x7f7f4010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 --2- 172.21.15.16:0/3726341793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40103740 0x7f7f40107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 --2- 172.21.15.16:0/3726341793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f40103170 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- 172.21.15.16:0/3726341793 >> 172.21.15.16:0/3726341793 conn(0x7f7f400fc7d0 msgr2=0x7f7f400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- 172.21.15.16:0/3726341793 shutdown_connections 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- 172.21.15.16:0/3726341793 wait complete. 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 Processor -- start 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.089+0000 7f7f4552e640 1 -- start start 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f401985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40108290 0x7f7f4019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40112090 con 0x7f7f40103740 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40111f10 con 0x7f7f40108290 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40112210 con 0x7f7f40102d90 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f3effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f401985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47644/0 (socket says 172.21.15.16:47644) 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 -- 172.21.15.16:0/1282846966 learned_addr learned my addr 172.21.15.16:0/1282846966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f3f7fe640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40108290 0x7f7f4019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 -- 172.21.15.16:0/1282846966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 msgr2=0x7f7f401985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f401985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 -- 172.21.15.16:0/1282846966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40108290 msgr2=0x7f7f4019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40108290 0x7f7f4019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 -- 172.21.15.16:0/1282846966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4019d580 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f37fff640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800e9f0 tx=0x7f7f2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2800ccc0 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.090+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4019d870 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.091+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f40100920 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.091+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f28005500 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.091+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28010430 con 0x7f7f40103740 2024-09-12T02:51:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.092+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f4006a7f0 con 0x7f7f40103740 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.093+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7f280046c0 con 0x7f7f40103740 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.094+0000 7f7f3cff9640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 0x7f7f2c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.094+0000 7f7f3effd640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 0x7f7f2c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.094+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7f280cf430 con 0x7f7f40103740 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.094+0000 7f7f3effd640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 0x7f7f2c0a5040 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009230 tx=0x7f7f300058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.099+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7f2809c3a0 con 0x7f7f40103740 2024-09-12T02:51:59.151 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:51:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:51:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:51:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:51:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:51:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:51:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:51:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:51:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:51:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.323+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f40199960 con 0x7f7f40103740 2024-09-12T02:51:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.324+0000 7f7f3cff9640 1 -- 172.21.15.16:0/1282846966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f7f40199960 con 0x7f7f40103740 2024-09-12T02:51:59.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:51:59.326 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:51:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:51:59.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 msgr2=0x7f7f2c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 0x7f7f2c0a5040 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009230 tx=0x7f7f300058d0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 msgr2=0x7f7f40198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800e9f0 tx=0x7f7f2800edf0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 shutdown_connections 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f40108290 0x7f7f4019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7f2c0a2b80 0x7f7f2c0a5040 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f40103740 0x7f7f40198b10 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 --2- 172.21.15.16:0/1282846966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f40102d90 0x7f7f401985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 >> 172.21.15.16:0/1282846966 conn(0x7f7f400fc7d0 msgr2=0x7f7f40105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 shutdown_connections 2024-09-12T02:51:59.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.326+0000 7f7f4552e640 1 -- 172.21.15.16:0/1282846966 wait complete. 2024-09-12T02:51:59.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 330162 9.63 MB/sec execute 61 sec latency 20.539 ms 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/3782538645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc109140 msgr2=0x7fb1dc110cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/3782538645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc109140 0x7fb1dc110cc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009ca0 tx=0x7fb1d402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/3782538645 shutdown_connections 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/3782538645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc109140 0x7fb1dc110cc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/3782538645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc1049c0 0x7fb1dc108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/3782538645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc104010 0x7fb1dc1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/3782538645 >> 172.21.15.16:0/3782538645 conn(0x7fb1dc0ffa50 msgr2=0x7fb1dc101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/3782538645 shutdown_connections 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.934+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/3782538645 wait complete. 2024-09-12T02:51:59.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 Processor -- start 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 -- start start 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1049c0 0x7fb1dc19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 0x7fb1dc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc114150 con 0x7fb1dc1049c0 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc113fd0 con 0x7fb1dc109140 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.935+0000 7fb1e2ee5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1142d0 con 0x7fb1dc104010 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1049c0 0x7fb1dc19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33090/0 (socket says 172.21.15.16:33090) 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 -- 172.21.15.16:0/4277604681 learned_addr learned my addr 172.21.15.16:0/4277604681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e1482640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 0x7fb1dc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 -- 172.21.15.16:0/4277604681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 msgr2=0x7fb1dc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 0x7fb1dc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 -- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1049c0 msgr2=0x7fb1dc19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1049c0 0x7fb1dc19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 -- 172.21.15.16:0/4277604681 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc19f640 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e1482640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 0x7fb1dc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000e9f0 tx=0x7fb1d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1d9ffb640 1 -- 172.21.15.16:0/4277604681 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000cd80 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc19f930 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1d9ffb640 1 -- 172.21.15.16:0/4277604681 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0010910 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.936+0000 7fb1d9ffb640 1 -- 172.21.15.16:0/4277604681 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00188f0 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.937+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc1a7210 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.939+0000 7fb1d9ffb640 1 -- 172.21.15.16:0/4277604681 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb1d0010ac0 con 0x7fb1dc104010 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.940+0000 7fb1d9ffb640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 0x7fb1cc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.940+0000 7fb1d9ffb640 1 -- 172.21.15.16:0/4277604681 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1d00cf470 con 0x7fb1dc104010 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.940+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 0x7fb1cc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.940+0000 7fb1e52be740 1 Processor -- start 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.940+0000 7fb1e52be740 1 -- start start 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 0x7fb1cc0a5040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0ffe70 tx=0x7fb1c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x55827218ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5582720f42f0 con 0x558272199370 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55827209bcf0 con 0x55827218e5d0 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e52be740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5582720f02b0 con 0x55827218fb40 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33092/0 (socket says 172.21.15.16:33092) 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e0c81640 1 -- 172.21.15.16:0/1848304954 learned_addr learned my addr 172.21.15.16:0/1848304954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x55827218ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 -- 172.21.15.16:0/1848304954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 msgr2=0x55827218e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 -- 172.21.15.16:0/1848304954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 msgr2=0x55827218ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x55827218ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 -- 172.21.15.16:0/1848304954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582720803a0 con 0x558272199370 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x55827218ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.941+0000 7fb1e1482640 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219b760 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009bf0 tx=0x7fb1d4005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917d407aa635b3b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.942+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/1848304954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4031db0 con 0x558272199370 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.942+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/1848304954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4036050 con 0x558272199370 2024-09-12T02:51:59.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.942+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/1848304954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40042a0 con 0x558272199370 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.942+0000 7fb1e52be740 1 -- 172.21.15.16:0/1848304954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 msgr2=0x55827219b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.942+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219b760 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009bf0 tx=0x7fb1d4005f60 comp rx=0 tx=0).stop 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 -- 172.21.15.16:0/1848304954 shutdown_connections 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219b760 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x55827218ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1848304954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x55827218e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 -- 172.21.15.16:0/1848304954 >> 172.21.15.16:0/1848304954 conn(0x558272157310 msgr2=0x55827218ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 -- 172.21.15.16:0/1848304954 shutdown_connections 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 -- 172.21.15.16:0/1848304954 wait complete. 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 -- start start 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.943+0000 7fb1e52be740 1 Processor -- start 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 0x558272198d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 0x558272196040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 0x558272198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272196650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 0x558272198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47690/0 (socket says 172.21.15.16:47690) 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e0c81640 1 -- 172.21.15.16:0/4101981832 learned_addr learned my addr 172.21.15.16:0/4101981832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 0x558272196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55827219be90 con 0x55827218e5d0 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5582720af060 con 0x55827218fb40 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55827219c010 con 0x558272199370 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e52be740 10 client.0 _ll_register_callbacks cb 0x558271e028c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1e1482640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 -- 172.21.15.16:0/4101981832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 msgr2=0x558272196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 -- 172.21.15.16:0/4101981832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 msgr2=0x558272198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 0x558272198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.944+0000 7fb1dbfff640 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558271e653c0 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 0x558272196040 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55827218bac0 tx=0x7fb1c8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f4b3f3e4d3a3d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c801e8f0 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1c801ea90 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d4036050 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1c801ec90 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1e52be740 10 client.25991 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.945+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582721a7a10 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c801ee30 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb1c8026020 con 0x55827218fb40 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 10 client.25991 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 10 client.25991 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:51:59.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 20 client.25991 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5582721a8ed0 0x5582721ab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 4 client.25991 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb1d4036050 con 0x5582721a8ed0 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5582721a8ed0 0x5582721ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5582721a8ed0 0x5582721ab330 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8110a9c1d5b1665d server_cookie=f9f920fa3de78a75 in_seq=0 out_seq=0 2024-09-12T02:51:59.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.946+0000 7fb1bbfff640 10 client.25991 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:59.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1bbfff640 1 -- 172.21.15.16:0/4101981832 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb1d4036050 con 0x5582721a8ed0 2024-09-12T02:51:59.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1bbfff640 10 client.25991 handle_command_reply: tid=0 2024-09-12T02:51:59.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 client.25991 shutdown 2024-09-12T02:51:59.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 msgr2=0x558272196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 0x558272196040 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55827218bac0 tx=0x7fb1c8004730 comp rx=0 tx=0).stop 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 shutdown_connections 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5582721a8ed0 0x5582721ab330 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218fb40 0x558272196040 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1bbfff640 0 client.25991 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 --2- 172.21.15.16:0/4101981832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218e5d0 0x558272198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.947+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 >> 172.21.15.16:0/4101981832 conn(0x558272157310 msgr2=0x558272190960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.948+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 shutdown_connections 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.948+0000 7fb1e52be740 1 -- 172.21.15.16:0/4101981832 wait complete. 2024-09-12T02:51:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.948+0000 7fb1e52be740 20 client.25991 trim_cache size 0 max 16384 2024-09-12T02:51:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.977+0000 7fb1e52be740 1 Processor -- start 2024-09-12T02:51:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.977+0000 7fb1e52be740 1 -- start start 2024-09-12T02:51:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.977+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x558272195900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.977+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.977+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e52be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1af800 con 0x55827218fb40 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e52be740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1b1340 con 0x55827218e5d0 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e52be740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1afed0 con 0x558272199370 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47694/0 (socket says 172.21.15.16:47694) 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 -- 172.21.15.16:0/1479850806 learned_addr learned my addr 172.21.15.16:0/1479850806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x558272195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e1482640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 -- 172.21.15.16:0/1479850806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 msgr2=0x558272197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 -- 172.21.15.16:0/1479850806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 msgr2=0x558272195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x558272195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 -- 172.21.15.16:0/1479850806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558272194ff0 con 0x55827218fb40 2024-09-12T02:51:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x558272195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.978+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8016040 tx=0x7fb1c8011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2077757183a1abe9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1c67fc640 1 -- 172.21.15.16:0/1479850806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c803a020 con 0x55827218fb40 2024-09-12T02:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1c67fc640 1 -- 172.21.15.16:0/1479850806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c801a3c0 con 0x55827218fb40 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1c67fc640 1 -- 172.21.15.16:0/1479850806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8033740 con 0x55827218fb40 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 -- 172.21.15.16:0/1479850806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 msgr2=0x5582721a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8016040 tx=0x7fb1c8011c60 comp rx=0 tx=0).stop 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 -- 172.21.15.16:0/1479850806 shutdown_connections 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558272199370 0x558272197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55827218fb40 0x5582721a5130 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 --2- 172.21.15.16:0/1479850806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x558272195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.979+0000 7fb1e52be740 1 -- 172.21.15.16:0/1479850806 >> 172.21.15.16:0/1479850806 conn(0x558272157310 msgr2=0x5582721978a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.980+0000 7fb1e52be740 1 -- 172.21.15.16:0/1479850806 shutdown_connections 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.980+0000 7fb1e52be740 1 -- 172.21.15.16:0/1479850806 wait complete. 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.980+0000 7fb1e52be740 1 -- start start 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.980+0000 7fb1e52be740 1 Processor -- start 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x5582721a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1b0c70 con 0x558272199370 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1b05a0 con 0x55827218e5d0 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1ad7e0 con 0x55827218fb40 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e52be740 10 client.0 _ll_register_callbacks cb 0x7fb1c8011d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e1482640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58990/0 (socket says 172.21.15.16:58990) 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 -- 172.21.15.16:0/788383341 learned_addr learned my addr 172.21.15.16:0/788383341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:51:59.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x5582721a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 -- 172.21.15.16:0/788383341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 msgr2=0x5582721a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x5582721a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 -- 172.21.15.16:0/788383341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 msgr2=0x55827219d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.981+0000 7fb1e0c81640 1 -- 172.21.15.16:0/788383341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55827219e050 con 0x55827218e5d0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e0c81640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1d009b9e0 tx=0x7fb1d0097f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23300e9bc3603e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00b9060 con 0x55827218e5d0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1d00cf0b0 con 0x55827218e5d0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1c801a3c0 con 0x55827218e5d0 2024-09-12T02:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1d00cf2b0 con 0x55827218e5d0 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 10 client.25997 fetch_fsmap learned FSMap version 32 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558272214720 con 0x55827218e5d0 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c801a3c0 con 0x55827218e5d0 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb1d0097bf0 con 0x55827218e5d0 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 10 client.25997 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 10 client.25997 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 20 client.25997 populate_metadata read hostname 'smithi016' 2024-09-12T02:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558272214da0 0x558272217190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 4 client.25997 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.982+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558272101860 con 0x558272214da0 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558272214da0 0x558272217190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1bbfff640 10 client.25997 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1dbfff640 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558272214da0 0x558272217190 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83f09aaa42de57 server_cookie=cb4460373f0355e3 in_seq=0 out_seq=0 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1bbfff640 1 -- 172.21.15.16:0/788383341 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558272101860 con 0x558272214da0 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1bbfff640 10 client.25997 handle_command_reply: tid=0 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 client.25997 shutdown 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 msgr2=0x5582721a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1d009b9e0 tx=0x7fb1d0097f80 comp rx=0 tx=0).stop 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 shutdown_connections 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558272214da0 0x558272217190 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558272199370 0x55827219d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55827218fb40 0x5582721a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1bbfff640 0 client.25997 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 --2- 172.21.15.16:0/788383341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55827218e5d0 0x5582721a1400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.983+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 >> 172.21.15.16:0/788383341 conn(0x558272157310 msgr2=0x558272190110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 shutdown_connections 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e52be740 1 -- 172.21.15.16:0/788383341 wait complete. 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e52be740 20 client.25997 trim_cache size 0 max 16384 2024-09-12T02:51:59.990 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 msgr2=0x7fb1cc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 0x7fb1cc0a5040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0ffe70 tx=0x7fb1c8005f90 comp rx=0 tx=0).stop 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 msgr2=0x7fb1dc19a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000e9f0 tx=0x7fb1d000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 shutdown_connections 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1dc109140 0x7fb1dc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1cc0a2b80 0x7fb1cc0a5040 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1049c0 0x7fb1dc19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 --2- 172.21.15.16:0/4277604681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1dc104010 0x7fb1dc19a690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:51:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.984+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 >> 172.21.15.16:0/4277604681 conn(0x7fb1dc0ffa50 msgr2=0x7fb1dc100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:51:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.985+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 shutdown_connections 2024-09-12T02:51:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:51:59.985+0000 7fb1e2ee5640 1 -- 172.21.15.16:0/4277604681 wait complete. 2024-09-12T02:52:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:59 smithi016 ceph-mon[30925]: pgmap v459: 97 pgs: 97 active+clean; 134 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 2.4 MiB/s wr, 90 op/s 2024-09-12T02:52:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:51:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1282846966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:59 smithi033 ceph-mon[35092]: pgmap v459: 97 pgs: 97 active+clean; 134 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 2.4 MiB/s wr, 90 op/s 2024-09-12T02:52:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:51:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1282846966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:59 smithi184 ceph-mon[35321]: pgmap v459: 97 pgs: 97 active+clean; 134 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 2.4 MiB/s wr, 90 op/s 2024-09-12T02:52:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:51:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1282846966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:00.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 333223 9.71 MB/sec execute 62 sec latency 13.206 ms 2024-09-12T02:52:00.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:52:01.260 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0' 2024-09-12T02:52:01.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0 2024-09-12T02:52:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:01.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335407 9.72 MB/sec execute 63 sec latency 21.667 ms 2024-09-12T02:52:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 -- 172.21.15.16:0/896121240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce38102d30 msgr2=0x7fce38103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 --2- 172.21.15.16:0/896121240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce38102d30 0x7fce38103110 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fce34009930 tx=0x7fce3402a420 comp rx=0 tx=0).stop 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 -- 172.21.15.16:0/896121240 shutdown_connections 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 --2- 172.21.15.16:0/896121240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 0x7fce3810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 --2- 172.21.15.16:0/896121240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce381036e0 0x7fce38107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 --2- 172.21.15.16:0/896121240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce38102d30 0x7fce38103110 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 -- 172.21.15.16:0/896121240 >> 172.21.15.16:0/896121240 conn(0x7fce380fc7d0 msgr2=0x7fce380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 -- 172.21.15.16:0/896121240 shutdown_connections 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.896+0000 7fce40552640 1 -- 172.21.15.16:0/896121240 wait complete. 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.897+0000 7fce40552640 1 Processor -- start 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.897+0000 7fce40552640 1 -- start start 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38102d30 0x7fce381985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 0x7fce3819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce381120a0 con 0x7fce381036e0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fce38111f20 con 0x7fce38102d30 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce40552640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce38112220 con 0x7fce381082a0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47726/0 (socket says 172.21.15.16:47726) 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 -- 172.21.15.16:0/1546101215 learned_addr learned my addr 172.21.15.16:0/1546101215 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38102d30 0x7fce381985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3eaef640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 0x7fce3819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 -- 172.21.15.16:0/1546101215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 msgr2=0x7fce3819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 0x7fce3819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 -- 172.21.15.16:0/1546101215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38102d30 msgr2=0x7fce381985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38102d30 0x7fce381985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 -- 172.21.15.16:0/1546101215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3819d570 con 0x7fce381036e0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce3daed640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fce2800d950 tx=0x7fce2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.898+0000 7fce2f7fe640 1 -- 172.21.15.16:0/1546101215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28017070 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.899+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce3819d860 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.899+0000 7fce2f7fe640 1 -- 172.21.15.16:0/1546101215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce28009c30 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.899+0000 7fce2f7fe640 1 -- 172.21.15.16:0/1546101215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce280047a0 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.899+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce381a5140 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.901+0000 7fce2f7fe640 1 -- 172.21.15.16:0/1546101215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fce28004020 con 0x7fce381036e0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.901+0000 7fce4292b740 1 Processor -- start 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.901+0000 7fce4292b740 1 -- start start 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 0x557295578a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295589b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5572954de2f0 con 0x557295579b40 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557295485cf0 con 0x5572955785d0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce4292b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5572954da2b0 con 0x557295587780 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3eaef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295589b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3eaef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47730/0 (socket says 172.21.15.16:47730) 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3eaef640 1 -- 172.21.15.16:0/3131426015 learned_addr learned my addr 172.21.15.16:0/3131426015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 0x557295578a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce2f7fe640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 0x7fce240a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 -- 172.21.15.16:0/3131426015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 msgr2=0x557295578a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 0x557295578a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 -- 172.21.15.16:0/3131426015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 msgr2=0x557295578090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.902+0000 7fce3daed640 1 -- 172.21.15.16:0/3131426015 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55729546a3a0 con 0x557295587780 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce2f7fe640 1 -- 172.21.15.16:0/1546101215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fce280cec90 con 0x7fce381036e0 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 0x557295578a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce3eaef640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce3eaef640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 0x7fce240a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce3daed640 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295589b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce280cc710 tx=0x7fce28022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e1eedd328ac9f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce2cff9640 1 -- 172.21.15.16:0/3131426015 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28016170 con 0x557295587780 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce2cff9640 1 -- 172.21.15.16:0/3131426015 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce2809bba0 con 0x557295587780 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.903+0000 7fce2cff9640 1 -- 172.21.15.16:0/3131426015 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce280a4110 con 0x557295587780 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- 172.21.15.16:0/3131426015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 msgr2=0x557295589b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295589b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce280cc710 tx=0x7fce28022000 comp rx=0 tx=0).stop 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce3eaef640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 0x7fce240a5040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fce300107d0 tx=0x7fce30004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- 172.21.15.16:0/3131426015 shutdown_connections 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295589b70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572955785d0 0x557295578a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 --2- 172.21.15.16:0/3131426015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295578090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- 172.21.15.16:0/3131426015 >> 172.21.15.16:0/3131426015 conn(0x557295541310 msgr2=0x557295574c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- 172.21.15.16:0/3131426015 shutdown_connections 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- 172.21.15.16:0/3131426015 wait complete. 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.904+0000 7fce4292b740 1 -- start start 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 Processor -- start 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295579b40 0x5572955843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x5572955849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55729558a3c0 con 0x5572955785d0 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557295499060 con 0x557295587780 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55729558a540 con 0x557295579b40 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce4292b740 10 client.0 _ll_register_callbacks cb 0x5572951ec8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47744/0 (socket says 172.21.15.16:47744) 2024-09-12T02:52:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3184231440 learned_addr learned my addr 172.21.15.16:0/3184231440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce3daed640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295579b40 0x5572955843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.905+0000 7fce3eaef640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x5572955849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3184231440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295579b40 msgr2=0x5572955843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295579b40 0x5572955843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 msgr2=0x5572955849e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x5572955849e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3184231440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55729524f3c0 con 0x5572955785d0 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fce34031550 tx=0x7fce34031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74b25496fd1338b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34035040 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.906+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fce34004450 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.907+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce2809bba0 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.910+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce34005020 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.910+0000 7fce4292b740 10 client.16443 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.910+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55729557b310 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce2809bba0 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fce340035f0 con 0x5572955785d0 2024-09-12T02:52:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 10 client.16443 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 20 client.16443 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55729557c7d0 0x55729557ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 4 client.16443 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572954eb860 con 0x55729557c7d0 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.911+0000 7fce3daed640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55729557c7d0 0x55729557ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.912+0000 7fce21ffb640 10 client.16443 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.912+0000 7fce3daed640 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55729557c7d0 0x55729557ebe0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5d505aa717c9a14 server_cookie=a21834723dd83269 in_seq=0 out_seq=0 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.912+0000 7fce21ffb640 1 -- 172.21.15.16:0/3184231440 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5572954eb860 con 0x55729557c7d0 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.912+0000 7fce21ffb640 10 client.16443 handle_command_reply: tid=0 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.912+0000 7fce4292b740 1 client.16443 shutdown 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 msgr2=0x557295587140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fce34031550 tx=0x7fce34031850 comp rx=0 tx=0).stop 2024-09-12T02:52:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 shutdown_connections 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55729557c7d0 0x55729557ebe0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce21ffb640 0 client.16443 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x5572955849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295579b40 0x5572955843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 --2- 172.21.15.16:0/3184231440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x557295587140 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 >> 172.21.15.16:0/3184231440 conn(0x557295541310 msgr2=0x55729557a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 shutdown_connections 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 1 -- 172.21.15.16:0/3184231440 wait complete. 2024-09-12T02:52:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.913+0000 7fce4292b740 20 client.16443 trim_cache size 0 max 16384 2024-09-12T02:52:01.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.929+0000 7fce4292b740 1 Processor -- start 2024-09-12T02:52:01.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.929+0000 7fce4292b740 1 -- start start 2024-09-12T02:52:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 0x557295581650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295581b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47756/0 (socket says 172.21.15.16:47756) 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3daed640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 0x557295581650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 -- 172.21.15.16:0/1832009861 learned_addr learned my addr 172.21.15.16:0/1832009861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce381ab420 con 0x5572955785d0 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fce381acf60 con 0x557295579b40 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce381abaf0 con 0x557295587780 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3eaef640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295581b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 -- 172.21.15.16:0/1832009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 msgr2=0x557295581b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295581b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 -- 172.21.15.16:0/1832009861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 msgr2=0x557295581650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 0x557295581650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3e2ee640 1 -- 172.21.15.16:0/1832009861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572955833a0 con 0x5572955785d0 2024-09-12T02:52:01.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.930+0000 7fce3daed640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 0x557295581650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:01.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.931+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fce3402a940 tx=0x7fce3402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7a1a0ec2a4b657 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.931+0000 7fce23fff640 1 -- 172.21.15.16:0/1832009861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34035070 con 0x5572955785d0 2024-09-12T02:52:01.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.931+0000 7fce23fff640 1 -- 172.21.15.16:0/1832009861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce34003e80 con 0x5572955785d0 2024-09-12T02:52:01.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.931+0000 7fce23fff640 1 -- 172.21.15.16:0/1832009861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340044b0 con 0x5572955785d0 2024-09-12T02:52:01.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 msgr2=0x55729557fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fce3402a940 tx=0x7fce3402a420 comp rx=0 tx=0).stop 2024-09-12T02:52:01.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 shutdown_connections 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557295587780 0x557295581b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295579b40 0x557295581650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 --2- 172.21.15.16:0/1832009861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5572955785d0 0x55729557fbe0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 >> 172.21.15.16:0/1832009861 conn(0x557295541310 msgr2=0x55729557f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 shutdown_connections 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- 172.21.15.16:0/1832009861 wait complete. 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.932+0000 7fce4292b740 1 -- start start 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 Processor -- start 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295598800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x557295598d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce381ac890 con 0x557295579b40 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fce381ac1c0 con 0x557295587780 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce381a9400 con 0x5572955785d0 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce4292b740 10 client.0 _ll_register_callbacks cb 0x7fce340314a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295598800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3e2ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33172/0 (socket says 172.21.15.16:33172) 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3810006203 learned_addr learned my addr 172.21.15.16:0/3810006203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3daed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295598800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47768/0 (socket says 172.21.15.16:47768) 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.933+0000 7fce3eaef640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x557295598d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 msgr2=0x557295598d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x557295598d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3810006203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 msgr2=0x557295598800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295598800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 -- 172.21.15.16:0/3810006203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557295599570 con 0x5572955785d0 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3eaef640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x557295598d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce3e2ee640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce34009850 tx=0x7fce34005950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a968abcf082f51cc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34035040 con 0x5572955785d0 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fce34004250 con 0x5572955785d0 2024-09-12T02:52:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.934+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5572954e5d70 con 0x5572955785d0 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce34004450 con 0x5572955785d0 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 10 client.25909 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5572955fdd70 con 0x5572955785d0 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5572954e5d70 con 0x5572955785d0 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fce34064430 con 0x5572955785d0 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 10 client.25909 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 20 client.25909 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5572955fe410 0x557295600830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 4 client.25909 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557295462170 con 0x5572955fe410 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.935+0000 7fce3daed640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5572955fe410 0x557295600830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce3daed640 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5572955fe410 0x557295600830 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7153599c6931c19 server_cookie=a49880d89e06be1b in_seq=0 out_seq=0 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce21ffb640 10 client.25909 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce21ffb640 1 -- 172.21.15.16:0/3810006203 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557295462170 con 0x5572955fe410 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce21ffb640 10 client.25909 handle_command_reply: tid=0 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce4292b740 1 client.25909 shutdown 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 msgr2=0x55729559b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce34009850 tx=0x7fce34005950 comp rx=0 tx=0).stop 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 shutdown_connections 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.936+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5572955fe410 0x557295600830 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557295587780 0x557295598d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce21ffb640 0 client.25909 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557295579b40 0x557295598800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 --2- 172.21.15.16:0/3810006203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5572955785d0 0x55729559b4b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 >> 172.21.15.16:0/3810006203 conn(0x557295541310 msgr2=0x55729557a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 shutdown_connections 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 1 -- 172.21.15.16:0/3810006203 wait complete. 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce4292b740 20 client.25909 trim_cache size 0 max 16384 2024-09-12T02:52:01.942 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 msgr2=0x7fce240a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 0x7fce240a5040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fce300107d0 tx=0x7fce30004a40 comp rx=0 tx=0).stop 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 msgr2=0x7fce38198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fce2800d950 tx=0x7fce2800de10 comp rx=0 tx=0).stop 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 shutdown_connections 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce381082a0 0x7fce3819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fce240a2b80 0x7fce240a5040 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce381036e0 0x7fce38198b00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.937+0000 7fce40552640 1 --2- 172.21.15.16:0/1546101215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fce38102d30 0x7fce381985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.938+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 >> 172.21.15.16:0/1546101215 conn(0x7fce380fc7d0 msgr2=0x7fce38105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.938+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 shutdown_connections 2024-09-12T02:52:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:01.938+0000 7fce40552640 1 -- 172.21.15.16:0/1546101215 wait complete. 2024-09-12T02:52:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:01 smithi016 ceph-mon[30925]: pgmap v460: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 178 op/s 2024-09-12T02:52:02.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:01 smithi033 ceph-mon[35092]: pgmap v460: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 178 op/s 2024-09-12T02:52:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:01 smithi184 ceph-mon[35321]: pgmap v460: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 178 op/s 2024-09-12T02:52:02.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 337731 9.78 MB/sec execute 64 sec latency 40.985 ms 2024-09-12T02:52:03.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- 172.21.15.16:0/1991417946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49841049a0 msgr2=0x7f49841089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 --2- 172.21.15.16:0/1991417946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49841049a0 0x7f49841089f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f497402a410 comp rx=0 tx=0).stop 2024-09-12T02:52:03.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- 172.21.15.16:0/1991417946 shutdown_connections 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 --2- 172.21.15.16:0/1991417946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 0x7f4984110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 --2- 172.21.15.16:0/1991417946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49841049a0 0x7f49841089f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 --2- 172.21.15.16:0/1991417946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4984103ff0 0x7f49841043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- 172.21.15.16:0/1991417946 >> 172.21.15.16:0/1991417946 conn(0x7f49840ffab0 msgr2=0x7f4984101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- 172.21.15.16:0/1991417946 shutdown_connections 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- 172.21.15.16:0/1991417946 wait complete. 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 Processor -- start 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.374+0000 7f498c770640 1 -- start start 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49841049a0 0x7f498419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 0x7f498419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4984114120 con 0x7f4984103ff0 2024-09-12T02:52:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4984113fa0 con 0x7f4984109120 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498c770640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49841142a0 con 0x7f49841049a0 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47776/0 (socket says 172.21.15.16:47776) 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 -- 172.21.15.16:0/2248163145 learned_addr learned my addr 172.21.15.16:0/2248163145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498ad0d640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 0x7f498419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f4989d0b640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49841049a0 0x7f498419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 -- 172.21.15.16:0/2248163145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49841049a0 msgr2=0x7f498419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49841049a0 0x7f498419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 -- 172.21.15.16:0/2248163145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 msgr2=0x7f498419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 0x7f498419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 -- 172.21.15.16:0/2248163145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498419f5f0 con 0x7f4984103ff0 2024-09-12T02:52:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.375+0000 7f498a50c640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f498000e9f0 tx=0x7f498000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.376+0000 7f497b7fe640 1 -- 172.21.15.16:0/2248163145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498000cd80 con 0x7f4984103ff0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.376+0000 7f497b7fe640 1 -- 172.21.15.16:0/2248163145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4980010910 con 0x7f4984103ff0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.376+0000 7f497b7fe640 1 -- 172.21.15.16:0/2248163145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4980018990 con 0x7f4984103ff0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.376+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498419f8e0 con 0x7f4984103ff0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.376+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498419fce0 con 0x7f4984103ff0 2024-09-12T02:52:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.379+0000 7f497b7fe640 1 -- 172.21.15.16:0/2248163145 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4980010ac0 con 0x7f4984103ff0 2024-09-12T02:52:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 Processor -- start 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 -- start start 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f497b7fe640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 0x7f49700a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f497b7fe640 1 -- 172.21.15.16:0/2248163145 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f49800cf300 con 0x7f4984103ff0 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498ad0d640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 0x7f49700a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 0x56250ecfdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56250ec62160 con 0x56250ecfd850 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.380+0000 7f498eb49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56250ec09720 con 0x56250ecfe170 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f498eb49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56250ec5e120 con 0x56250ed08a30 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f498a50c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 0x56250ecfdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59098/0 (socket says 172.21.15.16:59098) 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 -- 172.21.15.16:0/1862494492 learned_addr learned my addr 172.21.15.16:0/1862494492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f498ad0d640 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f498ad0d640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 0x7f49700a50c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f497c005fd0 tx=0x7f497c0107a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 -- 172.21.15.16:0/1862494492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 msgr2=0x56250ed0ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 -- 172.21.15.16:0/1862494492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 msgr2=0x56250ecfdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 0x56250ecfdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 -- 172.21.15.16:0/1862494492 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56250e9d3ba0 con 0x56250ecfe170 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f498a50c640 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 0x56250ecfdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4989d0b640 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f4974005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4f048fa830d798 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.381+0000 7f4978ff9640 1 -- 172.21.15.16:0/1862494492 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974040070 con 0x56250ecfe170 2024-09-12T02:52:03.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f4978ff9640 1 -- 172.21.15.16:0/1862494492 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49740318f0 con 0x56250ecfe170 2024-09-12T02:52:03.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f4978ff9640 1 -- 172.21.15.16:0/1862494492 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974035040 con 0x56250ecfe170 2024-09-12T02:52:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 -- 172.21.15.16:0/1862494492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 msgr2=0x56250ed084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f4974005e00 comp rx=0 tx=0).stop 2024-09-12T02:52:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 -- 172.21.15.16:0/1862494492 shutdown_connections 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed084f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 --2- 172.21.15.16:0/1862494492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfd850 0x56250ecfdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 -- 172.21.15.16:0/1862494492 >> 172.21.15.16:0/1862494492 conn(0x56250ecf9de0 msgr2=0x56250ecfa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 -- 172.21.15.16:0/1862494492 shutdown_connections 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.382+0000 7f498eb49740 1 -- 172.21.15.16:0/1862494492 wait complete. 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 -- start start 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 Processor -- start 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed05e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56250ed0b600 con 0x56250ed08a30 2024-09-12T02:52:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56250ebea900 con 0x56250ecfe170 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56250ed0b780 con 0x56250ecfd850 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498eb49740 10 client.0 _ll_register_callbacks cb 0x56250ebe8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498a50c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498ad0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498a50c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33208/0 (socket says 172.21.15.16:33208) 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498ad0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47798/0 (socket says 172.21.15.16:47798) 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f498a50c640 1 -- 172.21.15.16:0/864311049 learned_addr learned my addr 172.21.15.16:0/864311049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.383+0000 7f4989d0b640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed05e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 -- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 msgr2=0x56250ed05e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed05e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 -- 172.21.15.16:0/864311049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 msgr2=0x56250ed063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 -- 172.21.15.16:0/864311049 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56250ec2ac50 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498ad0d640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498a50c640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49800ccdb0 tx=0x7f498009ff80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee52adf533183b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f496dffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498009b010 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f498009b1b0 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49740318f0 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4980010430 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498eb49740 10 client.25921 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.384+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56250ed17320 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49740318f0 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f498009aa80 con 0x56250ecfd850 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 10 client.25921 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:03.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 10 client.25921 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 20 client.25921 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed1aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 4 client.25921 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56250ec6f890 con 0x56250ed186c0 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f4989d0b640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed1aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f496dffb640 10 client.25921 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.385+0000 7f4989d0b640 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed1aad0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da41a8082ffa5c53 server_cookie=45d771d86f8a0052 in_seq=0 out_seq=0 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f496dffb640 1 -- 172.21.15.16:0/864311049 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56250ec6f890 con 0x56250ed186c0 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f496dffb640 10 client.25921 handle_command_reply: tid=0 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 client.25921 shutdown 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 msgr2=0x56250ed05930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49800ccdb0 tx=0x7f498009ff80 comp rx=0 tx=0).stop 2024-09-12T02:52:03.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 shutdown_connections 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed1aad0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfe170 0x56250ed05e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f496dffb640 0 client.25921 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 --2- 172.21.15.16:0/864311049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfd850 0x56250ed05930 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 >> 172.21.15.16:0/864311049 conn(0x56250ecf9de0 msgr2=0x56250ecffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 shutdown_connections 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.386+0000 7f498eb49740 1 -- 172.21.15.16:0/864311049 wait complete. 2024-09-12T02:52:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.387+0000 7f498eb49740 20 client.25921 trim_cache size 0 max 16384 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.404+0000 7f498eb49740 1 Processor -- start 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.404+0000 7f498eb49740 1 -- start start 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed00c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfe170 0x56250ed06000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49841af7a0 con 0x56250ed08a30 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f49841b12e0 con 0x56250ecfd850 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498eb49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49841afe70 con 0x56250ecfe170 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f4989d0b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfe170 0x56250ed06000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47810/0 (socket says 172.21.15.16:47810) 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 -- 172.21.15.16:0/1101383484 learned_addr learned my addr 172.21.15.16:0/1101383484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498a50c640 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed00c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 -- 172.21.15.16:0/1101383484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfe170 msgr2=0x56250ed06000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfe170 0x56250ed06000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 -- 172.21.15.16:0/1101383484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 msgr2=0x56250ed00c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed00c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 -- 172.21.15.16:0/1101383484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56250ecff980 con 0x56250ed08a30 2024-09-12T02:52:03.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.405+0000 7f498ad0d640 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f497c013430 tx=0x7f497c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8552a8b2d63a6f65 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f496ffff640 1 -- 172.21.15.16:0/1101383484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c0115c0 con 0x56250ed08a30 2024-09-12T02:52:03.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f496ffff640 1 -- 172.21.15.16:0/1101383484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497c011c40 con 0x56250ed08a30 2024-09-12T02:52:03.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f496ffff640 1 -- 172.21.15.16:0/1101383484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c01ed80 con 0x56250ed08a30 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 -- 172.21.15.16:0/1101383484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 msgr2=0x56250ed06540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f497c013430 tx=0x7f497c015000 comp rx=0 tx=0).stop 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 -- 172.21.15.16:0/1101383484 shutdown_connections 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ed08a30 0x56250ed06540 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ecfe170 0x56250ed06000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 --2- 172.21.15.16:0/1101383484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed00c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 -- 172.21.15.16:0/1101383484 >> 172.21.15.16:0/1101383484 conn(0x56250ecf9de0 msgr2=0x56250ed058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 -- 172.21.15.16:0/1101383484 shutdown_connections 2024-09-12T02:52:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.406+0000 7f498eb49740 1 -- 172.21.15.16:0/1101383484 wait complete. 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 -- start start 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 Processor -- start 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed10cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49841b0c10 con 0x56250ecfe170 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f49841b0540 con 0x56250ecfd850 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49841ad780 con 0x56250ed08a30 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498eb49740 10 client.0 _ll_register_callbacks cb 0x7f4980005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f4989d0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498ad0d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f4989d0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47816/0 (socket says 172.21.15.16:47816) 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f4989d0b640 1 -- 172.21.15.16:0/3815858496 learned_addr learned my addr 172.21.15.16:0/3815858496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.407+0000 7f498a50c640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed10cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 -- 172.21.15.16:0/3815858496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 msgr2=0x56250ed0dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 -- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 msgr2=0x56250ed10cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed10cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 -- 172.21.15.16:0/3815858496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56250ecfdfa0 con 0x56250ecfe170 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f498a50c640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed10cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f4989d0b640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f49740054e0 tx=0x7f4974005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fd6cdc2bf97481 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f496dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974040040 con 0x56250ecfe170 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f497c011c40 con 0x56250ecfe170 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4974004240 con 0x56250ecfe170 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4974060050 con 0x56250ecfe170 2024-09-12T02:52:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f498eb49740 10 client.16467 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56250ed83da0 con 0x56250ecfe170 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.408+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497c011c40 con 0x56250ecfe170 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4974050420 con 0x56250ecfe170 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 10 client.16467 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 10 client.16467 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 20 client.16467 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed86220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:03.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 4 client.16467 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}] 2024-09-12T02:52:03.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}) -- 0x56250ec69be0 con 0x56250ed186c0 2024-09-12T02:52:03.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498a50c640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed86220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:03.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f498a50c640 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed86220 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c195431b592281 server_cookie=fc159de7ee1131c6 in_seq=0 out_seq=0 2024-09-12T02:52:03.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.409+0000 7f496dffb640 10 client.16467 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f496dffb640 1 -- 172.21.15.16:0/3815858496 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56250ec69be0 con 0x56250ed186c0 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f496dffb640 10 client.16467 handle_command_reply: tid=0 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 client.16467 shutdown 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 msgr2=0x56250ed11230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f49740054e0 tx=0x7f4974005470 comp rx=0 tx=0).stop 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 shutdown_connections 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56250ed186c0 0x56250ed86220 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56250ed08a30 0x56250ed0dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56250ecfe170 0x56250ed11230 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 --2- 172.21.15.16:0/3815858496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250ecfd850 0x56250ed10cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 >> 172.21.15.16:0/3815858496 conn(0x56250ecf9de0 msgr2=0x56250ed06110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f496dffb640 0 client.16467 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 shutdown_connections 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 1 -- 172.21.15.16:0/3815858496 wait complete. 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498eb49740 20 client.16467 trim_cache size 0 max 16384 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:52:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 18, 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "bf2304de": { 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 18, 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:52:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.2, 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:03.545 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 msgr2=0x7f49700a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 0x7f49700a50c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f497c005fd0 tx=0x7f497c0107a0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 msgr2=0x7f498419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f498000e9f0 tx=0x7f498000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.540+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 shutdown_connections 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4984109120 0x7f498419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49841049a0 0x7f498419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f49700a2c00 0x7f49700a50c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 --2- 172.21.15.16:0/2248163145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4984103ff0 0x7f498419a640 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 >> 172.21.15.16:0/2248163145 conn(0x7f49840ffab0 msgr2=0x7f4984100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 shutdown_connections 2024-09-12T02:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:03.541+0000 7f498c770640 1 -- 172.21.15.16:0/2248163145 wait complete. 2024-09-12T02:52:03.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.80 MB/sec execute 65 sec latency 241.570 ms 2024-09-12T02:52:03.741 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:52:03.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:52:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:03 smithi033 ceph-mon[35092]: pgmap v461: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-12T02:52:04.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:03 smithi016 ceph-mon[30925]: pgmap v461: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-12T02:52:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:03 smithi184 ceph-mon[35321]: pgmap v461: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-09-12T02:52:04.302 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bf2304de', quiesce took 7.2 seconds. Will release after: 12.95 2024-09-12T02:52:04.357 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:04.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.65 MB/sec execute 66 sec latency 1241.669 ms 2024-09-12T02:52:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 -- 172.21.15.16:0/3129384690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 msgr2=0x7f780410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 --2- 172.21.15.16:0/3129384690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780410ec50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009930 tx=0x7f77fc02a440 comp rx=0 tx=0).stop 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 -- 172.21.15.16:0/3129384690 shutdown_connections 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 --2- 172.21.15.16:0/3129384690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780410ec50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 --2- 172.21.15.16:0/3129384690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804103760 0x7f7804107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 --2- 172.21.15.16:0/3129384690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804102db0 0x7f7804103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 -- 172.21.15.16:0/3129384690 >> 172.21.15.16:0/3129384690 conn(0x7f78040fc7d0 msgr2=0x7f78040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 -- 172.21.15.16:0/3129384690 shutdown_connections 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.819+0000 7f780c0ca640 1 -- 172.21.15.16:0/3129384690 wait complete. 2024-09-12T02:52:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 Processor -- start 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 -- start start 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804102db0 0x7f78041985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804103760 0x7f7804198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78041120f0 con 0x7f78041082e0 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7804111f70 con 0x7f7804103760 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780c0ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7804112270 con 0x7f7804102db0 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.820+0000 7f780a667640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47838/0 (socket says 172.21.15.16:47838) 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 -- 172.21.15.16:0/1916393609 learned_addr learned my addr 172.21.15.16:0/1916393609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f7809665640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804103760 0x7f7804198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f7809e66640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804102db0 0x7f78041985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 -- 172.21.15.16:0/1916393609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804102db0 msgr2=0x7f78041985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804102db0 0x7f78041985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 -- 172.21.15.16:0/1916393609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804103760 msgr2=0x7f7804198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804103760 0x7f7804198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 -- 172.21.15.16:0/1916393609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780419d580 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780a667640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009900 tx=0x7f77fc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f7802ffd640 1 -- 172.21.15.16:0/1916393609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc03f070 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f7802ffd640 1 -- 172.21.15.16:0/1916393609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77fc031e30 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780419d810 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f7802ffd640 1 -- 172.21.15.16:0/1916393609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc035030 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.821+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041a5150 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.824+0000 7f7802ffd640 1 -- 172.21.15.16:0/1916393609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f77fc042070 con 0x7f78041082e0 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.824+0000 7f780e4a3740 1 Processor -- start 2024-09-12T02:52:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.824+0000 7f780e4a3740 1 -- start start 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3df350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3ea150 0x56309e3ec540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56309e340280 con 0x56309e3def70 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56309e2e7b10 con 0x56309e3df890 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780e4a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56309e33c240 con 0x56309e3ea150 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7802ffd640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 0x7f77ec0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f780a667640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 0x7f77ec0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809665640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3ea150 0x56309e3ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59156/0 (socket says 172.21.15.16:59156) 2024-09-12T02:52:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 -- 172.21.15.16:0/2384198780 learned_addr learned my addr 172.21.15.16:0/2384198780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809665640 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3df350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7802ffd640 1 -- 172.21.15.16:0/1916393609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f77fc0ef9b0 con 0x7f78041082e0 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 -- 172.21.15.16:0/2384198780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3ea150 msgr2=0x56309e3ec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3ea150 0x56309e3ec540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 -- 172.21.15.16:0/2384198780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 msgr2=0x56309e3df350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3df350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809e66640 1 -- 172.21.15.16:0/2384198780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309e2f41f0 con 0x56309e3df890 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.825+0000 7f7809665640 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3df350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.826+0000 7f780a667640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 0x7f77ec0a50c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0b6cc0 tx=0x7f77fc0e6db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.826+0000 7f7809e66640 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f77f400ae90 tx=0x7f77f400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae9ea9d3ab03d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.826+0000 7f77ebfff640 1 -- 172.21.15.16:0/2384198780 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4011850 con 0x56309e3df890 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.826+0000 7f77ebfff640 1 -- 172.21.15.16:0/2384198780 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f401a480 con 0x56309e3df890 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.826+0000 7f77ebfff640 1 -- 172.21.15.16:0/2384198780 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4010630 con 0x56309e3df890 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- 172.21.15.16:0/2384198780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 msgr2=0x56309e3e9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f77f400ae90 tx=0x7f77f400f730 comp rx=0 tx=0).stop 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- 172.21.15.16:0/2384198780 shutdown_connections 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3ea150 0x56309e3ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e9c10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2384198780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3df350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- 172.21.15.16:0/2384198780 >> 172.21.15.16:0/2384198780 conn(0x56309e3db500 msgr2=0x56309e3db8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- 172.21.15.16:0/2384198780 shutdown_connections 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- 172.21.15.16:0/2384198780 wait complete. 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 -- start start 2024-09-12T02:52:04.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.827+0000 7f780e4a3740 1 Processor -- start 2024-09-12T02:52:04.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e3e7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3ea150 0x56309e3e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56309e3ecd20 con 0x56309e3def70 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56309e2faf10 con 0x56309e3ea150 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56309e3ecea0 con 0x56309e3df890 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 10 client.0 _ll_register_callbacks cb 0x56309e2c3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47854/0 (socket says 172.21.15.16:47854) 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 -- 172.21.15.16:0/2302293984 learned_addr learned my addr 172.21.15.16:0/2302293984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809e66640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e3e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 -- 172.21.15.16:0/2302293984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 msgr2=0x56309e3e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e3e7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 -- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3ea150 msgr2=0x56309e3e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3ea150 0x56309e3e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 -- 172.21.15.16:0/2302293984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56309e0b16a0 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f7809665640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f78040697d0 tx=0x7f77f8016ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a97540e3ffa4e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f77e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8015cb0 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77f401a480 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.828+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f77f8018480 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f77f8015e50 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 10 client.16479 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56309e3f8990 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f401a480 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f77f8015040 con 0x56309e3def70 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 10 client.16479 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 20 client.16479 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e3f9e50 0x56309e3fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 4 client.16479 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.829+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56309e34d7f0 con 0x56309e3f9e50 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.830+0000 7f7809e66640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e3f9e50 0x56309e3fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.830+0000 7f7809e66640 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e3f9e50 0x56309e3fc260 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f209e560874079cd server_cookie=da1deaf45be3b38 in_seq=0 out_seq=0 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.830+0000 7f77e97fa640 10 client.16479 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f77e97fa640 1 -- 172.21.15.16:0/2302293984 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56309e34d7f0 con 0x56309e3f9e50 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f77e97fa640 10 client.16479 handle_command_reply: tid=0 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 client.16479 shutdown 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 msgr2=0x56309e3e7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f78040697d0 tx=0x7f77f8016ed0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 shutdown_connections 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e3f9e50 0x56309e3fc260 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3ea150 0x56309e3e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e3e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 --2- 172.21.15.16:0/2302293984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3def70 0x56309e3e7040 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 >> 172.21.15.16:0/2302293984 conn(0x56309e3db500 msgr2=0x56309e3d6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f77e97fa640 0 client.16479 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 shutdown_connections 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 1 -- 172.21.15.16:0/2302293984 wait complete. 2024-09-12T02:52:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.831+0000 7f780e4a3740 20 client.16479 trim_cache size 0 max 16384 2024-09-12T02:52:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.848+0000 7f780e4a3740 1 Processor -- start 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.848+0000 7f780e4a3740 1 -- start start 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3def70 0x56309e401b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e402070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78041ab3f0 con 0x56309e3ea150 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f78041acf30 con 0x56309e3def70 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f780e4a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78041abac0 con 0x56309e3df890 2024-09-12T02:52:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47862/0 (socket says 172.21.15.16:47862) 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 -- 172.21.15.16:0/3663386595 learned_addr learned my addr 172.21.15.16:0/3663386595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3def70 0x56309e401b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809e66640 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e402070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 -- 172.21.15.16:0/3663386595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 msgr2=0x56309e402070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e402070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 -- 172.21.15.16:0/3663386595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3def70 msgr2=0x56309e401b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3def70 0x56309e401b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 -- 172.21.15.16:0/3663386595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309e3db330 con 0x56309e3ea150 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f7809665640 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f77f80166e0 tx=0x7f77f80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c66a6893ce4df08 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f77eb7fe640 1 -- 172.21.15.16:0/3663386595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8015e50 con 0x56309e3ea150 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f77eb7fe640 1 -- 172.21.15.16:0/3663386595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f8005ea0 con 0x56309e3ea150 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.849+0000 7f77eb7fe640 1 -- 172.21.15.16:0/3663386595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80062a0 con 0x56309e3ea150 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- 172.21.15.16:0/3663386595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 msgr2=0x56309e3d5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f77f80166e0 tx=0x7f77f80047d0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- 172.21.15.16:0/3663386595 shutdown_connections 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3d5a50 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3df890 0x56309e402070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 --2- 172.21.15.16:0/3663386595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3def70 0x56309e401b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- 172.21.15.16:0/3663386595 >> 172.21.15.16:0/3663386595 conn(0x56309e3db500 msgr2=0x56309e3e5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- 172.21.15.16:0/3663386595 shutdown_connections 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- 172.21.15.16:0/3663386595 wait complete. 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 -- start start 2024-09-12T02:52:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.850+0000 7f780e4a3740 1 Processor -- start 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3def70 0x56309e3e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78041ac860 con 0x56309e3ea150 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f78041ac190 con 0x56309e3df890 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78041a93f0 con 0x56309e3def70 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f780e4a3740 10 client.0 _ll_register_callbacks cb 0x7f77f8006f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2024-09-12T02:52:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 -- 172.21.15.16:0/1998198319 learned_addr learned my addr 172.21.15.16:0/1998198319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809e66640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3def70 0x56309e3e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 -- 172.21.15.16:0/1998198319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3def70 msgr2=0x56309e3e3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3def70 0x56309e3e3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 -- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 msgr2=0x56309e3e4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.851+0000 7f7809665640 1 -- 172.21.15.16:0/1998198319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56309e463b00 con 0x56309e3ea150 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f7809665640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f77f8016790 tx=0x7f77f80185c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7799944c2bd18b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80059c0 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77f8005ea0 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f77f8005b60 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f77f80060a0 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 10 client.16485 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56309e464d60 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f8005ea0 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f77f8023590 con 0x56309e3ea150 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 10 client.16485 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 20 client.16485 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e465220 0x56309e467610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 4 client.16485 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.852+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56309e347d00 con 0x56309e465220 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.853+0000 7f7809e66640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e465220 0x56309e467610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.853+0000 7f7809e66640 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e465220 0x56309e467610 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6fefea3622c445 server_cookie=9995239736553960 in_seq=0 out_seq=0 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.853+0000 7f77e97fa640 10 client.16485 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f77e97fa640 1 -- 172.21.15.16:0/1998198319 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56309e347d00 con 0x56309e465220 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f77e97fa640 10 client.16485 handle_command_reply: tid=0 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 client.16485 shutdown 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 msgr2=0x56309e3e4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f77f8016790 tx=0x7f77f80185c0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 shutdown_connections 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56309e465220 0x56309e467610 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56309e3ea150 0x56309e3e4a50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56309e3df890 0x56309e3e4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 --2- 172.21.15.16:0/1998198319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56309e3def70 0x56309e3e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 >> 172.21.15.16:0/1998198319 conn(0x56309e3db500 msgr2=0x56309e3e7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f77e97fa640 0 client.16485 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 shutdown_connections 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 1 -- 172.21.15.16:0/1998198319 wait complete. 2024-09-12T02:52:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.854+0000 7f780e4a3740 20 client.16485 trim_cache size 0 max 16384 2024-09-12T02:52:04.863 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:04.863 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:52:04.863 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "7860bffd-68e0-4211-b5ac-92ba7059e31b", 2024-09-12T02:52:04.863 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:52:04.863 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 msgr2=0x7f77ec0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 0x7f77ec0a50c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0b6cc0 tx=0x7f77fc0e6db0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 msgr2=0x7f780419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009900 tx=0x7f77fc003c10 comp rx=0 tx=0).stop 2024-09-12T02:52:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 shutdown_connections 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f77ec0a2c00 0x7f77ec0a50c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78041082e0 0x7f780419cea0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7804103760 0x7f7804198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 --2- 172.21.15.16:0/1916393609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7804102db0 0x7f78041985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 >> 172.21.15.16:0/1916393609 conn(0x7f78040fc7d0 msgr2=0x7f7804106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 shutdown_connections 2024-09-12T02:52:04.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:04.855+0000 7f780c0ca640 1 -- 172.21.15.16:0/1916393609 wait complete. 2024-09-12T02:52:05.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:52:05.715 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.51 MB/sec execute 67 sec latency 2241.957 ms 2024-09-12T02:52:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: pgmap v462: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:05 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: pgmap v462: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:05 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: pgmap v462: 97 pgs: 97 active+clean; 135 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-09-12T02:52:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:52:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:52:06.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:52:06.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:05 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- 172.21.15.16:0/1481087171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8101280 msgr2=0x7f5ab810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1481087171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8101280 0x7f5ab810df40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009900 tx=0x7f5aa802a410 comp rx=0 tx=0).stop 2024-09-12T02:52:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- 172.21.15.16:0/1481087171 shutdown_connections 2024-09-12T02:52:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1481087171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab810e480 0x7f5ab8110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1481087171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8101280 0x7f5ab810df40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1481087171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab81008d0 0x7f5ab8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- 172.21.15.16:0/1481087171 >> 172.21.15.16:0/1481087171 conn(0x7f5ab80fc7a0 msgr2=0x7f5ab80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- 172.21.15.16:0/1481087171 shutdown_connections 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- 172.21.15.16:0/1481087171 wait complete. 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 Processor -- start 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.320+0000 7f5abfc06640 1 -- start start 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab81008d0 0x7f5ab819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab8101280 0x7f5ab819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8114200 con 0x7f5ab810e480 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8114080 con 0x7f5ab81008d0 2024-09-12T02:52:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8114380 con 0x7f5ab8101280 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55018/0 (socket says 172.21.15.16:55018) 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 -- 172.21.15.16:0/1569223707 learned_addr learned my addr 172.21.15.16:0/1569223707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab8101280 0x7f5ab819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab81008d0 0x7f5ab819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 -- 172.21.15.16:0/1569223707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab8101280 msgr2=0x7f5ab819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab8101280 0x7f5ab819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 -- 172.21.15.16:0/1569223707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab81008d0 msgr2=0x7f5ab819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab81008d0 0x7f5ab819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 -- 172.21.15.16:0/1569223707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab819f6f0 con 0x7f5ab810e480 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abe1a3640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b9e0 tx=0x7f5ab000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5aaeffd640 1 -- 172.21.15.16:0/1569223707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0019070 con 0x7f5ab810e480 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5aaeffd640 1 -- 172.21.15.16:0/1569223707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab0014440 con 0x7f5ab810e480 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.321+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab819f9e0 con 0x7f5ab810e480 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.322+0000 7f5aaeffd640 1 -- 172.21.15.16:0/1569223707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00125a0 con 0x7f5ab810e480 2024-09-12T02:52:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.322+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab81a72c0 con 0x7f5ab810e480 2024-09-12T02:52:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5aaeffd640 1 -- 172.21.15.16:0/1569223707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5ab001c070 con 0x7f5ab810e480 2024-09-12T02:52:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 Processor -- start 2024-09-12T02:52:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 -- start start 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3da880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3e5680 0x5600fc3e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5600fc3402e0 con 0x5600fc3dadc0 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.324+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5600fc2e7ce0 con 0x5600fc3da4a0 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5600fc33c2a0 con 0x5600fc3e5680 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55020/0 (socket says 172.21.15.16:55020) 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2795048703 learned_addr learned my addr 172.21.15.16:0/2795048703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3e5680 0x5600fc3e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5aaeffd640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 0x7f5aa40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2795048703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3e5680 msgr2=0x5600fc3e7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3e5680 0x5600fc3e7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2795048703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 msgr2=0x5600fc3da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3da880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2795048703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600fc2cc440 con 0x5600fc3dadc0 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abe1a3640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 0x7f5aa40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009d20 tx=0x7f5ab4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee3565862ea0642 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5aaeffd640 1 -- 172.21.15.16:0/1569223707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5ab00ce5b0 con 0x7f5ab810e480 2024-09-12T02:52:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.325+0000 7f5aa3fff640 1 -- 172.21.15.16:0/2795048703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab401e070 con 0x5600fc3dadc0 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5aa3fff640 1 -- 172.21.15.16:0/2795048703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab4019b10 con 0x5600fc3dadc0 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5aa3fff640 1 -- 172.21.15.16:0/2795048703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab400ebb0 con 0x5600fc3dadc0 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2795048703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 msgr2=0x5600fc3e5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5abe1a3640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 0x7f5aa40a50c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b9b0 tx=0x7f5ab0090350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009d20 tx=0x7f5ab4011810 comp rx=0 tx=0).stop 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2795048703 shutdown_connections 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3e5680 0x5600fc3e7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3dadc0 0x5600fc3e5140 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2795048703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.326+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2795048703 >> 172.21.15.16:0/2795048703 conn(0x5600fc3d6a30 msgr2=0x5600fc3d6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2795048703 shutdown_connections 2024-09-12T02:52:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2795048703 wait complete. 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- start start 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 Processor -- start 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3da4a0 0x5600fc3e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3e2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5600fc3e8250 con 0x5600fc3e5680 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5600fc2fb050 con 0x5600fc3dadc0 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5600fc3e83d0 con 0x5600fc3da4a0 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.327+0000 7f5ac1fdf740 10 client.0 _ll_register_callbacks cb 0x5600fc33b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55026/0 (socket says 172.21.15.16:55026) 2024-09-12T02:52:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 -- 172.21.15.16:0/2727119753 learned_addr learned my addr 172.21.15.16:0/2727119753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3da4a0 0x5600fc3e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 -- 172.21.15.16:0/2727119753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3da4a0 msgr2=0x5600fc3e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3da4a0 0x5600fc3e4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 -- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 msgr2=0x5600fc3e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3e2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 -- 172.21.15.16:0/2727119753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600fc0b13c0 con 0x5600fc3e5680 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8031550 tx=0x7f5aa802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0c8c8351894e8e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5aa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8035040 con 0x5600fc3e5680 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5aa8004020 con 0x5600fc3e5680 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.328+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab4019b10 con 0x5600fc3e5680 2024-09-12T02:52:06.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5aa8004220 con 0x5600fc3e5680 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 10 client.16497 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600fc3f3b40 con 0x5600fc3e5680 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab4019b10 con 0x5600fc3e5680 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5aa8003b10 con 0x5600fc3e5680 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 10 client.16497 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 10 client.16497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 20 client.16497 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3f5000 0x5600fc3f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 4 client.16497 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600fc34d850 con 0x5600fc3f5000 2024-09-12T02:52:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.329+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3f5000 0x5600fc3f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.330+0000 7f5aa17fa640 10 client.16497 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.330+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3f5000 0x5600fc3f73f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e6eaa0e8d044d36 server_cookie=eb308fae55939753 in_seq=0 out_seq=0 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5aa17fa640 1 -- 172.21.15.16:0/2727119753 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600fc34d850 con 0x5600fc3f5000 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5aa17fa640 10 client.16497 handle_command_reply: tid=0 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 client.16497 shutdown 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 msgr2=0x5600fc3e2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8031550 tx=0x7f5aa802a410 comp rx=0 tx=0).stop 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 shutdown_connections 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3f5000 0x5600fc3f73f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3e5680 0x5600fc3e2780 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2727119753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3da4a0 0x5600fc3e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 >> 172.21.15.16:0/2727119753 conn(0x5600fc3d6a30 msgr2=0x5600fc3d5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5aa17fa640 0 client.16497 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 shutdown_connections 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2727119753 wait complete. 2024-09-12T02:52:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.331+0000 7f5ac1fdf740 20 client.16497 trim_cache size 0 max 16384 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 Processor -- start 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 -- start start 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 0x5600fc3f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81af870 con 0x5600fc3da4a0 2024-09-12T02:52:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b13b0 con 0x5600fc3dadc0 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.348+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81aff40 con 0x5600fc3d6a30 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55032/0 (socket says 172.21.15.16:55032) 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2228073716 learned_addr learned my addr 172.21.15.16:0/2228073716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 0x5600fc3f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2228073716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 msgr2=0x5600fc3f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 0x5600fc3f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2228073716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 msgr2=0x5600fc3fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3fcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 -- 172.21.15.16:0/2228073716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600fc3dabf0 con 0x5600fc3da4a0 2024-09-12T02:52:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 0x5600fc3f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009ab0 tx=0x7f5ab401aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f44aba6edc6a3a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5aa37fe640 1 -- 172.21.15.16:0/2228073716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab401e040 con 0x5600fc3da4a0 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5aa37fe640 1 -- 172.21.15.16:0/2228073716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab400e860 con 0x5600fc3da4a0 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.349+0000 7f5aa37fe640 1 -- 172.21.15.16:0/2228073716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40048a0 con 0x5600fc3da4a0 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2228073716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 msgr2=0x5600fc3dd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009ab0 tx=0x7f5ab401aee0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2228073716 shutdown_connections 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3dadc0 0x5600fc3fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3da4a0 0x5600fc3dd740 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/2228073716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3d6a30 0x5600fc3f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2228073716 >> 172.21.15.16:0/2228073716 conn(0x5600fc3e28e0 msgr2=0x5600fc3dd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2228073716 shutdown_connections 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/2228073716 wait complete. 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 -- start start 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.350+0000 7f5ac1fdf740 1 Processor -- start 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3e9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3dadc0 0x5600fc3e9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b0ce0 con 0x5600fc3d6a30 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81b0610 con 0x5600fc3da4a0 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab81ad850 con 0x5600fc3dadc0 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5ac1fdf740 10 client.0 _ll_register_callbacks cb 0x7f5aa8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55046/0 (socket says 172.21.15.16:55046) 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 -- 172.21.15.16:0/488300576 learned_addr learned my addr 172.21.15.16:0/488300576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 -- 172.21.15.16:0/488300576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3dadc0 msgr2=0x5600fc3e9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3dadc0 0x5600fc3e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 -- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 msgr2=0x5600fc3e9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3e9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 -- 172.21.15.16:0/488300576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600fc3e9fa0 con 0x5600fc3d6a30 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.351+0000 7f5abd1a1640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8003f60 tx=0x7f5aa804b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af831a9cd28b675 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8035040 con 0x5600fc3d6a30 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5aa80043c0 con 0x5600fc3d6a30 2024-09-12T02:52:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab400e860 con 0x5600fc3d6a30 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5aa8009520 con 0x5600fc3d6a30 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 10 client.16503 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600fc45fd10 con 0x5600fc3d6a30 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab400e860 con 0x5600fc3d6a30 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5aa8004770 con 0x5600fc3d6a30 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 10 client.16503 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 20 client.16503 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3e5680 0x5600fc462380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 4 client.16503 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.352+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5600fc347d60 con 0x5600fc3e5680 2024-09-12T02:52:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3e5680 0x5600fc462380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5abd9a2640 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3e5680 0x5600fc462380 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b02c0e60f81a8cf0 server_cookie=e818a85ab54ab9de in_seq=0 out_seq=0 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5aa17fa640 10 client.16503 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5aa17fa640 1 -- 172.21.15.16:0/488300576 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5600fc347d60 con 0x5600fc3e5680 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5aa17fa640 10 client.16503 handle_command_reply: tid=0 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5ac1fdf740 1 client.16503 shutdown 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 msgr2=0x5600fc3ede50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8003f60 tx=0x7f5aa804b780 comp rx=0 tx=0).stop 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.353+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 shutdown_connections 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5600fc3e5680 0x5600fc462380 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5600fc3dadc0 0x5600fc3e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5600fc3da4a0 0x5600fc3e9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 --2- 172.21.15.16:0/488300576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5600fc3d6a30 0x5600fc3ede50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5aa17fa640 0 client.16503 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 >> 172.21.15.16:0/488300576 conn(0x5600fc3e28e0 msgr2=0x5600fc3dd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 shutdown_connections 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 1 -- 172.21.15.16:0/488300576 wait complete. 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5ac1fdf740 20 client.16503 trim_cache size 0 max 16384 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:52:06.362 INFO:teuthology.orchestra.run.smithi016.stdout: "7860bffd-68e0-4211-b5ac-92ba7059e31b": { 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "7860bffd-68e0-4211-b5ac-92ba7059e31b", 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 msgr2=0x7f5aa40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 0x7f5aa40a50c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b9b0 tx=0x7f5ab0090350 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 msgr2=0x7f5ab819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b9e0 tx=0x7f5ab000bea0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 shutdown_connections 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5aa40a2c00 0x7f5aa40a50c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab810e480 0x7f5ab819f010 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab8101280 0x7f5ab819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 --2- 172.21.15.16:0/1569223707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ab81008d0 0x7f5ab819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 >> 172.21.15.16:0/1569223707 conn(0x7f5ab80fc7a0 msgr2=0x7f5ab810f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 shutdown_connections 2024-09-12T02:52:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:06.354+0000 7f5abfc06640 1 -- 172.21.15.16:0/1569223707 wait complete. 2024-09-12T02:52:06.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.37 MB/sec execute 68 sec latency 3242.050 ms 2024-09-12T02:52:07.222 INFO:tasks.cephfs.filesystem:scrub status for tag:7860bffd-68e0-4211-b5ac-92ba7059e31b - {'path': '/', 'tag': '7860bffd-68e0-4211-b5ac-92ba7059e31b', 'options': 'recursive,force'} 2024-09-12T02:52:07.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.23 MB/sec execute 69 sec latency 4242.166 ms 2024-09-12T02:52:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:07 smithi033 ceph-mon[35092]: pgmap v463: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-09-12T02:52:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:07 smithi184 ceph-mon[35321]: pgmap v463: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-09-12T02:52:08.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:07 smithi016 ceph-mon[30925]: pgmap v463: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-09-12T02:52:08.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 9.10 MB/sec execute 70 sec latency 5242.287 ms 2024-09-12T02:52:09.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.97 MB/sec execute 71 sec latency 6242.411 ms 2024-09-12T02:52:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:09 smithi033 ceph-mon[35092]: pgmap v464: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-12T02:52:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:09 smithi184 ceph-mon[35321]: pgmap v464: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-12T02:52:10.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:09 smithi016 ceph-mon[30925]: pgmap v464: 97 pgs: 97 active+clean; 137 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-09-12T02:52:10.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.85 MB/sec execute 72 sec latency 7242.507 ms 2024-09-12T02:52:11.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.72 MB/sec execute 73 sec latency 8242.611 ms 2024-09-12T02:52:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:11 smithi033 ceph-mon[35092]: pgmap v465: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-09-12T02:52:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:11 smithi184 ceph-mon[35321]: pgmap v465: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-09-12T02:52:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:11 smithi016 ceph-mon[30925]: pgmap v465: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-09-12T02:52:12.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.61 MB/sec execute 74 sec latency 9242.706 ms 2024-09-12T02:52:13.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:12 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:52:13.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:12 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:52:13.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:12 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:52:13.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:12 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:52:13.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:12 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:52:13.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:12 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:52:13.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.49 MB/sec execute 75 sec latency 10242.816 ms 2024-09-12T02:52:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:13 smithi033 ceph-mon[35092]: pgmap v466: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 138 op/s 2024-09-12T02:52:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:13 smithi184 ceph-mon[35321]: pgmap v466: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 138 op/s 2024-09-12T02:52:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:13 smithi016 ceph-mon[30925]: pgmap v466: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 138 op/s 2024-09-12T02:52:14.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.38 MB/sec execute 76 sec latency 11242.911 ms 2024-09-12T02:52:15.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.27 MB/sec execute 77 sec latency 12243.050 ms 2024-09-12T02:52:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:15 smithi033 ceph-mon[35092]: pgmap v467: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-12T02:52:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:15 smithi184 ceph-mon[35321]: pgmap v467: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-12T02:52:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:15 smithi016 ceph-mon[30925]: pgmap v467: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-09-12T02:52:16.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.16 MB/sec execute 78 sec latency 13243.122 ms 2024-09-12T02:52:17.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:52:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:17.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 8.06 MB/sec execute 79 sec latency 14243.223 ms 2024-09-12T02:52:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.969+0000 7f9e6cef1640 1 -- 172.21.15.16:0/307939861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68102790 msgr2=0x7f9e681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.969+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/307939861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68102790 0x7f9e681067e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009900 tx=0x7f9e5002a410 comp rx=0 tx=0).stop 2024-09-12T02:52:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 -- 172.21.15.16:0/307939861 shutdown_connections 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/307939861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68106f10 0x7f9e6810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/307939861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68102790 0x7f9e681067e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/307939861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68101de0 0x7f9e681021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 -- 172.21.15.16:0/307939861 >> 172.21.15.16:0/307939861 conn(0x7f9e680fd8c0 msgr2=0x7f9e680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 -- 172.21.15.16:0/307939861 shutdown_connections 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 -- 172.21.15.16:0/307939861 wait complete. 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.970+0000 7f9e6cef1640 1 Processor -- start 2024-09-12T02:52:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 -- start start 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 0x7f9e68198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68106f10 0x7f9e6819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68111f70 con 0x7f9e68101de0 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68111df0 con 0x7f9e68106f10 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6cef1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681120f0 con 0x7f9e68102790 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43022/0 (socket says 172.21.15.16:43022) 2024-09-12T02:52:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 -- 172.21.15.16:0/283453727 learned_addr learned my addr 172.21.15.16:0/283453727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e65d9b640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 0x7f9e68198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e66d9d640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68106f10 0x7f9e6819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 -- 172.21.15.16:0/283453727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 msgr2=0x7f9e68198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 0x7f9e68198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 -- 172.21.15.16:0/283453727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68106f10 msgr2=0x7f9e6819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68106f10 0x7f9e6819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e6659c640 1 -- 172.21.15.16:0/283453727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6819d3c0 con 0x7f9e68101de0 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.971+0000 7f9e65d9b640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 0x7f9e68198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e6659c640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d950 tx=0x7f9e5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017070 con 0x7f9e68101de0 2024-09-12T02:52:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58009c30 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6819d6b0 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5800f6b0 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.972+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e681a4f90 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.973+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e681021c0 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.974+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9e580109d0 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.975+0000 7f9e5f7fe640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 0x7f9e540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.975+0000 7f9e65d9b640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 0x7f9e540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.975+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9e580cedd0 con 0x7f9e68101de0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.975+0000 7f9e65d9b640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 0x7f9e540a5040 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e50000c00 tx=0x7f9e50032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:17.979+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9e58009de0 con 0x7f9e68101de0 2024-09-12T02:52:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.190+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e68199770 con 0x7f9e68101de0 2024-09-12T02:52:18.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:17 smithi016 ceph-mon[30925]: pgmap v468: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-09-12T02:52:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.191+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/283453727 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f9e5809bce0 con 0x7f9e68101de0 2024-09-12T02:52:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:52:18.194 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 msgr2=0x7f9e540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 0x7f9e540a5040 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e50000c00 tx=0x7f9e50032c70 comp rx=0 tx=0).stop 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 msgr2=0x7f9e68198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d950 tx=0x7f9e5800de10 comp rx=0 tx=0).stop 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 shutdown_connections 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e68106f10 0x7f9e6819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e68102790 0x7f9e68198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e540a2b80 0x7f9e540a5040 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 --2- 172.21.15.16:0/283453727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e68101de0 0x7f9e68198410 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 >> 172.21.15.16:0/283453727 conn(0x7f9e680fd8c0 msgr2=0x7f9e680fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 shutdown_connections 2024-09-12T02:52:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:18.194+0000 7f9e6cef1640 1 -- 172.21.15.16:0/283453727 wait complete. 2024-09-12T02:52:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:17 smithi033 ceph-mon[35092]: pgmap v468: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-09-12T02:52:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:17 smithi184 ceph-mon[35321]: pgmap v468: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-09-12T02:52:18.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 7.96 MB/sec execute 80 sec latency 15243.317 ms 2024-09-12T02:52:18.934 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id bf2304de --release --await' 2024-09-12T02:52:18.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id bf2304de --release --await 2024-09-12T02:52:19.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/283453727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:19.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/283453727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/283453727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 -- 172.21.15.16:0/1915655761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0103660 msgr2=0x7f4ee010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/1915655761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0103660 0x7f4ee010bb50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009900 tx=0x7f4ec802a480 comp rx=0 tx=0).stop 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 -- 172.21.15.16:0/1915655761 shutdown_connections 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/1915655761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee010c170 0x7f4ee010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/1915655761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0103660 0x7f4ee010bb50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/1915655761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 0x7f4ee0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 -- 172.21.15.16:0/1915655761 >> 172.21.15.16:0/1915655761 conn(0x7f4ee00fc7a0 msgr2=0x7f4ee00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 -- 172.21.15.16:0/1915655761 shutdown_connections 2024-09-12T02:52:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.678+0000 7f4ee48c8640 1 -- 172.21.15.16:0/1915655761 wait complete. 2024-09-12T02:52:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 Processor -- start 2024-09-12T02:52:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 -- start start 2024-09-12T02:52:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 0x7f4ee01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee0103660 0x7f4ee0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee0111f70 con 0x7f4ee010c170 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee0111df0 con 0x7f4ee0103660 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ee48c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01120f0 con 0x7f4ee0102d40 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4eded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4eded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43034/0 (socket says 172.21.15.16:43034) 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4eded9d640 1 -- 172.21.15.16:0/3498611938 learned_addr learned my addr 172.21.15.16:0/3498611938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ede59c640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 0x7f4ee01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.679+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee0103660 0x7f4ee0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 -- 172.21.15.16:0/3498611938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 msgr2=0x7f4ee01009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 0x7f4ee01009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 -- 172.21.15.16:0/3498611938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee0103660 msgr2=0x7f4ee0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee0103660 0x7f4ee0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 -- 172.21.15.16:0/3498611938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee01a6930 con 0x7f4ee010c170 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009fd0 tx=0x7f4ed800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4ed7fff640 1 -- 172.21.15.16:0/3498611938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8017070 con 0x7f4ee010c170 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee01a6c20 con 0x7f4ee010c170 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4ed7fff640 1 -- 172.21.15.16:0/3498611938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed8003e50 con 0x7f4ee010c170 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4ed7fff640 1 -- 172.21.15.16:0/3498611938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80106c0 con 0x7f4ee010c170 2024-09-12T02:52:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.680+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee01a7160 con 0x7f4ee010c170 2024-09-12T02:52:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.682+0000 7f4ed7fff640 1 -- 172.21.15.16:0/3498611938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4ed8003970 con 0x7f4ee010c170 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.682+0000 7f4ee6ca1740 1 Processor -- start 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 -- start start 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c30d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c33630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5652d3b8b620 con 0x5652d3c26060 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5652d3b32e50 con 0x5652d3c31240 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5652d3b875e0 con 0x5652d3c26980 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed7fff640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 0x7f4ecc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43036/0 (socket says 172.21.15.16:43036) 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/2559574929 learned_addr learned my addr 172.21.15.16:0/2559574929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c30d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 0x7f4ecc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c33630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.683+0000 7f4ed7fff640 1 -- 172.21.15.16:0/3498611938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4ed80cedc0 con 0x7f4ee010c170 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/2559574929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 msgr2=0x5652d3c30d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c30d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/2559574929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 msgr2=0x5652d3c33630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c33630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/2559574929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d3b23d40 con 0x5652d3c26060 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8005940 tx=0x7f4ec80314d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7975c4c25fe6bf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 0x7f4ecc0a5040 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ed809b560 tx=0x7f4ed8066250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed4d99640 1 -- 172.21.15.16:0/2559574929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8035030 con 0x5652d3c26060 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed4d99640 1 -- 172.21.15.16:0/2559574929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec8004030 con 0x5652d3c26060 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.684+0000 7f4ed4d99640 1 -- 172.21.15.16:0/2559574929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8005610 con 0x5652d3c26060 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/2559574929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 msgr2=0x5652d3c26440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8005940 tx=0x7f4ec80314d0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/2559574929 shutdown_connections 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c33630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c30d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/2559574929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c26440 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/2559574929 >> 172.21.15.16:0/2559574929 conn(0x5652d3c225f0 msgr2=0x5652d3c229d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/2559574929 shutdown_connections 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/2559574929 wait complete. 2024-09-12T02:52:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 -- start start 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.685+0000 7f4ee6ca1740 1 Processor -- start 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26060 0x5652d3c2e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c2ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5652d3c33e10 con 0x5652d3c26980 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5652d3b13f10 con 0x5652d3c31240 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5652d3c33f90 con 0x5652d3c26060 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ee6ca1740 10 client.0 _ll_register_callbacks cb 0x5652d3b37db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43050/0 (socket says 172.21.15.16:43050) 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 -- 172.21.15.16:0/1452076379 learned_addr learned my addr 172.21.15.16:0/1452076379 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26060 0x5652d3c2e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c2ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 -- 172.21.15.16:0/1452076379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26060 msgr2=0x5652d3c2e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26060 0x5652d3c2e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 -- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 msgr2=0x5652d3c2ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c2ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 -- 172.21.15.16:0/1452076379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d389a580 con 0x5652d3c26980 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c2ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000bb00 tx=0x7f4ed000bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe473977f324f6f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.686+0000 7f4ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003ee0 con 0x5652d3c26980 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4ed0005020 con 0x5652d3c26980 2024-09-12T02:52:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec8004030 con 0x5652d3c26980 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ed0005260 con 0x5652d3c26980 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 10 client.16521 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5652d3964960 con 0x5652d3c26980 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec8004030 con 0x5652d3c26980 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4ed001b620 con 0x5652d3c26980 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 10 client.16521 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 10 client.16521 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 20 client.16521 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3c40ee0 0x5652d3c43300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 4 client.16521 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.687+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652d3b98b90 con 0x5652d3c40ee0 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.688+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3c40ee0 0x5652d3c43300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.688+0000 7f4ec5ffb640 10 client.16521 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.688+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3c40ee0 0x5652d3c43300 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7552ecf3593be02 server_cookie=f2e9605c23c8b9e5 in_seq=0 out_seq=0 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.689+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1452076379 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652d3b98b90 con 0x5652d3c40ee0 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.689+0000 7f4ec5ffb640 10 client.16521 handle_command_reply: tid=0 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.689+0000 7f4ee6ca1740 1 client.16521 shutdown 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 msgr2=0x5652d3c2e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000bb00 tx=0x7f4ed000bf00 comp rx=0 tx=0).stop 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 shutdown_connections 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3c40ee0 0x5652d3c43300 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c2ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26980 0x5652d3c2e670 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1452076379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26060 0x5652d3c2e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 >> 172.21.15.16:0/1452076379 conn(0x5652d3c225f0 msgr2=0x5652d3c28320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ec5ffb640 0 client.16521 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 shutdown_connections 2024-09-12T02:52:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1452076379 wait complete. 2024-09-12T02:52:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.690+0000 7f4ee6ca1740 20 client.16521 trim_cache size 0 max 16384 2024-09-12T02:52:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 Processor -- start 2024-09-12T02:52:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 -- start start 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c48bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c49110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01af630 con 0x5652d3c26060 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01b1170 con 0x5652d3c31240 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01afd00 con 0x5652d3c26980 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.707+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43052/0 (socket says 172.21.15.16:43052) 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/871114465 learned_addr learned my addr 172.21.15.16:0/871114465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ede59c640 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c48bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/871114465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 msgr2=0x5652d3c48bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c48bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/871114465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 msgr2=0x5652d3c49110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c49110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/871114465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d3c267b0 con 0x5652d3c26060 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8033000 tx=0x7f4ec8005150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59236fc0e6a4467 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ec7fff640 1 -- 172.21.15.16:0/871114465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8035040 con 0x5652d3c26060 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ec7fff640 1 -- 172.21.15.16:0/871114465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec8053020 con 0x5652d3c26060 2024-09-12T02:52:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.708+0000 7f4ec7fff640 1 -- 172.21.15.16:0/871114465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80048e0 con 0x5652d3c26060 2024-09-12T02:52:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/871114465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 msgr2=0x5652d3c300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8033000 tx=0x7f4ec8005150 comp rx=0 tx=0).stop 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/871114465 shutdown_connections 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c31240 0x5652d3c49110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c48bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/871114465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c26060 0x5652d3c300a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/871114465 >> 172.21.15.16:0/871114465 conn(0x5652d3c225f0 msgr2=0x5652d3c2f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/871114465 shutdown_connections 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/871114465 wait complete. 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 -- start start 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.709+0000 7f4ee6ca1740 1 Processor -- start 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c26060 0x5652d3c397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c2ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01b0aa0 con 0x5652d3c31240 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01b03d0 con 0x5652d3c26060 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee01ad610 con 0x5652d3c26980 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ee6ca1740 10 client.0 _ll_register_callbacks cb 0x7f4ed0004480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43068/0 (socket says 172.21.15.16:43068) 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/1699145652 learned_addr learned my addr 172.21.15.16:0/1699145652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c26060 0x5652d3c397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c2ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/1699145652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 msgr2=0x5652d3c2ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c2ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c26060 msgr2=0x5652d3c397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c26060 0x5652d3c397e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 -- 172.21.15.16:0/1699145652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d3c2b910 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ed5d9b640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x5652d3c2a5a0 tx=0x7f4ec80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c76ed6295daa678 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.710+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8035070 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4ec8054040 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec8053020 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ec8031630 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 10 client.16527 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5652d3caba90 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec8053020 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4ec8068510 con 0x5652d3c31240 2024-09-12T02:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 10 client.16527 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 10 client.16527 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 20 client.16527 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3cabcb0 0x5652d3cae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 4 client.16527 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bf2304de", "release": true, "await": true}] 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.711+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bf2304de", "release": true, "await": true}) -- 0x5652d3b930a0 con 0x5652d3cabcb0 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.712+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3cabcb0 0x5652d3cae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.712+0000 7f4ede59c640 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3cabcb0 0x5652d3cae0a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f0ef53c21e9972 server_cookie=61d24e9a65053319 in_seq=0 out_seq=0 2024-09-12T02:52:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.712+0000 7f4ec5ffb640 10 client.16527 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:19.716 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339750 7.86 MB/sec execute 81 sec latency 16243.345 ms 2024-09-12T02:52:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/1699145652 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5652d3b930a0 con 0x5652d3cabcb0 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ec5ffb640 10 client.16527 handle_command_reply: tid=0 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 client.16527 shutdown 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 msgr2=0x5652d3c2b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x5652d3c2a5a0 tx=0x7f4ec80046c0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 shutdown_connections 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5652d3cabcb0 0x5652d3cae0a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652d3c31240 0x5652d3c2b270 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5652d3c26980 0x5652d3c2ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 --2- 172.21.15.16:0/1699145652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5652d3c26060 0x5652d3c397e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 >> 172.21.15.16:0/1699145652 conn(0x5652d3c225f0 msgr2=0x5652d3c21750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ec5ffb640 0 client.16527 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 shutdown_connections 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 1 -- 172.21.15.16:0/1699145652 wait complete. 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.723+0000 7f4ee6ca1740 20 client.16527 trim_cache size 0 max 16384 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 20, 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:52:19.725 INFO:teuthology.orchestra.run.smithi016.stdout: "bf2304de": { 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 20, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.2, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:52:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:19.727 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 msgr2=0x7f4ecc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 0x7f4ecc0a5040 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ed809b560 tx=0x7f4ed8066250 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 msgr2=0x7f4ee0101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009fd0 tx=0x7f4ed800b260 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 shutdown_connections 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ecc0a2b80 0x7f4ecc0a5040 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee010c170 0x7f4ee0101420 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ee0103660 0x7f4ee0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 --2- 172.21.15.16:0/3498611938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ee0102d40 0x7f4ee01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 >> 172.21.15.16:0/3498611938 conn(0x7f4ee00fc7a0 msgr2=0x7f4ee00fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 shutdown_connections 2024-09-12T02:52:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:19.724+0000 7f4ee48c8640 1 -- 172.21.15.16:0/3498611938 wait complete. 2024-09-12T02:52:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:19 smithi033 ceph-mon[35092]: pgmap v469: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 583 KiB/s wr, 57 op/s 2024-09-12T02:52:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:19 smithi184 ceph-mon[35321]: pgmap v469: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 583 KiB/s wr, 57 op/s 2024-09-12T02:52:20.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:19 smithi016 ceph-mon[30925]: pgmap v469: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 583 KiB/s wr, 57 op/s 2024-09-12T02:52:20.562 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bf2304de', total seconds elapsed: 23.5 2024-09-12T02:52:20.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 341966 7.93 MB/sec execute 82 sec latency 16252.247 ms 2024-09-12T02:52:21.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 344734 8.00 MB/sec execute 83 sec latency 11.010 ms 2024-09-12T02:52:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:21 smithi033 ceph-mon[35092]: pgmap v470: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 57 KiB/s wr, 23 op/s 2024-09-12T02:52:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:21 smithi184 ceph-mon[35321]: pgmap v470: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 57 KiB/s wr, 23 op/s 2024-09-12T02:52:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:21 smithi016 ceph-mon[30925]: pgmap v470: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 57 KiB/s wr, 23 op/s 2024-09-12T02:52:22.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 347905 8.13 MB/sec execute 84 sec latency 11.628 ms 2024-09-12T02:52:23.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 350878 8.21 MB/sec execute 85 sec latency 12.613 ms 2024-09-12T02:52:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:23 smithi033 ceph-mon[35092]: pgmap v471: 97 pgs: 97 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 196 KiB/s wr, 41 op/s 2024-09-12T02:52:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:23 smithi184 ceph-mon[35321]: pgmap v471: 97 pgs: 97 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 196 KiB/s wr, 41 op/s 2024-09-12T02:52:24.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:23 smithi016 ceph-mon[30925]: pgmap v471: 97 pgs: 97 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 196 KiB/s wr, 41 op/s 2024-09-12T02:52:24.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 353710 8.26 MB/sec execute 86 sec latency 11.380 ms 2024-09-12T02:52:25.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356153 8.35 MB/sec execute 87 sec latency 12.852 ms 2024-09-12T02:52:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:25 smithi033 ceph-mon[35092]: pgmap v472: 97 pgs: 97 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 395 KiB/s wr, 75 op/s 2024-09-12T02:52:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:25 smithi184 ceph-mon[35321]: pgmap v472: 97 pgs: 97 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 395 KiB/s wr, 75 op/s 2024-09-12T02:52:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:25 smithi016 ceph-mon[30925]: pgmap v472: 97 pgs: 97 active+clean; 140 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 395 KiB/s wr, 75 op/s 2024-09-12T02:52:26.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 358861 8.42 MB/sec execute 88 sec latency 19.134 ms 2024-09-12T02:52:27.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 362102 8.53 MB/sec execute 89 sec latency 24.813 ms 2024-09-12T02:52:28.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:27 smithi033 ceph-mon[35092]: pgmap v473: 97 pgs: 97 active+clean; 141 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 548 KiB/s wr, 87 op/s 2024-09-12T02:52:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:27 smithi184 ceph-mon[35321]: pgmap v473: 97 pgs: 97 active+clean; 141 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 548 KiB/s wr, 87 op/s 2024-09-12T02:52:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:27 smithi016 ceph-mon[30925]: pgmap v473: 97 pgs: 97 active+clean; 141 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 548 KiB/s wr, 87 op/s 2024-09-12T02:52:28.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 365304 8.63 MB/sec execute 90 sec latency 13.736 ms 2024-09-12T02:52:29.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367735 8.64 MB/sec execute 91 sec latency 60.281 ms 2024-09-12T02:52:30.209 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:29 smithi016 ceph-mon[30925]: pgmap v474: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 646 KiB/s wr, 91 op/s 2024-09-12T02:52:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:29 smithi033 ceph-mon[35092]: pgmap v474: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 646 KiB/s wr, 91 op/s 2024-09-12T02:52:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:29 smithi184 ceph-mon[35321]: pgmap v474: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 646 KiB/s wr, 91 op/s 2024-09-12T02:52:30.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370038 8.70 MB/sec execute 92 sec latency 13.993 ms 2024-09-12T02:52:31.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 373020 8.79 MB/sec execute 93 sec latency 19.043 ms 2024-09-12T02:52:32.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:31 smithi033 ceph-mon[35092]: pgmap v475: 97 pgs: 97 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-09-12T02:52:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:31 smithi184 ceph-mon[35321]: pgmap v475: 97 pgs: 97 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-09-12T02:52:32.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:31 smithi016 ceph-mon[30925]: pgmap v475: 97 pgs: 97 active+clean; 147 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-09-12T02:52:32.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376203 8.89 MB/sec execute 94 sec latency 11.692 ms 2024-09-12T02:52:33.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 378596 8.89 MB/sec execute 95 sec latency 53.333 ms 2024-09-12T02:52:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:33 smithi033 ceph-mon[35092]: pgmap v476: 97 pgs: 97 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-09-12T02:52:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:33 smithi184 ceph-mon[35321]: pgmap v476: 97 pgs: 97 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-09-12T02:52:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:33 smithi016 ceph-mon[30925]: pgmap v476: 97 pgs: 97 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-09-12T02:52:34.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 380742 8.93 MB/sec execute 96 sec latency 71.623 ms 2024-09-12T02:52:35.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 383582 9.01 MB/sec execute 97 sec latency 17.328 ms 2024-09-12T02:52:36.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:35 smithi033 ceph-mon[35092]: pgmap v477: 97 pgs: 97 active+clean; 151 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-12T02:52:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:35 smithi184 ceph-mon[35321]: pgmap v477: 97 pgs: 97 active+clean; 151 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-12T02:52:36.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:35 smithi016 ceph-mon[30925]: pgmap v477: 97 pgs: 97 active+clean; 151 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-12T02:52:36.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 386782 9.10 MB/sec execute 98 sec latency 16.157 ms 2024-09-12T02:52:37.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:52:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:37.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 389266 9.11 MB/sec execute 99 sec latency 18.933 ms 2024-09-12T02:52:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 -- 172.21.15.16:0/3277643560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8100a60 msgr2=0x7f17a8100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 --2- 172.21.15.16:0/3277643560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8100a60 0x7f17a8100ec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1790009890 tx=0x7f179002a360 comp rx=0 tx=0).stop 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 -- 172.21.15.16:0/3277643560 shutdown_connections 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 --2- 172.21.15.16:0/3277643560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8101400 0x7f17a810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 --2- 172.21.15.16:0/3277643560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8100a60 0x7f17a8100ec0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 --2- 172.21.15.16:0/3277643560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8106a60 0x7f17a8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 -- 172.21.15.16:0/3277643560 >> 172.21.15.16:0/3277643560 conn(0x7f17a80fc780 msgr2=0x7f17a80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 -- 172.21.15.16:0/3277643560 shutdown_connections 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.038+0000 7f17ae046640 1 -- 172.21.15.16:0/3277643560 wait complete. 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 Processor -- start 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 -- start start 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8106a60 0x7f17a810f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17a8111f10 con 0x7f17a8100a60 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a8111d90 con 0x7f17a8106a60 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17ae046640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f17a8112090 con 0x7f17a8101400 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.039+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57116/0 (socket says 172.21.15.16:57116) 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50114/0 (socket says 172.21.15.16:50114) 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 -- 172.21.15.16:0/3632423396 learned_addr learned my addr 172.21.15.16:0/3632423396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8106a60 0x7f17a810f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 -- 172.21.15.16:0/3632423396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8106a60 msgr2=0x7f17a810f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8106a60 0x7f17a810f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 -- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 msgr2=0x7f17a810e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 -- 172.21.15.16:0/3632423396 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a81a6970 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f179effd640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1790000c00 tx=0x7f1790033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a57fa640 1 -- 172.21.15.16:0/3632423396 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179003f020 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a81a6c00 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a57fa640 1 -- 172.21.15.16:0/3632423396 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1790032770 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17a57fa640 1 -- 172.21.15.16:0/3632423396 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790037040 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.040+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a81a70e0 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.043+0000 7f17a57fa640 1 -- 172.21.15.16:0/3632423396 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1790042020 con 0x7f17a8101400 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.043+0000 7f17b041f740 1 Processor -- start 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.043+0000 7f17b041f740 1 -- start start 2024-09-12T02:52:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1834164a0 0x55a183416880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a183423a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a18337c2e0 con 0x55a183416dc0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a183323ce0 con 0x55a1834164a0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17b041f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a1833782a0 con 0x55a183421680 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50116/0 (socket says 172.21.15.16:50116) 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 -- 172.21.15.16:0/3194933213 learned_addr learned my addr 172.21.15.16:0/3194933213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a57fa640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 0x7f17940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1834164a0 0x55a183416880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f179effd640 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a183423a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 0x7f17940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 -- 172.21.15.16:0/3194933213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 msgr2=0x55a183423a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a183423a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 -- 172.21.15.16:0/3194933213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1834164a0 msgr2=0x55a183416880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1834164a0 0x55a183416880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.044+0000 7f17a77fe640 1 -- 172.21.15.16:0/3194933213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a183308440 con 0x55a183416dc0 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f17a57fa640 1 -- 172.21.15.16:0/3632423396 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f17900ef100 con 0x7f17a8101400 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1798011470 tx=0x7f1798011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8159caccd6bd94d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f179dffb640 1 -- 172.21.15.16:0/3194933213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798005020 con 0x55a183416dc0 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f179dffb640 1 -- 172.21.15.16:0/3194933213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17980047a0 con 0x55a183416dc0 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f179dffb640 1 -- 172.21.15.16:0/3194933213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179800f970 con 0x55a183416dc0 2024-09-12T02:52:38.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.045+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 0x7f17940a5040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009fd0 tx=0x7f17a0004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- 172.21.15.16:0/3194933213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 msgr2=0x55a183421140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1798011470 tx=0x7f1798011930 comp rx=0 tx=0).stop 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- 172.21.15.16:0/3194933213 shutdown_connections 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a183423a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183421140 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 --2- 172.21.15.16:0/3194933213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1834164a0 0x55a183416880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- 172.21.15.16:0/3194933213 >> 172.21.15.16:0/3194933213 conn(0x55a183412a30 msgr2=0x55a183412e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- 172.21.15.16:0/3194933213 shutdown_connections 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- 172.21.15.16:0/3194933213 wait complete. 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 -- start start 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.046+0000 7f17b041f740 1 Processor -- start 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 0x55a18341e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a18341e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a183424250 con 0x55a1834164a0 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a183337050 con 0x55a183416dc0 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a1834243d0 con 0x55a183421680 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17b041f740 10 client.0 _ll_register_callbacks cb 0x55a183377160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 0x55a18341e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50128/0 (socket says 172.21.15.16:50128) 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17a77fe640 1 -- 172.21.15.16:0/3818347116 learned_addr learned my addr 172.21.15.16:0/3818347116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a18341e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 -- 172.21.15.16:0/3818347116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 msgr2=0x55a18341e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a18341e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 -- 172.21.15.16:0/3818347116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 msgr2=0x55a183420f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 -- 172.21.15.16:0/3818347116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1830ed3c0 con 0x55a183416dc0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f179effd640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 0x55a18341e1e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f17900b7d70 tx=0x7f1790004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e93a03e61ac793da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.047+0000 7f1786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17900c4580 con 0x55a183416dc0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f17900c40f0 con 0x55a183416dc0 2024-09-12T02:52:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17980047a0 con 0x55a183416dc0 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f17900bb910 con 0x55a183416dc0 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 10 client.26087 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a18342fb70 con 0x55a183416dc0 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17980047a0 con 0x55a183416dc0 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f179003a030 con 0x55a183416dc0 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 10 client.26087 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 20 client.26087 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183430d90 0x55a183433230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 4 client.26087 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a183389850 con 0x55a183430d90 2024-09-12T02:52:38.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.048+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183430d90 0x55a183433230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.049+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183430d90 0x55a183433230 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6318535f17b3645a server_cookie=23df62622deb3e6c in_seq=0 out_seq=0 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.049+0000 7f1786ffd640 10 client.26087 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.049+0000 7f1786ffd640 1 -- 172.21.15.16:0/3818347116 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a183389850 con 0x55a183430d90 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.049+0000 7f1786ffd640 10 client.26087 handle_command_reply: tid=0 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 client.26087 shutdown 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 msgr2=0x55a18341e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 0x55a18341e1e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f17900b7d70 tx=0x7f1790004570 comp rx=0 tx=0).stop 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 shutdown_connections 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183430d90 0x55a183433230 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a183421680 0x55a18341e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183416dc0 0x55a18341e1e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f1786ffd640 0 client.26087 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 --2- 172.21.15.16:0/3818347116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1834164a0 0x55a183420f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 >> 172.21.15.16:0/3818347116 conn(0x55a183412a30 msgr2=0x55a183418760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 shutdown_connections 2024-09-12T02:52:38.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 1 -- 172.21.15.16:0/3818347116 wait complete. 2024-09-12T02:52:38.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.050+0000 7f17b041f740 20 client.26087 trim_cache size 0 max 16384 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 Processor -- start 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 -- start start 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a18341e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183419740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81af700 con 0x55a183416dc0 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81b1240 con 0x55a183412a30 2024-09-12T02:52:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17b041f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81afdd0 con 0x55a1834164a0 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52556/0 (socket says 172.21.15.16:52556) 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17a77fe640 1 -- 172.21.15.16:0/1750703033 learned_addr learned my addr 172.21.15.16:0/1750703033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f179effd640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a18341e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.070+0000 7f17a7fff640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183419740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 -- 172.21.15.16:0/1750703033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 msgr2=0x55a18341e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a18341e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 -- 172.21.15.16:0/1750703033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 msgr2=0x55a183419740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183419740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 -- 172.21.15.16:0/1750703033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1834212a0 con 0x55a183412a30 2024-09-12T02:52:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a7fff640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183419740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f17a77fe640 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17980050a0 tx=0x7f1798005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15fdcbecbab791b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f179d7fa640 1 -- 172.21.15.16:0/1750703033 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798004c60 con 0x55a183412a30 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f179d7fa640 1 -- 172.21.15.16:0/1750703033 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179800f540 con 0x55a183412a30 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.071+0000 7f179d7fa640 1 -- 172.21.15.16:0/1750703033 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179802f6c0 con 0x55a183412a30 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- 172.21.15.16:0/1750703033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 msgr2=0x55a18342d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17980050a0 tx=0x7f1798005a00 comp rx=0 tx=0).stop 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- 172.21.15.16:0/1750703033 shutdown_connections 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a183419740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a18341e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 --2- 172.21.15.16:0/1750703033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a18342d4b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- 172.21.15.16:0/1750703033 >> 172.21.15.16:0/1750703033 conn(0x55a18341e880 msgr2=0x55a183419030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- 172.21.15.16:0/1750703033 shutdown_connections 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- 172.21.15.16:0/1750703033 wait complete. 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.072+0000 7f17b041f740 1 -- start start 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 Processor -- start 2024-09-12T02:52:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a183424fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a1834254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81b0b70 con 0x55a183416dc0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81b04a0 con 0x55a183412a30 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81ad6e0 con 0x55a1834164a0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17b041f740 10 client.0 _ll_register_callbacks cb 0x7f17900097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52566/0 (socket says 172.21.15.16:52566) 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 -- 172.21.15.16:0/3783294247 learned_addr learned my addr 172.21.15.16:0/3783294247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f179effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a183424fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a7fff640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a1834254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 -- 172.21.15.16:0/3783294247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 msgr2=0x55a183424fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a183424fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 -- 172.21.15.16:0/3783294247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 msgr2=0x55a1834254e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a1834254e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f17a77fe640 1 -- 172.21.15.16:0/3783294247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a183425d10 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.073+0000 7f179effd640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a183424fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17a77fe640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1798000c00 tx=0x7f1798009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145142a374974441 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179803e060 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f179800f540 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f17980053d0 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1798004c60 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 10 client.26093 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a18349b9e0 con 0x55a183412a30 2024-09-12T02:52:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179800f540 con 0x55a183412a30 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f179802f880 con 0x55a183412a30 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 10 client.26093 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.074+0000 7f17b041f740 20 client.26093 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183421680 0x55a18349dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 4 client.26093 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a183383d60 con 0x55a183421680 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f179effd640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183421680 0x55a18349dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f179effd640 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183421680 0x55a18349dec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4dfd7ffb43feb42 server_cookie=4e54ed829a17c62b in_seq=0 out_seq=0 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f1786ffd640 10 client.26093 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f1786ffd640 1 -- 172.21.15.16:0/3783294247 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a183383d60 con 0x55a183421680 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f1786ffd640 10 client.26093 handle_command_reply: tid=0 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 1 client.26093 shutdown 2024-09-12T02:52:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 msgr2=0x55a183429c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.075+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1798000c00 tx=0x7f1798009530 comp rx=0 tx=0).stop 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 shutdown_connections 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a183421680 0x55a18349dec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a183416dc0 0x55a1834254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1834164a0 0x55a183424fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 --2- 172.21.15.16:0/3783294247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a183412a30 0x55a183429c70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f1786ffd640 0 client.26093 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 >> 172.21.15.16:0/3783294247 conn(0x55a18341e880 msgr2=0x55a183420260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 shutdown_connections 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 1 -- 172.21.15.16:0/3783294247 wait complete. 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17b041f740 20 client.26093 trim_cache size 0 max 16384 2024-09-12T02:52:38.081 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 msgr2=0x7f17940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 0x7f17940a5040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009fd0 tx=0x7f17a0004ab0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 msgr2=0x7f17a810eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1790000c00 tx=0x7f1790033d10 comp rx=0 tx=0).stop 2024-09-12T02:52:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 shutdown_connections 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17a8106a60 0x7f17a810f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17a8101400 0x7f17a810eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17940a2b80 0x7f17940a5040 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 --2- 172.21.15.16:0/3632423396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17a8100a60 0x7f17a810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 >> 172.21.15.16:0/3632423396 conn(0x7f17a80fc780 msgr2=0x7f17a80fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 shutdown_connections 2024-09-12T02:52:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:38.076+0000 7f17ae046640 1 -- 172.21.15.16:0/3632423396 wait complete. 2024-09-12T02:52:38.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:37 smithi033 ceph-mon[35092]: pgmap v478: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-09-12T02:52:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:37 smithi184 ceph-mon[35321]: pgmap v478: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-09-12T02:52:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:37 smithi016 ceph-mon[30925]: pgmap v478: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-09-12T02:52:38.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391438 9.15 MB/sec execute 100 sec latency 15.079 ms 2024-09-12T02:52:38.947 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:52:38.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:52:39.218 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 -- 172.21.15.16:0/3794221391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4100e20 msgr2=0x7f8ed4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 --2- 172.21.15.16:0/3794221391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4100e20 0x7f8ed4101200 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009900 tx=0x7f8ec402a480 comp rx=0 tx=0).stop 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 -- 172.21.15.16:0/3794221391 shutdown_connections 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 --2- 172.21.15.16:0/3794221391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 0x7f8ed410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 --2- 172.21.15.16:0/3794221391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4102570 0x7f8ed41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 --2- 172.21.15.16:0/3794221391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4100e20 0x7f8ed4101200 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 -- 172.21.15.16:0/3794221391 >> 172.21.15.16:0/3794221391 conn(0x7f8ed40fc7d0 msgr2=0x7f8ed40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.663+0000 7f8eda47b640 1 -- 172.21.15.16:0/3794221391 shutdown_connections 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.664+0000 7f8eda47b640 1 -- 172.21.15.16:0/3794221391 wait complete. 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.664+0000 7f8eda47b640 1 Processor -- start 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.664+0000 7f8eda47b640 1 -- start start 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.664+0000 7f8eda47b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 0x7f8ed419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.664+0000 7f8eda47b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8eda47b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 0x7f8ed419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8eda47b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4110f80 con 0x7f8ed4102570 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8eda47b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4110e00 con 0x7f8ed4100e20 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8eda47b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4111100 con 0x7f8ed4102ef0 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 0x7f8ed419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50164/0 (socket says 172.21.15.16:50164) 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 -- 172.21.15.16:0/1151388369 learned_addr learned my addr 172.21.15.16:0/1151388369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed8a18640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 0x7f8ed419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 -- 172.21.15.16:0/1151388369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 msgr2=0x7f8ed419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 0x7f8ed419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 -- 172.21.15.16:0/1151388369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 msgr2=0x7f8ed419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 0x7f8ed419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 -- 172.21.15.16:0/1151388369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed419f660 con 0x7f8ed4102570 2024-09-12T02:52:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed3fff640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 0x7f8ed419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed37fe640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800ea60 tx=0x7f8ec800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800cde0 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed419f950 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec80035f0 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.665+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8010670 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.666+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed41a7230 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.666+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed4101200 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.668+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8ec80037a0 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.668+0000 7f8ed17fa640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 0x7f8ebc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.669+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8ec80cf3a0 con 0x7f8ed4102570 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.669+0000 7f8ed3fff640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 0x7f8ebc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.669+0000 7f8ed3fff640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 0x7f8ebc0a5040 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009230 tx=0x7f8ec40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.672+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8ec809c3a0 con 0x7f8ed4102570 2024-09-12T02:52:39.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 394349 9.23 MB/sec execute 101 sec latency 10.895 ms 2024-09-12T02:52:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.898+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ed419ba10 con 0x7f8ed4102570 2024-09-12T02:52:39.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.898+0000 7f8ed17fa640 1 -- 172.21.15.16:0/1151388369 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f8ed419ba10 con 0x7f8ed4102570 2024-09-12T02:52:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:52:39.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:52:39.899 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:52:39.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 msgr2=0x7f8ebc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 0x7f8ebc0a5040 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009230 tx=0x7f8ec40058d0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 msgr2=0x7f8ed419abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800ea60 tx=0x7f8ec800ef20 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 shutdown_connections 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ed4102ef0 0x7f8ed419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8ebc0a2b80 0x7f8ebc0a5040 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ed4102570 0x7f8ed419abf0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 --2- 172.21.15.16:0/1151388369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ed4100e20 0x7f8ed419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 >> 172.21.15.16:0/1151388369 conn(0x7f8ed40fc7d0 msgr2=0x7f8ed406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 shutdown_connections 2024-09-12T02:52:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:39.901+0000 7f8eda47b640 1 -- 172.21.15.16:0/1151388369 wait complete. 2024-09-12T02:52:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:39 smithi033 ceph-mon[35092]: pgmap v479: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-09-12T02:52:40.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:39 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1151388369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:40.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:39 smithi016 ceph-mon[30925]: pgmap v479: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-09-12T02:52:40.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:39 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1151388369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:39 smithi184 ceph-mon[35321]: pgmap v479: 97 pgs: 97 active+clean; 155 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-09-12T02:52:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:39 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1151388369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:52:40.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:52:40.717 INFO:tasks.workunit.client.0.smithi016.stdout: 1 397428 9.31 MB/sec execute 102 sec latency 11.687 ms 2024-09-12T02:52:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 -- 172.21.15.16:0/2437469193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 msgr2=0x7fea8410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 --2- 172.21.15.16:0/2437469193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea8410ec40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009930 tx=0x7fea7c02a420 comp rx=0 tx=0).stop 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 -- 172.21.15.16:0/2437469193 shutdown_connections 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 --2- 172.21.15.16:0/2437469193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea8410ec40 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 --2- 172.21.15.16:0/2437469193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea841036e0 0x7fea84107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 --2- 172.21.15.16:0/2437469193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea84102d30 0x7fea84103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 -- 172.21.15.16:0/2437469193 >> 172.21.15.16:0/2437469193 conn(0x7fea840fc7d0 msgr2=0x7fea840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 -- 172.21.15.16:0/2437469193 shutdown_connections 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.480+0000 7fea8908d640 1 -- 172.21.15.16:0/2437469193 wait complete. 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 Processor -- start 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 -- start start 2024-09-12T02:52:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 0x7fea84100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea841036e0 0x7fea84100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea84101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea841120a0 con 0x7fea841082a0 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea84111f20 con 0x7fea84102d30 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8908d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea84112220 con 0x7fea841036e0 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea84101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 0x7fea84100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea84101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50182/0 (socket says 172.21.15.16:50182) 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8359e640 1 -- 172.21.15.16:0/1873443384 learned_addr learned my addr 172.21.15.16:0/1873443384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.481+0000 7fea8259c640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea841036e0 0x7fea84100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 -- 172.21.15.16:0/1873443384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea841036e0 msgr2=0x7fea84100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea841036e0 0x7fea84100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 -- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 msgr2=0x7fea84101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea84101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 -- 172.21.15.16:0/1873443384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea841a6a10 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 0x7fea84100990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fea7400eb00 tx=0x7fea7400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea7bfff640 1 -- 172.21.15.16:0/1873443384 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74010430 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea841a6d00 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea7bfff640 1 -- 172.21.15.16:0/1873443384 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea74010ab0 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea7bfff640 1 -- 172.21.15.16:0/1873443384 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74017bb0 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.482+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea841a71e0 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.484+0000 7fea7bfff640 1 -- 172.21.15.16:0/1873443384 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fea74010c60 con 0x7fea84102d30 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 Processor -- start 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 -- start start 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882c4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55968822a2f0 con 0x5596882c5b40 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5596881d1cf0 con 0x5596882cf370 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8b466740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596882262b0 con 0x5596882c45d0 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57174/0 (socket says 172.21.15.16:57174) 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea8259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52608/0 (socket says 172.21.15.16:52608) 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.485+0000 7fea82d9d640 1 -- 172.21.15.16:0/271676034 learned_addr learned my addr 172.21.15.16:0/271676034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 -- 172.21.15.16:0/271676034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 msgr2=0x5596882c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 -- 172.21.15.16:0/271676034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 msgr2=0x5596882c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 -- 172.21.15.16:0/271676034 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596881b63a0 con 0x5596882cf370 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea82d9d640 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea7bfff640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 0x7fea700a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8359e640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 0x7fea700a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea8259c640 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea8406a370 tx=0x7fea6c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd1c38f85b5314a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea7bfff640 1 -- 172.21.15.16:0/1873443384 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea740cf9e0 con 0x7fea84102d30 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea797fa640 1 -- 172.21.15.16:0/271676034 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c004050 con 0x5596882cf370 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea797fa640 1 -- 172.21.15.16:0/271676034 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea6c005500 con 0x5596882cf370 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.486+0000 7fea797fa640 1 -- 172.21.15.16:0/271676034 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c004d40 con 0x5596882cf370 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 -- 172.21.15.16:0/271676034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 msgr2=0x5596882d1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8359e640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 0x7fea700a5040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0096c0 tx=0x7fea7c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea8406a370 tx=0x7fea6c011810 comp rx=0 tx=0).stop 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 -- 172.21.15.16:0/271676034 shutdown_connections 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d1760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 --2- 172.21.15.16:0/271676034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 -- 172.21.15.16:0/271676034 >> 172.21.15.16:0/271676034 conn(0x55968828d310 msgr2=0x5596882c0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 -- 172.21.15.16:0/271676034 shutdown_connections 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.487+0000 7fea8b466740 1 -- 172.21.15.16:0/271676034 wait complete. 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 -- start start 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 Processor -- start 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 0x5596882ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5596882d2070 con 0x5596882cf370 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5596881e5060 con 0x5596882c45d0 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596882d21f0 con 0x5596882c5b40 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8b466740 10 client.0 _ll_register_callbacks cb 0x559687f388c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 0x5596882ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50200/0 (socket says 172.21.15.16:50200) 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.488+0000 7fea8359e640 1 -- 172.21.15.16:0/3537304112 learned_addr learned my addr 172.21.15.16:0/3537304112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 -- 172.21.15.16:0/3537304112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 msgr2=0x5596882cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 -- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 msgr2=0x5596882cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 -- 172.21.15.16:0/3537304112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559687f9b3c0 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8359e640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea82d9d640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 0x5596882ced20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea740a4990 tx=0x7fea74005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91db0eb8c1dd4da2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740cfe60 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea740988c0 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea6c005500 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea7409bd90 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 10 client.26111 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5596882ddbd0 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea6c005500 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea740bc0b0 con 0x5596882c45d0 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 10 client.26111 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 10 client.26111 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.489+0000 7fea8b466740 20 client.26111 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5596882df090 0x5596882e1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea8b466740 4 client.26111 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559688237860 con 0x5596882df090 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea8259c640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5596882df090 0x5596882e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea8259c640 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5596882df090 0x5596882e1480 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c6ef68bd83d09bd3 server_cookie=79742ac78d7a38fe in_seq=0 out_seq=0 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.490+0000 7fea6a7fc640 10 client.26111 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea6a7fc640 1 -- 172.21.15.16:0/3537304112 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559688237860 con 0x5596882df090 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea6a7fc640 10 client.26111 handle_command_reply: tid=0 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 client.26111 shutdown 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 msgr2=0x5596882ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 0x5596882ced20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea740a4990 tx=0x7fea74005f10 comp rx=0 tx=0).stop 2024-09-12T02:52:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 shutdown_connections 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882cf370 0x5596882cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5596882df090 0x5596882e1480 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 --2- 172.21.15.16:0/3537304112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882c45d0 0x5596882ced20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 >> 172.21.15.16:0/3537304112 conn(0x55968828d310 msgr2=0x5596882c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea6a7fc640 0 client.26111 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 shutdown_connections 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 1 -- 172.21.15.16:0/3537304112 wait complete. 2024-09-12T02:52:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.491+0000 7fea8b466740 20 client.26111 trim_cache size 0 max 16384 2024-09-12T02:52:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.506+0000 7fea8b466740 1 Processor -- start 2024-09-12T02:52:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.506+0000 7fea8b466740 1 -- start start 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea841af7b0 con 0x5596882c5b40 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea841b12f0 con 0x5596882cf370 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8b466740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea841afe80 con 0x5596882c45d0 2024-09-12T02:52:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50202/0 (socket says 172.21.15.16:50202) 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 -- 172.21.15.16:0/2490846415 learned_addr learned my addr 172.21.15.16:0/2490846415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea82d9d640 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 -- 172.21.15.16:0/2490846415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 msgr2=0x5596882c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 -- 172.21.15.16:0/2490846415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 msgr2=0x5596882e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8259c640 1 -- 172.21.15.16:0/2490846415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596882e1a60 con 0x5596882c5b40 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.507+0000 7fea8359e640 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.508+0000 7fea8259c640 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009230 tx=0x7fea6c004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305d7f1cd3134a4b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.508+0000 7fea78ff9640 1 -- 172.21.15.16:0/2490846415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c030070 con 0x5596882c5b40 2024-09-12T02:52:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.508+0000 7fea78ff9640 1 -- 172.21.15.16:0/2490846415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea6c004050 con 0x5596882c5b40 2024-09-12T02:52:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.508+0000 7fea78ff9640 1 -- 172.21.15.16:0/2490846415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c01d040 con 0x5596882c5b40 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- 172.21.15.16:0/2490846415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 msgr2=0x5596882db370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009230 tx=0x7fea6c004810 comp rx=0 tx=0).stop 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- 172.21.15.16:0/2490846415 shutdown_connections 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c5b40 0x5596882db370 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 --2- 172.21.15.16:0/2490846415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c45d0 0x5596882c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- 172.21.15.16:0/2490846415 >> 172.21.15.16:0/2490846415 conn(0x55968828d310 msgr2=0x5596882c0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- 172.21.15.16:0/2490846415 shutdown_connections 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- 172.21.15.16:0/2490846415 wait complete. 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 -- start start 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.509+0000 7fea8b466740 1 Processor -- start 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea841b0c20 con 0x5596882c45d0 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea841b0550 con 0x5596882cf370 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea841ad790 con 0x5596882c5b40 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8b466740 10 client.0 _ll_register_callbacks cb 0x7fea740173c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50206/0 (socket says 172.21.15.16:50206) 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 -- 172.21.15.16:0/1406990901 learned_addr learned my addr 172.21.15.16:0/1406990901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8259c640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea8359e640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 -- 172.21.15.16:0/1406990901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 msgr2=0x5596882d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882d2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 -- 172.21.15.16:0/1406990901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 msgr2=0x5596882d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 -- 172.21.15.16:0/1406990901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596882d3d80 con 0x5596882c45d0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea82d9d640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fea7409c1e0 tx=0x7fea7400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570b4072eb3cda20 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740b4070 con 0x5596882c45d0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.510+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea740a0310 con 0x5596882c45d0 2024-09-12T02:52:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea6c004050 con 0x5596882c45d0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea7409bbb0 con 0x5596882c45d0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 10 client.16569 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559688352500 con 0x5596882c45d0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea6c004050 con 0x5596882c45d0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea7409b6a0 con 0x5596882c45d0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 10 client.16569 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 10 client.16569 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 20 client.16569 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559688352c10 0x559688355000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 4 client.16569 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559688231d70 con 0x559688352c10 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.511+0000 7fea8259c640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559688352c10 0x559688355000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea6a7fc640 10 client.16569 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:52:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8259c640 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559688352c10 0x559688355000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3495e649e82d28a1 server_cookie=b2636563eda03f14 in_seq=0 out_seq=0 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea6a7fc640 1 -- 172.21.15.16:0/1406990901 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559688231d70 con 0x559688352c10 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea6a7fc640 10 client.16569 handle_command_reply: tid=0 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 client.16569 shutdown 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 msgr2=0x5596882d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fea7409c1e0 tx=0x7fea7400cf10 comp rx=0 tx=0).stop 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 shutdown_connections 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559688352c10 0x559688355000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596882cf370 0x5596882d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596882c5b40 0x5596882d2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 --2- 172.21.15.16:0/1406990901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596882c45d0 0x5596882d7cc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.512+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 >> 172.21.15.16:0/1406990901 conn(0x55968828d310 msgr2=0x5596882c6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea6a7fc640 0 client.16569 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:52:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 shutdown_connections 2024-09-12T02:52:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8b466740 1 -- 172.21.15.16:0/1406990901 wait complete. 2024-09-12T02:52:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8b466740 20 client.16569 trim_cache size 0 max 16384 2024-09-12T02:52:41.518 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 msgr2=0x7fea700a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 0x7fea700a5040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0096c0 tx=0x7fea7c005a20 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 msgr2=0x7fea84100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 0x7fea84100990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fea7400eb00 tx=0x7fea7400c2d0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 shutdown_connections 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea700a2b80 0x7fea700a5040 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea841082a0 0x7fea84101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea841036e0 0x7fea84100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 --2- 172.21.15.16:0/1873443384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea84102d30 0x7fea84100990 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fea7400eb00 tx=0x7fea7400c2d0 comp rx=0 tx=0).stop 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.513+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 >> 172.21.15.16:0/1873443384 conn(0x7fea840fc7d0 msgr2=0x7fea84105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:41.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.514+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 shutdown_connections 2024-09-12T02:52:41.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:41.514+0000 7fea8908d640 1 -- 172.21.15.16:0/1873443384 wait complete. 2024-09-12T02:52:41.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399764 9.30 MB/sec execute 103 sec latency 18.946 ms 2024-09-12T02:52:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:41 smithi033 ceph-mon[35092]: pgmap v480: 97 pgs: 97 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-09-12T02:52:42.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:41 smithi016 ceph-mon[30925]: pgmap v480: 97 pgs: 97 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-09-12T02:52:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:41 smithi184 ceph-mon[35321]: pgmap v480: 97 pgs: 97 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-09-12T02:52:42.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:52:42.668 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:42.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402346 9.38 MB/sec execute 104 sec latency 11.424 ms 2024-09-12T02:52:43.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.142+0000 7f477851f640 1 -- 172.21.15.16:0/387324157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770102790 msgr2=0x7f47701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.142+0000 7f477851f640 1 --2- 172.21.15.16:0/387324157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770102790 0x7f47701067e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4764009900 tx=0x7f476402a410 comp rx=0 tx=0).stop 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 -- 172.21.15.16:0/387324157 shutdown_connections 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 --2- 172.21.15.16:0/387324157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 0x7f477010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 --2- 172.21.15.16:0/387324157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770102790 0x7f47701067e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 --2- 172.21.15.16:0/387324157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770101de0 0x7f47701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 -- 172.21.15.16:0/387324157 >> 172.21.15.16:0/387324157 conn(0x7f47700fd8c0 msgr2=0x7f47700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 -- 172.21.15.16:0/387324157 shutdown_connections 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 -- 172.21.15.16:0/387324157 wait complete. 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 Processor -- start 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.143+0000 7f477851f640 1 -- start start 2024-09-12T02:52:43.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770102790 0x7f4770198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 0x7f477019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4770111f70 con 0x7f4770101de0 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4770111df0 con 0x7f4770106f10 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47701120f0 con 0x7f4770102790 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50212/0 (socket says 172.21.15.16:50212) 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 -- 172.21.15.16:0/3577445821 learned_addr learned my addr 172.21.15.16:0/3577445821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f4775aba640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770102790 0x7f4770198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f4776abc640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 0x7f477019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 -- 172.21.15.16:0/3577445821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770102790 msgr2=0x7f4770198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770102790 0x7f4770198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 -- 172.21.15.16:0/3577445821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 msgr2=0x7f477019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 0x7f477019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 -- 172.21.15.16:0/3577445821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477019d400 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f47762bb640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f476000eb90 tx=0x7f476000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f476f7fe640 1 -- 172.21.15.16:0/3577445821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760004d20 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477019d6f0 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f476f7fe640 1 -- 172.21.15.16:0/3577445821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476000ee90 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.144+0000 7f476f7fe640 1 -- 172.21.15.16:0/3577445821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760010830 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.145+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47701a4fd0 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f476f7fe640 1 -- 172.21.15.16:0/3577445821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4760005020 con 0x7f4770101de0 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 Processor -- start 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 -- start start 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b2154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b2165b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a4b20ba2f0 con 0x55a4b2155b40 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4b2061cf0 con 0x55a4b21545d0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.147+0000 7f477a8f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a4b20b62b0 con 0x55a4b2163780 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f476f7fe640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 0x7f47580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b2165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 -- 172.21.15.16:0/1659859296 learned_addr learned my addr 172.21.15.16:0/1659859296 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f47762bb640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b2154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 0x7f47580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f476f7fe640 1 -- 172.21.15.16:0/3577445821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47600cf660 con 0x7f4770101de0 2024-09-12T02:52:43.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 -- 172.21.15.16:0/1659859296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 msgr2=0x55a4b2154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b2154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 -- 172.21.15.16:0/1659859296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 msgr2=0x55a4b2154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 -- 172.21.15.16:0/1659859296 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4b20463a0 con 0x55a4b2163780 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f47762bb640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b2154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4775aba640 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b2165b70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4764005940 tx=0x7f4764005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1775887becd1418f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.148+0000 7f4776abc640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 0x7f47580a5040 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f47680058c0 tx=0x7f4768004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f476cff9640 1 -- 172.21.15.16:0/1659859296 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764040070 con 0x55a4b2163780 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f476cff9640 1 -- 172.21.15.16:0/1659859296 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4764031870 con 0x55a4b2163780 2024-09-12T02:52:43.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f476cff9640 1 -- 172.21.15.16:0/1659859296 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764035040 con 0x55a4b2163780 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 -- 172.21.15.16:0/1659859296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 msgr2=0x55a4b2165b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b2165b70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4764005940 tx=0x7f4764005be0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 -- 172.21.15.16:0/1659859296 shutdown_connections 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b2165b70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b2154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 --2- 172.21.15.16:0/1659859296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.149+0000 7f477a8f8740 1 -- 172.21.15.16:0/1659859296 >> 172.21.15.16:0/1659859296 conn(0x55a4b211d310 msgr2=0x55a4b2150c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- 172.21.15.16:0/1659859296 shutdown_connections 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- 172.21.15.16:0/1659859296 wait complete. 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- start start 2024-09-12T02:52:43.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 Processor -- start 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21630d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a4b2166420 con 0x55a4b2155b40 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4b2075060 con 0x55a4b21545d0 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a4b21665a0 con 0x55a4b2163780 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f477a8f8740 10 client.0 _ll_register_callbacks cb 0x55a4b1dc88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f4775aba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f4775aba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50240/0 (socket says 172.21.15.16:50240) 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.150+0000 7f4775aba640 1 -- 172.21.15.16:0/3380196211 learned_addr learned my addr 172.21.15.16:0/3380196211 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f47762bb640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4776abc640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 -- 172.21.15.16:0/3380196211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 msgr2=0x55a4b21608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 -- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 msgr2=0x55a4b21630d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21630d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 -- 172.21.15.16:0/3380196211 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4b1e2b3c0 con 0x55a4b2155b40 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f47762bb640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4776abc640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4775aba640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f476402a930 tx=0x7f4764005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01a9db426680792 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764040030 con 0x55a4b2155b40 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4764031670 con 0x55a4b2155b40 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4764031870 con 0x55a4b2155b40 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4764035070 con 0x55a4b2155b40 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f477a8f8740 10 client.16581 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4b2171da0 con 0x55a4b2155b40 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.151+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4764031870 con 0x55a4b2155b40 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f47640035f0 con 0x55a4b2155b40 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 10 client.16581 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 10 client.16581 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 20 client.16581 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21730f0 0x55a4b21754e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 4 client.16581 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4b20c7860 con 0x55a4b21730f0 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f47762bb640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21730f0 0x55a4b21754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f4755ffb640 10 client.16581 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.152+0000 7f47762bb640 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21730f0 0x55a4b21754e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c94fefe318dfdf9a server_cookie=60c561f8fa34b76 in_seq=0 out_seq=0 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f4755ffb640 1 -- 172.21.15.16:0/3380196211 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4b20c7860 con 0x55a4b21730f0 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f4755ffb640 10 client.16581 handle_command_reply: tid=0 2024-09-12T02:52:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 client.16581 shutdown 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 msgr2=0x55a4b2160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f476402a930 tx=0x7f4764005250 comp rx=0 tx=0).stop 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 shutdown_connections 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21730f0 0x55a4b21754e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f4755ffb640 0 client.16581 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2155b40 0x55a4b2160380 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3380196211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 >> 172.21.15.16:0/3380196211 conn(0x55a4b211d310 msgr2=0x55a4b2156960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 shutdown_connections 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 1 -- 172.21.15.16:0/3380196211 wait complete. 2024-09-12T02:52:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.153+0000 7f477a8f8740 20 client.16581 trim_cache size 0 max 16384 2024-09-12T02:52:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 Processor -- start 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 -- start start 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 0x55a4b215d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 0x55a4b215db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47701ab2f0 con 0x55a4b2163780 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47701ace30 con 0x55a4b21545d0 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f477a8f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47701ab9c0 con 0x55a4b2155b40 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 0x55a4b215db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4775aba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 0x55a4b215d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4775aba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 0x55a4b215d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57254/0 (socket says 172.21.15.16:57254) 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4775aba640 1 -- 172.21.15.16:0/3878029620 learned_addr learned my addr 172.21.15.16:0/3878029620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f47762bb640 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 -- 172.21.15.16:0/3878029620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 msgr2=0x55a4b215d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 0x55a4b215d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 -- 172.21.15.16:0/3878029620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 msgr2=0x55a4b21622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 -- 172.21.15.16:0/3878029620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4b2162cf0 con 0x55a4b2163780 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.167+0000 7f4776abc640 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 0x55a4b215db20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4768012040 tx=0x7f4768009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ea746eb95be540 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f4757fff640 1 -- 172.21.15.16:0/3878029620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768010980 con 0x55a4b2163780 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f4757fff640 1 -- 172.21.15.16:0/3878029620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4768026070 con 0x55a4b2163780 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f4757fff640 1 -- 172.21.15.16:0/3878029620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476802bd70 con 0x55a4b2163780 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 -- 172.21.15.16:0/3878029620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 msgr2=0x55a4b215db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 0x55a4b215db20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4768012040 tx=0x7f4768009830 comp rx=0 tx=0).stop 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 -- 172.21.15.16:0/3878029620 shutdown_connections 2024-09-12T02:52:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b2163780 0x55a4b215db20 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2155b40 0x55a4b215d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3878029620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b21545d0 0x55a4b21622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 -- 172.21.15.16:0/3878029620 >> 172.21.15.16:0/3878029620 conn(0x55a4b211d310 msgr2=0x55a4b2161be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 -- 172.21.15.16:0/3878029620 shutdown_connections 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.168+0000 7f477a8f8740 1 -- 172.21.15.16:0/3878029620 wait complete. 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 -- start start 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 Processor -- start 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b2155b40 0x55a4b2176050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21765d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47701ac760 con 0x55a4b21545d0 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47701ac090 con 0x55a4b2155b40 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47701a92d0 con 0x55a4b2163780 2024-09-12T02:52:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f477a8f8740 10 client.0 _ll_register_callbacks cb 0x7f4768009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50246/0 (socket says 172.21.15.16:50246) 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 -- 172.21.15.16:0/3129183668 learned_addr learned my addr 172.21.15.16:0/3129183668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f4775aba640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b2155b40 0x55a4b2176050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f4776abc640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21765d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 -- 172.21.15.16:0/3129183668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 msgr2=0x55a4b21765d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21765d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 -- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b2155b40 msgr2=0x55a4b2176050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.169+0000 7f47762bb640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b2155b40 0x55a4b2176050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f47762bb640 1 -- 172.21.15.16:0/3129183668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4b2176e00 con 0x55a4b21545d0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f47762bb640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f47600049d0 tx=0x7f476006ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eef8efa676be212 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600a0280 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4760005310 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4768026070 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f47600a4b40 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 10 client.16587 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4b21ddb30 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4768026070 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f47600a46b0 con 0x55a4b21545d0 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 10 client.16587 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 10 client.16587 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 20 client.16587 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21de200 0x55a4b21e05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 4 client.16587 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.170+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4b20c1d70 con 0x55a4b21de200 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f4775aba640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21de200 0x55a4b21e05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f4775aba640 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21de200 0x55a4b21e05f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=90619d9bafe1047 server_cookie=21174243f7a25740 in_seq=0 out_seq=0 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f4755ffb640 10 client.16587 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f4755ffb640 1 -- 172.21.15.16:0/3129183668 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4b20c1d70 con 0x55a4b21de200 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f4755ffb640 10 client.16587 handle_command_reply: tid=0 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 client.16587 shutdown 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 msgr2=0x55a4b2178d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f47600049d0 tx=0x7f476006ae50 comp rx=0 tx=0).stop 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 shutdown_connections 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55a4b21de200 0x55a4b21e05f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a4b2163780 0x55a4b21765d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4b2155b40 0x55a4b2176050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 --2- 172.21.15.16:0/3129183668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4b21545d0 0x55a4b2178d00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.171+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 >> 172.21.15.16:0/3129183668 conn(0x55a4b211d310 msgr2=0x55a4b2156110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f4755ffb640 0 client.16587 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:52:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 shutdown_connections 2024-09-12T02:52:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477a8f8740 1 -- 172.21.15.16:0/3129183668 wait complete. 2024-09-12T02:52:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477a8f8740 20 client.16587 trim_cache size 0 max 16384 2024-09-12T02:52:43.183 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 msgr2=0x7f47580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 0x7f47580a5040 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f47680058c0 tx=0x7f4768004730 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 msgr2=0x7f4770198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f476000eb90 tx=0x7f476000c6a0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 shutdown_connections 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4770106f10 0x7f477019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4770102790 0x7f4770198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47580a2b80 0x7f47580a5040 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 --2- 172.21.15.16:0/3577445821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4770101de0 0x7f4770198450 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 >> 172.21.15.16:0/3577445821 conn(0x7f47700fd8c0 msgr2=0x7f47700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 shutdown_connections 2024-09-12T02:52:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:43.172+0000 7f477851f640 1 -- 172.21.15.16:0/3577445821 wait complete. 2024-09-12T02:52:43.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 405005 9.42 MB/sec execute 105 sec latency 63.098 ms 2024-09-12T02:52:44.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:52:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:43 smithi033 ceph-mon[35092]: pgmap v481: 97 pgs: 97 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-09-12T02:52:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:43 smithi184 ceph-mon[35321]: pgmap v481: 97 pgs: 97 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-09-12T02:52:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:44.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:43 smithi016 ceph-mon[30925]: pgmap v481: 97 pgs: 97 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 281 op/s 2024-09-12T02:52:44.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 408190 9.51 MB/sec execute 106 sec latency 11.770 ms 2024-09-12T02:52:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 -- 172.21.15.16:0/2876821638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc100600 msgr2=0x7f3ecc10e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2876821638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc100600 0x7f3ecc10e900 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009cd0 tx=0x7f3ec402a8d0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 -- 172.21.15.16:0/2876821638 shutdown_connections 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2876821638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc100600 0x7f3ecc10e900 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2876821638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2876821638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 0x7f3ecc107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 -- 172.21.15.16:0/2876821638 >> 172.21.15.16:0/2876821638 conn(0x7f3ecc0fba90 msgr2=0x7f3ecc0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 -- 172.21.15.16:0/2876821638 shutdown_connections 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.775+0000 7f3ed2983640 1 -- 172.21.15.16:0/2876821638 wait complete. 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 Processor -- start 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 -- start start 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 0x7f3ecc198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 0x7f3ecc19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc111f70 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc111df0 con 0x7f3ecc106c80 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed2983640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1120f0 con 0x7f3ecc100600 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50258/0 (socket says 172.21.15.16:50258) 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ecbfff640 1 -- 172.21.15.16:0/2935555828 learned_addr learned my addr 172.21.15.16:0/2935555828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 0x7f3ecc198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.776+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 0x7f3ecc19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 -- 172.21.15.16:0/2935555828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 msgr2=0x7f3ecc198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 0x7f3ecc198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 -- 172.21.15.16:0/2935555828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 msgr2=0x7f3ecc19ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 0x7f3ecc19ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 -- 172.21.15.16:0/2935555828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc19d3a0 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 0x7f3ecc198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ecbfff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00d950 tx=0x7f3ebc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ec9ffb640 1 -- 172.21.15.16:0/2935555828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc017030 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ec9ffb640 1 -- 172.21.15.16:0/2935555828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ebc004020 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ec9ffb640 1 -- 172.21.15.16:0/2935555828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc0047a0 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc19d690 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.777+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1a4f70 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 Processor -- start 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ec9ffb640 1 -- 172.21.15.16:0/2935555828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3ebc0041d0 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 -- start start 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d5bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563772cb02f0 con 0x563772d4a5d0 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563772c57cf0 con 0x563772d4bb40 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563772cac2b0 con 0x563772d59780 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d4aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52688/0 (socket says 172.21.15.16:52688) 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d4aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50264/0 (socket says 172.21.15.16:50264) 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 -- 172.21.15.16:0/3886855567 learned_addr learned my addr 172.21.15.16:0/3886855567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d5bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 -- 172.21.15.16:0/3886855567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 msgr2=0x563772d5bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d5bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 -- 172.21.15.16:0/3886855567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 msgr2=0x563772d4aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d4aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.780+0000 7f3ed0f20640 1 -- 172.21.15.16:0/3886855567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563772c3c3a0 con 0x563772d4bb40 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3ec402adf0 tx=0x7f3ec4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413a71e1c32c9930 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ec9ffb640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 0x7f3eb80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ecbfff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 0x7f3eb80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ec27fc640 1 -- 172.21.15.16:0/3886855567 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4031d60 con 0x563772d4bb40 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ec27fc640 1 -- 172.21.15.16:0/3886855567 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec4036030 con 0x563772d4bb40 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ec27fc640 1 -- 172.21.15.16:0/3886855567 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec40043b0 con 0x563772d4bb40 2024-09-12T02:52:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.781+0000 7f3ec9ffb640 1 -- 172.21.15.16:0/2935555828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ebc0ce960 con 0x7f3ecc0ffc40 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3886855567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 msgr2=0x563772d4a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ecbfff640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 0x7f3eb80a5040 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0cc7e0 tx=0x7f3ebc052040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3ec402adf0 tx=0x7f3ec4009920 comp rx=0 tx=0).stop 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3886855567 shutdown_connections 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d5bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d4aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3886855567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d4a090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3886855567 >> 172.21.15.16:0/3886855567 conn(0x563772d13310 msgr2=0x563772d46c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3886855567 shutdown_connections 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3886855567 wait complete. 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 -- start start 2024-09-12T02:52:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.782+0000 7f3ed4d5c740 1 Processor -- start 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d563e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d56980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563772d5c420 con 0x563772d4a5d0 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563772c6b060 con 0x563772d4bb40 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563772d5c5a0 con 0x563772d59780 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed4d5c740 10 client.0 _ll_register_callbacks cb 0x5637729be8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50274/0 (socket says 172.21.15.16:50274) 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2064439800 learned_addr learned my addr 172.21.15.16:0/2064439800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d56980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 msgr2=0x563772d56980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d56980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2064439800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 msgr2=0x563772d563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2064439800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563772a213c0 con 0x563772d4a5d0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4003c50 tx=0x7f3ec4003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d19ba29f3c7df8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.783+0000 7f3eab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4059060 con 0x563772d4a5d0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ec4031830 con 0x563772d4a5d0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec4036030 con 0x563772d4a5d0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3ec4004b40 con 0x563772d4a5d0 2024-09-12T02:52:44.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 10 client.16599 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563772d5c420 con 0x563772d4a5d0 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec4036030 con 0x563772d4a5d0 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3ec4043070 con 0x563772d4a5d0 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 10 client.16599 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 10 client.16599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 20 client.16599 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772d69020 0x563772d6b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 4 client.16599 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563772cbd860 con 0x563772d69020 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.784+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772d69020 0x563772d6b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.785+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772d69020 0x563772d6b410 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7e6eed954fd2133 server_cookie=f0ea3d4a8116a04a in_seq=0 out_seq=0 2024-09-12T02:52:44.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.785+0000 7f3eab7fe640 10 client.16599 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2064439800 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563772cbd860 con 0x563772d69020 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3eab7fe640 10 client.16599 handle_command_reply: tid=0 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 client.16599 shutdown 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 msgr2=0x563772d59130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4003c50 tx=0x7f3ec4003ef0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 shutdown_connections 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772d69020 0x563772d6b410 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d56980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2064439800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d59130 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3eab7fe640 0 client.16599 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:52:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 >> 172.21.15.16:0/2064439800 conn(0x563772d13310 msgr2=0x563772d4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 shutdown_connections 2024-09-12T02:52:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2064439800 wait complete. 2024-09-12T02:52:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.786+0000 7f3ed4d5c740 20 client.16599 trim_cache size 0 max 16384 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 Processor -- start 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 -- start start 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d46510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d46a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d70b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1ab330 con 0x563772d4a5d0 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1ace70 con 0x563772d4bb40 2024-09-12T02:52:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1aba00 con 0x563772d59780 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d46510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ec3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d46a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d46510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50278/0 (socket says 172.21.15.16:50278) 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.800+0000 7f3ed0f20640 1 -- 172.21.15.16:0/3645606955 learned_addr learned my addr 172.21.15.16:0/3645606955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 -- 172.21.15.16:0/3645606955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 msgr2=0x563772d70b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d70b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d70b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 -- 172.21.15.16:0/3645606955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 msgr2=0x563772d46510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d46510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 -- 172.21.15.16:0/3645606955 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563772d54930 con 0x563772d4bb40 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d70b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:52:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d46a50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x563772d57a20 tx=0x7f3eb40039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf48996e118308e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec1ffb640 1 -- 172.21.15.16:0/3645606955 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4022070 con 0x563772d4bb40 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec1ffb640 1 -- 172.21.15.16:0/3645606955 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb4011b20 con 0x563772d4bb40 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.801+0000 7f3ec1ffb640 1 -- 172.21.15.16:0/3645606955 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb401cc00 con 0x563772d4bb40 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3645606955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 msgr2=0x563772d46a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d46a50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x563772d57a20 tx=0x7f3eb40039c0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3645606955 shutdown_connections 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d59780 0x563772d70b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4bb40 0x563772d46a50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/3645606955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d4a5d0 0x563772d46510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3645606955 >> 172.21.15.16:0/3645606955 conn(0x563772d13310 msgr2=0x563772d4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3645606955 shutdown_connections 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/3645606955 wait complete. 2024-09-12T02:52:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 -- start start 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.802+0000 7f3ed4d5c740 1 Processor -- start 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4a5d0 0x563772d5d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 0x563772d5dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1ac7a0 con 0x563772d59780 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1ac0d0 con 0x563772d4a5d0 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1a9310 con 0x563772d4bb40 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed4d5c740 10 client.0 _ll_register_callbacks cb 0x7f3ebc0d1f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ec3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 0x563772d5dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50294/0 (socket says 172.21.15.16:50294) 2024-09-12T02:52:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2674692399 learned_addr learned my addr 172.21.15.16:0/2674692399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4a5d0 0x563772d5d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 msgr2=0x563772d5dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 0x563772d5dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2674692399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4a5d0 msgr2=0x563772d5d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4a5d0 0x563772d5d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 -- 172.21.15.16:0/2674692399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563772dd2b00 con 0x563772d59780 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 0x563772d5dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3ed0f20640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005e30 tx=0x7f3ec403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbbc0ff020c0bad server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.803+0000 7f3eab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4064050 con 0x563772d59780 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ec4003f60 con 0x563772d59780 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eb4011b20 con 0x563772d59780 2024-09-12T02:52:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3ec4059040 con 0x563772d59780 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 10 client.16605 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563772dd3d20 con 0x563772d59780 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb4011b20 con 0x563772d59780 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3ec4043070 con 0x563772d59780 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 10 client.16605 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 10 client.16605 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 20 client.16605 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772dd4380 0x563772dd67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 4 client.16605 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563772cb7d70 con 0x563772dd4380 2024-09-12T02:52:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.804+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772dd4380 0x563772dd67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.805+0000 7f3ec3fff640 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772dd4380 0x563772dd67a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d78cb1b54da2855 server_cookie=c571563c11e6fc5c in_seq=0 out_seq=0 2024-09-12T02:52:44.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.805+0000 7f3eab7fe640 10 client.16605 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3eab7fe640 1 -- 172.21.15.16:0/2674692399 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563772cb7d70 con 0x563772dd4380 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3eab7fe640 10 client.16605 handle_command_reply: tid=0 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 client.16605 shutdown 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 msgr2=0x563772d5e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005e30 tx=0x7f3ec403e040 comp rx=0 tx=0).stop 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 shutdown_connections 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563772d59780 0x563772d5e000 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x563772dd4380 0x563772dd67a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563772d4bb40 0x563772d5dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 --2- 172.21.15.16:0/2674692399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563772d4a5d0 0x563772d5d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3eab7fe640 0 client.16605 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 >> 172.21.15.16:0/2674692399 conn(0x563772d13310 msgr2=0x563772d57c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 shutdown_connections 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 1 -- 172.21.15.16:0/2674692399 wait complete. 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed4d5c740 20 client.16605 trim_cache size 0 max 16384 2024-09-12T02:52:44.813 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 msgr2=0x7f3eb80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 0x7f3eb80a5040 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0cc7e0 tx=0x7f3ebc052040 comp rx=0 tx=0).stop 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 msgr2=0x7f3ecc1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.807+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00d950 tx=0x7f3ebc00de10 comp rx=0 tx=0).stop 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 shutdown_connections 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ecc106c80 0x7f3ecc19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ecc100600 0x7f3ecc198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3eb80a2b80 0x7f3eb80a5040 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 --2- 172.21.15.16:0/2935555828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ecc0ffc40 0x7f3ecc1983f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 >> 172.21.15.16:0/2935555828 conn(0x7f3ecc0fba90 msgr2=0x7f3ecc0fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 shutdown_connections 2024-09-12T02:52:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:44.808+0000 7f3ed2983640 1 -- 172.21.15.16:0/2935555828 wait complete. 2024-09-12T02:52:45.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:52:45.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 411121 9.55 MB/sec execute 107 sec latency 12.969 ms 2024-09-12T02:52:45.869 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:45 smithi033 ceph-mon[35092]: pgmap v482: 97 pgs: 97 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 256 op/s 2024-09-12T02:52:46.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:45 smithi016 ceph-mon[30925]: pgmap v482: 97 pgs: 97 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 256 op/s 2024-09-12T02:52:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:45 smithi184 ceph-mon[35321]: pgmap v482: 97 pgs: 97 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 256 op/s 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 -- 172.21.15.16:0/1144476849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8344102df0 msgr2=0x7f83441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 --2- 172.21.15.16:0/1144476849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8344102df0 0x7f83441031d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8334009900 tx=0x7f833402a410 comp rx=0 tx=0).stop 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 -- 172.21.15.16:0/1144476849 shutdown_connections 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 --2- 172.21.15.16:0/1144476849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f834410c2f0 0x7f834410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 --2- 172.21.15.16:0/1144476849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83441037a0 0x7f834410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 --2- 172.21.15.16:0/1144476849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8344102df0 0x7f83441031d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 -- 172.21.15.16:0/1144476849 >> 172.21.15.16:0/1144476849 conn(0x7f83440fc7d0 msgr2=0x7f83440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 -- 172.21.15.16:0/1144476849 shutdown_connections 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 -- 172.21.15.16:0/1144476849 wait complete. 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.381+0000 7f8349ad6640 1 Processor -- start 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 -- start start 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 0x7f8344100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 0x7f83441014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8344112150 con 0x7f83441037a0 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8344111fd0 con 0x7f834410c2f0 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8349ad6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f83441122d0 con 0x7f8344102df0 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f83437fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 0x7f8344100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f83437fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 0x7f8344100a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43758/0 (socket says 172.21.15.16:43758) 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47102/0 (socket says 172.21.15.16:47102) 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f83437fe640 1 -- 172.21.15.16:0/3650227131 learned_addr learned my addr 172.21.15.16:0/3650227131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8343fff640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 0x7f83441014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 -- 172.21.15.16:0/3650227131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 msgr2=0x7f8344100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 0x7f8344100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 -- 172.21.15.16:0/3650227131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 msgr2=0x7f83441014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 0x7f83441014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.382+0000 7f8342ffd640 1 -- 172.21.15.16:0/3650227131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83441a6b30 con 0x7f83441037a0 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8343fff640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 0x7f83441014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f83440697d0 tx=0x7f833800b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8340ff9640 1 -- 172.21.15.16:0/3650227131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338015020 con 0x7f83441037a0 2024-09-12T02:52:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8340ff9640 1 -- 172.21.15.16:0/3650227131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338011d10 con 0x7f83441037a0 2024-09-12T02:52:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83441a6e20 con 0x7f83441037a0 2024-09-12T02:52:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8340ff9640 1 -- 172.21.15.16:0/3650227131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800f2e0 con 0x7f83441037a0 2024-09-12T02:52:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.383+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83441a7320 con 0x7f83441037a0 2024-09-12T02:52:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f8340ff9640 1 -- 172.21.15.16:0/3650227131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8338018020 con 0x7f83441037a0 2024-09-12T02:52:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 Processor -- start 2024-09-12T02:52:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 -- start start 2024-09-12T02:52:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1645a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd1652760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dcd15ab2f0 con 0x55dcd1650370 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dcd1552cf0 con 0x55dcd16455d0 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f834beaf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dcd15a72b0 con 0x55dcd1646b40 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f8343fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f8343fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43774/0 (socket says 172.21.15.16:43774) 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f8343fff640 1 -- 172.21.15.16:0/3773522676 learned_addr learned my addr 172.21.15.16:0/3773522676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.386+0000 7f8340ff9640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 0x7f832c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd1652760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f83437fe640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1645a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 0x7f832c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8340ff9640 1 -- 172.21.15.16:0/3650227131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f83380ce760 con 0x7f83441037a0 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 -- 172.21.15.16:0/3773522676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 msgr2=0x55dcd1645a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1645a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 -- 172.21.15.16:0/3773522676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 msgr2=0x55dcd1652760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd1652760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 -- 172.21.15.16:0/3773522676 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd15373a0 con 0x55dcd1646b40 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd1652760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f833c013b20 tx=0x7f833c011380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f397f6812d556f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8343fff640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 0x7f832c0a50c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f833c01b000 tx=0x7f833c01d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8329ffb640 1 -- 172.21.15.16:0/3773522676 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c01cbb0 con 0x55dcd1646b40 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8329ffb640 1 -- 172.21.15.16:0/3773522676 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c01cd50 con 0x55dcd1646b40 2024-09-12T02:52:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.387+0000 7f8329ffb640 1 -- 172.21.15.16:0/3773522676 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c00e5b0 con 0x55dcd1646b40 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 -- 172.21.15.16:0/3773522676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 msgr2=0x55dcd1645090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f833c013b20 tx=0x7f833c011380 comp rx=0 tx=0).stop 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 -- 172.21.15.16:0/3773522676 shutdown_connections 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd1652760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1645a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 --2- 172.21.15.16:0/3773522676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd1645090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 -- 172.21.15.16:0/3773522676 >> 172.21.15.16:0/3773522676 conn(0x55dcd160e310 msgr2=0x55dcd1641c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 -- 172.21.15.16:0/3773522676 shutdown_connections 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.388+0000 7f834beaf740 1 -- 172.21.15.16:0/3773522676 wait complete. 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 -- start start 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 Processor -- start 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd164fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dcd1652fb0 con 0x55dcd1650370 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dcd1566060 con 0x55dcd16455d0 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dcd1653130 con 0x55dcd1646b40 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.389+0000 7f834beaf740 10 client.0 _ll_register_callbacks cb 0x55dcd12b98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f83437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd164fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8343fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43780/0 (socket says 172.21.15.16:43780) 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 -- 172.21.15.16:0/686247351 learned_addr learned my addr 172.21.15.16:0/686247351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f83437fe640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd164fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36374/0 (socket says 172.21.15.16:36374) 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 -- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 msgr2=0x55dcd164fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd164fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 -- 172.21.15.16:0/686247351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 msgr2=0x55dcd164d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 -- 172.21.15.16:0/686247351 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd131c3c0 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8342ffd640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8338097430 tx=0x7f833809a9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333882e9b9544d7b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833809b6d0 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8338010510 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.390+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c01cd50 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f833809afd0 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 10 client.26065 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dcd165eaa0 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c01cd50 con 0x55dcd1646b40 2024-09-12T02:52:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f833809a420 con 0x55dcd1646b40 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 10 client.26065 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 10 client.26065 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 20 client.26065 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd165ff60 0x55dcd1662350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 4 client.26065 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcd15b8860 con 0x55dcd165ff60 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.391+0000 7f83437fe640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd165ff60 0x55dcd1662350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.392+0000 7f83437fe640 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd165ff60 0x55dcd1662350 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2321d9a67a29c4f7 server_cookie=a218d943125ec9d4 in_seq=0 out_seq=0 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.392+0000 7f8322ffd640 10 client.26065 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.392+0000 7f8322ffd640 1 -- 172.21.15.16:0/686247351 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dcd15b8860 con 0x55dcd165ff60 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.392+0000 7f8322ffd640 10 client.26065 handle_command_reply: tid=0 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.392+0000 7f834beaf740 1 client.26065 shutdown 2024-09-12T02:52:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 msgr2=0x55dcd164d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8338097430 tx=0x7f833809a9f0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 shutdown_connections 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd165ff60 0x55dcd1662350 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164d060 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 --2- 172.21.15.16:0/686247351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd164fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f8322ffd640 0 client.26065 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 >> 172.21.15.16:0/686247351 conn(0x55dcd160e310 msgr2=0x55dcd1647960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 shutdown_connections 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 1 -- 172.21.15.16:0/686247351 wait complete. 2024-09-12T02:52:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.393+0000 7f834beaf740 20 client.26065 trim_cache size 0 max 16384 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.406+0000 7f834beaf740 1 Processor -- start 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.406+0000 7f834beaf740 1 -- start start 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1646300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83441af7f0 con 0x55dcd1650370 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f83441b1330 con 0x55dcd16455d0 2024-09-12T02:52:46.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f834beaf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f83441afec0 con 0x55dcd1646b40 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47126/0 (socket says 172.21.15.16:47126) 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 -- 172.21.15.16:0/481489737 learned_addr learned my addr 172.21.15.16:0/481489737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f83437fe640 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1646300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 -- 172.21.15.16:0/481489737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 msgr2=0x55dcd164c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 -- 172.21.15.16:0/481489737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 msgr2=0x55dcd1646300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1646300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 -- 172.21.15.16:0/481489737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd1662980 con 0x55dcd1650370 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.407+0000 7f8343fff640 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f833c01de10 tx=0x7f833c005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1d6440d6d0107b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f83297fa640 1 -- 172.21.15.16:0/481489737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c01de40 con 0x55dcd1650370 2024-09-12T02:52:46.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f83297fa640 1 -- 172.21.15.16:0/481489737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c013520 con 0x55dcd1650370 2024-09-12T02:52:46.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f83297fa640 1 -- 172.21.15.16:0/481489737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c003cf0 con 0x55dcd1650370 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 -- 172.21.15.16:0/481489737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 msgr2=0x55dcd164c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f833c01de10 tx=0x7f833c005e30 comp rx=0 tx=0).stop 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 -- 172.21.15.16:0/481489737 shutdown_connections 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1650370 0x55dcd164c040 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1646b40 0x55dcd164c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 --2- 172.21.15.16:0/481489737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd1646300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.408+0000 7f834beaf740 1 -- 172.21.15.16:0/481489737 >> 172.21.15.16:0/481489737 conn(0x55dcd160e310 msgr2=0x55dcd164b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.409+0000 7f834beaf740 1 -- 172.21.15.16:0/481489737 shutdown_connections 2024-09-12T02:52:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.409+0000 7f834beaf740 1 -- 172.21.15.16:0/481489737 wait complete. 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.409+0000 7f834beaf740 1 -- start start 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.409+0000 7f834beaf740 1 Processor -- start 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 0x55dcd1658ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 0x55dcd16539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83441b0c60 con 0x55dcd1646b40 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f83441b0590 con 0x55dcd16455d0 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f83441ad7d0 con 0x55dcd1650370 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f834beaf740 10 client.0 _ll_register_callbacks cb 0x7f833809adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36384/0 (socket says 172.21.15.16:36384) 2024-09-12T02:52:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 -- 172.21.15.16:0/2750952975 learned_addr learned my addr 172.21.15.16:0/2750952975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f8343fff640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 0x55dcd16539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f8342ffd640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 0x55dcd1658ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 -- 172.21.15.16:0/2750952975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 msgr2=0x55dcd16539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 0x55dcd16539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 -- 172.21.15.16:0/2750952975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 msgr2=0x55dcd1658ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 0x55dcd1658ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f83437fe640 1 -- 172.21.15.16:0/2750952975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd16541e0 con 0x55dcd16455d0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f8343fff640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 0x55dcd16539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.410+0000 7f8342ffd640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 0x55dcd1658ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f83437fe640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f833402a450 tx=0x7f8334004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fadb4ca0bbc515f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833404a070 con 0x55dcd16455d0 2024-09-12T02:52:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8334005040 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c013520 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8334056050 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 10 client.26168 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dcd16ca730 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c013520 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f83340316b0 con 0x55dcd16455d0 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 10 client.26168 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 10 client.26168 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 20 client.26168 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd16cae00 0x55dcd16cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 4 client.26168 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.411+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcd15b2d70 con 0x55dcd16cae00 2024-09-12T02:52:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8342ffd640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd16cae00 0x55dcd16cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8322ffd640 10 client.26168 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8342ffd640 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd16cae00 0x55dcd16cd1f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfa8c73444bc4476 server_cookie=85481e8b6e7c8b15 in_seq=0 out_seq=0 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8322ffd640 1 -- 172.21.15.16:0/2750952975 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcd15b2d70 con 0x55dcd16cae00 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8322ffd640 10 client.26168 handle_command_reply: tid=0 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 client.26168 shutdown 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 msgr2=0x55dcd16585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f833402a450 tx=0x7f8334004040 comp rx=0 tx=0).stop 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 shutdown_connections 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55dcd16cae00 0x55dcd16cd1f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dcd1650370 0x55dcd16539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f8322ffd640 0 client.26168 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dcd1646b40 0x55dcd1658ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 --2- 172.21.15.16:0/2750952975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcd16455d0 0x55dcd16585a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 >> 172.21.15.16:0/2750952975 conn(0x55dcd160e310 msgr2=0x55dcd164c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.412+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 shutdown_connections 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f834beaf740 1 -- 172.21.15.16:0/2750952975 wait complete. 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f834beaf740 20 client.26168 trim_cache size 0 max 16384 2024-09-12T02:52:46.419 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 msgr2=0x7f832c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 0x7f832c0a50c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f833c01b000 tx=0x7f833c01d000 comp rx=0 tx=0).stop 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 msgr2=0x7f8344100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f83440697d0 tx=0x7f833800b850 comp rx=0 tx=0).stop 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 shutdown_connections 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f834410c2f0 0x7f83441014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f832c0a2c00 0x7f832c0a50c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83441037a0 0x7f8344100f90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 --2- 172.21.15.16:0/3650227131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8344102df0 0x7f8344100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.413+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 >> 172.21.15.16:0/3650227131 conn(0x7f83440fc7d0 msgr2=0x7f83440fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.414+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 shutdown_connections 2024-09-12T02:52:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:46.414+0000 7f8349ad6640 1 -- 172.21.15.16:0/3650227131 wait complete. 2024-09-12T02:52:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413692 9.57 MB/sec execute 108 sec latency 40.985 ms 2024-09-12T02:52:47.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:52:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:47.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416033 9.61 MB/sec execute 109 sec latency 11.660 ms 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 -- 172.21.15.16:0/1251779514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 msgr2=0x7fae2c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 --2- 172.21.15.16:0/1251779514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c1031b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009930 tx=0x7fae1c02a420 comp rx=0 tx=0).stop 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 -- 172.21.15.16:0/1251779514 shutdown_connections 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 --2- 172.21.15.16:0/1251779514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 0x7fae2c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 --2- 172.21.15.16:0/1251779514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 0x7fae2c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 --2- 172.21.15.16:0/1251779514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c1031b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 -- 172.21.15.16:0/1251779514 >> 172.21.15.16:0/1251779514 conn(0x7fae2c0fc7d0 msgr2=0x7fae2c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 -- 172.21.15.16:0/1251779514 shutdown_connections 2024-09-12T02:52:47.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 -- 172.21.15.16:0/1251779514 wait complete. 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.977+0000 7fae32853640 1 Processor -- start 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 -- start start 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 0x7fae2c100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 0x7fae2c1014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1120b0 con 0x7fae2c102dd0 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c111f30 con 0x7fae2c108300 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae32853640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c112230 con 0x7fae2c103780 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47146/0 (socket says 172.21.15.16:47146) 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 -- 172.21.15.16:0/3134209820 learned_addr learned my addr 172.21.15.16:0/3134209820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae30df0640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 0x7fae2c1014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 0x7fae2c100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 -- 172.21.15.16:0/3134209820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 msgr2=0x7fae2c100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 0x7fae2c100f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 -- 172.21.15.16:0/3134209820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 msgr2=0x7fae2c1014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 0x7fae2c1014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.978+0000 7fae2bfff640 1 -- 172.21.15.16:0/3134209820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c1a6ab0 con 0x7fae2c102dd0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae2bfff640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009230 tx=0x7fae1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae297fa640 1 -- 172.21.15.16:0/3134209820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c03f030 con 0x7fae2c102dd0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae297fa640 1 -- 172.21.15.16:0/3134209820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae1c003ad0 con 0x7fae2c102dd0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c1a6d40 con 0x7fae2c102dd0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae297fa640 1 -- 172.21.15.16:0/3134209820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c035070 con 0x7fae2c102dd0 2024-09-12T02:52:47.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.979+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c1a71c0 con 0x7fae2c102dd0 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.981+0000 7fae297fa640 1 -- 172.21.15.16:0/3134209820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fae1c042030 con 0x7fae2c102dd0 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 Processor -- start 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 -- start start 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf95a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cfa2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55575cefb2f0 con 0x55575cf96b40 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55575cea2cf0 con 0x55575cf955d0 2024-09-12T02:52:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae34c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55575cef72b0 con 0x55575cfa0370 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae297fa640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 0x7fae140a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf95a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae30df0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae30df0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47158/0 (socket says 172.21.15.16:47158) 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae30df0640 1 -- 172.21.15.16:0/1186990440 learned_addr learned my addr 172.21.15.16:0/1186990440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.982+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cfa2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae30df0640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 0x7fae140a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae297fa640 1 -- 172.21.15.16:0/3134209820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fae1c0ef430 con 0x7fae2c102dd0 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 -- 172.21.15.16:0/1186990440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 msgr2=0x55575cfa2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cfa2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 -- 172.21.15.16:0/1186990440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 msgr2=0x55575cf95090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 -- 172.21.15.16:0/1186990440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55575ce873a0 con 0x55575cf955d0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae30df0640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cfa2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf95a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0b86c0 tx=0x7fae1c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea00de8596226541 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae30df0640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 0x7fae140a5040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae24005f40 tx=0x7fae24004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae127fc640 1 -- 172.21.15.16:0/1186990440 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c049070 con 0x55575cf955d0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.983+0000 7fae127fc640 1 -- 172.21.15.16:0/1186990440 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae1c03b460 con 0x55575cf955d0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae127fc640 1 -- 172.21.15.16:0/1186990440 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c003c80 con 0x55575cf955d0 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 -- 172.21.15.16:0/1186990440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 msgr2=0x55575cf95a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf95a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0b86c0 tx=0x7fae1c009260 comp rx=0 tx=0).stop 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 -- 172.21.15.16:0/1186990440 shutdown_connections 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cfa2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf95a10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1186990440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf95090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 -- 172.21.15.16:0/1186990440 >> 172.21.15.16:0/1186990440 conn(0x55575cf5e310 msgr2=0x55575cf91c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 -- 172.21.15.16:0/1186990440 shutdown_connections 2024-09-12T02:52:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.984+0000 7fae34c2c740 1 -- 172.21.15.16:0/1186990440 wait complete. 2024-09-12T02:52:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 -- start start 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 Processor -- start 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf9d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cf9d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55575cfa2e30 con 0x55575cf96b40 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55575ceb6060 con 0x55575cf955d0 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55575cfa2fb0 con 0x55575cfa0370 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.985+0000 7fae34c2c740 10 client.0 _ll_register_callbacks cb 0x55575cc098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae30df0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cf9d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36424/0 (socket says 172.21.15.16:36424) 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 -- 172.21.15.16:0/2845399357 learned_addr learned my addr 172.21.15.16:0/2845399357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf9d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 -- 172.21.15.16:0/2845399357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 msgr2=0x55575cf9d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cf9d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 -- 172.21.15.16:0/2845399357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 msgr2=0x55575cf9d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf9d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 -- 172.21.15.16:0/2845399357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55575cc6c3c0 con 0x55575cf955d0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae2bfff640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0c4380 tx=0x7fae1c0bfff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e1fe408c6fe0e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c049050 con 0x55575cf955d0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fae1c0bbf50 con 0x55575cf955d0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55575cf08860 con 0x55575cf955d0 2024-09-12T02:52:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae1c0bb830 con 0x55575cf955d0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae34c2c740 10 client.26180 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.986+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55575cfa2e30 con 0x55575cf955d0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55575cf08860 con 0x55575cf955d0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae1c03bb10 con 0x55575cf955d0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 10 client.26180 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 10 client.26180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 20 client.26180 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575cfafb10 0x55575cfb1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 4 client.26180 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55575cf02d70 con 0x55575cfafb10 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575cfafb10 0x55575cfb1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae0b7fe640 10 client.26180 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.987+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575cfafb10 0x55575cfb1fb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b987a9910cff0269 server_cookie=aa79f4a1af72173d in_seq=0 out_seq=0 2024-09-12T02:52:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae0b7fe640 1 -- 172.21.15.16:0/2845399357 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55575cf02d70 con 0x55575cfafb10 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae0b7fe640 10 client.26180 handle_command_reply: tid=0 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 client.26180 shutdown 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 msgr2=0x55575cf9fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0c4380 tx=0x7fae1c0bfff0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 shutdown_connections 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575cfafb10 0x55575cfb1fb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cfa0370 0x55575cf9d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cf9d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae0b7fe640 0 client.26180 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 --2- 172.21.15.16:0/2845399357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cf955d0 0x55575cf9fdc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 >> 172.21.15.16:0/2845399357 conn(0x55575cf5e310 msgr2=0x55575cf979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.988+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 shutdown_connections 2024-09-12T02:52:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.989+0000 7fae34c2c740 1 -- 172.21.15.16:0/2845399357 wait complete. 2024-09-12T02:52:47.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:47.989+0000 7fae34c2c740 20 client.26180 trim_cache size 0 max 16384 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 Processor -- start 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 -- start start 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cf90bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cfa1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1af760 con 0x55575cf96b40 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1b12a0 con 0x55575cfa0370 2024-09-12T02:52:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae34c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1afe30 con 0x55575cf955d0 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47178/0 (socket says 172.21.15.16:47178) 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae30df0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cfa1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1305384077 learned_addr learned my addr 172.21.15.16:0/1305384077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cf90bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1305384077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 msgr2=0x55575cf90bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cf90bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1305384077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 msgr2=0x55575cfa1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cfa1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1305384077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55575cf92960 con 0x55575cf96b40 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.005+0000 7fae30df0640 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cfa1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.006+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fae2c1024c0 tx=0x7fae20005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a682faf4a84a602a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.006+0000 7fae11ffb640 1 -- 172.21.15.16:0/1305384077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20005220 con 0x55575cf96b40 2024-09-12T02:52:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.006+0000 7fae11ffb640 1 -- 172.21.15.16:0/1305384077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae20004c80 con 0x55575cf96b40 2024-09-12T02:52:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.006+0000 7fae11ffb640 1 -- 172.21.15.16:0/1305384077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2001eab0 con 0x55575cf96b40 2024-09-12T02:52:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- 172.21.15.16:0/1305384077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 msgr2=0x55575cfa1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fae2c1024c0 tx=0x7fae20005f90 comp rx=0 tx=0).stop 2024-09-12T02:52:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- 172.21.15.16:0/1305384077 shutdown_connections 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cfa1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa1760 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1305384077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cf90bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- 172.21.15.16:0/1305384077 >> 172.21.15.16:0/1305384077 conn(0x55575cf5e310 msgr2=0x55575cf904e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- 172.21.15.16:0/1305384077 shutdown_connections 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- 172.21.15.16:0/1305384077 wait complete. 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 -- start start 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.007+0000 7fae34c2c740 1 Processor -- start 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cfa85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cf9a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1b0bd0 con 0x55575cf96b40 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1b0500 con 0x55575cfa0370 2024-09-12T02:52:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1ad740 con 0x55575cf955d0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae34c2c740 10 client.0 _ll_register_callbacks cb 0x7fae24009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47188/0 (socket says 172.21.15.16:47188) 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cfa85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1242620848 learned_addr learned my addr 172.21.15.16:0/1242620848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae30df0640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cf9a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1242620848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 msgr2=0x55575cfa85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cfa85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 msgr2=0x55575cf9a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cf9a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 -- 172.21.15.16:0/1242620848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55575cf9add0 con 0x55575cf96b40 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae30df0640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cf9a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae2b7fe640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fae20009710 tx=0x7fae20009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545440c6fd81eb9f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.008+0000 7fae0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20037060 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae20004c80 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fae200048d0 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fae20005020 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 10 client.16641 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55575cf9a0d0 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae20004c80 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fae2001ec60 con 0x55575cf96b40 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 10 client.16641 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 20 client.16641 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575d01bd00 0x55575d01e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 4 client.16641 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:52:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.009+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55575ce7f170 con 0x55575d01bd00 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575d01bd00 0x55575d01e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae0b7fe640 10 client.16641 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae2bfff640 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575d01bd00 0x55575d01e120 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a60f3103ca38f7 server_cookie=64de2a5457e2e7fd in_seq=0 out_seq=0 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae0b7fe640 1 -- 172.21.15.16:0/1242620848 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55575ce7f170 con 0x55575d01bd00 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae0b7fe640 10 client.16641 handle_command_reply: tid=0 2024-09-12T02:52:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.010+0000 7fae34c2c740 1 client.16641 shutdown 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 msgr2=0x55575cfa8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fae20009710 tx=0x7fae20009f70 comp rx=0 tx=0).stop 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 shutdown_connections 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55575d01bd00 0x55575d01e120 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55575cfa0370 0x55575cf9a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55575cf96b40 0x55575cfa8b30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 --2- 172.21.15.16:0/1242620848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55575cf955d0 0x55575cfa85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 >> 172.21.15.16:0/1242620848 conn(0x55575cf5e310 msgr2=0x55575cf97110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae0b7fe640 0 client.16641 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 shutdown_connections 2024-09-12T02:52:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 1 -- 172.21.15.16:0/1242620848 wait complete. 2024-09-12T02:52:48.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:47 smithi016 ceph-mon[30925]: pgmap v483: 97 pgs: 97 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2024-09-12T02:52:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.011+0000 7fae34c2c740 20 client.16641 trim_cache size 0 max 16384 2024-09-12T02:52:48.016 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 msgr2=0x7fae140a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 0x7fae140a5040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae24005f40 tx=0x7fae24004ab0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 msgr2=0x7fae2c100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009230 tx=0x7fae1c004270 comp rx=0 tx=0).stop 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 shutdown_connections 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae2c108300 0x7fae2c1014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae2c103780 0x7fae2c100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fae140a2b80 0x7fae140a5040 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 --2- 172.21.15.16:0/3134209820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae2c102dd0 0x7fae2c100a30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 >> 172.21.15.16:0/3134209820 conn(0x7fae2c0fc7d0 msgr2=0x7fae2c105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 shutdown_connections 2024-09-12T02:52:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:48.012+0000 7fae32853640 1 -- 172.21.15.16:0/3134209820 wait complete. 2024-09-12T02:52:48.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:47 smithi033 ceph-mon[35092]: pgmap v483: 97 pgs: 97 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2024-09-12T02:52:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:47 smithi184 ceph-mon[35321]: pgmap v483: 97 pgs: 97 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2024-09-12T02:52:48.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 419127 9.69 MB/sec execute 110 sec latency 10.525 ms 2024-09-12T02:52:49.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 422216 9.76 MB/sec execute 111 sec latency 40.785 ms 2024-09-12T02:52:49.860 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:52:49.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:52:50.125 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:49 smithi016 ceph-mon[30925]: pgmap v484: 97 pgs: 97 active+clean; 164 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-12T02:52:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:49 smithi033 ceph-mon[35092]: pgmap v484: 97 pgs: 97 active+clean; 164 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-12T02:52:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:49 smithi184 ceph-mon[35321]: pgmap v484: 97 pgs: 97 active+clean; 164 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-12T02:52:50.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.612+0000 7fe28d63c640 1 -- 172.21.15.16:0/4242375268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 msgr2=0x7fe28810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.612+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4242375268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28810e610 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe278009ca0 tx=0x7fe27802a950 comp rx=0 tx=0).stop 2024-09-12T02:52:50.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 -- 172.21.15.16:0/4242375268 shutdown_connections 2024-09-12T02:52:50.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4242375268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28810e610 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4242375268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288103660 0x7fe28810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4242375268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288102d40 0x7fe288103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 -- 172.21.15.16:0/4242375268 >> 172.21.15.16:0/4242375268 conn(0x7fe2880fc7a0 msgr2=0x7fe2880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 -- 172.21.15.16:0/4242375268 shutdown_connections 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 -- 172.21.15.16:0/4242375268 wait complete. 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.613+0000 7fe28d63c640 1 Processor -- start 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 -- start start 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288103660 0x7fe288198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe288111f70 con 0x7fe28810c170 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe288111df0 con 0x7fe288102d40 2024-09-12T02:52:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe28d63c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881120f0 con 0x7fe288103660 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2867fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288103660 0x7fe288198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36464/0 (socket says 172.21.15.16:36464) 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe286ffd640 1 -- 172.21.15.16:0/4257524791 learned_addr learned my addr 172.21.15.16:0/4257524791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 -- 172.21.15.16:0/4257524791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288103660 msgr2=0x7fe288198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288103660 0x7fe288198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 -- 172.21.15.16:0/4257524791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 msgr2=0x7fe288198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe2877fe640 1 -- 172.21.15.16:0/4257524791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28819d3c0 con 0x7fe28810c170 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.614+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe2877fe640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28819cce0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe278005cc0 tx=0x7fe27802f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe27ffff640 1 -- 172.21.15.16:0/4257524791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278034040 con 0x7fe28810c170 2024-09-12T02:52:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe27ffff640 1 -- 172.21.15.16:0/4257524791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe278030920 con 0x7fe28810c170 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe27ffff640 1 -- 172.21.15.16:0/4257524791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278004610 con 0x7fe28810c170 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28819d650 con 0x7fe28810c170 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.615+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2881a4f90 con 0x7fe28810c170 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.617+0000 7fe27ffff640 1 -- 172.21.15.16:0/4257524791 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe278037020 con 0x7fe28810c170 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.617+0000 7fe28fa15740 1 Processor -- start 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 -- start start 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1511c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1512150 0x55f6d1514540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6d1468280 con 0x55f6d1503340 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6d140fb10 con 0x55f6d1512150 2024-09-12T02:52:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe28fa15740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f6d1464240 con 0x55f6d1502a20 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1511c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe2867fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe2867fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43878/0 (socket says 172.21.15.16:43878) 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.618+0000 7fe2867fc640 1 -- 172.21.15.16:0/4065656570 learned_addr learned my addr 172.21.15.16:0/4065656570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1512150 0x55f6d1514540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 -- 172.21.15.16:0/4065656570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1512150 msgr2=0x55f6d1514540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1512150 0x55f6d1514540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 -- 172.21.15.16:0/4065656570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 msgr2=0x55f6d1511c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1511c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 -- 172.21.15.16:0/4065656570 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d141c1f0 con 0x55f6d1502a20 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1511c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe27ffff640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 0x7fe2740a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe2867fc640 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe270012740 tx=0x7fe27000b700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37231208e25e8d75 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 0x7fe2740a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe27ffff640 1 -- 172.21.15.16:0/4257524791 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe2780efe30 con 0x7fe28810c170 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe27d7fa640 1 -- 172.21.15.16:0/4065656570 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270012cc0 con 0x55f6d1502a20 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe27d7fa640 1 -- 172.21.15.16:0/4065656570 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe270010c50 con 0x55f6d1502a20 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.619+0000 7fe27d7fa640 1 -- 172.21.15.16:0/4065656570 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27001ea50 con 0x55f6d1502a20 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 -- 172.21.15.16:0/4065656570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 msgr2=0x55f6d1502e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe270012740 tx=0x7fe27000b700 comp rx=0 tx=0).stop 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 0x7fe2740a5040 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe280009fd0 tx=0x7fe280005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 -- 172.21.15.16:0/4065656570 shutdown_connections 2024-09-12T02:52:50.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1512150 0x55f6d1514540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1511c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4065656570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1502a20 0x55f6d1502e00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 -- 172.21.15.16:0/4065656570 >> 172.21.15.16:0/4065656570 conn(0x55f6d14ff0f0 msgr2=0x55f6d14ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 -- 172.21.15.16:0/4065656570 shutdown_connections 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.620+0000 7fe28fa15740 1 -- 172.21.15.16:0/4065656570 wait complete. 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 -- start start 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 Processor -- start 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1503340 0x55f6d150f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d150fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6d1514d20 con 0x55f6d1502a20 2024-09-12T02:52:50.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6d1422f10 con 0x55f6d1503340 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f6d1514ea0 con 0x55f6d1512150 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe28fa15740 10 client.0 _ll_register_callbacks cb 0x55f6d13eba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47236/0 (socket says 172.21.15.16:47236) 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 -- 172.21.15.16:0/2619857918 learned_addr learned my addr 172.21.15.16:0/2619857918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe2877fe640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d150fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe2867fc640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1503340 0x55f6d150f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 -- 172.21.15.16:0/2619857918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 msgr2=0x55f6d150fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d150fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 -- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1503340 msgr2=0x55f6d150f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1503340 0x55f6d150f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe286ffd640 1 -- 172.21.15.16:0/2619857918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d11d96a0 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.621+0000 7fe2877fe640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d150fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe286ffd640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe28000f8c0 tx=0x7fe280004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0375710a1a8b10 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280011940 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe280011ae0 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe270010c50 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe280011ce0 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe28fa15740 10 client.16653 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f6d1520a00 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe270010c50 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.622+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe280027070 con 0x55f6d1502a20 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 10 client.16653 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:50.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 20 client.16653 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d1521e60 0x55f6d15242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 4 client.16653 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6d14757f0 con 0x55f6d1521e60 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe2867fc640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d1521e60 0x55f6d15242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe2867fc640 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d1521e60 0x55f6d15242c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1113ba4a080adcc server_cookie=84d0dee5c672329e in_seq=0 out_seq=0 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.623+0000 7fe26e7fc640 10 client.16653 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe26e7fc640 1 -- 172.21.15.16:0/2619857918 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f6d14757f0 con 0x55f6d1521e60 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe26e7fc640 10 client.16653 handle_command_reply: tid=0 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 client.16653 shutdown 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 msgr2=0x55f6d150f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe28000f8c0 tx=0x7fe280004730 comp rx=0 tx=0).stop 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 shutdown_connections 2024-09-12T02:52:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d1521e60 0x55f6d15242c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d150fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1503340 0x55f6d150f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe26e7fc640 0 client.16653 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 --2- 172.21.15.16:0/2619857918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1502a20 0x55f6d150f080 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 >> 172.21.15.16:0/2619857918 conn(0x55f6d14ff0f0 msgr2=0x55f6d1504e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 shutdown_connections 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 1 -- 172.21.15.16:0/2619857918 wait complete. 2024-09-12T02:52:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.624+0000 7fe28fa15740 20 client.16653 trim_cache size 0 max 16384 2024-09-12T02:52:50.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 Processor -- start 2024-09-12T02:52:50.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 -- start start 2024-09-12T02:52:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1529b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d152a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ab2f0 con 0x55f6d1503340 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ace30 con 0x55f6d1502a20 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.643+0000 7fe28fa15740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ab9c0 con 0x55f6d1512150 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe2867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1529b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36492/0 (socket says 172.21.15.16:36492) 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 -- 172.21.15.16:0/4131630555 learned_addr learned my addr 172.21.15.16:0/4131630555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe2877fe640 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d152a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 -- 172.21.15.16:0/4131630555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 msgr2=0x55f6d152a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d152a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 -- 172.21.15.16:0/4131630555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 msgr2=0x55f6d1529b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1529b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 -- 172.21.15.16:0/4131630555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d14fa830 con 0x55f6d1502a20 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe286ffd640 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe28000ffd0 tx=0x7fe28000f4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422467a13f8ea1fb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe27cff9640 1 -- 172.21.15.16:0/4131630555 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280011f00 con 0x55f6d1502a20 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.644+0000 7fe27cff9640 1 -- 172.21.15.16:0/4131630555 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe280038020 con 0x55f6d1502a20 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe27cff9640 1 -- 172.21.15.16:0/4131630555 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2800054f0 con 0x55f6d1502a20 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 -- 172.21.15.16:0/4131630555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 msgr2=0x55f6d1510ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe28000ffd0 tx=0x7fe28000f4a0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 -- 172.21.15.16:0/4131630555 shutdown_connections 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d152a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d1529b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 --2- 172.21.15.16:0/4131630555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d1510ee0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 -- 172.21.15.16:0/4131630555 >> 172.21.15.16:0/4131630555 conn(0x55f6d14ff0f0 msgr2=0x55f6d15107d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 -- 172.21.15.16:0/4131630555 shutdown_connections 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.645+0000 7fe28fa15740 1 -- 172.21.15.16:0/4131630555 wait complete. 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 -- start start 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 Processor -- start 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d15159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d1515f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ac760 con 0x55f6d1503340 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881ac090 con 0x55f6d1502a20 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2881a92f0 con 0x55f6d1512150 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe28fa15740 10 client.0 _ll_register_callbacks cb 0x7fe280009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe2867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d15159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe286ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36506/0 (socket says 172.21.15.16:36506) 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe286ffd640 1 -- 172.21.15.16:0/1314117829 learned_addr learned my addr 172.21.15.16:0/1314117829 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.646+0000 7fe2877fe640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d1515f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 -- 172.21.15.16:0/1314117829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 msgr2=0x55f6d1515f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d1515f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 -- 172.21.15.16:0/1314117829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 msgr2=0x55f6d15159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d15159c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 -- 172.21.15.16:0/1314117829 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d15164c0 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe2867fc640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d15159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe286ffd640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe28001f730 tx=0x7fe280005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcff46c04301c433 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe26e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280011810 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe28001f800 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe280038020 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe28001fa00 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe28fa15740 10 client.26204 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f6d158c970 con 0x55f6d1502a20 2024-09-12T02:52:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.647+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe280038020 con 0x55f6d1502a20 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe280027040 con 0x55f6d1502a20 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 10 client.26204 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 10 client.26204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 20 client.26204 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d158cfd0 0x55f6d158f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 4 client.26204 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f6d146fd00 con 0x55f6d158cfd0 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe2867fc640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d158cfd0 0x55f6d158f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe2867fc640 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d158cfd0 0x55f6d158f3c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=133c2b08596a1f57 server_cookie=47bbbca6cdcf0bce in_seq=0 out_seq=0 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.648+0000 7fe26e7fc640 10 client.26204 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe26e7fc640 1 -- 172.21.15.16:0/1314117829 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f6d146fd00 con 0x55f6d158cfd0 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe26e7fc640 10 client.26204 handle_command_reply: tid=0 2024-09-12T02:52:50.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 client.26204 shutdown 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 msgr2=0x55f6d151a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe28001f730 tx=0x7fe280005760 comp rx=0 tx=0).stop 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 shutdown_connections 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f6d158cfd0 0x55f6d158f3c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6d1512150 0x55f6d1515f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6d1503340 0x55f6d15159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 --2- 172.21.15.16:0/1314117829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6d1502a20 0x55f6d151a640 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 >> 172.21.15.16:0/1314117829 conn(0x55f6d14ff0f0 msgr2=0x55f6d14fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe26e7fc640 0 client.26204 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 shutdown_connections 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 1 -- 172.21.15.16:0/1314117829 wait complete. 2024-09-12T02:52:50.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.649+0000 7fe28fa15740 20 client.26204 trim_cache size 0 max 16384 2024-09-12T02:52:50.657 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:52:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "8a74ccb1-367a-4b94-9d22-27c6fd056c08", 2024-09-12T02:52:50.657 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:52:50.657 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 msgr2=0x7fe2740a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 0x7fe2740a5040 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe280009fd0 tx=0x7fe280005f90 comp rx=0 tx=0).stop 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 msgr2=0x7fe28819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28819cce0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe278005cc0 tx=0x7fe27802f420 comp rx=0 tx=0).stop 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 shutdown_connections 2024-09-12T02:52:50.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2740a2b80 0x7fe2740a5040 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe28810c170 0x7fe28819cce0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe288103660 0x7fe288198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 --2- 172.21.15.16:0/4257524791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe288102d40 0x7fe288198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 >> 172.21.15.16:0/4257524791 conn(0x7fe2880fc7a0 msgr2=0x7fe2880fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 shutdown_connections 2024-09-12T02:52:50.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:50.650+0000 7fe28d63c640 1 -- 172.21.15.16:0/4257524791 wait complete. 2024-09-12T02:52:50.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 424611 9.76 MB/sec execute 112 sec latency 37.487 ms 2024-09-12T02:52:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:50 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:52:51.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:50 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:51.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:50 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:52:51.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:50 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:50 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:52:51.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:50 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:51.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:52:51.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 426852 9.79 MB/sec execute 113 sec latency 17.086 ms 2024-09-12T02:52:51.925 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:52:52.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:51 smithi033 ceph-mon[35092]: pgmap v485: 97 pgs: 97 active+clean; 169 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 286 op/s 2024-09-12T02:52:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:51 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:52:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:51 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:51 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:52:52.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:51 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:51 smithi184 ceph-mon[35321]: pgmap v485: 97 pgs: 97 active+clean; 169 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 286 op/s 2024-09-12T02:52:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:51 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:52:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:51 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:51 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:52:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:51 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:51 smithi016 ceph-mon[30925]: pgmap v485: 97 pgs: 97 active+clean; 169 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 286 op/s 2024-09-12T02:52:52.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:51 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:52:52.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:51 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:51 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:52:52.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:51 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:52:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 -- 172.21.15.16:0/3332478187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0102570 msgr2=0x7f91c01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 --2- 172.21.15.16:0/3332478187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0102570 0x7f91c01029b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009900 tx=0x7f91b402a480 comp rx=0 tx=0).stop 2024-09-12T02:52:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 -- 172.21.15.16:0/3332478187 shutdown_connections 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 --2- 172.21.15.16:0/3332478187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 0x7f91c010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 --2- 172.21.15.16:0/3332478187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0102570 0x7f91c01029b0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 --2- 172.21.15.16:0/3332478187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0100e20 0x7f91c0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 -- 172.21.15.16:0/3332478187 >> 172.21.15.16:0/3332478187 conn(0x7f91c00fc7d0 msgr2=0x7f91c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 -- 172.21.15.16:0/3332478187 shutdown_connections 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.402+0000 7f91c6972640 1 -- 172.21.15.16:0/3332478187 wait complete. 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 Processor -- start 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 -- start start 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 0x7f91c019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0110f80 con 0x7f91c0100e20 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0110e00 con 0x7f91c0102570 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.403+0000 7f91c6972640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0111100 con 0x7f91c0102ef0 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47282/0 (socket says 172.21.15.16:47282) 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36518/0 (socket says 172.21.15.16:36518) 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bffff640 1 -- 172.21.15.16:0/2424060914 learned_addr learned my addr 172.21.15.16:0/2424060914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91c4f0f640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 0x7f91c019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 -- 172.21.15.16:0/2424060914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 msgr2=0x7f91c019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 0x7f91c019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 -- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 msgr2=0x7f91c0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 -- 172.21.15.16:0/2424060914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c019d4e0 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bffff640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f91b4005940 tx=0x7f91b4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bd7fa640 1 -- 172.21.15.16:0/2424060914 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b403f070 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bd7fa640 1 -- 172.21.15.16:0/2424060914 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b4031c30 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91bd7fa640 1 -- 172.21.15.16:0/2424060914 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b4035030 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.404+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c019d770 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.405+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c00fcbb0 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91bd7fa640 1 -- 172.21.15.16:0/2424060914 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f91b4042070 con 0x7f91c0102570 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 Processor -- start 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 -- start start 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 0x563f8e889140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e88ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f8e7e42e0 con 0x563f8e87e4a0 2024-09-12T02:52:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563f8e78bce0 con 0x563f8e87edc0 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563f8e7e02a0 con 0x563f8e889680 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.407+0000 7f91c4f0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 0x563f8e889140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47294/0 (socket says 172.21.15.16:47294) 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 -- 172.21.15.16:0/16439608 learned_addr learned my addr 172.21.15.16:0/16439608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e88ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 -- 172.21.15.16:0/16439608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 msgr2=0x563f8e88ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e88ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 -- 172.21.15.16:0/16439608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 msgr2=0x563f8e889140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 0x563f8e889140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 -- 172.21.15.16:0/16439608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f8e770440 con 0x563f8e87e4a0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bffff640 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 0x563f8e889140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bd7fa640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 0x7f91a80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91c4f0f640 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009f50 tx=0x7f91b8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36aa4a227129d0d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bffff640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 0x7f91a80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91bd7fa640 1 -- 172.21.15.16:0/2424060914 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91b40ef590 con 0x7f91c0102570 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91a67fc640 1 -- 172.21.15.16:0/16439608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b8003c30 con 0x563f8e87e4a0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91a67fc640 1 -- 172.21.15.16:0/16439608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b801f070 con 0x563f8e87e4a0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.408+0000 7f91a67fc640 1 -- 172.21.15.16:0/16439608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b800b9e0 con 0x563f8e87e4a0 2024-09-12T02:52:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 -- 172.21.15.16:0/16439608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 msgr2=0x563f8e87e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91bffff640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 0x7f91a80a5040 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x563f8e87b890 tx=0x7f91b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009f50 tx=0x7f91b8009500 comp rx=0 tx=0).stop 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 -- 172.21.15.16:0/16439608 shutdown_connections 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e88ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87edc0 0x563f8e889140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/16439608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87e4a0 0x563f8e87e880 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 -- 172.21.15.16:0/16439608 >> 172.21.15.16:0/16439608 conn(0x563f8e87aa30 msgr2=0x563f8e87ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 -- 172.21.15.16:0/16439608 shutdown_connections 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.409+0000 7f91c8d4b740 1 -- 172.21.15.16:0/16439608 wait complete. 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 -- start start 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 Processor -- start 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e886250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e886790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f8e88c250 con 0x563f8e87edc0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563f8e79f050 con 0x563f8e87e4a0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.410+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563f8e88c3d0 con 0x563f8e889680 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91c8d4b740 10 client.0 _ll_register_callbacks cb 0x563f8e7df160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36544/0 (socket says 172.21.15.16:36544) 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 -- 172.21.15.16:0/3792387646 learned_addr learned my addr 172.21.15.16:0/3792387646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91c4f0f640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e886790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e886250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 -- 172.21.15.16:0/3792387646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 msgr2=0x563f8e886790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e886790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 -- 172.21.15.16:0/3792387646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 msgr2=0x563f8e886250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e886250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 -- 172.21.15.16:0/3792387646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f8e5553c0 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e886250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.411+0000 7f91bffff640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f91b000ffa0 tx=0x7f91b0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ad176df707c056 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0004020 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91b0018030 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91b801f070 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f91b00041c0 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 10 client.26216 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f8e8a0440 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b801f070 con 0x563f8e87e4a0 2024-09-12T02:52:52.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f91b00038c0 con 0x563f8e87e4a0 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 10 client.26216 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 20 client.26216 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e8a1900 0x563f8e8a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 4 client.26216 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.412+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f8e7f1850 con 0x563f8e8a1900 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.413+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e8a1900 0x563f8e8a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.413+0000 7f919f7fe640 10 client.26216 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.413+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e8a1900 0x563f8e8a3d60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a138fe11c47f181f server_cookie=738b5d95372e4c7b in_seq=0 out_seq=0 2024-09-12T02:52:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f919f7fe640 1 -- 172.21.15.16:0/3792387646 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563f8e7f1850 con 0x563f8e8a1900 2024-09-12T02:52:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f919f7fe640 10 client.26216 handle_command_reply: tid=0 2024-09-12T02:52:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 client.26216 shutdown 2024-09-12T02:52:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 msgr2=0x563f8e888fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f91b000ffa0 tx=0x7f91b0004730 comp rx=0 tx=0).stop 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 shutdown_connections 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e8a1900 0x563f8e8a3d60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e889680 0x563f8e886790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e886250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/3792387646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e888fa0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 >> 172.21.15.16:0/3792387646 conn(0x563f8e87aa30 msgr2=0x563f8e879b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f919f7fe640 0 client.26216 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 shutdown_connections 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 1 -- 172.21.15.16:0/3792387646 wait complete. 2024-09-12T02:52:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.414+0000 7f91c8d4b740 20 client.26216 trim_cache size 0 max 16384 2024-09-12T02:52:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 Processor -- start 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 -- start start 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87e4a0 0x563f8e8a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e8834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ab2e0 con 0x563f8e87edc0 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ace20 con 0x563f8e87aa30 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ab9b0 con 0x563f8e87e4a0 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36546/0 (socket says 172.21.15.16:36546) 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 -- 172.21.15.16:0/1884368900 learned_addr learned my addr 172.21.15.16:0/1884368900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91c4f0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e8834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87e4a0 0x563f8e8a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 -- 172.21.15.16:0/1884368900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87e4a0 msgr2=0x563f8e8a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87e4a0 0x563f8e8a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 -- 172.21.15.16:0/1884368900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 msgr2=0x563f8e8834b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e8834b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.433+0000 7f91bffff640 1 -- 172.21.15.16:0/1884368900 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f8e87e2d0 con 0x563f8e87aa30 2024-09-12T02:52:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.434+0000 7f91bffff640 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f91b0019040 tx=0x7f91b0004130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6166be53bb4badf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.434+0000 7f91a5ffb640 1 -- 172.21.15.16:0/1884368900 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0038070 con 0x563f8e87aa30 2024-09-12T02:52:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.434+0000 7f91a5ffb640 1 -- 172.21.15.16:0/1884368900 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b0003ad0 con 0x563f8e87aa30 2024-09-12T02:52:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.434+0000 7f91a5ffb640 1 -- 172.21.15.16:0/1884368900 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b003d460 con 0x563f8e87aa30 2024-09-12T02:52:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1884368900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 msgr2=0x563f8e8a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f91b0019040 tx=0x7f91b0004130 comp rx=0 tx=0).stop 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1884368900 shutdown_connections 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e8834b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87e4a0 0x563f8e8a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1884368900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87aa30 0x563f8e8a9680 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1884368900 >> 172.21.15.16:0/1884368900 conn(0x563f8e8868f0 msgr2=0x563f8e884b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1884368900 shutdown_connections 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1884368900 wait complete. 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 -- start start 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.435+0000 7f91c8d4b740 1 Processor -- start 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 0x563f8e896400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e896940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e90b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ac750 con 0x563f8e87edc0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ac080 con 0x563f8e87e4a0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01a92c0 con 0x563f8e87aa30 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c8d4b740 10 client.0 _ll_register_callbacks cb 0x7f91b000fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c4f0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e90b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e896940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c4f0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e90b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47334/0 (socket says 172.21.15.16:47334) 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91c4f0f640 1 -- 172.21.15.16:0/1185377438 learned_addr learned my addr 172.21.15.16:0/1185377438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bffff640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 0x563f8e896400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 -- 172.21.15.16:0/1185377438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 msgr2=0x563f8e896400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 0x563f8e896400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 -- 172.21.15.16:0/1185377438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 msgr2=0x563f8e90b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:52:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e90b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 -- 172.21.15.16:0/1185377438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f8e90bc90 con 0x563f8e87e4a0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bffff640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 0x563f8e896400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f91bf7fe640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e896940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91b40ed070 tx=0x7f91b4034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eb2e99c8d434ff0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.436+0000 7f919f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b4049040 con 0x563f8e87e4a0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91b40ba1b0 con 0x563f8e87e4a0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91b0003ad0 con 0x563f8e87e4a0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f91b40bb810 con 0x563f8e87e4a0 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f91c8d4b740 10 client.26222 fetch_fsmap learned FSMap version 32 2024-09-12T02:52:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f8e90cfc0 con 0x563f8e87e4a0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b0003ad0 con 0x563f8e87e4a0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.437+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f91b40c45e0 con 0x563f8e87e4a0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 10 client.26222 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 20 client.26222 populate_metadata read hostname 'smithi016' 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e889680 0x563f8e90f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 4 client.26222 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f8e8873a0 con 0x563f8e889680 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91bffff640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e889680 0x563f8e90f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91bffff640 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e889680 0x563f8e90f730 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3aa80692f019ed5 server_cookie=92fac94cd182f991 in_seq=0 out_seq=0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f919f7fe640 10 client.26222 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f919f7fe640 1 -- 172.21.15.16:0/1185377438 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563f8e8873a0 con 0x563f8e889680 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f919f7fe640 10 client.26222 handle_command_reply: tid=0 2024-09-12T02:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.438+0000 7f91c8d4b740 1 client.26222 shutdown 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 msgr2=0x563f8e896940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e896940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91b40ed070 tx=0x7f91b4034000 comp rx=0 tx=0).stop 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 shutdown_connections 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563f8e889680 0x563f8e90f730 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f8e87edc0 0x563f8e90b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563f8e87e4a0 0x563f8e896940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 --2- 172.21.15.16:0/1185377438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563f8e87aa30 0x563f8e896400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f919f7fe640 0 client.26222 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 >> 172.21.15.16:0/1185377438 conn(0x563f8e8868f0 msgr2=0x563f8e87ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 shutdown_connections 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 1 -- 172.21.15.16:0/1185377438 wait complete. 2024-09-12T02:52:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c8d4b740 20 client.26222 trim_cache size 0 max 16384 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "8a74ccb1-367a-4b94-9d22-27c6fd056c08": { 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "8a74ccb1-367a-4b94-9d22-27c6fd056c08", 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:52:52.445 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 msgr2=0x7f91a80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 0x7f91a80a5040 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x563f8e87b890 tx=0x7f91b0005f90 comp rx=0 tx=0).stop 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 msgr2=0x7f91c0198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f91b4005940 tx=0x7f91b4003c10 comp rx=0 tx=0).stop 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 shutdown_connections 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91c0102ef0 0x7f91c019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91c0102570 0x7f91c0198a70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91a80a2b80 0x7f91a80a5040 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 --2- 172.21.15.16:0/2424060914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c0100e20 0x7f91c0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:52:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.439+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 >> 172.21.15.16:0/2424060914 conn(0x7f91c00fc7d0 msgr2=0x7f91c006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:52:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.440+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 shutdown_connections 2024-09-12T02:52:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:52:52.440+0000 7f91c6972640 1 -- 172.21.15.16:0/2424060914 wait complete. 2024-09-12T02:52:52.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 429650 9.85 MB/sec execute 114 sec latency 20.264 ms 2024-09-12T02:52:53.290 INFO:tasks.cephfs.filesystem:scrub status for tag:8a74ccb1-367a-4b94-9d22-27c6fd056c08 - {'path': '/', 'tag': '8a74ccb1-367a-4b94-9d22-27c6fd056c08', 'options': 'recursive,force'} 2024-09-12T02:52:53.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 432591 9.89 MB/sec execute 115 sec latency 13.482 ms 2024-09-12T02:52:54.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:53 smithi033 ceph-mon[35092]: pgmap v486: 97 pgs: 97 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-12T02:52:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:53 smithi184 ceph-mon[35321]: pgmap v486: 97 pgs: 97 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-12T02:52:54.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:53 smithi016 ceph-mon[30925]: pgmap v486: 97 pgs: 97 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-09-12T02:52:54.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434542 9.89 MB/sec execute 116 sec latency 55.502 ms 2024-09-12T02:52:55.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 436994 9.93 MB/sec execute 117 sec latency 20.111 ms 2024-09-12T02:52:56.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:55 smithi033 ceph-mon[35092]: pgmap v487: 97 pgs: 97 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-12T02:52:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:55 smithi184 ceph-mon[35321]: pgmap v487: 97 pgs: 97 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-12T02:52:56.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:55 smithi016 ceph-mon[30925]: pgmap v487: 97 pgs: 97 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-12T02:52:56.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 440126 10.00 MB/sec execute 118 sec latency 9.665 ms 2024-09-12T02:52:57.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 442865 10.03 MB/sec execute 119 sec latency 16.282 ms 2024-09-12T02:52:58.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:57 smithi033 ceph-mon[35092]: pgmap v488: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-09-12T02:52:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:57 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:52:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:57 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:52:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:57 smithi184 ceph-mon[35321]: pgmap v488: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-09-12T02:52:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:57 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:52:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:57 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:52:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:57 smithi016 ceph-mon[30925]: pgmap v488: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-09-12T02:52:58.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:57 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:52:58.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:57 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:52:58.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445108 10.04 MB/sec execute 120 sec latency 11.144 ms 2024-09-12T02:52:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:58 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:52:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:58 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:52:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:58 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:52:59.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 447521 10.07 MB/sec execute 121 sec latency 29.953 ms 2024-09-12T02:53:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:59 smithi033 ceph-mon[35092]: pgmap v489: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-12T02:53:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:53:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:53:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:52:59 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:53:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:59 smithi184 ceph-mon[35321]: pgmap v489: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-12T02:53:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:53:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:53:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:52:59 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:53:00.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:59 smithi016 ceph-mon[30925]: pgmap v489: 97 pgs: 97 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-12T02:53:00.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:53:00.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:53:00.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:52:59 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:53:00.718 INFO:tasks.workunit.client.0.smithi016.stdout: 1 450489 10.13 MB/sec execute 122 sec latency 50.563 ms 2024-09-12T02:53:01.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452942 10.12 MB/sec execute 123 sec latency 11.915 ms 2024-09-12T02:53:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:01 smithi033 ceph-mon[35092]: pgmap v490: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2024-09-12T02:53:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:01 smithi184 ceph-mon[35321]: pgmap v490: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2024-09-12T02:53:02.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:01 smithi016 ceph-mon[30925]: pgmap v490: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2024-09-12T02:53:02.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455290 10.15 MB/sec execute 124 sec latency 13.501 ms 2024-09-12T02:53:03.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458187 10.21 MB/sec execute 125 sec latency 11.968 ms 2024-09-12T02:53:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:03 smithi033 ceph-mon[35092]: pgmap v491: 97 pgs: 97 active+clean; 183 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 246 op/s 2024-09-12T02:53:04.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:03 smithi184 ceph-mon[35321]: pgmap v491: 97 pgs: 97 active+clean; 183 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 246 op/s 2024-09-12T02:53:04.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:03 smithi016 ceph-mon[30925]: pgmap v491: 97 pgs: 97 active+clean; 183 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 246 op/s 2024-09-12T02:53:04.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460811 10.36 MB/sec execute 126 sec latency 64.480 ms 2024-09-12T02:53:05.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462896 10.37 MB/sec execute 127 sec latency 47.805 ms 2024-09-12T02:53:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:05 smithi184 ceph-mon[35321]: pgmap v492: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-09-12T02:53:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:05 smithi016 ceph-mon[30925]: pgmap v492: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-09-12T02:53:06.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:05 smithi033 ceph-mon[35092]: pgmap v492: 97 pgs: 97 active+clean; 182 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-09-12T02:53:06.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 465979 10.43 MB/sec execute 128 sec latency 10.801 ms 2024-09-12T02:53:07.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469235 10.50 MB/sec execute 129 sec latency 11.343 ms 2024-09-12T02:53:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:07 smithi184 ceph-mon[35321]: pgmap v493: 97 pgs: 97 active+clean; 184 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-09-12T02:53:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:07 smithi016 ceph-mon[30925]: pgmap v493: 97 pgs: 97 active+clean; 184 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-09-12T02:53:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:07 smithi033 ceph-mon[35092]: pgmap v493: 97 pgs: 97 active+clean; 184 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 330 op/s 2024-09-12T02:53:08.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472018 10.52 MB/sec execute 130 sec latency 13.927 ms 2024-09-12T02:53:09.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 474916 10.54 MB/sec execute 131 sec latency 11.961 ms 2024-09-12T02:53:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:10 smithi184 ceph-mon[35321]: pgmap v494: 97 pgs: 97 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-09-12T02:53:10.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:10 smithi016 ceph-mon[30925]: pgmap v494: 97 pgs: 97 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-09-12T02:53:10.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:10 smithi033 ceph-mon[35092]: pgmap v494: 97 pgs: 97 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-09-12T02:53:10.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 477011 10.56 MB/sec execute 132 sec latency 116.893 ms 2024-09-12T02:53:11.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 480269 10.62 MB/sec execute 133 sec latency 17.639 ms 2024-09-12T02:53:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:12 smithi184 ceph-mon[35321]: pgmap v495: 97 pgs: 97 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 342 op/s 2024-09-12T02:53:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:12 smithi016 ceph-mon[30925]: pgmap v495: 97 pgs: 97 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 342 op/s 2024-09-12T02:53:12.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:12 smithi033 ceph-mon[35092]: pgmap v495: 97 pgs: 97 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 342 op/s 2024-09-12T02:53:12.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 483586 10.68 MB/sec execute 134 sec latency 12.281 ms 2024-09-12T02:53:13.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 486735 10.73 MB/sec execute 135 sec latency 24.678 ms 2024-09-12T02:53:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:14 smithi184 ceph-mon[35321]: pgmap v496: 97 pgs: 97 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-09-12T02:53:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:14 smithi016 ceph-mon[30925]: pgmap v496: 97 pgs: 97 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-09-12T02:53:14.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:14 smithi033 ceph-mon[35092]: pgmap v496: 97 pgs: 97 active+clean; 176 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-09-12T02:53:14.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 489479 10.74 MB/sec execute 136 sec latency 11.966 ms 2024-09-12T02:53:15.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 491444 10.74 MB/sec execute 137 sec latency 55.476 ms 2024-09-12T02:53:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:16 smithi184 ceph-mon[35321]: pgmap v497: 97 pgs: 97 active+clean; 175 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-12T02:53:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:16 smithi016 ceph-mon[30925]: pgmap v497: 97 pgs: 97 active+clean; 175 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-12T02:53:16.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:16 smithi033 ceph-mon[35092]: pgmap v497: 97 pgs: 97 active+clean; 175 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 252 op/s 2024-09-12T02:53:16.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 494297 10.79 MB/sec execute 138 sec latency 22.119 ms 2024-09-12T02:53:17.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 497339 10.83 MB/sec execute 139 sec latency 53.506 ms 2024-09-12T02:53:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:18 smithi184 ceph-mon[35321]: pgmap v498: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:53:18.298 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:18 smithi033 ceph-mon[35092]: pgmap v498: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:53:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:18 smithi016 ceph-mon[30925]: pgmap v498: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:53:18.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499813 10.82 MB/sec execute 140 sec latency 39.807 ms 2024-09-12T02:53:19.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 501922 10.83 MB/sec execute 141 sec latency 17.997 ms 2024-09-12T02:53:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:20 smithi184 ceph-mon[35321]: pgmap v499: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 3.5 MiB/s wr, 257 op/s 2024-09-12T02:53:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:20 smithi016 ceph-mon[30925]: pgmap v499: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 3.5 MiB/s wr, 257 op/s 2024-09-12T02:53:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:20 smithi033 ceph-mon[35092]: pgmap v499: 97 pgs: 97 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 3.5 MiB/s wr, 257 op/s 2024-09-12T02:53:20.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504860 10.89 MB/sec execute 142 sec latency 50.110 ms 2024-09-12T02:53:21.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 507838 10.91 MB/sec execute 143 sec latency 16.959 ms 2024-09-12T02:53:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:22 smithi184 ceph-mon[35321]: pgmap v500: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:53:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:22 smithi016 ceph-mon[30925]: pgmap v500: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:53:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:22 smithi033 ceph-mon[35092]: pgmap v500: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:53:22.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 510452 10.92 MB/sec execute 144 sec latency 11.159 ms 2024-09-12T02:53:23.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:53:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:23.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 512760 10.94 MB/sec execute 145 sec latency 22.652 ms 2024-09-12T02:53:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 -- 172.21.15.16:0/3961807360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181011e0 msgr2=0x7f7418105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 --2- 172.21.15.16:0/3961807360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181011e0 0x7f7418105980 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7408009900 tx=0x7f740802a410 comp rx=0 tx=0).stop 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 -- 172.21.15.16:0/3961807360 shutdown_connections 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 --2- 172.21.15.16:0/3961807360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7418105ec0 0x7f7418106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 --2- 172.21.15.16:0/3961807360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181011e0 0x7f7418105980 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 --2- 172.21.15.16:0/3961807360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74181008c0 0x7f7418100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 -- 172.21.15.16:0/3961807360 >> 172.21.15.16:0/3961807360 conn(0x7f74180fc7d0 msgr2=0x7f74180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 -- 172.21.15.16:0/3961807360 shutdown_connections 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.057+0000 7f74203ff640 1 -- 172.21.15.16:0/3961807360 wait complete. 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 Processor -- start 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 -- start start 2024-09-12T02:53:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 0x7f74181a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7418105ec0 0x7f741819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7418118630 con 0x7f74181008c0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74181184b0 con 0x7f74181011e0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f74203ff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74181187b0 con 0x7f7418105ec0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f741e19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f741d99a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 0x7f74181a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f741e19b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35136/0 (socket says 172.21.15.16:35136) 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f741e19b640 1 -- 172.21.15.16:0/941063289 learned_addr learned my addr 172.21.15.16:0/941063289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.058+0000 7f741e99c640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7418105ec0 0x7f741819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 -- 172.21.15.16:0/941063289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7418105ec0 msgr2=0x7f741819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7418105ec0 0x7f741819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 -- 172.21.15.16:0/941063289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 msgr2=0x7f74181a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 0x7f74181a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 -- 172.21.15.16:0/941063289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741819f070 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741d99a640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 0x7f74181a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f741e19b640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f741400e9f0 tx=0x7f741400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f740f7fe640 1 -- 172.21.15.16:0/941063289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741400cd80 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741819f360 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74181ab650 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f740f7fe640 1 -- 172.21.15.16:0/941063289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7414005500 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.059+0000 7f740f7fe640 1 -- 172.21.15.16:0/941063289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414010690 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.062+0000 7f74227d8740 1 Processor -- start 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.062+0000 7f740f7fe640 1 -- 172.21.15.16:0/941063289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f74140046c0 con 0x7f74181008c0 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.062+0000 7f74227d8740 1 -- start start 2024-09-12T02:53:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b84140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b86a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564833adf2e0 con 0x564833b794a0 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564833a86ce0 con 0x564833b79dc0 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f74227d8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564833adb2a0 con 0x564833b84680 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35150/0 (socket says 172.21.15.16:35150) 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 -- 172.21.15.16:0/3622531260 learned_addr learned my addr 172.21.15.16:0/3622531260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741d99a640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b86a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e19b640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b84140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 -- 172.21.15.16:0/3622531260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 msgr2=0x564833b86a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b86a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 -- 172.21.15.16:0/3622531260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 msgr2=0x564833b84140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b84140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e99c640 1 -- 172.21.15.16:0/3622531260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564833a6b440 con 0x564833b794a0 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.063+0000 7f741e19b640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b84140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f740f7fe640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 0x7f74040a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f741e99c640 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7410009f50 tx=0x7f74100118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753a7934cf1b460 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f740f7fe640 1 -- 172.21.15.16:0/941063289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74140cf320 con 0x7f74181008c0 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f741e19b640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 0x7f74040a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f740cdea640 1 -- 172.21.15.16:0/3622531260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410004d20 con 0x564833b794a0 2024-09-12T02:53:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f740cdea640 1 -- 172.21.15.16:0/3622531260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7410004050 con 0x564833b794a0 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.064+0000 7f740cdea640 1 -- 172.21.15.16:0/3622531260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7410005280 con 0x564833b794a0 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 -- 172.21.15.16:0/3622531260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 msgr2=0x564833b79880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7410009f50 tx=0x7f74100118d0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f741e19b640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 0x7f74040a50c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7414004d70 tx=0x7f7414009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 -- 172.21.15.16:0/3622531260 shutdown_connections 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b86a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b84140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 --2- 172.21.15.16:0/3622531260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b79880 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 -- 172.21.15.16:0/3622531260 >> 172.21.15.16:0/3622531260 conn(0x564833b75a30 msgr2=0x564833b75e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 -- 172.21.15.16:0/3622531260 shutdown_connections 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.066+0000 7f74227d8740 1 -- 172.21.15.16:0/3622531260 wait complete. 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 -- start start 2024-09-12T02:53:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 Processor -- start 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 0x564833b83f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b84680 0x564833b81710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564833b87250 con 0x564833b84680 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564833a9a050 con 0x564833b79dc0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564833b873d0 con 0x564833b794a0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f74227d8740 10 client.0 _ll_register_callbacks cb 0x564833ada160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f741e99c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 0x564833b83f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f741e99c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 0x564833b83f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:45098/0 (socket says 172.21.15.16:45098) 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f741d99a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.067+0000 7f741e99c640 1 -- 172.21.15.16:0/3927827279 learned_addr learned my addr 172.21.15.16:0/3927827279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741e99c640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b84680 0x564833b81710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 -- 172.21.15.16:0/3927827279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 msgr2=0x564833b83f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 0x564833b83f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 -- 172.21.15.16:0/3927827279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b84680 msgr2=0x564833b81710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b84680 0x564833b81710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 -- 172.21.15.16:0/3927827279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648338503c0 con 0x564833b79dc0 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f741d99a640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b811d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7408005940 tx=0x7f74080035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d3485077141fc56 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f7401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408035070 con 0x564833b79dc0 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7408003a30 con 0x564833b79dc0 2024-09-12T02:53:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.068+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7410004050 con 0x564833b79dc0 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7408003c30 con 0x564833b79dc0 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 10 client.26234 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564833b92bd0 con 0x564833b79dc0 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7410004050 con 0x564833b79dc0 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f740803c560 con 0x564833b79dc0 2024-09-12T02:53:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 10 client.26234 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 20 client.26234 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833b94130 0x564833b96520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 4 client.26234 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564833aec850 con 0x564833b94130 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.069+0000 7f741e99c640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833b94130 0x564833b96520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.070+0000 7f7401ffb640 10 client.26234 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.070+0000 7f741e99c640 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833b94130 0x564833b96520 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dc30f476b3567f1 server_cookie=5d713b3cf40cbd14 in_seq=0 out_seq=0 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.070+0000 7f7401ffb640 1 -- 172.21.15.16:0/3927827279 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564833aec850 con 0x564833b94130 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.070+0000 7f7401ffb640 10 client.26234 handle_command_reply: tid=0 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.070+0000 7f74227d8740 1 client.26234 shutdown 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 msgr2=0x564833b811d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b811d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7408005940 tx=0x7f74080035f0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 shutdown_connections 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833b94130 0x564833b96520 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b84680 0x564833b81710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b811d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f7401ffb640 0 client.26234 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 --2- 172.21.15.16:0/3927827279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b794a0 0x564833b83f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 >> 172.21.15.16:0/3927827279 conn(0x564833b75a30 msgr2=0x564833b74b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 shutdown_connections 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 1 -- 172.21.15.16:0/3927827279 wait complete. 2024-09-12T02:53:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.071+0000 7f74227d8740 20 client.26234 trim_cache size 0 max 16384 2024-09-12T02:53:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 Processor -- start 2024-09-12T02:53:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 -- start start 2024-09-12T02:53:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b9c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00e400 con 0x564833b794a0 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00ff40 con 0x564833b79dc0 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f74227d8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00ead0 con 0x564833b84680 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741d99a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35160/0 (socket says 172.21.15.16:35160) 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741d99a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60864/0 (socket says 172.21.15.16:60864) 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741e99c640 1 -- 172.21.15.16:0/3367323096 learned_addr learned my addr 172.21.15.16:0/3367323096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.091+0000 7f741e99c640 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b9c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 -- 172.21.15.16:0/3367323096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 msgr2=0x564833b9c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b9c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 -- 172.21.15.16:0/3367323096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 msgr2=0x564833b90510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 -- 172.21.15.16:0/3367323096 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564833b96b50 con 0x564833b79dc0 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741e99c640 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f741d99a640 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7408009850 tx=0x7f7408031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d13f7cb3f9439cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f7403fff640 1 -- 172.21.15.16:0/3367323096 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408056070 con 0x564833b79dc0 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f7403fff640 1 -- 172.21.15.16:0/3367323096 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7408003dd0 con 0x564833b79dc0 2024-09-12T02:53:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.092+0000 7f7403fff640 1 -- 172.21.15.16:0/3367323096 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408035070 con 0x564833b79dc0 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- 172.21.15.16:0/3367323096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 msgr2=0x564833b9bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7408009850 tx=0x7f7408031400 comp rx=0 tx=0).stop 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- 172.21.15.16:0/3367323096 shutdown_connections 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833b9c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b9bde0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 --2- 172.21.15.16:0/3367323096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b90510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- 172.21.15.16:0/3367323096 >> 172.21.15.16:0/3367323096 conn(0x564833b75a30 msgr2=0x564833b83250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- 172.21.15.16:0/3367323096 shutdown_connections 2024-09-12T02:53:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- 172.21.15.16:0/3367323096 wait complete. 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 -- start start 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.093+0000 7f74227d8740 1 Processor -- start 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b891a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833bf7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00f870 con 0x564833b794a0 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00f1a0 con 0x564833b79dc0 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc00c3e0 con 0x564833b84680 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f74227d8740 10 client.0 _ll_register_callbacks cb 0x7f7408009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741e99c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35166/0 (socket says 172.21.15.16:35166) 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741e99c640 1 -- 172.21.15.16:0/3481158690 learned_addr learned my addr 172.21.15.16:0/3481158690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b891a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 -- 172.21.15.16:0/3481158690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 msgr2=0x564833bf7df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833bf7df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 -- 172.21.15.16:0/3481158690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 msgr2=0x564833b88c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741d99a640 1 -- 172.21.15.16:0/3481158690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564833b76ae0 con 0x564833b79dc0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.094+0000 7f741e99c640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f741d99a640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b891a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x564833b7a890 tx=0x7f7408000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d951cd1a2d9d4891 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408035070 con 0x564833b79dc0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7408003a30 con 0x564833b79dc0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7408003dd0 con 0x564833b79dc0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7408056040 con 0x564833b79dc0 2024-09-12T02:53:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 10 client.26240 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564833bff5d0 con 0x564833b79dc0 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7408003dd0 con 0x564833b79dc0 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7408004af0 con 0x564833b79dc0 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 10 client.26240 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 20 client.26240 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833bffb60 0x564833c01f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 4 client.26240 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.095+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564833ae6d60 con 0x564833bffb60 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f741e99c640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833bffb60 0x564833c01f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f7401ffb640 10 client.26240 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f741e99c640 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833bffb60 0x564833c01f80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d512fef64ffb90b server_cookie=896d8ee3c9cc10c5 in_seq=0 out_seq=0 2024-09-12T02:53:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f7401ffb640 1 -- 172.21.15.16:0/3481158690 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564833ae6d60 con 0x564833bffb60 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f7401ffb640 10 client.26240 handle_command_reply: tid=0 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 client.26240 shutdown 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 msgr2=0x564833b891a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b891a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x564833b7a890 tx=0x7f7408000bc0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 shutdown_connections 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564833bffb60 0x564833c01f80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564833b84680 0x564833bf7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564833b79dc0 0x564833b891a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 --2- 172.21.15.16:0/3481158690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564833b794a0 0x564833b88c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f7401ffb640 0 client.26240 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.096+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 >> 172.21.15.16:0/3481158690 conn(0x564833b75a30 msgr2=0x564833b74350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 shutdown_connections 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74227d8740 1 -- 172.21.15.16:0/3481158690 wait complete. 2024-09-12T02:53:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74227d8740 20 client.26240 trim_cache size 0 max 16384 2024-09-12T02:53:24.103 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:53:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:53:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:53:24.103 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 msgr2=0x7f74040a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 0x7f74040a50c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7414004d70 tx=0x7f7414009310 comp rx=0 tx=0).stop 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 msgr2=0x7f74181a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f741400e9f0 tx=0x7f741400edf0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 shutdown_connections 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7418105ec0 0x7f741819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74181011e0 0x7f74181a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74040a2c00 0x7f74040a50c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 --2- 172.21.15.16:0/941063289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74181008c0 0x7f74181a4780 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.097+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 >> 172.21.15.16:0/941063289 conn(0x7f74180fc7d0 msgr2=0x7f7418103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.098+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 shutdown_connections 2024-09-12T02:53:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:24.098+0000 7f74203ff640 1 -- 172.21.15.16:0/941063289 wait complete. 2024-09-12T02:53:24.265 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:24 smithi016 ceph-mon[30925]: pgmap v501: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-12T02:53:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:24 smithi184 ceph-mon[35321]: pgmap v501: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-12T02:53:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:24 smithi033 ceph-mon[35092]: pgmap v501: 97 pgs: 97 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-09-12T02:53:24.719 INFO:tasks.workunit.client.0.smithi016.stdout: 1 515734 10.98 MB/sec execute 146 sec latency 31.408 ms 2024-09-12T02:53:24.918 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:53:24.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:53:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:25.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 518756 11.03 MB/sec execute 147 sec latency 66.614 ms 2024-09-12T02:53:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 -- 172.21.15.16:0/1839958354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc101de0 msgr2=0x7f50bc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 --2- 172.21.15.16:0/1839958354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc101de0 0x7f50bc1021c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009900 tx=0x7f50b402a410 comp rx=0 tx=0).stop 2024-09-12T02:53:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 -- 172.21.15.16:0/1839958354 shutdown_connections 2024-09-12T02:53:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 --2- 172.21.15.16:0/1839958354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc106f10 0x7f50bc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 --2- 172.21.15.16:0/1839958354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 --2- 172.21.15.16:0/1839958354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc101de0 0x7f50bc1021c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 -- 172.21.15.16:0/1839958354 >> 172.21.15.16:0/1839958354 conn(0x7f50bc0fd8c0 msgr2=0x7f50bc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 -- 172.21.15.16:0/1839958354 shutdown_connections 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.738+0000 7f50c3cca640 1 -- 172.21.15.16:0/1839958354 wait complete. 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 Processor -- start 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 -- start start 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 0x7f50bc19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc111f30 con 0x7f50bc106f10 2024-09-12T02:53:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc111db0 con 0x7f50bc102790 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c3cca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc1120b0 con 0x7f50bc101de0 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c1a66640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c1265640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c2267640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 0x7f50bc19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c1a66640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59878/0 (socket says 172.21.15.16:59878) 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c1a66640 1 -- 172.21.15.16:0/2933422014 learned_addr learned my addr 172.21.15.16:0/2933422014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.739+0000 7f50c1265640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57856/0 (socket says 172.21.15.16:57856) 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 -- 172.21.15.16:0/2933422014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 msgr2=0x7f50bc198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 -- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 msgr2=0x7f50bc19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 0x7f50bc19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 -- 172.21.15.16:0/2933422014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc19d4c0 con 0x7f50bc101de0 2024-09-12T02:53:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c2267640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 0x7f50bc19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c1a66640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b40042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b403f030 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b4003ce0 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc19d750 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4035070 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.740+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc074420 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.741+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc1032c0 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.742+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f50b4042030 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.743+0000 7f50baffd640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 0x7f50a40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.743+0000 7f50c1265640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 0x7f50a40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.743+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50b40ef750 con 0x7f50bc101de0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.744+0000 7f50c1265640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 0x7f50a40a50c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50bc1087b0 tx=0x7f50b000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:25.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.746+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f50b40bc0d0 con 0x7f50bc101de0 2024-09-12T02:53:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.964+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50bc199870 con 0x7f50bc101de0 2024-09-12T02:53:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.965+0000 7f50baffd640 1 -- 172.21.15.16:0/2933422014 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f50b40c03d0 con 0x7f50bc101de0 2024-09-12T02:53:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:53:25.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:53:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:53:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 msgr2=0x7f50a40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 0x7f50a40a50c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50bc1087b0 tx=0x7f50b000b410 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 msgr2=0x7f50bc198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b40042f0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 shutdown_connections 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50a40a2c00 0x7f50a40a50c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc106f10 0x7f50bc19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50bc102790 0x7f50bc198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 --2- 172.21.15.16:0/2933422014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc101de0 0x7f50bc198510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 >> 172.21.15.16:0/2933422014 conn(0x7f50bc0fd8c0 msgr2=0x7f50bc0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 shutdown_connections 2024-09-12T02:53:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:25.968+0000 7f50c3cca640 1 -- 172.21.15.16:0/2933422014 wait complete. 2024-09-12T02:53:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:26 smithi184 ceph-mon[35321]: pgmap v502: 97 pgs: 97 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-12T02:53:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:26 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2933422014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:26.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:26 smithi016 ceph-mon[30925]: pgmap v502: 97 pgs: 97 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-12T02:53:26.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:26 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2933422014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:26 smithi033 ceph-mon[35092]: pgmap v502: 97 pgs: 97 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-09-12T02:53:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:26 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2933422014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:26.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 521592 11.05 MB/sec execute 148 sec latency 16.859 ms 2024-09-12T02:53:26.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:53:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:27.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 524474 11.06 MB/sec execute 149 sec latency 44.112 ms 2024-09-12T02:53:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 -- 172.21.15.16:0/3325315339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 msgr2=0x7f1f50100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3325315339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50100cc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009900 tx=0x7f1f4002a410 comp rx=0 tx=0).stop 2024-09-12T02:53:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 -- 172.21.15.16:0/3325315339 shutdown_connections 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3325315339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 0x7f1f5010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3325315339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 0x7f1f50107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3325315339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50100cc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 -- 172.21.15.16:0/3325315339 >> 172.21.15.16:0/3325315339 conn(0x7f1f50077210 msgr2=0x7f1f50077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.725+0000 7f1f558f7640 1 -- 172.21.15.16:0/3325315339 shutdown_connections 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.726+0000 7f1f558f7640 1 -- 172.21.15.16:0/3325315339 wait complete. 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.726+0000 7f1f558f7640 1 Processor -- start 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.726+0000 7f1f558f7640 1 -- start start 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.726+0000 7f1f558f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.726+0000 7f1f558f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 0x7f1f50101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f558f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 0x7f1f501022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f558f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f50112070 con 0x7f1f501008e0 2024-09-12T02:53:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f558f7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f50111ef0 con 0x7f1f50108180 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f558f7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501121f0 con 0x7f1f50101290 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49756/0 (socket says 172.21.15.16:49756) 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 -- 172.21.15.16:0/3894510121 learned_addr learned my addr 172.21.15.16:0/3894510121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 0x7f1f50101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4f7fe640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 0x7f1f501022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 -- 172.21.15.16:0/3894510121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 msgr2=0x7f1f50101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 0x7f1f50101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 -- 172.21.15.16:0/3894510121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 msgr2=0x7f1f501022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 0x7f1f501022e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 -- 172.21.15.16:0/3894510121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f501a69c0 con 0x7f1f501008e0 2024-09-12T02:53:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1f40006010 tx=0x7f1f40004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f548f5640 1 -- 172.21.15.16:0/3894510121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4003f030 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f548f5640 1 -- 172.21.15.16:0/3894510121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f40004440 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f501a6c50 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.727+0000 7f1f548f5640 1 -- 172.21.15.16:0/3894510121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40035070 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.728+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f501a7180 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f548f5640 1 -- 172.21.15.16:0/3894510121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1f40042030 con 0x7f1f501008e0 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 Processor -- start 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 -- start start 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fc9d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fc9da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcaa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562e2fc032f0 con 0x562e2fc9eb40 2024-09-12T02:53:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562e2fbaacf0 con 0x562e2fca8370 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562e2fbff2b0 con 0x562e2fc9d5d0 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fc9da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.730+0000 7f1f4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fc9d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fc9d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49772/0 (socket says 172.21.15.16:49772) 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f4f7fe640 1 -- 172.21.15.16:0/3514167931 learned_addr learned my addr 172.21.15.16:0/3514167931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcaa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 -- 172.21.15.16:0/3514167931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 msgr2=0x562e2fc9da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fc9da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f548f5640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 0x7f1f3c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 -- 172.21.15.16:0/3514167931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 msgr2=0x562e2fc9d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fc9d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 -- 172.21.15.16:0/3514167931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2fb8f3a0 con 0x562e2fca8370 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f4f7fe640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 0x7f1f3c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f548f5640 1 -- 172.21.15.16:0/3894510121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f400ef2f0 con 0x7f1f501008e0 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcaa760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1f5006a760 tx=0x7f1f38011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc06751027eaf69 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f45ffb640 1 -- 172.21.15.16:0/3514167931 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38004230 con 0x562e2fca8370 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f45ffb640 1 -- 172.21.15.16:0/3514167931 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f38009e10 con 0x562e2fca8370 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.731+0000 7f1f45ffb640 1 -- 172.21.15.16:0/3514167931 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38004d70 con 0x562e2fca8370 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3514167931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 msgr2=0x562e2fcaa760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f4f7fe640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 0x7f1f3c0a50c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1f48010610 tx=0x7f1f48004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcaa760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1f5006a760 tx=0x7f1f38011810 comp rx=0 tx=0).stop 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3514167931 shutdown_connections 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcaa760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fc9da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3514167931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fc9d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3514167931 >> 172.21.15.16:0/3514167931 conn(0x562e2fc66310 msgr2=0x562e2fc99c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3514167931 shutdown_connections 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.732+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3514167931 wait complete. 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 -- start start 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 Processor -- start 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 0x562e2fca4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fca8370 0x562e2fca54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562e2fcab070 con 0x562e2fca8370 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562e2fbbe060 con 0x562e2fc9d5d0 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562e2fcab1f0 con 0x562e2fc9eb40 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f57cd0740 10 client.0 _ll_register_callbacks cb 0x562e2f9118c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 0x562e2fca4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fca8370 0x562e2fca54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 0x562e2fca4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59928/0 (socket says 172.21.15.16:59928) 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57872/0 (socket says 172.21.15.16:57872) 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.733+0000 7f1f47fff640 1 -- 172.21.15.16:0/3711468098 learned_addr learned my addr 172.21.15.16:0/3711468098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 -- 172.21.15.16:0/3711468098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 msgr2=0x562e2fca4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 0x562e2fca4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 -- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fca8370 msgr2=0x562e2fca54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fca8370 0x562e2fca54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 -- 172.21.15.16:0/3711468098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e2f9743c0 con 0x562e2fc9d5d0 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f4effd640 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f400ece50 tx=0x7f1f40005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dfff16d448742fd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f2effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40049040 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f400b8280 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f38009e10 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1f400bb750 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f57cd0740 10 client.26258 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.734+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562e2fcb6a90 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f38009e10 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1f400bbe70 con 0x562e2fc9d5d0 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 10 client.26258 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 10 client.26258 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 20 client.26258 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fcb7e80 0x562e2fcba340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 4 client.26258 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e2fc10860 con 0x562e2fcb7e80 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fcb7e80 0x562e2fcba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f2effd640 10 client.26258 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.735+0000 7f1f47fff640 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fcb7e80 0x562e2fcba340 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6fc378545515c0d0 server_cookie=42855c3378bc4c38 in_seq=0 out_seq=0 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f2effd640 1 -- 172.21.15.16:0/3711468098 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562e2fc10860 con 0x562e2fcb7e80 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f2effd640 10 client.26258 handle_command_reply: tid=0 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 client.26258 shutdown 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 msgr2=0x562e2fca7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f400ece50 tx=0x7f1f40005c90 comp rx=0 tx=0).stop 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 shutdown_connections 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fca8370 0x562e2fca54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9eb40 0x562e2fca4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fcb7e80 0x562e2fcba340 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/3711468098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fca7cb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f2effd640 0 client.26258 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:53:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 >> 172.21.15.16:0/3711468098 conn(0x562e2fc66310 msgr2=0x562e2fc9f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.736+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 shutdown_connections 2024-09-12T02:53:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.737+0000 7f1f57cd0740 1 -- 172.21.15.16:0/3711468098 wait complete. 2024-09-12T02:53:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.737+0000 7f1f57cd0740 20 client.26258 trim_cache size 0 max 16384 2024-09-12T02:53:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.754+0000 7f1f57cd0740 1 Processor -- start 2024-09-12T02:53:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.754+0000 7f1f57cd0740 1 -- start start 2024-09-12T02:53:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fc99e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fca8370 0x562e2fcbfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501af670 con 0x562e2fc9eb40 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501b11b0 con 0x562e2fc9d5d0 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501afd40 con 0x562e2fca8370 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49800/0 (socket says 172.21.15.16:49800) 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 -- 172.21.15.16:0/1967507356 learned_addr learned my addr 172.21.15.16:0/1967507356 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f4effd640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fc99e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f4f7fe640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fca8370 0x562e2fcbfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 -- 172.21.15.16:0/1967507356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fca8370 msgr2=0x562e2fcbfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fca8370 0x562e2fcbfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 -- 172.21.15.16:0/1967507356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 msgr2=0x562e2fc99e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fc99e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f47fff640 1 -- 172.21.15.16:0/1967507356 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2fca78d0 con 0x562e2fc9eb40 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.755+0000 7f1f4effd640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fc99e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.756+0000 7f1f47fff640 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1f380048f0 tx=0x7f1f38005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b2d876f67a3cea server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.756+0000 7f1f457fa640 1 -- 172.21.15.16:0/1967507356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3801e040 con 0x562e2fc9eb40 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.756+0000 7f1f457fa640 1 -- 172.21.15.16:0/1967507356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f38009e10 con 0x562e2fc9eb40 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.756+0000 7f1f457fa640 1 -- 172.21.15.16:0/1967507356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38003d30 con 0x562e2fc9eb40 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- 172.21.15.16:0/1967507356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 msgr2=0x562e2fcb4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1f380048f0 tx=0x7f1f38005f90 comp rx=0 tx=0).stop 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- 172.21.15.16:0/1967507356 shutdown_connections 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fca8370 0x562e2fcbfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcb4340 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/1967507356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fc9d5d0 0x562e2fc99e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- 172.21.15.16:0/1967507356 >> 172.21.15.16:0/1967507356 conn(0x562e2fc66310 msgr2=0x562e2fc99700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- 172.21.15.16:0/1967507356 shutdown_connections 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- 172.21.15.16:0/1967507356 wait complete. 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 -- start start 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.757+0000 7f1f57cd0740 1 Processor -- start 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fcb0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcabe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcac350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501b0ae0 con 0x562e2fc9eb40 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501b0410 con 0x562e2fca8370 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f501ad650 con 0x562e2fc9d5d0 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f57cd0740 10 client.0 _ll_register_callbacks cb 0x7f1f40005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fcb0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcabe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fcb0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59950/0 (socket says 172.21.15.16:59950) 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f4effd640 1 -- 172.21.15.16:0/2829090898 learned_addr learned my addr 172.21.15.16:0/2829090898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f4f7fe640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 -- 172.21.15.16:0/2829090898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 msgr2=0x562e2fcb0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fcb0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 -- 172.21.15.16:0/2829090898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 msgr2=0x562e2fcac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 -- 172.21.15.16:0/2829090898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e2fcac990 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f47fff640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcabe10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1f38004160 tx=0x7f1f38005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4c9cf7f679af03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f2effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.758+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38004290 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f38009e10 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f380035f0 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1f380037f0 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 10 client.16719 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562e2fd22a30 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f38009e10 con 0x562e2fc9eb40 2024-09-12T02:53:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1f38018070 con 0x562e2fc9eb40 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 10 client.16719 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 10 client.16719 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 20 client.16719 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fd22ea0 0x562e2fd25290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 4 client.16719 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e2fc0ad70 con 0x562e2fd22ea0 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.759+0000 7f1f4effd640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fd22ea0 0x562e2fd25290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f2effd640 10 client.16719 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f4effd640 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fd22ea0 0x562e2fd25290 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5f8bcda028d8093 server_cookie=298114fe8d0efaf0 in_seq=0 out_seq=0 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f2effd640 1 -- 172.21.15.16:0/2829090898 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e2fc0ad70 con 0x562e2fd22ea0 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f2effd640 10 client.16719 handle_command_reply: tid=0 2024-09-12T02:53:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 client.16719 shutdown 2024-09-12T02:53:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 msgr2=0x562e2fcabe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcabe10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1f38004160 tx=0x7f1f38005d30 comp rx=0 tx=0).stop 2024-09-12T02:53:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 shutdown_connections 2024-09-12T02:53:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562e2fd22ea0 0x562e2fd25290 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562e2fca8370 0x562e2fcac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562e2fc9eb40 0x562e2fcabe10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 --2- 172.21.15.16:0/2829090898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e2fc9d5d0 0x562e2fcb0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 >> 172.21.15.16:0/2829090898 conn(0x562e2fc66310 msgr2=0x562e2fc98d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f2effd640 0 client.16719 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 shutdown_connections 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.760+0000 7f1f57cd0740 1 -- 172.21.15.16:0/2829090898 wait complete. 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f57cd0740 20 client.16719 trim_cache size 0 max 16384 2024-09-12T02:53:27.768 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:53:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 msgr2=0x7f1f3c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 0x7f1f3c0a50c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1f48010610 tx=0x7f1f48004a40 comp rx=0 tx=0).stop 2024-09-12T02:53:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 msgr2=0x7f1f50101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1f40006010 tx=0x7f1f40004040 comp rx=0 tx=0).stop 2024-09-12T02:53:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 shutdown_connections 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f50108180 0x7f1f501022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f50101290 0x7f1f50101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f3c0a2c00 0x7f1f3c0a50c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 --2- 172.21.15.16:0/3894510121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f501008e0 0x7f1f50101860 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 >> 172.21.15.16:0/3894510121 conn(0x7f1f50077210 msgr2=0x7f1f50105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.761+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 shutdown_connections 2024-09-12T02:53:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:27.762+0000 7f1f558f7640 1 -- 172.21.15.16:0/3894510121 wait complete. 2024-09-12T02:53:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:28 smithi184 ceph-mon[35321]: pgmap v503: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 305 op/s 2024-09-12T02:53:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:28 smithi016 ceph-mon[30925]: pgmap v503: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 305 op/s 2024-09-12T02:53:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:28 smithi033 ceph-mon[35092]: pgmap v503: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 305 op/s 2024-09-12T02:53:28.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:53:28.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 526971 11.08 MB/sec execute 150 sec latency 13.646 ms 2024-09-12T02:53:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 -- 172.21.15.16:0/3496903202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 msgr2=0x7f31d01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/3496903202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d01029b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009ca0 tx=0x7f31c002a950 comp rx=0 tx=0).stop 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 -- 172.21.15.16:0/3496903202 shutdown_connections 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/3496903202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/3496903202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d01029b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/3496903202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 -- 172.21.15.16:0/3496903202 >> 172.21.15.16:0/3496903202 conn(0x7f31d00fc7d0 msgr2=0x7f31d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 -- 172.21.15.16:0/3496903202 shutdown_connections 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.375+0000 7f31d6ab5640 1 -- 172.21.15.16:0/3496903202 wait complete. 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 Processor -- start 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 -- start start 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0110fd0 con 0x7f31d0102570 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0110e50 con 0x7f31d0102ef0 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d6ab5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0111150 con 0x7f31d0100e20 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49818/0 (socket says 172.21.15.16:49818) 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d4851640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31cffff640 1 -- 172.21.15.16:0/2310692712 learned_addr learned my addr 172.21.15.16:0/2310692712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.376+0000 7f31d5052640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 -- 172.21.15.16:0/2310692712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 msgr2=0x7f31d00fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d00fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 -- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 msgr2=0x7f31d00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 -- 172.21.15.16:0/2310692712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d01a69e0 con 0x7f31d0100e20 2024-09-12T02:53:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31cffff640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d5052640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d00fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d4851640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d00fcc80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9f0 tx=0x7f31b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31cdffb640 1 -- 172.21.15.16:0/2310692712 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800ccc0 con 0x7f31d0100e20 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d01a6cd0 con 0x7f31d0100e20 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31cdffb640 1 -- 172.21.15.16:0/2310692712 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b8005020 con 0x7f31d0100e20 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31cdffb640 1 -- 172.21.15.16:0/2310692712 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b80049e0 con 0x7f31d0100e20 2024-09-12T02:53:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.377+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d01a71c0 con 0x7f31d0100e20 2024-09-12T02:53:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.379+0000 7f31cdffb640 1 -- 172.21.15.16:0/2310692712 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f31b8005560 con 0x7f31d0100e20 2024-09-12T02:53:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.380+0000 7f31d8e8e740 1 Processor -- start 2024-09-12T02:53:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.380+0000 7f31d8e8e740 1 -- start start 2024-09-12T02:53:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.380+0000 7f31cdffb640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 0x7f31bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d14a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 0x558dc7d21760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558dc7c7a2f0 con 0x558dc7d15b40 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558dc7c21cf0 con 0x558dc7d1f370 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558dc7c762b0 con 0x558dc7d145d0 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31cdffb640 1 -- 172.21.15.16:0/2310692712 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31b80ceff0 con 0x7f31d0100e20 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49834/0 (socket says 172.21.15.16:49834) 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 -- 172.21.15.16:0/3152201041 learned_addr learned my addr 172.21.15.16:0/3152201041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31cffff640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d14a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d5052640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 0x558dc7d21760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d5052640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 0x7f31bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 -- 172.21.15.16:0/3152201041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 msgr2=0x558dc7d14a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d14a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 -- 172.21.15.16:0/3152201041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 msgr2=0x558dc7d21760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 0x558dc7d21760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d4851640 1 -- 172.21.15.16:0/3152201041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dc7c063a0 con 0x558dc7d15b40 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.381+0000 7f31d5052640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 0x558dc7d21760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.382+0000 7f31d4851640 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f31b809b7b0 tx=0x7f31b8005290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f4311a59b727b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.382+0000 7f31d5052640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 0x7f31bc0a5040 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f31c80058c0 tx=0x7f31c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.382+0000 7f31c6ffd640 1 -- 172.21.15.16:0/3152201041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b801c040 con 0x558dc7d15b40 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.382+0000 7f31c6ffd640 1 -- 172.21.15.16:0/3152201041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b809fbd0 con 0x558dc7d15b40 2024-09-12T02:53:29.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.382+0000 7f31c6ffd640 1 -- 172.21.15.16:0/3152201041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8086070 con 0x558dc7d15b40 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3152201041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 msgr2=0x558dc7d14090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f31b809b7b0 tx=0x7f31b8005290 comp rx=0 tx=0).stop 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3152201041 shutdown_connections 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d1f370 0x558dc7d21760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d14a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3152201041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d14090 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3152201041 >> 172.21.15.16:0/3152201041 conn(0x558dc7cdd310 msgr2=0x558dc7d10c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3152201041 shutdown_connections 2024-09-12T02:53:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3152201041 wait complete. 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 -- start start 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.383+0000 7f31d8e8e740 1 Processor -- start 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d1c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d1c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558dc7d22070 con 0x558dc7d15b40 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558dc7c35060 con 0x558dc7d145d0 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558dc7d221f0 con 0x558dc7d1f370 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d8e8e740 10 client.0 _ll_register_callbacks cb 0x558dc79888c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57920/0 (socket says 172.21.15.16:57920) 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 -- 172.21.15.16:0/2546706995 learned_addr learned my addr 172.21.15.16:0/2546706995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31cffff640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d1c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d5052640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d1c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 -- 172.21.15.16:0/2546706995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 msgr2=0x558dc7d1c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d1c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 -- 172.21.15.16:0/2546706995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 msgr2=0x558dc7d1c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d1c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 -- 172.21.15.16:0/2546706995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dc79eb3c0 con 0x558dc7d145d0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31cffff640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d1c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31d4851640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c6a0 tx=0x7f31b80a4490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99921271a3ac1406 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.384+0000 7f31affff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b801c040 con 0x558dc7d145d0 2024-09-12T02:53:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31b80a4000 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b809fbd0 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f31b80a4200 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 10 client.26276 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558dc7d2db10 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b809fbd0 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f31b809ac70 con 0x558dc7d145d0 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 10 client.26276 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 10 client.26276 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 20 client.26276 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d31250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 4 client.26276 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558dc7c87860 con 0x558dc7d2ee60 2024-09-12T02:53:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.385+0000 7f31cffff640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d31250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.386+0000 7f31cffff640 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d31250 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=680a747d6ca397be server_cookie=95359faf92395bd3 in_seq=0 out_seq=0 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.386+0000 7f31affff640 10 client.26276 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.386+0000 7f31affff640 1 -- 172.21.15.16:0/2546706995 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558dc7c87860 con 0x558dc7d2ee60 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.386+0000 7f31affff640 10 client.26276 handle_command_reply: tid=0 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.386+0000 7f31d8e8e740 1 client.26276 shutdown 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 msgr2=0x558dc7d1ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c6a0 tx=0x7f31b80a4490 comp rx=0 tx=0).stop 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 shutdown_connections 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d31250 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31affff640 0 client.26276 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d1c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d15b40 0x558dc7d1c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/2546706995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d145d0 0x558dc7d1ed60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 >> 172.21.15.16:0/2546706995 conn(0x558dc7cdd310 msgr2=0x558dc7d16990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 shutdown_connections 2024-09-12T02:53:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 1 -- 172.21.15.16:0/2546706995 wait complete. 2024-09-12T02:53:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.387+0000 7f31d8e8e740 20 client.26276 trim_cache size 0 max 16384 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 Processor -- start 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 -- start start 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d1b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d17120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01af810 con 0x558dc7d1f370 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01b1350 con 0x558dc7d15b40 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01afee0 con 0x558dc7d145d0 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d5052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d4851640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d1b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d5052640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49856/0 (socket says 172.21.15.16:49856) 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31d5052640 1 -- 172.21.15.16:0/3210544740 learned_addr learned my addr 172.21.15.16:0/3210544740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.401+0000 7f31cffff640 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d17120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 -- 172.21.15.16:0/3210544740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 msgr2=0x558dc7d1b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d1b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 -- 172.21.15.16:0/3210544740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 msgr2=0x558dc7d17120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d17120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 -- 172.21.15.16:0/3210544740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dc7d11960 con 0x558dc7d1f370 2024-09-12T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d4851640 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d1b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31d5052640 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f31c80109e0 tx=0x7f31c8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184e008b70d3e568 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31c67fc640 1 -- 172.21.15.16:0/3210544740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8020920 con 0x558dc7d1f370 2024-09-12T02:53:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31c67fc640 1 -- 172.21.15.16:0/3210544740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31c8005020 con 0x558dc7d1f370 2024-09-12T02:53:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.402+0000 7f31c67fc640 1 -- 172.21.15.16:0/3210544740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8004b60 con 0x558dc7d1f370 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3210544740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 msgr2=0x558dc7d17660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f31c80109e0 tx=0x7f31c8005850 comp rx=0 tx=0).stop 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3210544740 shutdown_connections 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d1f370 0x558dc7d17660 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d17120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/3210544740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d145d0 0x558dc7d1b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3210544740 >> 172.21.15.16:0/3210544740 conn(0x558dc7cdd310 msgr2=0x558dc7d1b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3210544740 shutdown_connections 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- 172.21.15.16:0/3210544740 wait complete. 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 -- start start 2024-09-12T02:53:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.403+0000 7f31d8e8e740 1 Processor -- start 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 0x558dc7d316d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d97790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01b0c80 con 0x558dc7d145d0 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01b05b0 con 0x558dc7d15b40 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01ad7f0 con 0x558dc7d1f370 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d8e8e740 10 client.0 _ll_register_callbacks cb 0x7f31b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d4851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 0x558dc7d316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d5052640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d97790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57936/0 (socket says 172.21.15.16:57936) 2024-09-12T02:53:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d5052640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d97790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59980/0 (socket says 172.21.15.16:59980) 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 -- 172.21.15.16:0/762763319 learned_addr learned my addr 172.21.15.16:0/762763319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31d4851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 0x558dc7d316d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49870/0 (socket says 172.21.15.16:49870) 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 -- 172.21.15.16:0/762763319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 msgr2=0x558dc7d97790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d97790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 -- 172.21.15.16:0/762763319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 msgr2=0x558dc7d316d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 0x558dc7d316d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.404+0000 7f31cffff640 1 -- 172.21.15.16:0/762763319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dc7d97cd0 con 0x558dc7d15b40 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31cffff640 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009620 tx=0x7f31c002fb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=424cae07e7396f45 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31affff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0034040 con 0x558dc7d15b40 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31c0047430 con 0x558dc7d15b40 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31c8005020 con 0x558dc7d15b40 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f31c002fed0 con 0x558dc7d15b40 2024-09-12T02:53:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31d8e8e740 10 client.26282 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.405+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558dc7d99000 con 0x558dc7d15b40 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31c8005020 con 0x558dc7d15b40 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f31c004fa60 con 0x558dc7d15b40 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 10 client.26282 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 10 client.26282 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 20 client.26282 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d9b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 4 client.26282 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558dc7c81d70 con 0x558dc7d2ee60 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.406+0000 7f31d4851640 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d9b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31affff640 10 client.26282 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d4851640 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d9b7f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1c7a7ced69560404 server_cookie=bb73f6a406be04e0 in_seq=0 out_seq=0 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31affff640 1 -- 172.21.15.16:0/762763319 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558dc7c81d70 con 0x558dc7d2ee60 2024-09-12T02:53:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31affff640 10 client.26282 handle_command_reply: tid=0 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 client.26282 shutdown 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 msgr2=0x558dc7d31c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009620 tx=0x7f31c002fb10 comp rx=0 tx=0).stop 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 shutdown_connections 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558dc7d2ee60 0x558dc7d9b7f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558dc7d1f370 0x558dc7d97790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558dc7d15b40 0x558dc7d31c10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 --2- 172.21.15.16:0/762763319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558dc7d145d0 0x558dc7d316d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31affff640 0 client.26282 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 >> 172.21.15.16:0/762763319 conn(0x558dc7cdd310 msgr2=0x558dc7d1b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 shutdown_connections 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 1 -- 172.21.15.16:0/762763319 wait complete. 2024-09-12T02:53:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.407+0000 7f31d8e8e740 20 client.26282 trim_cache size 0 max 16384 2024-09-12T02:53:29.413 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:53:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 msgr2=0x7f31bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 0x7f31bc0a5040 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f31c80058c0 tx=0x7f31c8005f90 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 msgr2=0x7f31d00fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d00fcc80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9f0 tx=0x7f31b800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 shutdown_connections 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31d0102ef0 0x7f31d00fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31bc0a2b80 0x7f31bc0a5040 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0102570 0x7f31d00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 --2- 172.21.15.16:0/2310692712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31d0100e20 0x7f31d00fcc80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 >> 172.21.15.16:0/2310692712 conn(0x7f31d00fc7d0 msgr2=0x7f31d006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 shutdown_connections 2024-09-12T02:53:29.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:29.408+0000 7f31d6ab5640 1 -- 172.21.15.16:0/2310692712 wait complete. 2024-09-12T02:53:29.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530096 11.13 MB/sec execute 151 sec latency 19.540 ms 2024-09-12T02:53:30.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:30 smithi016 ceph-mon[30925]: pgmap v504: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 3.4 MiB/s wr, 231 op/s 2024-09-12T02:53:30.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:53:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:30 smithi184 ceph-mon[35321]: pgmap v504: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 3.4 MiB/s wr, 231 op/s 2024-09-12T02:53:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:30 smithi033 ceph-mon[35092]: pgmap v504: 97 pgs: 97 active+clean; 187 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 3.4 MiB/s wr, 231 op/s 2024-09-12T02:53:30.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 533255 11.18 MB/sec execute 152 sec latency 49.791 ms 2024-09-12T02:53:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 -- 172.21.15.16:0/3316651357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670100aa0 msgr2=0x7f067010ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 --2- 172.21.15.16:0/3316651357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670100aa0 0x7f067010ea20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0668009ca0 tx=0x7f066802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 -- 172.21.15.16:0/3316651357 shutdown_connections 2024-09-12T02:53:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 --2- 172.21.15.16:0/3316651357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670100aa0 0x7f067010ea20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 --2- 172.21.15.16:0/3316651357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f0670100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 --2- 172.21.15.16:0/3316651357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670106b80 0x7f0670106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 -- 172.21.15.16:0/3316651357 >> 172.21.15.16:0/3316651357 conn(0x7f0670077230 msgr2=0x7f06700fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 -- 172.21.15.16:0/3316651357 shutdown_connections 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.005+0000 7f067592c640 1 -- 172.21.15.16:0/3316651357 wait complete. 2024-09-12T02:53:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 Processor -- start 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 -- start start 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f06701983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670100aa0 0x7f0670198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0670112050 con 0x7f0670106b80 2024-09-12T02:53:31.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0670111ed0 con 0x7f0670100aa0 2024-09-12T02:53:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f067592c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06701121d0 con 0x7f0670100100 2024-09-12T02:53:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49882/0 (socket says 172.21.15.16:49882) 2024-09-12T02:53:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f066f7fe640 1 -- 172.21.15.16:0/1890857937 learned_addr learned my addr 172.21.15.16:0/1890857937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.006+0000 7f066effd640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f06701983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066e7fc640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670100aa0 0x7f0670198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 -- 172.21.15.16:0/1890857937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 msgr2=0x7f06701983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f06701983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 -- 172.21.15.16:0/1890857937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670100aa0 msgr2=0x7f0670198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670100aa0 0x7f0670198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 -- 172.21.15.16:0/1890857937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067019d3a0 con 0x7f0670106b80 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066effd640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f06701983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f066f7fe640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f06680319f0 tx=0x7f0668031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f067492a640 1 -- 172.21.15.16:0/1890857937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668003b70 con 0x7f0670106b80 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f067492a640 1 -- 172.21.15.16:0/1890857937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0668003d10 con 0x7f0670106b80 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067019d630 con 0x7f0670106b80 2024-09-12T02:53:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f067492a640 1 -- 172.21.15.16:0/1890857937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066802fc40 con 0x7f0670106b80 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.007+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06701a4f70 con 0x7f0670106b80 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.009+0000 7f067492a640 1 -- 172.21.15.16:0/1890857937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0668004050 con 0x7f0670106b80 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 Processor -- start 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 -- start start 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f067492a640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 0x7f06580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x56348946da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x56348947eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5634893d32f0 con 0x56348946d5d0 2024-09-12T02:53:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56348937acf0 con 0x56348946eb40 2024-09-12T02:53:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f0677d05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5634893cf2b0 con 0x56348947c780 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f067492a640 1 -- 172.21.15.16:0/1890857937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f06680f0240 con 0x7f0670106b80 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.010+0000 7f066e7fc640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 0x7f06580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57962/0 (socket says 172.21.15.16:57962) 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 -- 172.21.15.16:0/1756564674 learned_addr learned my addr 172.21.15.16:0/1756564674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066f7fe640 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x56348947eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066e7fc640 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x56348946da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 -- 172.21.15.16:0/1756564674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 msgr2=0x56348947eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x56348947eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 -- 172.21.15.16:0/1756564674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 msgr2=0x56348946da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x56348946da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 -- 172.21.15.16:0/1756564674 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56348935f3a0 con 0x56348946eb40 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066e7fc640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 0x7f06580a5040 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fd0 tx=0x7f0664009b80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f066effd640 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f066000f4c0 tx=0x7f066000f980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a817ff52deb635f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f0655ffb640 1 -- 172.21.15.16:0/1756564674 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660005220 con 0x56348946eb40 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f0655ffb640 1 -- 172.21.15.16:0/1756564674 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0660004c80 con 0x56348946eb40 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.011+0000 7f0655ffb640 1 -- 172.21.15.16:0/1756564674 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660010b30 con 0x56348946eb40 2024-09-12T02:53:31.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- 172.21.15.16:0/1756564674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 msgr2=0x56348946d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f066000f4c0 tx=0x7f066000f980 comp rx=0 tx=0).stop 2024-09-12T02:53:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- 172.21.15.16:0/1756564674 shutdown_connections 2024-09-12T02:53:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x56348947eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x56348946da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 --2- 172.21.15.16:0/1756564674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x56348946d090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- 172.21.15.16:0/1756564674 >> 172.21.15.16:0/1756564674 conn(0x563489436310 msgr2=0x563489469c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- 172.21.15.16:0/1756564674 shutdown_connections 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- 172.21.15.16:0/1756564674 wait complete. 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.012+0000 7f0677d05740 1 -- start start 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 Processor -- start 2024-09-12T02:53:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348946d5d0 0x56348947c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489479380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56348947f480 con 0x56348947c780 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56348938e060 con 0x56348946eb40 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56348947f600 con 0x56348946d5d0 2024-09-12T02:53:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f0677d05740 10 client.0 _ll_register_callbacks cb 0x5634890e18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:31.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348946d5d0 0x56348947c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49894/0 (socket says 172.21.15.16:49894) 2024-09-12T02:53:31.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 -- 172.21.15.16:0/4172502457 learned_addr learned my addr 172.21.15.16:0/4172502457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:31.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066e7fc640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489479380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 -- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348946d5d0 msgr2=0x56348947c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348946d5d0 0x56348947c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 -- 172.21.15.16:0/4172502457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 msgr2=0x563489479380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.013+0000 7f066f7fe640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489479380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f066f7fe640 1 -- 172.21.15.16:0/4172502457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634891443c0 con 0x56348947c780 2024-09-12T02:53:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f066f7fe640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0668006010 tx=0x7f06680bc6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f3a37f4d1b01d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680bcab0 con 0x56348947c780 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0668004730 con 0x56348947c780 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0660004c80 con 0x56348947c780 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f06680c0e60 con 0x56348947c780 2024-09-12T02:53:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f0677d05740 10 client.16749 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56348947f480 con 0x56348947c780 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0660004c80 con 0x56348947c780 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.014+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f06680db330 con 0x56348947c780 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 10 client.16749 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 10 client.16749 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:53:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 20 client.16749 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56348948c290 0x56348948e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 4 client.16749 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634893e0860 con 0x56348948c290 2024-09-12T02:53:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f066effd640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56348948c290 0x56348948e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f066effd640 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56348948c290 0x56348948e6f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35fee7490c09f0c9 server_cookie=60ad1c70079933c9 in_seq=0 out_seq=0 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.015+0000 7f064effd640 10 client.16749 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.016+0000 7f064effd640 1 -- 172.21.15.16:0/4172502457 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5634893e0860 con 0x56348948c290 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.016+0000 7f064effd640 10 client.16749 handle_command_reply: tid=0 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 client.16749 shutdown 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 msgr2=0x5634894798e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0668006010 tx=0x7f06680bc6b0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 shutdown_connections 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348947c780 0x5634894798e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489479380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56348948c290 0x56348948e6f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 --2- 172.21.15.16:0/4172502457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348946d5d0 0x56348947c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 >> 172.21.15.16:0/4172502457 conn(0x563489436310 msgr2=0x56348946f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f064effd640 0 client.16749 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:53:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 shutdown_connections 2024-09-12T02:53:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 1 -- 172.21.15.16:0/4172502457 wait complete. 2024-09-12T02:53:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.017+0000 7f0677d05740 20 client.16749 trim_cache size 0 max 16384 2024-09-12T02:53:31.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.039+0000 7f0677d05740 1 Processor -- start 2024-09-12T02:53:31.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.039+0000 7f0677d05740 1 -- start start 2024-09-12T02:53:31.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.039+0000 7f0677d05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x563489488720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f0677d05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489494000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f0677d05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f0677d05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06701ab150 con 0x56348946d5d0 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f0677d05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06701acc90 con 0x56348946eb40 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f0677d05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06701ab820 con 0x56348947c780 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60010/0 (socket says 172.21.15.16:60010) 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 -- 172.21.15.16:0/673856560 learned_addr learned my addr 172.21.15.16:0/673856560 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066effd640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x563489488720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066e7fc640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489494000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 -- 172.21.15.16:0/673856560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 msgr2=0x563489494000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489494000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 -- 172.21.15.16:0/673856560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 msgr2=0x563489488720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x563489488720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.040+0000 7f066f7fe640 1 -- 172.21.15.16:0/673856560 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56348947bc80 con 0x56348947c780 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.041+0000 7f066effd640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x563489488720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.041+0000 7f066f7fe640 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06680bca40 tx=0x7f06680bafc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5eb678f2c0d77d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.041+0000 7f06557fa640 1 -- 172.21.15.16:0/673856560 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680bcab0 con 0x56348947c780 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.041+0000 7f06557fa640 1 -- 172.21.15.16:0/673856560 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0668036060 con 0x56348947c780 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.041+0000 7f06557fa640 1 -- 172.21.15.16:0/673856560 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680db2a0 con 0x56348947c780 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 -- 172.21.15.16:0/673856560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 msgr2=0x563489494540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06680bca40 tx=0x7f06680bafc0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 -- 172.21.15.16:0/673856560 shutdown_connections 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x563489494540 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489494000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 --2- 172.21.15.16:0/673856560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x563489488720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 -- 172.21.15.16:0/673856560 >> 172.21.15.16:0/673856560 conn(0x563489436310 msgr2=0x563489479340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 -- 172.21.15.16:0/673856560 shutdown_connections 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.042+0000 7f0677d05740 1 -- 172.21.15.16:0/673856560 wait complete. 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.043+0000 7f0677d05740 1 -- start start 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.043+0000 7f0677d05740 1 Processor -- start 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x5634894809d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x5634894f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06701ac5c0 con 0x56348946d5d0 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06701abef0 con 0x56348946eb40 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06701a9150 con 0x56348947c780 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57988/0 (socket says 172.21.15.16:57988) 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 -- 172.21.15.16:0/2811415914 learned_addr learned my addr 172.21.15.16:0/2811415914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066effd640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x5634894809d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f0677d05740 10 client.0 _ll_register_callbacks cb 0x7f0668009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 -- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 msgr2=0x5634894f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066f7fe640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x5634894f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x5634894f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 -- 172.21.15.16:0/2811415914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 msgr2=0x5634894809d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:31.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x5634894809d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.044+0000 7f066e7fc640 1 -- 172.21.15.16:0/2811415914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563489481810 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f066effd640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x5634894809d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f066f7fe640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x5634894f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f066e7fc640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0664009710 tx=0x7f0664005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564a430671c5127d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f064effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664026040 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f066401a070 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.045+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0668036060 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0664009780 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 10 client.26297 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5634894f7610 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0668036060 con 0x56348946eb40 2024-09-12T02:53:31.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0664029070 con 0x56348946eb40 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 10 client.26297 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 10 client.26297 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 20 client.26297 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5634894f7e40 0x5634894fa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 4 client.26297 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.046+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634893dad70 con 0x5634894f7e40 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.047+0000 7f066effd640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5634894f7e40 0x5634894fa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.047+0000 7f066effd640 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5634894f7e40 0x5634894fa260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa18f79791edeaf7 server_cookie=cee2f2e46fe88841 in_seq=0 out_seq=0 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.047+0000 7f064effd640 10 client.26297 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f064effd640 1 -- 172.21.15.16:0/2811415914 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634893dad70 con 0x5634894f7e40 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f064effd640 10 client.26297 handle_command_reply: tid=0 2024-09-12T02:53:31.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 client.26297 shutdown 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 msgr2=0x563489480f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0664009710 tx=0x7f0664005f90 comp rx=0 tx=0).stop 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 shutdown_connections 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5634894f7e40 0x5634894fa260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56348947c780 0x5634894f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f064effd640 0 client.26297 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56348946eb40 0x563489480f10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 --2- 172.21.15.16:0/2811415914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56348946d5d0 0x5634894809d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 >> 172.21.15.16:0/2811415914 conn(0x563489436310 msgr2=0x5634894684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 shutdown_connections 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.050+0000 7f0677d05740 1 -- 172.21.15.16:0/2811415914 wait complete. 2024-09-12T02:53:31.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.051+0000 7f0677d05740 20 client.26297 trim_cache size 0 max 16384 2024-09-12T02:53:31.063 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:53:31.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 msgr2=0x7f06580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 0x7f06580a5040 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fd0 tx=0x7f0664009b80 comp rx=0 tx=0).stop 2024-09-12T02:53:31.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 msgr2=0x7f067019ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:31.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f06680319f0 tx=0x7f0668031a20 comp rx=0 tx=0).stop 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 shutdown_connections 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f06580a2b80 0x7f06580a5040 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0670106b80 0x7f067019ccc0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0670100aa0 0x7f0670198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 --2- 172.21.15.16:0/1890857937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0670100100 0x7f06701983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 >> 172.21.15.16:0/1890857937 conn(0x7f0670077230 msgr2=0x7f0670104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 shutdown_connections 2024-09-12T02:53:31.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:31.052+0000 7f067592c640 1 -- 172.21.15.16:0/1890857937 wait complete. 2024-09-12T02:53:31.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 536262 11.20 MB/sec execute 153 sec latency 21.607 ms 2024-09-12T02:53:31.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:53:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:32 smithi016 ceph-mon[30925]: pgmap v505: 97 pgs: 97 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-09-12T02:53:32.173 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:32 smithi184 ceph-mon[35321]: pgmap v505: 97 pgs: 97 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-09-12T02:53:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:32 smithi033 ceph-mon[35092]: pgmap v505: 97 pgs: 97 active+clean; 192 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.691+0000 7f6ca3da7640 1 -- 172.21.15.16:0/2787469019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100350 msgr2=0x7f6c9c100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.691+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/2787469019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100350 0x7f6c9c100730 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009930 tx=0x7f6c9802a420 comp rx=0 tx=0).stop 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.691+0000 7f6ca3da7640 1 -- 172.21.15.16:0/2787469019 shutdown_connections 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.691+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/2787469019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c106f50 0x7f6c9c10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/2787469019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c100d00 0x7f6c9c106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/2787469019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100350 0x7f6c9c100730 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 -- 172.21.15.16:0/2787469019 >> 172.21.15.16:0/2787469019 conn(0x7f6c9c077210 msgr2=0x7f6c9c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 -- 172.21.15.16:0/2787469019 shutdown_connections 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 -- 172.21.15.16:0/2787469019 wait complete. 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 Processor -- start 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.692+0000 7f6ca3da7640 1 -- start start 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 0x7f6c9c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c106f50 0x7f6c9c101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c111fa0 con 0x7f6c9c100d00 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c111e20 con 0x7f6c9c106f50 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c112120 con 0x7f6c9c100350 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 0x7f6c9c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 0x7f6c9c1012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60042/0 (socket says 172.21.15.16:60042) 2024-09-12T02:53:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49928/0 (socket says 172.21.15.16:49928) 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1b43640 1 -- 172.21.15.16:0/3600334225 learned_addr learned my addr 172.21.15.16:0/3600334225 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca2344640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c106f50 0x7f6c9c101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 -- 172.21.15.16:0/3600334225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 msgr2=0x7f6c9c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 0x7f6c9c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 -- 172.21.15.16:0/3600334225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c106f50 msgr2=0x7f6c9c101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c106f50 0x7f6c9c101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 -- 172.21.15.16:0/3600334225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c1a6900 con 0x7f6c9c100d00 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca1342640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c940 tx=0x7f6c8c00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6c92ffd640 1 -- 172.21.15.16:0/3600334225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c007bb0 con 0x7f6c9c100d00 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.693+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c1a6bf0 con 0x7f6c9c100d00 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.694+0000 7f6c92ffd640 1 -- 172.21.15.16:0/3600334225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c007d50 con 0x7f6c9c100d00 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.694+0000 7f6c92ffd640 1 -- 172.21.15.16:0/3600334225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c018760 con 0x7f6c9c100d00 2024-09-12T02:53:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.694+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c1a7180 con 0x7f6c9c100d00 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.696+0000 7f6c92ffd640 1 -- 172.21.15.16:0/3600334225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6c8c00f450 con 0x7f6c9c100d00 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.696+0000 7f6ca6180740 1 Processor -- start 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 -- start start 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b33090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b40760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561a22a992f0 con 0x561a22b34b40 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a22a40cf0 con 0x561a22b3e370 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca6180740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561a22a952b0 con 0x561a22b335d0 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6c92ffd640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 0x7f6c880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca1342640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b40760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60054/0 (socket says 172.21.15.16:60054) 2024-09-12T02:53:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca2344640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b33090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca1b43640 1 -- 172.21.15.16:0/1208557012 learned_addr learned my addr 172.21.15.16:0/1208557012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6ca2344640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 0x7f6c880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.697+0000 7f6c92ffd640 1 -- 172.21.15.16:0/3600334225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6c8c0cf060 con 0x7f6c9c100d00 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 -- 172.21.15.16:0/1208557012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 msgr2=0x561a22b40760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b40760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 -- 172.21.15.16:0/1208557012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 msgr2=0x561a22b33090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b33090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 -- 172.21.15.16:0/1208557012 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a22a253a0 con 0x561a22b335d0 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1342640 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b40760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca2344640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 0x7f6c880a5040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c94009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006010 tx=0x7f6c98009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fe8d66c5f09f06f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6c87fff640 1 -- 172.21.15.16:0/1208557012 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98040070 con 0x561a22b335d0 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6c87fff640 1 -- 172.21.15.16:0/1208557012 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98031990 con 0x561a22b335d0 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.698+0000 7f6c87fff640 1 -- 172.21.15.16:0/1208557012 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98035040 con 0x561a22b335d0 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 -- 172.21.15.16:0/1208557012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 msgr2=0x561a22b33a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006010 tx=0x7f6c98009680 comp rx=0 tx=0).stop 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 -- 172.21.15.16:0/1208557012 shutdown_connections 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b40760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b335d0 0x561a22b33a10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 --2- 172.21.15.16:0/1208557012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b33090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.699+0000 7f6ca6180740 1 -- 172.21.15.16:0/1208557012 >> 172.21.15.16:0/1208557012 conn(0x561a22afc310 msgr2=0x561a22b2fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.700+0000 7f6ca6180740 1 -- 172.21.15.16:0/1208557012 shutdown_connections 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.700+0000 7f6ca6180740 1 -- 172.21.15.16:0/1208557012 wait complete. 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.700+0000 7f6ca6180740 1 -- start start 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.700+0000 7f6ca6180740 1 Processor -- start 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b3af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b3e370 0x561a22b3b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561a22b41010 con 0x561a22b34b40 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a22a54060 con 0x561a22b335d0 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561a22b41190 con 0x561a22b3e370 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca6180740 10 client.0 _ll_register_callbacks cb 0x561a227a78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58034/0 (socket says 172.21.15.16:58034) 2024-09-12T02:53:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca1b43640 1 -- 172.21.15.16:0/4052888806 learned_addr learned my addr 172.21.15.16:0/4052888806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca1342640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b3af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca2344640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b3e370 0x561a22b3b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.701+0000 7f6ca1b43640 1 -- 172.21.15.16:0/4052888806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b3e370 msgr2=0x561a22b3b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b3e370 0x561a22b3b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1b43640 1 -- 172.21.15.16:0/4052888806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 msgr2=0x561a22b3af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b3af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1b43640 1 -- 172.21.15.16:0/4052888806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a2280a3c0 con 0x561a22b335d0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1342640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b3af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009a50 tx=0x7f6c98005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2666a45da039a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6c857fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98040030 con 0x561a22b335d0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6c98004260 con 0x561a22b335d0 2024-09-12T02:53:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c98031990 con 0x561a22b335d0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c98035040 con 0x561a22b335d0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca6180740 10 client.26306 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561a22b4ca20 con 0x561a22b335d0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.702+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98031990 con 0x561a22b335d0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c980038f0 con 0x561a22b335d0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 10 client.26306 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 10 client.26306 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 20 client.26306 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22b4dd70 0x561a22b50160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 4 client.26306 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a22aa6860 con 0x561a22b4dd70 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca1342640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22b4dd70 0x561a22b50160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6c857fa640 10 client.26306 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.703+0000 7f6ca1342640 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22b4dd70 0x561a22b50160 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54a0ad49af2052cc server_cookie=8b9853354c9e870d in_seq=0 out_seq=0 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.704+0000 7f6c857fa640 1 -- 172.21.15.16:0/4052888806 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561a22aa6860 con 0x561a22b4dd70 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.704+0000 7f6c857fa640 10 client.26306 handle_command_reply: tid=0 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.704+0000 7f6ca6180740 1 client.26306 shutdown 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.704+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 msgr2=0x561a22b3dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.704+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009a50 tx=0x7f6c98005250 comp rx=0 tx=0).stop 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 shutdown_connections 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22b4dd70 0x561a22b50160 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b3e370 0x561a22b3b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b34b40 0x561a22b3af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 --2- 172.21.15.16:0/4052888806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b3dce0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6c857fa640 0 client.26306 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:53:32.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 >> 172.21.15.16:0/4052888806 conn(0x561a22afc310 msgr2=0x561a22b35960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 shutdown_connections 2024-09-12T02:53:32.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 1 -- 172.21.15.16:0/4052888806 wait complete. 2024-09-12T02:53:32.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.705+0000 7f6ca6180740 20 client.26306 trim_cache size 0 max 16384 2024-09-12T02:53:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 Processor -- start 2024-09-12T02:53:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 -- start start 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 0x561a22b34410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b2aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b36120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1af6c0 con 0x561a22b335d0 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1b1200 con 0x561a22b3e370 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca6180740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1afd90 con 0x561a22b34b40 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 0x561a22b34410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 0x561a22b34410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49956/0 (socket says 172.21.15.16:49956) 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca1342640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b2aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca1b43640 1 -- 172.21.15.16:0/2108730989 learned_addr learned my addr 172.21.15.16:0/2108730989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:32.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.719+0000 7f6ca2344640 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b36120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 -- 172.21.15.16:0/2108730989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 msgr2=0x561a22b36120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b36120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 -- 172.21.15.16:0/2108730989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 msgr2=0x561a22b34410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 538988 11.21 MB/sec execute 154 sec latency 16.128 ms 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 0x561a22b34410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 -- 172.21.15.16:0/2108730989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a22b394e0 con 0x561a22b34b40 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca2344640 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b36120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b2aad0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ccb10 tx=0x7f6c8c0d1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded321a873446942 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6c877fe640 1 -- 172.21.15.16:0/2108730989 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c09fc20 con 0x561a22b34b40 2024-09-12T02:53:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6c877fe640 1 -- 172.21.15.16:0/2108730989 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c0a44e0 con 0x561a22b34b40 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.720+0000 7f6c877fe640 1 -- 172.21.15.16:0/2108730989 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c00fb70 con 0x561a22b34b40 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- 172.21.15.16:0/2108730989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 msgr2=0x561a22b2aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b2aad0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ccb10 tx=0x7f6c8c0d1040 comp rx=0 tx=0).stop 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- 172.21.15.16:0/2108730989 shutdown_connections 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b3e370 0x561a22b36120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b2aad0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2108730989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b335d0 0x561a22b34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- 172.21.15.16:0/2108730989 >> 172.21.15.16:0/2108730989 conn(0x561a22afc310 msgr2=0x561a22b3a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- 172.21.15.16:0/2108730989 shutdown_connections 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- 172.21.15.16:0/2108730989 wait complete. 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 -- start start 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.721+0000 7f6ca6180740 1 Processor -- start 2024-09-12T02:53:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b507c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b3e370 0x561a22bb66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1b0b30 con 0x561a22b3e370 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1b0460 con 0x561a22b335d0 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1ad6a0 con 0x561a22b34b40 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca6180740 10 client.0 _ll_register_callbacks cb 0x7f6c94009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca1b43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58044/0 (socket says 172.21.15.16:58044) 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca1b43640 1 -- 172.21.15.16:0/2216023242 learned_addr learned my addr 172.21.15.16:0/2216023242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca2344640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b3e370 0x561a22bb66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.722+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 -- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 msgr2=0x561a22b50280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 -- 172.21.15.16:0/2216023242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b3e370 msgr2=0x561a22bb66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b3e370 0x561a22bb66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 -- 172.21.15.16:0/2216023242 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a22b510a0 con 0x561a22b34b40 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca1342640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b507c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ccfd0 tx=0x7f6c8c00fe50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf3a67942a1bdba server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0b4060 con 0x561a22b34b40 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c0a44e0 con 0x561a22b34b40 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6c8c0ccc20 con 0x561a22b34b40 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c8c081050 con 0x561a22b34b40 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca6180740 10 client.26215 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561a22bb7cf0 con 0x561a22b34b40 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c0a44e0 con 0x561a22b34b40 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.723+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c8c097ee0 con 0x561a22b34b40 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 10 client.26215 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 10 client.26215 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 20 client.26215 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22bb83a0 0x561a22bba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 4 client.26215 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a22aa0d70 con 0x561a22bb83a0 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22bb83a0 0x561a22bba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6c857fa640 10 client.26215 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca1b43640 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22bb83a0 0x561a22bba7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6536757e9fdf6221 server_cookie=42e68f95d2204cef in_seq=0 out_seq=0 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6c857fa640 1 -- 172.21.15.16:0/2216023242 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a22aa0d70 con 0x561a22bb83a0 2024-09-12T02:53:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6c857fa640 10 client.26215 handle_command_reply: tid=0 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.724+0000 7f6ca6180740 1 client.26215 shutdown 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 msgr2=0x561a22b507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b507c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ccfd0 tx=0x7f6c8c00fe50 comp rx=0 tx=0).stop 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 shutdown_connections 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561a22bb83a0 0x561a22bba7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a22b3e370 0x561a22bb66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6c857fa640 0 client.26215 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561a22b34b40 0x561a22b507c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 --2- 172.21.15.16:0/2216023242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a22b335d0 0x561a22b50280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 >> 172.21.15.16:0/2216023242 conn(0x561a22afc310 msgr2=0x561a22b2e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 shutdown_connections 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 1 -- 172.21.15.16:0/2216023242 wait complete. 2024-09-12T02:53:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca6180740 20 client.26215 trim_cache size 0 max 16384 2024-09-12T02:53:32.731 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:53:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 msgr2=0x7f6c880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 0x7f6c880a5040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c94009500 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 msgr2=0x7f6c9c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.725+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c940 tx=0x7f6c8c00ce00 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 shutdown_connections 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c9c106f50 0x7f6c9c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6c880a2b80 0x7f6c880a5040 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c100d00 0x7f6c9c101810 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 --2- 172.21.15.16:0/3600334225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c100350 0x7f6c9c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 >> 172.21.15.16:0/3600334225 conn(0x7f6c9c077210 msgr2=0x7f6c9c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 shutdown_connections 2024-09-12T02:53:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:32.726+0000 7f6ca3da7640 1 -- 172.21.15.16:0/3600334225 wait complete. 2024-09-12T02:53:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:33 smithi016 ceph-mon[30925]: pgmap v506: 97 pgs: 97 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.9 MiB/s wr, 284 op/s 2024-09-12T02:53:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:33 smithi033 ceph-mon[35092]: pgmap v506: 97 pgs: 97 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.9 MiB/s wr, 284 op/s 2024-09-12T02:53:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:33 smithi184 ceph-mon[35321]: pgmap v506: 97 pgs: 97 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 3.9 MiB/s wr, 284 op/s 2024-09-12T02:53:33.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:53:33.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541389 11.24 MB/sec execute 155 sec latency 22.381 ms 2024-09-12T02:53:33.825 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:34.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 -- 172.21.15.16:0/4262540135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d481037a0 msgr2=0x7f5d4810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 --2- 172.21.15.16:0/4262540135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d481037a0 0x7f5d4810bc00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009900 tx=0x7f5d3802a410 comp rx=0 tx=0).stop 2024-09-12T02:53:34.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 -- 172.21.15.16:0/4262540135 shutdown_connections 2024-09-12T02:53:34.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 --2- 172.21.15.16:0/4262540135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d4810c2f0 0x7f5d4810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 --2- 172.21.15.16:0/4262540135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d481037a0 0x7f5d4810bc00 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 --2- 172.21.15.16:0/4262540135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d48102df0 0x7f5d481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 -- 172.21.15.16:0/4262540135 >> 172.21.15.16:0/4262540135 conn(0x7f5d480fc7d0 msgr2=0x7f5d480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 -- 172.21.15.16:0/4262540135 shutdown_connections 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.345+0000 7f5d4facb640 1 -- 172.21.15.16:0/4262540135 wait complete. 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 Processor -- start 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 -- start start 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d481037a0 0x7f5d48198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d4810c2f0 0x7f5d4819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481120b0 con 0x7f5d48102df0 2024-09-12T02:53:34.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48111f30 con 0x7f5d4810c2f0 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4facb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48112230 con 0x7f5d481037a0 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.346+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49972/0 (socket says 172.21.15.16:49972) 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 -- 172.21.15.16:0/1641272117 learned_addr learned my addr 172.21.15.16:0/1641272117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4e068640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d4810c2f0 0x7f5d4819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d481037a0 0x7f5d48198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 -- 172.21.15.16:0/1641272117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d481037a0 msgr2=0x7f5d48198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d481037a0 0x7f5d48198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 -- 172.21.15.16:0/1641272117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d4810c2f0 msgr2=0x7f5d4819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d4810c2f0 0x7f5d4819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 -- 172.21.15.16:0/1641272117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4819d5e0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4d867640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400ebc0 tx=0x7f5d4400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d3effd640 1 -- 172.21.15.16:0/1641272117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44010630 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4819d8d0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d3effd640 1 -- 172.21.15.16:0/1641272117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d44010cb0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d3effd640 1 -- 172.21.15.16:0/1641272117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44018bd0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.347+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d481a51b0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.350+0000 7f5d51ea4740 1 Processor -- start 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.350+0000 7f5d3effd640 1 -- 172.21.15.16:0/1641272117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5d440107d0 con 0x7f5d48102df0 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.350+0000 7f5d51ea4740 1 -- start start 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 0x55938ff00090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938ff00a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55938fe662f0 con 0x55938ff005d0 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55938fe0dcf0 con 0x55938ff0b370 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55938fe622b0 con 0x55938ff01b40 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938ff00a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4e068640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 0x55938ff00090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4e068640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 0x55938ff00090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60112/0 (socket says 172.21.15.16:60112) 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58066/0 (socket says 172.21.15.16:58066) 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4e068640 1 -- 172.21.15.16:0/1274641790 learned_addr learned my addr 172.21.15.16:0/1274641790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 -- 172.21.15.16:0/1274641790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 msgr2=0x55938ff00090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 0x55938ff00090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 -- 172.21.15.16:0/1274641790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 msgr2=0x55938ff00a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938ff00a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 -- 172.21.15.16:0/1274641790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938fdf23a0 con 0x55938ff0b370 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d3effd640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 0x7f5d340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.351+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d3802a930 tx=0x7f5d38005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3816e04deb9c5e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.352+0000 7f5d4e068640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 0x7f5d340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.352+0000 7f5d3effd640 1 -- 172.21.15.16:0/1641272117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5d440cf650 con 0x7f5d48102df0 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.352+0000 7f5d33fff640 1 -- 172.21.15.16:0/1274641790 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38040070 con 0x55938ff0b370 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.352+0000 7f5d33fff640 1 -- 172.21.15.16:0/1274641790 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d38004030 con 0x55938ff0b370 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.352+0000 7f5d33fff640 1 -- 172.21.15.16:0/1274641790 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38035040 con 0x55938ff0b370 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1274641790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 msgr2=0x55938ff0d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d4e068640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 0x7f5d340a5040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009e60 tx=0x7f5d40005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d3802a930 tx=0x7f5d38005d50 comp rx=0 tx=0).stop 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1274641790 shutdown_connections 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff0b370 0x55938ff0d760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938ff00a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1274641790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff01b40 0x55938ff00090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1274641790 >> 172.21.15.16:0/1274641790 conn(0x55938fec9310 msgr2=0x55938fefcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1274641790 shutdown_connections 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1274641790 wait complete. 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 -- start start 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.353+0000 7f5d51ea4740 1 Processor -- start 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 0x55938ff08000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55938ff0e010 con 0x55938ff01b40 2024-09-12T02:53:34.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 0x55938ff08000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55938fe21060 con 0x55938ff005d0 2024-09-12T02:53:34.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55938ff0e190 con 0x55938ff0b370 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58082/0 (socket says 172.21.15.16:58082) 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 -- 172.21.15.16:0/4090941840 learned_addr learned my addr 172.21.15.16:0/4090941840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4e068640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 0x55938ff08000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49992/0 (socket says 172.21.15.16:49992) 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d51ea4740 10 client.0 _ll_register_callbacks cb 0x55938fb748c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 -- 172.21.15.16:0/4090941840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 msgr2=0x55938ff085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 -- 172.21.15.16:0/4090941840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 msgr2=0x55938ff08000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 0x55938ff08000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.354+0000 7f5d4d867640 1 -- 172.21.15.16:0/4090941840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55938fbd73c0 con 0x55938ff005d0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d4e068640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d4d867640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400eb90 tx=0x7f5d44097af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=356c5932e2f79bd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44010e60 con 0x55938ff005d0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5d4409b3c0 con 0x55938ff005d0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d38004030 con 0x55938ff005d0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d4409bab0 con 0x55938ff005d0 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d51ea4740 10 client.26324 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55938ff19ae0 con 0x55938ff005d0 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.355+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d38004030 con 0x55938ff005d0 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5d440bab00 con 0x55938ff005d0 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 10 client.26324 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 20 client.26324 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff1af40 0x55938ff1d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 4 client.26324 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55938fe73860 con 0x55938ff1af40 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d4d066640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff1af40 0x55938ff1d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d4d066640 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff1af40 0x55938ff1d390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c79b91a44e8262a4 server_cookie=c8d9b52cf6a7c485 in_seq=0 out_seq=0 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.356+0000 7f5d317fa640 10 client.26324 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d317fa640 1 -- 172.21.15.16:0/4090941840 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55938fe73860 con 0x55938ff1af40 2024-09-12T02:53:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d317fa640 10 client.26324 handle_command_reply: tid=0 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 client.26324 shutdown 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 msgr2=0x55938ff0ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400eb90 tx=0x7f5d44097af0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 shutdown_connections 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff1af40 0x55938ff1d390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff01b40 0x55938ff08000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/4090941840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff005d0 0x55938ff0ad50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 >> 172.21.15.16:0/4090941840 conn(0x55938fec9310 msgr2=0x55938ff02960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d317fa640 0 client.26324 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 shutdown_connections 2024-09-12T02:53:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.357+0000 7f5d51ea4740 1 -- 172.21.15.16:0/4090941840 wait complete. 2024-09-12T02:53:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.358+0000 7f5d51ea4740 20 client.26324 trim_cache size 0 max 16384 2024-09-12T02:53:34.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.371+0000 7f5d51ea4740 1 Processor -- start 2024-09-12T02:53:34.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 -- start start 2024-09-12T02:53:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938fefbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff03660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ad530 con 0x55938ff005d0 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481af070 con 0x55938ff01b40 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481adc00 con 0x55938ff0b370 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938fefbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938fefbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50008/0 (socket says 172.21.15.16:50008) 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d867640 1 -- 172.21.15.16:0/1632650726 learned_addr learned my addr 172.21.15.16:0/1632650726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:58084/0 (socket says 172.21.15.16:58084) 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4e068640 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff03660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 -- 172.21.15.16:0/1632650726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 msgr2=0x55938ff03660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff03660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 -- 172.21.15.16:0/1632650726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 msgr2=0x55938fefbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938fefbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4d066640 1 -- 172.21.15.16:0/1632650726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938ff1d9e0 con 0x55938ff01b40 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.372+0000 7f5d4e068640 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff03660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d4d066640 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005940 tx=0x7f5d38000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d3e6e5b26857b12 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d337fe640 1 -- 172.21.15.16:0/1632650726 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38040050 con 0x55938ff01b40 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d337fe640 1 -- 172.21.15.16:0/1632650726 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d38004030 con 0x55938ff01b40 2024-09-12T02:53:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d337fe640 1 -- 172.21.15.16:0/1632650726 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38035040 con 0x55938ff01b40 2024-09-12T02:53:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1632650726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 msgr2=0x55938ff03120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.373+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005940 tx=0x7f5d38000bc0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1632650726 shutdown_connections 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff0b370 0x55938ff03660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff03120 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/1632650726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff005d0 0x55938fefbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1632650726 >> 172.21.15.16:0/1632650726 conn(0x55938fec9310 msgr2=0x55938fefb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1632650726 shutdown_connections 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- 172.21.15.16:0/1632650726 wait complete. 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- start start 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 Processor -- start 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff04d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff0b370 0x55938ff05290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ae9a0 con 0x55938ff0b370 2024-09-12T02:53:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ae2d0 con 0x55938ff01b40 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.374+0000 7f5d51ea4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ab510 con 0x55938ff005d0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d51ea4740 10 client.0 _ll_register_callbacks cb 0x7f5d440099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60156/0 (socket says 172.21.15.16:60156) 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 -- 172.21.15.16:0/2829423487 learned_addr learned my addr 172.21.15.16:0/2829423487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d066640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff04d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4e068640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff0b370 0x55938ff05290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 -- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 msgr2=0x55938ff04d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff04d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 -- 172.21.15.16:0/2829423487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff0b370 msgr2=0x55938ff05290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff0b370 0x55938ff05290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 -- 172.21.15.16:0/2829423487 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55938ff05ac0 con 0x55938ff005d0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d4d867640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5d440988f0 tx=0x7f5d44009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40bf5534a00f7b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d440b7070 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.375+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5d4409b3c0 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d38004030 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d44010e60 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 10 client.26233 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55938ff85c90 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d38004030 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5d440ba020 con 0x55938ff005d0 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 10 client.26233 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 10 client.26233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 20 client.26233 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff86430 0x55938ff88850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 4 client.26233 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.376+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55938fe6dd70 con 0x55938ff86430 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d4d066640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff86430 0x55938ff88850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d317fa640 10 client.26233 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d4d066640 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff86430 0x55938ff88850 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4454ed0135d551fa server_cookie=a4c9adbdd636e77f in_seq=0 out_seq=0 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d317fa640 1 -- 172.21.15.16:0/2829423487 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55938fe6dd70 con 0x55938ff86430 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d317fa640 10 client.26233 handle_command_reply: tid=0 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d51ea4740 1 client.26233 shutdown 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 msgr2=0x55938ff13b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.377+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5d440988f0 tx=0x7f5d44009590 comp rx=0 tx=0).stop 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 shutdown_connections 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55938ff86430 0x55938ff88850 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55938ff0b370 0x55938ff05290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55938ff01b40 0x55938ff04d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d317fa640 0 client.26233 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 --2- 172.21.15.16:0/2829423487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55938ff005d0 0x55938ff13b70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 >> 172.21.15.16:0/2829423487 conn(0x55938fec9310 msgr2=0x55938fefbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 shutdown_connections 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 1 -- 172.21.15.16:0/2829423487 wait complete. 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d51ea4740 20 client.26233 trim_cache size 0 max 16384 2024-09-12T02:53:34.383 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 msgr2=0x7f5d340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 0x7f5d340a5040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009e60 tx=0x7f5d40005f90 comp rx=0 tx=0).stop 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 msgr2=0x7f5d48198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.378+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400ebc0 tx=0x7f5d4400c6a0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 shutdown_connections 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d4810c2f0 0x7f5d4819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d481037a0 0x7f5d48198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5d340a2b80 0x7f5d340a5040 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 --2- 172.21.15.16:0/1641272117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d48102df0 0x7f5d48198630 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 >> 172.21.15.16:0/1641272117 conn(0x7f5d480fc7d0 msgr2=0x7f5d480fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 shutdown_connections 2024-09-12T02:53:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:34.379+0000 7f5d4facb640 1 -- 172.21.15.16:0/1641272117 wait complete. 2024-09-12T02:53:34.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 544187 11.26 MB/sec execute 156 sec latency 57.244 ms 2024-09-12T02:53:35.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:35 smithi016 ceph-mon[30925]: pgmap v507: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 3.5 MiB/s wr, 279 op/s 2024-09-12T02:53:35.405 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.1 2024-09-12T02:53:35.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:53:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:35 smithi033 ceph-mon[35092]: pgmap v507: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 3.5 MiB/s wr, 279 op/s 2024-09-12T02:53:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:35 smithi184 ceph-mon[35321]: pgmap v507: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 3.5 MiB/s wr, 279 op/s 2024-09-12T02:53:35.706 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:35.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 547059 11.30 MB/sec execute 157 sec latency 73.253 ms 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.202+0000 7fc7162af640 1 -- 172.21.15.16:0/1703843890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc710106f20 msgr2=0x7fc71010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.202+0000 7fc7162af640 1 --2- 172.21.15.16:0/1703843890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc710106f20 0x7fc71010eaa0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc708009ca0 tx=0x7fc70802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 -- 172.21.15.16:0/1703843890 shutdown_connections 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 --2- 172.21.15.16:0/1703843890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc710106f20 0x7fc71010eaa0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 --2- 172.21.15.16:0/1703843890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7101027a0 0x7fc7101067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 --2- 172.21.15.16:0/1703843890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7101021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 -- 172.21.15.16:0/1703843890 >> 172.21.15.16:0/1703843890 conn(0x7fc7100fd8b0 msgr2=0x7fc7100ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 -- 172.21.15.16:0/1703843890 shutdown_connections 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 -- 172.21.15.16:0/1703843890 wait complete. 2024-09-12T02:53:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 Processor -- start 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.203+0000 7fc7162af640 1 -- start start 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7100744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc710106f20 0x7fc710074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc710111f80 con 0x7fc7101027a0 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc710111e00 con 0x7fc710106f20 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc7162af640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc710112100 con 0x7fc710101df0 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51944/0 (socket says 172.21.15.16:51944) 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7100744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 -- 172.21.15.16:0/1970177809 learned_addr learned my addr 172.21.15.16:0/1970177809 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:36.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc71484c640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc710106f20 0x7fc710074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 -- 172.21.15.16:0/1970177809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 msgr2=0x7fc7100744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7100744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 -- 172.21.15.16:0/1970177809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc710106f20 msgr2=0x7fc710074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc710106f20 0x7fc710074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70f7fe640 1 -- 172.21.15.16:0/1970177809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7101a6860 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.204+0000 7fc70ffff640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7100744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc70f7fe640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9f0 tx=0x7fc70400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70400cd80 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc704010910 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7101a6b50 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704018990 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.205+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7101a7090 con 0x7fc7101027a0 2024-09-12T02:53:36.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.206+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7101021d0 con 0x7fc7101027a0 2024-09-12T02:53:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.207+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc704010ac0 con 0x7fc7101027a0 2024-09-12T02:53:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.208+0000 7fc70d7fa640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 0x7fc6f80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.208+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc7040cf320 con 0x7fc7101027a0 2024-09-12T02:53:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.208+0000 7fc70ffff640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 0x7fc6f80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.209+0000 7fc70ffff640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 0x7fc6f80a5040 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc700005d00 tx=0x7fc70000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.213+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc70409bbe0 con 0x7fc7101027a0 2024-09-12T02:53:36.358 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:53:36.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:53:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.451+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc710109b70 con 0x7fc7101027a0 2024-09-12T02:53:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.452+0000 7fc70d7fa640 1 -- 172.21.15.16:0/1970177809 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fc70409fee0 con 0x7fc7101027a0 2024-09-12T02:53:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:53:36.454 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:53:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:53:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 msgr2=0x7fc6f80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 0x7fc6f80a5040 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc700005d00 tx=0x7fc70000b410 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 msgr2=0x7fc710074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9f0 tx=0x7fc70400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 shutdown_connections 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc710106f20 0x7fc710074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6f80a2b80 0x7fc6f80a5040 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7101027a0 0x7fc710074a30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 --2- 172.21.15.16:0/1970177809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc710101df0 0x7fc7100744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.455+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 >> 172.21.15.16:0/1970177809 conn(0x7fc7100fd8b0 msgr2=0x7fc7100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.456+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 shutdown_connections 2024-09-12T02:53:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:36.456+0000 7fc7162af640 1 -- 172.21.15.16:0/1970177809 wait complete. 2024-09-12T02:53:36.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 548931 11.27 MB/sec execute 158 sec latency 18.105 ms 2024-09-12T02:53:37.255 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 140.1 --await-for 45.0' 2024-09-12T02:53:37.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 140.1 --await-for 45.0 2024-09-12T02:53:37.369 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:37.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:37 smithi016 ceph-mon[30925]: pgmap v508: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-12T02:53:37.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:37 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1970177809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:37.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:37 smithi033 ceph-mon[35092]: pgmap v508: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-12T02:53:37.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:37 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1970177809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:37 smithi184 ceph-mon[35321]: pgmap v508: 97 pgs: 97 active+clean; 191 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-09-12T02:53:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:37 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1970177809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:37.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 551398 11.29 MB/sec execute 159 sec latency 12.064 ms 2024-09-12T02:53:37.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 -- 172.21.15.16:0/1740451309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 msgr2=0x7f2b7c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 --2- 172.21.15.16:0/1740451309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c10ec30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009930 tx=0x7f2b6c02a420 comp rx=0 tx=0).stop 2024-09-12T02:53:37.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 -- 172.21.15.16:0/1740451309 shutdown_connections 2024-09-12T02:53:37.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 --2- 172.21.15.16:0/1740451309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c10ec30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 --2- 172.21.15.16:0/1740451309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 0x7f2b7c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 --2- 172.21.15.16:0/1740451309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 -- 172.21.15.16:0/1740451309 >> 172.21.15.16:0/1740451309 conn(0x7f2b7c0fc7d0 msgr2=0x7f2b7c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 -- 172.21.15.16:0/1740451309 shutdown_connections 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.946+0000 7f2b81385640 1 -- 172.21.15.16:0/1740451309 wait complete. 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 Processor -- start 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 -- start start 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 0x7f2b7c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c112090 con 0x7f2b7c108290 2024-09-12T02:53:37.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c111f10 con 0x7f2b7c102d90 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b81385640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c112210 con 0x7f2b7c103740 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51954/0 (socket says 172.21.15.16:51954) 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 0x7f2b7c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.947+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3271128329 learned_addr learned my addr 172.21.15.16:0/3271128329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3271128329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 msgr2=0x7f2b7c100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 0x7f2b7c100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3271128329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 msgr2=0x7f2b7c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3271128329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c1a6a10 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7affd640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005ab0 tx=0x7f2b6c031b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b73fff640 1 -- 172.21.15.16:0/3271128329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c03f030 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7c1a6ca0 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b73fff640 1 -- 172.21.15.16:0/3271128329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b6c0043d0 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b73fff640 1 -- 172.21.15.16:0/3271128329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c035070 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.948+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7c1a7180 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.950+0000 7f2b73fff640 1 -- 172.21.15.16:0/3271128329 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2b6c042030 con 0x7f2b7c108290 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.951+0000 7f2b8375e740 1 Processor -- start 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.951+0000 7f2b8375e740 1 -- start start 2024-09-12T02:53:37.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.951+0000 7f2b73fff640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 0x7f2b680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.951+0000 7f2b73fff640 1 -- 172.21.15.16:0/3271128329 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b6c0ef2d0 con 0x7f2b7c108290 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.951+0000 7f2b7a7fc640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 0x7f2b680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x559159080f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 0x55915908b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x55915908e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559158fe6280 con 0x559159081480 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559158f8db10 con 0x55915908bd40 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b8375e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559158fe2240 con 0x559159080b60 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 0x55915908b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x559159080f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 0x55915908b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51962/0 (socket says 172.21.15.16:51962) 2024-09-12T02:53:37.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7a7fc640 1 -- 172.21.15.16:0/2436507424 learned_addr learned my addr 172.21.15.16:0/2436507424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x55915908e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/2436507424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 msgr2=0x559159080f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x559159080f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/2436507424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 msgr2=0x55915908b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 0x55915908b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7a7fc640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 0x7f2b680a5040 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2b64013460 tx=0x7f2b64013860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/2436507424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559158f9a1f0 con 0x55915908bd40 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x55915908e130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0bb8f0 tx=0x7f2b6c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5814b64a0f80683 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b717fa640 1 -- 172.21.15.16:0/2436507424 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c049040 con 0x55915908bd40 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.952+0000 7f2b717fa640 1 -- 172.21.15.16:0/2436507424 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b6c0a6040 con 0x55915908bd40 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b717fa640 1 -- 172.21.15.16:0/2436507424 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0c4320 con 0x55915908bd40 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 -- 172.21.15.16:0/2436507424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 msgr2=0x55915908e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x55915908e130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0bb8f0 tx=0x7f2b6c009260 comp rx=0 tx=0).stop 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 -- 172.21.15.16:0/2436507424 shutdown_connections 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x55915908e130 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159081480 0x55915908b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 --2- 172.21.15.16:0/2436507424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x559159080f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 -- 172.21.15.16:0/2436507424 >> 172.21.15.16:0/2436507424 conn(0x55915907d0f0 msgr2=0x55915907d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 -- 172.21.15.16:0/2436507424 shutdown_connections 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.953+0000 7f2b8375e740 1 -- 172.21.15.16:0/2436507424 wait complete. 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 -- start start 2024-09-12T02:53:37.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 Processor -- start 2024-09-12T02:53:37.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x559159088bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x559159089100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55915908e6d0 con 0x559159080b60 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559158fa0f10 con 0x559159081480 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55915908e850 con 0x55915908bd40 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.954+0000 7f2b8375e740 10 client.0 _ll_register_callbacks cb 0x559158f69a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58408/0 (socket says 172.21.15.16:58408) 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/1783381860 learned_addr learned my addr 172.21.15.16:0/1783381860 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7a7fc640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x559159089100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7affd640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x559159088bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 msgr2=0x559159089100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x559159089100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/1783381860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 msgr2=0x559159088bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x559159088bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/1783381860 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559158d576a0 con 0x55915908bd40 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005c40 tx=0x7f2b6c0c47b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a6bb84ec2761dc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b627fc640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c09f060 con 0x55915908bd40 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b6c0c4320 con 0x55915908bd40 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.955+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c0a6040 con 0x55915908bd40 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2b6c0da070 con 0x55915908bd40 2024-09-12T02:53:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 10 client.26251 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55915908e6d0 con 0x55915908bd40 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b6c0a6040 con 0x55915908bd40 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2b6c0da2a0 con 0x55915908bd40 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 10 client.26251 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 10 client.26251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 20 client.26251 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5591590a3e20 0x5591590a6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 4 client.26251 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.956+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559158ff37f0 con 0x5591590a3e20 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.957+0000 7f2b7affd640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5591590a3e20 0x5591590a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.957+0000 7f2b627fc640 10 client.26251 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.957+0000 7f2b7affd640 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5591590a3e20 0x5591590a6260 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a9c3b3d637b5e4a server_cookie=318884332d1d05d9 in_seq=0 out_seq=0 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b627fc640 1 -- 172.21.15.16:0/1783381860 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559158ff37f0 con 0x5591590a3e20 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b627fc640 10 client.26251 handle_command_reply: tid=0 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 client.26251 shutdown 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 msgr2=0x559159089640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005c40 tx=0x7f2b6c0c47b0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 shutdown_connections 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5591590a3e20 0x5591590a6260 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55915908bd40 0x559159089640 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x559159089100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 --2- 172.21.15.16:0/1783381860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x559159088bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 >> 172.21.15.16:0/1783381860 conn(0x55915907d0f0 msgr2=0x559159082e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b627fc640 0 client.26251 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 shutdown_connections 2024-09-12T02:53:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.958+0000 7f2b8375e740 1 -- 172.21.15.16:0/1783381860 wait complete. 2024-09-12T02:53:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.959+0000 7f2b8375e740 20 client.26251 trim_cache size 0 max 16384 2024-09-12T02:53:37.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.979+0000 7f2b8375e740 1 Processor -- start 2024-09-12T02:53:37.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.979+0000 7f2b8375e740 1 -- start start 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x55915908aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159081480 0x5591590abb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x5591590ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1ab340 con 0x559159080b60 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1ace80 con 0x55915908bd40 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b8375e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1aba10 con 0x559159081480 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x55915908aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x5591590ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159081480 0x5591590abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x55915908aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51982/0 (socket says 172.21.15.16:51982) 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7affd640 1 -- 172.21.15.16:0/3506498039 learned_addr learned my addr 172.21.15.16:0/3506498039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3506498039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159081480 msgr2=0x5591590abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159081480 0x5591590abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3506498039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 msgr2=0x55915908aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x55915908aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.980+0000 7f2b7b7fe640 1 -- 172.21.15.16:0/3506498039 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55915907cf20 con 0x55915908bd40 2024-09-12T02:53:37.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.981+0000 7f2b7b7fe640 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x5591590ac0b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0c4280 tx=0x7f2b6c0ba100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8dd16bcd28eb892 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.981+0000 7f2b70ff9640 1 -- 172.21.15.16:0/3506498039 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c09f070 con 0x55915908bd40 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.981+0000 7f2b70ff9640 1 -- 172.21.15.16:0/3506498039 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b6c0a6040 con 0x55915908bd40 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.981+0000 7f2b70ff9640 1 -- 172.21.15.16:0/3506498039 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c049040 con 0x55915908bd40 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- 172.21.15.16:0/3506498039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 msgr2=0x5591590ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x5591590ac0b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0c4280 tx=0x7f2b6c0ba100 comp rx=0 tx=0).stop 2024-09-12T02:53:37.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- 172.21.15.16:0/3506498039 shutdown_connections 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55915908bd40 0x5591590ac0b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159081480 0x5591590abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 --2- 172.21.15.16:0/3506498039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559159080b60 0x55915908aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- 172.21.15.16:0/3506498039 >> 172.21.15.16:0/3506498039 conn(0x55915907d0f0 msgr2=0x55915908a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- 172.21.15.16:0/3506498039 shutdown_connections 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- 172.21.15.16:0/3506498039 wait complete. 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.982+0000 7f2b8375e740 1 -- start start 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 Processor -- start 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x55915909c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915908bd40 0x559159099270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1ac7b0 con 0x55915908bd40 2024-09-12T02:53:37.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1ac0e0 con 0x559159081480 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c1a9320 con 0x559159080b60 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b8375e740 10 client.0 _ll_register_callbacks cb 0x7f2b6400f540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x55915909c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915908bd40 0x559159099270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58422/0 (socket says 172.21.15.16:58422) 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 -- 172.21.15.16:0/955978197 learned_addr learned my addr 172.21.15.16:0/955978197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 -- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 msgr2=0x55915909c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x55915909c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.983+0000 7f2b7affd640 1 -- 172.21.15.16:0/955978197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915908bd40 msgr2=0x559159099270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b7affd640 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915908bd40 0x559159099270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b7affd640 1 -- 172.21.15.16:0/955978197 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559159078830 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b7affd640 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400fa10 tx=0x7f2b7400fbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bd6f5778bcb037 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b627fc640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7402b060 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2b74003970 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c0a6040 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2b74003b70 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b8375e740 10 client.26257 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.984+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55915910ef90 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b74003d10 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2b74025070 con 0x559159080b60 2024-09-12T02:53:37.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 10 client.26257 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 10 client.26257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 20 client.26257 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55915910f740 0x559159111b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 4 client.26257 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f2b6c0a6040 con 0x55915910f740 2024-09-12T02:53:37.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b7a7fc640 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55915910f740 0x559159111b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:37.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.985+0000 7f2b627fc640 10 client.26257 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:37.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.986+0000 7f2b7a7fc640 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55915910f740 0x559159111b60 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f671b6616d52a0f server_cookie=5c247748cb963b37 in_seq=0 out_seq=0 2024-09-12T02:53:37.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.986+0000 7f2b627fc640 1 -- 172.21.15.16:0/955978197 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f2b6c0a6040 con 0x55915910f740 2024-09-12T02:53:37.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.987+0000 7f2b627fc640 10 client.26257 handle_command_reply: tid=0 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.987+0000 7f2b8375e740 1 client.26257 shutdown 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.987+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 msgr2=0x55915909c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.987+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400fa10 tx=0x7f2b7400fbf0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 shutdown_connections 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55915910f740 0x559159111b60 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915908bd40 0x559159099270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559159081480 0x55915909c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 --2- 172.21.15.16:0/955978197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559159080b60 0x55915909c2a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 >> 172.21.15.16:0/955978197 conn(0x55915907d0f0 msgr2=0x55915907c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b627fc640 0 client.26257 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 shutdown_connections 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 1 -- 172.21.15.16:0/955978197 wait complete. 2024-09-12T02:53:37.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.988+0000 7f2b8375e740 20 client.26257 trim_cache size 0 max 16384 2024-09-12T02:53:37.992 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:53:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:53:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "7984094e-85e9-4128-8940-109d22e0f868", 2024-09-12T02:53:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:53:37.992 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 msgr2=0x7f2b680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 0x7f2b680a5040 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2b64013460 tx=0x7f2b64013860 comp rx=0 tx=0).stop 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 msgr2=0x7f2b7c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005ab0 tx=0x7f2b6c031b50 comp rx=0 tx=0).stop 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 shutdown_connections 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2b680a2b80 0x7f2b680a5040 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b7c108290 0x7f2b7c101470 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b7c103740 0x7f2b7c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 --2- 172.21.15.16:0/3271128329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b7c102d90 0x7f2b7c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:37.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 >> 172.21.15.16:0/3271128329 conn(0x7f2b7c0fc7d0 msgr2=0x7f2b7c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:37.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 shutdown_connections 2024-09-12T02:53:37.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:37.989+0000 7f2b81385640 1 -- 172.21.15.16:0/3271128329 wait complete. 2024-09-12T02:53:37.997 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:38.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 554383 11.33 MB/sec execute 160 sec latency 46.161 ms 2024-09-12T02:53:38.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:53:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 -- 172.21.15.16:0/2620726725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 msgr2=0x7efd7c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 --2- 172.21.15.16:0/2620726725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c101200 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009900 tx=0x7efd6c02a480 comp rx=0 tx=0).stop 2024-09-12T02:53:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 -- 172.21.15.16:0/2620726725 shutdown_connections 2024-09-12T02:53:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 --2- 172.21.15.16:0/2620726725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 0x7efd7c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 --2- 172.21.15.16:0/2620726725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 --2- 172.21.15.16:0/2620726725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c101200 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 -- 172.21.15.16:0/2620726725 >> 172.21.15.16:0/2620726725 conn(0x7efd7c0fc7d0 msgr2=0x7efd7c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 -- 172.21.15.16:0/2620726725 shutdown_connections 2024-09-12T02:53:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 -- 172.21.15.16:0/2620726725 wait complete. 2024-09-12T02:53:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.268+0000 7efd81810640 1 Processor -- start 2024-09-12T02:53:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 -- start start 2024-09-12T02:53:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 0x7efd7c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c110f50 con 0x7efd7c100e20 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c110dd0 con 0x7efd7c102570 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd81810640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1110d0 con 0x7efd7c102ef0 2024-09-12T02:53:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52006/0 (socket says 172.21.15.16:52006) 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 -- 172.21.15.16:0/1970420616 learned_addr learned my addr 172.21.15.16:0/1970420616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 0x7efd7c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 -- 172.21.15.16:0/1970420616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 msgr2=0x7efd7c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 0x7efd7c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 -- 172.21.15.16:0/1970420616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 msgr2=0x7efd7c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7affd640 1 -- 172.21.15.16:0/1970420616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7c19f6e0 con 0x7efd7c100e20 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.269+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd7affd640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006010 tx=0x7efd6c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd8080e640 1 -- 172.21.15.16:0/1970420616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c03f070 con 0x7efd7c100e20 2024-09-12T02:53:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7c19f970 con 0x7efd7c100e20 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd8080e640 1 -- 172.21.15.16:0/1970420616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd6c031c30 con 0x7efd7c100e20 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd8080e640 1 -- 172.21.15.16:0/1970420616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c035030 con 0x7efd7c100e20 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.270+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd7c1a72b0 con 0x7efd7c100e20 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd8080e640 1 -- 172.21.15.16:0/1970420616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efd6c042070 con 0x7efd7c100e20 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 Processor -- start 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 -- start start 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf78400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56068ced32a0 con 0x56068cf6d760 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56068ce7ab10 con 0x56068cf78940 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd83be9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56068cecf260 con 0x56068cf6e080 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd7affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf78400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52014/0 (socket says 172.21.15.16:52014) 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.273+0000 7efd7b7fe640 1 -- 172.21.15.16:0/840383642 learned_addr learned my addr 172.21.15.16:0/840383642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd8080e640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 0x7efd640a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 -- 172.21.15.16:0/840383642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 msgr2=0x56068cf78400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf78400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 -- 172.21.15.16:0/840383642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 msgr2=0x56068cf7ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 -- 172.21.15.16:0/840383642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56068cc44b90 con 0x56068cf6d760 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7affd640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf78400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd8080e640 1 -- 172.21.15.16:0/1970420616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd6c0ef690 con 0x7efd7c100e20 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 0x7efd640a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7efd74009f50 tx=0x7efd740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0605daaebcca89a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.274+0000 7efd61ffb640 1 -- 172.21.15.16:0/840383642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd74004250 con 0x56068cf6d760 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.275+0000 7efd61ffb640 1 -- 172.21.15.16:0/840383642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd7400fe80 con 0x56068cf6d760 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.275+0000 7efd61ffb640 1 -- 172.21.15.16:0/840383642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7401bbc0 con 0x56068cf6d760 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.275+0000 7efd83be9740 1 -- 172.21.15.16:0/840383642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 msgr2=0x56068cf6db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.275+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 0x7efd640a5040 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efd74012fd0 tx=0x7efd74021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.275+0000 7efd83be9740 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7efd74009f50 tx=0x7efd740118d0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 -- 172.21.15.16:0/840383642 shutdown_connections 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf78400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 --2- 172.21.15.16:0/840383642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf6db40 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 -- 172.21.15.16:0/840383642 >> 172.21.15.16:0/840383642 conn(0x56068cf69cf0 msgr2=0x56068cf6a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 -- 172.21.15.16:0/840383642 shutdown_connections 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 -- 172.21.15.16:0/840383642 wait complete. 2024-09-12T02:53:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 -- start start 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.276+0000 7efd83be9740 1 Processor -- start 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf75800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf76280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56068cf7b510 con 0x56068cf78940 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56068ce5b6e0 con 0x56068cf6d760 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56068cf7b690 con 0x56068cf6e080 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd83be9740 10 client.0 _ll_register_callbacks cb 0x56068ce7fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58440/0 (socket says 172.21.15.16:58440) 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 -- 172.21.15.16:0/3823374644 learned_addr learned my addr 172.21.15.16:0/3823374644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf76280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 -- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 msgr2=0x56068cf75800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7affd640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf75800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf75800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 -- 172.21.15.16:0/3823374644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 msgr2=0x56068cf76280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf76280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7a7fc640 1 -- 172.21.15.16:0/3823374644 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56068cf849d0 con 0x56068cf6e080 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.277+0000 7efd7b7fe640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf76280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd70013d30 tx=0x7efd700117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c729a59861ca02b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd5affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd7affd640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf75800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70020060 con 0x56068cf6e080 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd7001b630 con 0x56068cf6e080 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56068cee09d0 con 0x56068cf6e080 2024-09-12T02:53:39.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd7001b830 con 0x56068cf6e080 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd83be9740 10 client.26269 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56068cf87330 con 0x56068cf6e080 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.278+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56068cee09d0 con 0x56068cf6e080 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd70026b90 con 0x56068cf6e080 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 10 client.26269 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 10 client.26269 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 20 client.26269 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cf8ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 4 client.26269 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56068cedad20 con 0x56068cf887f0 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd7affd640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cf8ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd5affd640 10 client.26269 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.279+0000 7efd7affd640 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cf8ac40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec55522df9efa1d server_cookie=4e3d5a83030e89b8 in_seq=0 out_seq=0 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd5affd640 1 -- 172.21.15.16:0/3823374644 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56068cedad20 con 0x56068cf887f0 2024-09-12T02:53:39.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd5affd640 10 client.26269 handle_command_reply: tid=0 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 client.26269 shutdown 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 msgr2=0x56068cf75d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd70013d30 tx=0x7efd700117b0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 shutdown_connections 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cf8ac40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf76280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf75d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd5affd640 0 client.26269 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 --2- 172.21.15.16:0/3823374644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf75800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 >> 172.21.15.16:0/3823374644 conn(0x56068cf69cf0 msgr2=0x56068cf68e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 shutdown_connections 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.280+0000 7efd83be9740 1 -- 172.21.15.16:0/3823374644 wait complete. 2024-09-12T02:53:39.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.281+0000 7efd83be9740 20 client.26269 trim_cache size 0 max 16384 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 Processor -- start 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 -- start start 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf68d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf90510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1af790 con 0x56068cf78940 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1b12d0 con 0x56068cf6d760 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.300+0000 7efd83be9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1afe60 con 0x56068cf6e080 2024-09-12T02:53:39.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34010/0 (socket says 172.21.15.16:34010) 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf90510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 -- 172.21.15.16:0/2313514768 learned_addr learned my addr 172.21.15.16:0/2313514768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf68d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 -- 172.21.15.16:0/2313514768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 msgr2=0x56068cf68d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf68d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 -- 172.21.15.16:0/2313514768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 msgr2=0x56068cf90510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf90510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 -- 172.21.15.16:0/2313514768 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56068cf747b0 con 0x56068cf6d760 2024-09-12T02:53:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd7affd640 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd6c005ab0 tx=0x7efd6c0f5550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c882681cb11c1a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd617fa640 1 -- 172.21.15.16:0/2313514768 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c049040 con 0x56068cf6d760 2024-09-12T02:53:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.301+0000 7efd617fa640 1 -- 172.21.15.16:0/2313514768 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd6c031de0 con 0x56068cf6d760 2024-09-12T02:53:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd617fa640 1 -- 172.21.15.16:0/2313514768 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0ba2b0 con 0x56068cf6d760 2024-09-12T02:53:39.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 -- 172.21.15.16:0/2313514768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 msgr2=0x56068cf701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd6c005ab0 tx=0x7efd6c0f5550 comp rx=0 tx=0).stop 2024-09-12T02:53:39.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 -- 172.21.15.16:0/2313514768 shutdown_connections 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf78940 0x56068cf90510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf68d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 --2- 172.21.15.16:0/2313514768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf6d760 0x56068cf701c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 -- 172.21.15.16:0/2313514768 >> 172.21.15.16:0/2313514768 conn(0x56068cf69cf0 msgr2=0x56068cf68650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 -- 172.21.15.16:0/2313514768 shutdown_connections 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.302+0000 7efd83be9740 1 -- 172.21.15.16:0/2313514768 wait complete. 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 -- start start 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 Processor -- start 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf80e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1b0c00 con 0x56068cf6d760 2024-09-12T02:53:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1b0530 con 0x56068cf78940 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1ad770 con 0x56068cf6e080 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.303+0000 7efd83be9740 10 client.0 _ll_register_callbacks cb 0x7efd70013940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf80e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf808e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52044/0 (socket says 172.21.15.16:52044) 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7affd640 1 -- 172.21.15.16:0/309185282 learned_addr learned my addr 172.21.15.16:0/309185282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 -- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 msgr2=0x56068cf7d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 -- 172.21.15.16:0/309185282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 msgr2=0x56068cf808e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf808e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 -- 172.21.15.16:0/309185282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56068cf7ddf0 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd7a7fc640 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf80e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efd70027020 tx=0x7efd7000f780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8feea730ba8c84b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd5affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.304+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70035060 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd7001bde0 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd6c031de0 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd7002b030 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 10 client.26275 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56068cff3b20 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd6c031de0 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd700035f0 con 0x56068cf6e080 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 10 client.26275 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 10 client.26275 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 20 client.26275 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cff6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 4 client.26275 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.1, "await_for": 45.0}] 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.305+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.1, "await_for": 45.0}) -- 0x56068ce56ec0 con 0x56068cf887f0 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.306+0000 7efd7affd640 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cff6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.306+0000 7efd7affd640 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cff6320 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a91c50f65fa04569 server_cookie=5b045ffd0c639a17 in_seq=0 out_seq=0 2024-09-12T02:53:39.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.306+0000 7efd5affd640 10 client.26275 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd5affd640 1 -- 172.21.15.16:0/309185282 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56068ce56ec0 con 0x56068cf887f0 2024-09-12T02:53:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd5affd640 10 client.26275 handle_command_reply: tid=0 2024-09-12T02:53:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 client.26275 shutdown 2024-09-12T02:53:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 msgr2=0x56068cf80e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf80e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efd70027020 tx=0x7efd7000f780 comp rx=0 tx=0).stop 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 shutdown_connections 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56068cf887f0 0x56068cff6320 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56068cf78940 0x56068cf7d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd5affd640 0 client.26275 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56068cf6e080 0x56068cf80e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 --2- 172.21.15.16:0/309185282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56068cf6d760 0x56068cf808e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 >> 172.21.15.16:0/309185282 conn(0x56068cf69cf0 msgr2=0x56068cf75fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 shutdown_connections 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 1 -- 172.21.15.16:0/309185282 wait complete. 2024-09-12T02:53:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.368+0000 7efd83be9740 20 client.26275 trim_cache size 0 max 16384 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 22, 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "76fbe22d": { 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 22, 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:53:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.1, 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 msgr2=0x7efd640a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 0x7efd640a5040 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efd74012fd0 tx=0x7efd74021000 comp rx=0 tx=0).stop 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 msgr2=0x7efd7c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006010 tx=0x7efd6c003c10 comp rx=0 tx=0).stop 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 shutdown_connections 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd7c102ef0 0x7efd7c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd7c102570 0x7efd7c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd640a2b80 0x7efd640a5040 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 --2- 172.21.15.16:0/1970420616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd7c100e20 0x7efd7c19a730 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 >> 172.21.15.16:0/1970420616 conn(0x7efd7c0fc7d0 msgr2=0x7efd7c06a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 shutdown_connections 2024-09-12T02:53:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:39.369+0000 7efd81810640 1 -- 172.21.15.16:0/1970420616 wait complete. 2024-09-12T02:53:39.450 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:53:39.450 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.450 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:53:39.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:53:39.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:39 smithi016 ceph-mon[30925]: pgmap v509: 97 pgs: 97 active+clean; 199 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:39 smithi033 ceph-mon[35092]: pgmap v509: 97 pgs: 97 active+clean; 199 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-09-12T02:53:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:53:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:53:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:53:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:53:39.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:39 smithi184 ceph-mon[35321]: pgmap v509: 97 pgs: 97 active+clean; 199 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-09-12T02:53:39.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 11.28 MB/sec execute 161 sec latency 351.862 ms 2024-09-12T02:53:40.454 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '76fbe22d', quiesce took 5.0 seconds. Will release after: 15.100000000000001 2024-09-12T02:53:40.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 -- 172.21.15.16:0/19341210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954102d80 msgr2=0x7fe954103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 --2- 172.21.15.16:0/19341210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954102d80 0x7fe954103160 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009ca0 tx=0x7fe94c02a950 comp rx=0 tx=0).stop 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 -- 172.21.15.16:0/19341210 shutdown_connections 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 --2- 172.21.15.16:0/19341210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95410c160 0x7fe95410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 --2- 172.21.15.16:0/19341210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9541036a0 0x7fe95410bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 --2- 172.21.15.16:0/19341210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954102d80 0x7fe954103160 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 -- 172.21.15.16:0/19341210 >> 172.21.15.16:0/19341210 conn(0x7fe9540fc780 msgr2=0x7fe9540feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 -- 172.21.15.16:0/19341210 shutdown_connections 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 -- 172.21.15.16:0/19341210 wait complete. 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.669+0000 7fe95c706640 1 Processor -- start 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 -- start start 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954102d80 0x7fe9541009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9541036a0 0x7fe954100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe954111f40 con 0x7fe95410c160 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe954111dc0 con 0x7fe9541036a0 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95c706640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541120c0 con 0x7fe954102d80 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52076/0 (socket says 172.21.15.16:52076) 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 -- 172.21.15.16:0/2986768818 learned_addr learned my addr 172.21.15.16:0/2986768818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954102d80 0x7fe9541009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe959ca1640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9541036a0 0x7fe954100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 -- 172.21.15.16:0/2986768818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954102d80 msgr2=0x7fe9541009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954102d80 0x7fe9541009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 -- 172.21.15.16:0/2986768818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9541036a0 msgr2=0x7fe954100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9541036a0 0x7fe954100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 -- 172.21.15.16:0/2986768818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9541a6930 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.670+0000 7fe95aca3640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe94400d750 tx=0x7fe94400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.671+0000 7fe9537fe640 1 -- 172.21.15.16:0/2986768818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944017070 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.671+0000 7fe9537fe640 1 -- 172.21.15.16:0/2986768818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe944005500 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.671+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9541a6c20 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.671+0000 7fe9537fe640 1 -- 172.21.15.16:0/2986768818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944004d90 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.671+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9541a7160 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.673+0000 7fe9537fe640 1 -- 172.21.15.16:0/2986768818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe944004f30 con 0x7fe95410c160 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.673+0000 7fe95eadf740 1 Processor -- start 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.673+0000 7fe95eadf740 1 -- start start 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a10552c2e0 con 0x55a1055c6dc0 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a1054d3ce0 con 0x55a1055c64a0 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95eadf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a1055282a0 con 0x55a1055d1680 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52092/0 (socket says 172.21.15.16:52092) 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95a4a2640 1 -- 172.21.15.16:0/1651111176 learned_addr learned my addr 172.21.15.16:0/1651111176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 -- 172.21.15.16:0/1651111176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 msgr2=0x55a1055c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 -- 172.21.15.16:0/1651111176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 msgr2=0x55a1055d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 -- 172.21.15.16:0/1651111176 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1054b8440 con 0x55a1055d1680 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe9537fe640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 0x7fe9400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 0x7fe9400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe959ca1640 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d3a70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe93c012450 tx=0x7fe93c0096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3ec508f8f8c1170 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.674+0000 7fe9537fe640 1 -- 172.21.15.16:0/2986768818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9440ce920 con 0x7fe95410c160 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe950ff9640 1 -- 172.21.15.16:0/1651111176 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c004230 con 0x55a1055d1680 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe950ff9640 1 -- 172.21.15.16:0/1651111176 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c0043d0 con 0x55a1055d1680 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe950ff9640 1 -- 172.21.15.16:0/1651111176 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c004e60 con 0x55a1055d1680 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 -- 172.21.15.16:0/1651111176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 msgr2=0x55a1055d3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d3a70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe93c012450 tx=0x7fe93c0096f0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 0x7fe9400a50c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x55a1055c3890 tx=0x7fe94c004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 -- 172.21.15.16:0/1651111176 shutdown_connections 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d3a70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c6dc0 0x55a1055d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 --2- 172.21.15.16:0/1651111176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.675+0000 7fe95eadf740 1 -- 172.21.15.16:0/1651111176 >> 172.21.15.16:0/1651111176 conn(0x55a1055c2a30 msgr2=0x55a1055c2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- 172.21.15.16:0/1651111176 shutdown_connections 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- 172.21.15.16:0/1651111176 wait complete. 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- start start 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 Processor -- start 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055d1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 0x55a1055ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 0x55a1055ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a1055d4250 con 0x55a1055d1680 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a1054e7050 con 0x55a1055c64a0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a1055d43d0 con 0x55a1055c6dc0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe95eadf740 10 client.0 _ll_register_callbacks cb 0x55a105527160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.676+0000 7fe959ca1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 0x55a1055ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 0x55a1055ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe959ca1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 0x55a1055ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58478/0 (socket says 172.21.15.16:58478) 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe959ca1640 1 -- 172.21.15.16:0/3361342324 learned_addr learned my addr 172.21.15.16:0/3361342324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 -- 172.21.15.16:0/3361342324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 msgr2=0x55a1055ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 0x55a1055ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 -- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 msgr2=0x55a1055d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055d1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 -- 172.21.15.16:0/3361342324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a10529d3c0 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95aca3640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 0x55a1055ce920 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe9440cc480 tx=0x7fe94409abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5077bc3687eda464 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe949ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944020050 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe944097800 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c0043d0 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe9440b4190 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95eadf740 10 client.16845 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.677+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a1055dfd30 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c0043d0 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe9440bb0b0 con 0x55a1055d1680 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 10 client.16845 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 20 client.16845 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a1055e1240 0x55a1055e3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 4 client.16845 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a105539850 con 0x55a1055e1240 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a1055e1240 0x55a1055e3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a1055e1240 0x55a1055e3650 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5aa7c62c0f16527 server_cookie=db43964977747f25 in_seq=0 out_seq=0 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.678+0000 7fe949ffb640 10 client.16845 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe949ffb640 1 -- 172.21.15.16:0/3361342324 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a105539850 con 0x55a1055e1240 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe949ffb640 10 client.16845 handle_command_reply: tid=0 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 client.16845 shutdown 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 msgr2=0x55a1055ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 0x55a1055ce920 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe9440cc480 tx=0x7fe94409abf0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 shutdown_connections 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a1055e1240 0x55a1055e3650 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055d1680 0x55a1055ce920 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055c6dc0 0x55a1055ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe949ffb640 0 client.16845 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3361342324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c64a0 0x55a1055d1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 >> 172.21.15.16:0/3361342324 conn(0x55a1055c2a30 msgr2=0x55a1055c8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.679+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 shutdown_connections 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.680+0000 7fe95eadf740 1 -- 172.21.15.16:0/3361342324 wait complete. 2024-09-12T02:53:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.680+0000 7fe95eadf740 20 client.16845 trim_cache size 0 max 16384 2024-09-12T02:53:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 Processor -- start 2024-09-12T02:53:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 -- start start 2024-09-12T02:53:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055dd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541af680 con 0x55a1055c64a0 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541b11c0 con 0x55a1055c6dc0 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95eadf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541afd50 con 0x55a1055d1680 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055dd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95aca3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055dd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52114/0 (socket says 172.21.15.16:52114) 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe959ca1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95a4a2640 1 -- 172.21.15.16:0/954737978 learned_addr learned my addr 172.21.15.16:0/954737978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.697+0000 7fe95aca3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58492/0 (socket says 172.21.15.16:58492) 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 -- 172.21.15.16:0/954737978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 msgr2=0x55a1055e8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 -- 172.21.15.16:0/954737978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 msgr2=0x55a1055dd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055dd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 -- 172.21.15.16:0/954737978 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1055cd2e0 con 0x55a1055c6dc0 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe95aca3640 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe959ca1640 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055ccc00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55a1055cf970 tx=0x7fe93c009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a72b014b57b211 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe94bfff640 1 -- 172.21.15.16:0/954737978 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c004040 con 0x55a1055c6dc0 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe94bfff640 1 -- 172.21.15.16:0/954737978 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c0046c0 con 0x55a1055c6dc0 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.698+0000 7fe94bfff640 1 -- 172.21.15.16:0/954737978 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c003d10 con 0x55a1055c6dc0 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- 172.21.15.16:0/954737978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 msgr2=0x55a1055ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055ccc00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55a1055cf970 tx=0x7fe93c009420 comp rx=0 tx=0).stop 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- 172.21.15.16:0/954737978 shutdown_connections 2024-09-12T02:53:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055ccc00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 --2- 172.21.15.16:0/954737978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055dd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- 172.21.15.16:0/954737978 >> 172.21.15.16:0/954737978 conn(0x55a1055c2a30 msgr2=0x55a1055d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- 172.21.15.16:0/954737978 shutdown_connections 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- 172.21.15.16:0/954737978 wait complete. 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.699+0000 7fe95eadf740 1 -- start start 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 Processor -- start 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541b0af0 con 0x55a1055c64a0 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541b0420 con 0x55a1055c6dc0 2024-09-12T02:53:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541ad660 con 0x55a1055d1680 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95eadf740 10 client.0 _ll_register_callbacks cb 0x7fe944009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52118/0 (socket says 172.21.15.16:52118) 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 -- 172.21.15.16:0/3651791520 learned_addr learned my addr 172.21.15.16:0/3651791520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95aca3640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe959ca1640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 -- 172.21.15.16:0/3651791520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 msgr2=0x55a1055d6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 -- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 msgr2=0x55a1055d9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055d9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 -- 172.21.15.16:0/3651791520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1055d6b60 con 0x55a1055c64a0 2024-09-12T02:53:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe959ca1640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055d9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe95a4a2640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004ad0 tx=0x7fe94c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a22842877bfd4a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.700+0000 7fe949ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c057020 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe94c004040 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c0046c0 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe94c034040 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 10 client.16851 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a10564c660 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c0046c0 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe94c0539d0 con 0x55a1055c64a0 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 10 client.16851 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 20 client.16851 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a10564cd30 0x55a10564f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 4 client.16851 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.701+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a105533d60 con 0x55a10564cd30 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe959ca1640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a10564cd30 0x55a10564f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe959ca1640 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a10564cd30 0x55a10564f120 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48552cf97ea3fd6a server_cookie=8d11f9f97a1f4ade in_seq=0 out_seq=0 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe949ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe949ffb640 1 -- 172.21.15.16:0/3651791520 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a105533d60 con 0x55a10564cd30 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe949ffb640 10 client.16851 handle_command_reply: tid=0 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 client.16851 shutdown 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 msgr2=0x55a1055d9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004ad0 tx=0x7fe94c0095c0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 shutdown_connections 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a10564cd30 0x55a10564f120 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe949ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a1055d1680 0x55a1055d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1055c6dc0 0x55a1055d9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 --2- 172.21.15.16:0/3651791520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a1055c64a0 0x55a1055d9650 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.702+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 >> 172.21.15.16:0/3651791520 conn(0x55a1055c2a30 msgr2=0x55a1055c1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 shutdown_connections 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95eadf740 1 -- 172.21.15.16:0/3651791520 wait complete. 2024-09-12T02:53:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95eadf740 20 client.16851 trim_cache size 0 max 16384 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "7984094e-85e9-4128-8940-109d22e0f868": { 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "7984094e-85e9-4128-8940-109d22e0f868", 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:40.715 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 msgr2=0x7fe9400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 0x7fe9400a50c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x55a1055c3890 tx=0x7fe94c004a60 comp rx=0 tx=0).stop 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 msgr2=0x7fe954101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe94400d750 tx=0x7fe94400dc10 comp rx=0 tx=0).stop 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 shutdown_connections 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9400a2c00 0x7fe9400a50c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95410c160 0x7fe954101460 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9541036a0 0x7fe954100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 --2- 172.21.15.16:0/2986768818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954102d80 0x7fe9541009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 >> 172.21.15.16:0/2986768818 conn(0x7fe9540fc780 msgr2=0x7fe9540fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.703+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 shutdown_connections 2024-09-12T02:53:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:40.704+0000 7fe95c706640 1 -- 172.21.15.16:0/2986768818 wait complete. 2024-09-12T02:53:40.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 11.21 MB/sec execute 162 sec latency 1351.904 ms 2024-09-12T02:53:41.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:41 smithi016 ceph-mon[30925]: pgmap v510: 97 pgs: 97 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-09-12T02:53:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:41 smithi033 ceph-mon[35092]: pgmap v510: 97 pgs: 97 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-09-12T02:53:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:41 smithi184 ceph-mon[35321]: pgmap v510: 97 pgs: 97 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-09-12T02:53:41.705 INFO:tasks.cephfs.filesystem:scrub status for tag:7984094e-85e9-4128-8940-109d22e0f868 - {'path': '/', 'tag': '7984094e-85e9-4128-8940-109d22e0f868', 'options': 'recursive,force'} 2024-09-12T02:53:41.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 11.14 MB/sec execute 163 sec latency 2351.988 ms 2024-09-12T02:53:42.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 11.07 MB/sec execute 164 sec latency 3352.090 ms 2024-09-12T02:53:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:43 smithi033 ceph-mon[35092]: pgmap v511: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 262 op/s 2024-09-12T02:53:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:43 smithi184 ceph-mon[35321]: pgmap v511: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 262 op/s 2024-09-12T02:53:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:43 smithi016 ceph-mon[30925]: pgmap v511: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 262 op/s 2024-09-12T02:53:43.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 11.00 MB/sec execute 165 sec latency 4352.131 ms 2024-09-12T02:53:44.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.94 MB/sec execute 166 sec latency 5352.225 ms 2024-09-12T02:53:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:45 smithi033 ceph-mon[35092]: pgmap v512: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 211 op/s 2024-09-12T02:53:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:45 smithi184 ceph-mon[35321]: pgmap v512: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 211 op/s 2024-09-12T02:53:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:45 smithi016 ceph-mon[30925]: pgmap v512: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 211 op/s 2024-09-12T02:53:45.720 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.87 MB/sec execute 167 sec latency 6352.300 ms 2024-09-12T02:53:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.81 MB/sec execute 168 sec latency 7352.409 ms 2024-09-12T02:53:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:47 smithi033 ceph-mon[35092]: pgmap v513: 97 pgs: 97 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-09-12T02:53:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:47 smithi184 ceph-mon[35321]: pgmap v513: 97 pgs: 97 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-09-12T02:53:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:47 smithi016 ceph-mon[30925]: pgmap v513: 97 pgs: 97 active+clean; 221 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-09-12T02:53:47.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.74 MB/sec execute 169 sec latency 8352.523 ms 2024-09-12T02:53:48.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:53:48.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:53:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:53:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:53:48.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:53:48.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:53:48.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.68 MB/sec execute 170 sec latency 9352.661 ms 2024-09-12T02:53:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:49 smithi033 ceph-mon[35092]: pgmap v514: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-09-12T02:53:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:49 smithi184 ceph-mon[35321]: pgmap v514: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-09-12T02:53:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:49 smithi016 ceph-mon[30925]: pgmap v514: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-09-12T02:53:49.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.62 MB/sec execute 171 sec latency 10352.787 ms 2024-09-12T02:53:50.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.56 MB/sec execute 172 sec latency 11352.915 ms 2024-09-12T02:53:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:51 smithi033 ceph-mon[35092]: pgmap v515: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-12T02:53:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:51 smithi184 ceph-mon[35321]: pgmap v515: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-12T02:53:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:51 smithi016 ceph-mon[30925]: pgmap v515: 97 pgs: 97 active+clean; 220 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-12T02:53:51.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.49 MB/sec execute 173 sec latency 12353.043 ms 2024-09-12T02:53:52.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.43 MB/sec execute 174 sec latency 13353.134 ms 2024-09-12T02:53:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:53 smithi033 ceph-mon[35092]: pgmap v516: 97 pgs: 97 active+clean; 219 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.4 MiB/s wr, 88 op/s 2024-09-12T02:53:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:53 smithi184 ceph-mon[35321]: pgmap v516: 97 pgs: 97 active+clean; 219 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.4 MiB/s wr, 88 op/s 2024-09-12T02:53:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:53 smithi016 ceph-mon[30925]: pgmap v516: 97 pgs: 97 active+clean; 219 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 2.4 MiB/s wr, 88 op/s 2024-09-12T02:53:53.721 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.37 MB/sec execute 175 sec latency 14353.260 ms 2024-09-12T02:53:54.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.32 MB/sec execute 176 sec latency 15353.362 ms 2024-09-12T02:53:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:55 smithi033 ceph-mon[35092]: pgmap v517: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-12T02:53:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:55 smithi184 ceph-mon[35321]: pgmap v517: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-12T02:53:55.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:53:55.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:55 smithi016 ceph-mon[30925]: pgmap v517: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.1 MiB/s wr, 85 op/s 2024-09-12T02:53:55.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.26 MB/sec execute 177 sec latency 16353.430 ms 2024-09-12T02:53:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 -- 172.21.15.16:0/2536239522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0102ef0 msgr2=0x7f4fb010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 --2- 172.21.15.16:0/2536239522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0102ef0 0x7f4fb010da40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009cc0 tx=0x7f4fa802a970 comp rx=0 tx=0).stop 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 -- 172.21.15.16:0/2536239522 shutdown_connections 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 --2- 172.21.15.16:0/2536239522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0102ef0 0x7f4fb010da40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 --2- 172.21.15.16:0/2536239522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 0x7f4fb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 --2- 172.21.15.16:0/2536239522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0100e20 0x7f4fb0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 -- 172.21.15.16:0/2536239522 >> 172.21.15.16:0/2536239522 conn(0x7f4fb00fc7d0 msgr2=0x7f4fb00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 -- 172.21.15.16:0/2536239522 shutdown_connections 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 -- 172.21.15.16:0/2536239522 wait complete. 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.335+0000 7f4fb664b640 1 Processor -- start 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 -- start start 2024-09-12T02:53:56.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 0x7f4fb0196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0102ef0 0x7f4fb019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb010fe80 con 0x7f4fb0100e20 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb010fd00 con 0x7f4fb0102570 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb664b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb0110000 con 0x7f4fb0102ef0 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60612/0 (socket says 172.21.15.16:60612) 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 -- 172.21.15.16:0/3449100448 learned_addr learned my addr 172.21.15.16:0/3449100448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fa7fff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 0x7f4fb0196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4fb4be8640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0102ef0 0x7f4fb019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 -- 172.21.15.16:0/3449100448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0102ef0 msgr2=0x7f4fb019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0102ef0 0x7f4fb019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 -- 172.21.15.16:0/3449100448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 msgr2=0x7f4fb0196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 0x7f4fb0196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.336+0000 7f4faffff640 1 -- 172.21.15.16:0/3449100448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb019b1b0 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4faffff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0010af0 tx=0x7f4fa000bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa001a020 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb019b4a0 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fa000ebd0 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00155e0 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.337+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb01a2d80 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.338+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb0101200 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.339+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4fa001d020 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.340+0000 7f4fadffb640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 0x7f4f9c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.340+0000 7f4fa7fff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 0x7f4f9c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.340+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4fa00d2fc0 con 0x7f4fb0100e20 2024-09-12T02:53:56.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.341+0000 7f4fa7fff640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 0x7f4f9c0a5040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00fec10 tx=0x7f4f9800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:56.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.344+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4fa0024070 con 0x7f4fb0100e20 2024-09-12T02:53:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.556+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fb01a2fb0 con 0x7f4fb0100e20 2024-09-12T02:53:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.557+0000 7f4fadffb640 1 -- 172.21.15.16:0/3449100448 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f4fa009fed0 con 0x7f4fb0100e20 2024-09-12T02:53:56.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:53:56.559 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:53:56.559 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.559+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 msgr2=0x7f4f9c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.559+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 0x7f4f9c0a5040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00fec10 tx=0x7f4f9800b390 comp rx=0 tx=0).stop 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.559+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 msgr2=0x7f4fb0196200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.559+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0010af0 tx=0x7f4fa000bf30 comp rx=0 tx=0).stop 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 shutdown_connections 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb0102ef0 0x7f4fb019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fb0102570 0x7f4fb0196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f9c0a2b80 0x7f4f9c0a5040 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 --2- 172.21.15.16:0/3449100448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fb0100e20 0x7f4fb0196200 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 >> 172.21.15.16:0/3449100448 conn(0x7f4fb00fc7d0 msgr2=0x7f4fb01067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 shutdown_connections 2024-09-12T02:53:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:56.560+0000 7f4fb664b640 1 -- 172.21.15.16:0/3449100448 wait complete. 2024-09-12T02:53:56.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.20 MB/sec execute 178 sec latency 17353.479 ms 2024-09-12T02:53:57.447 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 76fbe22d --release --await' 2024-09-12T02:53:57.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 76fbe22d --release --await 2024-09-12T02:53:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:57 smithi033 ceph-mon[35092]: pgmap v518: 97 pgs: 97 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:53:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:57 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3449100448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:57.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:57 smithi016 ceph-mon[30925]: pgmap v518: 97 pgs: 97 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:53:57.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:57 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3449100448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:57 smithi184 ceph-mon[35321]: pgmap v518: 97 pgs: 97 active+clean; 216 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-12T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:57 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3449100448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:53:57.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555286 10.14 MB/sec execute 179 sec latency 18353.526 ms 2024-09-12T02:53:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 -- 172.21.15.16:0/2716645322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102790 msgr2=0x7f9b6c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/2716645322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102790 0x7f9b6c1067e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009900 tx=0x7f9b5c02a410 comp rx=0 tx=0).stop 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 -- 172.21.15.16:0/2716645322 shutdown_connections 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/2716645322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c106f10 0x7f9b6c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/2716645322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c102790 0x7f9b6c1067e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/2716645322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c101de0 0x7f9b6c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 -- 172.21.15.16:0/2716645322 >> 172.21.15.16:0/2716645322 conn(0x7f9b6c0fd8c0 msgr2=0x7f9b6c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 -- 172.21.15.16:0/2716645322 shutdown_connections 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.244+0000 7f9b72e9a640 1 -- 172.21.15.16:0/2716645322 wait complete. 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 Processor -- start 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 -- start start 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c101de0 0x7f9b6c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102790 0x7f9b6c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c111f90 con 0x7f9b6c106f10 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c111e10 con 0x7f9b6c102790 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b72e9a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c112110 con 0x7f9b6c101de0 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b71437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b71437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60638/0 (socket says 172.21.15.16:60638) 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.245+0000 7f9b71437640 1 -- 172.21.15.16:0/1357920889 learned_addr learned my addr 172.21.15.16:0/1357920889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c101de0 0x7f9b6c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102790 0x7f9b6c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 -- 172.21.15.16:0/1357920889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c101de0 msgr2=0x7f9b6c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c101de0 0x7f9b6c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 -- 172.21.15.16:0/1357920889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102790 msgr2=0x7f9b6c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102790 0x7f9b6c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 -- 172.21.15.16:0/1357920889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c1a68e0 con 0x7f9b6c106f10 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b71437640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b970 tx=0x7f9b6400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b61ffb640 1 -- 172.21.15.16:0/1357920889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64019070 con 0x7f9b6c106f10 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b61ffb640 1 -- 172.21.15.16:0/1357920889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b64014440 con 0x7f9b6c106f10 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c1a6bd0 con 0x7f9b6c106f10 2024-09-12T02:53:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b61ffb640 1 -- 172.21.15.16:0/1357920889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b640075d0 con 0x7f9b6c106f10 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.246+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c1a7110 con 0x7f9b6c106f10 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b61ffb640 1 -- 172.21.15.16:0/1357920889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9b6401c070 con 0x7f9b6c106f10 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 Processor -- start 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 -- start start 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4882c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b61ffb640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 0x7f9b540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 0x556a48839630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556a48791620 con 0x556a4882c980 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a48738e50 con 0x556a48837240 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b75273740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556a4878d5e0 con 0x556a4882c060 2024-09-12T02:53:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b70c36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b70c36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60642/0 (socket says 172.21.15.16:60642) 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b70c36640 1 -- 172.21.15.16:0/2479216899 learned_addr learned my addr 172.21.15.16:0/2479216899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.249+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 0x7f9b540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b61ffb640 1 -- 172.21.15.16:0/1357920889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b640ce680 con 0x7f9b6c106f10 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b63fff640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4882c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b71437640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 0x556a48839630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 -- 172.21.15.16:0/2479216899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 msgr2=0x556a4882c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4882c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 -- 172.21.15.16:0/2479216899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 msgr2=0x556a48839630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 0x556a48839630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 -- 172.21.15.16:0/2479216899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a48729d40 con 0x556a4882c980 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b71437640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 0x556a48839630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b63fff640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4882c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800ad00 tx=0x7f9b6800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21f6f2c92fcac43 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 0x7f9b540a5040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c075f80 tx=0x7f9b5c0314e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b52ffd640 1 -- 172.21.15.16:0/2479216899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6801d070 con 0x556a4882c980 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b52ffd640 1 -- 172.21.15.16:0/2479216899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b68018b20 con 0x556a4882c980 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.250+0000 7f9b52ffd640 1 -- 172.21.15.16:0/2479216899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68010bc0 con 0x556a4882c980 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- 172.21.15.16:0/2479216899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 msgr2=0x556a48836d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800ad00 tx=0x7f9b6800f790 comp rx=0 tx=0).stop 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- 172.21.15.16:0/2479216899 shutdown_connections 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a48837240 0x556a48839630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48836d00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 --2- 172.21.15.16:0/2479216899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4882c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- 172.21.15.16:0/2479216899 >> 172.21.15.16:0/2479216899 conn(0x556a488285f0 msgr2=0x556a488289d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- 172.21.15.16:0/2479216899 shutdown_connections 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- 172.21.15.16:0/2479216899 wait complete. 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.251+0000 7f9b75273740 1 -- start start 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 Processor -- start 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c060 0x556a488340e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a48834b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556a48839e10 con 0x556a4882c980 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a48719f10 con 0x556a4882c060 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556a48839f90 con 0x556a48837240 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b75273740 10 client.0 _ll_register_callbacks cb 0x556a4873ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60646/0 (socket says 172.21.15.16:60646) 2024-09-12T02:53:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 -- 172.21.15.16:0/2634176933 learned_addr learned my addr 172.21.15.16:0/2634176933 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c060 0x556a488340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b71437640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a48834b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 -- 172.21.15.16:0/2634176933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 msgr2=0x556a48834b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a48834b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 -- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c060 msgr2=0x556a488340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c060 0x556a488340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.252+0000 7f9b63fff640 1 -- 172.21.15.16:0/2634176933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a484a0580 con 0x556a4882c980 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b63fff640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009900 tx=0x7f9b5c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65e96acde758163 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c04f070 con 0x556a4882c980 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b5c0035f0 con 0x556a4882c980 2024-09-12T02:53:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b68018b20 con 0x556a4882c980 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9b5c035040 con 0x556a4882c980 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b75273740 10 client.16869 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556a4856a960 con 0x556a4882c980 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b68018b20 con 0x556a4882c980 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9b5c03a040 con 0x556a4882c980 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.253+0000 7f9b75273740 10 client.16869 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b75273740 10 client.16869 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b75273740 20 client.16869 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a48846cb0 0x556a48849170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b75273740 4 client.16869 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a4879eb90 con 0x556a48846cb0 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.254+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a48846cb0 0x556a48849170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.255+0000 7f9b4bfff640 10 client.16869 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.255+0000 7f9b70c36640 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a48846cb0 0x556a48849170 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50675f76a3bc94d3 server_cookie=556a256701d3e2fc in_seq=0 out_seq=0 2024-09-12T02:53:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b4bfff640 1 -- 172.21.15.16:0/2634176933 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a4879eb90 con 0x556a48846cb0 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b4bfff640 10 client.16869 handle_command_reply: tid=0 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 client.16869 shutdown 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 msgr2=0x556a48834620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009900 tx=0x7f9b5c02a410 comp rx=0 tx=0).stop 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 shutdown_connections 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a48846cb0 0x556a48849170 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a48834b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c980 0x556a48834620 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 --2- 172.21.15.16:0/2634176933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c060 0x556a488340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 >> 172.21.15.16:0/2634176933 conn(0x556a488285f0 msgr2=0x556a4882e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b4bfff640 0 client.16869 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 shutdown_connections 2024-09-12T02:53:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 1 -- 172.21.15.16:0/2634176933 wait complete. 2024-09-12T02:53:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.256+0000 7f9b75273740 20 client.16869 trim_cache size 0 max 16384 2024-09-12T02:53:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 Processor -- start 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 -- start start 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a48836050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a4884ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1af730 con 0x556a4882c060 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1b1270 con 0x556a4882c980 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b75273740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1afe00 con 0x556a48837240 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b70c36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b70c36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60658/0 (socket says 172.21.15.16:60658) 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.273+0000 7f9b70c36640 1 -- 172.21.15.16:0/1336027399 learned_addr learned my addr 172.21.15.16:0/1336027399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b71437640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a4884ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a48836050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 -- 172.21.15.16:0/1336027399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 msgr2=0x556a4884ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a4884ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 -- 172.21.15.16:0/1336027399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 msgr2=0x556a48836050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a48836050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 -- 172.21.15.16:0/1336027399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a48827610 con 0x556a4882c060 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a48836050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b71437640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a4884ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800aac0 tx=0x7f9b68010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446ebb2471c712d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b527fc640 1 -- 172.21.15.16:0/1336027399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68031070 con 0x556a4882c060 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b527fc640 1 -- 172.21.15.16:0/1336027399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b6800a730 con 0x556a4882c060 2024-09-12T02:53:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.274+0000 7f9b527fc640 1 -- 172.21.15.16:0/1336027399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6801d040 con 0x556a4882c060 2024-09-12T02:53:58.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- 172.21.15.16:0/1336027399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 msgr2=0x556a4882f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800aac0 tx=0x7f9b68010800 comp rx=0 tx=0).stop 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- 172.21.15.16:0/1336027399 shutdown_connections 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a48837240 0x556a4884ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a48836050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 --2- 172.21.15.16:0/1336027399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a4882c060 0x556a4882f3e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- 172.21.15.16:0/1336027399 >> 172.21.15.16:0/1336027399 conn(0x556a488285f0 msgr2=0x556a48835940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- 172.21.15.16:0/1336027399 shutdown_connections 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- 172.21.15.16:0/1336027399 wait complete. 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.275+0000 7f9b75273740 1 -- start start 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 Processor -- start 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a4883bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1b0ba0 con 0x556a48837240 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1b04d0 con 0x556a4882c980 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1ad710 con 0x556a4882c060 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b75273740 10 client.0 _ll_register_callbacks cb 0x7f9b5c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b70c36640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60660/0 (socket says 172.21.15.16:60660) 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 -- 172.21.15.16:0/1512500013 learned_addr learned my addr 172.21.15.16:0/1512500013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b63fff640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a4883bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 -- 172.21.15.16:0/1512500013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 msgr2=0x556a4883ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 -- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 msgr2=0x556a4883bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a4883bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b71437640 1 -- 172.21.15.16:0/1512500013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4883f6d0 con 0x556a48837240 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.276+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b71437640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b940 tx=0x7f9b6409f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395fb7b310ac70e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b640149a0 con 0x556a48837240 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9b640bc2c0 con 0x556a48837240 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b6800a730 con 0x556a48837240 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9b640c01b0 con 0x556a48837240 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 10 client.16875 fetch_fsmap learned FSMap version 32 2024-09-12T02:53:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556a488b2610 con 0x556a48837240 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b6800a730 con 0x556a48837240 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9b6407b270 con 0x556a48837240 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 10 client.16875 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 10 client.16875 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 20 client.16875 populate_metadata read hostname 'smithi016' 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a488b2d80 0x556a488b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 4 client.16875 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "76fbe22d", "release": true, "await": true}] 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.277+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "76fbe22d", "release": true, "await": true}) -- 0x556a487990a0 con 0x556a488b2d80 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.278+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a488b2d80 0x556a488b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.278+0000 7f9b4bfff640 10 client.16875 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.278+0000 7f9b70c36640 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a488b2d80 0x556a488b5170 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69c3e229cf5c513a server_cookie=56b56f266854342b in_seq=0 out_seq=0 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b4bfff640 1 -- 172.21.15.16:0/1512500013 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556a487990a0 con 0x556a488b2d80 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b4bfff640 10 client.16875 handle_command_reply: tid=0 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 client.16875 shutdown 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 msgr2=0x556a4883f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b940 tx=0x7f9b6409f240 comp rx=0 tx=0).stop 2024-09-12T02:53:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 shutdown_connections 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556a488b2d80 0x556a488b5170 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a48837240 0x556a4883f030 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a4882c980 0x556a4883bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b4bfff640 0 client.16875 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 --2- 172.21.15.16:0/1512500013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a4882c060 0x556a4883ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 >> 172.21.15.16:0/1512500013 conn(0x556a488285f0 msgr2=0x556a488348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 shutdown_connections 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 1 -- 172.21.15.16:0/1512500013 wait complete. 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.287+0000 7f9b75273740 20 client.16875 trim_cache size 0 max 16384 2024-09-12T02:53:58.290 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 24, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "76fbe22d": { 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 24, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:53:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 140.1, 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:53:58.292 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:53:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 msgr2=0x7f9b540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 0x7f9b540a5040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c075f80 tx=0x7f9b5c0314e0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 msgr2=0x7f9b6c074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:53:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b970 tx=0x7f9b6400bd70 comp rx=0 tx=0).stop 2024-09-12T02:53:58.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 shutdown_connections 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9b540a2b80 0x7f9b540a5040 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b6c106f10 0x7f9b6c074f70 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9b6c102790 0x7f9b6c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 --2- 172.21.15.16:0/1357920889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b6c101de0 0x7f9b6c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 >> 172.21.15.16:0/1357920889 conn(0x7f9b6c0fd8c0 msgr2=0x7f9b6c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 shutdown_connections 2024-09-12T02:53:58.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:53:58.288+0000 7f9b72e9a640 1 -- 172.21.15.16:0/1357920889 wait complete. 2024-09-12T02:53:58.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 556928 10.14 MB/sec execute 180 sec latency 18922.516 ms 2024-09-12T02:53:59.093 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '76fbe22d', total seconds elapsed: 23.7 2024-09-12T02:53:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:53:59 smithi016 ceph-mon[30925]: pgmap v519: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-12T02:53:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:53:59 smithi033 ceph-mon[35092]: pgmap v519: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-12T02:53:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:53:59 smithi184 ceph-mon[35321]: pgmap v519: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-09-12T02:53:59.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559177 10.15 MB/sec execute 181 sec latency 16.334 ms 2024-09-12T02:54:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:54:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:54:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:54:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:54:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:54:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:54:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:54:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:54:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:54:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:54:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:54:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:54:00.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 561942 10.18 MB/sec execute 182 sec latency 11.389 ms 2024-09-12T02:54:01.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 564880 10.22 MB/sec execute 183 sec latency 70.468 ms 2024-09-12T02:54:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:01 smithi016 ceph-mon[30925]: pgmap v520: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-09-12T02:54:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:01 smithi033 ceph-mon[35092]: pgmap v520: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-09-12T02:54:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:01 smithi184 ceph-mon[35321]: pgmap v520: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-09-12T02:54:02.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 567243 10.21 MB/sec execute 184 sec latency 23.532 ms 2024-09-12T02:54:03.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 569596 10.23 MB/sec execute 185 sec latency 12.204 ms 2024-09-12T02:54:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:03 smithi016 ceph-mon[30925]: pgmap v521: 97 pgs: 97 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-09-12T02:54:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:03 smithi033 ceph-mon[35092]: pgmap v521: 97 pgs: 97 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-09-12T02:54:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:03 smithi184 ceph-mon[35321]: pgmap v521: 97 pgs: 97 active+clean; 215 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-09-12T02:54:04.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 572491 10.26 MB/sec execute 186 sec latency 11.445 ms 2024-09-12T02:54:05.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 575686 10.31 MB/sec execute 187 sec latency 15.957 ms 2024-09-12T02:54:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:05 smithi016 ceph-mon[30925]: pgmap v522: 97 pgs: 97 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 128 op/s 2024-09-12T02:54:06.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:05 smithi033 ceph-mon[35092]: pgmap v522: 97 pgs: 97 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 128 op/s 2024-09-12T02:54:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:05 smithi184 ceph-mon[35321]: pgmap v522: 97 pgs: 97 active+clean; 213 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 128 op/s 2024-09-12T02:54:06.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 578186 10.32 MB/sec execute 188 sec latency 75.950 ms 2024-09-12T02:54:07.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 580184 10.31 MB/sec execute 189 sec latency 41.825 ms 2024-09-12T02:54:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:07 smithi016 ceph-mon[30925]: pgmap v523: 97 pgs: 97 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-09-12T02:54:08.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:07 smithi033 ceph-mon[35092]: pgmap v523: 97 pgs: 97 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-09-12T02:54:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:07 smithi184 ceph-mon[35321]: pgmap v523: 97 pgs: 97 active+clean; 216 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-09-12T02:54:08.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 582999 10.35 MB/sec execute 190 sec latency 9.947 ms 2024-09-12T02:54:09.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 586183 10.39 MB/sec execute 191 sec latency 11.731 ms 2024-09-12T02:54:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:09 smithi016 ceph-mon[30925]: pgmap v524: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-09-12T02:54:10.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:09 smithi033 ceph-mon[35092]: pgmap v524: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-09-12T02:54:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:09 smithi184 ceph-mon[35321]: pgmap v524: 97 pgs: 97 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-09-12T02:54:10.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 588679 10.38 MB/sec execute 192 sec latency 18.493 ms 2024-09-12T02:54:11.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:54:11.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 590762 10.40 MB/sec execute 193 sec latency 38.322 ms 2024-09-12T02:54:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:12.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:11 smithi016 ceph-mon[30925]: pgmap v525: 97 pgs: 97 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:54:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:11 smithi033 ceph-mon[35092]: pgmap v525: 97 pgs: 97 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:54:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:11 smithi184 ceph-mon[35321]: pgmap v525: 97 pgs: 97 active+clean; 219 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- 172.21.15.16:0/95361963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd0810e4f0 msgr2=0x7efd08110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/95361963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd0810e4f0 0x7efd08110900 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd00009ca0 tx=0x7efd0002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- 172.21.15.16:0/95361963 shutdown_connections 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/95361963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd0810e4f0 0x7efd08110900 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/95361963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd08101380 0x7efd0810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/95361963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd081009d0 0x7efd08100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- 172.21.15.16:0/95361963 >> 172.21.15.16:0/95361963 conn(0x7efd080fc7d0 msgr2=0x7efd080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- 172.21.15.16:0/95361963 shutdown_connections 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- 172.21.15.16:0/95361963 wait complete. 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 Processor -- start 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.457+0000 7efd0e7d8640 1 -- start start 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 0x7efd0819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 0x7efd0819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd08114250 con 0x7efd0810e4f0 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd081140d0 con 0x7efd08101380 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0e7d8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd081143d0 con 0x7efd081009d0 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 0x7efd0819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0cd75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 0x7efd0819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 0x7efd0819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51468/0 (socket says 172.21.15.16:51468) 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0cd75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34968/0 (socket says 172.21.15.16:34968) 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd07fff640 1 -- 172.21.15.16:0/524273573 learned_addr learned my addr 172.21.15.16:0/524273573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 -- 172.21.15.16:0/524273573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 msgr2=0x7efd0819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 0x7efd0819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 -- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 msgr2=0x7efd0819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd077fe640 1 -- 172.21.15.16:0/524273573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0819f6c0 con 0x7efd08101380 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.458+0000 7efd0cd75640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd077fe640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 0x7efd0819ac50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e9f0 tx=0x7efcfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd057fa640 1 -- 172.21.15.16:0/524273573 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00ccc0 con 0x7efd08101380 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd057fa640 1 -- 172.21.15.16:0/524273573 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc005020 con 0x7efd08101380 2024-09-12T02:54:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd057fa640 1 -- 172.21.15.16:0/524273573 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0049e0 con 0x7efd08101380 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0819f9b0 con 0x7efd08101380 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.459+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd081a7290 con 0x7efd08101380 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.461+0000 7efd057fa640 1 -- 172.21.15.16:0/524273573 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efcfc005560 con 0x7efd08101380 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.461+0000 7efd10bb1740 1 Processor -- start 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.461+0000 7efd10bb1740 1 -- start start 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a132680 0x563e1a134a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563e1a08d2e0 con 0x563e1a132680 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563e1a034ce0 con 0x563e1a127dc0 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd10bb1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563e1a0892a0 con 0x563e1a1274a0 2024-09-12T02:54:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a132680 0x563e1a134a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd07fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51470/0 (socket says 172.21.15.16:51470) 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 -- 172.21.15.16:0/2579174582 learned_addr learned my addr 172.21.15.16:0/2579174582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 -- 172.21.15.16:0/2579174582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 msgr2=0x563e1a132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 -- 172.21.15.16:0/2579174582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a132680 msgr2=0x563e1a134a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a132680 0x563e1a134a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 -- 172.21.15.16:0/2579174582 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1a019440 con 0x563e1a1274a0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd057fa640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 0x7efcf00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.462+0000 7efd0cd75640 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd00031f80 tx=0x7efd00005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e2de09d734a989 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efd07fff640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 0x7efcf00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efd057fa640 1 -- 172.21.15.16:0/524273573 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efcfc0ceeb0 con 0x7efd08101380 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efcee7fc640 1 -- 172.21.15.16:0/2579174582 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00003b70 con 0x563e1a1274a0 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efcee7fc640 1 -- 172.21.15.16:0/2579174582 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd00036040 con 0x563e1a1274a0 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efcee7fc640 1 -- 172.21.15.16:0/2579174582 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00004560 con 0x563e1a1274a0 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efd10bb1740 1 -- 172.21.15.16:0/2579174582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 msgr2=0x563e1a127880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efd07fff640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 0x7efcf00a50c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x563e1a124890 tx=0x7efcf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.463+0000 7efd10bb1740 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd00031f80 tx=0x7efd00005ef0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- 172.21.15.16:0/2579174582 shutdown_connections 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a132680 0x563e1a134a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- 172.21.15.16:0/2579174582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a127880 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- 172.21.15.16:0/2579174582 >> 172.21.15.16:0/2579174582 conn(0x563e1a123a30 msgr2=0x563e1a123e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- 172.21.15.16:0/2579174582 shutdown_connections 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- 172.21.15.16:0/2579174582 wait complete. 2024-09-12T02:54:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- start start 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 Processor -- start 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a131f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a132680 0x563e1a12f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.464+0000 7efd10bb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563e1a1350d0 con 0x563e1a127dc0 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd10bb1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563e1a048050 con 0x563e1a132680 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd10bb1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563e1a135250 con 0x563e1a1274a0 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd10bb1740 10 client.0 _ll_register_callbacks cb 0x563e1a088160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a131f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34996/0 (socket says 172.21.15.16:34996) 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a131f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51478/0 (socket says 172.21.15.16:51478) 2024-09-12T02:54:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 -- 172.21.15.16:0/3260179757 learned_addr learned my addr 172.21.15.16:0/3260179757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd0cd75640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a132680 0x563e1a12f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 -- 172.21.15.16:0/3260179757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 msgr2=0x563e1a131f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a131f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 -- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a132680 msgr2=0x563e1a12f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a132680 0x563e1a12f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 -- 172.21.15.16:0/3260179757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e19dfe3c0 con 0x563e1a127dc0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd077fe640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7efcfc098090 tx=0x7efcfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c14667e0e5a8610 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc09b020 con 0x563e1a127dc0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efcfc09b920 con 0x563e1a127dc0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.465+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd00036040 con 0x563e1a127dc0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efcfc0051d0 con 0x563e1a127dc0 2024-09-12T02:54:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 10 client.16887 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e1a140a80 con 0x563e1a127dc0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd00036040 con 0x563e1a127dc0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efcfc0a3ea0 con 0x563e1a127dc0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 10 client.16887 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 10 client.16887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 20 client.16887 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 4 client.16887 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e1a09a850 con 0x563e1a141dd0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.466+0000 7efd07fff640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.467+0000 7efd07fff640 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1441c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da467c4f2a1972 server_cookie=f18af6db6e8177d9 in_seq=0 out_seq=0 2024-09-12T02:54:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.467+0000 7efce77fe640 10 client.16887 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.467+0000 7efce77fe640 1 -- 172.21.15.16:0/3260179757 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e1a09a850 con 0x563e1a141dd0 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.467+0000 7efce77fe640 10 client.16887 handle_command_reply: tid=0 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 client.16887 shutdown 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 msgr2=0x563e1a12f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7efcfc098090 tx=0x7efcfc004730 comp rx=0 tx=0).stop 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 shutdown_connections 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1441c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a132680 0x563e1a12f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efce77fe640 0 client.16887 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a12f1f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3260179757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a1274a0 0x563e1a131f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 >> 172.21.15.16:0/3260179757 conn(0x563e1a123a30 msgr2=0x563e1a129710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 shutdown_connections 2024-09-12T02:54:12.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 1 -- 172.21.15.16:0/3260179757 wait complete. 2024-09-12T02:54:12.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.468+0000 7efd10bb1740 20 client.16887 trim_cache size 0 max 16384 2024-09-12T02:54:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 Processor -- start 2024-09-12T02:54:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 -- start start 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 0x563e1a13e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a131160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd081af680 con 0x563e1a1274a0 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd081b11c0 con 0x563e1a127dc0 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd10bb1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd081afd50 con 0x563e1a132680 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 0x563e1a13e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 0x563e1a13e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35004/0 (socket says 172.21.15.16:35004) 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd07fff640 1 -- 172.21.15.16:0/681112647 learned_addr learned my addr 172.21.15.16:0/681112647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60548/0 (socket says 172.21.15.16:60548) 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd0cd75640 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a131160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 -- 172.21.15.16:0/681112647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 msgr2=0x563e1a131160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a131160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 -- 172.21.15.16:0/681112647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 msgr2=0x563e1a13e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 0x563e1a13e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd077fe640 1 -- 172.21.15.16:0/681112647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e1a12e1a0 con 0x563e1a127dc0 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.486+0000 7efd0cd75640 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a131160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd077fe640 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efcfc097b20 tx=0x7efcfc0054d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2702e4d9c2262d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efcedffb640 1 -- 172.21.15.16:0/681112647 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc09b490 con 0x563e1a127dc0 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efcedffb640 1 -- 172.21.15.16:0/681112647 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc09bdc0 con 0x563e1a127dc0 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efcedffb640 1 -- 172.21.15.16:0/681112647 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0a3ea0 con 0x563e1a127dc0 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd10bb1740 1 -- 172.21.15.16:0/681112647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 msgr2=0x563e1a123e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd10bb1740 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efcfc097b20 tx=0x7efcfc0054d0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd10bb1740 1 -- 172.21.15.16:0/681112647 shutdown_connections 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd10bb1740 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a131160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.487+0000 7efd10bb1740 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a127dc0 0x563e1a123e10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 --2- 172.21.15.16:0/681112647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a1274a0 0x563e1a13e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- 172.21.15.16:0/681112647 >> 172.21.15.16:0/681112647 conn(0x563e1a123a30 msgr2=0x563e1a130a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- 172.21.15.16:0/681112647 shutdown_connections 2024-09-12T02:54:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- 172.21.15.16:0/681112647 wait complete. 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- start start 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 Processor -- start 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 0x563e1a13a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a136c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd081b0af0 con 0x563e1a127dc0 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd081b0420 con 0x563e1a1274a0 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd081ad660 con 0x563e1a132680 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.488+0000 7efd10bb1740 10 client.0 _ll_register_callbacks cb 0x7efcfc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35018/0 (socket says 172.21.15.16:35018) 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 -- 172.21.15.16:0/3337007055 learned_addr learned my addr 172.21.15.16:0/3337007055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:12.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd07fff640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 0x563e1a13a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd0cd75640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 -- 172.21.15.16:0/3337007055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 msgr2=0x563e1a136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a136c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 -- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 msgr2=0x563e1a13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 0x563e1a13a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 -- 172.21.15.16:0/3337007055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e1a1322b0 con 0x563e1a127dc0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd07fff640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 0x563e1a13a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd077fe640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0a3e00 tx=0x7efcfc09bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5986529f9d4839a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc09b490 con 0x563e1a127dc0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efcfc0afbe0 con 0x563e1a127dc0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcfc09bdc0 con 0x563e1a127dc0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efcfc0a4330 con 0x563e1a127dc0 2024-09-12T02:54:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd10bb1740 10 client.16893 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.489+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e1a1ad460 con 0x563e1a127dc0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc09bdc0 con 0x563e1a127dc0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efcfc09a740 con 0x563e1a127dc0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 10 client.16893 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 10 client.16893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 20 client.16893 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1afa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 4 client.16893 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563e1a094d60 con 0x563e1a141dd0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.490+0000 7efd07fff640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd07fff640 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1afa30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92dcc73a6dc6e899 server_cookie=ee0444b4fc68419a in_seq=0 out_seq=0 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efce77fe640 10 client.16893 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efce77fe640 1 -- 172.21.15.16:0/3337007055 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563e1a094d60 con 0x563e1a141dd0 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efce77fe640 10 client.16893 handle_command_reply: tid=0 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 client.16893 shutdown 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 msgr2=0x563e1a13ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0a3e00 tx=0x7efcfc09bbf0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 shutdown_connections 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563e1a141dd0 0x563e1a1afa30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e1a132680 0x563e1a136c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e1a127dc0 0x563e1a13ad30 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 --2- 172.21.15.16:0/3337007055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e1a1274a0 0x563e1a13a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 >> 172.21.15.16:0/3337007055 conn(0x563e1a123a30 msgr2=0x563e1a122350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efce77fe640 0 client.16893 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 shutdown_connections 2024-09-12T02:54:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 1 -- 172.21.15.16:0/3337007055 wait complete. 2024-09-12T02:54:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.491+0000 7efd10bb1740 20 client.16893 trim_cache size 0 max 16384 2024-09-12T02:54:12.498 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:12.498 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:54:12.498 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:54:12.498 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 msgr2=0x7efcf00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 0x7efcf00a50c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x563e1a124890 tx=0x7efcf8005f90 comp rx=0 tx=0).stop 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 msgr2=0x7efd0819ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 0x7efd0819ac50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e9f0 tx=0x7efcfc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 shutdown_connections 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcf00a2c00 0x7efcf00a50c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd0810e4f0 0x7efd0819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd08101380 0x7efd0819ac50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 --2- 172.21.15.16:0/524273573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd081009d0 0x7efd0819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 >> 172.21.15.16:0/524273573 conn(0x7efd080fc7d0 msgr2=0x7efd080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 shutdown_connections 2024-09-12T02:54:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:12.492+0000 7efd0e7d8640 1 -- 172.21.15.16:0/524273573 wait complete. 2024-09-12T02:54:12.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 593545 10.43 MB/sec execute 194 sec latency 20.330 ms 2024-09-12T02:54:13.289 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:54:13.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:54:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:13.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 596540 10.45 MB/sec execute 195 sec latency 19.374 ms 2024-09-12T02:54:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:14 smithi184 ceph-mon[35321]: pgmap v526: 97 pgs: 97 active+clean; 221 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-09-12T02:54:14.339 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:14 smithi016 ceph-mon[30925]: pgmap v526: 97 pgs: 97 active+clean; 221 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-09-12T02:54:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 -- 172.21.15.16:0/1545385807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a810c190 msgr2=0x7f53a810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/1545385807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a810c190 0x7f53a810e610 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ca0 tx=0x7f53a002a950 comp rx=0 tx=0).stop 2024-09-12T02:54:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 -- 172.21.15.16:0/1545385807 shutdown_connections 2024-09-12T02:54:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/1545385807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a810c190 0x7f53a810e610 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/1545385807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a81036d0 0x7f53a810bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/1545385807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a8102db0 0x7f53a8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 -- 172.21.15.16:0/1545385807 >> 172.21.15.16:0/1545385807 conn(0x7f53a80fc7d0 msgr2=0x7f53a80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 -- 172.21.15.16:0/1545385807 shutdown_connections 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 -- 172.21.15.16:0/1545385807 wait complete. 2024-09-12T02:54:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.341+0000 7f53ad2a9640 1 Processor -- start 2024-09-12T02:54:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 -- start start 2024-09-12T02:54:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 0x7f53a8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a810c190 0x7f53a819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8111f70 con 0x7f53a8102db0 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8111df0 con 0x7f53a81036d0 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53ad2a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81120f0 con 0x7f53a810c190 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 0x7f53a8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35030/0 (socket says 172.21.15.16:35030) 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a6ffd640 1 -- 172.21.15.16:0/172302295 learned_addr learned my addr 172.21.15.16:0/172302295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a77fe640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a810c190 0x7f53a819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 -- 172.21.15.16:0/172302295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a810c190 msgr2=0x7f53a819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a810c190 0x7f53a819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 -- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 msgr2=0x7f53a8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 -- 172.21.15.16:0/172302295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a819d400 con 0x7f53a81036d0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a6ffd640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.342+0000 7f53a67fc640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 0x7f53a8198990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f539c00e9f0 tx=0x7f539c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.343+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c00ccc0 con 0x7f53a81036d0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.343+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f539c010910 con 0x7f53a81036d0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.343+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a819d6f0 con 0x7f53a81036d0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.343+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c018960 con 0x7f53a81036d0 2024-09-12T02:54:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.343+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81a4fd0 con 0x7f53a81036d0 2024-09-12T02:54:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.344+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a8103190 con 0x7f53a81036d0 2024-09-12T02:54:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.345+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f539c010ac0 con 0x7f53a81036d0 2024-09-12T02:54:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.346+0000 7f538ffff640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 0x7f53900a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.346+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f539c0cf2e0 con 0x7f53a81036d0 2024-09-12T02:54:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.346+0000 7f53a6ffd640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 0x7f53900a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.347+0000 7f53a6ffd640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 0x7f53900a50c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5398011550 tx=0x7f5398005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.351+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f539c09bc00 con 0x7f53a81036d0 2024-09-12T02:54:14.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:14 smithi033 ceph-mon[35092]: pgmap v526: 97 pgs: 97 active+clean; 221 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 281 op/s 2024-09-12T02:54:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.587+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53a8107230 con 0x7f53a81036d0 2024-09-12T02:54:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.587+0000 7f538ffff640 1 -- 172.21.15.16:0/172302295 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f539c09ff00 con 0x7f53a81036d0 2024-09-12T02:54:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:54:14.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:54:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 msgr2=0x7f53900a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 0x7f53900a50c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5398011550 tx=0x7f5398005f90 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 msgr2=0x7f53a8198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 0x7f53a8198990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f539c00e9f0 tx=0x7f539c00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 shutdown_connections 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a810c190 0x7f53a819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53a81036d0 0x7f53a8198990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f53900a2c00 0x7f53900a50c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 --2- 172.21.15.16:0/172302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a8102db0 0x7f53a8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.590+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 >> 172.21.15.16:0/172302295 conn(0x7f53a80fc7d0 msgr2=0x7f53a80fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.591+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 shutdown_connections 2024-09-12T02:54:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:14.591+0000 7f53ad2a9640 1 -- 172.21.15.16:0/172302295 wait complete. 2024-09-12T02:54:14.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 598603 10.45 MB/sec execute 196 sec latency 40.465 ms 2024-09-12T02:54:15.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:15 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/172302295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:15.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:54:15.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:15 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/172302295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:15.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:15 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/172302295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:15.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 600965 10.47 MB/sec execute 197 sec latency 62.976 ms 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 -- 172.21.15.16:0/2345213594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50102d90 msgr2=0x7fec50103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 --2- 172.21.15.16:0/2345213594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50102d90 0x7fec50103170 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009930 tx=0x7fec4c02a420 comp rx=0 tx=0).stop 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 -- 172.21.15.16:0/2345213594 shutdown_connections 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 --2- 172.21.15.16:0/2345213594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 0x7fec5010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 --2- 172.21.15.16:0/2345213594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50103740 0x7fec50107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 --2- 172.21.15.16:0/2345213594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50102d90 0x7fec50103170 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 -- 172.21.15.16:0/2345213594 >> 172.21.15.16:0/2345213594 conn(0x7fec500fc7d0 msgr2=0x7fec500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 -- 172.21.15.16:0/2345213594 shutdown_connections 2024-09-12T02:54:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.070+0000 7fec57a98640 1 -- 172.21.15.16:0/2345213594 wait complete. 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 Processor -- start 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 -- start start 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 0x7fec501009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 0x7fec50101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec50112090 con 0x7fec50103740 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec50111f10 con 0x7fec50102d90 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.071+0000 7fec57a98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec50112210 con 0x7fec50108290 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41434/0 (socket says 172.21.15.16:41434) 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55834640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 0x7fec501009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 -- 172.21.15.16:0/1668401334 learned_addr learned my addr 172.21.15.16:0/1668401334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec56035640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 0x7fec50101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 -- 172.21.15.16:0/1668401334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 msgr2=0x7fec50101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 0x7fec50101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 -- 172.21.15.16:0/1668401334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 msgr2=0x7fec501009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 0x7fec501009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 -- 172.21.15.16:0/1668401334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec501a6ab0 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55834640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 0x7fec501009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec55033640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fec4000e9c0 tx=0x7fec4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec46ffd640 1 -- 172.21.15.16:0/1668401334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4000cd80 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec46ffd640 1 -- 172.21.15.16:0/1668401334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec40005500 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec501a6da0 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec46ffd640 1 -- 172.21.15.16:0/1668401334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40010690 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.072+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec501a7260 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.075+0000 7fec46ffd640 1 -- 172.21.15.16:0/1668401334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fec400046c0 con 0x7fec50103740 2024-09-12T02:54:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.075+0000 7fec59e71740 1 Processor -- start 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.075+0000 7fec59e71740 1 -- start start 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6564522f0 con 0x55f6564edb40 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6563f9cf0 con 0x55f6564ec5d0 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec59e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f65644e2b0 con 0x55f6564f7370 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec56035640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec56035640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41448/0 (socket says 172.21.15.16:41448) 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec56035640 1 -- 172.21.15.16:0/2369015085 learned_addr learned my addr 172.21.15.16:0/2369015085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec46ffd640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 0x7fec3c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55033640 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 -- 172.21.15.16:0/2369015085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 msgr2=0x55f6564f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec56035640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 0x7fec3c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 -- 172.21.15.16:0/2369015085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 msgr2=0x55f6564ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec55834640 1 -- 172.21.15.16:0/2369015085 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6563de3a0 con 0x55f6564ec5d0 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.076+0000 7fec46ffd640 1 -- 172.21.15.16:0/1668401334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec400cf320 con 0x7fec50103740 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.077+0000 7fec55834640 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564eca10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005940 tx=0x7fec4c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5782c8d5b894f039 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.077+0000 7fec56035640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 0x7fec3c0a50c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fec480049d0 tx=0x7fec48004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.077+0000 7fec3bfff640 1 -- 172.21.15.16:0/2369015085 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c035040 con 0x55f6564ec5d0 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.077+0000 7fec3bfff640 1 -- 172.21.15.16:0/2369015085 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec4c031870 con 0x55f6564ec5d0 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.077+0000 7fec3bfff640 1 -- 172.21.15.16:0/2369015085 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0047a0 con 0x55f6564ec5d0 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- 172.21.15.16:0/2369015085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 msgr2=0x55f6564eca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564eca10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005940 tx=0x7fec4c005c00 comp rx=0 tx=0).stop 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- 172.21.15.16:0/2369015085 shutdown_connections 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564eca10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 --2- 172.21.15.16:0/2369015085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- 172.21.15.16:0/2369015085 >> 172.21.15.16:0/2369015085 conn(0x55f6564b5310 msgr2=0x55f6564e8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- 172.21.15.16:0/2369015085 shutdown_connections 2024-09-12T02:54:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- 172.21.15.16:0/2369015085 wait complete. 2024-09-12T02:54:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 -- start start 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.078+0000 7fec59e71740 1 Processor -- start 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 0x55f6564f4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6564fa010 con 0x55f6564edb40 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f65640d060 con 0x55f6564f7370 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f6564fa190 con 0x55f6564ec5d0 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec59e71740 10 client.0 _ll_register_callbacks cb 0x55f6561608c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41458/0 (socket says 172.21.15.16:41458) 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 -- 172.21.15.16:0/161294038 learned_addr learned my addr 172.21.15.16:0/161294038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec56035640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 0x55f6564f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 -- 172.21.15.16:0/161294038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 msgr2=0x55f6564f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55834640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 -- 172.21.15.16:0/161294038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 msgr2=0x55f6564f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 0x55f6564f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 -- 172.21.15.16:0/161294038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6561c33c0 con 0x55f6564edb40 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec56035640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 0x55f6564f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55834640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec55033640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fec40098560 tx=0x7fec4009b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cd7ce154519e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec397fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.079+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4009bff0 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec4009fab0 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec4c031870 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec4009c290 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 10 client.16911 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f656505aa0 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec4c031870 con 0x55f6564edb40 2024-09-12T02:54:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fec400ce030 con 0x55f6564edb40 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 10 client.16911 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 10 client.16911 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 20 client.16911 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f656506f60 0x55f656509390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 4 client.16911 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f65645f860 con 0x55f656506f60 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.080+0000 7fec55834640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f656506f60 0x55f656509390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.081+0000 7fec397fa640 10 client.16911 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.081+0000 7fec55834640 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f656506f60 0x55f656509390 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6df3dde6930cd128 server_cookie=9a8754e66b92e82e in_seq=0 out_seq=0 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec397fa640 1 -- 172.21.15.16:0/161294038 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f65645f860 con 0x55f656506f60 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec397fa640 10 client.16911 handle_command_reply: tid=0 2024-09-12T02:54:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 client.16911 shutdown 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 msgr2=0x55f6564f3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fec40098560 tx=0x7fec4009b790 comp rx=0 tx=0).stop 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 shutdown_connections 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f656506f60 0x55f656509390 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564f7370 0x55f6564f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f3fa0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec397fa640 0 client.16911 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 --2- 172.21.15.16:0/161294038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 >> 172.21.15.16:0/161294038 conn(0x55f6564b5310 msgr2=0x55f6564ee960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 shutdown_connections 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 1 -- 172.21.15.16:0/161294038 wait complete. 2024-09-12T02:54:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.082+0000 7fec59e71740 20 client.16911 trim_cache size 0 max 16384 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 Processor -- start 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 -- start start 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564e8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564edb40 0x55f6565032b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec501af880 con 0x55f6564f7370 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec501b13c0 con 0x55f6564edb40 2024-09-12T02:54:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec59e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec501aff50 con 0x55f6564ec5d0 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec56035640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec55834640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564e8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec56035640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41466/0 (socket says 172.21.15.16:41466) 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec56035640 1 -- 172.21.15.16:0/2088757325 learned_addr learned my addr 172.21.15.16:0/2088757325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.096+0000 7fec55033640 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564edb40 0x55f6565032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 -- 172.21.15.16:0/2088757325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 msgr2=0x55f6564e8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564e8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 -- 172.21.15.16:0/2088757325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564edb40 msgr2=0x55f6565032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564edb40 0x55f6565032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 -- 172.21.15.16:0/2088757325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6564f4890 con 0x55f6564f7370 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec55834640 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec56035640 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fec48013270 tx=0x7fec48005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0222d7b4729c800 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec3b7fe640 1 -- 172.21.15.16:0/2088757325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48004b20 con 0x55f6564f7370 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec3b7fe640 1 -- 172.21.15.16:0/2088757325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec48010e10 con 0x55f6564f7370 2024-09-12T02:54:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.097+0000 7fec3b7fe640 1 -- 172.21.15.16:0/2088757325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48005560 con 0x55f6564f7370 2024-09-12T02:54:16.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- 172.21.15.16:0/2088757325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 msgr2=0x55f6564f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fec48013270 tx=0x7fec48005bd0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- 172.21.15.16:0/2088757325 shutdown_connections 2024-09-12T02:54:16.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564f7370 0x55f6564f3860 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564edb40 0x55f6565032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 --2- 172.21.15.16:0/2088757325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564ec5d0 0x55f6564e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- 172.21.15.16:0/2088757325 >> 172.21.15.16:0/2088757325 conn(0x55f6564b5310 msgr2=0x55f6564e8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- 172.21.15.16:0/2088757325 shutdown_connections 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- 172.21.15.16:0/2088757325 wait complete. 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 -- start start 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.098+0000 7fec59e71740 1 Processor -- start 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec501b0cf0 con 0x55f6564edb40 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec501b0620 con 0x55f6564ec5d0 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec501ad860 con 0x55f6564f7370 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec59e71740 10 client.0 _ll_register_callbacks cb 0x7fec40009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec55033640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41480/0 (socket says 172.21.15.16:41480) 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec55033640 1 -- 172.21.15.16:0/2008566888 learned_addr learned my addr 172.21.15.16:0/2008566888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec56035640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.099+0000 7fec55834640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 -- 172.21.15.16:0/2008566888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 msgr2=0x55f6564f1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 -- 172.21.15.16:0/2008566888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 msgr2=0x55f6564ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 -- 172.21.15.16:0/2008566888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f656570c30 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec55033640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fec4009b890 tx=0x7fec40097e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3ce9dccb4b84ba server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.101+0000 7fec397fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4009ff40 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec400a10d0 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec48010e10 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec4009bb90 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec59e71740 10 client.16917 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.102+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f656571d90 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec48010e10 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fec40081030 con 0x55f6564edb40 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 10 client.16917 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:16.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 10 client.16917 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 20 client.16917 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f6565722a0 0x55f6565746c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 4 client.16917 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f656459d70 con 0x55f6565722a0 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.103+0000 7fec55834640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f6565722a0 0x55f6565746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec397fa640 10 client.16917 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec55834640 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f6565722a0 0x55f6565746c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=29bede48bcec0893 server_cookie=51cd702ec5cbd518 in_seq=0 out_seq=0 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec397fa640 1 -- 172.21.15.16:0/2008566888 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f656459d70 con 0x55f6565722a0 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec397fa640 10 client.16917 handle_command_reply: tid=0 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec59e71740 1 client.16917 shutdown 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 msgr2=0x55f6564f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.104+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fec4009b890 tx=0x7fec40097e30 comp rx=0 tx=0).stop 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 shutdown_connections 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6564f7370 0x55f6564f1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6564edb40 0x55f6564f0e50 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55f6565722a0 0x55f6565746c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 --2- 172.21.15.16:0/2008566888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6564ec5d0 0x55f6564ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 >> 172.21.15.16:0/2008566888 conn(0x55f6564b5310 msgr2=0x55f6564f5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec397fa640 0 client.16917 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 shutdown_connections 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 1 -- 172.21.15.16:0/2008566888 wait complete. 2024-09-12T02:54:16.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.105+0000 7fec59e71740 20 client.16917 trim_cache size 0 max 16384 2024-09-12T02:54:16.115 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:54:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 msgr2=0x7fec3c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 0x7fec3c0a50c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fec480049d0 tx=0x7fec48004ab0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 msgr2=0x7fec50100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fec4000e9c0 tx=0x7fec4000ee80 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 shutdown_connections 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec50108290 0x7fec50101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec3c0a2c00 0x7fec3c0a50c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec50103740 0x7fec50100f30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 --2- 172.21.15.16:0/1668401334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec50102d90 0x7fec501009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.106+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 >> 172.21.15.16:0/1668401334 conn(0x7fec500fc7d0 msgr2=0x7fec50106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.107+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 shutdown_connections 2024-09-12T02:54:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:16.107+0000 7fec57a98640 1 -- 172.21.15.16:0/1668401334 wait complete. 2024-09-12T02:54:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:16 smithi184 ceph-mon[35321]: pgmap v527: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 275 op/s 2024-09-12T02:54:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:16 smithi016 ceph-mon[30925]: pgmap v527: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 275 op/s 2024-09-12T02:54:16.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:16 smithi033 ceph-mon[35092]: pgmap v527: 97 pgs: 97 active+clean; 222 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 275 op/s 2024-09-12T02:54:16.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 603918 10.50 MB/sec execute 198 sec latency 44.602 ms 2024-09-12T02:54:16.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:54:17.184 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- 172.21.15.16:0/4110235349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330100a80 msgr2=0x7f0330100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 --2- 172.21.15.16:0/4110235349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330100a80 0x7f0330100ee0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0320009930 tx=0x7f032002a420 comp rx=0 tx=0).stop 2024-09-12T02:54:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- 172.21.15.16:0/4110235349 shutdown_connections 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 --2- 172.21.15.16:0/4110235349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330101420 0x7f033010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 --2- 172.21.15.16:0/4110235349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330100a80 0x7f0330100ee0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 --2- 172.21.15.16:0/4110235349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330106a80 0x7f0330106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- 172.21.15.16:0/4110235349 >> 172.21.15.16:0/4110235349 conn(0x7f03300fc7a0 msgr2=0x7f03300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- 172.21.15.16:0/4110235349 shutdown_connections 2024-09-12T02:54:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- 172.21.15.16:0/4110235349 wait complete. 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 Processor -- start 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.665+0000 7f03385be640 1 -- start start 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 0x7f03301983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330106a80 0x7f033019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0330110e40 con 0x7f0330101420 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0330110cc0 con 0x7f0330100a80 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f03385be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0330110fc0 con 0x7f0330106a80 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f033635a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 0x7f03301983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41496/0 (socket says 172.21.15.16:41496) 2024-09-12T02:54:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 -- 172.21.15.16:0/1909805716 learned_addr learned my addr 172.21.15.16:0/1909805716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0336b5b640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330106a80 0x7f033019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 -- 172.21.15.16:0/1909805716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330106a80 msgr2=0x7f033019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330106a80 0x7f033019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 -- 172.21.15.16:0/1909805716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 msgr2=0x7f03301983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 0x7f03301983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 -- 172.21.15.16:0/1909805716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033019d3a0 con 0x7f0330101420 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f033635a640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 0x7f03301983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.666+0000 7f0335b59640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0320009230 tx=0x7f0320004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.667+0000 7f03277fe640 1 -- 172.21.15.16:0/1909805716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032003f030 con 0x7f0330101420 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.667+0000 7f03277fe640 1 -- 172.21.15.16:0/1909805716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0320003ad0 con 0x7f0330101420 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.667+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033019d630 con 0x7f0330101420 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.667+0000 7f03277fe640 1 -- 172.21.15.16:0/1909805716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320035070 con 0x7f0330101420 2024-09-12T02:54:17.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.667+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03301a4f70 con 0x7f0330101420 2024-09-12T02:54:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.669+0000 7f03277fe640 1 -- 172.21.15.16:0/1909805716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0320042030 con 0x7f0330101420 2024-09-12T02:54:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 Processor -- start 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 -- start start 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eaf8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 0x560a7eb05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560a7ea5e2f0 con 0x560a7eaf9b40 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560a7ea05cf0 con 0x560a7eb03370 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f033a997740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560a7ea5a2b0 con 0x560a7eaf85d0 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f0336b5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f0336b5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41510/0 (socket says 172.21.15.16:41510) 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f03277fe640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 0x7f031c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f0335b59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 0x560a7eb05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f0336b5b640 1 -- 172.21.15.16:0/104981108 learned_addr learned my addr 172.21.15.16:0/104981108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.670+0000 7f0336b5b640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 0x7f031c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f033635a640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eaf8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 -- 172.21.15.16:0/104981108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 msgr2=0x560a7eaf8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eaf8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 -- 172.21.15.16:0/104981108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 msgr2=0x560a7eaf8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 -- 172.21.15.16:0/104981108 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a7e9ea3a0 con 0x560a7eb03370 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f033635a640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eaf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0336b5b640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f03277fe640 1 -- 172.21.15.16:0/1909805716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f03200ef500 con 0x7f0330101420 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0335b59640 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 0x560a7eb05760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f03200b8790 tx=0x7f032004b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdc209c3a52999b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0336b5b640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 0x7f031c0a5040 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0328009fd0 tx=0x7f0328004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0324ff9640 1 -- 172.21.15.16:0/104981108 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320049070 con 0x560a7eb03370 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0324ff9640 1 -- 172.21.15.16:0/104981108 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03200c00c0 con 0x560a7eb03370 2024-09-12T02:54:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.671+0000 7f0324ff9640 1 -- 172.21.15.16:0/104981108 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200b9e60 con 0x560a7eb03370 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 -- 172.21.15.16:0/104981108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 msgr2=0x560a7eb05760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 0x560a7eb05760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f03200b8790 tx=0x7f032004b000 comp rx=0 tx=0).stop 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 -- 172.21.15.16:0/104981108 shutdown_connections 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eb03370 0x560a7eb05760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eaf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 --2- 172.21.15.16:0/104981108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eaf8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 -- 172.21.15.16:0/104981108 >> 172.21.15.16:0/104981108 conn(0x560a7eac1310 msgr2=0x560a7eaf4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 -- 172.21.15.16:0/104981108 shutdown_connections 2024-09-12T02:54:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.672+0000 7f033a997740 1 -- 172.21.15.16:0/104981108 wait complete. 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 -- start start 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 Processor -- start 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eb03370 0x560a7eb005d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560a7eb05e30 con 0x560a7eaf9b40 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560a7ea19060 con 0x560a7eaf85d0 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560a7eb05fb0 con 0x560a7eb03370 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033a997740 10 client.0 _ll_register_callbacks cb 0x560a7e76c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033635a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033635a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39446/0 (socket says 172.21.15.16:39446) 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41526/0 (socket says 172.21.15.16:41526) 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033635a640 1 -- 172.21.15.16:0/3635743845 learned_addr learned my addr 172.21.15.16:0/3635743845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0336b5b640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eb03370 0x560a7eb005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 -- 172.21.15.16:0/3635743845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eb03370 msgr2=0x560a7eb005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eb03370 0x560a7eb005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 -- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 msgr2=0x560a7eb02d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f0335b59640 1 -- 172.21.15.16:0/3635743845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a7e7cf3c0 con 0x560a7eaf9b40 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.673+0000 7f033635a640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0335b59640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f032003b660 tx=0x7f0320005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711ab6b95d3b7a62 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320049040 con 0x560a7eaf9b40 2024-09-12T02:54:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f03200ba0c0 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560a7ea6b860 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03200c4980 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f033a997740 10 client.16929 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a7eb05e30 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560a7ea6b860 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f03200ef0f0 con 0x560a7eaf9b40 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f033a997740 10 client.16929 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.674+0000 7f033a997740 10 client.16929 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033a997740 20 client.16929 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb12a30 0x560a7eb14e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033a997740 4 client.16929 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a7ea65d70 con 0x560a7eb12a30 2024-09-12T02:54:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033635a640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb12a30 0x560a7eb14e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f0319ffb640 10 client.16929 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.675+0000 7f033635a640 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb12a30 0x560a7eb14e20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ad019d1f55e14842 server_cookie=2ec0ea523bac64f5 in_seq=0 out_seq=0 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f0319ffb640 1 -- 172.21.15.16:0/3635743845 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560a7ea65d70 con 0x560a7eb12a30 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f0319ffb640 10 client.16929 handle_command_reply: tid=0 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 client.16929 shutdown 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 msgr2=0x560a7eb00000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f032003b660 tx=0x7f0320005e50 comp rx=0 tx=0).stop 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 shutdown_connections 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb12a30 0x560a7eb14e20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eb03370 0x560a7eb005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eaf9b40 0x560a7eb00000 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 --2- 172.21.15.16:0/3635743845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb02d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 >> 172.21.15.16:0/3635743845 conn(0x560a7eac1310 msgr2=0x560a7eafa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f0319ffb640 0 client.16929 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:54:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 shutdown_connections 2024-09-12T02:54:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 1 -- 172.21.15.16:0/3635743845 wait complete. 2024-09-12T02:54:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.676+0000 7f033a997740 20 client.16929 trim_cache size 0 max 16384 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 Processor -- start 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 -- start start 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eb0f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03301ad4d0 con 0x560a7eb03370 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03301af010 con 0x560a7eaf9b40 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033a997740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f03301adba0 con 0x560a7eaf85d0 2024-09-12T02:54:17.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0336b5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39448/0 (socket says 172.21.15.16:39448) 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0336b5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41540/0 (socket says 172.21.15.16:41540) 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 -- 172.21.15.16:0/184155557 learned_addr learned my addr 172.21.15.16:0/184155557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f033635a640 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eb0f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 -- 172.21.15.16:0/184155557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 msgr2=0x560a7eb0f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eb0f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 -- 172.21.15.16:0/184155557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 msgr2=0x560a7eafeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0335b59640 1 -- 172.21.15.16:0/184155557 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a7eaf5960 con 0x560a7eaf9b40 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.691+0000 7f0336b5b640 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f0335b59640 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03200c0280 tx=0x7f03200ed030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cda6c165c72e3b46 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f031bfff640 1 -- 172.21.15.16:0/184155557 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320049070 con 0x560a7eaf9b40 2024-09-12T02:54:17.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f031bfff640 1 -- 172.21.15.16:0/184155557 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03200ba0c0 con 0x560a7eaf9b40 2024-09-12T02:54:17.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f031bfff640 1 -- 172.21.15.16:0/184155557 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032003be20 con 0x560a7eaf9b40 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 -- 172.21.15.16:0/184155557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 msgr2=0x560a7eaf4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03200c0280 tx=0x7f03200ed030 comp rx=0 tx=0).stop 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 -- 172.21.15.16:0/184155557 shutdown_connections 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eafeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf9b40 0x560a7eaf4d70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 --2- 172.21.15.16:0/184155557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf85d0 0x560a7eb0f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.692+0000 7f033a997740 1 -- 172.21.15.16:0/184155557 >> 172.21.15.16:0/184155557 conn(0x560a7eac1310 msgr2=0x560a7eafe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- 172.21.15.16:0/184155557 shutdown_connections 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- 172.21.15.16:0/184155557 wait complete. 2024-09-12T02:54:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- start start 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 Processor -- start 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf9b40 0x560a7eb0ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eb06910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03301ae940 con 0x560a7eb03370 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03301ae270 con 0x560a7eaf85d0 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f03301ab4b0 con 0x560a7eaf9b40 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033a997740 10 client.0 _ll_register_callbacks cb 0x7f0328009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033635a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033635a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39462/0 (socket says 172.21.15.16:39462) 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f033635a640 1 -- 172.21.15.16:0/3681162445 learned_addr learned my addr 172.21.15.16:0/3681162445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f0335b59640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf9b40 0x560a7eb0ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.693+0000 7f0336b5b640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eb06910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 -- 172.21.15.16:0/3681162445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf9b40 msgr2=0x560a7eb0ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf9b40 0x560a7eb0ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 -- 172.21.15.16:0/3681162445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 msgr2=0x560a7eb06910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eb06910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 -- 172.21.15.16:0/3681162445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a7eaf44f0 con 0x560a7eaf85d0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033635a640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f032c012220 tx=0x7f032c012ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de7deb49bfd0561a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f0319ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c005020 con 0x560a7eaf85d0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f032c0051c0 con 0x560a7eaf85d0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03200ba0c0 con 0x560a7eaf85d0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.694+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f032c02f050 con 0x560a7eaf85d0 2024-09-12T02:54:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 10 client.26465 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a7eb81500 con 0x560a7eaf85d0 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032c01fb10 con 0x560a7eaf85d0 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f032c004aa0 con 0x560a7eaf85d0 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 10 client.26465 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 10 client.26465 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 20 client.26465 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb81c80 0x560a7eb840a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 4 client.26465 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f03200ba0c0 con 0x560a7eb81c80 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f0335b59640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb81c80 0x560a7eb840a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f0335b59640 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb81c80 0x560a7eb840a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=868249d18d71e093 server_cookie=439f2b087f0915eb in_seq=0 out_seq=0 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.695+0000 7f0319ffb640 10 client.26465 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:54:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f0319ffb640 1 -- 172.21.15.16:0/3681162445 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f03200ba0c0 con 0x560a7eb81c80 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f0319ffb640 10 client.26465 handle_command_reply: tid=0 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 client.26465 shutdown 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 msgr2=0x560a7eb0b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f032c012220 tx=0x7f032c012ac0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 shutdown_connections 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560a7eb81c80 0x560a7eb840a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560a7eb03370 0x560a7eb06910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f0319ffb640 0 client.26465 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a7eaf9b40 0x560a7eb0ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 --2- 172.21.15.16:0/3681162445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560a7eaf85d0 0x560a7eb0b530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 >> 172.21.15.16:0/3681162445 conn(0x560a7eac1310 msgr2=0x560a7eb01860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 shutdown_connections 2024-09-12T02:54:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 1 -- 172.21.15.16:0/3681162445 wait complete. 2024-09-12T02:54:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.696+0000 7f033a997740 20 client.26465 trim_cache size 0 max 16384 2024-09-12T02:54:17.703 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:54:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 msgr2=0x7f031c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 0x7f031c0a5040 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0328009fd0 tx=0x7f0328004ab0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 msgr2=0x7f0330198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0320009230 tx=0x7f0320004270 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 shutdown_connections 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0330106a80 0x7f033019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f031c0a2b80 0x7f031c0a5040 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0330101420 0x7f0330198930 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 --2- 172.21.15.16:0/1909805716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0330100a80 0x7f03301983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 >> 172.21.15.16:0/1909805716 conn(0x7f03300fc7a0 msgr2=0x7f033006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 shutdown_connections 2024-09-12T02:54:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:17.697+0000 7f03385be640 1 -- 172.21.15.16:0/1909805716 wait complete. 2024-09-12T02:54:17.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 606212 10.49 MB/sec execute 199 sec latency 16.597 ms 2024-09-12T02:54:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:18 smithi184 ceph-mon[35321]: pgmap v528: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-09-12T02:54:18.298 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:18 smithi033 ceph-mon[35092]: pgmap v528: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-09-12T02:54:18.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:18 smithi016 ceph-mon[30925]: pgmap v528: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-09-12T02:54:18.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:54:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:18.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 608055 10.50 MB/sec execute 200 sec latency 203.760 ms 2024-09-12T02:54:19.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 -- 172.21.15.16:0/1731490958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c101de0 msgr2=0x7fbc6c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 --2- 172.21.15.16:0/1731490958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1021c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009900 tx=0x7fbc5c02a410 comp rx=0 tx=0).stop 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 -- 172.21.15.16:0/1731490958 shutdown_connections 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 --2- 172.21.15.16:0/1731490958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c106f10 0x7fbc6c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 --2- 172.21.15.16:0/1731490958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c102790 0x7fbc6c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 --2- 172.21.15.16:0/1731490958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1021c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 -- 172.21.15.16:0/1731490958 >> 172.21.15.16:0/1731490958 conn(0x7fbc6c0fd8c0 msgr2=0x7fbc6c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.145+0000 7fbc72a17640 1 -- 172.21.15.16:0/1731490958 shutdown_connections 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.146+0000 7fbc72a17640 1 -- 172.21.15.16:0/1731490958 wait complete. 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.146+0000 7fbc72a17640 1 Processor -- start 2024-09-12T02:54:19.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.146+0000 7fbc72a17640 1 -- start start 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c102790 0x7fbc6c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 0x7fbc6c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c111f70 con 0x7fbc6c106f10 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c111df0 con 0x7fbc6c102790 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc72a17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1120f0 con 0x7fbc6c101de0 2024-09-12T02:54:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 0x7fbc6c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35396/0 (socket says 172.21.15.16:35396) 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc6bfff640 1 -- 172.21.15.16:0/3621014136 learned_addr learned my addr 172.21.15.16:0/3621014136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c102790 0x7fbc6c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 -- 172.21.15.16:0/3621014136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 msgr2=0x7fbc6c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 -- 172.21.15.16:0/3621014136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c102790 msgr2=0x7fbc6c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c102790 0x7fbc6c198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 -- 172.21.15.16:0/3621014136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c19d480 con 0x7fbc6c106f10 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.147+0000 7fbc70fb4640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 0x7fbc6c19cda0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009fd0 tx=0x7fbc6400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.148+0000 7fbc697fa640 1 -- 172.21.15.16:0/3621014136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6400ce20 con 0x7fbc6c106f10 2024-09-12T02:54:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.148+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6c19d770 con 0x7fbc6c106f10 2024-09-12T02:54:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.148+0000 7fbc697fa640 1 -- 172.21.15.16:0/3621014136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc64005020 con 0x7fbc6c106f10 2024-09-12T02:54:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.148+0000 7fbc697fa640 1 -- 172.21.15.16:0/3621014136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64010720 con 0x7fbc6c106f10 2024-09-12T02:54:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.148+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6c074420 con 0x7fbc6c106f10 2024-09-12T02:54:19.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.150+0000 7fbc697fa640 1 -- 172.21.15.16:0/3621014136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbc64005540 con 0x7fbc6c106f10 2024-09-12T02:54:19.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.150+0000 7fbc74df0740 1 Processor -- start 2024-09-12T02:54:19.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 -- start start 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed7118d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed7119a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ed710f32f0 con 0x55ed71198370 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ed7109acf0 con 0x55ed7118d5d0 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc74df0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed710ef2b0 con 0x55ed7118eb40 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc697fa640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 0x7fbc540a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39488/0 (socket says 172.21.15.16:39488) 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed7118d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 -- 172.21.15.16:0/1211571290 learned_addr learned my addr 172.21.15.16:0/1211571290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed7119a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 -- 172.21.15.16:0/1211571290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 msgr2=0x55ed7118d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed7118d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc70fb4640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 0x7fbc540a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 -- 172.21.15.16:0/1211571290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 msgr2=0x55ed7119a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed7119a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6bfff640 1 -- 172.21.15.16:0/1211571290 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed7107f3a0 con 0x55ed7118d5d0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.151+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed7118d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.152+0000 7fbc697fa640 1 -- 172.21.15.16:0/3621014136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbc640cf7c0 con 0x7fbc6c106f10 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.152+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a20 tx=0x7fbc5c005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42c62c08b5d25f0c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.152+0000 7fbc527fc640 1 -- 172.21.15.16:0/1211571290 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c040070 con 0x55ed7118d5d0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.152+0000 7fbc527fc640 1 -- 172.21.15.16:0/1211571290 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc5c004030 con 0x55ed7118d5d0 2024-09-12T02:54:19.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.152+0000 7fbc527fc640 1 -- 172.21.15.16:0/1211571290 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c035040 con 0x55ed7118d5d0 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- 172.21.15.16:0/1211571290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 msgr2=0x55ed7118da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc70fb4640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 0x7fbc540a9450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbc640989a0 tx=0x7fbc640bc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a20 tx=0x7fbc5c005dd0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- 172.21.15.16:0/1211571290 shutdown_connections 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed7119a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed7118da10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 --2- 172.21.15.16:0/1211571290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed7118d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- 172.21.15.16:0/1211571290 >> 172.21.15.16:0/1211571290 conn(0x55ed71156310 msgr2=0x55ed71189c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- 172.21.15.16:0/1211571290 shutdown_connections 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- 172.21.15.16:0/1211571290 wait complete. 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 -- start start 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.153+0000 7fbc74df0740 1 Processor -- start 2024-09-12T02:54:19.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed71195000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711955d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ed7119b010 con 0x55ed7118eb40 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ed710ae060 con 0x55ed71198370 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed7119b190 con 0x55ed7118d5d0 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc74df0740 10 client.0 _ll_register_callbacks cb 0x55ed70e018c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed71195000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35404/0 (socket says 172.21.15.16:35404) 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6b7fe640 1 -- 172.21.15.16:0/3598937942 learned_addr learned my addr 172.21.15.16:0/3598937942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 -- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 msgr2=0x55ed71197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 -- 172.21.15.16:0/3598937942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 msgr2=0x55ed711955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711955d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6bfff640 1 -- 172.21.15.16:0/3598937942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed70e643c0 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.154+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed71195000 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c005ab0 tx=0x7fbc5c005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e931463b1f323f63 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c040070 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5c004030 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbc5c0043e0 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc5c035040 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc74df0740 10 client.16947 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ed711a6a90 con 0x55ed7118eb40 2024-09-12T02:54:19.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc5c004030 con 0x55ed7118eb40 2024-09-12T02:54:19.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.155+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbc5c004df0 con 0x55ed7118eb40 2024-09-12T02:54:19.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 10 client.16947 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:19.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 10 client.16947 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:54:19.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 20 client.16947 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed711a7de0 0x55ed711aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 4 client.16947 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed71100860 con 0x55ed711a7de0 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed711a7de0 0x55ed711aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed711a7de0 0x55ed711aa1d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b48ae706c4622368 server_cookie=ead25fed87ea0628 in_seq=0 out_seq=0 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.156+0000 7fbc4b7fe640 10 client.16947 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3598937942 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ed71100860 con 0x55ed711a7de0 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc4b7fe640 10 client.16947 handle_command_reply: tid=0 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 client.16947 shutdown 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 msgr2=0x55ed71195000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed71195000 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c005ab0 tx=0x7fbc5c005370 comp rx=0 tx=0).stop 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 shutdown_connections 2024-09-12T02:54:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711955d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed71195000 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed711a7de0 0x55ed711aa1d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 --2- 172.21.15.16:0/3598937942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc4b7fe640 0 client.16947 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 >> 172.21.15.16:0/3598937942 conn(0x55ed71156310 msgr2=0x55ed7118f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 shutdown_connections 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 1 -- 172.21.15.16:0/3598937942 wait complete. 2024-09-12T02:54:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.158+0000 7fbc74df0740 20 client.16947 trim_cache size 0 max 16384 2024-09-12T02:54:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.173+0000 7fbc74df0740 1 Processor -- start 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.173+0000 7fbc74df0740 1 -- start start 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71194020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1ab260 con 0x55ed7118eb40 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1acda0 con 0x55ed71198370 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc74df0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1ab930 con 0x55ed7118d5d0 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41632/0 (socket says 172.21.15.16:41632) 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 -- 172.21.15.16:0/386541117 learned_addr learned my addr 172.21.15.16:0/386541117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71194020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 -- 172.21.15.16:0/386541117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 msgr2=0x55ed71194020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71194020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 -- 172.21.15.16:0/386541117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 msgr2=0x55ed711ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 -- 172.21.15.16:0/386541117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed7118a730 con 0x55ed7118eb40 2024-09-12T02:54:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a20 tx=0x7fbc5c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5cd80dab665e3d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.174+0000 7fbc51ffb640 1 -- 172.21.15.16:0/386541117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c040070 con 0x55ed7118eb40 2024-09-12T02:54:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc51ffb640 1 -- 172.21.15.16:0/386541117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc5c054030 con 0x55ed7118eb40 2024-09-12T02:54:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc51ffb640 1 -- 172.21.15.16:0/386541117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004ba0 con 0x55ed7118eb40 2024-09-12T02:54:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 -- 172.21.15.16:0/386541117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 msgr2=0x55ed711ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a20 tx=0x7fbc5c0039c0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 -- 172.21.15.16:0/386541117 shutdown_connections 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed71198370 0x55ed711ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed7118eb40 0x55ed711ad1f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 --2- 172.21.15.16:0/386541117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118d5d0 0x55ed71194020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 -- 172.21.15.16:0/386541117 >> 172.21.15.16:0/386541117 conn(0x55ed71156310 msgr2=0x55ed71193910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 -- 172.21.15.16:0/386541117 shutdown_connections 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.175+0000 7fbc74df0740 1 -- 172.21.15.16:0/386541117 wait complete. 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 -- start start 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 Processor -- start 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed711ada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed711aad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1ac6d0 con 0x55ed71198370 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1ac000 con 0x55ed7118d5d0 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c1a9240 con 0x55ed7118eb40 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.176+0000 7fbc74df0740 10 client.0 _ll_register_callbacks cb 0x7fbc60009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed711aad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41634/0 (socket says 172.21.15.16:41634) 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 -- 172.21.15.16:0/4243470016 learned_addr learned my addr 172.21.15.16:0/4243470016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed711ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 -- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 msgr2=0x55ed711aad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed711aad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 -- 172.21.15.16:0/4243470016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 msgr2=0x55ed711ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed711ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 -- 172.21.15.16:0/4243470016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed711abb30 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed711aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x55ed7109ae00 tx=0x7fbc60005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80ffead43f5e52e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6001b040 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5c054030 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.177+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbc60004040 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbc60004240 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 10 client.16953 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ed712128a0 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc5c054030 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbc60021040 con 0x55ed71198370 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 10 client.16953 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 10 client.16953 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:54:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 20 client.16953 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed71212f80 0x55ed712153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 4 client.16953 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed710fad70 con 0x55ed71212f80 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.178+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed71212f80 0x55ed712153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc4b7fe640 10 client.16953 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc6bfff640 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed71212f80 0x55ed712153a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e16e7fc23554ed73 server_cookie=af84cdc7b1b35365 in_seq=0 out_seq=0 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/4243470016 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed710fad70 con 0x55ed71212f80 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc4b7fe640 10 client.16953 handle_command_reply: tid=0 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 client.16953 shutdown 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 msgr2=0x55ed711ab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x55ed7109ae00 tx=0x7fbc60005f90 comp rx=0 tx=0).stop 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 shutdown_connections 2024-09-12T02:54:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed71198370 0x55ed711ab300 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55ed71212f80 0x55ed712153a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed7118eb40 0x55ed711aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc4b7fe640 0 client.16953 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 --2- 172.21.15.16:0/4243470016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ed7118d5d0 0x55ed711ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.179+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 >> 172.21.15.16:0/4243470016 conn(0x55ed71156310 msgr2=0x55ed71194130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 shutdown_connections 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc74df0740 1 -- 172.21.15.16:0/4243470016 wait complete. 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc74df0740 20 client.16953 trim_cache size 0 max 16384 2024-09-12T02:54:19.189 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:54:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 msgr2=0x7fbc540a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 0x7fbc540a9450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbc640989a0 tx=0x7fbc640bc040 comp rx=0 tx=0).stop 2024-09-12T02:54:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 msgr2=0x7fbc6c19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 0x7fbc6c19cda0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009fd0 tx=0x7fbc6400ef10 comp rx=0 tx=0).stop 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 shutdown_connections 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc540a6f90 0x7fbc540a9450 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c106f10 0x7fbc6c19cda0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc6c102790 0x7fbc6c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 --2- 172.21.15.16:0/3621014136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc6c101de0 0x7fbc6c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.180+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 >> 172.21.15.16:0/3621014136 conn(0x7fbc6c0fd8c0 msgr2=0x7fbc6c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.181+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 shutdown_connections 2024-09-12T02:54:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:19.181+0000 7fbc72a17640 1 -- 172.21.15.16:0/3621014136 wait complete. 2024-09-12T02:54:19.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 611108 10.54 MB/sec execute 201 sec latency 40.420 ms 2024-09-12T02:54:19.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:54:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:20 smithi016 ceph-mon[30925]: pgmap v529: 97 pgs: 97 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 257 op/s 2024-09-12T02:54:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:20 smithi184 ceph-mon[35321]: pgmap v529: 97 pgs: 97 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 257 op/s 2024-09-12T02:54:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:20 smithi033 ceph-mon[35092]: pgmap v529: 97 pgs: 97 active+clean; 223 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 257 op/s 2024-09-12T02:54:20.722 INFO:tasks.workunit.client.0.smithi016.stdout: 1 613659 10.55 MB/sec execute 202 sec latency 40.679 ms 2024-09-12T02:54:20.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 -- 172.21.15.16:0/140415118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c101df0 msgr2=0x7f619c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 --2- 172.21.15.16:0/140415118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c101df0 0x7f619c1021d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6194009ca0 tx=0x7f619402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 -- 172.21.15.16:0/140415118 shutdown_connections 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 --2- 172.21.15.16:0/140415118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c106f20 0x7f619c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 --2- 172.21.15.16:0/140415118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c1027a0 0x7f619c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 --2- 172.21.15.16:0/140415118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c101df0 0x7f619c1021d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 -- 172.21.15.16:0/140415118 >> 172.21.15.16:0/140415118 conn(0x7f619c0fd8f0 msgr2=0x7f619c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 -- 172.21.15.16:0/140415118 shutdown_connections 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 -- 172.21.15.16:0/140415118 wait complete. 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.826+0000 7f61a16f3640 1 Processor -- start 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 -- start start 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c101df0 0x7f619c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 0x7f619c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f619c111f80 con 0x7f619c1027a0 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f619c111e00 con 0x7f619c101df0 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f61a16f3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f619c112100 con 0x7f619c106f20 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f619affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c101df0 0x7f619c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41652/0 (socket says 172.21.15.16:41652) 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 -- 172.21.15.16:0/3649470218 learned_addr learned my addr 172.21.15.16:0/3649470218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f619b7fe640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 0x7f619c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 -- 172.21.15.16:0/3649470218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 msgr2=0x7f619c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 0x7f619c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 -- 172.21.15.16:0/3649470218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c101df0 msgr2=0x7f619c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c101df0 0x7f619c19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f618a7fc640 1 -- 172.21.15.16:0/3649470218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c19f670 con 0x7f619c1027a0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.827+0000 7f619b7fe640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 0x7f619c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f618a7fc640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f618400e9f0 tx=0x7f618400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f6198ff9640 1 -- 172.21.15.16:0/3649470218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618400cd80 con 0x7f619c1027a0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c19f960 con 0x7f619c1027a0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f6198ff9640 1 -- 172.21.15.16:0/3649470218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6184005500 con 0x7f619c1027a0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f6198ff9640 1 -- 172.21.15.16:0/3649470218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184010690 con 0x7f619c1027a0 2024-09-12T02:54:20.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.828+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c1a7240 con 0x7f619c1027a0 2024-09-12T02:54:20.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 Processor -- start 2024-09-12T02:54:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 -- start start 2024-09-12T02:54:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f6198ff9640 1 -- 172.21.15.16:0/3649470218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f61840046c0 con 0x7f619c1027a0 2024-09-12T02:54:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 0x561555e3fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 0x561555e4c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561555da52f0 con 0x561555e40b40 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561555d4ccf0 con 0x561555e4a370 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.831+0000 7f61a3acc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561555da12b0 con 0x561555e3f5d0 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 0x561555e3fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 0x561555e3fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35432/0 (socket says 172.21.15.16:35432) 2024-09-12T02:54:20.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619affd640 1 -- 172.21.15.16:0/261536142 learned_addr learned my addr 172.21.15.16:0/261536142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:20.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f618a7fc640 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 0x561555e4c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 -- 172.21.15.16:0/261536142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 msgr2=0x561555e3fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 0x561555e3fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 -- 172.21.15.16:0/261536142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 msgr2=0x561555e4c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 0x561555e4c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 -- 172.21.15.16:0/261536142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561555d313a0 con 0x561555e40b40 2024-09-12T02:54:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f618a7fc640 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 0x561555e4c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f6198ff9640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 0x7f617c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619b7fe640 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e3f090 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f618c009f30 tx=0x7f618c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e77f823f2dbf1e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f619affd640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 0x7f617c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f6198ff9640 1 -- 172.21.15.16:0/3649470218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f61840cf410 con 0x7f619c1027a0 2024-09-12T02:54:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.832+0000 7f6191ffb640 1 -- 172.21.15.16:0/261536142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c004310 con 0x561555e40b40 2024-09-12T02:54:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f6191ffb640 1 -- 172.21.15.16:0/261536142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f618c0044b0 con 0x561555e40b40 2024-09-12T02:54:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f6191ffb640 1 -- 172.21.15.16:0/261536142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c0035f0 con 0x561555e40b40 2024-09-12T02:54:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 -- 172.21.15.16:0/261536142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 msgr2=0x561555e3f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f619affd640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 0x7f617c0a5040 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x561555e3cac0 tx=0x7f619403d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e3f090 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f618c009f30 tx=0x7f618c0118d0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 -- 172.21.15.16:0/261536142 shutdown_connections 2024-09-12T02:54:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e4a370 0x561555e4c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e3f5d0 0x561555e3fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 --2- 172.21.15.16:0/261536142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e3f090 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.833+0000 7f61a3acc740 1 -- 172.21.15.16:0/261536142 >> 172.21.15.16:0/261536142 conn(0x561555e08310 msgr2=0x561555e3bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.834+0000 7f61a3acc740 1 -- 172.21.15.16:0/261536142 shutdown_connections 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.834+0000 7f61a3acc740 1 -- 172.21.15.16:0/261536142 wait complete. 2024-09-12T02:54:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.834+0000 7f61a3acc740 1 -- start start 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.834+0000 7f61a3acc740 1 Processor -- start 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e49d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e4a370 0x561555e47650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561555e4d070 con 0x561555e4a370 2024-09-12T02:54:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561555d60060 con 0x561555e3f5d0 2024-09-12T02:54:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561555e4d1f0 con 0x561555e40b40 2024-09-12T02:54:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f61a3acc740 10 client.0 _ll_register_callbacks cb 0x561555ab38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f619affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e49d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35444/0 (socket says 172.21.15.16:35444) 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f619affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e49d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39552/0 (socket says 172.21.15.16:39552) 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 -- 172.21.15.16:0/2283911891 learned_addr learned my addr 172.21.15.16:0/2283911891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f619b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e4a370 0x561555e47650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 -- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 msgr2=0x561555e49d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e49d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 -- 172.21.15.16:0/2283911891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e4a370 msgr2=0x561555e47650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e4a370 0x561555e47650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 -- 172.21.15.16:0/2283911891 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561555b163c0 con 0x561555e40b40 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618a7fc640 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f618400c6a0 tx=0x7f61840056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bae8cc46a7d62e70 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618409b7c0 con 0x561555e40b40 2024-09-12T02:54:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.835+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f61840a4660 con 0x561555e40b40 2024-09-12T02:54:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f618c0044b0 con 0x561555e40b40 2024-09-12T02:54:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f61840b9050 con 0x561555e40b40 2024-09-12T02:54:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 10 client.26398 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561555e58bf0 con 0x561555e40b40 2024-09-12T02:54:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f618c0044b0 con 0x561555e40b40 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f61840a4bd0 con 0x561555e40b40 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 10 client.26398 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 10 client.26398 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 20 client.26398 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555e5a0b0 0x561555e5c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 4 client.26398 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561555db2860 con 0x561555e5a0b0 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.836+0000 7f619affd640 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555e5a0b0 0x561555e5c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.837+0000 7f619affd640 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555e5a0b0 0x561555e5c4a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=976d821bcf3e56f7 server_cookie=5c2ed1569aad047 in_seq=0 out_seq=0 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.837+0000 7f618affd640 10 client.26398 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.837+0000 7f618affd640 1 -- 172.21.15.16:0/2283911891 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561555db2860 con 0x561555e5a0b0 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.837+0000 7f618affd640 10 client.26398 handle_command_reply: tid=0 2024-09-12T02:54:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.837+0000 7f61a3acc740 1 client.26398 shutdown 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 msgr2=0x561555e47040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f618400c6a0 tx=0x7f61840056b0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 shutdown_connections 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555e5a0b0 0x561555e5c4a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e4a370 0x561555e47650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f618affd640 0 client.26398 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e40b40 0x561555e47040 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 --2- 172.21.15.16:0/2283911891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e49d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 >> 172.21.15.16:0/2283911891 conn(0x561555e08310 msgr2=0x561555e41960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 shutdown_connections 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 1 -- 172.21.15.16:0/2283911891 wait complete. 2024-09-12T02:54:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.838+0000 7f61a3acc740 20 client.26398 trim_cache size 0 max 16384 2024-09-12T02:54:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.851+0000 7f61a3acc740 1 Processor -- start 2024-09-12T02:54:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 -- start start 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e40b40 0x561555e42660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e61cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1af5f0 con 0x561555e3f5d0 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1b1130 con 0x561555e40b40 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f61a3acc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1afcc0 con 0x561555e4a370 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41692/0 (socket says 172.21.15.16:41692) 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e61cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 -- 172.21.15.16:0/1616866141 learned_addr learned my addr 172.21.15.16:0/1616866141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e40b40 0x561555e42660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 -- 172.21.15.16:0/1616866141 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 msgr2=0x561555e61cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e61cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 -- 172.21.15.16:0/1616866141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e40b40 msgr2=0x561555e42660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e40b40 0x561555e42660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 -- 172.21.15.16:0/1616866141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561555e45590 con 0x561555e3f5d0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.852+0000 7f619affd640 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6194009bf0 tx=0x7f61940095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e418c06e99ac1c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61917fa640 1 -- 172.21.15.16:0/1616866141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619402fce0 con 0x561555e3f5d0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61917fa640 1 -- 172.21.15.16:0/1616866141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6194036040 con 0x561555e3f5d0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61917fa640 1 -- 172.21.15.16:0/1616866141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619405c690 con 0x561555e3f5d0 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 -- 172.21.15.16:0/1616866141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 msgr2=0x561555e42120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6194009bf0 tx=0x7f61940095c0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 -- 172.21.15.16:0/1616866141 shutdown_connections 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e61cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e40b40 0x561555e42660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1616866141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e3f5d0 0x561555e42120 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 -- 172.21.15.16:0/1616866141 >> 172.21.15.16:0/1616866141 conn(0x561555e08310 msgr2=0x561555e490c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 -- 172.21.15.16:0/1616866141 shutdown_connections 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.853+0000 7f61a3acc740 1 -- 172.21.15.16:0/1616866141 wait complete. 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 -- start start 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 Processor -- start 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e52ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1b0a60 con 0x561555e40b40 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1b0390 con 0x561555e3f5d0 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1ad5d0 con 0x561555e4a370 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f61a3acc740 10 client.0 _ll_register_callbacks cb 0x7f61840d2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e52ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f619b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f619affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f619b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35466/0 (socket says 172.21.15.16:35466) 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.854+0000 7f619b7fe640 1 -- 172.21.15.16:0/1847892078 learned_addr learned my addr 172.21.15.16:0/1847892078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 -- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 msgr2=0x561555e52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 -- 172.21.15.16:0/1847892078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 msgr2=0x561555e52ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e52ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 -- 172.21.15.16:0/1847892078 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561555e4fb10 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619affd640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f619b7fe640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f618c009ed0 tx=0x7f618c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b990b48ab03aa01 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f618affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c030060 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6194036040 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f618c0037f0 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f618c0039f0 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f61a3acc740 10 client.26404 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.855+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561555ec5650 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6194036040 con 0x561555e4a370 2024-09-12T02:54:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f618c00fbc0 con 0x561555e4a370 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 10 client.26404 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 10 client.26404 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 20 client.26404 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555ec5b80 0x561555ec7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 4 client.26404 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561555dacd70 con 0x561555ec5b80 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f619affd640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555ec5b80 0x561555ec7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f619affd640 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555ec5b80 0x561555ec7f70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=458406e427b1f7cf server_cookie=d89a35e6862bf6cc in_seq=0 out_seq=0 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.856+0000 7f618affd640 10 client.26404 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f618affd640 1 -- 172.21.15.16:0/1847892078 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561555dacd70 con 0x561555ec5b80 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f618affd640 10 client.26404 handle_command_reply: tid=0 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 client.26404 shutdown 2024-09-12T02:54:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 msgr2=0x561555e4f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f618c009ed0 tx=0x7f618c009310 comp rx=0 tx=0).stop 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 shutdown_connections 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x561555ec5b80 0x561555ec7f70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561555e4a370 0x561555e4f450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561555e40b40 0x561555e52ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f618affd640 0 client.26404 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 --2- 172.21.15.16:0/1847892078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561555e3f5d0 0x561555e52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 >> 172.21.15.16:0/1847892078 conn(0x561555e08310 msgr2=0x561555e3a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 shutdown_connections 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 1 -- 172.21.15.16:0/1847892078 wait complete. 2024-09-12T02:54:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a3acc740 20 client.26404 trim_cache size 0 max 16384 2024-09-12T02:54:20.870 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:54:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 msgr2=0x7f617c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.857+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 0x7f617c0a5040 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x561555e3cac0 tx=0x7f619403d040 comp rx=0 tx=0).stop 2024-09-12T02:54:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 msgr2=0x7f619c19ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f618400e9f0 tx=0x7f618400edf0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 shutdown_connections 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f619c106f20 0x7f619c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f617c0a2b80 0x7f617c0a5040 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619c1027a0 0x7f619c19ac00 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 --2- 172.21.15.16:0/3649470218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f619c101df0 0x7f619c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 >> 172.21.15.16:0/3649470218 conn(0x7f619c0fd8f0 msgr2=0x7f619c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 shutdown_connections 2024-09-12T02:54:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:20.858+0000 7f61a16f3640 1 -- 172.21.15.16:0/3649470218 wait complete. 2024-09-12T02:54:21.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:54:21.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 615621 10.55 MB/sec execute 203 sec latency 15.397 ms 2024-09-12T02:54:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:22.202 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:22 smithi016 ceph-mon[30925]: pgmap v530: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 228 op/s 2024-09-12T02:54:22.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 -- 172.21.15.16:0/1554700729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c103740 msgr2=0x7f141c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 --2- 172.21.15.16:0/1554700729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c103740 0x7f141c107b80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1410009930 tx=0x7f141002a420 comp rx=0 tx=0).stop 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 -- 172.21.15.16:0/1554700729 shutdown_connections 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 --2- 172.21.15.16:0/1554700729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c108290 0x7f141c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 --2- 172.21.15.16:0/1554700729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c103740 0x7f141c107b80 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 --2- 172.21.15.16:0/1554700729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 -- 172.21.15.16:0/1554700729 >> 172.21.15.16:0/1554700729 conn(0x7f141c0fc7d0 msgr2=0x7f141c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 -- 172.21.15.16:0/1554700729 shutdown_connections 2024-09-12T02:54:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.290+0000 7f14223f6640 1 -- 172.21.15.16:0/1554700729 wait complete. 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 Processor -- start 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 -- start start 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 0x7f141c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c108290 0x7f141c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f141c112050 con 0x7f141c108290 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f141c111ed0 con 0x7f141c102d90 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f14223f6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1121d0 con 0x7f141c103740 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f141bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f141bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c1a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39590/0 (socket says 172.21.15.16:39590) 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 0x7f141c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f141bfff640 1 -- 172.21.15.16:0/3167262283 learned_addr learned my addr 172.21.15.16:0/3167262283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.291+0000 7f1420993640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c108290 0x7f141c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 -- 172.21.15.16:0/3167262283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 msgr2=0x7f141c1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 -- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c108290 msgr2=0x7f141c19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c108290 0x7f141c19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 -- 172.21.15.16:0/3167262283 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c19ad10 con 0x7f141c103740 2024-09-12T02:54:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 0x7f141c1a0960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1410009230 tx=0x7f14100040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f14197fa640 1 -- 172.21.15.16:0/3167262283 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141003f070 con 0x7f141c103740 2024-09-12T02:54:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f14197fa640 1 -- 172.21.15.16:0/3167262283 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14100044c0 con 0x7f141c103740 2024-09-12T02:54:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c19afa0 con 0x7f141c103740 2024-09-12T02:54:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f14197fa640 1 -- 172.21.15.16:0/3167262283 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410035070 con 0x7f141c103740 2024-09-12T02:54:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.292+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c1a72f0 con 0x7f141c103740 2024-09-12T02:54:22.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:22 smithi184 ceph-mon[35321]: pgmap v530: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 228 op/s 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14197fa640 1 -- 172.21.15.16:0/3167262283 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f141003b460 con 0x7f141c103740 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 Processor -- start 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 -- start start 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83d1370 0x55a3b83d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3b832c2f0 con 0x55a3b83d1370 2024-09-12T02:54:22.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a3b82d3cf0 con 0x55a3b83c7b40 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f14247cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a3b83282b0 con 0x55a3b83c65d0 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f1420993640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f1420993640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39594/0 (socket says 172.21.15.16:39594) 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f1420993640 1 -- 172.21.15.16:0/1418215597 learned_addr learned my addr 172.21.15.16:0/1418215597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.295+0000 7f141bfff640 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f141b7fe640 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83d1370 0x55a3b83d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 -- 172.21.15.16:0/1418215597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 msgr2=0x55a3b83c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 -- 172.21.15.16:0/1418215597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83d1370 msgr2=0x55a3b83d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83d1370 0x55a3b83d3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 -- 172.21.15.16:0/1418215597 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3b82b83a0 con 0x55a3b83c7b40 2024-09-12T02:54:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f14197fa640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 0x7f14040a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f141bfff640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 0x7f14040a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f14197fa640 1 -- 172.21.15.16:0/3167262283 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14100ef450 con 0x7f141c103740 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f1420993640 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b6b0 tx=0x7f140c004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adfd79608064370e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f14027fc640 1 -- 172.21.15.16:0/1418215597 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c003a70 con 0x55a3b83c7b40 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f14027fc640 1 -- 172.21.15.16:0/1418215597 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c015070 con 0x55a3b83c7b40 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.296+0000 7f14027fc640 1 -- 172.21.15.16:0/1418215597 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c0116a0 con 0x55a3b83c7b40 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 -- 172.21.15.16:0/1418215597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 msgr2=0x55a3b83c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f141bfff640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 0x7f14040a50c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b6b0 tx=0x7f140c004610 comp rx=0 tx=0).stop 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 -- 172.21.15.16:0/1418215597 shutdown_connections 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83d1370 0x55a3b83d3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 --2- 172.21.15.16:0/1418215597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c7b40 0x55a3b83c6090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 -- 172.21.15.16:0/1418215597 >> 172.21.15.16:0/1418215597 conn(0x55a3b838f310 msgr2=0x55a3b83c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 -- 172.21.15.16:0/1418215597 shutdown_connections 2024-09-12T02:54:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.297+0000 7f14247cf740 1 -- 172.21.15.16:0/1418215597 wait complete. 2024-09-12T02:54:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 -- start start 2024-09-12T02:54:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 Processor -- start 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3b83d4070 con 0x55a3b83c7b40 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a3b82e7060 con 0x55a3b83d1370 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a3b83d41f0 con 0x55a3b83c65d0 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f14247cf740 10 client.0 _ll_register_callbacks cb 0x55a3b803a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f141bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f1420993640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41748/0 (socket says 172.21.15.16:41748) 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f141bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35518/0 (socket says 172.21.15.16:35518) 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.298+0000 7f141b7fe640 1 -- 172.21.15.16:0/1148327991 learned_addr learned my addr 172.21.15.16:0/1148327991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 -- 172.21.15.16:0/1148327991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 msgr2=0x55a3b83d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 -- 172.21.15.16:0/1148327991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 msgr2=0x55a3b83ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 -- 172.21.15.16:0/1148327991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3b809d3c0 con 0x55a3b83d1370 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f1420993640 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83ce7f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b6b0 tx=0x7f140c01a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43da2f55434b79c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f13fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c028060 con 0x55a3b83d1370 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f140c005240 con 0x55a3b83d1370 2024-09-12T02:54:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f140c015070 con 0x55a3b83d1370 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f140c01a890 con 0x55a3b83d1370 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f14247cf740 10 client.26513 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.299+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3b83dfb60 con 0x55a3b83d1370 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c015070 con 0x55a3b83d1370 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f140c004920 con 0x55a3b83d1370 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 10 client.26513 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 20 client.26513 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b83e0f60 0x55a3b83e3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 4 client.26513 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3b8339860 con 0x55a3b83e0f60 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.300+0000 7f141bfff640 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b83e0f60 0x55a3b83e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f13fb7fe640 10 client.26513 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f141bfff640 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b83e0f60 0x55a3b83e3350 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6277cef3c58a1a22 server_cookie=24f5cf25f80efc32 in_seq=0 out_seq=0 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f13fb7fe640 1 -- 172.21.15.16:0/1148327991 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a3b8339860 con 0x55a3b83e0f60 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f13fb7fe640 10 client.26513 handle_command_reply: tid=0 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 client.26513 shutdown 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 msgr2=0x55a3b83ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83ce7f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b6b0 tx=0x7f140c01a6a0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 shutdown_connections 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b83e0f60 0x55a3b83e3350 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83ce7f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 --2- 172.21.15.16:0/1148327991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f13fb7fe640 0 client.26513 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.301+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 >> 172.21.15.16:0/1148327991 conn(0x55a3b838f310 msgr2=0x55a3b83c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.302+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 shutdown_connections 2024-09-12T02:54:22.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.302+0000 7f14247cf740 1 -- 172.21.15.16:0/1148327991 wait complete. 2024-09-12T02:54:22.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.302+0000 7f14247cf740 20 client.26513 trim_cache size 0 max 16384 2024-09-12T02:54:22.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 Processor -- start 2024-09-12T02:54:22.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 -- start start 2024-09-12T02:54:22.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83d1370 0x55a3b83e6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1af7d0 con 0x55a3b83c7b40 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1b1310 con 0x55a3b83c65d0 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.315+0000 7f14247cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1afea0 con 0x55a3b83d1370 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f1420993640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83d1370 0x55a3b83e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39616/0 (socket says 172.21.15.16:39616) 2024-09-12T02:54:22.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141bfff640 1 -- 172.21.15.16:0/454263440 learned_addr learned my addr 172.21.15.16:0/454263440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:22.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 -- 172.21.15.16:0/454263440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83d1370 msgr2=0x55a3b83e6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83d1370 0x55a3b83e6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 -- 172.21.15.16:0/454263440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 msgr2=0x55a3b83c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 -- 172.21.15.16:0/454263440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3b83c24f0 con 0x55a3b83c7b40 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141bfff640 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f141b7fe640 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83c9660 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1410006010 tx=0x7f141003b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f947e441d8af5f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f1401ffb640 1 -- 172.21.15.16:0/454263440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410048040 con 0x55a3b83c7b40 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f1401ffb640 1 -- 172.21.15.16:0/454263440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14100bbd70 con 0x55a3b83c7b40 2024-09-12T02:54:22.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.316+0000 7f1401ffb640 1 -- 172.21.15.16:0/454263440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100ba100 con 0x55a3b83c7b40 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- 172.21.15.16:0/454263440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 msgr2=0x55a3b83c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83c9660 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1410006010 tx=0x7f141003b800 comp rx=0 tx=0).stop 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- 172.21.15.16:0/454263440 shutdown_connections 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83d1370 0x55a3b83e6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83c9660 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 --2- 172.21.15.16:0/454263440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83c65d0 0x55a3b83c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- 172.21.15.16:0/454263440 >> 172.21.15.16:0/454263440 conn(0x55a3b838f310 msgr2=0x55a3b83cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- 172.21.15.16:0/454263440 shutdown_connections 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- 172.21.15.16:0/454263440 wait complete. 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.317+0000 7f14247cf740 1 -- start start 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 Processor -- start 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83e4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1b0c40 con 0x55a3b83c7b40 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1b0570 con 0x55a3b83d1370 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1ad7b0 con 0x55a3b83c65d0 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f14247cf740 10 client.0 _ll_register_callbacks cb 0x7f140c004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:22.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f141bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f1420993640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f141b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41776/0 (socket says 172.21.15.16:41776) 2024-09-12T02:54:22.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f141bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83e6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35538/0 (socket says 172.21.15.16:35538) 2024-09-12T02:54:22.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.318+0000 7f141b7fe640 1 -- 172.21.15.16:0/3025787743 learned_addr learned my addr 172.21.15.16:0/3025787743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 -- 172.21.15.16:0/3025787743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 msgr2=0x55a3b83e6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83e6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 -- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 msgr2=0x55a3b83e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83e4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 -- 172.21.15.16:0/3025787743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3b83e4c70 con 0x55a3b83c7b40 2024-09-12T02:54:22.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f1420993640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f141b7fe640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f14100bba70 tx=0x7f14100bb6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201f4a06cbed5da0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f13fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410048070 con 0x55a3b83c7b40 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14100ef0f0 con 0x55a3b83c7b40 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14100bbd70 con 0x55a3b83c7b40 2024-09-12T02:54:22.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f14100d9050 con 0x55a3b83c7b40 2024-09-12T02:54:22.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f14247cf740 10 client.16989 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:22.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3b844b940 con 0x55a3b83c7b40 2024-09-12T02:54:22.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.319+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14100bbd70 con 0x55a3b83c7b40 2024-09-12T02:54:22.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f141004b060 con 0x55a3b83c7b40 2024-09-12T02:54:22.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 10 client.16989 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 20 client.16989 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b844bfb0 0x55a3b844e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 4 client.16989 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3b8333d70 con 0x55a3b844bfb0 2024-09-12T02:54:22.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f141bfff640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b844bfb0 0x55a3b844e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f13fb7fe640 10 client.16989 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f141bfff640 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b844bfb0 0x55a3b844e3d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=debfe73716cdb185 server_cookie=8c5930bf35bc2fb7 in_seq=0 out_seq=0 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f13fb7fe640 1 -- 172.21.15.16:0/3025787743 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3b8333d70 con 0x55a3b844bfb0 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.320+0000 7f13fb7fe640 10 client.16989 handle_command_reply: tid=0 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 client.16989 shutdown 2024-09-12T02:54:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 msgr2=0x55a3b83e3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f14100bba70 tx=0x7f14100bb6d0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 shutdown_connections 2024-09-12T02:54:22.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a3b844bfb0 0x55a3b844e3d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3b83d1370 0x55a3b83e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3b83c7b40 0x55a3b83e3ec0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 --2- 172.21.15.16:0/3025787743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3b83c65d0 0x55a3b83e6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 >> 172.21.15.16:0/3025787743 conn(0x55a3b838f310 msgr2=0x55a3b83cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f13fb7fe640 0 client.16989 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 shutdown_connections 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 1 -- 172.21.15.16:0/3025787743 wait complete. 2024-09-12T02:54:22.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.321+0000 7f14247cf740 20 client.16989 trim_cache size 0 max 16384 2024-09-12T02:54:22.342 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:54:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 msgr2=0x7f14040a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 0x7f14040a50c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414009500 comp rx=0 tx=0).stop 2024-09-12T02:54:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 msgr2=0x7f141c1a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 0x7f141c1a0960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1410009230 tx=0x7f14100040c0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 shutdown_connections 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14040a2c00 0x7f14040a50c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f141c108290 0x7f141c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f141c103740 0x7f141c1a0960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 --2- 172.21.15.16:0/3167262283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f141c102d90 0x7f141c1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 >> 172.21.15.16:0/3167262283 conn(0x7f141c0fc7d0 msgr2=0x7f141c100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 shutdown_connections 2024-09-12T02:54:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:22.322+0000 7f14223f6640 1 -- 172.21.15.16:0/3167262283 wait complete. 2024-09-12T02:54:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:22 smithi033 ceph-mon[35092]: pgmap v530: 97 pgs: 97 active+clean; 225 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 228 op/s 2024-09-12T02:54:22.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 618519 10.58 MB/sec execute 204 sec latency 8.925 ms 2024-09-12T02:54:23.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 621687 10.61 MB/sec execute 205 sec latency 13.332 ms 2024-09-12T02:54:24.096 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:54:24.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:54:24.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:24 smithi184 ceph-mon[35321]: pgmap v531: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-09-12T02:54:24.378 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:24.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:24 smithi016 ceph-mon[30925]: pgmap v531: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-09-12T02:54:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:24 smithi033 ceph-mon[35092]: pgmap v531: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-09-12T02:54:24.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 624134 10.61 MB/sec execute 206 sec latency 42.931 ms 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.884+0000 7fac24689640 1 -- 172.21.15.16:0/3565854850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c103740 msgr2=0x7fac1c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.884+0000 7fac24689640 1 --2- 172.21.15.16:0/3565854850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c103740 0x7fac1c107b80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac10009860 tx=0x7fac1002a340 comp rx=0 tx=0).stop 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 -- 172.21.15.16:0/3565854850 shutdown_connections 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 --2- 172.21.15.16:0/3565854850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c108290 0x7fac1c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 --2- 172.21.15.16:0/3565854850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c103740 0x7fac1c107b80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 --2- 172.21.15.16:0/3565854850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 -- 172.21.15.16:0/3565854850 >> 172.21.15.16:0/3565854850 conn(0x7fac1c0fc7d0 msgr2=0x7fac1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 -- 172.21.15.16:0/3565854850 shutdown_connections 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 -- 172.21.15.16:0/3565854850 wait complete. 2024-09-12T02:54:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 Processor -- start 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.885+0000 7fac24689640 1 -- start start 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c103740 0x7fac1c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c108290 0x7fac1c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c112050 con 0x7fac1c102d90 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c111ed0 con 0x7fac1c103740 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac24689640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1121d0 con 0x7fac1c108290 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41792/0 (socket says 172.21.15.16:41792) 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 -- 172.21.15.16:0/1600491062 learned_addr learned my addr 172.21.15.16:0/1600491062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22c26640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c108290 0x7fac1c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac21c24640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c103740 0x7fac1c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 -- 172.21.15.16:0/1600491062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c108290 msgr2=0x7fac1c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c108290 0x7fac1c19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 -- 172.21.15.16:0/1600491062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c103740 msgr2=0x7fac1c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c103740 0x7fac1c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 -- 172.21.15.16:0/1600491062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c19d560 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.886+0000 7fac22425640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fac1800e9c0 tx=0x7fac1800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.887+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1600491062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1800ccc0 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.887+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1600491062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac18005500 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.887+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1c19d850 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.887+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1600491062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18010430 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.887+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1c1a5130 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.889+0000 7fac26a62740 1 Processor -- start 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.889+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1600491062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fac180046c0 con 0x7fac1c102d90 2024-09-12T02:54:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.889+0000 7fac26a62740 1 -- start start 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556f26c09280 con 0x556f26ca3b60 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f26bb0b10 con 0x556f26caed40 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac26a62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556f26c05240 con 0x556f26ca4480 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41794/0 (socket says 172.21.15.16:41794) 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 -- 172.21.15.16:0/870379869 learned_addr learned my addr 172.21.15.16:0/870379869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac21c24640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22425640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 -- 172.21.15.16:0/870379869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 msgr2=0x556f26cae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 -- 172.21.15.16:0/870379869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 msgr2=0x556f26cb1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac22c26640 1 -- 172.21.15.16:0/870379869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f26bbd1f0 con 0x556f26ca3b60 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac21c24640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.890+0000 7fac0f7fe640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 0x7fac080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac22c26640 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fac14009f60 tx=0x7fac14009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cc68ea44c9809 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac22425640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 0x7fac080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1600491062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac180cf3d0 con 0x7fac1c102d90 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac0cff9640 1 -- 172.21.15.16:0/870379869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1401e070 con 0x556f26ca3b60 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac0cff9640 1 -- 172.21.15.16:0/870379869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac14003c30 con 0x556f26ca3b60 2024-09-12T02:54:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.891+0000 7fac0cff9640 1 -- 172.21.15.16:0/870379869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1400f9e0 con 0x556f26ca3b60 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- 172.21.15.16:0/870379869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 msgr2=0x556f26ca3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac22425640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 0x7fac080a5040 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac180cd250 tx=0x7fac180d2eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fac14009f60 tx=0x7fac14009f90 comp rx=0 tx=0).stop 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- 172.21.15.16:0/870379869 shutdown_connections 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 --2- 172.21.15.16:0/870379869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26ca3f40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- 172.21.15.16:0/870379869 >> 172.21.15.16:0/870379869 conn(0x556f26ca00f0 msgr2=0x556f26ca04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- 172.21.15.16:0/870379869 shutdown_connections 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- 172.21.15.16:0/870379869 wait complete. 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 -- start start 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.892+0000 7fac26a62740 1 Processor -- start 2024-09-12T02:54:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cabc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556f26cb1910 con 0x556f26ca4480 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f26bc3f10 con 0x556f26caed40 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556f26cb1a90 con 0x556f26ca3b60 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac26a62740 10 client.0 _ll_register_callbacks cb 0x556f26b8ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac22c26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac22c26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39678/0 (socket says 172.21.15.16:39678) 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac22c26640 1 -- 172.21.15.16:0/1796595928 learned_addr learned my addr 172.21.15.16:0/1796595928 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac22c26640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cabc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 -- 172.21.15.16:0/1796595928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 msgr2=0x556f26cabc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cabc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 -- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 msgr2=0x556f26cac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 -- 172.21.15.16:0/1796595928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f2697a6a0 con 0x556f26ca4480 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac22c26640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac21c24640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cac150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fac10009780 tx=0x7fac100303d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f7300b7f0d35e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.893+0000 7fac05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10037040 con 0x556f26ca4480 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac10004050 con 0x556f26ca4480 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac14003c30 con 0x556f26ca4480 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fac10004270 con 0x556f26ca4480 2024-09-12T02:54:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 10 client.17001 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f26cbd510 con 0x556f26ca4480 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac14003c30 con 0x556f26ca4480 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fac10004820 con 0x556f26ca4480 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 10 client.17001 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 20 client.17001 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26cbea20 0x556f26cc0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 4 client.17001 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.894+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f26c167f0 con 0x556f26cbea20 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.895+0000 7fac22c26640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26cbea20 0x556f26cc0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.895+0000 7fac22c26640 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26cbea20 0x556f26cc0e10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8241b3fb4874ad server_cookie=5972cc7819d1e509 in_seq=0 out_seq=0 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.895+0000 7fac05ffb640 10 client.17001 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac05ffb640 1 -- 172.21.15.16:0/1796595928 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556f26c167f0 con 0x556f26cbea20 2024-09-12T02:54:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac05ffb640 10 client.17001 handle_command_reply: tid=0 2024-09-12T02:54:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 client.17001 shutdown 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 msgr2=0x556f26cac150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cac150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fac10009780 tx=0x7fac100303d0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 shutdown_connections 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26cbea20 0x556f26cc0e10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cac690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cac150 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 --2- 172.21.15.16:0/1796595928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cabc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 >> 172.21.15.16:0/1796595928 conn(0x556f26ca00f0 msgr2=0x556f26c9f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac05ffb640 0 client.17001 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 shutdown_connections 2024-09-12T02:54:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 1 -- 172.21.15.16:0/1796595928 wait complete. 2024-09-12T02:54:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.896+0000 7fac26a62740 20 client.17001 trim_cache size 0 max 16384 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 Processor -- start 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 -- start start 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cc6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1ad640 con 0x556f26ca4480 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1af180 con 0x556f26caed40 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac26a62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1add10 con 0x556f26ca3b60 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac21c24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac22c26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac21c24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41804/0 (socket says 172.21.15.16:41804) 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac21c24640 1 -- 172.21.15.16:0/3166832925 learned_addr learned my addr 172.21.15.16:0/3166832925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.915+0000 7fac22c26640 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cc6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 -- 172.21.15.16:0/3166832925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 msgr2=0x556f26cac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cac4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 -- 172.21.15.16:0/3166832925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 msgr2=0x556f26cc6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cc6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 -- 172.21.15.16:0/3166832925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f26cb1470 con 0x556f26ca4480 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac22c26640 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac21c24640 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac1002d020 tx=0x7fac10005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6285884acf92205 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac07fff640 1 -- 172.21.15.16:0/3166832925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10037070 con 0x556f26ca4480 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac07fff640 1 -- 172.21.15.16:0/3166832925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac100035f0 con 0x556f26ca4480 2024-09-12T02:54:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.916+0000 7fac07fff640 1 -- 172.21.15.16:0/3166832925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1004b040 con 0x556f26ca4480 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- 172.21.15.16:0/3166832925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 msgr2=0x556f26cc66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac1002d020 tx=0x7fac10005dc0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- 172.21.15.16:0/3166832925 shutdown_connections 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cc6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca4480 0x556f26cc66e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 --2- 172.21.15.16:0/3166832925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca3b60 0x556f26cac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- 172.21.15.16:0/3166832925 >> 172.21.15.16:0/3166832925 conn(0x556f26ca00f0 msgr2=0x556f26cadb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- 172.21.15.16:0/3166832925 shutdown_connections 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- 172.21.15.16:0/3166832925 wait complete. 2024-09-12T02:54:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 -- start start 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.917+0000 7fac26a62740 1 Processor -- start 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cb7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1aeab0 con 0x556f26ca3b60 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1ae3e0 con 0x556f26caed40 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1ab620 con 0x556f26ca4480 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac26a62740 10 client.0 _ll_register_callbacks cb 0x7fac18005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41818/0 (socket says 172.21.15.16:41818) 2024-09-12T02:54:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac21c24640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cb7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 -- 172.21.15.16:0/2932707898 learned_addr learned my addr 172.21.15.16:0/2932707898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 -- 172.21.15.16:0/2932707898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 msgr2=0x556f26cb7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cb7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 -- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 msgr2=0x556f26cb2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 -- 172.21.15.16:0/2932707898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f26cb2a30 con 0x556f26ca3b60 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac21c24640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cb7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac22c26640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac1400f4f0 tx=0x7fac10005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ff65bb6ce5b649 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.918+0000 7fac05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1401e030 con 0x556f26ca3b60 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac14004370 con 0x556f26ca3b60 2024-09-12T02:54:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac100035f0 con 0x556f26ca3b60 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fac14003c60 con 0x556f26ca3b60 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 10 client.17007 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f26d29210 con 0x556f26ca3b60 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac100035f0 con 0x556f26ca3b60 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fac140194c0 con 0x556f26ca3b60 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 10 client.17007 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 20 client.17007 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26d29740 0x556f26d2bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 4 client.17007 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.919+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556f26c10d00 con 0x556f26d29740 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.920+0000 7fac21c24640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26d29740 0x556f26d2bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.920+0000 7fac21c24640 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26d29740 0x556f26d2bb30 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=488bd674872aec30 server_cookie=627dbfad51709c47 in_seq=0 out_seq=0 2024-09-12T02:54:24.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.920+0000 7fac05ffb640 10 client.17007 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:24.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac05ffb640 1 -- 172.21.15.16:0/2932707898 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556f26c10d00 con 0x556f26d29740 2024-09-12T02:54:24.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac05ffb640 10 client.17007 handle_command_reply: tid=0 2024-09-12T02:54:24.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 client.17007 shutdown 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 msgr2=0x556f26cb6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac1400f4f0 tx=0x7fac10005d50 comp rx=0 tx=0).stop 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 shutdown_connections 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556f26d29740 0x556f26d2bb30 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f26caed40 0x556f26cb2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556f26ca4480 0x556f26cb7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 --2- 172.21.15.16:0/2932707898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f26ca3b60 0x556f26cb6e40 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac05ffb640 0 client.17007 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 >> 172.21.15.16:0/2932707898 conn(0x556f26ca00f0 msgr2=0x556f26caa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 shutdown_connections 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 1 -- 172.21.15.16:0/2932707898 wait complete. 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.921+0000 7fac26a62740 20 client.17007 trim_cache size 0 max 16384 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "12fe430f-6db4-443b-b231-495628051baa", 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:54:24.928 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 msgr2=0x7fac080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 0x7fac080a5040 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac180cd250 tx=0x7fac180d2eb0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 msgr2=0x7fac1c1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fac1800e9c0 tx=0x7fac1800edc0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 shutdown_connections 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac1c108290 0x7fac1c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac1c103740 0x7fac1c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fac080a2b80 0x7fac080a5040 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 --2- 172.21.15.16:0/1600491062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c102d90 0x7fac1c1985b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:24.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 >> 172.21.15.16:0/1600491062 conn(0x7fac1c0fc7d0 msgr2=0x7fac1c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:24.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 shutdown_connections 2024-09-12T02:54:24.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:24.922+0000 7fac24689640 1 -- 172.21.15.16:0/1600491062 wait complete. 2024-09-12T02:54:25.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:54:25.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 626545 10.63 MB/sec execute 207 sec latency 16.372 ms 2024-09-12T02:54:25.924 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: pgmap v532: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:26 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: pgmap v532: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-12T02:54:26.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:54:26.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:54:26.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:54:26.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:26 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 -- 172.21.15.16:0/1597614653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c103660 msgr2=0x7fee0c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 --2- 172.21.15.16:0/1597614653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c103660 0x7fee0c10bb50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee0002a480 comp rx=0 tx=0).stop 2024-09-12T02:54:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 -- 172.21.15.16:0/1597614653 shutdown_connections 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 --2- 172.21.15.16:0/1597614653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 --2- 172.21.15.16:0/1597614653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c103660 0x7fee0c10bb50 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 --2- 172.21.15.16:0/1597614653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c102d40 0x7fee0c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 -- 172.21.15.16:0/1597614653 >> 172.21.15.16:0/1597614653 conn(0x7fee0c0fc7a0 msgr2=0x7fee0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 -- 172.21.15.16:0/1597614653 shutdown_connections 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.406+0000 7fee12497640 1 -- 172.21.15.16:0/1597614653 wait complete. 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 Processor -- start 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 -- start start 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 0x7fee0c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c111fc0 con 0x7fee0c102d40 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c111e40 con 0x7fee0c103660 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.407+0000 7fee12497640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c112140 con 0x7fee0c10c170 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33846/0 (socket says 172.21.15.16:33846) 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 -- 172.21.15.16:0/2769161652 learned_addr learned my addr 172.21.15.16:0/2769161652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 0x7fee0c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee10a34640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 -- 172.21.15.16:0/2769161652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 msgr2=0x7fee0c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 -- 172.21.15.16:0/2769161652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 msgr2=0x7fee0c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 0x7fee0c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 -- 172.21.15.16:0/2769161652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c1aae00 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 0x7fee0c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee10a34640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.408+0000 7fee0bfff640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00e9f0 tx=0x7fedfc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.409+0000 7fee097fa640 1 -- 172.21.15.16:0/2769161652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00cd80 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.409+0000 7fee097fa640 1 -- 172.21.15.16:0/2769161652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc010910 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.409+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c1aafd0 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.409+0000 7fee097fa640 1 -- 172.21.15.16:0/2769161652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc018990 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.409+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1ab460 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.411+0000 7fee097fa640 1 -- 172.21.15.16:0/2769161652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fedfc018b70 con 0x7fee0c102d40 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.411+0000 7fee14870740 1 Processor -- start 2024-09-12T02:54:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 -- start start 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0c4a0 0x5628a6e0c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 0x5628a6e17140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e19a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5628a6d722e0 con 0x5628a6e0c4a0 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628a6d19ce0 con 0x5628a6e17680 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee14870740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628a6d6e2a0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 0x5628a6e17140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee10a34640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0c4a0 0x5628a6e0c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 0x5628a6e17140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51932/0 (socket says 172.21.15.16:51932) 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee0bfff640 1 -- 172.21.15.16:0/1418971053 learned_addr learned my addr 172.21.15.16:0/1418971053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:26.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.412+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e19a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 -- 172.21.15.16:0/1418971053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 msgr2=0x5628a6e17140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 0x5628a6e17140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 -- 172.21.15.16:0/1418971053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0c4a0 msgr2=0x5628a6e0c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0c4a0 0x5628a6e0c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 -- 172.21.15.16:0/1418971053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a6cfe440 con 0x5628a6e17680 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e19a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee0002aa70 tx=0x7fee00000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f20250d4240ba3a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee097fa640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 0x7fedf40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fedf27fc640 1 -- 172.21.15.16:0/1418971053 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00035040 con 0x5628a6e17680 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fedf27fc640 1 -- 172.21.15.16:0/1418971053 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00004760 con 0x5628a6e17680 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fedf27fc640 1 -- 172.21.15.16:0/1418971053 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee000038e0 con 0x5628a6e17680 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee10a34640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 0x7fedf40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.413+0000 7fee097fa640 1 -- 172.21.15.16:0/2769161652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedfc0cf390 con 0x7fee0c102d40 2024-09-12T02:54:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 -- 172.21.15.16:0/1418971053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 msgr2=0x5628a6e19a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee10a34640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 0x7fedf40a50c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e19a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee0002aa70 tx=0x7fee00000bc0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 -- 172.21.15.16:0/1418971053 shutdown_connections 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e19a70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0cdc0 0x5628a6e17140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 --2- 172.21.15.16:0/1418971053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0c4a0 0x5628a6e0c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 -- 172.21.15.16:0/1418971053 >> 172.21.15.16:0/1418971053 conn(0x5628a6e08a30 msgr2=0x5628a6e08e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 -- 172.21.15.16:0/1418971053 shutdown_connections 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.414+0000 7fee14870740 1 -- 172.21.15.16:0/1418971053 wait complete. 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 -- start start 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 Processor -- start 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e14200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e14740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5628a6e1a250 con 0x5628a6e0cdc0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.415+0000 7fee14870740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628a6d2d050 con 0x5628a6e17680 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee14870740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628a6e1a3d0 con 0x5628a6e0c4a0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee14870740 10 client.0 _ll_register_callbacks cb 0x5628a6d6d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e14200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:51942/0 (socket says 172.21.15.16:51942) 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 -- 172.21.15.16:0/807093134 learned_addr learned my addr 172.21.15.16:0/807093134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee10a34640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e14740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 -- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 msgr2=0x5628a6e14740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e14740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 -- 172.21.15.16:0/807093134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 msgr2=0x5628a6e14200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e14200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 -- 172.21.15.16:0/807093134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628a6ae33c0 con 0x5628a6e0c4a0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e14200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee10a34640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e14740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee0bfff640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0a4340 tx=0x7fedfc09b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53dccb31b9ceaf13 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fedeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc018e60 con 0x5628a6e0c4a0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedfc098270 con 0x5628a6e0c4a0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.416+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee00004760 con 0x5628a6e0c4a0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fedfc09c300 con 0x5628a6e0c4a0 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 10 client.26449 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5628a6e25c30 con 0x5628a6e0c4a0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00004760 con 0x5628a6e0c4a0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fedfc09b050 con 0x5628a6e0c4a0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 10 client.26449 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 20 client.26449 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e270f0 0x5628a6e29580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 4 client.26449 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628a6d7f850 con 0x5628a6e270f0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.417+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e270f0 0x5628a6e29580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.418+0000 7fedeb7fe640 10 client.26449 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.418+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e270f0 0x5628a6e29580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ebed85aabe29380 server_cookie=2af25aca6e848d63 in_seq=0 out_seq=0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.418+0000 7fedeb7fe640 1 -- 172.21.15.16:0/807093134 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5628a6d7f850 con 0x5628a6e270f0 2024-09-12T02:54:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.418+0000 7fedeb7fe640 10 client.26449 handle_command_reply: tid=0 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.418+0000 7fee14870740 1 client.26449 shutdown 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 msgr2=0x5628a6e16f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0a4340 tx=0x7fedfc09b9d0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 shutdown_connections 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e270f0 0x5628a6e29580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e17680 0x5628a6e14740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e14200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 --2- 172.21.15.16:0/807093134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e16f50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 >> 172.21.15.16:0/807093134 conn(0x5628a6e08a30 msgr2=0x5628a6e0e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fedeb7fe640 0 client.26449 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 shutdown_connections 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 1 -- 172.21.15.16:0/807093134 wait complete. 2024-09-12T02:54:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.419+0000 7fee14870740 20 client.26449 trim_cache size 0 max 16384 2024-09-12T02:54:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 Processor -- start 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 -- start start 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0c4a0 0x5628a6e234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e17680 0x5628a6e2ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b3c20 con 0x5628a6e0cdc0 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b5760 con 0x5628a6e0c4a0 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee14870740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b42f0 con 0x5628a6e17680 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33862/0 (socket says 172.21.15.16:33862) 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3821323714 learned_addr learned my addr 172.21.15.16:0/3821323714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee10a34640 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e17680 0x5628a6e2ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.440+0000 7fee0bfff640 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0c4a0 0x5628a6e234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3821323714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e17680 msgr2=0x5628a6e2ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e17680 0x5628a6e2ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3821323714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0c4a0 msgr2=0x5628a6e234e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0c4a0 0x5628a6e234e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3821323714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a6e08860 con 0x5628a6e0cdc0 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee00031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dfdd579efd2054 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fedf1ffb640 1 -- 172.21.15.16:0/3821323714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00035040 con 0x5628a6e0cdc0 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fedf1ffb640 1 -- 172.21.15.16:0/3821323714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee000315b0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.441+0000 7fedf1ffb640 1 -- 172.21.15.16:0/3821323714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0004f6a0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 -- 172.21.15.16:0/3821323714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 msgr2=0x5628a6e0f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee00031510 comp rx=0 tx=0).stop 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 -- 172.21.15.16:0/3821323714 shutdown_connections 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e17680 0x5628a6e2ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e0cdc0 0x5628a6e0f7f0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 --2- 172.21.15.16:0/3821323714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0c4a0 0x5628a6e234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 -- 172.21.15.16:0/3821323714 >> 172.21.15.16:0/3821323714 conn(0x5628a6e08a30 msgr2=0x5628a6e0f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 -- 172.21.15.16:0/3821323714 shutdown_connections 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.442+0000 7fee14870740 1 -- 172.21.15.16:0/3821323714 wait complete. 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 -- start start 2024-09-12T02:54:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 Processor -- start 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e1abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e17680 0x5628a6e8fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b5090 con 0x5628a6e17680 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b49c0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b1c00 con 0x5628a6e0c4a0 2024-09-12T02:54:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee14870740 10 client.0 _ll_register_callbacks cb 0x7fedfc0d2980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee0b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35954/0 (socket says 172.21.15.16:35954) 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.443+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3901618004 learned_addr learned my addr 172.21.15.16:0/3901618004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee10a34640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e17680 0x5628a6e8fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0bfff640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e1abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3901618004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 msgr2=0x5628a6e1abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e1abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3901618004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e17680 msgr2=0x5628a6e8fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e17680 0x5628a6e8fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 -- 172.21.15.16:0/3901618004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628a6e90300 con 0x5628a6e0cdc0 2024-09-12T02:54:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee0b7fe640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fee00006010 tx=0x7fee00031f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b8c6f81b3a9da9e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fedeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00031c30 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fee000038c0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee000315b0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee00003ac0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee14870740 10 client.26555 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.444+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5628a6e915f0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee000315b0 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee00062020 con 0x5628a6e0cdc0 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 10 client.26555 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 10 client.26555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 20 client.26555 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e918e0 0x5628a6e93d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 4 client.26555 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5628a6d79d60 con 0x5628a6e918e0 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee0bfff640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e918e0 0x5628a6e93d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fee0bfff640 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e918e0 0x5628a6e93d00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf6b673d6ae3b587 server_cookie=13ce826b666f0b8b in_seq=0 out_seq=0 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.445+0000 7fedeb7fe640 10 client.26555 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fedeb7fe640 1 -- 172.21.15.16:0/3901618004 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5628a6d79d60 con 0x5628a6e918e0 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fedeb7fe640 10 client.26555 handle_command_reply: tid=0 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 client.26555 shutdown 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 msgr2=0x5628a6e1b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fee00006010 tx=0x7fee00031f40 comp rx=0 tx=0).stop 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 shutdown_connections 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628a6e918e0 0x5628a6e93d00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628a6e17680 0x5628a6e8fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628a6e0cdc0 0x5628a6e1b0f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 --2- 172.21.15.16:0/3901618004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628a6e0c4a0 0x5628a6e1abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 >> 172.21.15.16:0/3901618004 conn(0x5628a6e08a30 msgr2=0x5628a6e07b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fedeb7fe640 0 client.26555 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 shutdown_connections 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 1 -- 172.21.15.16:0/3901618004 wait complete. 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.446+0000 7fee14870740 20 client.26555 trim_cache size 0 max 16384 2024-09-12T02:54:26.452 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "12fe430f-6db4-443b-b231-495628051baa": { 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "12fe430f-6db4-443b-b231-495628051baa", 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:26.453 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 msgr2=0x7fedf40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 0x7fedf40a50c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04009500 comp rx=0 tx=0).stop 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 msgr2=0x7fee0c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00e9f0 tx=0x7fedfc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 shutdown_connections 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c10c170 0x7fee0c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fee0c103660 0x7fee0c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedf40a2c00 0x7fedf40a50c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 --2- 172.21.15.16:0/2769161652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee0c102d40 0x7fee0c1009a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 >> 172.21.15.16:0/2769161652 conn(0x7fee0c0fc7a0 msgr2=0x7fee0c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 shutdown_connections 2024-09-12T02:54:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:26.447+0000 7fee12497640 1 -- 172.21.15.16:0/2769161652 wait complete. 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: pgmap v532: 97 pgs: 97 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:26 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:54:26.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 629283 10.65 MB/sec execute 208 sec latency 13.675 ms 2024-09-12T02:54:27.199 INFO:tasks.cephfs.filesystem:scrub status for tag:12fe430f-6db4-443b-b231-495628051baa - {'path': '/', 'tag': '12fe430f-6db4-443b-b231-495628051baa', 'options': 'recursive,force'} 2024-09-12T02:54:27.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 632404 10.69 MB/sec execute 209 sec latency 12.669 ms 2024-09-12T02:54:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:28 smithi184 ceph-mon[35321]: pgmap v533: 97 pgs: 97 active+clean; 233 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 238 op/s 2024-09-12T02:54:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:28 smithi016 ceph-mon[30925]: pgmap v533: 97 pgs: 97 active+clean; 233 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 238 op/s 2024-09-12T02:54:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:28 smithi033 ceph-mon[35092]: pgmap v533: 97 pgs: 97 active+clean; 233 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 238 op/s 2024-09-12T02:54:28.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 635085 10.70 MB/sec execute 210 sec latency 13.344 ms 2024-09-12T02:54:29.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 637571 10.70 MB/sec execute 211 sec latency 12.603 ms 2024-09-12T02:54:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:30 smithi184 ceph-mon[35321]: pgmap v534: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-09-12T02:54:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:30 smithi016 ceph-mon[30925]: pgmap v534: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-09-12T02:54:30.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:30 smithi033 ceph-mon[35092]: pgmap v534: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-09-12T02:54:30.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 640046 10.72 MB/sec execute 212 sec latency 78.528 ms 2024-09-12T02:54:31.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 643143 10.76 MB/sec execute 213 sec latency 11.835 ms 2024-09-12T02:54:32.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:32 smithi016 ceph-mon[30925]: pgmap v535: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-09-12T02:54:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:32 smithi033 ceph-mon[35092]: pgmap v535: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-09-12T02:54:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:32 smithi184 ceph-mon[35321]: pgmap v535: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-09-12T02:54:32.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 645715 10.77 MB/sec execute 214 sec latency 36.945 ms 2024-09-12T02:54:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:54:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:54:33.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:54:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:54:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:54:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:54:33.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 648054 10.77 MB/sec execute 215 sec latency 12.341 ms 2024-09-12T02:54:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:34 smithi016 ceph-mon[30925]: pgmap v536: 97 pgs: 97 active+clean; 241 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-12T02:54:34.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:34 smithi033 ceph-mon[35092]: pgmap v536: 97 pgs: 97 active+clean; 241 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-12T02:54:34.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:34 smithi184 ceph-mon[35321]: pgmap v536: 97 pgs: 97 active+clean; 241 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-09-12T02:54:34.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 650764 10.79 MB/sec execute 216 sec latency 11.784 ms 2024-09-12T02:54:35.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:35 smithi016 ceph-mon[30925]: pgmap v537: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-12T02:54:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:35 smithi033 ceph-mon[35092]: pgmap v537: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-12T02:54:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:35 smithi184 ceph-mon[35321]: pgmap v537: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-09-12T02:54:35.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 653849 10.82 MB/sec execute 217 sec latency 8.775 ms 2024-09-12T02:54:36.423 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 16.6 2024-09-12T02:54:36.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:54:36.713 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:36.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 656965 10.85 MB/sec execute 218 sec latency 14.755 ms 2024-09-12T02:54:37.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 -- 172.21.15.16:0/1050755349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081082a0 msgr2=0x7f260810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:37.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 --2- 172.21.15.16:0/1050755349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081082a0 0x7f260810ec40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2600009cd0 tx=0x7f260002a8f0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 -- 172.21.15.16:0/1050755349 shutdown_connections 2024-09-12T02:54:37.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 --2- 172.21.15.16:0/1050755349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081082a0 0x7f260810ec40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 --2- 172.21.15.16:0/1050755349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081036e0 0x7f2608107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 --2- 172.21.15.16:0/1050755349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 0x7f2608103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 -- 172.21.15.16:0/1050755349 >> 172.21.15.16:0/1050755349 conn(0x7f26080fc7d0 msgr2=0x7f26080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 -- 172.21.15.16:0/1050755349 shutdown_connections 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 -- 172.21.15.16:0/1050755349 wait complete. 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.221+0000 7f260ddb9640 1 Processor -- start 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 -- start start 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 0x7f2608100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081082a0 0x7f2608101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26081120a0 con 0x7f26081036e0 2024-09-12T02:54:37.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2608111f20 con 0x7f2608102d30 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f260ddb9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2608112220 con 0x7f26081082a0 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53956/0 (socket says 172.21.15.16:53956) 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 -- 172.21.15.16:0/2590789561 learned_addr learned my addr 172.21.15.16:0/2590789561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f26077fe640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 0x7f2608100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2607fff640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081082a0 0x7f2608101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 -- 172.21.15.16:0/2590789561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081082a0 msgr2=0x7f2608101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081082a0 0x7f2608101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 -- 172.21.15.16:0/2590789561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 msgr2=0x7f2608100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 0x7f2608100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 -- 172.21.15.16:0/2590789561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26081a69c0 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.222+0000 7f2606ffd640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00e9f0 tx=0x7f25fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.223+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc00ccc0 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.223+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25fc010910 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.223+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26081a6cb0 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.223+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc018920 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.223+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26081a71a0 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.224+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2608103110 con 0x7f26081036e0 2024-09-12T02:54:37.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.225+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f25fc010ac0 con 0x7f26081036e0 2024-09-12T02:54:37.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.226+0000 7f2604ff9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 0x7f25f00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:37.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.226+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f25fc0cf2e0 con 0x7f26081036e0 2024-09-12T02:54:37.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.226+0000 7f26077fe640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 0x7f25f00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:37.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.227+0000 7f26077fe640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 0x7f25f00a5040 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f25f800bbe0 tx=0x7f25f800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:37.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.231+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f25fc09c250 con 0x7f26081036e0 2024-09-12T02:54:37.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:37 smithi016 ceph-mon[30925]: pgmap v538: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 258 op/s 2024-09-12T02:54:37.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.484+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26081a73d0 con 0x7f26081036e0 2024-09-12T02:54:37.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.484+0000 7f2604ff9640 1 -- 172.21.15.16:0/2590789561 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f25fc09bb50 con 0x7f26081036e0 2024-09-12T02:54:37.485 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:54:37.485 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:54:37.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:54:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 msgr2=0x7f25f00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 0x7f25f00a5040 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f25f800bbe0 tx=0x7f25f800b390 comp rx=0 tx=0).stop 2024-09-12T02:54:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 msgr2=0x7f2608100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00e9f0 tx=0x7f25fc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 shutdown_connections 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26081082a0 0x7f2608101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f25f00a2b80 0x7f25f00a5040 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26081036e0 0x7f2608100ed0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 --2- 172.21.15.16:0/2590789561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2608102d30 0x7f2608100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 >> 172.21.15.16:0/2590789561 conn(0x7f26080fc7d0 msgr2=0x7f2608105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 shutdown_connections 2024-09-12T02:54:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:37.487+0000 7f260ddb9640 1 -- 172.21.15.16:0/2590789561 wait complete. 2024-09-12T02:54:37.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:37 smithi033 ceph-mon[35092]: pgmap v538: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 258 op/s 2024-09-12T02:54:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:37 smithi184 ceph-mon[35321]: pgmap v538: 97 pgs: 97 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 258 op/s 2024-09-12T02:54:37.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 658574 10.84 MB/sec execute 219 sec latency 73.830 ms 2024-09-12T02:54:38.248 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 136.6 --await-for 45.0' 2024-09-12T02:54:38.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 136.6 --await-for 45.0 2024-09-12T02:54:38.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:38 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2590789561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:38 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2590789561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:38.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:38 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2590789561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:38.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 661310 10.85 MB/sec execute 220 sec latency 11.276 ms 2024-09-12T02:54:39.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 -- 172.21.15.16:0/337459830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 msgr2=0x7f2370106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 --2- 172.21.15.16:0/337459830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f2370106e60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2360009cd0 tx=0x7f236002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 -- 172.21.15.16:0/337459830 shutdown_connections 2024-09-12T02:54:39.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 --2- 172.21.15.16:0/337459830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370101420 0x7f237010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 --2- 172.21.15.16:0/337459830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370100a80 0x7f2370100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 --2- 172.21.15.16:0/337459830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f2370106e60 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 -- 172.21.15.16:0/337459830 >> 172.21.15.16:0/337459830 conn(0x7f23700fc7a0 msgr2=0x7f23700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 -- 172.21.15.16:0/337459830 shutdown_connections 2024-09-12T02:54:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 -- 172.21.15.16:0/337459830 wait complete. 2024-09-12T02:54:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.031+0000 7f2377380640 1 Processor -- start 2024-09-12T02:54:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 -- start start 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370101420 0x7f23700fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f23700fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2370110e80 con 0x7f2370106a80 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2370110d00 con 0x7f2370100a80 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f2377380640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2370111000 con 0x7f2370101420 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f23700fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f23700fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53962/0 (socket says 172.21.15.16:53962) 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237591d640 1 -- 172.21.15.16:0/3938298244 learned_addr learned my addr 172.21.15.16:0/3938298244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:56948/0 (socket says 172.21.15.16:56948) 2024-09-12T02:54:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237491b640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370101420 0x7f23700fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 -- 172.21.15.16:0/3938298244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370101420 msgr2=0x7f23700fd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370101420 0x7f23700fd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 -- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 msgr2=0x7f23700fd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f23700fd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.032+0000 7f237511c640 1 -- 172.21.15.16:0/3938298244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23701a68e0 con 0x7f2370100a80 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f237511c640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2360000f80 tx=0x7f2360031e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f236e7fc640 1 -- 172.21.15.16:0/3938298244 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360003e40 con 0x7f2370100a80 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f236e7fc640 1 -- 172.21.15.16:0/3938298244 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360004520 con 0x7f2370100a80 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23701a6b70 con 0x7f2370100a80 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f236e7fc640 1 -- 172.21.15.16:0/3938298244 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360030d10 con 0x7f2370100a80 2024-09-12T02:54:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.033+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23701a70b0 con 0x7f2370100a80 2024-09-12T02:54:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.036+0000 7f2379759740 1 Processor -- start 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.036+0000 7f2379759740 1 -- start start 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb44fac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fb444d2a0 con 0x559fb44fb150 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559fb43f4b10 con 0x559fb44f0890 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f2379759740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559fb4449260 con 0x559fb44eff70 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:50136/0 (socket says 172.21.15.16:50136) 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 -- 172.21.15.16:0/2742513183 learned_addr learned my addr 172.21.15.16:0/2742513183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237491b640 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 -- 172.21.15.16:0/2742513183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 msgr2=0x559fb44fac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb44fac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 -- 172.21.15.16:0/2742513183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 msgr2=0x559fb44fd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f236e7fc640 1 -- 172.21.15.16:0/3938298244 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f23600046d0 con 0x7f2370100a80 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 -- 172.21.15.16:0/2742513183 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb41beb90 con 0x559fb44eff70 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237491b640 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.037+0000 7f237591d640 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2368009d20 tx=0x7f2368011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c64a696de578de09 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f23577fe640 1 -- 172.21.15.16:0/2742513183 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236801e070 con 0x559fb44eff70 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f23577fe640 1 -- 172.21.15.16:0/2742513183 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2368019b10 con 0x559fb44eff70 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f23577fe640 1 -- 172.21.15.16:0/2742513183 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236800fa50 con 0x559fb44eff70 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f236e7fc640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 0x7f23580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f237511c640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 0x7f23580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.038+0000 7f236e7fc640 1 -- 172.21.15.16:0/3938298244 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f236002e030 con 0x7f2370100a80 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 -- 172.21.15.16:0/2742513183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 msgr2=0x559fb44f0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2368009d20 tx=0x7f2368011810 comp rx=0 tx=0).stop 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f237511c640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 0x7f23580a5040 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23600b95a0 tx=0x7f236003d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 -- 172.21.15.16:0/2742513183 shutdown_connections 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb44fac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 --2- 172.21.15.16:0/2742513183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb44f0350 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 -- 172.21.15.16:0/2742513183 >> 172.21.15.16:0/2742513183 conn(0x559fb44ec500 msgr2=0x559fb44ec8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 -- 172.21.15.16:0/2742513183 shutdown_connections 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.039+0000 7f2379759740 1 -- 172.21.15.16:0/2742513183 wait complete. 2024-09-12T02:54:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.040+0000 7f2379759740 1 -- start start 2024-09-12T02:54:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 Processor -- start 2024-09-12T02:54:39.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44f89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fb44fdd20 con 0x559fb44fb150 2024-09-12T02:54:39.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559fb43d56e0 con 0x559fb44eff70 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559fb44fdea0 con 0x559fb44f0890 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f2379759740 10 client.0 _ll_register_callbacks cb 0x559fb43f9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.041+0000 7f237591d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:56954/0 (socket says 172.21.15.16:56954) 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 -- 172.21.15.16:0/3669594088 learned_addr learned my addr 172.21.15.16:0/3669594088 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44f89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 -- 172.21.15.16:0/3669594088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 msgr2=0x559fb44f8480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f8480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 -- 172.21.15.16:0/3669594088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 msgr2=0x559fb44f89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44f89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.042+0000 7f237591d640 1 -- 172.21.15.16:0/3669594088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb45071c0 con 0x559fb44eff70 2024-09-12T02:54:39.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f237591d640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f236800f530 tx=0x7f236800f630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e437d7c58b922bd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236801e070 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f236800fbe0 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2368019b10 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2368019930 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2379759740 10 client.26570 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559fb4509ad0 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.043+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f236802e070 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2368029410 con 0x559fb44eff70 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 10 client.26570 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 10 client.26570 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 20 client.26570 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb450af30 0x559fb450d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 4 client.26570 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2368019b10 con 0x559fb450af30 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.044+0000 7f237491b640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb450af30 0x559fb450d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.045+0000 7f2354ff9640 10 client.26570 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.045+0000 7f237491b640 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb450af30 0x559fb450d320 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64e115b0572b0498 server_cookie=9f52d58ff13cda98 in_seq=0 out_seq=0 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.045+0000 7f2354ff9640 1 -- 172.21.15.16:0/3669594088 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2368019b10 con 0x559fb450af30 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.045+0000 7f2354ff9640 10 client.26570 handle_command_reply: tid=0 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.045+0000 7f2379759740 1 client.26570 shutdown 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 msgr2=0x559fb44f7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f236800f530 tx=0x7f236800f630 comp rx=0 tx=0).stop 2024-09-12T02:54:39.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 shutdown_connections 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb450af30 0x559fb450d320 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44f89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 --2- 172.21.15.16:0/3669594088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f7f40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2354ff9640 0 client.26570 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 >> 172.21.15.16:0/3669594088 conn(0x559fb44ec500 msgr2=0x559fb44e2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 shutdown_connections 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 1 -- 172.21.15.16:0/3669594088 wait complete. 2024-09-12T02:54:39.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.046+0000 7f2379759740 20 client.26570 trim_cache size 0 max 16384 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 Processor -- start 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 -- start start 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23701ab2d0 con 0x559fb44fb150 2024-09-12T02:54:39.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23701ace10 con 0x559fb44eff70 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f2379759740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23701ab9a0 con 0x559fb44f0890 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f237591d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53980/0 (socket says 172.21.15.16:53980) 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f237591d640 1 -- 172.21.15.16:0/3642733637 learned_addr learned my addr 172.21.15.16:0/3642733637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.068+0000 7f237491b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 -- 172.21.15.16:0/3642733637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 msgr2=0x559fb44f9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 -- 172.21.15.16:0/3642733637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 msgr2=0x559fb44f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 -- 172.21.15.16:0/3642733637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb44ec330 con 0x559fb44fb150 2024-09-12T02:54:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f237591d640 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f236800f600 tx=0x7f2368000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec197956c8781ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f2356ffd640 1 -- 172.21.15.16:0/3642733637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236801e070 con 0x559fb44fb150 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f2356ffd640 1 -- 172.21.15.16:0/3642733637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f236802e040 con 0x559fb44fb150 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.069+0000 7f2356ffd640 1 -- 172.21.15.16:0/3642733637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368035480 con 0x559fb44fb150 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- 172.21.15.16:0/3642733637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 msgr2=0x559fb4512ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f236800f600 tx=0x7f2368000bc0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- 172.21.15.16:0/3642733637 shutdown_connections 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb4512ab0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44f0890 0x559fb44f9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 --2- 172.21.15.16:0/3642733637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44eff70 0x559fb44f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- 172.21.15.16:0/3642733637 >> 172.21.15.16:0/3642733637 conn(0x559fb44ec500 msgr2=0x559fb44f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- 172.21.15.16:0/3642733637 shutdown_connections 2024-09-12T02:54:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- 172.21.15.16:0/3642733637 wait complete. 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.070+0000 7f2379759740 1 -- start start 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 Processor -- start 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb4503310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb4503850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23701ac740 con 0x559fb44fb150 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23701ac070 con 0x559fb44f0890 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23701a92b0 con 0x559fb44eff70 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f2379759740 10 client.0 _ll_register_callbacks cb 0x7f2368009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f237591d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fe0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53994/0 (socket says 172.21.15.16:53994) 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f237591d640 1 -- 172.21.15.16:0/2982438921 learned_addr learned my addr 172.21.15.16:0/2982438921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f237491b640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb4503850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.071+0000 7f237591d640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb4503310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 -- 172.21.15.16:0/2982438921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 msgr2=0x559fb4503310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb4503310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 -- 172.21.15.16:0/2982438921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 msgr2=0x559fb44fe0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fe0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 -- 172.21.15.16:0/2982438921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb44df260 con 0x559fb44f0890 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f237491b640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb4503850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x559fb450da50 tx=0x7f236401c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ddad655fcf253e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364010840 con 0x559fb44f0890 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f23640109e0 con 0x559fb44f0890 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f236802e040 con 0x559fb44f0890 2024-09-12T02:54:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2364010be0 con 0x559fb44f0890 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2379759740 10 client.26576 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.072+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559fb4576260 con 0x559fb44f0890 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f236802e040 con 0x559fb44f0890 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2364030460 con 0x559fb44f0890 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 10 client.26576 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 10 client.26576 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 20 client.26576 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb45769f0 0x559fb4578e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 4 client.26576 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 136.6, "await_for": 45.0}] 2024-09-12T02:54:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 136.6, "await_for": 45.0}) -- 0x559fb445a9d0 con 0x559fb45769f0 2024-09-12T02:54:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.073+0000 7f237591d640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb45769f0 0x559fb4578e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.074+0000 7f2354ff9640 10 client.26576 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.074+0000 7f237591d640 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb45769f0 0x559fb4578e10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33a21cebe1a9d87b server_cookie=698963c6cfc71f60 in_seq=0 out_seq=0 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2354ff9640 1 -- 172.21.15.16:0/2982438921 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559fb445a9d0 con 0x559fb45769f0 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2354ff9640 10 client.26576 handle_command_reply: tid=0 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 client.26576 shutdown 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 msgr2=0x559fb4503850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb4503850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x559fb450da50 tx=0x7f236401c7a0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 shutdown_connections 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fb45769f0 0x559fb4578e10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fb44fb150 0x559fb44fe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fb44f0890 0x559fb4503850 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 --2- 172.21.15.16:0/2982438921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fb44eff70 0x559fb4503310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2354ff9640 0 client.26576 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 >> 172.21.15.16:0/2982438921 conn(0x559fb44ec500 msgr2=0x559fb44f9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 shutdown_connections 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 1 -- 172.21.15.16:0/2982438921 wait complete. 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.157+0000 7f2379759740 20 client.26576 trim_cache size 0 max 16384 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:54:39.159 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 26, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "ea1d6443": { 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 26, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 136.6, 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:54:39.160 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:39.161 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 msgr2=0x7f23580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 0x7f23580a5040 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23600b95a0 tx=0x7f236003d040 comp rx=0 tx=0).stop 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 msgr2=0x7f23700fceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2360000f80 tx=0x7f2360031e40 comp rx=0 tx=0).stop 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 shutdown_connections 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f23580a2b80 0x7f23580a5040 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2370106a80 0x7f23700fd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2370101420 0x7f23700fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 --2- 172.21.15.16:0/3938298244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2370100a80 0x7f23700fceb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 >> 172.21.15.16:0/3938298244 conn(0x7f23700fc7a0 msgr2=0x7f237006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 shutdown_connections 2024-09-12T02:54:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:39.158+0000 7f2377380640 1 -- 172.21.15.16:0/3938298244 wait complete. 2024-09-12T02:54:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:39 smithi033 ceph-mon[35092]: pgmap v539: 97 pgs: 97 active+clean; 244 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-09-12T02:54:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:39 smithi184 ceph-mon[35321]: pgmap v539: 97 pgs: 97 active+clean; 244 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-09-12T02:54:39.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:39 smithi016 ceph-mon[30925]: pgmap v539: 97 pgs: 97 active+clean; 244 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-09-12T02:54:39.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.83 MB/sec execute 221 sec latency 638.793 ms 2024-09-12T02:54:40.293 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ea1d6443', quiesce took 3.9 seconds. Will release after: 12.700000000000001 2024-09-12T02:54:40.723 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.78 MB/sec execute 222 sec latency 1638.911 ms 2024-09-12T02:54:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:41 smithi033 ceph-mon[35092]: pgmap v540: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 301 op/s 2024-09-12T02:54:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:41 smithi184 ceph-mon[35321]: pgmap v540: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 301 op/s 2024-09-12T02:54:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:41 smithi016 ceph-mon[30925]: pgmap v540: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 301 op/s 2024-09-12T02:54:41.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.73 MB/sec execute 223 sec latency 2639.011 ms 2024-09-12T02:54:42.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.68 MB/sec execute 224 sec latency 3639.082 ms 2024-09-12T02:54:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:43 smithi033 ceph-mon[35092]: pgmap v541: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-09-12T02:54:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:43 smithi184 ceph-mon[35321]: pgmap v541: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-09-12T02:54:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:43 smithi016 ceph-mon[30925]: pgmap v541: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-09-12T02:54:43.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.63 MB/sec execute 225 sec latency 4639.122 ms 2024-09-12T02:54:44.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.59 MB/sec execute 226 sec latency 5639.234 ms 2024-09-12T02:54:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:45 smithi033 ceph-mon[35092]: pgmap v542: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-09-12T02:54:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:45 smithi184 ceph-mon[35321]: pgmap v542: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-09-12T02:54:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:45 smithi016 ceph-mon[30925]: pgmap v542: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-09-12T02:54:45.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.54 MB/sec execute 227 sec latency 6639.324 ms 2024-09-12T02:54:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.49 MB/sec execute 228 sec latency 7639.446 ms 2024-09-12T02:54:47.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:47 smithi033 ceph-mon[35092]: pgmap v543: 97 pgs: 97 active+clean; 247 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-09-12T02:54:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:47 smithi184 ceph-mon[35321]: pgmap v543: 97 pgs: 97 active+clean; 247 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-09-12T02:54:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:47 smithi016 ceph-mon[30925]: pgmap v543: 97 pgs: 97 active+clean; 247 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-09-12T02:54:47.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.45 MB/sec execute 229 sec latency 8639.541 ms 2024-09-12T02:54:48.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.40 MB/sec execute 230 sec latency 9639.647 ms 2024-09-12T02:54:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:49 smithi033 ceph-mon[35092]: pgmap v544: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-09-12T02:54:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:49 smithi184 ceph-mon[35321]: pgmap v544: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-09-12T02:54:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:49 smithi016 ceph-mon[30925]: pgmap v544: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-09-12T02:54:49.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.36 MB/sec execute 231 sec latency 10639.772 ms 2024-09-12T02:54:50.724 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.31 MB/sec execute 232 sec latency 11639.876 ms 2024-09-12T02:54:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:51 smithi033 ceph-mon[35092]: pgmap v545: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 185 op/s 2024-09-12T02:54:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:51 smithi184 ceph-mon[35321]: pgmap v545: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 185 op/s 2024-09-12T02:54:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:51 smithi016 ceph-mon[30925]: pgmap v545: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 185 op/s 2024-09-12T02:54:51.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.27 MB/sec execute 233 sec latency 12639.993 ms 2024-09-12T02:54:52.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.22 MB/sec execute 234 sec latency 13640.115 ms 2024-09-12T02:54:52.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:54:53.269 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:53.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:53 smithi016 ceph-mon[30925]: pgmap v546: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:54:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:53 smithi033 ceph-mon[35092]: pgmap v546: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:54:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:53 smithi184 ceph-mon[35321]: pgmap v546: 97 pgs: 97 active+clean; 249 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-09-12T02:54:53.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.18 MB/sec execute 235 sec latency 14640.179 ms 2024-09-12T02:54:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.730+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3558193548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70101de0 msgr2=0x7f1b701021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.730+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3558193548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70101de0 0x7f1b701021c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009ca0 tx=0x7f1b6002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3558193548 shutdown_connections 2024-09-12T02:54:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3558193548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3558193548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70102790 0x7f1b701067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3558193548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70101de0 0x7f1b701021c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3558193548 >> 172.21.15.16:0/3558193548 conn(0x7f1b700fd8c0 msgr2=0x7f1b700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3558193548 shutdown_connections 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3558193548 wait complete. 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 Processor -- start 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- start start 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70101de0 0x7f1b701983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b70111f70 con 0x7f1b70102790 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b70111df0 con 0x7f1b70101de0 2024-09-12T02:54:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.731+0000 7f1b76f2b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1b701120f0 con 0x7f1b70106f10 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38540/0 (socket says 172.21.15.16:38540) 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 -- 172.21.15.16:0/3467532807 learned_addr learned my addr 172.21.15.16:0/3467532807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b74cc7640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70101de0 0x7f1b701983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b754c8640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 -- 172.21.15.16:0/3467532807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 msgr2=0x7f1b7019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 -- 172.21.15.16:0/3467532807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70101de0 msgr2=0x7f1b701983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70101de0 0x7f1b701983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 -- 172.21.15.16:0/3467532807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7019d380 con 0x7f1b70102790 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b754c8640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:54:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6ffff640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c890 tx=0x7f1b5800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b58004220 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7019d670 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b580043c0 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b58004a00 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.732+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b701a4f50 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.733+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b701021c0 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.735+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1b580079b0 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.735+0000 7f1b6dffb640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 0x7f1b5c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.735+0000 7f1b74cc7640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 0x7f1b5c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.735+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1b580cedc0 con 0x7f1b70102790 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.736+0000 7f1b74cc7640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 0x7f1b5c0a5040 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1b6002f540 tx=0x7f1b6002f310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.740+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b58014030 con 0x7f1b70102790 2024-09-12T02:54:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.954+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b70199730 con 0x7f1b70102790 2024-09-12T02:54:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.955+0000 7f1b6dffb640 1 -- 172.21.15.16:0/3467532807 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f1b70199730 con 0x7f1b70102790 2024-09-12T02:54:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:54:53.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:54:53.956 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:54:53.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 msgr2=0x7f1b5c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:53.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 0x7f1b5c0a5040 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1b6002f540 tx=0x7f1b6002f310 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 msgr2=0x7f1b70198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c890 tx=0x7f1b5800cd50 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 shutdown_connections 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b70106f10 0x7f1b7019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1b5c0a2b80 0x7f1b5c0a5040 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b70102790 0x7f1b70198910 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 --2- 172.21.15.16:0/3467532807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b70101de0 0x7f1b701983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 >> 172.21.15.16:0/3467532807 conn(0x7f1b700fd8c0 msgr2=0x7f1b700febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 shutdown_connections 2024-09-12T02:54:53.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:53.957+0000 7f1b76f2b640 1 -- 172.21.15.16:0/3467532807 wait complete. 2024-09-12T02:54:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:54 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3467532807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:54 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3467532807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:54.646 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3467532807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:54:54.675 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id ea1d6443 --release --await' 2024-09-12T02:54:54.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id ea1d6443 --release --await 2024-09-12T02:54:54.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 662222 10.14 MB/sec execute 236 sec latency 15640.210 ms 2024-09-12T02:54:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:55.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:55 smithi016 ceph-mon[30925]: pgmap v547: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-09-12T02:54:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 -- 172.21.15.16:0/683675463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc103660 msgr2=0x7f74fc10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 --2- 172.21.15.16:0/683675463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc103660 0x7f74fc10bb50 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009900 tx=0x7f74e402a480 comp rx=0 tx=0).stop 2024-09-12T02:54:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 -- 172.21.15.16:0/683675463 shutdown_connections 2024-09-12T02:54:55.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 --2- 172.21.15.16:0/683675463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc10c170 0x7f74fc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 --2- 172.21.15.16:0/683675463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc103660 0x7f74fc10bb50 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 --2- 172.21.15.16:0/683675463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 -- 172.21.15.16:0/683675463 >> 172.21.15.16:0/683675463 conn(0x7f74fc0fc7a0 msgr2=0x7f74fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 -- 172.21.15.16:0/683675463 shutdown_connections 2024-09-12T02:54:55.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.430+0000 7f7500d02640 1 -- 172.21.15.16:0/683675463 wait complete. 2024-09-12T02:54:55.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.431+0000 7f7500d02640 1 Processor -- start 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.431+0000 7f7500d02640 1 -- start start 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.431+0000 7f7500d02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.431+0000 7f7500d02640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc103660 0x7f74fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.431+0000 7f7500d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f7500d02640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc111f70 con 0x7f74fc10c170 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f7500d02640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc111df0 con 0x7f74fc103660 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f7500d02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1120f0 con 0x7f74fc102d40 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35746/0 (socket says 172.21.15.16:35746) 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 -- 172.21.15.16:0/494216631 learned_addr learned my addr 172.21.15.16:0/494216631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc103660 0x7f74fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fa59c640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 -- 172.21.15.16:0/494216631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 msgr2=0x7f74fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 -- 172.21.15.16:0/494216631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc103660 msgr2=0x7f74fc198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc103660 0x7f74fc198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 -- 172.21.15.16:0/494216631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74fc19d3c0 con 0x7f74fc10c170 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fa59c640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74fad9d640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f74f400eaa0 tx=0x7f74f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74f3fff640 1 -- 172.21.15.16:0/494216631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f400cdb0 con 0x7f74fc10c170 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74fc19d6b0 con 0x7f74fc10c170 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74f3fff640 1 -- 172.21.15.16:0/494216631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4005020 con 0x7f74fc10c170 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.432+0000 7f74f3fff640 1 -- 172.21.15.16:0/494216631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f40116b0 con 0x7f74fc10c170 2024-09-12T02:54:55.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.433+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74fc1a4f90 con 0x7f74fc10c170 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.435+0000 7f74f3fff640 1 -- 172.21.15.16:0/494216631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f74f4005560 con 0x7f74fc10c170 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.435+0000 7f75030db740 1 Processor -- start 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.435+0000 7f75030db740 1 -- start start 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.435+0000 7f75030db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f060 0x56552c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f75030db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 0x56552c159d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f75030db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f75030db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56552c0b4620 con 0x56552c15a240 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f75030db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56552c05be50 con 0x56552c14f060 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f75030db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56552c0b05e0 con 0x56552c14f980 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35750/0 (socket says 172.21.15.16:35750) 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2520881698 learned_addr learned my addr 172.21.15.16:0/2520881698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:55.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f3fff640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 0x7f74e80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f060 0x56552c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74fad9d640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 0x7f74e80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 0x56552c159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2520881698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 msgr2=0x56552c159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 0x56552c159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2520881698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f060 msgr2=0x56552c14f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f060 0x56552c14f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2520881698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56552c04cd40 con 0x56552c15a240 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f3fff640 1 -- 172.21.15.16:0/494216631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74f40cf2a0 con 0x7f74fc10c170 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 0x56552c159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009900 tx=0x7f74e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea1b8ec9c6329c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.436+0000 7f74fad9d640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 0x7f74e80a5040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74f4015800 tx=0x7f74f406dac0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f74f0d99640 1 -- 172.21.15.16:0/2520881698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035070 con 0x56552c15a240 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f74f0d99640 1 -- 172.21.15.16:0/2520881698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e4004030 con 0x56552c15a240 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f74f0d99640 1 -- 172.21.15.16:0/2520881698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4005620 con 0x56552c15a240 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 -- 172.21.15.16:0/2520881698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 msgr2=0x56552c15c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009900 tx=0x7f74e4000bc0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 -- 172.21.15.16:0/2520881698 shutdown_connections 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c15c630 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f980 0x56552c159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 --2- 172.21.15.16:0/2520881698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f060 0x56552c14f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 -- 172.21.15.16:0/2520881698 >> 172.21.15.16:0/2520881698 conn(0x56552c14b5f0 msgr2=0x56552c14b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 -- 172.21.15.16:0/2520881698 shutdown_connections 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.437+0000 7f75030db740 1 -- 172.21.15.16:0/2520881698 wait complete. 2024-09-12T02:54:55.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 -- start start 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 Processor -- start 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c157130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c157670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56552c15ce10 con 0x56552c15a240 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56552c03cf10 con 0x56552c14f980 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56552c15cf90 con 0x56552c14f060 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f75030db740 10 client.0 _ll_register_callbacks cb 0x56552c060db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35762/0 (socket says 172.21.15.16:35762) 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f74f1d9b640 1 -- 172.21.15.16:0/164925098 learned_addr learned my addr 172.21.15.16:0/164925098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f74fa59c640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.438+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 -- 172.21.15.16:0/164925098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 msgr2=0x56552c157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 -- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 msgr2=0x56552c157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 -- 172.21.15.16:0/164925098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56552bdc3580 con 0x56552c15a240 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f74e4032d60 tx=0x7f74e40052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c34280f7794339f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035040 con 0x56552c15a240 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74e4031830 con 0x56552c15a240 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74e4004030 con 0x56552c15a240 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f74e4031a30 con 0x56552c15a240 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f75030db740 10 client.17067 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:55.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56552be8d960 con 0x56552c15a240 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.439+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e4004030 con 0x56552c15a240 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f74e4003780 con 0x56552c15a240 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 10 client.17067 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 10 client.17067 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 20 client.17067 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c169bb0 0x56552c16bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 4 client.17067 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56552c0c1b90 con 0x56552c169bb0 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f74fa59c640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c169bb0 0x56552c16bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f74fa59c640 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c169bb0 0x56552c16bfa0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0e423c520438cb4 server_cookie=8ee1d27edc03060 in_seq=0 out_seq=0 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.440+0000 7f74e1ffb640 10 client.17067 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f74e1ffb640 1 -- 172.21.15.16:0/164925098 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56552c0c1b90 con 0x56552c169bb0 2024-09-12T02:54:55.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f74e1ffb640 10 client.17067 handle_command_reply: tid=0 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 client.17067 shutdown 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 msgr2=0x56552c157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f74e4032d60 tx=0x7f74e40052e0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 shutdown_connections 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c169bb0 0x56552c16bfa0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c15a240 0x56552c157bb0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 --2- 172.21.15.16:0/164925098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f74e1ffb640 0 client.17067 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 >> 172.21.15.16:0/164925098 conn(0x56552c14b5f0 msgr2=0x56552c1512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 shutdown_connections 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 1 -- 172.21.15.16:0/164925098 wait complete. 2024-09-12T02:54:55.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.441+0000 7f75030db740 20 client.17067 trim_cache size 0 max 16384 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 Processor -- start 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 -- start start 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c1543b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c15a240 0x56552c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1ad480 con 0x56552c14f060 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1aefc0 con 0x56552c14f980 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f75030db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1adb50 con 0x56552c15a240 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f74f1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35778/0 (socket says 172.21.15.16:35778) 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.459+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2649700453 learned_addr learned my addr 172.21.15.16:0/2649700453 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c1543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c15a240 0x56552c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2649700453 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c15a240 msgr2=0x56552c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c15a240 0x56552c1548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2649700453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 msgr2=0x56552c1543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c1543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 -- 172.21.15.16:0/2649700453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56552c150200 con 0x56552c14f060 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f74e4032c00 tx=0x7f74e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ecf885e7b03200 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74e3fff640 1 -- 172.21.15.16:0/2649700453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e405f030 con 0x56552c14f060 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74e3fff640 1 -- 172.21.15.16:0/2649700453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e4005850 con 0x56552c14f060 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.460+0000 7f74e3fff640 1 -- 172.21.15.16:0/2649700453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035030 con 0x56552c14f060 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- 172.21.15.16:0/2649700453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 msgr2=0x56552c14a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f74e4032c00 tx=0x7f74e4000bc0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- 172.21.15.16:0/2649700453 shutdown_connections 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c15a240 0x56552c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c14f980 0x56552c1543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 --2- 172.21.15.16:0/2649700453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f060 0x56552c14a640 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- 172.21.15.16:0/2649700453 >> 172.21.15.16:0/2649700453 conn(0x56552c14b5f0 msgr2=0x56552c149f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- 172.21.15.16:0/2649700453 shutdown_connections 2024-09-12T02:54:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- 172.21.15.16:0/2649700453 wait complete. 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 -- start start 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.461+0000 7f75030db740 1 Processor -- start 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c1cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c15a240 0x56552c16d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1ae8f0 con 0x56552c14f980 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1ae220 con 0x56552c15a240 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1ab460 con 0x56552c14f060 2024-09-12T02:54:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 10 client.0 _ll_register_callbacks cb 0x7f74e4032c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35786/0 (socket says 172.21.15.16:35786) 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 -- 172.21.15.16:0/2572109787 learned_addr learned my addr 172.21.15.16:0/2572109787 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c1cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 -- 172.21.15.16:0/2572109787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 msgr2=0x56552c1cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c1cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 -- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c15a240 msgr2=0x56552c16d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c15a240 0x56552c16d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 -- 172.21.15.16:0/2572109787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56552c16dfb0 con 0x56552c14f980 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c1cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74fa59c640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x56552c1569c0 tx=0x7f74ec011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351eacee0d5c8095 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec01a760 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74ec01a900 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.462+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74e4005850 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f74ec01ab00 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 10 client.17073 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56552c16fa70 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e4005850 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f74ec02d520 con 0x56552c14f980 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 10 client.17073 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 10 client.17073 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 20 client.17073 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c1cc880 0x56552c1ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 4 client.17073 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ea1d6443", "release": true, "await": true}] 2024-09-12T02:54:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ea1d6443", "release": true, "await": true}) -- 0x56552c0bc0a0 con 0x56552c1cc880 2024-09-12T02:54:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.463+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c1cc880 0x56552c1ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.464+0000 7f74f1d9b640 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c1cc880 0x56552c1ccc90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a05ac6c6d083db server_cookie=1c20d3aeedd01c17 in_seq=0 out_seq=0 2024-09-12T02:54:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.464+0000 7f74e1ffb640 10 client.17073 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f74e1ffb640 1 -- 172.21.15.16:0/2572109787 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56552c0bc0a0 con 0x56552c1cc880 2024-09-12T02:54:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f74e1ffb640 10 client.17073 handle_command_reply: tid=0 2024-09-12T02:54:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 client.17073 shutdown 2024-09-12T02:54:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 msgr2=0x56552c1cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x56552c1569c0 tx=0x7f74ec011b00 comp rx=0 tx=0).stop 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 shutdown_connections 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56552c1cc880 0x56552c1ccc90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56552c15a240 0x56552c16d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552c14f980 0x56552c1cfdf0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 --2- 172.21.15.16:0/2572109787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56552c14f060 0x56552c1cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 >> 172.21.15.16:0/2572109787 conn(0x56552c14b5f0 msgr2=0x56552c157910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f74e1ffb640 0 client.17073 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 shutdown_connections 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 1 -- 172.21.15.16:0/2572109787 wait complete. 2024-09-12T02:54:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.473+0000 7f75030db740 20 client.17073 trim_cache size 0 max 16384 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 28, 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "ea1d6443": { 2024-09-12T02:54:55.476 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 28, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 136.6, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:54:55.477 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:54:55.478 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 msgr2=0x7f74e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 0x7f74e80a5040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74f4015800 tx=0x7f74f406dac0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 msgr2=0x7f74fc19cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f74f400eaa0 tx=0x7f74f400ef60 comp rx=0 tx=0).stop 2024-09-12T02:54:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 shutdown_connections 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74e80a2b80 0x7f74e80a5040 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74fc10c170 0x7f74fc19cce0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74fc103660 0x7f74fc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 --2- 172.21.15.16:0/494216631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74fc102d40 0x7f74fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 >> 172.21.15.16:0/494216631 conn(0x7f74fc0fc7a0 msgr2=0x7f74fc0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 shutdown_connections 2024-09-12T02:54:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:55.474+0000 7f7500d02640 1 -- 172.21.15.16:0/494216631 wait complete. 2024-09-12T02:54:55.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:55 smithi033 ceph-mon[35092]: pgmap v547: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-09-12T02:54:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:55 smithi184 ceph-mon[35321]: pgmap v547: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-09-12T02:54:55.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 663133 10.11 MB/sec execute 237 sec latency 16392.126 ms 2024-09-12T02:54:56.143 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ea1d6443', total seconds elapsed: 19.7 2024-09-12T02:54:56.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 665658 10.13 MB/sec execute 238 sec latency 9.918 ms 2024-09-12T02:54:57.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:54:57.493 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:57.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:57 smithi033 ceph-mon[35092]: pgmap v548: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:54:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:57 smithi184 ceph-mon[35321]: pgmap v548: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:54:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:57 smithi016 ceph-mon[30925]: pgmap v548: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.5 MiB/s wr, 94 op/s 2024-09-12T02:54:57.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 668218 10.15 MB/sec execute 239 sec latency 54.840 ms 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 -- 172.21.15.16:0/3568963929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 msgr2=0x7fc67c10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 --2- 172.21.15.16:0/3568963929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c10e600 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc674009ca0 tx=0x7fc67402a950 comp rx=0 tx=0).stop 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 -- 172.21.15.16:0/3568963929 shutdown_connections 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 --2- 172.21.15.16:0/3568963929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c10e600 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 --2- 172.21.15.16:0/3568963929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c103650 0x7fc67c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 --2- 172.21.15.16:0/3568963929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c102d30 0x7fc67c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.978+0000 7fc684137640 1 -- 172.21.15.16:0/3568963929 >> 172.21.15.16:0/3568963929 conn(0x7fc67c0fc7d0 msgr2=0x7fc67c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.979+0000 7fc684137640 1 -- 172.21.15.16:0/3568963929 shutdown_connections 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.979+0000 7fc684137640 1 -- 172.21.15.16:0/3568963929 wait complete. 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.979+0000 7fc684137640 1 Processor -- start 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.979+0000 7fc684137640 1 -- start start 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 0x7fc67c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c111f60 con 0x7fc67c10c160 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c111de0 con 0x7fc67c102d30 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc684137640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1120e0 con 0x7fc67c103650 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc6826d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 0x7fc67c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39722/0 (socket says 172.21.15.16:39722) 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 -- 172.21.15.16:0/2593474617 learned_addr learned my addr 172.21.15.16:0/2593474617 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 -- 172.21.15.16:0/2593474617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 msgr2=0x7fc67c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 0x7fc67c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 -- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 msgr2=0x7fc67c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc681ed3640 1 -- 172.21.15.16:0/2593474617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67c19d390 con 0x7fc67c102d30 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 0x7fc67c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.980+0000 7fc6826d4640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc67800e9f0 tx=0x7fc67800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc672ffd640 1 -- 172.21.15.16:0/2593474617 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67800cd80 con 0x7fc67c102d30 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc672ffd640 1 -- 172.21.15.16:0/2593474617 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc678005500 con 0x7fc67c102d30 2024-09-12T02:54:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67c19d680 con 0x7fc67c102d30 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc672ffd640 1 -- 172.21.15.16:0/2593474617 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678010690 con 0x7fc67c102d30 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.981+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67c1a4f60 con 0x7fc67c102d30 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.984+0000 7fc672ffd640 1 -- 172.21.15.16:0/2593474617 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc6780046c0 con 0x7fc67c102d30 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.984+0000 7fc672ffd640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 0x7fc6680a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.984+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 0x7fc6680a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.984+0000 7fc672ffd640 1 -- 172.21.15.16:0/2593474617 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc6780cf3b0 con 0x7fc67c102d30 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 Processor -- start 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 0x7fc6680a50c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0fed60 tx=0x7fc66c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 -- start start 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b79140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b79680 0x563727b7ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563727ad42e0 con 0x563727b79680 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563727a7bce0 con 0x563727b6edc0 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.985+0000 7fc686510740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563727ad02a0 con 0x563727b6e4a0 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b79140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc6826d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b79680 0x563727b7ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59584/0 (socket says 172.21.15.16:59584) 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 -- 172.21.15.16:0/11685662 learned_addr learned my addr 172.21.15.16:0/11685662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 -- 172.21.15.16:0/11685662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 msgr2=0x563727b79140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b79140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 -- 172.21.15.16:0/11685662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b79680 msgr2=0x563727b7ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:57.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b79680 0x563727b7ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 -- 172.21.15.16:0/11685662 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563727a60440 con 0x563727b6e4a0 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc6816d2640 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b79140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc681ed3640 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc678098080 tx=0x7fc67809b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bdfa19efe5b8064 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc667fff640 1 -- 172.21.15.16:0/11685662 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67809ffd0 con 0x563727b6e4a0 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc667fff640 1 -- 172.21.15.16:0/11685662 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6780a4890 con 0x563727b6e4a0 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.986+0000 7fc667fff640 1 -- 172.21.15.16:0/11685662 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67809b070 con 0x563727b6e4a0 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 -- 172.21.15.16:0/11685662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 msgr2=0x563727b6e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc678098080 tx=0x7fc67809b580 comp rx=0 tx=0).stop 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 -- 172.21.15.16:0/11685662 shutdown_connections 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b79680 0x563727b7ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b79140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 --2- 172.21.15.16:0/11685662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b6e880 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 -- 172.21.15.16:0/11685662 >> 172.21.15.16:0/11685662 conn(0x563727b6aa30 msgr2=0x563727b6ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 -- 172.21.15.16:0/11685662 shutdown_connections 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.987+0000 7fc686510740 1 -- 172.21.15.16:0/11685662 wait complete. 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 -- start start 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 Processor -- start 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b76200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727b76740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563727b7c250 con 0x563727b6e4a0 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563727a8f050 con 0x563727b6edc0 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563727b7c3d0 con 0x563727b79680 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc686510740 10 client.0 _ll_register_callbacks cb 0x563727acf160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc6826d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727b76740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35818/0 (socket says 172.21.15.16:35818) 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc681ed3640 1 -- 172.21.15.16:0/2852513550 learned_addr learned my addr 172.21.15.16:0/2852513550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.988+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b76200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 -- 172.21.15.16:0/2852513550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 msgr2=0x563727b76740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727b76740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 -- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 msgr2=0x563727b76200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b76200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 -- 172.21.15.16:0/2852513550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637278453c0 con 0x563727b6e4a0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc6780a4360 tx=0x7fc678005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84aec307c88d7ee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc6657fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6780a9070 con 0x563727b6e4a0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc678099f20 con 0x563727b6e4a0 2024-09-12T02:54:58.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.989+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6780a4890 con 0x563727b6e4a0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc678098290 con 0x563727b6e4a0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 10 client.17085 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563727b87ba0 con 0x563727b6e4a0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6780a4890 con 0x563727b6e4a0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc678097970 con 0x563727b6e4a0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 10 client.17085 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 10 client.17085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 20 client.17085 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727b88ef0 0x563727b8b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 4 client.17085 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563727ae1850 con 0x563727b88ef0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.990+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727b88ef0 0x563727b8b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.991+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727b88ef0 0x563727b8b2e0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77d5fd402474f950 server_cookie=9dc2d56ccd8ae5e0 in_seq=0 out_seq=0 2024-09-12T02:54:58.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.991+0000 7fc6657fa640 10 client.17085 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.991+0000 7fc6657fa640 1 -- 172.21.15.16:0/2852513550 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563727ae1850 con 0x563727b88ef0 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.991+0000 7fc6657fa640 10 client.17085 handle_command_reply: tid=0 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.991+0000 7fc686510740 1 client.17085 shutdown 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 msgr2=0x563727b78f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc6780a4360 tx=0x7fc678005760 comp rx=0 tx=0).stop 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 shutdown_connections 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727b88ef0 0x563727b8b2e0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727b76740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727b76200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc6657fa640 0 client.17085 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 --2- 172.21.15.16:0/2852513550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727b78f50 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 >> 172.21.15.16:0/2852513550 conn(0x563727b6aa30 msgr2=0x563727b69b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 shutdown_connections 2024-09-12T02:54:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 1 -- 172.21.15.16:0/2852513550 wait complete. 2024-09-12T02:54:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:57.992+0000 7fc686510740 20 client.17085 trim_cache size 0 max 16384 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 Processor -- start 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 -- start start 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b69e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 0x563727b8e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ad5b0 con 0x563727b6edc0 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1af0f0 con 0x563727b79680 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc686510740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1adc80 con 0x563727b6e4a0 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 0x563727b8e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35826/0 (socket says 172.21.15.16:35826) 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6816d2640 1 -- 172.21.15.16:0/2031915011 learned_addr learned my addr 172.21.15.16:0/2031915011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b69e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 -- 172.21.15.16:0/2031915011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 msgr2=0x563727b69e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b69e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 -- 172.21.15.16:0/2031915011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 msgr2=0x563727b8e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6826d4640 1 -- 172.21.15.16:0/2031915011 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563727b79280 con 0x563727b79680 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.010+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:58.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.011+0000 7fc6826d4640 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 0x563727b8e850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc674005cc0 tx=0x7fc6740098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14461285b7bc432f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.011+0000 7fc6677fe640 1 -- 172.21.15.16:0/2031915011 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674034040 con 0x563727b79680 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.011+0000 7fc6677fe640 1 -- 172.21.15.16:0/2031915011 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc67404a070 con 0x563727b79680 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.011+0000 7fc6677fe640 1 -- 172.21.15.16:0/2031915011 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674004780 con 0x563727b79680 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- 172.21.15.16:0/2031915011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 msgr2=0x563727b8e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 0x563727b8e850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc674005cc0 tx=0x7fc6740098d0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- 172.21.15.16:0/2031915011 shutdown_connections 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b79680 0x563727b8e850 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6edc0 0x563727b8e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 --2- 172.21.15.16:0/2031915011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b6e4a0 0x563727b69e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- 172.21.15.16:0/2031915011 >> 172.21.15.16:0/2031915011 conn(0x563727b6aa30 msgr2=0x563727b78280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- 172.21.15.16:0/2031915011 shutdown_connections 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- 172.21.15.16:0/2031915011 wait complete. 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.012+0000 7fc686510740 1 -- start start 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 Processor -- start 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727be77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727be8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1aea20 con 0x563727b6e4a0 2024-09-12T02:54:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ae350 con 0x563727b6edc0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ab590 con 0x563727b79680 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc681ed3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727be77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc686510740 10 client.0 _ll_register_callbacks cb 0x7fc67809cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc6816d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39760/0 (socket says 172.21.15.16:39760) 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc6816d2640 1 -- 172.21.15.16:0/2872080660 learned_addr learned my addr 172.21.15.16:0/2872080660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.013+0000 7fc6826d4640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727be8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 -- 172.21.15.16:0/2872080660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 msgr2=0x563727be8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727be8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 -- 172.21.15.16:0/2872080660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 msgr2=0x563727be77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727be77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 -- 172.21.15.16:0/2872080660 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563727bf2890 con 0x563727b6edc0 2024-09-12T02:54:58.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6816d2640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc66c003760 tx=0x7fc66c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2081082fec7d50de server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6657fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c01e660 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc66c01e800 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc67404a070 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc66c01ea00 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc686510740 10 client.26612 fetch_fsmap learned FSMap version 32 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.014+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563727bf3c30 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc67404a070 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc66c01a040 con 0x563727b6edc0 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 10 client.26612 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 10 client.26612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 20 client.26612 populate_metadata read hostname 'smithi016' 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727bf43c0 0x563727bf67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 4 client.26612 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:54:58.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563727adbd60 con 0x563727bf43c0 2024-09-12T02:54:58.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727bf43c0 0x563727bf67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc6657fa640 10 client.26612 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.015+0000 7fc681ed3640 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727bf43c0 0x563727bf67e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c60766d287c4cbf7 server_cookie=dc2e840173025f46 in_seq=0 out_seq=0 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc6657fa640 1 -- 172.21.15.16:0/2872080660 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563727adbd60 con 0x563727bf43c0 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc6657fa640 10 client.26612 handle_command_reply: tid=0 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 client.26612 shutdown 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 msgr2=0x563727be7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc66c003760 tx=0x7fc66c004730 comp rx=0 tx=0).stop 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 shutdown_connections 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563727bf43c0 0x563727bf67e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563727b79680 0x563727be8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563727b6edc0 0x563727be7d00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 --2- 172.21.15.16:0/2872080660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563727b6e4a0 0x563727be77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 >> 172.21.15.16:0/2872080660 conn(0x563727b6aa30 msgr2=0x563727b74750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc6657fa640 0 client.26612 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 shutdown_connections 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 1 -- 172.21.15.16:0/2872080660 wait complete. 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.016+0000 7fc686510740 20 client.26612 trim_cache size 0 max 16384 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:54:58.021 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 msgr2=0x7fc6680a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 0x7fc6680a50c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0fed60 tx=0x7fc66c005f90 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 msgr2=0x7fc67c1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc67800e9f0 tx=0x7fc67800edf0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 shutdown_connections 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc6680a2c00 0x7fc6680a50c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc67c10c160 0x7fc67c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc67c103650 0x7fc67c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 --2- 172.21.15.16:0/2593474617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc67c102d30 0x7fc67c1983e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 >> 172.21.15.16:0/2593474617 conn(0x7fc67c0fc7d0 msgr2=0x7fc67c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 shutdown_connections 2024-09-12T02:54:58.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:58.017+0000 7fc684137640 1 -- 172.21.15.16:0/2593474617 wait complete. 2024-09-12T02:54:58.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 671165 10.18 MB/sec execute 240 sec latency 41.162 ms 2024-09-12T02:54:58.764 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:54:58.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:54:59.045 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:54:59.488 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:54:59 smithi016 ceph-mon[30925]: pgmap v549: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:54:59.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:54:59 smithi033 ceph-mon[35092]: pgmap v549: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:54:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:54:59 smithi184 ceph-mon[35321]: pgmap v549: 97 pgs: 97 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 2.2 MiB/s wr, 88 op/s 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 -- 172.21.15.16:0/3290069402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c102d30 msgr2=0x7fd00c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 --2- 172.21.15.16:0/3290069402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c102d30 0x7fd00c103110 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009930 tx=0x7fcffc02a420 comp rx=0 tx=0).stop 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 -- 172.21.15.16:0/3290069402 shutdown_connections 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 --2- 172.21.15.16:0/3290069402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 --2- 172.21.15.16:0/3290069402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c1036e0 0x7fd00c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 --2- 172.21.15.16:0/3290069402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c102d30 0x7fd00c103110 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.574+0000 7fd01190d640 1 -- 172.21.15.16:0/3290069402 >> 172.21.15.16:0/3290069402 conn(0x7fd00c0fc7d0 msgr2=0x7fd00c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.575+0000 7fd01190d640 1 -- 172.21.15.16:0/3290069402 shutdown_connections 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.575+0000 7fd01190d640 1 -- 172.21.15.16:0/3290069402 wait complete. 2024-09-12T02:54:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.575+0000 7fd01190d640 1 Processor -- start 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.575+0000 7fd01190d640 1 -- start start 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102d30 0x7fd00c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1120a0 con 0x7fd00c1036e0 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c111f20 con 0x7fd00c102d30 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd01190d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c112220 con 0x7fd00c1082a0 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35838/0 (socket says 172.21.15.16:35838) 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 -- 172.21.15.16:0/446999719 learned_addr learned my addr 172.21.15.16:0/446999719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00affd640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102d30 0x7fd00c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 -- 172.21.15.16:0/446999719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 msgr2=0x7fd00c19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 -- 172.21.15.16:0/446999719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102d30 msgr2=0x7fd00c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102d30 0x7fd00c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 -- 172.21.15.16:0/446999719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c19d570 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00b7fe640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.576+0000 7fd00a7fc640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd00000d740 tx=0x7fd00000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.577+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000017070 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.577+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd00000de50 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.577+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c19d860 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.577+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00000f6b0 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.577+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c1a5140 con 0x7fd00c1036e0 2024-09-12T02:54:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.578+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c06a850 con 0x7fd00c1036e0 2024-09-12T02:54:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.579+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd00001a070 con 0x7fd00c1036e0 2024-09-12T02:54:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.580+0000 7fd01090b640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 0x7fcff40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:54:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.580+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd0000ce9e0 con 0x7fd00c1036e0 2024-09-12T02:54:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.580+0000 7fd00affd640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 0x7fcff40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:54:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.581+0000 7fd00affd640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 0x7fcff40a5040 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009900 tx=0x7fcffc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:54:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.586+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd00009b2a0 con 0x7fd00c1036e0 2024-09-12T02:54:59.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 673648 10.17 MB/sec execute 241 sec latency 19.463 ms 2024-09-12T02:54:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.820+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd00c10a2b0 con 0x7fd00c1036e0 2024-09-12T02:54:59.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.821+0000 7fd01090b640 1 -- 172.21.15.16:0/446999719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fd00009f5a0 con 0x7fd00c1036e0 2024-09-12T02:54:59.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:54:59.821 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:54:59.822 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 msgr2=0x7fcff40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 0x7fcff40a5040 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009900 tx=0x7fcffc0058d0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 msgr2=0x7fd00c198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd00000d740 tx=0x7fd00000dc00 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 shutdown_connections 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd00c1082a0 0x7fd00c19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcff40a2b80 0x7fcff40a5040 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd00c1036e0 0x7fd00c198b00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 --2- 172.21.15.16:0/446999719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00c102d30 0x7fd00c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 >> 172.21.15.16:0/446999719 conn(0x7fd00c0fc7d0 msgr2=0x7fd00c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:54:59.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 shutdown_connections 2024-09-12T02:54:59.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:54:59.823+0000 7fd01190d640 1 -- 172.21.15.16:0/446999719 wait complete. 2024-09-12T02:55:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:00 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/446999719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:00 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:00 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/446999719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:00 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:00 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/446999719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:00 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:55:00.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:55:00.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 675703 10.18 MB/sec execute 242 sec latency 16.327 ms 2024-09-12T02:55:00.879 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:01.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: pgmap v550: 97 pgs: 97 active+clean; 243 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 115 op/s 2024-09-12T02:55:01.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:55:01.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2455453406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 msgr2=0x7f359c10b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2455453406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c10b9a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3584009ca0 tx=0x7f358402a950 comp rx=0 tx=0).stop 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2455453406 shutdown_connections 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2455453406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 0x7f359c10e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2455453406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c10b9a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2455453406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 0x7f359c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.394+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2455453406 >> 172.21.15.16:0/2455453406 conn(0x7f359c0fc610 msgr2=0x7f359c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2455453406 shutdown_connections 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2455453406 wait complete. 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 Processor -- start 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 -- start start 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 0x7f359c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 0x7f359c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.395+0000 7f35a1ba1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f359c111e10 con 0x7f359c1034b0 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f35a1ba1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f359c111c90 con 0x7f359c10bfc0 2024-09-12T02:55:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f35a1ba1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f359c111f90 con 0x7f359c102b90 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35858/0 (socket says 172.21.15.16:35858) 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 -- 172.21.15.16:0/2103569499 learned_addr learned my addr 172.21.15.16:0/2103569499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 0x7f359c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f35a13a0640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 0x7f359c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 -- 172.21.15.16:0/2103569499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 msgr2=0x7f359c198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 0x7f359c198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 -- 172.21.15.16:0/2103569499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 msgr2=0x7f359c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 0x7f359c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 -- 172.21.15.16:0/2103569499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c19d2e0 con 0x7f359c1034b0 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f359bfff640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3584005e30 tx=0x7f3584003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f3599ffb640 1 -- 172.21.15.16:0/2103569499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584034040 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359c19d570 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f3599ffb640 1 -- 172.21.15.16:0/2103569499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358402fe20 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.396+0000 7f3599ffb640 1 -- 172.21.15.16:0/2103569499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35840047c0 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.397+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f359c1a4eb0 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f3599ffb640 1 -- 172.21.15.16:0/2103569499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3584037020 con 0x7f359c1034b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 Processor -- start 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 -- start start 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 0x56361c3d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56361c32a2f0 con 0x56361c3cf550 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56361c2d1cf0 con 0x56361c3c47b0 2024-09-12T02:55:01.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a51dc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56361c3262b0 con 0x56361c3c5d20 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39800/0 (socket says 172.21.15.16:39800) 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59676/0 (socket says 172.21.15.16:59676) 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 -- 172.21.15.16:0/2037804130 learned_addr learned my addr 172.21.15.16:0/2037804130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f359bfff640 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 0x56361c3d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 -- 172.21.15.16:0/2037804130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 msgr2=0x56361c3c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 -- 172.21.15.16:0/2037804130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 msgr2=0x56361c3d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.400+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 0x56361c3d1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f35a0b9f640 1 -- 172.21.15.16:0/2037804130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56361c2b63a0 con 0x56361c3c47b0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f359bfff640 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 0x56361c3d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f3599ffb640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 0x7f35880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f359400b760 tx=0x7f359400bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d38a198ed463c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f35a13a0640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 0x7f35880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f3599ffb640 1 -- 172.21.15.16:0/2103569499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f35840eff10 con 0x7f359c1034b0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f3592ffd640 1 -- 172.21.15.16:0/2037804130 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594003dd0 con 0x56361c3c47b0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f3592ffd640 1 -- 172.21.15.16:0/2037804130 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3594015070 con 0x56361c3c47b0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.401+0000 7f3592ffd640 1 -- 172.21.15.16:0/2037804130 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594011440 con 0x56361c3c47b0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- 172.21.15.16:0/2037804130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 msgr2=0x56361c3c4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f359400b760 tx=0x7f359400bc20 comp rx=0 tx=0).stop 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a13a0640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 0x7f35880a5040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f358c009c70 tx=0x7f358c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- 172.21.15.16:0/2037804130 shutdown_connections 2024-09-12T02:55:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3cf550 0x56361c3d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3c4bf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2037804130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- 172.21.15.16:0/2037804130 >> 172.21.15.16:0/2037804130 conn(0x56361c38d310 msgr2=0x56361c3c0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- 172.21.15.16:0/2037804130 shutdown_connections 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- 172.21.15.16:0/2037804130 wait complete. 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 -- start start 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.402+0000 7f35a51dc740 1 Processor -- start 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56361c3d2250 con 0x56361c3c47b0 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56361c2e5060 con 0x56361c3cf550 2024-09-12T02:55:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56361c3d23d0 con 0x56361c3c5d20 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a51dc740 10 client.0 _ll_register_callbacks cb 0x56361c0388c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35884/0 (socket says 172.21.15.16:35884) 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a0b9f640 1 -- 172.21.15.16:0/1471858332 learned_addr learned my addr 172.21.15.16:0/1471858332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39812/0 (socket says 172.21.15.16:39812) 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f359bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 -- 172.21.15.16:0/1471858332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 msgr2=0x56361c3cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 -- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 msgr2=0x56361c3ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a13a0640 1 -- 172.21.15.16:0/1471858332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56361c09b3c0 con 0x56361c3cf550 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f359bfff640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.403+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f358c020520 tx=0x7f358c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b1243d9cc019f71 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c02d060 con 0x56361c3cf550 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3594015070 con 0x56361c3cf550 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f358c018030 con 0x56361c3cf550 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f358c011710 con 0x56361c3cf550 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 10 client.26630 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56361c3ddd50 con 0x56361c3cf550 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3594015070 con 0x56361c3cf550 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f358c028b40 con 0x56361c3cf550 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 10 client.26630 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 10 client.26630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 20 client.26630 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c3df160 0x56361c3e1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 4 client.26630 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.404+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56361c337860 con 0x56361c3df160 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.405+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c3df160 0x56361c3e1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.405+0000 7f357bfff640 10 client.26630 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.405+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c3df160 0x56361c3e1570 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3752f5a9f1b1bd30 server_cookie=ef5b7bd400d30ef0 in_seq=0 out_seq=0 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f357bfff640 1 -- 172.21.15.16:0/1471858332 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56361c337860 con 0x56361c3df160 2024-09-12T02:55:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f357bfff640 10 client.26630 handle_command_reply: tid=0 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 client.26630 shutdown 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 msgr2=0x56361c3cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f358c020520 tx=0x7f358c004730 comp rx=0 tx=0).stop 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 shutdown_connections 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c3df160 0x56361c3e1570 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3cc740 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c5d20 0x56361c3cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1471858332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c47b0 0x56361c3ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f357bfff640 0 client.26630 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 >> 172.21.15.16:0/1471858332 conn(0x56361c38d310 msgr2=0x56361c3c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 shutdown_connections 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 1 -- 172.21.15.16:0/1471858332 wait complete. 2024-09-12T02:55:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.406+0000 7f35a51dc740 20 client.26630 trim_cache size 0 max 16384 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 Processor -- start 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 -- start start 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 0x56361c3c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3db4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1ad350 con 0x56361c3c5d20 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1aee90 con 0x56361c3cf550 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a51dc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1ada20 con 0x56361c3c47b0 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3db4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 0x56361c3c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3db4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35888/0 (socket says 172.21.15.16:35888) 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39822/0 (socket says 172.21.15.16:39822) 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 -- 172.21.15.16:0/1904961730 learned_addr learned my addr 172.21.15.16:0/1904961730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 0x56361c3c0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59704/0 (socket says 172.21.15.16:59704) 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 -- 172.21.15.16:0/1904961730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 msgr2=0x56361c3c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 0x56361c3c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 -- 172.21.15.16:0/1904961730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 msgr2=0x56361c3db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3db4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.422+0000 7f35a13a0640 1 -- 172.21.15.16:0/1904961730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56361c3e1bf0 con 0x56361c3cf550 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35a13a0640 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f358c028ee0 tx=0x7f358c01d6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23256f3250b961cd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35927fc640 1 -- 172.21.15.16:0/1904961730 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c038030 con 0x56361c3cf550 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35927fc640 1 -- 172.21.15.16:0/1904961730 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358c004510 con 0x56361c3cf550 2024-09-12T02:55:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35927fc640 1 -- 172.21.15.16:0/1904961730 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c01d890 con 0x56361c3cf550 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35a51dc740 1 -- 172.21.15.16:0/1904961730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 msgr2=0x56361c3e6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.423+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f358c028ee0 tx=0x7f358c01d6f0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- 172.21.15.16:0/1904961730 shutdown_connections 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3cf550 0x56361c3e6e40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3db4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- 172.21.15.16:0/1904961730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3c47b0 0x56361c3c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- 172.21.15.16:0/1904961730 >> 172.21.15.16:0/1904961730 conn(0x56361c38d310 msgr2=0x56361c3c08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- 172.21.15.16:0/1904961730 shutdown_connections 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- 172.21.15.16:0/1904961730 wait complete. 2024-09-12T02:55:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- start start 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 Processor -- start 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 0x56361c3c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1ae7c0 con 0x56361c3c5d20 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1ae0f0 con 0x56361c3c47b0 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f359c1ab330 con 0x56361c3cf550 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.424+0000 7f35a51dc740 10 client.0 _ll_register_callbacks cb 0x7f358c0098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a13a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 0x56361c3c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35894/0 (socket says 172.21.15.16:35894) 2024-09-12T02:55:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 -- 172.21.15.16:0/2596019639 learned_addr learned my addr 172.21.15.16:0/2596019639 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a0b9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39824/0 (socket says 172.21.15.16:39824) 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 -- 172.21.15.16:0/2596019639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 msgr2=0x56361c3c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 0x56361c3c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 -- 172.21.15.16:0/2596019639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 msgr2=0x56361c3d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 -- 172.21.15.16:0/2596019639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56361c3c8e40 con 0x56361c3c5d20 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a13a0640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 0x56361c3c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f359bfff640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3584000f80 tx=0x7f358402ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72677fb3884581fb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f357bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:01.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35840d5060 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f35840c4f60 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f358c004510 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f358402c040 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.425+0000 7f35a51dc740 10 client.17115 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56361c4497d0 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358c004510 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f35840b8040 con 0x56361c3c5d20 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 10 client.17115 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 10 client.17115 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 20 client.17115 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c449ed0 0x56361c44c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 4 client.17115 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:01.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56361c331d70 con 0x56361c449ed0 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.426+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c449ed0 0x56361c44c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f357bfff640 10 client.17115 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a0b9f640 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c449ed0 0x56361c44c2f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=808ebc9633c8ea4 server_cookie=6918b242845d42c5 in_seq=0 out_seq=0 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f357bfff640 1 -- 172.21.15.16:0/2596019639 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56361c331d70 con 0x56361c449ed0 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f357bfff640 10 client.17115 handle_command_reply: tid=0 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 client.17115 shutdown 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 msgr2=0x56361c3d7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3584000f80 tx=0x7f358402ffd0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 shutdown_connections 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56361c3cf550 0x56361c3c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56361c3c5d20 0x56361c3d7d80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56361c449ed0 0x56361c44c2f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 --2- 172.21.15.16:0/2596019639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56361c3c47b0 0x56361c3d7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f357bfff640 0 client.17115 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 >> 172.21.15.16:0/2596019639 conn(0x56361c38d310 msgr2=0x56361c3ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 shutdown_connections 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 1 -- 172.21.15.16:0/2596019639 wait complete. 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.427+0000 7f35a51dc740 20 client.17115 trim_cache size 0 max 16384 2024-09-12T02:55:01.435 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 msgr2=0x7f35880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 0x7f35880a5040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f358c009c70 tx=0x7f358c005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 msgr2=0x7f359c198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3584005e30 tx=0x7f3584003c90 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 shutdown_connections 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f359c10bfc0 0x7f359c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35880a2b80 0x7f35880a5040 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359c1034b0 0x7f359c198870 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 --2- 172.21.15.16:0/2103569499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c102b90 0x7f359c198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:01.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 >> 172.21.15.16:0/2103569499 conn(0x7f359c0fc610 msgr2=0x7f359c0fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 shutdown_connections 2024-09-12T02:55:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:01.428+0000 7f35a1ba1640 1 -- 172.21.15.16:0/2103569499 wait complete. 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: pgmap v550: 97 pgs: 97 active+clean; 243 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 115 op/s 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:55:01.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:55:01.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:55:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: pgmap v550: 97 pgs: 97 active+clean; 243 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 115 op/s 2024-09-12T02:55:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:55:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:55:01.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 678623 10.22 MB/sec execute 243 sec latency 62.450 ms 2024-09-12T02:55:02.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:55:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:02.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 681223 10.23 MB/sec execute 244 sec latency 64.376 ms 2024-09-12T02:55:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 -- 172.21.15.16:0/3278267168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ea0 msgr2=0x7f01801005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 --2- 172.21.15.16:0/3278267168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ea0 0x7f01801005f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0174009900 tx=0x7f017402a480 comp rx=0 tx=0).stop 2024-09-12T02:55:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 -- 172.21.15.16:0/3278267168 shutdown_connections 2024-09-12T02:55:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 --2- 172.21.15.16:0/3278267168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 0x7f0180110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 --2- 172.21.15.16:0/3278267168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ea0 0x7f01801005f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 --2- 172.21.15.16:0/3278267168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ac0 0x7f01801000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 -- 172.21.15.16:0/3278267168 >> 172.21.15.16:0/3278267168 conn(0x7f0180077210 msgr2=0x7f01800fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 -- 172.21.15.16:0/3278267168 shutdown_connections 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 -- 172.21.15.16:0/3278267168 wait complete. 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.014+0000 7f0185bfc640 1 Processor -- start 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 -- start start 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 0x7f018019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 0x7f018019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0180114160 con 0x7f0180102ac0 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0180113fe0 con 0x7f0180102ea0 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f0185bfc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01801142e0 con 0x7f0180100c10 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35906/0 (socket says 172.21.15.16:35906) 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 -- 172.21.15.16:0/2582509985 learned_addr learned my addr 172.21.15.16:0/2582509985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017ffff640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 0x7f018019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017f7fe640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 0x7f018019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 -- 172.21.15.16:0/2582509985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 msgr2=0x7f018019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 0x7f018019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 -- 172.21.15.16:0/2582509985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 msgr2=0x7f018019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.015+0000 7f017effd640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 0x7f018019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017effd640 1 -- 172.21.15.16:0/2582509985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018019f5f0 con 0x7f0180102ac0 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017ffff640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 0x7f018019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017effd640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f01740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017cff9640 1 -- 172.21.15.16:0/2582509985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017403f030 con 0x7f0180102ac0 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018019f880 con 0x7f0180102ac0 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017cff9640 1 -- 172.21.15.16:0/2582509985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0174003ce0 con 0x7f0180102ac0 2024-09-12T02:55:03.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f017cff9640 1 -- 172.21.15.16:0/2582509985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174035070 con 0x7f0180102ac0 2024-09-12T02:55:03.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.016+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01801a71c0 con 0x7f0180102ac0 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.019+0000 7f0187fd5740 1 Processor -- start 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.019+0000 7f0187fd5740 1 -- start start 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.019+0000 7f017cff9640 1 -- 172.21.15.16:0/2582509985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0174042030 con 0x7f0180102ac0 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0f5b40 0x56260f0f4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 0x56260f101760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56260f05a2f0 con 0x56260f0f45d0 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56260f001cf0 con 0x56260f0ff370 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f0187fd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56260f0562b0 con 0x56260f0f5b40 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0f5b40 0x56260f0f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35908/0 (socket says 172.21.15.16:35908) 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 -- 172.21.15.16:0/4140418615 learned_addr learned my addr 172.21.15.16:0/4140418615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017effd640 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 0x56260f101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 -- 172.21.15.16:0/4140418615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0f5b40 msgr2=0x56260f0f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0f5b40 0x56260f0f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 -- 172.21.15.16:0/4140418615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 msgr2=0x56260f101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 0x56260f101760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 -- 172.21.15.16:0/4140418615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260efe63a0 con 0x56260f0f45d0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017effd640 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 0x56260f101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017cff9640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 0x7f01680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.020+0000 7f017f7fe640 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f017000bc90 tx=0x7f01700107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3279ddaaf3dc954 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.021+0000 7f017cff9640 1 -- 172.21.15.16:0/2582509985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01740ef470 con 0x7f0180102ac0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.021+0000 7f017ffff640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 0x7f01680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.021+0000 7f0165ffb640 1 -- 172.21.15.16:0/4140418615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017001c480 con 0x56260f0f45d0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.021+0000 7f0165ffb640 1 -- 172.21.15.16:0/4140418615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f017001cb00 con 0x56260f0f45d0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.021+0000 7f0165ffb640 1 -- 172.21.15.16:0/4140418615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170011c60 con 0x56260f0f45d0 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- 172.21.15.16:0/4140418615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 msgr2=0x56260f0f4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f017000bc90 tx=0x7f01700107a0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f017ffff640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 0x7f01680a5040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0178009fd0 tx=0x7f0178009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- 172.21.15.16:0/4140418615 shutdown_connections 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0ff370 0x56260f101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0f4a10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 --2- 172.21.15.16:0/4140418615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0f5b40 0x56260f0f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- 172.21.15.16:0/4140418615 >> 172.21.15.16:0/4140418615 conn(0x56260f0bd310 msgr2=0x56260f0f0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- 172.21.15.16:0/4140418615 shutdown_connections 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- 172.21.15.16:0/4140418615 wait complete. 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 -- start start 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.022+0000 7f0187fd5740 1 Processor -- start 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0fc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56260f102070 con 0x56260f0f45d0 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56260f015060 con 0x56260f0f5b40 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56260f1021f0 con 0x56260f0ff370 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 10 client.0 _ll_register_callbacks cb 0x56260ed688c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0fc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39864/0 (socket says 172.21.15.16:39864) 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 -- 172.21.15.16:0/1260501924 learned_addr learned my addr 172.21.15.16:0/1260501924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0fc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59746/0 (socket says 172.21.15.16:59746) 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 -- 172.21.15.16:0/1260501924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 msgr2=0x56260f0fc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0fc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 -- 172.21.15.16:0/1260501924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 msgr2=0x56260f0fedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0fedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 -- 172.21.15.16:0/1260501924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260edcb3c0 con 0x56260f0f5b40 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017f7fe640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f017effd640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f01740b81f0 tx=0x7f0174031a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40fc84c600f0fc7e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f015effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049070 con 0x56260f0f5b40 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01740ef080 con 0x56260f0f5b40 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.023+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f017001cb00 con 0x56260f0f5b40 2024-09-12T02:55:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f01740b8030 con 0x56260f0f5b40 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 10 client.26648 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56260f10db60 con 0x56260f0f5b40 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f017001cb00 con 0x56260f0f5b40 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f01740c44c0 con 0x56260f0f5b40 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 10 client.26648 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 10 client.26648 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 20 client.26648 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f10efa0 0x56260f111430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 4 client.26648 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56260f067860 con 0x56260f10efa0 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.024+0000 7f017f7fe640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f10efa0 0x56260f111430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.025+0000 7f015effd640 10 client.26648 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.025+0000 7f017f7fe640 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f10efa0 0x56260f111430 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4f78e04ed3ba9bf4 server_cookie=aa707309cf61d943 in_seq=0 out_seq=0 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.025+0000 7f015effd640 1 -- 172.21.15.16:0/1260501924 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56260f067860 con 0x56260f10efa0 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.025+0000 7f015effd640 10 client.26648 handle_command_reply: tid=0 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.025+0000 7f0187fd5740 1 client.26648 shutdown 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 msgr2=0x56260f0fc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f01740b81f0 tx=0x7f0174031a60 comp rx=0 tx=0).stop 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 shutdown_connections 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f10efa0 0x56260f111430 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0fc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0fc060 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 --2- 172.21.15.16:0/1260501924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f0fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 >> 172.21.15.16:0/1260501924 conn(0x56260f0bd310 msgr2=0x56260f0f6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f015effd640 0 client.26648 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 shutdown_connections 2024-09-12T02:55:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 1 -- 172.21.15.16:0/1260501924 wait complete. 2024-09-12T02:55:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.026+0000 7f0187fd5740 20 client.26648 trim_cache size 0 max 16384 2024-09-12T02:55:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 Processor -- start 2024-09-12T02:55:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 -- start start 2024-09-12T02:55:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 0x56260f0f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f116d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01801af740 con 0x56260f0f5b40 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01801b1280 con 0x56260f0f45d0 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f0187fd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01801afe10 con 0x56260f0ff370 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f017effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 0x56260f0f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39874/0 (socket says 172.21.15.16:39874) 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f017f7fe640 1 -- 172.21.15.16:0/262512776 learned_addr learned my addr 172.21.15.16:0/262512776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.040+0000 7f017ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f116d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 -- 172.21.15.16:0/262512776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 msgr2=0x56260f116d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f116d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 -- 172.21.15.16:0/262512776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 msgr2=0x56260f0f0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 -- 172.21.15.16:0/262512776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260f0fcb60 con 0x56260f0f5b40 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017f7fe640 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f017effd640 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 0x56260f0f0c40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x56260f0fd610 tx=0x7f01740ba2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d944c42efe4831f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f01657fa640 1 -- 172.21.15.16:0/262512776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049030 con 0x56260f0f5b40 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f01657fa640 1 -- 172.21.15.16:0/262512776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01740f2c40 con 0x56260f0f5b40 2024-09-12T02:55:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.041+0000 7f01657fa640 1 -- 172.21.15.16:0/262512776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01740bb6f0 con 0x56260f0f5b40 2024-09-12T02:55:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- 172.21.15.16:0/262512776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 msgr2=0x56260f0f0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 0x56260f0f0c40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x56260f0fd610 tx=0x7f01740ba2a0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- 172.21.15.16:0/262512776 shutdown_connections 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f116d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f5b40 0x56260f0f0c40 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 --2- 172.21.15.16:0/262512776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f45d0 0x56260f0f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- 172.21.15.16:0/262512776 >> 172.21.15.16:0/262512776 conn(0x56260f0bd310 msgr2=0x56260f0f7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- 172.21.15.16:0/262512776 shutdown_connections 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- 172.21.15.16:0/262512776 wait complete. 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.042+0000 7f0187fd5740 1 -- start start 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 Processor -- start 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0f9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01801b0bb0 con 0x56260f0f45d0 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01801b04e0 con 0x56260f0f5b40 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01801ad720 con 0x56260f0ff370 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f0187fd5740 10 client.0 _ll_register_callbacks cb 0x7f0174005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35932/0 (socket says 172.21.15.16:35932) 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 -- 172.21.15.16:0/239053759 learned_addr learned my addr 172.21.15.16:0/239053759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017ffff640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017effd640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 -- 172.21.15.16:0/239053759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 msgr2=0x56260f0f9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0f9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 -- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 msgr2=0x56260f0f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0f8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.043+0000 7f017f7fe640 1 -- 172.21.15.16:0/239053759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260f0f9680 con 0x56260f0f45d0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f017f7fe640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f017001f010 tx=0x7f0170010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b0c7b5054ec607 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170005850 con 0x56260f0f45d0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f017000bd80 con 0x56260f0f45d0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01740f2c40 con 0x56260f0f45d0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f017003a050 con 0x56260f0f45d0 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f0187fd5740 10 client.17133 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56260f179b30 con 0x56260f0f45d0 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01740f2c40 con 0x56260f0f45d0 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.044+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f017002b450 con 0x56260f0f45d0 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 10 client.17133 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 10 client.17133 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 20 client.17133 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f17a000 0x56260f17c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 4 client.17133 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56260f061d70 con 0x56260f17a000 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f017effd640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f17a000 0x56260f17c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f015effd640 10 client.17133 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f017effd640 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f17a000 0x56260f17c420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4b2379665f61decd server_cookie=160e01a88ae60c7b in_seq=0 out_seq=0 2024-09-12T02:55:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f015effd640 1 -- 172.21.15.16:0/239053759 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56260f061d70 con 0x56260f17a000 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f015effd640 10 client.17133 handle_command_reply: tid=0 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 1 client.17133 shutdown 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 msgr2=0x56260f107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.045+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f017001f010 tx=0x7f0170010fd0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 shutdown_connections 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56260f17a000 0x56260f17c420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56260f0ff370 0x56260f0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56260f0f5b40 0x56260f0f8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f015effd640 0 client.17133 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 --2- 172.21.15.16:0/239053759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56260f0f45d0 0x56260f107970 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 >> 172.21.15.16:0/239053759 conn(0x56260f0bd310 msgr2=0x56260f0f0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 shutdown_connections 2024-09-12T02:55:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 1 -- 172.21.15.16:0/239053759 wait complete. 2024-09-12T02:55:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0187fd5740 20 client.17133 trim_cache size 0 max 16384 2024-09-12T02:55:03.054 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 msgr2=0x7f01680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 0x7f01680a5040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0178009fd0 tx=0x7f0178009500 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 msgr2=0x7f018019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f01740042f0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 shutdown_connections 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0180102ea0 0x7f018019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01680a2b80 0x7f01680a5040 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0180102ac0 0x7f018019ab80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.046+0000 7f0185bfc640 1 --2- 172.21.15.16:0/2582509985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0180100c10 0x7f018019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.047+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 >> 172.21.15.16:0/2582509985 conn(0x7f0180077210 msgr2=0x7f01800fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.047+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 shutdown_connections 2024-09-12T02:55:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:03.047+0000 7f0185bfc640 1 -- 172.21.15.16:0/2582509985 wait complete. 2024-09-12T02:55:03.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:03 smithi033 ceph-mon[35092]: pgmap v551: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-09-12T02:55:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:03 smithi184 ceph-mon[35321]: pgmap v551: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-09-12T02:55:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:03 smithi016 ceph-mon[30925]: pgmap v551: 97 pgs: 97 active+clean; 246 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-09-12T02:55:03.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:55:03.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 683267 10.23 MB/sec execute 245 sec latency 46.993 ms 2024-09-12T02:55:03.950 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 -- 172.21.15.16:0/2101683777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 msgr2=0x7f87cc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 --2- 172.21.15.16:0/2101683777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc103110 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009930 tx=0x7f87bc02a420 comp rx=0 tx=0).stop 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 -- 172.21.15.16:0/2101683777 shutdown_connections 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 --2- 172.21.15.16:0/2101683777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1082a0 0x7f87cc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 --2- 172.21.15.16:0/2101683777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1036e0 0x7f87cc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 --2- 172.21.15.16:0/2101683777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc103110 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 -- 172.21.15.16:0/2101683777 >> 172.21.15.16:0/2101683777 conn(0x7f87cc0fc7d0 msgr2=0x7f87cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 -- 172.21.15.16:0/2101683777 shutdown_connections 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.449+0000 7f87d1652640 1 -- 172.21.15.16:0/2101683777 wait complete. 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 Processor -- start 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 -- start start 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 0x7f87cc198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1082a0 0x7f87cc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1120a0 con 0x7f87cc102d30 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc111f20 con 0x7f87cc1036e0 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87d1652640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc112220 con 0x7f87cc1082a0 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35944/0 (socket says 172.21.15.16:35944) 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87caffd640 1 -- 172.21.15.16:0/781392700 learned_addr learned my addr 172.21.15.16:0/781392700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.450+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 0x7f87cc198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1082a0 0x7f87cc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 -- 172.21.15.16:0/781392700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1082a0 msgr2=0x7f87cc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1082a0 0x7f87cc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 -- 172.21.15.16:0/781392700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 msgr2=0x7f87cc198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 0x7f87cc198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 -- 172.21.15.16:0/781392700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc19d550 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 0x7f87cc198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87caffd640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009230 tx=0x7f87bc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87b3fff640 1 -- 172.21.15.16:0/781392700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc03f030 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc19d7e0 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87b3fff640 1 -- 172.21.15.16:0/781392700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87bc004440 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87b3fff640 1 -- 172.21.15.16:0/781392700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc035070 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.451+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc1a5120 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87b3fff640 1 -- 172.21.15.16:0/781392700 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f87bc042030 con 0x7f87cc102d30 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 Processor -- start 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 -- start start 2024-09-12T02:55:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d5b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87b3fff640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 0x7f87b40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d68760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562149cc12f0 con 0x562149d5b5d0 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562149c68cf0 con 0x562149d66370 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562149cbd2b0 con 0x562149d5cb40 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87ca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87b3fff640 1 -- 172.21.15.16:0/781392700 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f87bc0ef350 con 0x7f87cc102d30 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d5b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.454+0000 7f87ca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35960/0 (socket says 172.21.15.16:35960) 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 -- 172.21.15.16:0/1189969833 learned_addr learned my addr 172.21.15.16:0/1189969833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:04.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87caffd640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 0x7f87b40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d68760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 -- 172.21.15.16:0/1189969833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 msgr2=0x562149d5b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d5b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 -- 172.21.15.16:0/1189969833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 msgr2=0x562149d68760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d68760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 -- 172.21.15.16:0/1189969833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562149c4d3a0 con 0x562149d5b5d0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87caffd640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 0x7f87b40a5040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0bbc00 tx=0x7f87bc02a420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f87c0011470 tx=0x7f87c0011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bc2ea1364587d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87b17fa640 1 -- 172.21.15.16:0/1189969833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c001e070 con 0x562149d5b5d0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87b17fa640 1 -- 172.21.15.16:0/1189969833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87c00035f0 con 0x562149d5b5d0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.455+0000 7f87b17fa640 1 -- 172.21.15.16:0/1189969833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c000f800 con 0x562149d5b5d0 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 -- 172.21.15.16:0/1189969833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 msgr2=0x562149d5ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f87c0011470 tx=0x7f87c0011930 comp rx=0 tx=0).stop 2024-09-12T02:55:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 -- 172.21.15.16:0/1189969833 shutdown_connections 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d68760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d5ba10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/1189969833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d5b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 -- 172.21.15.16:0/1189969833 >> 172.21.15.16:0/1189969833 conn(0x562149d24310 msgr2=0x562149d57c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 -- 172.21.15.16:0/1189969833 shutdown_connections 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.456+0000 7f87d3a2b740 1 -- 172.21.15.16:0/1189969833 wait complete. 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 -- start start 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 Processor -- start 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d62ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562149d69070 con 0x562149d5b5d0 2024-09-12T02:55:04.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562149c7c060 con 0x562149d66370 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562149d691f0 con 0x562149d5cb40 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87d3a2b740 10 client.0 _ll_register_callbacks cb 0x5621499cf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35962/0 (socket says 172.21.15.16:35962) 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 -- 172.21.15.16:0/4010198459 learned_addr learned my addr 172.21.15.16:0/4010198459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d62ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 -- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 msgr2=0x562149d62ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d62ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 -- 172.21.15.16:0/4010198459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 msgr2=0x562149d635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.457+0000 7f87cb7fe640 1 -- 172.21.15.16:0/4010198459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562149a323c0 con 0x562149d5b5d0 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f87c40119f0 tx=0x7f87c4011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1332dc2a93928cc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4023060 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f87c4010c40 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87c00035f0 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f87c4010e40 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 10 client.17145 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562149d74b40 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87c00035f0 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f87c401d070 con 0x562149d5b5d0 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 10 client.17145 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 10 client.17145 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 20 client.17145 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149d76000 0x562149d783f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 4 client.17145 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.458+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562149cce860 con 0x562149d76000 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.459+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149d76000 0x562149d783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.459+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149d76000 0x562149d783f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ec9ae479a7da73c server_cookie=f2ef22b417e5ef5b in_seq=0 out_seq=0 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.459+0000 7f87aeff5640 10 client.17145 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.460+0000 7f87aeff5640 1 -- 172.21.15.16:0/4010198459 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562149cce860 con 0x562149d76000 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.460+0000 7f87aeff5640 10 client.17145 handle_command_reply: tid=0 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 client.17145 shutdown 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 msgr2=0x562149d65d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f87c40119f0 tx=0x7f87c4011eb0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 shutdown_connections 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d66370 0x562149d635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149d76000 0x562149d783f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d5cb40 0x562149d62ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/4010198459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d65d40 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 >> 172.21.15.16:0/4010198459 conn(0x562149d24310 msgr2=0x562149d5d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87aeff5640 0 client.17145 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 shutdown_connections 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 1 -- 172.21.15.16:0/4010198459 wait complete. 2024-09-12T02:55:04.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.461+0000 7f87d3a2b740 20 client.17145 trim_cache size 0 max 16384 2024-09-12T02:55:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 Processor -- start 2024-09-12T02:55:04.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 -- start start 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d57e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d72310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1ad550 con 0x562149d5b5d0 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1af090 con 0x562149d5cb40 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1adc20 con 0x562149d66370 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d57e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d57e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35968/0 (socket says 172.21.15.16:35968) 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87cb7fe640 1 -- 172.21.15.16:0/788260087 learned_addr learned my addr 172.21.15.16:0/788260087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.474+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d72310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 -- 172.21.15.16:0/788260087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 msgr2=0x562149d5e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 -- 172.21.15.16:0/788260087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 msgr2=0x562149d57e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d57e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 -- 172.21.15.16:0/788260087 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562149d52ab0 con 0x562149d5cb40 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d72310 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87c0011440 tx=0x7f87c0009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468614f5ff7d400c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87b0ff9640 1 -- 172.21.15.16:0/788260087 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0029070 con 0x562149d5cb40 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87b0ff9640 1 -- 172.21.15.16:0/788260087 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87c00050e0 con 0x562149d5cb40 2024-09-12T02:55:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.475+0000 7f87b0ff9640 1 -- 172.21.15.16:0/788260087 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c001e040 con 0x562149d5cb40 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- 172.21.15.16:0/788260087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 msgr2=0x562149d72310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d72310 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87c0011440 tx=0x7f87c0009b90 comp rx=0 tx=0).stop 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- 172.21.15.16:0/788260087 shutdown_connections 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d72310 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/788260087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d57e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- 172.21.15.16:0/788260087 >> 172.21.15.16:0/788260087 conn(0x562149d24310 msgr2=0x562149d57700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- 172.21.15.16:0/788260087 shutdown_connections 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- 172.21.15.16:0/788260087 wait complete. 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 -- start start 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.476+0000 7f87d3a2b740 1 Processor -- start 2024-09-12T02:55:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d6e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1ae9c0 con 0x562149d5b5d0 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1ae2f0 con 0x562149d5cb40 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1ab530 con 0x562149d66370 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87d3a2b740 10 client.0 _ll_register_callbacks cb 0x7f87c4011810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35976/0 (socket says 172.21.15.16:35976) 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87cb7fe640 1 -- 172.21.15.16:0/2120193607 learned_addr learned my addr 172.21.15.16:0/2120193607 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d6e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 -- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 msgr2=0x562149d5f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 -- 172.21.15.16:0/2120193607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 msgr2=0x562149d6e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 -- 172.21.15.16:0/2120193607 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562149d5fb80 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87ca7fc640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d6e9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x562149d72d00 tx=0x7f87c00036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3dcad226a667fc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.477+0000 7f87aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0032020 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87c00050e0 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f87c00053d0 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f87c001e040 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 10 client.26669 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562149de0340 con 0x562149d5cb40 2024-09-12T02:55:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87c00050e0 con 0x562149d5cb40 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f87c000fb00 con 0x562149d5cb40 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 10 client.26669 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 10 client.26669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 20 client.26669 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149de07c0 0x562149de2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 4 client.26669 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.478+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562149cc8d70 con 0x562149de07c0 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149de07c0 0x562149de2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87aeff5640 10 client.26669 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87cb7fe640 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149de07c0 0x562149de2bb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d046506e2f88ae9 server_cookie=b23385356011a5f2 in_seq=0 out_seq=0 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87aeff5640 1 -- 172.21.15.16:0/2120193607 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562149cc8d70 con 0x562149de07c0 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87aeff5640 10 client.26669 handle_command_reply: tid=0 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 client.26669 shutdown 2024-09-12T02:55:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 msgr2=0x562149d6e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d6e9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x562149d72d00 tx=0x7f87c00036f0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 shutdown_connections 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562149de07c0 0x562149de2bb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562149d66370 0x562149d5f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562149d5cb40 0x562149d6e9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87aeff5640 0 client.26669 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 --2- 172.21.15.16:0/2120193607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562149d5b5d0 0x562149d6e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 >> 172.21.15.16:0/2120193607 conn(0x562149d24310 msgr2=0x562149d564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.479+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 shutdown_connections 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d3a2b740 1 -- 172.21.15.16:0/2120193607 wait complete. 2024-09-12T02:55:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d3a2b740 20 client.26669 trim_cache size 0 max 16384 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 msgr2=0x7f87b40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 0x7f87b40a5040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0bbc00 tx=0x7f87bc02a420 comp rx=0 tx=0).stop 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 msgr2=0x7f87cc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009230 tx=0x7f87bc004040 comp rx=0 tx=0).stop 2024-09-12T02:55:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 shutdown_connections 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc1082a0 0x7f87cc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1036e0 0x7f87cc198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f87b40a2b80 0x7f87b40a5040 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 --2- 172.21.15.16:0/781392700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87cc102d30 0x7f87cc1985a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 >> 172.21.15.16:0/781392700 conn(0x7f87cc0fc7d0 msgr2=0x7f87cc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 shutdown_connections 2024-09-12T02:55:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:04.480+0000 7f87d1652640 1 -- 172.21.15.16:0/781392700 wait complete. 2024-09-12T02:55:04.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 685907 10.25 MB/sec execute 246 sec latency 75.169 ms 2024-09-12T02:55:05.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:55:05.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:05 smithi016 ceph-mon[30925]: pgmap v552: 97 pgs: 97 active+clean; 251 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 218 op/s 2024-09-12T02:55:05.479 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:05 smithi033 ceph-mon[35092]: pgmap v552: 97 pgs: 97 active+clean; 251 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 218 op/s 2024-09-12T02:55:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:05 smithi184 ceph-mon[35321]: pgmap v552: 97 pgs: 97 active+clean; 251 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 218 op/s 2024-09-12T02:55:05.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 688928 10.28 MB/sec execute 247 sec latency 57.694 ms 2024-09-12T02:55:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1235391547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c01002f0 msgr2=0x7fa8c01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1235391547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c01002f0 0x7fa8c01006d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009900 tx=0x7fa8b002a3f0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1235391547 shutdown_connections 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1235391547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1235391547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c0100ca0 0x7fa8c0106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1235391547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c01002f0 0x7fa8c01006d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1235391547 >> 172.21.15.16:0/1235391547 conn(0x7fa8c0077210 msgr2=0x7fa8c00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1235391547 shutdown_connections 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.966+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1235391547 wait complete. 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 Processor -- start 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 -- start start 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 0x7fa8c01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0111fa0 con 0x7fa8c01002f0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0111e20 con 0x7fa8c0106f50 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8c4ce4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0112120 con 0x7fa8c0100ca0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58496/0 (socket says 172.21.15.16:58496) 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8be59c640 1 -- 172.21.15.16:0/1232388690 learned_addr learned my addr 172.21.15.16:0/1232388690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 0x7fa8c01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 -- 172.21.15.16:0/1232388690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 msgr2=0x7fa8c019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 -- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 msgr2=0x7fa8c0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bdd9b640 1 -- 172.21.15.16:0/1232388690 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c019d420 con 0x7fa8c0100ca0 2024-09-12T02:55:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.967+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 0x7fa8c01989b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e9c0 tx=0x7fa8b400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8af7fe640 1 -- 172.21.15.16:0/1232388690 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400ccc0 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8af7fe640 1 -- 172.21.15.16:0/1232388690 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b4010910 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8af7fe640 1 -- 172.21.15.16:0/1232388690 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40188c0 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c019d710 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.968+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c01a4ff0 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.970+0000 7fa8af7fe640 1 -- 172.21.15.16:0/1232388690 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa8b4010ac0 con 0x7fa8c0100ca0 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 Processor -- start 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 -- start start 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaebba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8af7fe640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 0x7fa8a80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55abaae212f0 con 0x55abaaec6370 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55abaadc8cf0 con 0x55abaaebcb40 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55abaae1d2b0 con 0x55abaaebb5d0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37008/0 (socket says 172.21.15.16:37008) 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8bed9d640 1 -- 172.21.15.16:0/107670179 learned_addr learned my addr 172.21.15.16:0/107670179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaebba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8af7fe640 1 -- 172.21.15.16:0/1232388690 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8b40cf3f0 con 0x7fa8c0100ca0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.971+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 0x7fa8a80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 -- 172.21.15.16:0/107670179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 msgr2=0x55abaaebba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaebba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 -- 172.21.15.16:0/107670179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 msgr2=0x55abaaec8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 -- 172.21.15.16:0/107670179 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abaadad3a0 con 0x55abaaebcb40 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bdd9b640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 0x7fa8a80a5040 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000c00 tx=0x7fa8b400efb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005c40 tx=0x7fa8b8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=febb60928498b946 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8acff9640 1 -- 172.21.15.16:0/107670179 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800fc20 con 0x55abaaebcb40 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8acff9640 1 -- 172.21.15.16:0/107670179 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55abaaebcb40 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.972+0000 7fa8acff9640 1 -- 172.21.15.16:0/107670179 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8004860 con 0x55abaaebcb40 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- 172.21.15.16:0/107670179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 msgr2=0x55abaaebb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005c40 tx=0x7fa8b8005bd0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- 172.21.15.16:0/107670179 shutdown_connections 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaebba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/107670179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaebb090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- 172.21.15.16:0/107670179 >> 172.21.15.16:0/107670179 conn(0x55abaae84310 msgr2=0x55abaaeb7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- 172.21.15.16:0/107670179 shutdown_connections 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- 172.21.15.16:0/107670179 wait complete. 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 -- start start 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.973+0000 7fa8c70bd740 1 Processor -- start 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaec3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55abaaec9010 con 0x55abaaec6370 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55abaaddc060 con 0x55abaaebcb40 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55abaaec9190 con 0x55abaaebb5d0 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8c70bd740 10 client.0 _ll_register_callbacks cb 0x55abaab2f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaec3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58508/0 (socket says 172.21.15.16:58508) 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8bed9d640 1 -- 172.21.15.16:0/1060092425 learned_addr learned my addr 172.21.15.16:0/1060092425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 -- 172.21.15.16:0/1060092425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 msgr2=0x55abaaec5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 -- 172.21.15.16:0/1060092425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 msgr2=0x55abaaec37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 -- 172.21.15.16:0/1060092425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abaab923c0 con 0x55abaaebcb40 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaec3050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009230 tx=0x7fa8b0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4022a0834226b740 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.974+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0035040 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b8003e50 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8b0003e20 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8b0031540 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 10 client.26681 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55abaaed4af0 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8b0046860 con 0x55abaaebcb40 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 10 client.26681 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 10 client.26681 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 20 client.26681 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaed83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 4 client.26681 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abaae2e860 con 0x55abaaed5fb0 2024-09-12T02:55:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.975+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaed83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaed83a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75c4d1c5c23e74b server_cookie=35f4e8076b87a350 in_seq=0 out_seq=0 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8a5ffb640 10 client.26681 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1060092425 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55abaae2e860 con 0x55abaaed5fb0 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8a5ffb640 10 client.26681 handle_command_reply: tid=0 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8c70bd740 1 client.26681 shutdown 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 msgr2=0x55abaaec3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.976+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaec3050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009230 tx=0x7fa8b0003820 comp rx=0 tx=0).stop 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 shutdown_connections 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaed83a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaec6370 0x55abaaec37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8a5ffb640 0 client.26681 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaec3050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1060092425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 >> 172.21.15.16:0/1060092425 conn(0x55abaae84310 msgr2=0x55abaaebd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 shutdown_connections 2024-09-12T02:55:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1060092425 wait complete. 2024-09-12T02:55:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.977+0000 7fa8c70bd740 20 client.26681 trim_cache size 0 max 16384 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.990+0000 7fa8c70bd740 1 Processor -- start 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.990+0000 7fa8c70bd740 1 -- start start 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaec6370 0x55abaaede1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01ad560 con 0x55abaaebcb40 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01af0a0 con 0x55abaaec6370 2024-09-12T02:55:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01adc30 con 0x55abaaebb5d0 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58524/0 (socket says 172.21.15.16:58524) 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 -- 172.21.15.16:0/1750214566 learned_addr learned my addr 172.21.15.16:0/1750214566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaec6370 0x55abaaede1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 -- 172.21.15.16:0/1750214566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 msgr2=0x55abaaec3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 -- 172.21.15.16:0/1750214566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaec6370 msgr2=0x55abaaede1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaec6370 0x55abaaede1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 -- 172.21.15.16:0/1750214566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abaaeb7510 con 0x55abaaebcb40 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0046da0 tx=0x7fa8b002a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e7ac42581e8c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.991+0000 7fa8a7fff640 1 -- 172.21.15.16:0/1750214566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0035030 con 0x55abaaebcb40 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8a7fff640 1 -- 172.21.15.16:0/1750214566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b0003e20 con 0x55abaaebcb40 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8a7fff640 1 -- 172.21.15.16:0/1750214566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00052a0 con 0x55abaaebcb40 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1750214566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 msgr2=0x55abaaeddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0046da0 tx=0x7fa8b002a3f0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1750214566 shutdown_connections 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaec6370 0x55abaaede1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebcb40 0x55abaaeddc70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1750214566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaebb5d0 0x55abaaec3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1750214566 >> 172.21.15.16:0/1750214566 conn(0x55abaae84310 msgr2=0x55abaaec2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1750214566 shutdown_connections 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.992+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1750214566 wait complete. 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 -- start start 2024-09-12T02:55:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 Processor -- start 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaec6370 0x55abaaebf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01ae9d0 con 0x55abaaebb5d0 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01ae300 con 0x55abaaebcb40 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01ab540 con 0x55abaaec6370 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8c70bd740 10 client.0 _ll_register_callbacks cb 0x7fa8b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58530/0 (socket says 172.21.15.16:58530) 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8bed9d640 1 -- 172.21.15.16:0/1886964550 learned_addr learned my addr 172.21.15.16:0/1886964550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37036/0 (socket says 172.21.15.16:37036) 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaec6370 0x55abaaebf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 -- 172.21.15.16:0/1886964550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaec6370 msgr2=0x55abaaebf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaec6370 0x55abaaebf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 -- 172.21.15.16:0/1886964550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 msgr2=0x55abaaece420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8be59c640 1 -- 172.21.15.16:0/1886964550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abaaebf960 con 0x55abaaebcb40 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.993+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8be59c640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55abaaeb3690 tx=0x7fa8b0046ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c371ccdbbadd745 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b005d070 con 0x55abaaebcb40 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8b0005250 con 0x55abaaebcb40 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b0003e20 con 0x55abaaebcb40 2024-09-12T02:55:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8b0035040 con 0x55abaaebcb40 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8c70bd740 10 client.26687 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55abaaf402d0 con 0x55abaaebcb40 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b0003e20 con 0x55abaaebcb40 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8b00316e0 con 0x55abaaebcb40 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.994+0000 7fa8c70bd740 10 client.26687 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 10 client.26687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 20 client.26687 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaf42890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 4 client.26687 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abaae28d70 con 0x55abaaed5fb0 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaf42890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8bed9d640 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaf42890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a59051b8b58d8b7 server_cookie=7f570037d444830d in_seq=0 out_seq=0 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8a5ffb640 10 client.26687 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8a5ffb640 1 -- 172.21.15.16:0/1886964550 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abaae28d70 con 0x55abaaed5fb0 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8a5ffb640 10 client.26687 handle_command_reply: tid=0 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 client.26687 shutdown 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 msgr2=0x55abaaece960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55abaaeb3690 tx=0x7fa8b0046ae0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 shutdown_connections 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55abaaed5fb0 0x55abaaf42890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55abaaec6370 0x55abaaebf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55abaaebcb40 0x55abaaece960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8a5ffb640 0 client.26687 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.995+0000 7fa8c70bd740 1 --2- 172.21.15.16:0/1886964550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55abaaebb5d0 0x55abaaece420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 >> 172.21.15.16:0/1886964550 conn(0x55abaae84310 msgr2=0x55abaaec48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 shutdown_connections 2024-09-12T02:55:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c70bd740 1 -- 172.21.15.16:0/1886964550 wait complete. 2024-09-12T02:55:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c70bd740 20 client.26687 trim_cache size 0 max 16384 2024-09-12T02:55:06.009 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:06.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 msgr2=0x7fa8a80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 0x7fa8a80a5040 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000c00 tx=0x7fa8b400efb0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 msgr2=0x7fa8c01989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 0x7fa8c01989b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e9c0 tx=0x7fa8b400ee80 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 shutdown_connections 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c0106f50 0x7fa8c019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8c0100ca0 0x7fa8c01989b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8a80a2b80 0x7fa8a80a5040 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 --2- 172.21.15.16:0/1232388690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c01002f0 0x7fa8c0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 >> 172.21.15.16:0/1232388690 conn(0x7fa8c0077210 msgr2=0x7fa8c00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 shutdown_connections 2024-09-12T02:55:06.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:05.996+0000 7fa8c4ce4640 1 -- 172.21.15.16:0/1232388690 wait complete. 2024-09-12T02:55:06.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:55:06.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 691403 10.27 MB/sec execute 248 sec latency 17.280 ms 2024-09-12T02:55:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:07.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:07 smithi016 ceph-mon[30925]: pgmap v553: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-09-12T02:55:07.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:07 smithi033 ceph-mon[35092]: pgmap v553: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-09-12T02:55:07.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:07 smithi184 ceph-mon[35321]: pgmap v553: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 -- 172.21.15.16:0/1619411948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0102570 msgr2=0x7f38a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 --2- 172.21.15.16:0/1619411948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0102570 0x7f38a01029b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3894009900 tx=0x7f389402a480 comp rx=0 tx=0).stop 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 -- 172.21.15.16:0/1619411948 shutdown_connections 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 --2- 172.21.15.16:0/1619411948 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102ef0 0x7f38a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 --2- 172.21.15.16:0/1619411948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0102570 0x7f38a01029b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 --2- 172.21.15.16:0/1619411948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0100e20 0x7f38a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 -- 172.21.15.16:0/1619411948 >> 172.21.15.16:0/1619411948 conn(0x7f38a00fc7d0 msgr2=0x7f38a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 -- 172.21.15.16:0/1619411948 shutdown_connections 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.492+0000 7f38a72ca640 1 -- 172.21.15.16:0/1619411948 wait complete. 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 Processor -- start 2024-09-12T02:55:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 -- start start 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102570 0x7f38a0199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0102ef0 0x7f38a01a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0110f50 con 0x7f38a0100e20 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0110dd0 con 0x7f38a0102ef0 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a72ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01110d0 con 0x7f38a0102570 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58534/0 (socket says 172.21.15.16:58534) 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 -- 172.21.15.16:0/861026189 learned_addr learned my addr 172.21.15.16:0/861026189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a4865640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102570 0x7f38a0199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5867640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0102ef0 0x7f38a01a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 -- 172.21.15.16:0/861026189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102570 msgr2=0x7f38a0199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102570 0x7f38a0199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 -- 172.21.15.16:0/861026189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0102ef0 msgr2=0x7f38a01a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0102ef0 0x7f38a01a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.493+0000 7f38a5066640 1 -- 172.21.15.16:0/861026189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a01a4980 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f38a5066640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f389000eab0 tx=0x7f389000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f389e7fc640 1 -- 172.21.15.16:0/861026189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389000ce40 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a01a4b50 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f389e7fc640 1 -- 172.21.15.16:0/861026189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3890011950 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f389e7fc640 1 -- 172.21.15.16:0/861026189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890010ae0 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.494+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a01a5060 con 0x7f38a0100e20 2024-09-12T02:55:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f389e7fc640 1 -- 172.21.15.16:0/861026189 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3890011b00 con 0x7f38a0100e20 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 Processor -- start 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 -- start start 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7bb40 0x55800be7a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be8bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55800bde02f0 con 0x55800be7a5d0 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55800bd87cf0 con 0x55800be7bb40 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a96a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55800bddc2b0 con 0x55800be89780 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a5066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a5066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58536/0 (socket says 172.21.15.16:58536) 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a5066640 1 -- 172.21.15.16:0/2847701430 learned_addr learned my addr 172.21.15.16:0/2847701430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:07.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a4865640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be8bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.497+0000 7f38a5867640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7bb40 0x55800be7a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f389e7fc640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 0x7f38880a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 -- 172.21.15.16:0/2847701430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 msgr2=0x55800be8bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be8bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 -- 172.21.15.16:0/2847701430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7bb40 msgr2=0x55800be7a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7bb40 0x55800be7a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 -- 172.21.15.16:0/2847701430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800bd6c3a0 con 0x55800be7a5d0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a4865640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5867640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 0x7f38880a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f389e7fc640 1 -- 172.21.15.16:0/861026189 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38900cf620 con 0x7f38a0100e20 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38a5066640 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3890064560 tx=0x7f389006ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656b5a31c0311715 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38877fe640 1 -- 172.21.15.16:0/2847701430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389009b7f0 con 0x55800be7a5d0 2024-09-12T02:55:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38877fe640 1 -- 172.21.15.16:0/2847701430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f389009c590 con 0x55800be7a5d0 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.498+0000 7f38877fe640 1 -- 172.21.15.16:0/2847701430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38900645c0 con 0x55800be7a5d0 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- 172.21.15.16:0/2847701430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 msgr2=0x55800be7aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a5867640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 0x7f38880a50c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f38980058c0 tx=0x7f3898004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3890064560 tx=0x7f389006ef50 comp rx=0 tx=0).stop 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- 172.21.15.16:0/2847701430 shutdown_connections 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7a5d0 0x55800be7aa10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2847701430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7bb40 0x55800be7a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- 172.21.15.16:0/2847701430 >> 172.21.15.16:0/2847701430 conn(0x55800be43310 msgr2=0x55800be76c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- 172.21.15.16:0/2847701430 shutdown_connections 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- 172.21.15.16:0/2847701430 wait complete. 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 -- start start 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.499+0000 7f38a96a3740 1 Processor -- start 2024-09-12T02:55:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be86970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55800be8c420 con 0x55800be7bb40 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55800bd9b060 con 0x55800be7a5d0 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55800be8c5a0 con 0x55800be89780 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a96a3740 10 client.0 _ll_register_callbacks cb 0x55800baee8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a4865640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be86970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a4865640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58544/0 (socket says 172.21.15.16:58544) 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a4865640 1 -- 172.21.15.16:0/994592011 learned_addr learned my addr 172.21.15.16:0/994592011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5066640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 -- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 msgr2=0x55800be89120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be89120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 -- 172.21.15.16:0/994592011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 msgr2=0x55800be863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5867640 1 -- 172.21.15.16:0/994592011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800bb513c0 con 0x55800be89780 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a5066640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be89120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.500+0000 7f38a4865640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a5867640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be86970 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3898020b80 tx=0x7f3898005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797cf5028f80df15 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898004460 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f389800f8c0 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f389009c590 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3898032050 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 10 client.26587 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55800be97ed0 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f389009c590 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f389801e460 con 0x55800be89780 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 10 client.26587 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 20 client.26587 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800be993e0 0x55800be9b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 4 client.26587 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.501+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55800bded860 con 0x55800be993e0 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.502+0000 7f38a5066640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800be993e0 0x55800be9b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.502+0000 7f3884ff9640 10 client.26587 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.502+0000 7f38a5066640 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800be993e0 0x55800be9b830 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20687229d09ce109 server_cookie=48c6c69ac119950 in_seq=0 out_seq=0 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f3884ff9640 1 -- 172.21.15.16:0/994592011 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55800bded860 con 0x55800be993e0 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f3884ff9640 10 client.26587 handle_command_reply: tid=0 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 client.26587 shutdown 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 msgr2=0x55800be86970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be86970 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3898020b80 tx=0x7f3898005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 shutdown_connections 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800be993e0 0x55800be9b830 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be89780 0x55800be86970 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f3884ff9640 0 client.26587 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 --2- 172.21.15.16:0/994592011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be89120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 >> 172.21.15.16:0/994592011 conn(0x55800be43310 msgr2=0x55800be7c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 shutdown_connections 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 1 -- 172.21.15.16:0/994592011 wait complete. 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.503+0000 7f38a96a3740 20 client.26587 trim_cache size 0 max 16384 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 Processor -- start 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 -- start start 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7bb40 0x55800be85c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be89780 0x55800be84ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ab300 con 0x55800be89780 2024-09-12T02:55:07.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ace40 con 0x55800be7a5d0 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a96a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ab9d0 con 0x55800be7bb40 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a5867640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be89780 0x55800be84ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a5066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a4865640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7bb40 0x55800be85c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a5066640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37086/0 (socket says 172.21.15.16:37086) 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.517+0000 7f38a5066640 1 -- 172.21.15.16:0/838869608 learned_addr learned my addr 172.21.15.16:0/838869608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 -- 172.21.15.16:0/838869608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7bb40 msgr2=0x55800be85c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7bb40 0x55800be85c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 -- 172.21.15.16:0/838869608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be89780 msgr2=0x55800be84ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be89780 0x55800be84ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 -- 172.21.15.16:0/838869608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800be8c590 con 0x55800be7a5d0 2024-09-12T02:55:07.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f38a5066640 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38900982f0 tx=0x7f38900094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce8f77411f98b14 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f3886ffd640 1 -- 172.21.15.16:0/838869608 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38900a0240 con 0x55800be7a5d0 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f3886ffd640 1 -- 172.21.15.16:0/838869608 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38900a4b00 con 0x55800be7a5d0 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.518+0000 7f3886ffd640 1 -- 172.21.15.16:0/838869608 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389009a220 con 0x55800be7a5d0 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- 172.21.15.16:0/838869608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 msgr2=0x55800be956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38900982f0 tx=0x7f38900094a0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- 172.21.15.16:0/838869608 shutdown_connections 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be89780 0x55800be84ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7bb40 0x55800be85c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 --2- 172.21.15.16:0/838869608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be7a5d0 0x55800be956c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- 172.21.15.16:0/838869608 >> 172.21.15.16:0/838869608 conn(0x55800be43310 msgr2=0x55800be86390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- 172.21.15.16:0/838869608 shutdown_connections 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- 172.21.15.16:0/838869608 wait complete. 2024-09-12T02:55:07.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 -- start start 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.519+0000 7f38a96a3740 1 Processor -- start 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be91830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be89780 0x55800be91d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ac770 con 0x55800be7bb40 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ac0a0 con 0x55800be89780 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01a9300 con 0x55800be7a5d0 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a96a3740 10 client.0 _ll_register_callbacks cb 0x7f3898009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a4865640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be91830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41150/0 (socket says 172.21.15.16:41150) 2024-09-12T02:55:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 -- 172.21.15.16:0/2455054786 learned_addr learned my addr 172.21.15.16:0/2455054786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5867640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be89780 0x55800be91d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 -- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be89780 msgr2=0x55800be91d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be89780 0x55800be91d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 -- 172.21.15.16:0/2455054786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 msgr2=0x55800be91830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be91830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 -- 172.21.15.16:0/2455054786 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800be922e0 con 0x55800be7a5d0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f38a5066640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f38900cd4a0 tx=0x7f38900bc0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdb2d6f3d9b4100 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.520+0000 7f3884ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389009b7f0 con 0x55800be7a5d0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f389009a190 con 0x55800be7a5d0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38900a4b00 con 0x55800be7a5d0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f389006f450 con 0x55800be7a5d0 2024-09-12T02:55:07.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 10 client.26593 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55800bf040d0 con 0x55800be7a5d0 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38900a4b00 con 0x55800be7a5d0 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f389000f040 con 0x55800be7a5d0 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 10 client.26593 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 10 client.26593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 20 client.26593 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800bf042a0 0x55800bf06690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 4 client.26593 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.521+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55800bde7d70 con 0x55800bf042a0 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a4865640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800bf042a0 0x55800bf06690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f3884ff9640 10 client.26593 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:07.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a4865640 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800bf042a0 0x55800bf06690 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83273d5cfdc0a5f5 server_cookie=b714f60b25f9cad0 in_seq=0 out_seq=0 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f3884ff9640 1 -- 172.21.15.16:0/2455054786 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55800bde7d70 con 0x55800bf042a0 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f3884ff9640 10 client.26593 handle_command_reply: tid=0 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 client.26593 shutdown 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 msgr2=0x55800be83c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f38900cd4a0 tx=0x7f38900bc0c0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 shutdown_connections 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55800bf042a0 0x55800bf06690 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55800be89780 0x55800be91d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55800be7bb40 0x55800be91830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f3884ff9640 0 client.26593 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 --2- 172.21.15.16:0/2455054786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55800be7a5d0 0x55800be83c80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 >> 172.21.15.16:0/2455054786 conn(0x55800be43310 msgr2=0x55800be848a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 shutdown_connections 2024-09-12T02:55:07.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.522+0000 7f38a96a3740 1 -- 172.21.15.16:0/2455054786 wait complete. 2024-09-12T02:55:07.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a96a3740 20 client.26593 trim_cache size 0 max 16384 2024-09-12T02:55:07.666 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 msgr2=0x7f38880a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 0x7f38880a50c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f38980058c0 tx=0x7f3898004730 comp rx=0 tx=0).stop 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 msgr2=0x7f38a01993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f389000eab0 tx=0x7f389000ef70 comp rx=0 tx=0).stop 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 shutdown_connections 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38a0102ef0 0x7f38a01a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a0102570 0x7f38a0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38880a2c00 0x7f38880a50c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 --2- 172.21.15.16:0/861026189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a0100e20 0x7f38a01993c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 >> 172.21.15.16:0/861026189 conn(0x7f38a00fc7d0 msgr2=0x7f38a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 shutdown_connections 2024-09-12T02:55:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:07.523+0000 7f38a72ca640 1 -- 172.21.15.16:0/861026189 wait complete. 2024-09-12T02:55:07.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 693415 10.28 MB/sec execute 249 sec latency 31.419 ms 2024-09-12T02:55:08.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 696401 10.31 MB/sec execute 250 sec latency 56.051 ms 2024-09-12T02:55:09.326 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:55:09.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:55:09.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:09 smithi184 ceph-mon[35321]: pgmap v554: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 225 op/s 2024-09-12T02:55:09.614 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:09.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:09 smithi016 ceph-mon[30925]: pgmap v554: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 225 op/s 2024-09-12T02:55:09.725 INFO:tasks.workunit.client.0.smithi016.stdout: 1 699395 10.33 MB/sec execute 251 sec latency 12.537 ms 2024-09-12T02:55:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:09 smithi033 ceph-mon[35092]: pgmap v554: 97 pgs: 97 active+clean; 250 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 225 op/s 2024-09-12T02:55:10.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- 172.21.15.16:0/2478204104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0112ba0 msgr2=0x7f50d0114f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 --2- 172.21.15.16:0/2478204104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0112ba0 0x7f50d0114f90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009ca0 tx=0x7f50c802a950 comp rx=0 tx=0).stop 2024-09-12T02:55:10.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- 172.21.15.16:0/2478204104 shutdown_connections 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 --2- 172.21.15.16:0/2478204104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0112ba0 0x7f50d0114f90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 --2- 172.21.15.16:0/2478204104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0074eb0 0x7f50d0112660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 --2- 172.21.15.16:0/2478204104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d0074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- 172.21.15.16:0/2478204104 >> 172.21.15.16:0/2478204104 conn(0x7f50d00fd8b0 msgr2=0x7f50d00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- 172.21.15.16:0/2478204104 shutdown_connections 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- 172.21.15.16:0/2478204104 wait complete. 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 Processor -- start 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.134+0000 7f50d785a640 1 -- start start 2024-09-12T02:55:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d01a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0112ba0 0x7f50d019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d0118560 con 0x7f50d0074eb0 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01183e0 con 0x7f50d0074590 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01186e0 con 0x7f50d0112ba0 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58580/0 (socket says 172.21.15.16:58580) 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 -- 172.21.15.16:0/4057668475 learned_addr learned my addr 172.21.15.16:0/4057668475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d01a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d5df7640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0112ba0 0x7f50d019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 -- 172.21.15.16:0/4057668475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0112ba0 msgr2=0x7f50d019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0112ba0 0x7f50d019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 -- 172.21.15.16:0/4057668475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 msgr2=0x7f50d01a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d01a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 -- 172.21.15.16:0/4057668475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d019efa0 con 0x7f50d0074eb0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d55f6640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d01a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d4df5640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f50c000e9f0 tx=0x7f50c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50c67fc640 1 -- 172.21.15.16:0/4057668475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c000cd80 con 0x7f50d0074eb0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50c67fc640 1 -- 172.21.15.16:0/4057668475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c0010910 con 0x7f50d0074eb0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d019f290 con 0x7f50d0074eb0 2024-09-12T02:55:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.135+0000 7f50c67fc640 1 -- 172.21.15.16:0/4057668475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0018990 con 0x7f50d0074eb0 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.136+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d01ab580 con 0x7f50d0074eb0 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50c67fc640 1 -- 172.21.15.16:0/4057668475 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f50c0010ac0 con 0x7f50d0074eb0 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 Processor -- start 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 -- start start 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e43800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e46130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557070d9e280 con 0x557070e38b60 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557070d45b10 con 0x557070e39480 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d9c33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557070d9a240 con 0x557070e43d40 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d5df7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d5df7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58590/0 (socket says 172.21.15.16:58590) 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d5df7640 1 -- 172.21.15.16:0/824721086 learned_addr learned my addr 172.21.15.16:0/824721086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d4df5640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e46130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.139+0000 7f50d55f6640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e43800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50c67fc640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 0x7f50bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 -- 172.21.15.16:0/824721086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 msgr2=0x557070e43800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e43800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 -- 172.21.15.16:0/824721086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 msgr2=0x557070e38f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 -- 172.21.15.16:0/824721086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557070d521f0 con 0x557070e43d40 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d5df7640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d5df7640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 0x7f50bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50c67fc640 1 -- 172.21.15.16:0/4057668475 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50c00cf2c0 con 0x7f50d0074eb0 2024-09-12T02:55:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50d4df5640 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e46130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50c00ccda0 tx=0x7f50c0097fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c217a43c33e384ec server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.140+0000 7f50bb7fe640 1 -- 172.21.15.16:0/824721086 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0086030 con 0x557070e43d40 2024-09-12T02:55:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50bb7fe640 1 -- 172.21.15.16:0/824721086 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c009bbe0 con 0x557070e43d40 2024-09-12T02:55:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50bb7fe640 1 -- 172.21.15.16:0/824721086 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0026030 con 0x557070e43d40 2024-09-12T02:55:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 -- 172.21.15.16:0/824721086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 msgr2=0x557070e46130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e46130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50c00ccda0 tx=0x7f50c0097fb0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d5df7640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 0x7f50bc0a5040 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f50c8030040 tx=0x7f50c803c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 -- 172.21.15.16:0/824721086 shutdown_connections 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e46130 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e43800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.141+0000 7f50d9c33740 1 --2- 172.21.15.16:0/824721086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e38f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.142+0000 7f50d9c33740 1 -- 172.21.15.16:0/824721086 >> 172.21.15.16:0/824721086 conn(0x557070e350f0 msgr2=0x557070e354d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.142+0000 7f50d9c33740 1 -- 172.21.15.16:0/824721086 shutdown_connections 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.142+0000 7f50d9c33740 1 -- 172.21.15.16:0/824721086 wait complete. 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.142+0000 7f50d9c33740 1 -- start start 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.142+0000 7f50d9c33740 1 Processor -- start 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e40bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e41110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e41650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557070e468b0 con 0x557070e38b60 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557070d58f10 con 0x557070e39480 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557070e46a30 con 0x557070e43d40 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d9c33740 10 client.0 _ll_register_callbacks cb 0x557070d21a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e40bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e41110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e40bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58598/0 (socket says 172.21.15.16:58598) 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d55f6640 1 -- 172.21.15.16:0/2330446079 learned_addr learned my addr 172.21.15.16:0/2330446079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d5df7640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e41650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 -- 172.21.15.16:0/2330446079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 msgr2=0x557070e41650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e41650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 -- 172.21.15.16:0/2330446079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 msgr2=0x557070e40bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e40bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 -- 172.21.15.16:0/2330446079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557070b0f6a0 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d5df7640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e41650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50d4df5640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e41110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50c00ccdd0 tx=0x7f50c0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645d3d8eb97565e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.143+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c009ce60 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50c009bbe0 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50c00cc5a0 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50c0086040 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 10 client.26717 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557070e52460 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c009bbe0 con 0x557070e39480 2024-09-12T02:55:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f50c0098250 con 0x557070e39480 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 10 client.26717 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 20 client.26717 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070e55ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 4 client.26717 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557070dab7f0 con 0x557070e537b0 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.144+0000 7f50d55f6640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070e55ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.145+0000 7f50b8ff9640 10 client.26717 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.145+0000 7f50d55f6640 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070e55ba0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465189000cb0fc9 server_cookie=8581ecde13c666a9 in_seq=0 out_seq=0 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.145+0000 7f50b8ff9640 1 -- 172.21.15.16:0/2330446079 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557070dab7f0 con 0x557070e537b0 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.145+0000 7f50b8ff9640 10 client.26717 handle_command_reply: tid=0 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.145+0000 7f50d9c33740 1 client.26717 shutdown 2024-09-12T02:55:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 msgr2=0x557070e41110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e41110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50c00ccdd0 tx=0x7f50c0009310 comp rx=0 tx=0).stop 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 shutdown_connections 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070e55ba0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e43d40 0x557070e41650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50b8ff9640 0 client.26717 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e39480 0x557070e41110 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2330446079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e40bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 >> 172.21.15.16:0/2330446079 conn(0x557070e350f0 msgr2=0x557070e3add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 shutdown_connections 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 1 -- 172.21.15.16:0/2330446079 wait complete. 2024-09-12T02:55:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.146+0000 7f50d9c33740 20 client.26717 trim_cache size 0 max 16384 2024-09-12T02:55:10.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 Processor -- start 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 -- start start 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e414a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e43d40 0x557070e5b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b3c20 con 0x557070e43d40 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b5760 con 0x557070e38b60 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d9c33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b42f0 con 0x557070e39480 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.165+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37132/0 (socket says 172.21.15.16:37132) 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 -- 172.21.15.16:0/2344428524 learned_addr learned my addr 172.21.15.16:0/2344428524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d5df7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e43d40 0x557070e5b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d4df5640 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 -- 172.21.15.16:0/2344428524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 msgr2=0x557070e414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 -- 172.21.15.16:0/2344428524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e43d40 msgr2=0x557070e5b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e43d40 0x557070e5b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 -- 172.21.15.16:0/2344428524 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557070e3b9d0 con 0x557070e38b60 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50d55f6640 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f50cc012960 tx=0x7f50cc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=683f9780acea3fd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50baffd640 1 -- 172.21.15.16:0/2344428524 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc004030 con 0x557070e38b60 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50baffd640 1 -- 172.21.15.16:0/2344428524 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc005500 con 0x557070e38b60 2024-09-12T02:55:10.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.166+0000 7f50baffd640 1 -- 172.21.15.16:0/2344428524 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc00fda0 con 0x557070e38b60 2024-09-12T02:55:10.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 -- 172.21.15.16:0/2344428524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 msgr2=0x557070e42df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f50cc012960 tx=0x7f50cc005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 -- 172.21.15.16:0/2344428524 shutdown_connections 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e43d40 0x557070e5b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 --2- 172.21.15.16:0/2344428524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e38b60 0x557070e42df0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 -- 172.21.15.16:0/2344428524 >> 172.21.15.16:0/2344428524 conn(0x557070e350f0 msgr2=0x557070e3add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 -- 172.21.15.16:0/2344428524 shutdown_connections 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.167+0000 7f50d9c33740 1 -- 172.21.15.16:0/2344428524 wait complete. 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 -- start start 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 Processor -- start 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e47c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e43d40 0x557070e48700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b5090 con 0x557070e38b60 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b49c0 con 0x557070e43d40 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01b1c00 con 0x557070e39480 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d9c33740 10 client.0 _ll_register_callbacks cb 0x7f50c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d4df5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d55f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e47c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d4df5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41190/0 (socket says 172.21.15.16:41190) 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d4df5640 1 -- 172.21.15.16:0/1517442562 learned_addr learned my addr 172.21.15.16:0/1517442562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.168+0000 7f50d5df7640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e43d40 0x557070e48700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 -- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e43d40 msgr2=0x557070e48700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e43d40 0x557070e48700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 -- 172.21.15.16:0/1517442562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 msgr2=0x557070e47c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e47c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 -- 172.21.15.16:0/1517442562 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557070e388e0 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d4df5640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f50c00985b0 tx=0x7f50c0009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a558a4c02ad322 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c00a9060 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50c009a1a0 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50cc005500 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50c009ce60 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d9c33740 10 client.26611 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.169+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557070ebe560 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c009bbe0 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f50c00a4310 con 0x557070e39480 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 10 client.26611 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:10.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 10 client.26611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 20 client.26611 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070ec0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 4 client.26611 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f50cc005500 con 0x557070e537b0 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d55f6640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070ec0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50d55f6640 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070ec0a30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0996aa04dec5a5b server_cookie=405cd2d69ab9fe1f in_seq=0 out_seq=0 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.170+0000 7f50b8ff9640 10 client.26611 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50b8ff9640 1 -- 172.21.15.16:0/1517442562 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f50cc005500 con 0x557070e537b0 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50b8ff9640 10 client.26611 handle_command_reply: tid=0 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 client.26611 shutdown 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 msgr2=0x557070e481c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f50c00985b0 tx=0x7f50c0009700 comp rx=0 tx=0).stop 2024-09-12T02:55:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 shutdown_connections 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557070e537b0 0x557070ec0a30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557070e43d40 0x557070e48700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557070e39480 0x557070e481c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50b8ff9640 0 client.26611 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 --2- 172.21.15.16:0/1517442562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557070e38b60 0x557070e47c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.171+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 >> 172.21.15.16:0/1517442562 conn(0x557070e350f0 msgr2=0x557070e33a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 shutdown_connections 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d9c33740 1 -- 172.21.15.16:0/1517442562 wait complete. 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d9c33740 20 client.26611 trim_cache size 0 max 16384 2024-09-12T02:55:10.178 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:55:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "1f7016ee-af1c-43ed-8d81-5f605e23021c", 2024-09-12T02:55:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:55:10.179 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 msgr2=0x7f50bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 0x7f50bc0a5040 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f50c8030040 tx=0x7f50c803c040 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 msgr2=0x7f50d01a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f50c000e9f0 tx=0x7f50c000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 shutdown_connections 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50d0112ba0 0x7f50d019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50bc0a2b80 0x7f50bc0a5040 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d0074eb0 0x7f50d01a4bf0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 --2- 172.21.15.16:0/4057668475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50d0074590 0x7f50d01a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 >> 172.21.15.16:0/4057668475 conn(0x7f50d00fd8b0 msgr2=0x7f50d00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 shutdown_connections 2024-09-12T02:55:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:10.172+0000 7f50d785a640 1 -- 172.21.15.16:0/4057668475 wait complete. 2024-09-12T02:55:10.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 701113 10.33 MB/sec execute 252 sec latency 39.557 ms 2024-09-12T02:55:10.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:55:11.206 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:11.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: pgmap v555: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:55:11.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:55:11.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:55:11.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:55:11.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:11 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: pgmap v555: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:11 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.709+0000 7fa4d50c8640 1 -- 172.21.15.16:0/4115179863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0101410 msgr2=0x7fa4d010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.709+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/4115179863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0101410 0x7fa4d010d940 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009cd0 tx=0x7fa4c802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 -- 172.21.15.16:0/4115179863 shutdown_connections 2024-09-12T02:55:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/4115179863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0101410 0x7fa4d010d940 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/4115179863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/4115179863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0106a70 0x7fa4d0106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 -- 172.21.15.16:0/4115179863 >> 172.21.15.16:0/4115179863 conn(0x7fa4d00fc790 msgr2=0x7fa4d00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 -- 172.21.15.16:0/4115179863 shutdown_connections 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 -- 172.21.15.16:0/4115179863 wait complete. 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.710+0000 7fa4d50c8640 1 Processor -- start 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 -- start start 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d00fce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0101410 0x7fa4d00fd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 0x7fa4d00fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0110e70 con 0x7fa4d0106a70 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0110cf0 con 0x7fa4d0101410 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4d50c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0110ff0 con 0x7fa4d0100a70 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d00fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 0x7fa4d00fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d00fce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41208/0 (socket says 172.21.15.16:41208) 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1160550017 learned_addr learned my addr 172.21.15.16:0/1160550017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0101410 0x7fa4d00fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 -- 172.21.15.16:0/1160550017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 msgr2=0x7fa4d00fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d00fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 -- 172.21.15.16:0/1160550017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0101410 msgr2=0x7fa4d00fd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0101410 0x7fa4d00fd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.711+0000 7fa4cf59e640 1 -- 172.21.15.16:0/1160550017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d01a6890 con 0x7fa4d0106a70 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4cf59e640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 0x7fa4d00fd8a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802ae00 tx=0x7fa4c8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4b7fff640 1 -- 172.21.15.16:0/1160550017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8004040 con 0x7fa4d0106a70 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4b7fff640 1 -- 172.21.15.16:0/1160550017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c80046c0 con 0x7fa4d0106a70 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d01a6b20 con 0x7fa4d0106a70 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4b7fff640 1 -- 172.21.15.16:0/1160550017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8046480 con 0x7fa4d0106a70 2024-09-12T02:55:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.712+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01a7060 con 0x7fa4d0106a70 2024-09-12T02:55:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.714+0000 7fa4b7fff640 1 -- 172.21.15.16:0/1160550017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa4c80305a0 con 0x7fa4d0106a70 2024-09-12T02:55:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 Processor -- start 2024-09-12T02:55:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 -- start start 2024-09-12T02:55:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cf4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022d01a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b022c5a2e0 con 0x55b022cf44a0 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b022c01ce0 con 0x55b022cff680 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b022c562a0 con 0x55b022cf4dc0 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cf4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41222/0 (socket says 172.21.15.16:41222) 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 -- 172.21.15.16:0/3602883964 learned_addr learned my addr 172.21.15.16:0/3602883964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022d01a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4b7fff640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 0x7fa4b80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 -- 172.21.15.16:0/3602883964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 msgr2=0x55b022d01a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4cf59e640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 0x7fa4b80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022d01a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 -- 172.21.15.16:0/3602883964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 msgr2=0x55b022cf4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cf4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4ced9d640 1 -- 172.21.15.16:0/3602883964 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b022be6440 con 0x55b022cf4dc0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.715+0000 7fa4b7fff640 1 -- 172.21.15.16:0/1160550017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa4c802e030 con 0x7fa4d0106a70 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.716+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00045e0 tx=0x7fa4c0004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2dc76a86475ff66 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.716+0000 7fa4b57fa640 1 -- 172.21.15.16:0/3602883964 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0003dd0 con 0x55b022cf4dc0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.716+0000 7fa4b57fa640 1 -- 172.21.15.16:0/3602883964 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c0015070 con 0x55b022cf4dc0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.716+0000 7fa4b57fa640 1 -- 172.21.15.16:0/3602883964 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0011730 con 0x55b022cf4dc0 2024-09-12T02:55:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- 172.21.15.16:0/3602883964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 msgr2=0x55b022cff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4cf59e640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 0x7fa4b80a5040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8005e30 tx=0x7fa4c80b24a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00045e0 tx=0x7fa4c0004610 comp rx=0 tx=0).stop 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- 172.21.15.16:0/3602883964 shutdown_connections 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022d01a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cff140 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/3602883964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cf4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- 172.21.15.16:0/3602883964 >> 172.21.15.16:0/3602883964 conn(0x55b022cf0a30 msgr2=0x55b022cf0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- 172.21.15.16:0/3602883964 shutdown_connections 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- 172.21.15.16:0/3602883964 wait complete. 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 -- start start 2024-09-12T02:55:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.717+0000 7fa4d74a1740 1 Processor -- start 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cfef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022cfc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b022d02250 con 0x55b022cf44a0 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b022c15050 con 0x55b022cff680 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b022d023d0 con 0x55b022cf4dc0 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4d74a1740 10 client.0 _ll_register_callbacks cb 0x55b022c55160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cfef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41226/0 (socket says 172.21.15.16:41226) 2024-09-12T02:55:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 -- 172.21.15.16:0/765461382 learned_addr learned my addr 172.21.15.16:0/765461382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 -- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 msgr2=0x55b022cfc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022cfc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 -- 172.21.15.16:0/765461382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 msgr2=0x55b022cfef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cfef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.718+0000 7fa4ced9d640 1 -- 172.21.15.16:0/765461382 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0229cb3c0 con 0x55b022cf4dc0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa4c001ae30 tx=0x7fa4c0004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d09d6968da8c318 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c002a060 con 0x55b022cf4dc0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa4c0003de0 con 0x55b022cf4dc0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c0015070 con 0x55b022cf4dc0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa4c000be50 con 0x55b022cf4dc0 2024-09-12T02:55:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4d74a1740 10 client.26623 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b022d0dbf0 con 0x55b022cf4dc0 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c0015070 con 0x55b022cf4dc0 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.719+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c0011a80 con 0x55b022cf4dc0 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 10 client.26623 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 10 client.26623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 20 client.26623 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d11440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 4 client.26623 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b022c67850 con 0x55b022d0f050 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d11440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4b2ff5640 10 client.26623 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.720+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d11440 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f3963aa2497d2f0 server_cookie=c5bda940367207d2 in_seq=0 out_seq=0 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/765461382 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b022c67850 con 0x55b022d0f050 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4b2ff5640 10 client.26623 handle_command_reply: tid=0 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 client.26623 shutdown 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 msgr2=0x55b022cfc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa4c001ae30 tx=0x7fa4c0004320 comp rx=0 tx=0).stop 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 shutdown_connections 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d11440 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cff680 0x55b022cfc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4b2ff5640 0 client.26623 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf4dc0 0x55b022cfc230 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/765461382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022cfef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 >> 172.21.15.16:0/765461382 conn(0x55b022cf0a30 msgr2=0x55b022cf6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 shutdown_connections 2024-09-12T02:55:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 1 -- 172.21.15.16:0/765461382 wait complete. 2024-09-12T02:55:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.721+0000 7fa4d74a1740 20 client.26623 trim_cache size 0 max 16384 2024-09-12T02:55:11.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 703818 10.34 MB/sec execute 253 sec latency 11.957 ms 2024-09-12T02:55:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 Processor -- start 2024-09-12T02:55:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 -- start start 2024-09-12T02:55:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 0x55b022d0b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 0x55b022d16fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01af710 con 0x55b022cff680 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b1250 con 0x55b022cf4dc0 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01afde0 con 0x55b022cf44a0 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 0x55b022d16fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.743+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37174/0 (socket says 172.21.15.16:37174) 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 0x55b022d16fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58670/0 (socket says 172.21.15.16:58670) 2024-09-12T02:55:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1423255017 learned_addr learned my addr 172.21.15.16:0/1423255017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 0x55b022d0b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1423255017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 msgr2=0x55b022d0b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 0x55b022d0b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1423255017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 msgr2=0x55b022d16fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 0x55b022d16fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1423255017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b022cf04d0 con 0x55b022cf4dc0 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 0x55b022d0b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c010 tx=0x7fa4c000b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6be3ef4b5358853d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4b4ff9640 1 -- 172.21.15.16:0/1423255017 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0035070 con 0x55b022cf4dc0 2024-09-12T02:55:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4b4ff9640 1 -- 172.21.15.16:0/1423255017 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c0015040 con 0x55b022cf4dc0 2024-09-12T02:55:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.744+0000 7fa4b4ff9640 1 -- 172.21.15.16:0/1423255017 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c002a070 con 0x55b022cf4dc0 2024-09-12T02:55:11.751 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: pgmap v555: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-09-12T02:55:11.751 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:55:11.752 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.752 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:55:11.752 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:55:11.752 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.752 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:11 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:55:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1423255017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 msgr2=0x55b022d16a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c010 tx=0x7fa4c000b630 comp rx=0 tx=0).stop 2024-09-12T02:55:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1423255017 shutdown_connections 2024-09-12T02:55:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cff680 0x55b022d16fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d16a70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1423255017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cf44a0 0x55b022d0b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1423255017 >> 172.21.15.16:0/1423255017 conn(0x55b022cf0a30 msgr2=0x55b022cfb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1423255017 shutdown_connections 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.745+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1423255017 wait complete. 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 -- start start 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 Processor -- start 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022d71e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 0x55b022d04d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b0b80 con 0x55b022cf44a0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b04b0 con 0x55b022cf4dc0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01ad6f0 con 0x55b022cff680 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4d74a1740 10 client.0 _ll_register_callbacks cb 0x7fa4c001cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022d71e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37182/0 (socket says 172.21.15.16:37182) 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022d71e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58680/0 (socket says 172.21.15.16:58680) 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1997969774 learned_addr learned my addr 172.21.15.16:0/1997969774 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.746+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 0x55b022d04d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1997969774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 msgr2=0x55b022d04d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 0x55b022d04d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1997969774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 msgr2=0x55b022d71e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022d71e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 -- 172.21.15.16:0/1997969774 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b022d053f0 con 0x55b022cf4dc0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 0x55b022d04d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4ced9d640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0011f70 tx=0x7fa4c000b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c96fbf14693305a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0035070 con 0x55b022cf4dc0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c0015040 con 0x55b022cf4dc0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa4c0010930 con 0x55b022cf4dc0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa4c002a040 con 0x55b022cf4dc0 2024-09-12T02:55:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4d74a1740 10 client.26738 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.747+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b022d79640 con 0x55b022cf4dc0 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c0015040 con 0x55b022cf4dc0 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c001a470 con 0x55b022cf4dc0 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 10 client.26738 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 10 client.26738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 20 client.26738 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d7bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 4 client.26738 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b022c61d60 con 0x55b022d0f050 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.748+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d7bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4ce59c640 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d7bcc0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90ebf9e0a550f47 server_cookie=13bd69ffa1feb85 in_seq=0 out_seq=0 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4b2ff5640 10 client.26738 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4b2ff5640 1 -- 172.21.15.16:0/1997969774 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b022c61d60 con 0x55b022d0f050 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4b2ff5640 10 client.26738 handle_command_reply: tid=0 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 client.26738 shutdown 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 msgr2=0x55b022d72360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0011f70 tx=0x7fa4c000b4a0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 shutdown_connections 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b022d0f050 0x55b022d7bcc0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b022cff680 0x55b022d04d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b022cf4dc0 0x55b022d72360 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 --2- 172.21.15.16:0/1997969774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b022cf44a0 0x55b022d71e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 >> 172.21.15.16:0/1997969774 conn(0x55b022cf0a30 msgr2=0x55b022cfda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4b2ff5640 0 client.26738 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 shutdown_connections 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 1 -- 172.21.15.16:0/1997969774 wait complete. 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.749+0000 7fa4d74a1740 20 client.26738 trim_cache size 0 max 16384 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:55:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "1f7016ee-af1c-43ed-8d81-5f605e23021c": { 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "1f7016ee-af1c-43ed-8d81-5f605e23021c", 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:11.762 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 msgr2=0x7fa4b80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 0x7fa4b80a5040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8005e30 tx=0x7fa4c80b24a0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 msgr2=0x7fa4d00fd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 0x7fa4d00fd8a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802ae00 tx=0x7fa4c8003970 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 shutdown_connections 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4b80a2b80 0x7fa4b80a5040 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0106a70 0x7fa4d00fd8a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa4d0101410 0x7fa4d00fd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 --2- 172.21.15.16:0/1160550017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4d0100a70 0x7fa4d00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 >> 172.21.15.16:0/1160550017 conn(0x7fa4d00fc790 msgr2=0x7fa4d006a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:11.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 shutdown_connections 2024-09-12T02:55:11.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:11.750+0000 7fa4d50c8640 1 -- 172.21.15.16:0/1160550017 wait complete. 2024-09-12T02:55:12.566 INFO:tasks.cephfs.filesystem:scrub status for tag:1f7016ee-af1c-43ed-8d81-5f605e23021c - {'path': '/', 'tag': '1f7016ee-af1c-43ed-8d81-5f605e23021c', 'options': 'recursive,force'} 2024-09-12T02:55:12.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 706918 10.38 MB/sec execute 254 sec latency 21.455 ms 2024-09-12T02:55:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:13 smithi184 ceph-mon[35321]: pgmap v556: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-09-12T02:55:13.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:13 smithi016 ceph-mon[30925]: pgmap v556: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-09-12T02:55:13.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 709588 10.39 MB/sec execute 255 sec latency 22.395 ms 2024-09-12T02:55:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:13 smithi033 ceph-mon[35092]: pgmap v556: 97 pgs: 97 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-09-12T02:55:14.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 712050 10.39 MB/sec execute 256 sec latency 12.155 ms 2024-09-12T02:55:15.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:15 smithi184 ceph-mon[35321]: pgmap v557: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-09-12T02:55:15.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:15 smithi016 ceph-mon[30925]: pgmap v557: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-09-12T02:55:15.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 714688 10.41 MB/sec execute 257 sec latency 12.150 ms 2024-09-12T02:55:15.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:15 smithi033 ceph-mon[35092]: pgmap v557: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-09-12T02:55:16.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 717780 10.44 MB/sec execute 258 sec latency 9.335 ms 2024-09-12T02:55:17.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:17 smithi184 ceph-mon[35321]: pgmap v558: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:55:17.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:17 smithi016 ceph-mon[30925]: pgmap v558: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:55:17.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 721080 10.47 MB/sec execute 259 sec latency 11.601 ms 2024-09-12T02:55:17.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:17 smithi033 ceph-mon[35092]: pgmap v558: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:55:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:55:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:55:18.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:55:18.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:55:18.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 723843 10.48 MB/sec execute 260 sec latency 12.770 ms 2024-09-12T02:55:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:55:18.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:55:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:19 smithi184 ceph-mon[35321]: pgmap v559: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-12T02:55:19.647 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:19 smithi033 ceph-mon[35092]: pgmap v559: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-12T02:55:19.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:19 smithi016 ceph-mon[30925]: pgmap v559: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 255 op/s 2024-09-12T02:55:19.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 726230 10.49 MB/sec execute 261 sec latency 15.934 ms 2024-09-12T02:55:20.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 728926 10.50 MB/sec execute 262 sec latency 11.333 ms 2024-09-12T02:55:21.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:21 smithi184 ceph-mon[35321]: pgmap v560: 97 pgs: 97 active+clean; 265 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 305 op/s 2024-09-12T02:55:21.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:21 smithi016 ceph-mon[30925]: pgmap v560: 97 pgs: 97 active+clean; 265 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 305 op/s 2024-09-12T02:55:21.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 732039 10.53 MB/sec execute 263 sec latency 10.337 ms 2024-09-12T02:55:21.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:21 smithi033 ceph-mon[35092]: pgmap v560: 97 pgs: 97 active+clean; 265 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 305 op/s 2024-09-12T02:55:22.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 735269 10.56 MB/sec execute 264 sec latency 13.779 ms 2024-09-12T02:55:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:23 smithi184 ceph-mon[35321]: pgmap v561: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 240 op/s 2024-09-12T02:55:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:23 smithi016 ceph-mon[30925]: pgmap v561: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 240 op/s 2024-09-12T02:55:23.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 738031 10.57 MB/sec execute 265 sec latency 13.137 ms 2024-09-12T02:55:23.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:23 smithi033 ceph-mon[35092]: pgmap v561: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 240 op/s 2024-09-12T02:55:24.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 740767 10.58 MB/sec execute 266 sec latency 14.546 ms 2024-09-12T02:55:25.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:25 smithi184 ceph-mon[35321]: pgmap v562: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-09-12T02:55:25.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:25 smithi016 ceph-mon[30925]: pgmap v562: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-09-12T02:55:25.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 743197 10.60 MB/sec execute 267 sec latency 27.207 ms 2024-09-12T02:55:25.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:25 smithi033 ceph-mon[35092]: pgmap v562: 97 pgs: 97 active+clean; 268 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-09-12T02:55:26.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 746233 10.62 MB/sec execute 268 sec latency 9.915 ms 2024-09-12T02:55:26.973 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.8) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T02:55:26.973 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.4 2024-09-12T02:55:26.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:55:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:27.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:27 smithi016 ceph-mon[30925]: pgmap v563: 97 pgs: 97 active+clean; 273 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 280 op/s 2024-09-12T02:55:27.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:27 smithi184 ceph-mon[35321]: pgmap v563: 97 pgs: 97 active+clean; 273 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 280 op/s 2024-09-12T02:55:27.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 749218 10.65 MB/sec execute 269 sec latency 20.543 ms 2024-09-12T02:55:27.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:27 smithi033 ceph-mon[35092]: pgmap v563: 97 pgs: 97 active+clean; 273 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 280 op/s 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 -- 172.21.15.16:0/3675958560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102570 msgr2=0x7f7c481029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3675958560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102570 0x7f7c481029b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009900 tx=0x7f7c3802a480 comp rx=0 tx=0).stop 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 -- 172.21.15.16:0/3675958560 shutdown_connections 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3675958560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102ef0 0x7f7c4810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3675958560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102570 0x7f7c481029b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3675958560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.786+0000 7f7c506a6640 1 -- 172.21.15.16:0/3675958560 >> 172.21.15.16:0/3675958560 conn(0x7f7c480fc7d0 msgr2=0x7f7c480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.787+0000 7f7c506a6640 1 -- 172.21.15.16:0/3675958560 shutdown_connections 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.787+0000 7f7c506a6640 1 -- 172.21.15.16:0/3675958560 wait complete. 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.787+0000 7f7c506a6640 1 Processor -- start 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.787+0000 7f7c506a6640 1 -- start start 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48110f80 con 0x7f7c48102ef0 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48110e00 con 0x7f7c48102570 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c506a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48111100 con 0x7f7c48100e20 2024-09-12T02:55:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4e442640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4dc41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50976/0 (socket says 172.21.15.16:50976) 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4dc41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54188/0 (socket says 172.21.15.16:54188) 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 -- 172.21.15.16:0/3946161012 learned_addr learned my addr 172.21.15.16:0/3946161012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 -- 172.21.15.16:0/3946161012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 msgr2=0x7f7c48197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 -- 172.21.15.16:0/3946161012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 msgr2=0x7f7c481976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 -- 172.21.15.16:0/3946161012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c481a69d0 con 0x7f7c48102ef0 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4e442640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4dc41640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.788+0000 7f7c4ec43640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000eb10 tx=0x7f7c4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.789+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40004b20 con 0x7f7c48102ef0 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.789+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c481a6cc0 con 0x7f7c48102ef0 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.789+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c40004cc0 con 0x7f7c48102ef0 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.789+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40010760 con 0x7f7c48102ef0 2024-09-12T02:55:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.789+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481a71d0 con 0x7f7c48102ef0 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.790+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c48101200 con 0x7f7c48102ef0 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.792+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7c40010900 con 0x7f7c48102ef0 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.793+0000 7f7c3f7fe640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 0x7f7c340a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.793+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c400cf740 con 0x7f7c48102ef0 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.793+0000 7f7c4e442640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 0x7f7c340a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.793+0000 7f7c4e442640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 0x7f7c340a50c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fc0 tx=0x7f7c44005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:27.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:27.796+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7c400541b0 con 0x7f7c48102ef0 2024-09-12T02:55:28.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.054+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c4806a5a0 con 0x7f7c48102ef0 2024-09-12T02:55:28.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.055+0000 7f7c3f7fe640 1 -- 172.21.15.16:0/3946161012 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f7c4806a5a0 con 0x7f7c48102ef0 2024-09-12T02:55:28.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:55:28.057 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:55:28.058 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 msgr2=0x7f7c340a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 0x7f7c340a50c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fc0 tx=0x7f7c44005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 msgr2=0x7f7c48197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000eb10 tx=0x7f7c4000c2d0 comp rx=0 tx=0).stop 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 shutdown_connections 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c340a2c00 0x7f7c340a50c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c48102ef0 0x7f7c48197be0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:28.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c48102570 0x7f7c481976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:28.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 --2- 172.21.15.16:0/3946161012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c48100e20 0x7f7c48197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:28.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.057+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 >> 172.21.15.16:0/3946161012 conn(0x7f7c480fc7d0 msgr2=0x7f7c480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:28.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.058+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 shutdown_connections 2024-09-12T02:55:28.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:28.058+0000 7f7c506a6640 1 -- 172.21.15.16:0/3946161012 wait complete. 2024-09-12T02:55:28.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:28 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3946161012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:28 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3946161012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:28.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 751517 10.64 MB/sec execute 270 sec latency 24.946 ms 2024-09-12T02:55:28.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:28 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3946161012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:28.822 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 139.4 --await-for 45.0' 2024-09-12T02:55:28.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 139.4 --await-for 45.0 2024-09-12T02:55:29.103 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:29.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:29 smithi016 ceph-mon[30925]: pgmap v564: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 205 op/s 2024-09-12T02:55:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:29 smithi184 ceph-mon[35321]: pgmap v564: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 205 op/s 2024-09-12T02:55:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/3421550719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b00745c0 msgr2=0x7fd7b00749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/3421550719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b00745c0 0x7fd7b00749a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009860 tx=0x7fd7a002a3d0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/3421550719 shutdown_connections 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/3421550719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b010e430 0x7fd7b0110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/3421550719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b0074ee0 0x7fd7b010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/3421550719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b00745c0 0x7fd7b00749a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/3421550719 >> 172.21.15.16:0/3421550719 conn(0x7fd7b00fd8f0 msgr2=0x7fd7b00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/3421550719 shutdown_connections 2024-09-12T02:55:29.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.636+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/3421550719 wait complete. 2024-09-12T02:55:29.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.637+0000 7fd7b4b1f640 1 Processor -- start 2024-09-12T02:55:29.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.637+0000 7fd7b4b1f640 1 -- start start 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 0x7fd7b019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 0x7fd7b019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01141d0 con 0x7fd7b0074ee0 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0114050 con 0x7fd7b010e430 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7b4b1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0114350 con 0x7fd7b00745c0 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 0x7fd7b019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48622/0 (socket says 172.21.15.16:48622) 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 -- 172.21.15.16:0/2213299452 learned_addr learned my addr 172.21.15.16:0/2213299452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 0x7fd7b019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 -- 172.21.15.16:0/2213299452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 msgr2=0x7fd7b019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 0x7fd7b019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 -- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 msgr2=0x7fd7b019acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 0x7fd7b019acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7ae59c640 1 -- 172.21.15.16:0/2213299452 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b019e6e0 con 0x7fd7b00745c0 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 0x7fd7b019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.638+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 0x7fd7b019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7ae59c640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009980 tx=0x7fd7a00309a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7a77fe640 1 -- 172.21.15.16:0/2213299452 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0040070 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b019e970 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7a77fe640 1 -- 172.21.15.16:0/2213299452 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a0005620 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7a77fe640 1 -- 172.21.15.16:0/2213299452 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00049b0 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.639+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b01a7340 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.641+0000 7fd7a77fe640 1 -- 172.21.15.16:0/2213299452 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd7a0043070 con 0x7fd7b00745c0 2024-09-12T02:55:29.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.641+0000 7fd7b6ef8740 1 Processor -- start 2024-09-12T02:55:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 -- start start 2024-09-12T02:55:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f719400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 0x55d02f71bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d02f6742a0 con 0x55d02f70f080 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d02f61bb10 con 0x55d02f719940 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d02f670260 con 0x55d02f70e760 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7ae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f719400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48624/0 (socket says 172.21.15.16:48624) 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7aed9d640 1 -- 172.21.15.16:0/2048744729 learned_addr learned my addr 172.21.15.16:0/2048744729 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.642+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 0x55d02f71bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 -- 172.21.15.16:0/2048744729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 msgr2=0x55d02f71bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 0x55d02f71bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 -- 172.21.15.16:0/2048744729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 msgr2=0x55d02f719400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f719400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 -- 172.21.15.16:0/2048744729 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02f3e5b90 con 0x55d02f70e760 2024-09-12T02:55:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a77fe640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 0x7fd79c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 0x55d02f71bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7ae59c640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 0x7fd79c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800e850 tx=0x7fd7a80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8b24b94003a559b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a77fe640 1 -- 172.21.15.16:0/2213299452 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd7a00efd90 con 0x7fd7b00745c0 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a4ff9640 1 -- 172.21.15.16:0/2048744729 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a801d070 con 0x55d02f70e760 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a4ff9640 1 -- 172.21.15.16:0/2048744729 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8003e50 con 0x55d02f70e760 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.643+0000 7fd7a4ff9640 1 -- 172.21.15.16:0/2048744729 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8018480 con 0x55d02f70e760 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2048744729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 msgr2=0x55d02f70eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7ae59c640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 0x7fd79c0a5040 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0031c50 tx=0x7fd7a00f37d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800e850 tx=0x7fd7a80095c0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2048744729 shutdown_connections 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f719940 0x55d02f71bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f719400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2048744729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f70eb40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2048744729 >> 172.21.15.16:0/2048744729 conn(0x55d02f70acf0 msgr2=0x55d02f70b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2048744729 shutdown_connections 2024-09-12T02:55:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2048744729 wait complete. 2024-09-12T02:55:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.644+0000 7fd7b6ef8740 1 -- start start 2024-09-12T02:55:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 Processor -- start 2024-09-12T02:55:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f717300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d02f71c510 con 0x55d02f70f080 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d02f5fc6e0 con 0x55d02f70e760 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d02f71c690 con 0x55d02f719940 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7b6ef8740 10 client.0 _ll_register_callbacks cb 0x55d02f620a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54228/0 (socket says 172.21.15.16:54228) 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1636777208 learned_addr learned my addr 172.21.15.16:0/1636777208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51018/0 (socket says 172.21.15.16:51018) 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f717300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1636777208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 msgr2=0x55d02f717300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f717300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1636777208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 msgr2=0x55d02f716dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1636777208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02f725a50 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.645+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd7a801b040 tx=0x7fd7a8004510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0ae5c1c6b8c4a39 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a802d060 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd7a8003970 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7a8003e50 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd7a801d030 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd7b6ef8740 10 client.26756 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d02f7283e0 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8003e50 con 0x55d02f70e760 2024-09-12T02:55:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.646+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd7a800b040 con 0x55d02f70e760 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 10 client.26756 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 10 client.26756 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 20 client.26756 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f7297f0 0x55d02f72bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 4 client.26756 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d02f6819d0 con 0x55d02f7297f0 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f7297f0 0x55d02f72bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd795ffb640 10 client.26756 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.647+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f7297f0 0x55d02f72bbe0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfb4bba62647e796 server_cookie=ad3b13c9ce36d546 in_seq=0 out_seq=0 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd795ffb640 1 -- 172.21.15.16:0/1636777208 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d02f6819d0 con 0x55d02f7297f0 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd795ffb640 10 client.26756 handle_command_reply: tid=0 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 client.26756 shutdown 2024-09-12T02:55:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 msgr2=0x55d02f716880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd7a801b040 tx=0x7fd7a8004510 comp rx=0 tx=0).stop 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 shutdown_connections 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f7297f0 0x55d02f72bbe0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f717300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f716dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1636777208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f716880 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 >> 172.21.15.16:0/1636777208 conn(0x55d02f70acf0 msgr2=0x55d02f710a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd795ffb640 0 client.26756 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 shutdown_connections 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1636777208 wait complete. 2024-09-12T02:55:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.648+0000 7fd7b6ef8740 20 client.26756 trim_cache size 0 max 16384 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 Processor -- start 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 -- start start 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f7157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01af830 con 0x55d02f719940 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01b1370 con 0x55d02f70f080 2024-09-12T02:55:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01aff00 con 0x55d02f70e760 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51024/0 (socket says 172.21.15.16:51024) 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54238/0 (socket says 172.21.15.16:54238) 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1365784188 learned_addr learned my addr 172.21.15.16:0/1365784188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f7157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1365784188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 msgr2=0x55d02f7157c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f7157c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1365784188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 msgr2=0x55d02f7311b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.668+0000 7fd7aed9d640 1 -- 172.21.15.16:0/1365784188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02f70eeb0 con 0x55d02f719940 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.669+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.669+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009ba0 tx=0x7fd7a8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc263ae8934e428 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.669+0000 7fd797fff640 1 -- 172.21.15.16:0/1365784188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8025690 con 0x55d02f719940 2024-09-12T02:55:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.669+0000 7fd797fff640 1 -- 172.21.15.16:0/1365784188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a801e040 con 0x55d02f719940 2024-09-12T02:55:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.669+0000 7fd797fff640 1 -- 172.21.15.16:0/1365784188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8003bd0 con 0x55d02f719940 2024-09-12T02:55:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1365784188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 msgr2=0x55d02f7316f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009ba0 tx=0x7fd7a8000f40 comp rx=0 tx=0).stop 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1365784188 shutdown_connections 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f719940 0x55d02f7316f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70f080 0x55d02f7311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/1365784188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f70e760 0x55d02f7157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1365784188 >> 172.21.15.16:0/1365784188 conn(0x55d02f70acf0 msgr2=0x55d02f709e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1365784188 shutdown_connections 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/1365784188 wait complete. 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 -- start start 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.670+0000 7fd7b6ef8740 1 Processor -- start 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f71d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f71db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f71e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01b0ca0 con 0x55d02f70f080 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01b05d0 con 0x55d02f70e760 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b01ad810 con 0x55d02f719940 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f71d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f71db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7b6ef8740 10 client.0 _ll_register_callbacks cb 0x7fd7a800dc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7aed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f71d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54250/0 (socket says 172.21.15.16:54250) 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7aed9d640 1 -- 172.21.15.16:0/2433924851 learned_addr learned my addr 172.21.15.16:0/2433924851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f71e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 -- 172.21.15.16:0/2433924851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 msgr2=0x55d02f71e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f71e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 -- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 msgr2=0x55d02f71d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f71d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.671+0000 7fd7a7fff640 1 -- 172.21.15.16:0/2433924851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02f79b9c0 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f71e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd7a7fff640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f71db90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x55d02f716110 tx=0x7fd798012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3215ebe3320c0f4a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798020ab0 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd798020c50 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7a801e040 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd798020e50 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd7b6ef8740 10 client.17247 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d02f79ccc0 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.672+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a801e040 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd7980295a0 con 0x55d02f70f080 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 10 client.17247 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 10 client.17247 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 20 client.17247 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f79d450 0x55d02f79f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 4 client.17247 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.4, "await_for": 45.0}] 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.4, "await_for": 45.0}) -- 0x55d02f67bd20 con 0x55d02f79d450 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f79d450 0x55d02f79f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd795ffb640 10 client.17247 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.673+0000 7fd7aed9d640 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f79d450 0x55d02f79f840 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2907562c1d381a1a server_cookie=d1e0fc134c0a3fe9 in_seq=0 out_seq=0 2024-09-12T02:55:29.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753596 10.65 MB/sec execute 271 sec latency 35.236 ms 2024-09-12T02:55:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.729+0000 7fd795ffb640 1 -- 172.21.15.16:0/2433924851 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d02f67bd20 con 0x55d02f79d450 2024-09-12T02:55:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.729+0000 7fd795ffb640 10 client.17247 handle_command_reply: tid=0 2024-09-12T02:55:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.729+0000 7fd7b6ef8740 1 client.17247 shutdown 2024-09-12T02:55:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 msgr2=0x55d02f71db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f71db90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x55d02f716110 tx=0x7fd798012da0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 shutdown_connections 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d02f79d450 0x55d02f79f840 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d02f719940 0x55d02f71e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d02f70f080 0x55d02f71db90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd795ffb640 0 client.17247 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 --2- 172.21.15.16:0/2433924851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d02f70e760 0x55d02f71d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 >> 172.21.15.16:0/2433924851 conn(0x55d02f70acf0 msgr2=0x55d02f7101b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 shutdown_connections 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 1 -- 172.21.15.16:0/2433924851 wait complete. 2024-09-12T02:55:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.730+0000 7fd7b6ef8740 20 client.17247 trim_cache size 0 max 16384 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 30, 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "ca0e83a6": { 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 30, 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:29.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.4, 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:29.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 msgr2=0x7fd79c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 0x7fd79c0a5040 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0031c50 tx=0x7fd7a00f37d0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 msgr2=0x7fd7b019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009980 tx=0x7fd7a00309a0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 shutdown_connections 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd7b010e430 0x7fd7b019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd79c0a2b80 0x7fd79c0a5040 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7b0074ee0 0x7fd7b019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 --2- 172.21.15.16:0/2213299452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7b00745c0 0x7fd7b019a790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 >> 172.21.15.16:0/2213299452 conn(0x7fd7b00fd8f0 msgr2=0x7fd7b00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 shutdown_connections 2024-09-12T02:55:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:29.731+0000 7fd7b4b1f640 1 -- 172.21.15.16:0/2213299452 wait complete. 2024-09-12T02:55:29.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:29 smithi033 ceph-mon[35092]: pgmap v564: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 205 op/s 2024-09-12T02:55:30.489 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ca0e83a6', quiesce took 3.5 seconds. Will release after: 15.899999999999999 2024-09-12T02:55:30.726 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.61 MB/sec execute 272 sec latency 993.836 ms 2024-09-12T02:55:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:31 smithi184 ceph-mon[35321]: pgmap v565: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-09-12T02:55:31.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:31 smithi016 ceph-mon[30925]: pgmap v565: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-09-12T02:55:31.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.57 MB/sec execute 273 sec latency 1993.954 ms 2024-09-12T02:55:31.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:31 smithi033 ceph-mon[35092]: pgmap v565: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-09-12T02:55:32.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.53 MB/sec execute 274 sec latency 2994.056 ms 2024-09-12T02:55:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:33 smithi184 ceph-mon[35321]: pgmap v566: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-09-12T02:55:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:33 smithi016 ceph-mon[30925]: pgmap v566: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-09-12T02:55:33.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.49 MB/sec execute 275 sec latency 3994.169 ms 2024-09-12T02:55:33.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:33 smithi033 ceph-mon[35092]: pgmap v566: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-09-12T02:55:34.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.45 MB/sec execute 276 sec latency 4994.219 ms 2024-09-12T02:55:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:35 smithi184 ceph-mon[35321]: pgmap v567: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-12T02:55:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:35 smithi016 ceph-mon[30925]: pgmap v567: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-12T02:55:35.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.41 MB/sec execute 277 sec latency 5994.266 ms 2024-09-12T02:55:35.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:35 smithi033 ceph-mon[35092]: pgmap v567: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-12T02:55:36.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.38 MB/sec execute 278 sec latency 6994.359 ms 2024-09-12T02:55:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:37 smithi184 ceph-mon[35321]: pgmap v568: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-09-12T02:55:37.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:37 smithi016 ceph-mon[30925]: pgmap v568: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-09-12T02:55:37.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.34 MB/sec execute 279 sec latency 7994.414 ms 2024-09-12T02:55:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:37 smithi033 ceph-mon[35092]: pgmap v568: 97 pgs: 97 active+clean; 281 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-09-12T02:55:38.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.30 MB/sec execute 280 sec latency 8994.521 ms 2024-09-12T02:55:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:39 smithi184 ceph-mon[35321]: pgmap v569: 97 pgs: 97 active+clean; 280 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-12T02:55:39.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:39 smithi016 ceph-mon[30925]: pgmap v569: 97 pgs: 97 active+clean; 280 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-12T02:55:39.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.27 MB/sec execute 281 sec latency 9994.626 ms 2024-09-12T02:55:39.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:39 smithi033 ceph-mon[35092]: pgmap v569: 97 pgs: 97 active+clean; 280 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-12T02:55:40.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.23 MB/sec execute 282 sec latency 10994.745 ms 2024-09-12T02:55:41.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:41 smithi016 ceph-mon[30925]: pgmap v570: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-09-12T02:55:41.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.19 MB/sec execute 283 sec latency 11994.872 ms 2024-09-12T02:55:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:41 smithi033 ceph-mon[35092]: pgmap v570: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-09-12T02:55:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:41 smithi184 ceph-mon[35321]: pgmap v570: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-09-12T02:55:42.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:55:42.727 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.16 MB/sec execute 284 sec latency 12994.924 ms 2024-09-12T02:55:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- 172.21.15.16:0/3094135051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0106f10 msgr2=0x7f28a010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 --2- 172.21.15.16:0/3094135051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0106f10 0x7f28a010ea90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2898009900 tx=0x7f289802a410 comp rx=0 tx=0).stop 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- 172.21.15.16:0/3094135051 shutdown_connections 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 --2- 172.21.15.16:0/3094135051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0106f10 0x7f28a010ea90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 --2- 172.21.15.16:0/3094135051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0102790 0x7f28a01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 --2- 172.21.15.16:0/3094135051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0101de0 0x7f28a01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- 172.21.15.16:0/3094135051 >> 172.21.15.16:0/3094135051 conn(0x7f28a00fd8c0 msgr2=0x7f28a00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- 172.21.15.16:0/3094135051 shutdown_connections 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- 172.21.15.16:0/3094135051 wait complete. 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 Processor -- start 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.306+0000 7f28a7892640 1 -- start start 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 0x7f28a0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0106f10 0x7f28a0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0111f70 con 0x7f28a0101de0 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0111df0 con 0x7f28a0102790 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a7892640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01120f0 con 0x7f28a0106f10 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55692/0 (socket says 172.21.15.16:55692) 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 -- 172.21.15.16:0/222005309 learned_addr learned my addr 172.21.15.16:0/222005309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 0x7f28a0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0106f10 0x7f28a0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 -- 172.21.15.16:0/222005309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0106f10 msgr2=0x7f28a0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0106f10 0x7f28a0074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 -- 172.21.15.16:0/222005309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 msgr2=0x7f28a0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 0x7f28a0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 -- 172.21.15.16:0/222005309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a01a68b0 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 0x7f28a0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28a562e640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f289c00eab0 tx=0x7f289c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28967fc640 1 -- 172.21.15.16:0/222005309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c00ce40 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.307+0000 7f28967fc640 1 -- 172.21.15.16:0/222005309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f289c011950 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.308+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01a6ba0 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.308+0000 7f28967fc640 1 -- 172.21.15.16:0/222005309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c010ae0 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.308+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a01a7130 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.310+0000 7f28967fc640 1 -- 172.21.15.16:0/222005309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f289c011b00 con 0x7f28a0101de0 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.310+0000 7f28a9c6b740 1 Processor -- start 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.310+0000 7f28a9c6b740 1 -- start start 2024-09-12T02:55:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfd3680 0x563aecfd5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563aecf2e2e0 con 0x563aecfc84a0 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563aeced5ce0 con 0x563aecfc8dc0 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563aecf2a2a0 con 0x563aecfd3680 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55708/0 (socket says 172.21.15.16:55708) 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 -- 172.21.15.16:0/147687535 learned_addr learned my addr 172.21.15.16:0/147687535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a562e640 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfd3680 0x563aecfd5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28967fc640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 0x7f288c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 -- 172.21.15.16:0/147687535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfd3680 msgr2=0x563aecfd5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfd3680 0x563aecfd5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 -- 172.21.15.16:0/147687535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 msgr2=0x563aecfd3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 -- 172.21.15.16:0/147687535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aeceba440 con 0x563aecfc84a0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28967fc640 1 -- 172.21.15.16:0/222005309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f289c0cf7c0 con 0x7f28a0101de0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.311+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2898005ab0 tx=0x7f2898032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81bdd935a7405b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.312+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 0x7f288c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.312+0000 7f288b7fe640 1 -- 172.21.15.16:0/147687535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289803f030 con 0x563aecfc84a0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.312+0000 7f288b7fe640 1 -- 172.21.15.16:0/147687535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2898031890 con 0x563aecfc84a0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.312+0000 7f288b7fe640 1 -- 172.21.15.16:0/147687535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898035070 con 0x563aecfc84a0 2024-09-12T02:55:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- 172.21.15.16:0/147687535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 msgr2=0x563aecfc8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 0x7f288c0a50c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2898042000 tx=0x7f2898004af0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2898005ab0 tx=0x7f2898032c30 comp rx=0 tx=0).stop 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- 172.21.15.16:0/147687535 shutdown_connections 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfd3680 0x563aecfd5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/147687535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc84a0 0x563aecfc8880 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- 172.21.15.16:0/147687535 >> 172.21.15.16:0/147687535 conn(0x563aecfc4a30 msgr2=0x563aecfc4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- 172.21.15.16:0/147687535 shutdown_connections 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- 172.21.15.16:0/147687535 wait complete. 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 -- start start 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.313+0000 7f28a9c6b740 1 Processor -- start 2024-09-12T02:55:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfd2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563aecfd6250 con 0x563aecfd3680 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563aecee9050 con 0x563aecfc8dc0 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563aecfd63d0 con 0x563aecfc84a0 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 10 client.0 _ll_register_callbacks cb 0x563aecf29160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55720/0 (socket says 172.21.15.16:55720) 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 -- 172.21.15.16:0/1433267853 learned_addr learned my addr 172.21.15.16:0/1433267853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a562e640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfd2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 -- 172.21.15.16:0/1433267853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 msgr2=0x563aecfd2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfd2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 -- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 msgr2=0x563aecfd0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 -- 172.21.15.16:0/1433267853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aecc9f3c0 con 0x563aecfd3680 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f28980056c0 tx=0x7f2898043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fd5c6119841435 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f2888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289803f040 con 0x563aecfd3680 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f28980046c0 con 0x563aecfd3680 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.314+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2898031890 con 0x563aecfd3680 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2898009520 con 0x563aecfd3680 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 10 client.17259 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563aecfe1b30 con 0x563aecfd3680 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2898031890 con 0x563aecfd3680 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2898059c30 con 0x563aecfd3680 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 10 client.17259 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 20 client.17259 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aecfe2e80 0x563aecfe5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 4 client.17259 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aecf3b850 con 0x563aecfe2e80 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.315+0000 7f28a562e640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aecfe2e80 0x563aecfe5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.316+0000 7f28a562e640 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aecfe2e80 0x563aecfe5270 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45e7d1375512ed33 server_cookie=d52f8ee3f5e16512 in_seq=0 out_seq=0 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.316+0000 7f2888ff9640 10 client.17259 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.316+0000 7f2888ff9640 1 -- 172.21.15.16:0/1433267853 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563aecf3b850 con 0x563aecfe2e80 2024-09-12T02:55:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.316+0000 7f2888ff9640 10 client.17259 handle_command_reply: tid=0 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.316+0000 7f28a9c6b740 1 client.17259 shutdown 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 msgr2=0x563aecfd06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f28980056c0 tx=0x7f2898043000 comp rx=0 tx=0).stop 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 shutdown_connections 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aecfe2e80 0x563aecfe5270 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfd3680 0x563aecfd06a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfc8dc0 0x563aecfd0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1433267853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfd2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 >> 172.21.15.16:0/1433267853 conn(0x563aecfc4a30 msgr2=0x563aecfc3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f2888ff9640 0 client.17259 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 shutdown_connections 2024-09-12T02:55:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1433267853 wait complete. 2024-09-12T02:55:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.317+0000 7f28a9c6b740 20 client.17259 trim_cache size 0 max 16384 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 Processor -- start 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 -- start start 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfe4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfea920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01af6d0 con 0x563aecfc8dc0 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b1210 con 0x563aecfd3680 2024-09-12T02:55:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01afda0 con 0x563aecfc84a0 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55732/0 (socket says 172.21.15.16:55732) 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 -- 172.21.15.16:0/1124784706 learned_addr learned my addr 172.21.15.16:0/1124784706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a562e640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfe4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 -- 172.21.15.16:0/1124784706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 msgr2=0x563aecfe4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfe4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 -- 172.21.15.16:0/1124784706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 msgr2=0x563aecfea920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfea920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.335+0000 7f28a4e2d640 1 -- 172.21.15.16:0/1124784706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aecfcf110 con 0x563aecfc8dc0 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f28a562e640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfe4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f28a0075f80 tx=0x7f28900127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f54667c3ee50d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f288affd640 1 -- 172.21.15.16:0/1124784706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890004020 con 0x563aecfc8dc0 2024-09-12T02:55:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f288affd640 1 -- 172.21.15.16:0/1124784706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2890005500 con 0x563aecfc8dc0 2024-09-12T02:55:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.336+0000 7f288affd640 1 -- 172.21.15.16:0/1124784706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890004e40 con 0x563aecfc8dc0 2024-09-12T02:55:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1124784706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 msgr2=0x563aecfd20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f28a0075f80 tx=0x7f28900127c0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1124784706 shutdown_connections 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd20d0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/1124784706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfe4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1124784706 >> 172.21.15.16:0/1124784706 conn(0x563aecfc4a30 msgr2=0x563aecfd19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1124784706 shutdown_connections 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- 172.21.15.16:0/1124784706 wait complete. 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 -- start start 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 Processor -- start 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.337+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfdbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfd95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b0b40 con 0x563aecfc8dc0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b0470 con 0x563aecfd3680 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01ad6b0 con 0x563aecfc84a0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 10 client.0 _ll_register_callbacks cb 0x7f2898005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55748/0 (socket says 172.21.15.16:55748) 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 -- 172.21.15.16:0/3433184721 learned_addr learned my addr 172.21.15.16:0/3433184721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a5e2f640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfd95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a562e640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfdbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 -- 172.21.15.16:0/3433184721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 msgr2=0x563aecfdbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfdbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 -- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 msgr2=0x563aecfd95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfd95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 -- 172.21.15.16:0/3433184721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aecfd9dc0 con 0x563aecfc8dc0 2024-09-12T02:55:43.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a562e640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfdbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a4e2d640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f289001c040 tx=0x7f28900098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bb2dfdc6a49c57 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f2888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890028030 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2890004b80 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.338+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2890005500 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f289001e040 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 10 client.17265 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563aed04d8c0 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2890005500 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2890019030 con 0x563aecfc8dc0 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 10 client.17265 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 10 client.17265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 20 client.17265 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aed04df80 0x563aed0503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 4 client.17265 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563aecf35d60 con 0x563aed04df80 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.339+0000 7f28a562e640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aed04df80 0x563aed0503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a562e640 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aed04df80 0x563aed0503a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a3d20bffd88cff server_cookie=92f9448c0a23d6c5 in_seq=0 out_seq=0 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f2888ff9640 10 client.17265 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f2888ff9640 1 -- 172.21.15.16:0/3433184721 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563aecf35d60 con 0x563aed04df80 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f2888ff9640 10 client.17265 handle_command_reply: tid=0 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 client.17265 shutdown 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 msgr2=0x563aecfd90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f289001c040 tx=0x7f28900098c0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 shutdown_connections 2024-09-12T02:55:43.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563aed04df80 0x563aed0503a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563aecfd3680 0x563aecfd95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563aecfc8dc0 0x563aecfd90a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 --2- 172.21.15.16:0/3433184721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563aecfc84a0 0x563aecfdbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f2888ff9640 0 client.17265 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 >> 172.21.15.16:0/3433184721 conn(0x563aecfc4a30 msgr2=0x563aecfd21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 shutdown_connections 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 1 -- 172.21.15.16:0/3433184721 wait complete. 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.340+0000 7f28a9c6b740 20 client.17265 trim_cache size 0 max 16384 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 msgr2=0x7f288c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 0x7f288c0a50c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2898042000 tx=0x7f2898004af0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 msgr2=0x7f28a00744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f289c00eab0 tx=0x7f289c00ef70 comp rx=0 tx=0).stop 2024-09-12T02:55:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 shutdown_connections 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a0106f10 0x7f28a0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a0102790 0x7f28a0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f288c0a2c00 0x7f288c0a50c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 --2- 172.21.15.16:0/222005309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a0101de0 0x7f28a00744f0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 >> 172.21.15.16:0/222005309 conn(0x7f28a00fd8c0 msgr2=0x7f28a00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 shutdown_connections 2024-09-12T02:55:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:43.341+0000 7f28a7892640 1 -- 172.21.15.16:0/222005309 wait complete. 2024-09-12T02:55:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:43 smithi016 ceph-mon[30925]: pgmap v571: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-09-12T02:55:43.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.12 MB/sec execute 285 sec latency 13994.967 ms 2024-09-12T02:55:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:43 smithi033 ceph-mon[35092]: pgmap v571: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-09-12T02:55:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:43 smithi184 ceph-mon[35321]: pgmap v571: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-09-12T02:55:44.197 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:55:44.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:55:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:44.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.09 MB/sec execute 286 sec latency 14995.062 ms 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 -- 172.21.15.16:0/1464062965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20101de0 msgr2=0x7fba201021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 --2- 172.21.15.16:0/1464062965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20101de0 0x7fba201021c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fba10009900 tx=0x7fba1002a410 comp rx=0 tx=0).stop 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 -- 172.21.15.16:0/1464062965 shutdown_connections 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 --2- 172.21.15.16:0/1464062965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 --2- 172.21.15.16:0/1464062965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20102790 0x7fba201067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 --2- 172.21.15.16:0/1464062965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20101de0 0x7fba201021c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 -- 172.21.15.16:0/1464062965 >> 172.21.15.16:0/1464062965 conn(0x7fba200fd8c0 msgr2=0x7fba200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 -- 172.21.15.16:0/1464062965 shutdown_connections 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.962+0000 7fba2587d640 1 -- 172.21.15.16:0/1464062965 wait complete. 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 Processor -- start 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 -- start start 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20101de0 0x7fba201985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba20112010 con 0x7fba20102790 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba20111e90 con 0x7fba20101de0 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.963+0000 7fba2587d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba20112190 con 0x7fba20106f10 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55772/0 (socket says 172.21.15.16:55772) 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 -- 172.21.15.16:0/2396215277 learned_addr learned my addr 172.21.15.16:0/2396215277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1f7fe640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1effd640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20101de0 0x7fba201985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 -- 172.21.15.16:0/2396215277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 msgr2=0x7fba2019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 -- 172.21.15.16:0/2396215277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20101de0 msgr2=0x7fba201985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20101de0 0x7fba201985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 -- 172.21.15.16:0/2396215277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2019d560 con 0x7fba20102790 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1f7fe640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba1e7fc640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fba1400b4a0 tx=0x7fba1400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14014070 con 0x7fba20102790 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2019d850 con 0x7fba20102790 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba14010af0 con 0x7fba20102790 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14017d10 con 0x7fba20102790 2024-09-12T02:55:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.964+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba201a5130 con 0x7fba20102790 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.965+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba20102240 con 0x7fba20102790 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.966+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fba14017070 con 0x7fba20102790 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.967+0000 7fba2487b640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 0x7fba080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.967+0000 7fba1effd640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 0x7fba080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.967+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fba140ce510 con 0x7fba20102790 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.968+0000 7fba1effd640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 0x7fba080a5040 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba100096c0 tx=0x7fba10031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:44.971+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba1409b420 con 0x7fba20102790 2024-09-12T02:55:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.182+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba2010a150 con 0x7fba20102790 2024-09-12T02:55:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.183+0000 7fba2487b640 1 -- 172.21.15.16:0/2396215277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fba1409ad20 con 0x7fba20102790 2024-09-12T02:55:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:55:45.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:55:45.184 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 msgr2=0x7fba080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 0x7fba080a5040 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba100096c0 tx=0x7fba10031490 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 msgr2=0x7fba20198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fba1400b4a0 tx=0x7fba1400b960 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 shutdown_connections 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba20106f10 0x7fba2019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fba080a2b80 0x7fba080a5040 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba20102790 0x7fba20198af0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 --2- 172.21.15.16:0/2396215277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba20101de0 0x7fba201985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:45.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.185+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 >> 172.21.15.16:0/2396215277 conn(0x7fba200fd8c0 msgr2=0x7fba200fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:45.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.186+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 shutdown_connections 2024-09-12T02:55:45.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:45.186+0000 7fba2587d640 1 -- 172.21.15.16:0/2396215277 wait complete. 2024-09-12T02:55:45.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:45 smithi016 ceph-mon[30925]: pgmap v572: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 2.1 MiB/s wr, 84 op/s 2024-09-12T02:55:45.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2396215277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:45.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.05 MB/sec execute 287 sec latency 15995.173 ms 2024-09-12T02:55:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:45 smithi033 ceph-mon[35092]: pgmap v572: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 2.1 MiB/s wr, 84 op/s 2024-09-12T02:55:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2396215277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:45 smithi184 ceph-mon[35321]: pgmap v572: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 2.1 MiB/s wr, 84 op/s 2024-09-12T02:55:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2396215277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:45.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:55:46.148 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:46.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:55:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 10.02 MB/sec execute 288 sec latency 16995.270 ms 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/3409696699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4108300 msgr2=0x7fb6c410ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/3409696699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4108300 0x7fb6c410ec70 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009cd0 tx=0x7fb6b402a8d0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/3409696699 shutdown_connections 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/3409696699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4108300 0x7fb6c410ec70 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/3409696699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 0x7fb6c4107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/3409696699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4102dd0 0x7fb6c41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/3409696699 >> 172.21.15.16:0/3409696699 conn(0x7fb6c40fc7d0 msgr2=0x7fb6c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/3409696699 shutdown_connections 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.627+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/3409696699 wait complete. 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 Processor -- start 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 -- start start 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 0x7fb6c4198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4108300 0x7fb6c419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41120d0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4111f50 con 0x7fb6c4103780 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6cbd4c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4112250 con 0x7fb6c4108300 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60626/0 (socket says 172.21.15.16:60626) 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/1745506807 learned_addr learned my addr 172.21.15.16:0/1745506807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4108300 0x7fb6c419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 0x7fb6c4198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/1745506807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4108300 msgr2=0x7fb6c419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4108300 0x7fb6c419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/1745506807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 msgr2=0x7fb6c4198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 0x7fb6c4198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.628+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/1745506807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c419d500 con 0x7fb6c4102dd0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d950 tx=0x7fb6bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6c2ffd640 1 -- 172.21.15.16:0/1745506807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc017070 con 0x7fb6c4102dd0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6c2ffd640 1 -- 172.21.15.16:0/1745506807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc009c30 con 0x7fb6c4102dd0 2024-09-12T02:55:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c419d7f0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6c2ffd640 1 -- 172.21.15.16:0/1745506807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00f6b0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.629+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c41a50d0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.631+0000 7fb6c2ffd640 1 -- 172.21.15.16:0/1745506807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb6bc0109d0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.631+0000 7fb6ce125740 1 Processor -- start 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 -- start start 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa598fd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa598632f0 con 0x55aa598fd5d0 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa5980acf0 con 0x55aa598feb40 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ce125740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa5985f2b0 con 0x55aa59908370 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60634/0 (socket says 172.21.15.16:60634) 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/449078912 learned_addr learned my addr 172.21.15.16:0/449078912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa598fd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c2ffd640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 0x7fb6ac0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 0x7fb6ac0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/449078912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 msgr2=0x55aa5990a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/449078912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 msgr2=0x55aa598fd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.632+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa598fd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6c2ffd640 1 -- 172.21.15.16:0/1745506807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb6bc0cecc0 con 0x7fb6c4102dd0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/449078912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa597ef3a0 con 0x55aa598fd5d0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc097f00 tx=0x7fb6bc050cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2cbbe7d51820b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 0x7fb6ac0a5040 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4006010 tx=0x7fb6b402a8d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6abfff640 1 -- 172.21.15.16:0/449078912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0d0020 con 0x55aa598fd5d0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6abfff640 1 -- 172.21.15.16:0/449078912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc009de0 con 0x55aa598fd5d0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.633+0000 7fb6abfff640 1 -- 172.21.15.16:0/449078912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc020030 con 0x55aa598fd5d0 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- 172.21.15.16:0/449078912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 msgr2=0x55aa598fda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc097f00 tx=0x7fb6bc050cf0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- 172.21.15.16:0/449078912 shutdown_connections 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598fda10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 --2- 172.21.15.16:0/449078912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa598fd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- 172.21.15.16:0/449078912 >> 172.21.15.16:0/449078912 conn(0x55aa598c6310 msgr2=0x55aa598f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- 172.21.15.16:0/449078912 shutdown_connections 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- 172.21.15.16:0/449078912 wait complete. 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 -- start start 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.634+0000 7fb6ce125740 1 Processor -- start 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa59907d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 0x55aa59904fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa5990afb0 con 0x55aa59908370 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa5981e060 con 0x55aa598fd5d0 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa5990b130 con 0x55aa598feb40 2024-09-12T02:55:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ce125740 10 client.0 _ll_register_callbacks cb 0x55aa595718c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60640/0 (socket says 172.21.15.16:60640) 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 -- 172.21.15.16:0/3288651117 learned_addr learned my addr 172.21.15.16:0/3288651117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa59907d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 0x55aa59904fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 -- 172.21.15.16:0/3288651117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 msgr2=0x55aa59904fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 0x55aa59904fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 -- 172.21.15.16:0/3288651117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 msgr2=0x55aa59907d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa59907d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6ca2e9640 1 -- 172.21.15.16:0/3288651117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa595d43c0 con 0x55aa59908370 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.635+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 0x55aa59904fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4035000 tx=0x7fb6b40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf183980227031da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4051070 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb6b4004040 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6bc009de0 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb6b4003e40 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 10 client.17283 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aa59916a10 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc009de0 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb6b404a050 con 0x55aa59908370 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 10 client.17283 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 10 client.17283 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 20 client.17283 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa59917e70 0x55aa5991a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 4 client.17283 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.636+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa59870860 con 0x55aa59917e70 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.637+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa59917e70 0x55aa5991a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.637+0000 7fb6a97fa640 10 client.17283 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.637+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa59917e70 0x55aa5991a2c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44bc1cf621338e36 server_cookie=cc71cd64d5037065 in_seq=0 out_seq=0 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.637+0000 7fb6a97fa640 1 -- 172.21.15.16:0/3288651117 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa59870860 con 0x55aa59917e70 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.637+0000 7fb6a97fa640 10 client.17283 handle_command_reply: tid=0 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 client.17283 shutdown 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 msgr2=0x55aa599055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4035000 tx=0x7fb6b40095c0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 shutdown_connections 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa59908370 0x55aa599055d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa598feb40 0x55aa59904fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa59917e70 0x55aa5991a2c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 --2- 172.21.15.16:0/3288651117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa59907d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6a97fa640 0 client.17283 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 >> 172.21.15.16:0/3288651117 conn(0x55aa598c6310 msgr2=0x55aa598ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 shutdown_connections 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 1 -- 172.21.15.16:0/3288651117 wait complete. 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.638+0000 7fb6ce125740 20 client.17283 trim_cache size 0 max 16384 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 Processor -- start 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 -- start start 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa59914250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa599048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ad5f0 con 0x55aa598fd5d0 2024-09-12T02:55:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41af130 con 0x55aa598feb40 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ce125740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41adcc0 con 0x55aa59908370 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6c9ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60656/0 (socket says 172.21.15.16:60656) 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/2901250460 learned_addr learned my addr 172.21.15.16:0/2901250460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.651+0000 7fb6ca2e9640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa599048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/2901250460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 msgr2=0x55aa599048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa599048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/2901250460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 msgr2=0x55aa59914250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa59914250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa59914250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 -- 172.21.15.16:0/2901250460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa59903f90 con 0x55aa598fd5d0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa59914250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc09b4c0 tx=0x7fb6bc00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07ea65d73c103d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6ab7fe640 1 -- 172.21.15.16:0/2901250460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0d0070 con 0x55aa598fd5d0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6ab7fe640 1 -- 172.21.15.16:0/2901250460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc009de0 con 0x55aa598fd5d0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6ab7fe640 1 -- 172.21.15.16:0/2901250460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc020040 con 0x55aa598fd5d0 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6ce125740 1 -- 172.21.15.16:0/2901250460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 msgr2=0x55aa598f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.652+0000 7fb6ce125740 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc09b4c0 tx=0x7fb6bc00bf60 comp rx=0 tx=0).stop 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- 172.21.15.16:0/2901250460 shutdown_connections 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa599048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598feb40 0x55aa59914250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- 172.21.15.16:0/2901250460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598fd5d0 0x55aa598f9e10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- 172.21.15.16:0/2901250460 >> 172.21.15.16:0/2901250460 conn(0x55aa598c6310 msgr2=0x55aa598f9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- 172.21.15.16:0/2901250460 shutdown_connections 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- 172.21.15.16:0/2901250460 wait complete. 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- start start 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 Processor -- start 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa5990c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41aea60 con 0x55aa598feb40 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ae390 con 0x55aa598fd5d0 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ab5d0 con 0x55aa59908370 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.653+0000 7fb6ce125740 10 client.0 _ll_register_callbacks cb 0x7fb6b4005f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ca2e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ca2e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37646/0 (socket says 172.21.15.16:37646) 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ca2e9640 1 -- 172.21.15.16:0/898467951 learned_addr learned my addr 172.21.15.16:0/898467951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60664/0 (socket says 172.21.15.16:60664) 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa5990c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 -- 172.21.15.16:0/898467951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 msgr2=0x55aa5990ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 -- 172.21.15.16:0/898467951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 msgr2=0x55aa5990c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa5990c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 -- 172.21.15.16:0/898467951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa59981db0 con 0x55aa598feb40 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6c92e7640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8005fd0 tx=0x7fb6b8010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e231164d009d9369 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800fa80 con 0x55aa598feb40 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb6b800fc20 con 0x55aa598feb40 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6bc009de0 con 0x55aa598feb40 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb6b800fe20 con 0x55aa598feb40 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ce125740 10 client.17289 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.654+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aa59983040 con 0x55aa598feb40 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc009de0 con 0x55aa598feb40 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb6b8003c90 con 0x55aa598feb40 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 10 client.17289 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 10 client.17289 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 20 client.17289 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa599837e0 0x55aa59985c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 4 client.17289 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa5986ad70 con 0x55aa599837e0 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa599837e0 0x55aa59985c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6c9ae8640 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa599837e0 0x55aa59985c00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f0a406b37f846 server_cookie=f3bb0faaad1a92ed in_seq=0 out_seq=0 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6a97fa640 10 client.17289 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.655+0000 7fb6a97fa640 1 -- 172.21.15.16:0/898467951 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa5986ad70 con 0x55aa599837e0 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6a97fa640 10 client.17289 handle_command_reply: tid=0 2024-09-12T02:55:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 client.17289 shutdown 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 msgr2=0x55aa5990c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8005fd0 tx=0x7fb6b8010c80 comp rx=0 tx=0).stop 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 shutdown_connections 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa59908370 0x55aa5990ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa598feb40 0x55aa5990c920 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.759 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55aa599837e0 0x55aa59985c00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 --2- 172.21.15.16:0/898467951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa598fd5d0 0x55aa5990c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 >> 172.21.15.16:0/898467951 conn(0x55aa598c6310 msgr2=0x55aa598f84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6a97fa640 0 client.17289 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 shutdown_connections 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 1 -- 172.21.15.16:0/898467951 wait complete. 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6ce125740 20 client.17289 trim_cache size 0 max 16384 2024-09-12T02:55:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 msgr2=0x7fb6ac0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 0x7fb6ac0a5040 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4006010 tx=0x7fb6b402a8d0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 msgr2=0x7fb6c4198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d950 tx=0x7fb6bc00de10 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 shutdown_connections 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6c4108300 0x7fb6c419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb6c4103780 0x7fb6c4198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb6ac0a2b80 0x7fb6ac0a5040 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 --2- 172.21.15.16:0/1745506807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c4102dd0 0x7fb6c4198550 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.656+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 >> 172.21.15.16:0/1745506807 conn(0x7fb6c40fc7d0 msgr2=0x7fb6c4105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.657+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 shutdown_connections 2024-09-12T02:55:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:46.657+0000 7fb6cbd4c640 1 -- 172.21.15.16:0/1745506807 wait complete. 2024-09-12T02:55:47.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:47 smithi016 ceph-mon[30925]: pgmap v573: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.2 MiB/s wr, 91 op/s 2024-09-12T02:55:47.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:55:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:47.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 9.98 MB/sec execute 289 sec latency 17995.313 ms 2024-09-12T02:55:47.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:47 smithi033 ceph-mon[35092]: pgmap v573: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.2 MiB/s wr, 91 op/s 2024-09-12T02:55:47.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:47 smithi184 ceph-mon[35321]: pgmap v573: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.2 MiB/s wr, 91 op/s 2024-09-12T02:55:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- 172.21.15.16:0/2970093003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 msgr2=0x7ff81410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 --2- 172.21.15.16:0/2970093003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81410ea90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009ca0 tx=0x7ff80c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- 172.21.15.16:0/2970093003 shutdown_connections 2024-09-12T02:55:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 --2- 172.21.15.16:0/2970093003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81410ea90 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 --2- 172.21.15.16:0/2970093003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814102790 0x7ff8141067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 --2- 172.21.15.16:0/2970093003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814101de0 0x7ff8141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- 172.21.15.16:0/2970093003 >> 172.21.15.16:0/2970093003 conn(0x7ff8140fd880 msgr2=0x7ff8140ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- 172.21.15.16:0/2970093003 shutdown_connections 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- 172.21.15.16:0/2970093003 wait complete. 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 Processor -- start 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.966+0000 7ff81c190640 1 -- start start 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814101de0 0x7ff814198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814102790 0x7ff8141988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111f70 con 0x7ff814106f10 2024-09-12T02:55:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111df0 con 0x7ff814101de0 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81c190640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141120f0 con 0x7ff814102790 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60680/0 (socket says 172.21.15.16:60680) 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 -- 172.21.15.16:0/2987887705 learned_addr learned my addr 172.21.15.16:0/2987887705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81972b640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814102790 0x7ff8141988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff819f2c640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814101de0 0x7ff814198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 -- 172.21.15.16:0/2987887705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814102790 msgr2=0x7ff8141988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814102790 0x7ff8141988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 -- 172.21.15.16:0/2987887705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814101de0 msgr2=0x7ff814198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814101de0 0x7ff814198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.967+0000 7ff81a72d640 1 -- 172.21.15.16:0/2987887705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81419d320 con 0x7ff814106f10 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff81a72d640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff80c031a20 tx=0x7ff80c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:47.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c003c30 con 0x7ff814106f10 2024-09-12T02:55:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81419d5b0 con 0x7ff814106f10 2024-09-12T02:55:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80c003dd0 con 0x7ff814106f10 2024-09-12T02:55:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c02fcf0 con 0x7ff814106f10 2024-09-12T02:55:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.968+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a4ef0 con 0x7ff814106f10 2024-09-12T02:55:47.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.969+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8141021c0 con 0x7ff814106f10 2024-09-12T02:55:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.971+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff80c004050 con 0x7ff814106f10 2024-09-12T02:55:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.971+0000 7ff80affd640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 0x7ff8000a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.972+0000 7ff819f2c640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 0x7ff8000a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.972+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff80c0f0330 con 0x7ff814106f10 2024-09-12T02:55:47.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.972+0000 7ff819f2c640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 0x7ff8000a5040 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff8100058c0 tx=0x7ff81000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:47.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:47.976+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff80c0bd2d0 con 0x7ff814106f10 2024-09-12T02:55:47.996 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.197+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff81410a150 con 0x7ff814106f10 2024-09-12T02:55:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.198+0000 7ff80affd640 1 -- 172.21.15.16:0/2987887705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7ff80c02fe90 con 0x7ff814106f10 2024-09-12T02:55:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:55:48.198 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:55:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 msgr2=0x7ff8000a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 0x7ff8000a5040 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff8100058c0 tx=0x7ff81000a250 comp rx=0 tx=0).stop 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 msgr2=0x7ff81419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff80c031a20 tx=0x7ff80c031a50 comp rx=0 tx=0).stop 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 shutdown_connections 2024-09-12T02:55:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8000a2b80 0x7ff8000a5040 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff814106f10 0x7ff81419cc40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff814102790 0x7ff8141988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 --2- 172.21.15.16:0/2987887705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff814101de0 0x7ff814198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 >> 172.21.15.16:0/2987887705 conn(0x7ff8140fd880 msgr2=0x7ff8140fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 shutdown_connections 2024-09-12T02:55:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.200+0000 7ff81c190640 1 -- 172.21.15.16:0/2987887705 wait complete. 2024-09-12T02:55:48.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:48 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2987887705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:48.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 9.95 MB/sec execute 290 sec latency 18995.359 ms 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 -- 172.21.15.16:0/1532248936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 msgr2=0x7f049010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 --2- 172.21.15.16:0/1532248936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049010ea90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f0488009ca0 tx=0x7f048802a8c0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 -- 172.21.15.16:0/1532248936 shutdown_connections 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 --2- 172.21.15.16:0/1532248936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049010ea90 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 --2- 172.21.15.16:0/1532248936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490102790 0x7f04901067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 --2- 172.21.15.16:0/1532248936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490101de0 0x7f04901021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.739+0000 7f0495707640 1 -- 172.21.15.16:0/1532248936 >> 172.21.15.16:0/1532248936 conn(0x7f04900fd8c0 msgr2=0x7f04900ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.740+0000 7f0495707640 1 -- 172.21.15.16:0/1532248936 shutdown_connections 2024-09-12T02:55:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.740+0000 7f0495707640 1 -- 172.21.15.16:0/1532248936 wait complete. 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 Processor -- start 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 -- start start 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 0x7f049019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 0x7f049019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0490111f70 con 0x7f0490106f10 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0490111df0 con 0x7f0490101de0 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.741+0000 7f0495707640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04901120f0 con 0x7f0490102790 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 0x7f049019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60690/0 (socket says 172.21.15.16:60690) 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f04867fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 0x7f049019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 -- 172.21.15.16:0/2461891709 learned_addr learned my addr 172.21.15.16:0/2461891709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 0x7f049019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59046/0 (socket says 172.21.15.16:59046) 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 -- 172.21.15.16:0/2461891709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 msgr2=0x7f049019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 0x7f049019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 -- 172.21.15.16:0/2461891709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 msgr2=0x7f049019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 0x7f049019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 -- 172.21.15.16:0/2461891709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049019f570 con 0x7f0490106f10 2024-09-12T02:55:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f04867fc640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 0x7f049019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048f7fe640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0488031a20 tx=0x7f0488031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048cff9640 1 -- 172.21.15.16:0/2461891709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0488003c30 con 0x7f0490106f10 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049019f800 con 0x7f0490106f10 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.742+0000 7f048cff9640 1 -- 172.21.15.16:0/2461891709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0488003dd0 con 0x7f0490106f10 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.743+0000 7f048cff9640 1 -- 172.21.15.16:0/2461891709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048802fcf0 con 0x7f0490106f10 2024-09-12T02:55:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.743+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04901a7140 con 0x7f0490106f10 2024-09-12T02:55:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.745+0000 7f0497ae0740 1 Processor -- start 2024-09-12T02:55:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.745+0000 7f048cff9640 1 -- 172.21.15.16:0/2461891709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f04880305c0 con 0x7f0490106f10 2024-09-12T02:55:48.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.745+0000 7f0497ae0740 1 -- start start 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754556a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754556aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545577760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5575454d02f0 con 0x557545575370 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557545477cf0 con 0x55754556bb40 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f0497ae0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5575454cc2b0 con 0x55754556a5d0 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f04867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754556a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f04867fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754556a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59054/0 (socket says 172.21.15.16:59054) 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f04867fc640 1 -- 172.21.15.16:0/2661342875 learned_addr learned my addr 172.21.15.16:0/2661342875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f048effd640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754556aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.746+0000 7f048effd640 1 -- 172.21.15.16:0/2661342875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 msgr2=0x55754556a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048cff9640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 0x7f047c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048effd640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754556a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f04867fc640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545577760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048effd640 1 -- 172.21.15.16:0/2661342875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 msgr2=0x557545577760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:48 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2987887705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048effd640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545577760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048effd640 1 -- 172.21.15.16:0/2661342875 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55754545c3a0 con 0x55754556a5d0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f04867fc640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545577760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048cff9640 1 -- 172.21.15.16:0/2461891709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f04880f02f0 con 0x7f0490106f10 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.747+0000 7f048f7fe640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 0x7f047c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.748+0000 7f048effd640 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754556aa10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0480005fd0 tx=0x7f04800107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d863c94b1cdcff21 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.748+0000 7f048f7fe640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 0x7f047c0a5040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04880f0650 tx=0x7f0488005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.748+0000 7f04857fa640 1 -- 172.21.15.16:0/2661342875 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048001c480 con 0x55754556a5d0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.748+0000 7f04857fa640 1 -- 172.21.15.16:0/2661342875 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f048001cb00 con 0x55754556a5d0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.748+0000 7f04857fa640 1 -- 172.21.15.16:0/2661342875 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480011c20 con 0x55754556a5d0 2024-09-12T02:55:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 -- 172.21.15.16:0/2661342875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 msgr2=0x55754556aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754556aa10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0480005fd0 tx=0x7f04800107a0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 -- 172.21.15.16:0/2661342875 shutdown_connections 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545577760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754556aa10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 --2- 172.21.15.16:0/2661342875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754556a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 -- 172.21.15.16:0/2661342875 >> 172.21.15.16:0/2661342875 conn(0x557545533310 msgr2=0x557545566c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 -- 172.21.15.16:0/2661342875 shutdown_connections 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.749+0000 7f0497ae0740 1 -- 172.21.15.16:0/2661342875 wait complete. 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 -- start start 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 Processor -- start 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545574df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x5575455720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557545578070 con 0x557545575370 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55754548b060 con 0x55754556bb40 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5575455781f0 con 0x55754556a5d0 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f0497ae0740 10 client.0 _ll_register_callbacks cb 0x5575451de8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60708/0 (socket says 172.21.15.16:60708) 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f04867fc640 1 -- 172.21.15.16:0/1269032561 learned_addr learned my addr 172.21.15.16:0/1269032561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f048effd640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x5575455720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.750+0000 7f04867fc640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545574df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 -- 172.21.15.16:0/1269032561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 msgr2=0x557545574df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545574df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 -- 172.21.15.16:0/1269032561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 msgr2=0x5575455727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 -- 172.21.15.16:0/1269032561 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575452413c0 con 0x55754556bb40 2024-09-12T02:55:48.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04867fc640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:48.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f048effd640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x5575455720a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0480005d00 tx=0x7f048000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c8bcc0c09d3e0df server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04767fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048002b060 con 0x55754556bb40 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f048001c810 con 0x55754556bb40 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f048001cb00 con 0x55754556bb40 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0480011920 con 0x55754556bb40 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f0497ae0740 10 client.26810 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557545583b10 con 0x55754556bb40 2024-09-12T02:55:48.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.751+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f048001cb00 con 0x55754556bb40 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0480004750 con 0x55754556bb40 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 10 client.26810 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 10 client.26810 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 20 client.26810 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x557545587250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 4 client.26810 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575454dd860 con 0x557545584e60 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f04867fc640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x557545587250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f04867fc640 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x557545587250 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=620ffcf187105f24 server_cookie=bb761b48a01d0b95 in_seq=0 out_seq=0 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.752+0000 7f04767fc640 10 client.26810 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f04767fc640 1 -- 172.21.15.16:0/1269032561 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5575454dd860 con 0x557545584e60 2024-09-12T02:55:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f04767fc640 10 client.26810 handle_command_reply: tid=0 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 client.26810 shutdown 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 msgr2=0x5575455720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x5575455720a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0480005d00 tx=0x7f048000bf50 comp rx=0 tx=0).stop 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 shutdown_connections 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x557545587250 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f04767fc640 0 client.26810 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x5575455727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x5575455720a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 --2- 172.21.15.16:0/1269032561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545574df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 >> 172.21.15.16:0/1269032561 conn(0x557545533310 msgr2=0x55754556c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 shutdown_connections 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 1 -- 172.21.15.16:0/1269032561 wait complete. 2024-09-12T02:55:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.753+0000 7f0497ae0740 20 client.26810 trim_cache size 0 max 16384 2024-09-12T02:55:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.766+0000 7f0497ae0740 1 Processor -- start 2024-09-12T02:55:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.766+0000 7f0497ae0740 1 -- start start 2024-09-12T02:55:48.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545581310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557545575370 0x55754558c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04901af770 con 0x55754556bb40 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b12b0 con 0x557545575370 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0497ae0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04901afe40 con 0x55754556a5d0 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60718/0 (socket says 172.21.15.16:60718) 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 -- 172.21.15.16:0/821171439 learned_addr learned my addr 172.21.15.16:0/821171439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f04867fc640 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545581310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 -- 172.21.15.16:0/821171439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 msgr2=0x557545581310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545581310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 -- 172.21.15.16:0/821171439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557545575370 msgr2=0x55754558c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557545575370 0x55754558c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 -- 172.21.15.16:0/821171439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557545567960 con 0x55754556bb40 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f048effd640 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0480004d10 tx=0x7f0480005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8a4ed27251e3d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0484ff9640 1 -- 172.21.15.16:0/821171439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480036020 con 0x55754556bb40 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0484ff9640 1 -- 172.21.15.16:0/821171439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480010ca0 con 0x55754556bb40 2024-09-12T02:55:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.767+0000 7f0484ff9640 1 -- 172.21.15.16:0/821171439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048002b070 con 0x55754556bb40 2024-09-12T02:55:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 -- 172.21.15.16:0/821171439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 msgr2=0x557545574010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0480004d10 tx=0x7f0480005fb0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 -- 172.21.15.16:0/821171439 shutdown_connections 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557545575370 0x55754558c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55754556bb40 0x557545574010 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 --2- 172.21.15.16:0/821171439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x557545581310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 -- 172.21.15.16:0/821171439 >> 172.21.15.16:0/821171439 conn(0x557545533310 msgr2=0x557545573900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 -- 172.21.15.16:0/821171439 shutdown_connections 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.768+0000 7f0497ae0740 1 -- 172.21.15.16:0/821171439 wait complete. 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 -- start start 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 Processor -- start 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754557d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754557dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b0be0 con 0x557545575370 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b0510 con 0x55754556bb40 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04901ad750 con 0x55754556a5d0 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f0497ae0740 10 client.0 _ll_register_callbacks cb 0x7f048000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754557d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60720/0 (socket says 172.21.15.16:60720) 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 -- 172.21.15.16:0/3073683750 learned_addr learned my addr 172.21.15.16:0/3073683750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f048effd640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754557dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 -- 172.21.15.16:0/3073683750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 msgr2=0x55754557d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754557d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 -- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 msgr2=0x55754557dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:48.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754557dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.769+0000 7f04867fc640 1 -- 172.21.15.16:0/3073683750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55754556d120 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04867fc640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0478005940 tx=0x7f0478004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb79924be71ace3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478025070 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0478005020 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0480010ca0 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0478005220 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 10 client.17310 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5575455ef250 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480010ca0 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0478022430 con 0x557545575370 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 10 client.17310 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 10 client.17310 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:55:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.770+0000 7f0497ae0740 20 client.17310 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x5575455f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 4 client.17310 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5575454d7d70 con 0x557545584e60 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f048effd640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x5575455f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f048effd640 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x5575455f1a70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6dfca57c13b802e8 server_cookie=b11a768dbc53b1e7 in_seq=0 out_seq=0 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f04767fc640 10 client.17310 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f04767fc640 1 -- 172.21.15.16:0/3073683750 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5575454d7d70 con 0x557545584e60 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f04767fc640 10 client.17310 handle_command_reply: tid=0 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 client.17310 shutdown 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 msgr2=0x557545578c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0478005940 tx=0x7f0478004730 comp rx=0 tx=0).stop 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 shutdown_connections 2024-09-12T02:55:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557545584e60 0x5575455f1a70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557545575370 0x557545578c30 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f04767fc640 0 client.17310 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55754556bb40 0x55754557dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 --2- 172.21.15.16:0/3073683750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55754556a5d0 0x55754557d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.771+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 >> 172.21.15.16:0/3073683750 conn(0x557545533310 msgr2=0x55754556c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 shutdown_connections 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0497ae0740 1 -- 172.21.15.16:0/3073683750 wait complete. 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0497ae0740 20 client.17310 trim_cache size 0 max 16384 2024-09-12T02:55:48.777 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 msgr2=0x7f047c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 0x7f047c0a5040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04880f0650 tx=0x7f0488005d80 comp rx=0 tx=0).stop 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 msgr2=0x7f049019ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0488031a20 tx=0x7f0488031a50 comp rx=0 tx=0).stop 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 shutdown_connections 2024-09-12T02:55:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f047c0a2b80 0x7f047c0a5040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0490106f10 0x7f049019ee90 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0490102790 0x7f049019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 --2- 172.21.15.16:0/2461891709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0490101de0 0x7f049019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 >> 172.21.15.16:0/2461891709 conn(0x7f04900fd8c0 msgr2=0x7f04900fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 shutdown_connections 2024-09-12T02:55:48.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:48.772+0000 7f0495707640 1 -- 172.21.15.16:0/2461891709 wait complete. 2024-09-12T02:55:48.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:48 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2987887705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:49.156 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id ca0e83a6 --release --await' 2024-09-12T02:55:49.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id ca0e83a6 --release --await 2024-09-12T02:55:49.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:49 smithi016 ceph-mon[30925]: pgmap v574: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-09-12T02:55:49.648 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:49 smithi033 ceph-mon[35092]: pgmap v574: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-09-12T02:55:49.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:49 smithi184 ceph-mon[35321]: pgmap v574: 97 pgs: 97 active+clean; 276 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-09-12T02:55:49.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 753624 9.91 MB/sec execute 291 sec latency 19995.433 ms 2024-09-12T02:55:49.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:55:49.869 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 -- 172.21.15.16:0/2604765326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088106f10 msgr2=0x7f708810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2604765326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088106f10 0x7f708810ea90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7080009ca0 tx=0x7f708002a8c0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 -- 172.21.15.16:0/2604765326 shutdown_connections 2024-09-12T02:55:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2604765326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088106f10 0x7f708810ea90 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2604765326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7088102790 0x7f70881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2604765326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088101de0 0x7f70881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 -- 172.21.15.16:0/2604765326 >> 172.21.15.16:0/2604765326 conn(0x7f70880fd8c0 msgr2=0x7f70880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 -- 172.21.15.16:0/2604765326 shutdown_connections 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.381+0000 7f708ef4b640 1 -- 172.21.15.16:0/2604765326 wait complete. 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 Processor -- start 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 -- start start 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 0x7f7088198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7088106f10 0x7f708819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7088111f70 con 0x7f7088101de0 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7088111df0 con 0x7f7088102790 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.382+0000 7f708ef4b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70881120f0 con 0x7f7088106f10 2024-09-12T02:55:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60732/0 (socket says 172.21.15.16:60732) 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 -- 172.21.15.16:0/2946904780 learned_addr learned my addr 172.21.15.16:0/2946904780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f7087fff640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 0x7f7088198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 -- 172.21.15.16:0/2946904780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7088106f10 msgr2=0x7f708819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7088106f10 0x7f708819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 -- 172.21.15.16:0/2946904780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 msgr2=0x7f7088198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 0x7f7088198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 -- 172.21.15.16:0/2946904780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708819d3c0 con 0x7f7088101de0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f7087fff640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 0x7f7088198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708cce7640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f7085ffb640 1 -- 172.21.15.16:0/2946904780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707000cd80 con 0x7f7088101de0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708819d6b0 con 0x7f7088101de0 2024-09-12T02:55:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f7085ffb640 1 -- 172.21.15.16:0/2946904780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7070010910 con 0x7f7088101de0 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f7085ffb640 1 -- 172.21.15.16:0/2946904780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700188f0 con 0x7f7088101de0 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.383+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70881a4f90 con 0x7f7088101de0 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7085ffb640 1 -- 172.21.15.16:0/2946904780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7070010ac0 con 0x7f7088101de0 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 Processor -- start 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 -- start start 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e302ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e302d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556e2f85620 con 0x5556e3020060 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5556e2f2ce50 con 0x5556e3020980 2024-09-12T02:55:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7091324740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5556e2f815e0 con 0x5556e302b240 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60736/0 (socket says 172.21.15.16:60736) 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f708d4e8640 1 -- 172.21.15.16:0/2241910060 learned_addr learned my addr 172.21.15.16:0/2241910060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f708cce7640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e302ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7085ffb640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 0x7f70740a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.386+0000 7f7087fff640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e302d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 -- 172.21.15.16:0/2241910060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 msgr2=0x5556e302d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e302d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 -- 172.21.15.16:0/2241910060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 msgr2=0x5556e302ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e302ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 -- 172.21.15.16:0/2241910060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e2f1dd40 con 0x5556e3020060 2024-09-12T02:55:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f7087fff640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e302d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f7085ffb640 1 -- 172.21.15.16:0/2946904780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f70700cf550 con 0x7f7088101de0 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 0x7f70740a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7080005ef0 tx=0x7f7080005f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a9ac6004434345 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f707effd640 1 -- 172.21.15.16:0/2241910060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708004d070 con 0x5556e3020060 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f707effd640 1 -- 172.21.15.16:0/2241910060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7080043070 con 0x5556e3020060 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f707effd640 1 -- 172.21.15.16:0/2241910060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080030de0 con 0x5556e3020060 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.387+0000 7f708d4e8640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 0x7f70740a5040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f708004a000 tx=0x7f708004d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- 172.21.15.16:0/2241910060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 msgr2=0x5556e3020440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7080005ef0 tx=0x7f7080005f20 comp rx=0 tx=0).stop 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- 172.21.15.16:0/2241910060 shutdown_connections 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e302d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e302ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 --2- 172.21.15.16:0/2241910060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3020440 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- 172.21.15.16:0/2241910060 >> 172.21.15.16:0/2241910060 conn(0x5556e301c5f0 msgr2=0x5556e301c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- 172.21.15.16:0/2241910060 shutdown_connections 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- 172.21.15.16:0/2241910060 wait complete. 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 -- start start 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.388+0000 7f7091324740 1 Processor -- start 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020060 0x5556e30280b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e3020980 0x5556e30285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556e302de10 con 0x5556e302b240 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5556e2f0df10 con 0x5556e3020060 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5556e302df90 con 0x5556e3020980 2024-09-12T02:55:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7091324740 10 client.0 _ll_register_callbacks cb 0x5556e2f31db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60738/0 (socket says 172.21.15.16:60738) 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 -- 172.21.15.16:0/1374629559 learned_addr learned my addr 172.21.15.16:0/1374629559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708cce7640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020060 0x5556e30280b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f7087fff640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e3020980 0x5556e30285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 -- 172.21.15.16:0/1374629559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e3020980 msgr2=0x5556e30285f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e3020980 0x5556e30285f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 -- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020060 msgr2=0x5556e30280b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020060 0x5556e30280b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.389+0000 7f708d4e8640 1 -- 172.21.15.16:0/1374629559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e2c94580 con 0x5556e302b240 2024-09-12T02:55:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f708d4e8640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7080031f70 tx=0x7f7080009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c3ece9f6e1f87c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708002f310 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f708002fa60 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7080043070 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f708002fc60 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 10 client.17322 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5556e2d5e960 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7080043070 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f70800605a0 con 0x5556e302b240 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 10 client.17322 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 10 client.17322 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 20 client.17322 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e303abc0 0x5556e303cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 4 client.17322 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.390+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556e2f92b90 con 0x5556e303abc0 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.391+0000 7f708cce7640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e303abc0 0x5556e303cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.391+0000 7f7067fff640 10 client.17322 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.391+0000 7f708cce7640 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e303abc0 0x5556e303cfb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b74292ce8a60e45c server_cookie=d0fde1e153fd168c in_seq=0 out_seq=0 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7067fff640 1 -- 172.21.15.16:0/1374629559 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556e2f92b90 con 0x5556e303abc0 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7067fff640 10 client.17322 handle_command_reply: tid=0 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 client.17322 shutdown 2024-09-12T02:55:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 msgr2=0x5556e3028b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7080031f70 tx=0x7f7080009360 comp rx=0 tx=0).stop 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 shutdown_connections 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e303abc0 0x5556e303cfb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e302b240 0x5556e3028b30 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e3020980 0x5556e30285f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7067fff640 0 client.17322 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 --2- 172.21.15.16:0/1374629559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020060 0x5556e30280b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 >> 172.21.15.16:0/1374629559 conn(0x5556e301c5f0 msgr2=0x5556e3022320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 shutdown_connections 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 1 -- 172.21.15.16:0/1374629559 wait complete. 2024-09-12T02:55:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.392+0000 7f7091324740 20 client.17322 trim_cache size 0 max 16384 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 Processor -- start 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 -- start start 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e301b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e3042550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70881ad5c0 con 0x5556e3020060 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70881af100 con 0x5556e3020980 2024-09-12T02:55:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f7091324740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70881adc90 con 0x5556e302b240 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f708cce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f708cce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60742/0 (socket says 172.21.15.16:60742) 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.410+0000 7f708cce7640 1 -- 172.21.15.16:0/3206046208 learned_addr learned my addr 172.21.15.16:0/3206046208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708d4e8640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e3042550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f7087fff640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e301b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 -- 172.21.15.16:0/3206046208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 msgr2=0x5556e3042550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e3042550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 -- 172.21.15.16:0/3206046208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 msgr2=0x5556e301b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e301b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 -- 172.21.15.16:0/3206046208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e301abb0 con 0x5556e3020060 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f7087fff640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e301b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f708cce7640 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f707000c6a0 tx=0x7f70700d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1955eb564b1985 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f707e7fc640 1 -- 172.21.15.16:0/3206046208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707009bdc0 con 0x5556e3020060 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f707e7fc640 1 -- 172.21.15.16:0/3206046208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70700a0170 con 0x5556e3020060 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.411+0000 7f707e7fc640 1 -- 172.21.15.16:0/3206046208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700cf0b0 con 0x5556e3020060 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 -- 172.21.15.16:0/3206046208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 msgr2=0x5556e3022ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f707000c6a0 tx=0x7f70700d1040 comp rx=0 tx=0).stop 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 -- 172.21.15.16:0/3206046208 shutdown_connections 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e3042550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e301b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 --2- 172.21.15.16:0/3206046208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e3020060 0x5556e3022ac0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 -- 172.21.15.16:0/3206046208 >> 172.21.15.16:0/3206046208 conn(0x5556e301c5f0 msgr2=0x5556e301af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.412+0000 7f7091324740 1 -- 172.21.15.16:0/3206046208 shutdown_connections 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.413+0000 7f7091324740 1 -- 172.21.15.16:0/3206046208 wait complete. 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.413+0000 7f7091324740 1 -- start start 2024-09-12T02:55:50.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.413+0000 7f7091324740 1 Processor -- start 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e3033720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e30301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70881aea30 con 0x5556e30306f0 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70881ae360 con 0x5556e3020980 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70881ab5a0 con 0x5556e302b240 2024-09-12T02:55:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7091324740 10 client.0 _ll_register_callbacks cb 0x7f70700d2f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60758/0 (socket says 172.21.15.16:60758) 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 -- 172.21.15.16:0/980957368 learned_addr learned my addr 172.21.15.16:0/980957368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f7087fff640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e30301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708cce7640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e3033720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 -- 172.21.15.16:0/980957368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 msgr2=0x5556e30301b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e30301b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 -- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 msgr2=0x5556e3033720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e3033720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.414+0000 7f708d4e8640 1 -- 172.21.15.16:0/980957368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e30a5530 con 0x5556e30306f0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f708d4e8640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f708002ade0 tx=0x7f7080004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96df8b848105881a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080030d70 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7080060480 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70700a0170 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7080071050 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 10 client.17328 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5556e30a6720 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7080043070 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7080068070 con 0x5556e30306f0 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 10 client.17328 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 10 client.17328 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 20 client.17328 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.415+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e30a6d80 0x5556e30a9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.416+0000 7f7091324740 4 client.17328 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ca0e83a6", "release": true, "await": true}] 2024-09-12T02:55:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.416+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ca0e83a6", "release": true, "await": true}) -- 0x7f70700a0170 con 0x5556e30a6d80 2024-09-12T02:55:50.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.416+0000 7f708cce7640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e30a6d80 0x5556e30a9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.416+0000 7f708cce7640 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e30a6d80 0x5556e30a9170 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c431a0681c824e server_cookie=84ab28fc6f75621e in_seq=0 out_seq=0 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.416+0000 7f7067fff640 10 client.17328 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7067fff640 1 -- 172.21.15.16:0/980957368 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f70700a0170 con 0x5556e30a6d80 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7067fff640 10 client.17328 handle_command_reply: tid=0 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 client.17328 shutdown 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 msgr2=0x5556e3030b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f708002ade0 tx=0x7f7080004ab0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 shutdown_connections 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5556e30a6d80 0x5556e30a9170 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e30306f0 0x5556e3030b70 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7067fff640 0 client.17328 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e302b240 0x5556e30301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 --2- 172.21.15.16:0/980957368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e3020980 0x5556e3033720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.426+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 >> 172.21.15.16:0/980957368 conn(0x5556e301c5f0 msgr2=0x5556e3028870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.427+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 shutdown_connections 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.427+0000 7f7091324740 1 -- 172.21.15.16:0/980957368 wait complete. 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.427+0000 7f7091324740 20 client.17328 trim_cache size 0 max 16384 2024-09-12T02:55:50.428 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 32, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ca0e83a6": { 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 32, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:55:50.429 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.4, 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:50.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.427+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 msgr2=0x7f70740a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.427+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 0x7f70740a5040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f708004a000 tx=0x7f708004d000 comp rx=0 tx=0).stop 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 msgr2=0x7f7088198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 shutdown_connections 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7088106f10 0x7f708819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7088102790 0x7f7088198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f70740a2b80 0x7f70740a5040 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 --2- 172.21.15.16:0/2946904780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7088101de0 0x7f7088198410 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 >> 172.21.15.16:0/2946904780 conn(0x7f70880fd8c0 msgr2=0x7f70880feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 shutdown_connections 2024-09-12T02:55:50.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:50.428+0000 7f708ef4b640 1 -- 172.21.15.16:0/2946904780 wait complete. 2024-09-12T02:55:50.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 754461 9.90 MB/sec execute 292 sec latency 20688.273 ms 2024-09-12T02:55:51.136 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ca0e83a6', total seconds elapsed: 24.2 2024-09-12T02:55:51.136 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.4 2024-09-12T02:55:51.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:55:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:51.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:51 smithi016 ceph-mon[30925]: pgmap v575: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.1 MiB/s wr, 98 op/s 2024-09-12T02:55:51.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 -- 172.21.15.16:0/3853598891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c102570 msgr2=0x7f311c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 --2- 172.21.15.16:0/3853598891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c102570 0x7f311c1029b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3104009900 tx=0x7f310402a480 comp rx=0 tx=0).stop 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 -- 172.21.15.16:0/3853598891 shutdown_connections 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 --2- 172.21.15.16:0/3853598891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 --2- 172.21.15.16:0/3853598891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c102570 0x7f311c1029b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 --2- 172.21.15.16:0/3853598891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c100e20 0x7f311c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 -- 172.21.15.16:0/3853598891 >> 172.21.15.16:0/3853598891 conn(0x7f311c0fc7d0 msgr2=0x7f311c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 -- 172.21.15.16:0/3853598891 shutdown_connections 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.693+0000 7f31210b9640 1 -- 172.21.15.16:0/3853598891 wait complete. 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 Processor -- start 2024-09-12T02:55:51.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 -- start start 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 0x7f311c1968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c10fe80 con 0x7f311c100e20 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f311c10fd00 con 0x7f311c102ef0 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f31210b9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f311c110000 con 0x7f311c102570 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60778/0 (socket says 172.21.15.16:60778) 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f311ad9d640 1 -- 172.21.15.16:0/714723770 learned_addr learned my addr 172.21.15.16:0/714723770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.694+0000 7f3113fff640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 0x7f311c1968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311b59e640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 -- 172.21.15.16:0/714723770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 msgr2=0x7f311c1968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 0x7f311c1968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 -- 172.21.15.16:0/714723770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 msgr2=0x7f311c19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 -- 172.21.15.16:0/714723770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311c19b310 con 0x7f311c100e20 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311b59e640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f3113fff640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 0x7f311c1968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f311ad9d640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9f0 tx=0x7f310c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f3118d99640 1 -- 172.21.15.16:0/714723770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00cd80 con 0x7f311c100e20 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311c19b600 con 0x7f311c100e20 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f3118d99640 1 -- 172.21.15.16:0/714723770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f310c005500 con 0x7f311c100e20 2024-09-12T02:55:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.695+0000 7f3118d99640 1 -- 172.21.15.16:0/714723770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c010690 con 0x7f311c100e20 2024-09-12T02:55:51.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.696+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311c1a2ee0 con 0x7f311c100e20 2024-09-12T02:55:51.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.699+0000 7f3118d99640 1 -- 172.21.15.16:0/714723770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f310c0046c0 con 0x7f311c100e20 2024-09-12T02:55:51.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.699+0000 7f3123492740 1 Processor -- start 2024-09-12T02:55:51.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.699+0000 7f3123492740 1 -- start start 2024-09-12T02:55:51.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.699+0000 7f3118d99640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f311b59e640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3118d99640 1 -- 172.21.15.16:0/714723770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f310c0cf460 con 0x7f311c100e20 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149eef9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef06760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56149ee5f2f0 con 0x56149eef95d0 2024-09-12T02:55:51.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56149ee06cf0 con 0x56149eefab40 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3123492740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56149ee5b2b0 con 0x56149ef04370 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f311b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef06760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60782/0 (socket says 172.21.15.16:60782) 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f3113fff640 1 -- 172.21.15.16:0/2149049177 learned_addr learned my addr 172.21.15.16:0/2149049177 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.700+0000 7f311ad9d640 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149eef9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 -- 172.21.15.16:0/2149049177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 msgr2=0x56149ef06760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef06760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 -- 172.21.15.16:0/2149049177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 msgr2=0x56149eef9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 -- 172.21.15.16:0/2149049177 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56149edeb3a0 con 0x56149eefab40 2024-09-12T02:55:51.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f3113fff640 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:51.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f311ad9d640 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149eef9090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f310c0ccf70 tx=0x7f310c0980a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6715a1fc5e2678d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f31117fa640 1 -- 172.21.15.16:0/2149049177 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c0a0080 con 0x56149eefab40 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.701+0000 7f31117fa640 1 -- 172.21.15.16:0/2149049177 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f310c0a4940 con 0x56149eefab40 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.702+0000 7f31117fa640 1 -- 172.21.15.16:0/2149049177 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c0a4710 con 0x56149eefab40 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.702+0000 7f3123492740 1 -- 172.21.15.16:0/2149049177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 msgr2=0x56149eef9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.702+0000 7f311b59e640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f311400bfd0 tx=0x7f3114014b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.702+0000 7f3123492740 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149eef9090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f310c0ccf70 tx=0x7f310c0980a0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 -- 172.21.15.16:0/2149049177 shutdown_connections 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef06760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149eef9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 --2- 172.21.15.16:0/2149049177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149eef9090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 -- 172.21.15.16:0/2149049177 >> 172.21.15.16:0/2149049177 conn(0x56149eec2310 msgr2=0x56149eef5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 -- 172.21.15.16:0/2149049177 shutdown_connections 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 -- 172.21.15.16:0/2149049177 wait complete. 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 -- start start 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.703+0000 7f3123492740 1 Processor -- start 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149ef03de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149ef01090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef017d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56149ef0b420 con 0x56149eef95d0 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56149ee1a060 con 0x56149eefab40 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56149ef0b5a0 con 0x56149ef04370 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149ef03de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3123492740 10 client.0 _ll_register_callbacks cb 0x56149eb6d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149ef03de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60784/0 (socket says 172.21.15.16:60784) 2024-09-12T02:55:51.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f311ad9d640 1 -- 172.21.15.16:0/3196309188 learned_addr learned my addr 172.21.15.16:0/3196309188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3113fff640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149ef01090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f311b59e640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3113fff640 1 -- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 msgr2=0x56149ef017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3113fff640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.704+0000 7f3113fff640 1 -- 172.21.15.16:0/3196309188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 msgr2=0x56149ef03de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3113fff640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149ef03de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3113fff640 1 -- 172.21.15.16:0/3196309188 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56149ebd03c0 con 0x56149eefab40 2024-09-12T02:55:51.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3113fff640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149ef01090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56149eef6ac0 tx=0x7f3104003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0670d0c19799b31 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f31027fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104035040 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3104003e20 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f310c0a4940 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3104005020 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3123492740 10 client.26840 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.705+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56149ef06e00 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f310c0a4940 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3104031540 con 0x56149eefab40 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 10 client.26840 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 10 client.26840 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 20 client.26840 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 4 client.26840 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56149ee6c860 con 0x56149ef082c0 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.706+0000 7f311ad9d640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.707+0000 7f311ad9d640 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef0a760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad6288740a518c92 server_cookie=f757d7a672c2597a in_seq=0 out_seq=0 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.707+0000 7f31027fc640 10 client.26840 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f31027fc640 1 -- 172.21.15.16:0/3196309188 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56149ee6c860 con 0x56149ef082c0 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f31027fc640 10 client.26840 handle_command_reply: tid=0 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 client.26840 shutdown 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 msgr2=0x56149ef01090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149ef01090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56149eef6ac0 tx=0x7f3104003820 comp rx=0 tx=0).stop 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 shutdown_connections 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef0a760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f31027fc640 0 client.26840 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eefab40 0x56149ef01090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 --2- 172.21.15.16:0/3196309188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eef95d0 0x56149ef03de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 >> 172.21.15.16:0/3196309188 conn(0x56149eec2310 msgr2=0x56149eefb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.708+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 shutdown_connections 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.709+0000 7f3123492740 1 -- 172.21.15.16:0/3196309188 wait complete. 2024-09-12T02:55:51.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.709+0000 7f3123492740 20 client.26840 trim_cache size 0 max 16384 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 Processor -- start 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 -- start start 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149ef04370 0x56149ef24630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1ab2d0 con 0x56149ef04370 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1ace10 con 0x56149eef95d0 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3123492740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1ab9a0 con 0x56149eefab40 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3113fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3113fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37820/0 (socket says 172.21.15.16:37820) 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f311b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149ef04370 0x56149ef24630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f3113fff640 1 -- 172.21.15.16:0/1645876934 learned_addr learned my addr 172.21.15.16:0/1645876934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.724+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 -- 172.21.15.16:0/1645876934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 msgr2=0x56149ef146c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef146c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 -- 172.21.15.16:0/1645876934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149ef04370 msgr2=0x56149ef24630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149ef04370 0x56149ef24630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 -- 172.21.15.16:0/1645876934 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56149ef14c00 con 0x56149eefab40 2024-09-12T02:55:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f311ad9d640 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.725+0000 7f3113fff640 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3104031ab0 tx=0x7f3104032fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1e8c7842b658f8c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.726+0000 7f3110ff9640 1 -- 172.21.15.16:0/1645876934 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104035070 con 0x56149eefab40 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.726+0000 7f3110ff9640 1 -- 172.21.15.16:0/1645876934 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3104055070 con 0x56149eefab40 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.726+0000 7f3110ff9640 1 -- 172.21.15.16:0/1645876934 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104003e80 con 0x56149eefab40 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- 172.21.15.16:0/1645876934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 msgr2=0x56149ef00070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3104031ab0 tx=0x7f3104032fc0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- 172.21.15.16:0/1645876934 shutdown_connections 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149ef04370 0x56149ef24630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149eefab40 0x56149ef00070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 --2- 172.21.15.16:0/1645876934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- 172.21.15.16:0/1645876934 >> 172.21.15.16:0/1645876934 conn(0x56149eec2310 msgr2=0x56149eeff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- 172.21.15.16:0/1645876934 shutdown_connections 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- 172.21.15.16:0/1645876934 wait complete. 2024-09-12T02:55:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 -- start start 2024-09-12T02:55:51.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.727+0000 7f3123492740 1 Processor -- start 2024-09-12T02:55:51.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eefab40 0x56149ef0c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef0c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 756984 9.91 MB/sec execute 293 sec latency 30.365 ms 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1ac740 con 0x56149eefab40 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1ac070 con 0x56149eef95d0 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1a92b0 con 0x56149ef04370 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef0c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eefab40 0x56149ef0c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59186/0 (socket says 172.21.15.16:59186) 2024-09-12T02:55:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef0c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37836/0 (socket says 172.21.15.16:37836) 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 -- 172.21.15.16:0/923351243 learned_addr learned my addr 172.21.15.16:0/923351243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 10 client.0 _ll_register_callbacks cb 0x7f3104009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 -- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 msgr2=0x56149ef0c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef0c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 -- 172.21.15.16:0/923351243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eefab40 msgr2=0x56149ef0c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eefab40 0x56149ef0c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f311ad9d640 1 -- 172.21.15.16:0/923351243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56149ef0cb70 con 0x56149eef95d0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.728+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3104055070 con 0x56149eef95d0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f311ad9d640 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f310c09c580 tx=0x7f310c09c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5326bd585c1860f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f31027fc640 10 client.26846 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f31027fc640 1 -- 172.21.15.16:0/923351243 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c08b050 con 0x56149eef95d0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f31027fc640 1 -- 172.21.15.16:0/923351243 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f310c0cf130 con 0x56149eef95d0 2024-09-12T02:55:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f31027fc640 1 -- 172.21.15.16:0/923351243 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f310c0ba040 con 0x56149eef95d0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 10 client.26846 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=32}) -- 0x56149ef87280 con 0x56149eef95d0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f31027fc640 1 -- 172.21.15.16:0/923351243 <== mon.2 v2:172.21.15.33:3300/0 4 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f310c08e060 con 0x56149eef95d0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 10 client.26846 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 10 client.26846 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 20 client.26846 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef89460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 4 client.26846 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.729+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56149ee66d70 con 0x56149ef082c0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f3113fff640 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef89460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f3113fff640 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef89460 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b2a2a926dfbb08d server_cookie=d536301f5c4d72cc in_seq=0 out_seq=0 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f31027fc640 10 client.26846 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f31027fc640 1 -- 172.21.15.16:0/923351243 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56149ee66d70 con 0x56149ef082c0 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f31027fc640 10 client.26846 handle_command_reply: tid=0 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f3123492740 1 client.26846 shutdown 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.730+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 msgr2=0x56149ef10d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f310c09c580 tx=0x7f310c09c510 comp rx=0 tx=0).stop 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 shutdown_connections 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x56149ef082c0 0x56149ef89460 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56149ef04370 0x56149ef0c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56149eefab40 0x56149ef0c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 --2- 172.21.15.16:0/923351243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56149eef95d0 0x56149ef10d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 >> 172.21.15.16:0/923351243 conn(0x56149eec2310 msgr2=0x56149ef00180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f31027fc640 0 client.26846 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:55:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 shutdown_connections 2024-09-12T02:55:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 1 -- 172.21.15.16:0/923351243 wait complete. 2024-09-12T02:55:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.731+0000 7f3123492740 20 client.26846 trim_cache size 0 max 16384 2024-09-12T02:55:51.736 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 msgr2=0x7f31080a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f311400bfd0 tx=0x7f3114014b50 comp rx=0 tx=0).stop 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 msgr2=0x7f311c196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9f0 tx=0x7f310c00edf0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 shutdown_connections 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c102ef0 0x7f311c19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102570 0x7f311c1968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 --2- 172.21.15.16:0/714723770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c100e20 0x7f311c196360 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 >> 172.21.15.16:0/714723770 conn(0x7f311c0fc7d0 msgr2=0x7f311c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 shutdown_connections 2024-09-12T02:55:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:51.732+0000 7f31210b9640 1 -- 172.21.15.16:0/714723770 wait complete. 2024-09-12T02:55:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:51.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:51 smithi033 ceph-mon[35092]: pgmap v575: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.1 MiB/s wr, 98 op/s 2024-09-12T02:55:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:51 smithi184 ceph-mon[35321]: pgmap v575: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 2.1 MiB/s wr, 98 op/s 2024-09-12T02:55:52.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:55:52.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 760108 9.94 MB/sec execute 294 sec latency 12.513 ms 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 -- 172.21.15.16:0/377820838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 msgr2=0x7f50c41043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 --2- 172.21.15.16:0/377820838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c41043f0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009900 tx=0x7f50b402a410 comp rx=0 tx=0).stop 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 -- 172.21.15.16:0/377820838 shutdown_connections 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 --2- 172.21.15.16:0/377820838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c4110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 --2- 172.21.15.16:0/377820838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c4108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 --2- 172.21.15.16:0/377820838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c41043f0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 -- 172.21.15.16:0/377820838 >> 172.21.15.16:0/377820838 conn(0x7f50c40ffa50 msgr2=0x7f50c4101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 -- 172.21.15.16:0/377820838 shutdown_connections 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.990+0000 7f50c9b44640 1 -- 172.21.15.16:0/377820838 wait complete. 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 Processor -- start 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 -- start start 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4114170 con 0x7f50c4104010 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113ff0 con 0x7f50c41049c0 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c9b44640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41142f0 con 0x7f50c4109140 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60828/0 (socket says 172.21.15.16:60828) 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c37fe640 1 -- 172.21.15.16:0/538934315 learned_addr learned my addr 172.21.15.16:0/538934315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.991+0000 7f50c3fff640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c2ffd640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 -- 172.21.15.16:0/538934315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 msgr2=0x7f50c419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 -- 172.21.15.16:0/538934315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 msgr2=0x7f50c419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 -- 172.21.15.16:0/538934315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c419f600 con 0x7f50c4104010 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c3fff640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c2ffd640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c37fe640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b403f030 con 0x7f50c4104010 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.992+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b4004440 con 0x7f50c4104010 2024-09-12T02:55:53.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.993+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c419f890 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.993+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4035070 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.993+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c41a71d0 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.994+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c41043f0 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.995+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f50b4042030 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.996+0000 7f50c0ff9640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 0x7f50ac0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.996+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50b40ef300 con 0x7f50c4104010 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.996+0000 7f50c2ffd640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 0x7f50ac0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:52.996+0000 7f50c2ffd640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 0x7f50ac0a50c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50b80095f0 tx=0x7f50b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:53.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.000+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f50b40bbc20 con 0x7f50c4104010 2024-09-12T02:55:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:53.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.259+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50c419b9b0 con 0x7f50c4104010 2024-09-12T02:55:53.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.259+0000 7f50c0ff9640 1 -- 172.21.15.16:0/538934315 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f50b40bff20 con 0x7f50c4104010 2024-09-12T02:55:53.260 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:55:53.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:55:53.260 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 msgr2=0x7f50ac0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 0x7f50ac0a50c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50b80095f0 tx=0x7f50b8005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 msgr2=0x7f50c419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b4004040 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 shutdown_connections 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50c4109140 0x7f50c419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50c41049c0 0x7f50c419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.261+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50ac0a2c00 0x7f50ac0a50c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.262+0000 7f50c9b44640 1 --2- 172.21.15.16:0/538934315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4104010 0x7f50c419a650 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.262+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 >> 172.21.15.16:0/538934315 conn(0x7f50c40ffa50 msgr2=0x7f50c4100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.262+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 shutdown_connections 2024-09-12T02:55:53.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:53.262+0000 7f50c9b44640 1 -- 172.21.15.16:0/538934315 wait complete. 2024-09-12T02:55:53.633 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:53 smithi016 ceph-mon[30925]: pgmap v576: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.5 MiB/s wr, 99 op/s 2024-09-12T02:55:53.633 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/538934315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:53.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 762780 9.95 MB/sec execute 295 sec latency 20.980 ms 2024-09-12T02:55:53.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:53 smithi033 ceph-mon[35092]: pgmap v576: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.5 MiB/s wr, 99 op/s 2024-09-12T02:55:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/538934315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:53 smithi184 ceph-mon[35321]: pgmap v576: 97 pgs: 97 active+clean; 274 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.5 MiB/s wr, 99 op/s 2024-09-12T02:55:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/538934315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:55:54.124 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 139.4 --await-for 45.0' 2024-09-12T02:55:54.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 139.4 --await-for 45.0 2024-09-12T02:55:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 -- 172.21.15.16:0/1359905518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 msgr2=0x7f41c4106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/1359905518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c4106e50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009930 tx=0x7f41b402a420 comp rx=0 tx=0).stop 2024-09-12T02:55:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 -- 172.21.15.16:0/1359905518 shutdown_connections 2024-09-12T02:55:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/1359905518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4101410 0x7f41c410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/1359905518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4100a70 0x7f41c4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/1359905518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c4106e50 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 -- 172.21.15.16:0/1359905518 >> 172.21.15.16:0/1359905518 conn(0x7f41c40fc790 msgr2=0x7f41c40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 -- 172.21.15.16:0/1359905518 shutdown_connections 2024-09-12T02:55:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.298+0000 7f41c9fb1640 1 -- 172.21.15.16:0/1359905518 wait complete. 2024-09-12T02:55:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 Processor -- start 2024-09-12T02:55:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 -- start start 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 0x7f41c41992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 0x7f41c4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4110e70 con 0x7f41c4106a70 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4110cf0 con 0x7f41c4101410 2024-09-12T02:55:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c9fb1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4110ff0 con 0x7f41c4100a70 2024-09-12T02:55:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60858/0 (socket says 172.21.15.16:60858) 2024-09-12T02:55:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 -- 172.21.15.16:0/3900641178 learned_addr learned my addr 172.21.15.16:0/3900641178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 0x7f41c41992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 0x7f41c4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 -- 172.21.15.16:0/3900641178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 msgr2=0x7f41c41992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 0x7f41c41992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 -- 172.21.15.16:0/3900641178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 msgr2=0x7f41c4199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 0x7f41c4199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c3fff640 1 -- 172.21.15.16:0/3900641178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c41a4880 con 0x7f41c4106a70 2024-09-12T02:55:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 0x7f41c41992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.299+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 0x7f41c4199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c3fff640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00eb10 tx=0x7f41bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c0ff9640 1 -- 172.21.15.16:0/3900641178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc004b20 con 0x7f41c4106a70 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c41a4a50 con 0x7f41c4106a70 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c0ff9640 1 -- 172.21.15.16:0/3900641178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41bc004cc0 con 0x7f41c4106a70 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c0ff9640 1 -- 172.21.15.16:0/3900641178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc010760 con 0x7f41c4106a70 2024-09-12T02:55:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.300+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c41a4f60 con 0x7f41c4106a70 2024-09-12T02:55:54.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.303+0000 7f41cc38a740 1 Processor -- start 2024-09-12T02:55:54.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.303+0000 7f41c0ff9640 1 -- 172.21.15.16:0/3900641178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f41bc005020 con 0x7f41c4106a70 2024-09-12T02:55:54.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.303+0000 7f41cc38a740 1 -- start start 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 0x55ad05e08090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e19b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad05d6e2f0 con 0x55ad05e17780 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad05d15cf0 con 0x55ad05e09b40 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41cc38a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ad05d6a2b0 con 0x55ad05e085d0 2024-09-12T02:55:54.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e19b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e19b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60862/0 (socket says 172.21.15.16:60862) 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 -- 172.21.15.16:0/3587381390 learned_addr learned my addr 172.21.15.16:0/3587381390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 0x55ad05e08090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c0ff9640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 0x7f41ac0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 -- 172.21.15.16:0/3587381390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 msgr2=0x55ad05e08a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e08a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 -- 172.21.15.16:0/3587381390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 msgr2=0x55ad05e19b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e19b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c2ffd640 1 -- 172.21.15.16:0/3587381390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad05cfa3a0 con 0x55ad05e09b40 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.304+0000 7f41c3fff640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 0x7f41ac0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.305+0000 7f41c0ff9640 1 -- 172.21.15.16:0/3900641178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41bc0cf520 con 0x7f41c4106a70 2024-09-12T02:55:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.305+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 0x55ad05e08090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c1ca25e475da94e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.305+0000 7f41a9ffb640 1 -- 172.21.15.16:0/3587381390 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011ca0 con 0x55ad05e09b40 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.305+0000 7f41a9ffb640 1 -- 172.21.15.16:0/3587381390 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b80118a0 con 0x55ad05e09b40 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.305+0000 7f41a9ffb640 1 -- 172.21.15.16:0/3587381390 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b801e3a0 con 0x55ad05e09b40 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- 172.21.15.16:0/3587381390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 msgr2=0x55ad05e08090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 0x55ad05e08090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41c3fff640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 0x7f41ac0a5040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0981f0 tx=0x7f41bc066eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- 172.21.15.16:0/3587381390 shutdown_connections 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e19b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3587381390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e09b40 0x55ad05e08090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- 172.21.15.16:0/3587381390 >> 172.21.15.16:0/3587381390 conn(0x55ad05dd1310 msgr2=0x55ad05e04c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- 172.21.15.16:0/3587381390 shutdown_connections 2024-09-12T02:55:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- 172.21.15.16:0/3587381390 wait complete. 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.306+0000 7f41cc38a740 1 -- start start 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 Processor -- start 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 0x55ad05e17160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad05e1a480 con 0x55ad05e17780 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad05d29060 con 0x55ad05e085d0 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ad05e1a600 con 0x55ad05e09b40 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41cc38a740 10 client.0 _ll_register_callbacks cb 0x55ad05a7c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 0x55ad05e17160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37874/0 (socket says 172.21.15.16:37874) 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41c37fe640 1 -- 172.21.15.16:0/3937815022 learned_addr learned my addr 172.21.15.16:0/3937815022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.307+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 -- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 msgr2=0x55ad05e17160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 0x55ad05e17160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 -- 172.21.15.16:0/3937815022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 msgr2=0x55ad05e149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 -- 172.21.15.16:0/3937815022 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad05adf3c0 con 0x55ad05e09b40 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 0x55ad05e17160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f41b402a940 tx=0x7f41b4031850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df481fe2733e3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4035040 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41b4004450 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b80118a0 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41b40035f0 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41cc38a740 10 client.26758 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.308+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad05e25f00 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b80118a0 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f41b4048a10 con 0x55ad05e09b40 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 10 client.26758 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 10 client.26758 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 20 client.26758 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:54.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e273c0 0x55ad05e297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 4 client.26758 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad05d7b860 con 0x55ad05e273c0 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.309+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e273c0 0x55ad05e297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.310+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e273c0 0x55ad05e297e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9105851c3eb8700a server_cookie=1eeffa193e59b942 in_seq=0 out_seq=0 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.310+0000 7f41a2ffd640 10 client.26758 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.310+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3937815022 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad05d7b860 con 0x55ad05e273c0 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.310+0000 7f41a2ffd640 10 client.26758 handle_command_reply: tid=0 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.310+0000 7f41cc38a740 1 client.26758 shutdown 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 msgr2=0x55ad05e14410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f41b402a940 tx=0x7f41b4031850 comp rx=0 tx=0).stop 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 shutdown_connections 2024-09-12T02:55:54.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e273c0 0x55ad05e297e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e17780 0x55ad05e149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e09b40 0x55ad05e14410 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41a2ffd640 0 client.26758 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3937815022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e085d0 0x55ad05e17160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 >> 172.21.15.16:0/3937815022 conn(0x55ad05dd1310 msgr2=0x55ad05e0a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 shutdown_connections 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 1 -- 172.21.15.16:0/3937815022 wait complete. 2024-09-12T02:55:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.311+0000 7f41cc38a740 20 client.26758 trim_cache size 0 max 16384 2024-09-12T02:55:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 Processor -- start 2024-09-12T02:55:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 -- start start 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e23720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e2f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41ab2c0 con 0x55ad05e09b40 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41ace00 con 0x55ad05e17780 2024-09-12T02:55:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.326+0000 7f41cc38a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41ab990 con 0x55ad05e085d0 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e23720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60886/0 (socket says 172.21.15.16:60886) 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c37fe640 1 -- 172.21.15.16:0/2477153743 learned_addr learned my addr 172.21.15.16:0/2477153743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e2f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 -- 172.21.15.16:0/2477153743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 msgr2=0x55ad05e2f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e2f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 -- 172.21.15.16:0/2477153743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 msgr2=0x55ad05e133f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 -- 172.21.15.16:0/2477153743 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad05e04ed0 con 0x55ad05e085d0 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c37fe640 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.327+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e23720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ad05e2a050 tx=0x7f41b800eae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77c5f2d56a7d3551 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41a97fa640 1 -- 172.21.15.16:0/2477153743 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b800fd60 con 0x55ad05e085d0 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41a97fa640 1 -- 172.21.15.16:0/2477153743 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b8011820 con 0x55ad05e085d0 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41a97fa640 1 -- 172.21.15.16:0/2477153743 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8005290 con 0x55ad05e085d0 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 -- 172.21.15.16:0/2477153743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 msgr2=0x55ad05e23720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e23720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ad05e2a050 tx=0x7f41b800eae0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 -- 172.21.15.16:0/2477153743 shutdown_connections 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e2f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 --2- 172.21.15.16:0/2477153743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e23720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.328+0000 7f41cc38a740 1 -- 172.21.15.16:0/2477153743 >> 172.21.15.16:0/2477153743 conn(0x55ad05dd1310 msgr2=0x55ad05e12ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.329+0000 7f41cc38a740 1 -- 172.21.15.16:0/2477153743 shutdown_connections 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.329+0000 7f41cc38a740 1 -- 172.21.15.16:0/2477153743 wait complete. 2024-09-12T02:55:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.329+0000 7f41cc38a740 1 -- start start 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.329+0000 7f41cc38a740 1 Processor -- start 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e1fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e1c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41ac730 con 0x55ad05e09b40 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41ac060 con 0x55ad05e17780 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41a92c0 con 0x55ad05e085d0 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41cc38a740 10 client.0 _ll_register_callbacks cb 0x7f41b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:37896/0 (socket says 172.21.15.16:37896) 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c2ffd640 1 -- 172.21.15.16:0/3425100897 learned_addr learned my addr 172.21.15.16:0/3425100897 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e1c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e1fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c37fe640 1 -- 172.21.15.16:0/3425100897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 msgr2=0x55ad05e1f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c37fe640 1 -- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 msgr2=0x55ad05e1c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.330+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e1c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41c37fe640 1 -- 172.21.15.16:0/3425100897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad05e0b120 con 0x55ad05e09b40 2024-09-12T02:55:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41c37fe640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e1fdd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f41b404aff0 tx=0x7f41b4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da24297f8fc5e2b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4035070 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41b4031dd0 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.331+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b8011820 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41b4005d50 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 10 client.17370 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad05e92630 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b8011820 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f41b403db50 con 0x55ad05e09b40 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 10 client.17370 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 10 client.17370 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 20 client.17370 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e92ca0 0x55ad05e95090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 4 client.17370 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.332+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad05d75d70 con 0x55ad05e92ca0 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e92ca0 0x55ad05e95090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41a2ffd640 10 client.17370 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41c2ffd640 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e92ca0 0x55ad05e95090 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d4c8c3af3e2ec9fc server_cookie=991b18451f42fd2f in_seq=0 out_seq=0 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41a2ffd640 1 -- 172.21.15.16:0/3425100897 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad05d75d70 con 0x55ad05e92ca0 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41a2ffd640 10 client.17370 handle_command_reply: tid=0 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.333+0000 7f41cc38a740 1 client.17370 shutdown 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 msgr2=0x55ad05e1fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e1fdd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f41b404aff0 tx=0x7f41b4005fd0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 shutdown_connections 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ad05e92ca0 0x55ad05e95090 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad05e17780 0x55ad05e1c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad05e09b40 0x55ad05e1fdd0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 --2- 172.21.15.16:0/3425100897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ad05e085d0 0x55ad05e1f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41a2ffd640 0 client.17370 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 >> 172.21.15.16:0/3425100897 conn(0x55ad05dd1310 msgr2=0x55ad05e15c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 shutdown_connections 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 1 -- 172.21.15.16:0/3425100897 wait complete. 2024-09-12T02:55:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.334+0000 7f41cc38a740 20 client.17370 trim_cache size 0 max 16384 2024-09-12T02:55:54.348 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 msgr2=0x7f41ac0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 0x7f41ac0a5040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0981f0 tx=0x7f41bc066eb0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 msgr2=0x7f41c41a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00eb10 tx=0x7f41bc00c2d0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 shutdown_connections 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41ac0a2b80 0x7f41ac0a5040 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4106a70 0x7f41c41a4340 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41c4101410 0x7f41c4199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 --2- 172.21.15.16:0/3900641178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c4100a70 0x7f41c41992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 >> 172.21.15.16:0/3900641178 conn(0x7f41c40fc790 msgr2=0x7f41c40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 shutdown_connections 2024-09-12T02:55:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:54.335+0000 7f41c9fb1640 1 -- 172.21.15.16:0/3900641178 wait complete. 2024-09-12T02:55:54.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 764322 9.94 MB/sec execute 296 sec latency 107.836 ms 2024-09-12T02:55:55.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:55:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:55.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:55 smithi016 ceph-mon[30925]: pgmap v577: 97 pgs: 97 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.1 MiB/s wr, 100 op/s 2024-09-12T02:55:55.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 -- 172.21.15.16:0/4193257190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 msgr2=0x7f4bbc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 --2- 172.21.15.16:0/4193257190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc10da40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009ca0 tx=0x7f4bb402a950 comp rx=0 tx=0).stop 2024-09-12T02:55:55.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 -- 172.21.15.16:0/4193257190 shutdown_connections 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 --2- 172.21.15.16:0/4193257190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc10da40 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 --2- 172.21.15.16:0/4193257190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 0x7f4bbc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 --2- 172.21.15.16:0/4193257190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 -- 172.21.15.16:0/4193257190 >> 172.21.15.16:0/4193257190 conn(0x7f4bbc0fc7d0 msgr2=0x7f4bbc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 -- 172.21.15.16:0/4193257190 shutdown_connections 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.681+0000 7f4bc272d640 1 -- 172.21.15.16:0/4193257190 wait complete. 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 Processor -- start 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 -- start start 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 0x7f4bbc19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc110fd0 con 0x7f4bbc102ef0 2024-09-12T02:55:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc110e50 con 0x7f4bbc100e20 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc272d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc111150 con 0x7f4bbc102570 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc0cca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc0cca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35320/0 (socket says 172.21.15.16:35320) 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bc0cca640 1 -- 172.21.15.16:0/1685626750 learned_addr learned my addr 172.21.15.16:0/1685626750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 0x7f4bbc19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bbbfff640 1 -- 172.21.15.16:0/1685626750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 msgr2=0x7f4bbc19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 0x7f4bbc19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bbbfff640 1 -- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 msgr2=0x7f4bbc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.682+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bbbfff640 1 -- 172.21.15.16:0/1685626750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bbc19f640 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc19a690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00e9f0 tx=0x7f4bac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bb9ffb640 1 -- 172.21.15.16:0/1685626750 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00ccc0 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bbc19f930 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bb9ffb640 1 -- 172.21.15.16:0/1685626750 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bac010910 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bb9ffb640 1 -- 172.21.15.16:0/1685626750 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac018960 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.683+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bbc1a7210 con 0x7f4bbc100e20 2024-09-12T02:55:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.685+0000 7f4bb9ffb640 1 -- 172.21.15.16:0/1685626750 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4bac010ac0 con 0x7f4bbc100e20 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 Processor -- start 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 -- start start 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bb9ffb640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 0x7f4ba80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f13400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f15d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564383e6e2a0 con 0x564383f09080 2024-09-12T02:55:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564383e15b10 con 0x564383f13940 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.686+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564383e6a260 con 0x564383f08760 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bb9ffb640 1 -- 172.21.15.16:0/1685626750 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bac0cf4d0 con 0x7f4bbc100e20 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 0x7f4ba80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:45670/0 (socket says 172.21.15.16:45670) 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 -- 172.21.15.16:0/377940921 learned_addr learned my addr 172.21.15.16:0/377940921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f15d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 -- 172.21.15.16:0/377940921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 msgr2=0x564383f15d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f15d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 -- 172.21.15.16:0/377940921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 msgr2=0x564383f13400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f13400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 -- 172.21.15.16:0/377940921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564383bdfb90 con 0x564383f08760 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 0x7f4ba80a5040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x564383f06070 tx=0x7f4bb4009970 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.687+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0a4420 tx=0x7f4bac09b640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b423593528e0ae49 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.688+0000 7f4bb27fc640 1 -- 172.21.15.16:0/377940921 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0217a0 con 0x564383f08760 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.688+0000 7f4bb27fc640 1 -- 172.21.15.16:0/377940921 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bac09c3e0 con 0x564383f08760 2024-09-12T02:55:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.688+0000 7f4bb27fc640 1 -- 172.21.15.16:0/377940921 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0cd090 con 0x564383f08760 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- 172.21.15.16:0/377940921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 msgr2=0x564383f08b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0a4420 tx=0x7f4bac09b640 comp rx=0 tx=0).stop 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- 172.21.15.16:0/377940921 shutdown_connections 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f15d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f13400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/377940921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f08b40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- 172.21.15.16:0/377940921 >> 172.21.15.16:0/377940921 conn(0x564383f04cf0 msgr2=0x564383f050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- 172.21.15.16:0/377940921 shutdown_connections 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- 172.21.15.16:0/377940921 wait complete. 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.689+0000 7f4bc4b06740 1 -- start start 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 Processor -- start 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564383f16510 con 0x564383f09080 2024-09-12T02:55:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564383df66e0 con 0x564383f13940 2024-09-12T02:55:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564383f16690 con 0x564383f08760 2024-09-12T02:55:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc0cca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc4b06740 10 client.0 _ll_register_callbacks cb 0x564383e1aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:45678/0 (socket says 172.21.15.16:45678) 2024-09-12T02:55:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bc0cca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57498/0 (socket says 172.21.15.16:57498) 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 -- 172.21.15.16:0/104824620 learned_addr learned my addr 172.21.15.16:0/104824620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 -- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 msgr2=0x564383f11280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 -- 172.21.15.16:0/104824620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 msgr2=0x564383f10d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f10d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.690+0000 7f4bbbfff640 1 -- 172.21.15.16:0/104824620 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564383f1f9d0 con 0x564383f08760 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0986b0 tx=0x7f4bac0a49d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=505f9f982a5042fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0cd000 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bac09c3e0 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bac0cf0e0 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bac07f050 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4bc4b06740 10 client.26776 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564383f16510 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.691+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bac09c3e0 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bac021bf0 con 0x564383f08760 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 10 client.26776 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 10 client.26776 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 20 client.26776 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f258d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 4 client.26776 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564383e7b9d0 con 0x564383f234e0 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4b9b7fe640 10 client.26776 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.692+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f258d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=961b3e10d4a5096e server_cookie=27e07f9ae831e002 in_seq=0 out_seq=0 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/104824620 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564383e7b9d0 con 0x564383f234e0 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4b9b7fe640 10 client.26776 handle_command_reply: tid=0 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 client.26776 shutdown 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 msgr2=0x564383f10800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0986b0 tx=0x7f4bac0a49d0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 shutdown_connections 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f258d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4b9b7fe640 0 client.26776 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f13940 0x564383f11280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f09080 0x564383f10d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/104824620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f10800 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 >> 172.21.15.16:0/104824620 conn(0x564383f04cf0 msgr2=0x564383f0aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 shutdown_connections 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.693+0000 7f4bc4b06740 1 -- 172.21.15.16:0/104824620 wait complete. 2024-09-12T02:55:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.694+0000 7f4bc4b06740 20 client.26776 trim_cache size 0 max 16384 2024-09-12T02:55:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 Processor -- start 2024-09-12T02:55:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 -- start start 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f110d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1af7d0 con 0x564383f13940 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1b1310 con 0x564383f09080 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1afea0 con 0x564383f08760 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:45680/0 (socket says 172.21.15.16:45680) 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57508/0 (socket says 172.21.15.16:57508) 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bbbfff640 1 -- 172.21.15.16:0/553752883 learned_addr learned my addr 172.21.15.16:0/553752883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f110d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 -- 172.21.15.16:0/553752883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 msgr2=0x564383f126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f126d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 -- 172.21.15.16:0/553752883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 msgr2=0x564383f110d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.714+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f110d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.715+0000 7f4bb3fff640 1 -- 172.21.15.16:0/553752883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564383f04b20 con 0x564383f09080 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.715+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x564383f26090 tx=0x7f4ba4014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=227e27f506ea9280 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.715+0000 7f4bb1ffb640 1 -- 172.21.15.16:0/553752883 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40140d0 con 0x564383f09080 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.715+0000 7f4bb1ffb640 1 -- 172.21.15.16:0/553752883 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba4014270 con 0x564383f09080 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.715+0000 7f4bb1ffb640 1 -- 172.21.15.16:0/553752883 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4012960 con 0x564383f09080 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- 172.21.15.16:0/553752883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 msgr2=0x564383f0f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x564383f26090 tx=0x7f4ba4014840 comp rx=0 tx=0).stop 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- 172.21.15.16:0/553752883 shutdown_connections 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f09080 0x564383f0f0d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/553752883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f08760 0x564383f126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- 172.21.15.16:0/553752883 >> 172.21.15.16:0/553752883 conn(0x564383f04cf0 msgr2=0x564383f03e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- 172.21.15.16:0/553752883 shutdown_connections 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- 172.21.15.16:0/553752883 wait complete. 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 -- start start 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.716+0000 7f4bc4b06740 1 Processor -- start 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f09080 0x564383f0d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f0d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1b0c40 con 0x564383f13940 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1b0570 con 0x564383f08760 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1ad7b0 con 0x564383f09080 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc4b06740 10 client.0 _ll_register_callbacks cb 0x7f4bac0978f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57516/0 (socket says 172.21.15.16:57516) 2024-09-12T02:55:55.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 -- 172.21.15.16:0/138368591 learned_addr learned my addr 172.21.15.16:0/138368591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:55.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f09080 0x564383f0d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f0d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 -- 172.21.15.16:0/138368591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f09080 msgr2=0x564383f0d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f09080 0x564383f0d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 -- 172.21.15.16:0/138368591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 msgr2=0x564383f0d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f0d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bbbfff640 1 -- 172.21.15.16:0/138368591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564383f05da0 con 0x564383f08760 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.717+0000 7f4bc0cca640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f0d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4bac09b9e0 tx=0x7f4bac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491afdbd71a6a2dd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:55.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac09b1d0 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bac0cf0e0 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ba4014270 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bac0215f0 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4bc4b06740 10 client.26888 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564383f96570 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.718+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bac09c3e0 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bac07ab50 con 0x564383f08760 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 10 client.26888 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 10 client.26888 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 20 client.26888 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f98b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:55.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 4 client.26888 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.4, "await_for": 45.0}] 2024-09-12T02:55:55.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.4, "await_for": 45.0}) -- 0x7f4ba4014270 con 0x564383f234e0 2024-09-12T02:55:55.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.719+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f98b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:55.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.720+0000 7f4b9b7fe640 10 client.26888 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:55.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.720+0000 7f4bb3fff640 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f98b80 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fe825cac4729288 server_cookie=79ae93bd82127256 in_seq=0 out_seq=0 2024-09-12T02:55:55.725 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:55.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767275 9.97 MB/sec execute 297 sec latency 38.810 ms 2024-09-12T02:55:55.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:55 smithi033 ceph-mon[35092]: pgmap v577: 97 pgs: 97 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.1 MiB/s wr, 100 op/s 2024-09-12T02:55:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:55 smithi184 ceph-mon[35321]: pgmap v577: 97 pgs: 97 active+clean; 275 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.1 MiB/s wr, 100 op/s 2024-09-12T02:55:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/138368591 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f4ba4014270 con 0x564383f234e0 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4b9b7fe640 10 client.26888 handle_command_reply: tid=0 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 client.26888 shutdown 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 msgr2=0x564383f1beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4bac09b9e0 tx=0x7f4bac0057f0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 shutdown_connections 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564383f234e0 0x564383f98b80 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564383f13940 0x564383f0d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564383f09080 0x564383f0d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4b9b7fe640 0 client.26888 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 --2- 172.21.15.16:0/138368591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564383f08760 0x564383f1beb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 >> 172.21.15.16:0/138368591 conn(0x564383f04cf0 msgr2=0x564383f1e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 shutdown_connections 2024-09-12T02:55:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 1 -- 172.21.15.16:0/138368591 wait complete. 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.810+0000 7f4bc4b06740 20 client.26888 trim_cache size 0 max 16384 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 34, 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "ae63e4e6": { 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 34, 2024-09-12T02:55:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.4, 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:55:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 msgr2=0x7f4ba80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 0x7f4ba80a5040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x564383f06070 tx=0x7f4bb4009970 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 msgr2=0x7f4bbc19a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc19a690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00e9f0 tx=0x7f4bac00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 shutdown_connections 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba80a2b80 0x7f4ba80a5040 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc102ef0 0x7f4bbc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc102570 0x7f4bbc19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 --2- 172.21.15.16:0/1685626750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100e20 0x7f4bbc19a690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 >> 172.21.15.16:0/1685626750 conn(0x7f4bbc0fc7d0 msgr2=0x7f4bbc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 shutdown_connections 2024-09-12T02:55:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:55.811+0000 7f4bc272d640 1 -- 172.21.15.16:0/1685626750 wait complete. 2024-09-12T02:55:56.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.93 MB/sec execute 298 sec latency 978.793 ms 2024-09-12T02:55:56.868 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ae63e4e6', quiesce took 5.7 seconds. Will release after: 13.7 2024-09-12T02:55:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 -- 172.21.15.16:0/869452951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c410c170 msgr2=0x7f94c410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/869452951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c410c170 0x7f94c410e610 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009ca0 tx=0x7f94bc02a950 comp rx=0 tx=0).stop 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 -- 172.21.15.16:0/869452951 shutdown_connections 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/869452951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c410c170 0x7f94c410e610 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/869452951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c4103660 0x7f94c410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/869452951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 0x7f94c4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 -- 172.21.15.16:0/869452951 >> 172.21.15.16:0/869452951 conn(0x7f94c40fc7a0 msgr2=0x7f94c40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 -- 172.21.15.16:0/869452951 shutdown_connections 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.092+0000 7f94ca5a0640 1 -- 172.21.15.16:0/869452951 wait complete. 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 Processor -- start 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 -- start start 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 0x7f94c4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c410c170 0x7f94c419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4111f70 con 0x7f94c4103660 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94c4111df0 con 0x7f94c410c170 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94ca5a0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41120f0 con 0x7f94c4102d40 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35378/0 (socket says 172.21.15.16:35378) 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 -- 172.21.15.16:0/1326552663 learned_addr learned my addr 172.21.15.16:0/1326552663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c410c170 0x7f94c419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 0x7f94c4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 -- 172.21.15.16:0/1326552663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 msgr2=0x7f94c4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 0x7f94c4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 -- 172.21.15.16:0/1326552663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c410c170 msgr2=0x7f94c419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c410c170 0x7f94c419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.093+0000 7f94c37fe640 1 -- 172.21.15.16:0/1326552663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c419d3c0 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f94b800d740 tx=0x7f94b800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94c17fa640 1 -- 172.21.15.16:0/1326552663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8017070 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c419d6b0 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94c17fa640 1 -- 172.21.15.16:0/1326552663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b800de50 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94c17fa640 1 -- 172.21.15.16:0/1326552663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b800f620 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.094+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c41a4f90 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.096+0000 7f94c17fa640 1 -- 172.21.15.16:0/1326552663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f94b8009c30 con 0x7f94c4103660 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 Processor -- start 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 -- start start 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c17fa640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 0x7f94ac0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be52090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be52a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be63b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b41bdb82f0 con 0x55b41be61780 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b41bd5fcf0 con 0x55b41be53b40 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94cc979740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b41bdb42b0 con 0x55b41be525d0 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 0x7f94ac0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c17fa640 1 -- 172.21.15.16:0/1326552663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94b80cec20 con 0x7f94c4103660 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c8b3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be63b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be52a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be52a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:45728/0 (socket says 172.21.15.16:45728) 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c37fe640 1 -- 172.21.15.16:0/2771930229 learned_addr learned my addr 172.21.15.16:0/2771930229 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.097+0000 7f94c3fff640 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be52090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 -- 172.21.15.16:0/2771930229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 msgr2=0x55b41be52a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be52a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 -- 172.21.15.16:0/2771930229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 msgr2=0x55b41be63b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be63b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 -- 172.21.15.16:0/2771930229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41bd443a0 con 0x55b41be53b40 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c3fff640 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be52090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94b400cb10 tx=0x7f94b400e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da37fdc1bb0d93c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94aa7fc640 1 -- 172.21.15.16:0/2771930229 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4004020 con 0x55b41be53b40 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94aa7fc640 1 -- 172.21.15.16:0/2771930229 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b40041c0 con 0x55b41be53b40 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94aa7fc640 1 -- 172.21.15.16:0/2771930229 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4010650 con 0x55b41be53b40 2024-09-12T02:55:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.098+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 0x7f94ac0a5040 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f94bc005e30 tx=0x7f94bc0098b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- 172.21.15.16:0/2771930229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 msgr2=0x55b41be52090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be52090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94b400cb10 tx=0x7f94b400e350 comp rx=0 tx=0).stop 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- 172.21.15.16:0/2771930229 shutdown_connections 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be63b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be52a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 --2- 172.21.15.16:0/2771930229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be52090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- 172.21.15.16:0/2771930229 >> 172.21.15.16:0/2771930229 conn(0x55b41be1b310 msgr2=0x55b41be4ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- 172.21.15.16:0/2771930229 shutdown_connections 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- 172.21.15.16:0/2771930229 wait complete. 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.099+0000 7f94cc979740 1 -- start start 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 Processor -- start 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be5e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b41be64480 con 0x55b41be61780 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b41bd73060 con 0x55b41be525d0 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b41be64600 con 0x55b41be53b40 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94cc979740 10 client.0 _ll_register_callbacks cb 0x55b41bac68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c8b3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be5e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57568/0 (socket says 172.21.15.16:57568) 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c3fff640 1 -- 172.21.15.16:0/1542823924 learned_addr learned my addr 172.21.15.16:0/1542823924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.100+0000 7f94c3fff640 1 -- 172.21.15.16:0/1542823924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 msgr2=0x55b41be5e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94c3fff640 1 -- 172.21.15.16:0/1542823924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 msgr2=0x55b41be5e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be5e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94c3fff640 1 -- 172.21.15.16:0/1542823924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41bb293c0 con 0x55b41be525d0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f94b4003e60 tx=0x7f94b40045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82c52831b091447d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4029060 con 0x55b41be525d0 2024-09-12T02:55:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94b4003fc0 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b40041c0 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f94b400c710 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94cc979740 10 client.26900 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.101+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b41be6fed0 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b40041c0 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f94b4004a60 con 0x55b41be525d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 10 client.26900 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 10 client.26900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 20 client.26900 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41be712d0 0x55b41be736c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 4 client.26900 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b41bdc5860 con 0x55b41be712d0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41be712d0 0x55b41be736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.102+0000 7f94a37fe640 10 client.26900 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.103+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41be712d0 0x55b41be736c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd0eef4e4aca2de7 server_cookie=ea71c8717e79c7ed in_seq=0 out_seq=0 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.103+0000 7f94a37fe640 1 -- 172.21.15.16:0/1542823924 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b41bdc5860 con 0x55b41be712d0 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.103+0000 7f94a37fe640 10 client.26900 handle_command_reply: tid=0 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.103+0000 7f94cc979740 1 client.26900 shutdown 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 msgr2=0x55b41be61120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f94b4003e60 tx=0x7f94b40045a0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 shutdown_connections 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41be712d0 0x55b41be736c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be5e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94a37fe640 0 client.26900 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 --2- 172.21.15.16:0/1542823924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be525d0 0x55b41be61120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 >> 172.21.15.16:0/1542823924 conn(0x55b41be1b310 msgr2=0x55b41be54960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 shutdown_connections 2024-09-12T02:55:57.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 1 -- 172.21.15.16:0/1542823924 wait complete. 2024-09-12T02:55:57.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.104+0000 7f94cc979740 20 client.26900 trim_cache size 0 max 16384 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.117+0000 7f94cc979740 1 Processor -- start 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.117+0000 7f94cc979740 1 -- start start 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be6d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be5dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41ab330 con 0x55b41be61780 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41ace70 con 0x55b41be53b40 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94cc979740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41aba00 con 0x55b41be525d0 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35406/0 (socket says 172.21.15.16:35406) 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 -- 172.21.15.16:0/2023880753 learned_addr learned my addr 172.21.15.16:0/2023880753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c3fff640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be6d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c37fe640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be5dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 -- 172.21.15.16:0/2023880753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 msgr2=0x55b41be6d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be6d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 -- 172.21.15.16:0/2023880753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 msgr2=0x55b41be5dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be5dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 -- 172.21.15.16:0/2023880753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41be60d40 con 0x55b41be61780 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c37fe640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be5dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f94bc035000 tx=0x7f94bc031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15af77e90f4ad4d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.118+0000 7f94a9ffb640 1 -- 172.21.15.16:0/2023880753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc034040 con 0x55b41be61780 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94a9ffb640 1 -- 172.21.15.16:0/2023880753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94bc003df0 con 0x55b41be61780 2024-09-12T02:55:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94a9ffb640 1 -- 172.21.15.16:0/2023880753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc04bac0 con 0x55b41be61780 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- 172.21.15.16:0/2023880753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 msgr2=0x55b41be49ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f94bc035000 tx=0x7f94bc031000 comp rx=0 tx=0).stop 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- 172.21.15.16:0/2023880753 shutdown_connections 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be61780 0x55b41be49ad0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be53b40 0x55b41be5dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 --2- 172.21.15.16:0/2023880753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be525d0 0x55b41be6d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- 172.21.15.16:0/2023880753 >> 172.21.15.16:0/2023880753 conn(0x55b41be1b310 msgr2=0x55b41be4dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- 172.21.15.16:0/2023880753 shutdown_connections 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- 172.21.15.16:0/2023880753 wait complete. 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.119+0000 7f94cc979740 1 -- start start 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 Processor -- start 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be61780 0x55b41be5b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41ac7a0 con 0x55b41be525d0 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41ac0d0 con 0x55b41be61780 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94c41a9310 con 0x55b41be53b40 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94cc979740 10 client.0 _ll_register_callbacks cb 0x7f94b4014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35420/0 (socket says 172.21.15.16:35420) 2024-09-12T02:55:57.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 -- 172.21.15.16:0/1882091075 learned_addr learned my addr 172.21.15.16:0/1882091075 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c8b3d640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be61780 0x55b41be5b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 -- 172.21.15.16:0/1882091075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 msgr2=0x55b41be5b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 -- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be61780 msgr2=0x55b41be5b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be61780 0x55b41be5b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.120+0000 7f94c3fff640 1 -- 172.21.15.16:0/1882091075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41be5be10 con 0x55b41be525d0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94c3fff640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f94b4005a90 tx=0x7f94b4010730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a824acf0df1bc8ae server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4034030 con 0x55b41be525d0 2024-09-12T02:55:57.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94b4021040 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94bc003df0 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f94b401e070 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94cc979740 10 client.17403 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b41bedbb40 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94bc003df0 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.121+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f94b4003fe0 con 0x55b41be525d0 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 10 client.17403 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 20 client.17403 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41bedc2e0 0x55b41bede700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 4 client.17403 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:55:57.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b41bdbfd70 con 0x55b41bedc2e0 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41bedc2e0 0x55b41bede700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94a37fe640 10 client.17403 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94c37fe640 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41bedc2e0 0x55b41bede700 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed95333c0672155 server_cookie=8b59a278260eb92b in_seq=0 out_seq=0 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94a37fe640 1 -- 172.21.15.16:0/1882091075 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b41bdbfd70 con 0x55b41bedc2e0 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94a37fe640 10 client.17403 handle_command_reply: tid=0 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 1 client.17403 shutdown 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.122+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 msgr2=0x55b41be69f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f94b4005a90 tx=0x7f94b4010730 comp rx=0 tx=0).stop 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 shutdown_connections 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b41bedc2e0 0x55b41bede700 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b41be61780 0x55b41be5b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b41be53b40 0x55b41be5b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 --2- 172.21.15.16:0/1882091075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41be525d0 0x55b41be69f40 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94a37fe640 0 client.17403 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 >> 172.21.15.16:0/1882091075 conn(0x55b41be1b310 msgr2=0x55b41be4d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 shutdown_connections 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 1 -- 172.21.15.16:0/1882091075 wait complete. 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94cc979740 20 client.17403 trim_cache size 0 max 16384 2024-09-12T02:55:57.129 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 msgr2=0x7f94ac0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 0x7f94ac0a5040 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f94bc005e30 tx=0x7f94bc0098b0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 msgr2=0x7f94c4198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.123+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f94b800d740 tx=0x7f94b800dc00 comp rx=0 tx=0).stop 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 shutdown_connections 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94c410c170 0x7f94c419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94ac0a2b80 0x7f94ac0a5040 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c4103660 0x7f94c4198950 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 --2- 172.21.15.16:0/1326552663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94c4102d40 0x7f94c4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:57.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 >> 172.21.15.16:0/1326552663 conn(0x7f94c40fc7a0 msgr2=0x7f94c40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:57.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 shutdown_connections 2024-09-12T02:55:57.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:57.124+0000 7f94ca5a0640 1 -- 172.21.15.16:0/1326552663 wait complete. 2024-09-12T02:55:57.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:57 smithi016 ceph-mon[30925]: pgmap v578: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 156 op/s 2024-09-12T02:55:57.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.90 MB/sec execute 299 sec latency 1978.854 ms 2024-09-12T02:55:57.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:57 smithi033 ceph-mon[35092]: pgmap v578: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 156 op/s 2024-09-12T02:55:57.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:57 smithi184 ceph-mon[35321]: pgmap v578: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 156 op/s 2024-09-12T02:55:58.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.87 MB/sec execute 300 sec latency 2978.904 ms 2024-09-12T02:55:58.750 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:55:58.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:55:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:55:59.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:55:59 smithi016 ceph-mon[30925]: pgmap v579: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 159 op/s 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 -- 172.21.15.16:0/2158843153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 msgr2=0x7f8f7410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/2158843153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f7410da40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009ca0 tx=0x7f8f6c02a950 comp rx=0 tx=0).stop 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 -- 172.21.15.16:0/2158843153 shutdown_connections 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/2158843153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f7410da40 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/2158843153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f741029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/2158843153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f74101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 -- 172.21.15.16:0/2158843153 >> 172.21.15.16:0/2158843153 conn(0x7f8f740fc7d0 msgr2=0x7f8f740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 -- 172.21.15.16:0/2158843153 shutdown_connections 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.547+0000 7f8f79ee1640 1 -- 172.21.15.16:0/2158843153 wait complete. 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 Processor -- start 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 -- start start 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f740fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f740fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74110f80 con 0x7f8f74102ef0 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74110e00 con 0x7f8f74102570 2024-09-12T02:55:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f79ee1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74111100 con 0x7f8f74100e20 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35446/0 (socket says 172.21.15.16:35446) 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f73fff640 1 -- 172.21.15.16:0/3463100643 learned_addr learned my addr 172.21.15.16:0/3463100643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f740fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.548+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f740fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 -- 172.21.15.16:0/3463100643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 msgr2=0x7f8f740fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f740fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 -- 172.21.15.16:0/3463100643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 msgr2=0x7f8f740fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f740fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 -- 172.21.15.16:0/3463100643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f741a68d0 con 0x7f8f74102ef0 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f740fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f740fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f73fff640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c004af0 tx=0x7f8f6c02f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f70ff9640 1 -- 172.21.15.16:0/3463100643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c034040 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f741a6b60 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f70ff9640 1 -- 172.21.15.16:0/3463100643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6c030920 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f70ff9640 1 -- 172.21.15.16:0/3463100643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c004630 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.549+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741a7090 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.552+0000 7f8f70ff9640 1 -- 172.21.15.16:0/3463100643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8f6c037020 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.552+0000 7f8f70ff9640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 0x7f8f5c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f70ff9640 1 -- 172.21.15.16:0/3463100643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f6c0efe00 con 0x7f8f74102ef0 2024-09-12T02:55:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 0x7f8f5c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 Processor -- start 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 -- start start 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 0x7f8f5c0a5040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009fc0 tx=0x7f8f64005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62dd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62e0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9d6238280 con 0x55e9d62d2b60 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e9d61dfb10 con 0x55e9d62ddd40 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.553+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e9d6234240 con 0x55e9d62d3480 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35456/0 (socket says 172.21.15.16:35456) 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 -- 172.21.15.16:0/3340767773 learned_addr learned my addr 172.21.15.16:0/3340767773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62dd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f73fff640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62e0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 -- 172.21.15.16:0/3340767773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 msgr2=0x55e9d62dd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62dd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 -- 172.21.15.16:0/3340767773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 msgr2=0x55e9d62e0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62e0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 -- 172.21.15.16:0/3340767773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9d61ec1f0 con 0x55e9d62d2b60 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62dd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f73fff640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62e0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f737fe640 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8f640042c0 tx=0x7f8f64014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0d840af27a288e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f59ffb640 1 -- 172.21.15.16:0/3340767773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6400f480 con 0x55e9d62d2b60 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f59ffb640 1 -- 172.21.15.16:0/3340767773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6400fb00 con 0x55e9d62d2b60 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.554+0000 7f8f59ffb640 1 -- 172.21.15.16:0/3340767773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6401cbe0 con 0x55e9d62d2b60 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/3340767773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 msgr2=0x55e9d62d2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8f640042c0 tx=0x7f8f64014000 comp rx=0 tx=0).stop 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/3340767773 shutdown_connections 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62e0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62dd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/3340767773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d2b60 0x55e9d62d2f40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/3340767773 >> 172.21.15.16:0/3340767773 conn(0x55e9d62cf0f0 msgr2=0x55e9d62cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/3340767773 shutdown_connections 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/3340767773 wait complete. 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.555+0000 7f8f7c2ba740 1 -- start start 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 Processor -- start 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d2b60 0x55e9d62dd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9d62e0910 con 0x55e9d62d3480 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e9d61f2f10 con 0x55e9d62ddd40 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e9d62e0a90 con 0x55e9d62d2b60 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f7c2ba740 10 client.0 _ll_register_callbacks cb 0x55e9d61bba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35458/0 (socket says 172.21.15.16:35458) 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f72ffd640 1 -- 172.21.15.16:0/493132395 learned_addr learned my addr 172.21.15.16:0/493132395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f73fff640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.556+0000 7f8f737fe640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d2b60 0x55e9d62dd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 -- 172.21.15.16:0/493132395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d2b60 msgr2=0x55e9d62dd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d2b60 0x55e9d62dd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 -- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 msgr2=0x55e9d62db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62db2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 -- 172.21.15.16:0/493132395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9d5fa96a0 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f73fff640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8f68012a10 tx=0x7f8f68012ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05395edbdec12 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800fac0 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f6800fc60 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f6400fb00 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f6800fe60 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f7c2ba740 10 client.17415 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9d62ec490 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.557+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6400fb00 con 0x55e9d62d3480 2024-09-12T02:55:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f680105a0 con 0x55e9d62d3480 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 10 client.17415 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 20 client.17415 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d62ed9a0 0x55e9d62efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 4 client.17415 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9d62457f0 con 0x55e9d62ed9a0 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.558+0000 7f8f737fe640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d62ed9a0 0x55e9d62efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.559+0000 7f8f737fe640 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d62ed9a0 0x55e9d62efdc0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60069dd4a4af07ab server_cookie=b7b964da7a35c04e in_seq=0 out_seq=0 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.559+0000 7f8f52ffd640 10 client.17415 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f52ffd640 1 -- 172.21.15.16:0/493132395 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9d62457f0 con 0x55e9d62ed9a0 2024-09-12T02:55:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f52ffd640 10 client.17415 handle_command_reply: tid=0 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 client.17415 shutdown 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 msgr2=0x55e9d62dab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8f68012a10 tx=0x7f8f68012ed0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 shutdown_connections 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d62ed9a0 0x55e9d62efdc0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62ddd40 0x55e9d62db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62d3480 0x55e9d62dab40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/493132395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d2b60 0x55e9d62dd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 >> 172.21.15.16:0/493132395 conn(0x55e9d62cf0f0 msgr2=0x55e9d62ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f52ffd640 0 client.17415 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 shutdown_connections 2024-09-12T02:55:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/493132395 wait complete. 2024-09-12T02:55:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.560+0000 7f8f7c2ba740 20 client.17415 trim_cache size 0 max 16384 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.577+0000 7f8f7c2ba740 1 Processor -- start 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.577+0000 7f8f7c2ba740 1 -- start start 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62f5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741af610 con 0x55e9d62ddd40 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741b1150 con 0x55e9d62d2b60 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741afce0 con 0x55e9d62d3480 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35470/0 (socket says 172.21.15.16:35470) 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 -- 172.21.15.16:0/1353578026 learned_addr learned my addr 172.21.15.16:0/1353578026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f737fe640 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 -- 172.21.15.16:0/1353578026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 msgr2=0x55e9d62f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 -- 172.21.15.16:0/1353578026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 msgr2=0x55e9d62f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 -- 172.21.15.16:0/1353578026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9d62d32b0 con 0x55e9d62ddd40 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.578+0000 7f8f73fff640 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b8a70 tx=0x7f8f6c03c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711eedd806a804b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.579+0000 7f8f597fa640 1 -- 172.21.15.16:0/1353578026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0c09c0 con 0x55e9d62ddd40 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.579+0000 7f8f597fa640 1 -- 172.21.15.16:0/1353578026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6c0b8040 con 0x55e9d62ddd40 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.579+0000 7f8f597fa640 1 -- 172.21.15.16:0/1353578026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0bcad0 con 0x55e9d62ddd40 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.579+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1353578026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 msgr2=0x55e9d62d7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.579+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b8a70 tx=0x7f8f6c03c040 comp rx=0 tx=0).stop 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1353578026 shutdown_connections 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7da0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1353578026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1353578026 >> 172.21.15.16:0/1353578026 conn(0x55e9d62cf0f0 msgr2=0x55e9d62d5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1353578026 shutdown_connections 2024-09-12T02:55:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1353578026 wait complete. 2024-09-12T02:55:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 -- start start 2024-09-12T02:55:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.580+0000 7f8f7c2ba740 1 Processor -- start 2024-09-12T02:55:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62d77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741b0a80 con 0x55e9d62ddd40 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741b03b0 con 0x55e9d62d2b60 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741ad5f0 con 0x55e9d62d3480 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f7c2ba740 10 client.0 _ll_register_callbacks cb 0x7f8f68009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62d77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35482/0 (socket says 172.21.15.16:35482) 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f73fff640 1 -- 172.21.15.16:0/1923702626 learned_addr learned my addr 172.21.15.16:0/1923702626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f737fe640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 -- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 msgr2=0x55e9d62e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 -- 172.21.15.16:0/1923702626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 msgr2=0x55e9d62d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 -- 172.21.15.16:0/1923702626 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9d62d8430 con 0x55e9d62d3480 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f73fff640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f72ffd640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62d77b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8f680109c0 tx=0x7f8f68011e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7786171aaa9741a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.581+0000 7f8f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800f790 con 0x55e9d62d3480 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f6c0b8040 con 0x55e9d62d3480 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f68010640 con 0x55e9d62d3480 2024-09-12T02:55:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f68005040 con 0x55e9d62d3480 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 10 client.26812 fetch_fsmap learned FSMap version 32 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9d6358660 con 0x55e9d62d3480 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6c0b8040 con 0x55e9d62d3480 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f68003a40 con 0x55e9d62d3480 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 10 client.26812 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 20 client.26812 populate_metadata read hostname 'smithi016' 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d6358db0 0x55e9d635b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 4 client.26812 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.582+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e9d623fd00 con 0x55e9d6358db0 2024-09-12T02:55:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.583+0000 7f8f737fe640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d6358db0 0x55e9d635b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.583+0000 7f8f737fe640 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d6358db0 0x55e9d635b1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33a4fcee5b1fe0b7 server_cookie=21cc2d684041fe07 in_seq=0 out_seq=0 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.583+0000 7f8f52ffd640 10 client.26812 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.584+0000 7f8f52ffd640 1 -- 172.21.15.16:0/1923702626 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e9d623fd00 con 0x55e9d6358db0 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.584+0000 7f8f52ffd640 10 client.26812 handle_command_reply: tid=0 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.584+0000 7f8f7c2ba740 1 client.26812 shutdown 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 msgr2=0x55e9d62d77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62d77b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8f680109c0 tx=0x7f8f68011e50 comp rx=0 tx=0).stop 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 shutdown_connections 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e9d6358db0 0x55e9d635b1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9d62ddd40 0x55e9d62d7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e9d62d3480 0x55e9d62d77b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 --2- 172.21.15.16:0/1923702626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e9d62d2b60 0x55e9d62e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 >> 172.21.15.16:0/1923702626 conn(0x55e9d62cf0f0 msgr2=0x55e9d62d9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f52ffd640 0 client.26812 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 shutdown_connections 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 1 -- 172.21.15.16:0/1923702626 wait complete. 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.585+0000 7f8f7c2ba740 20 client.26812 trim_cache size 0 max 16384 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:55:59.591 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:55:59.592 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "dedcc50b-7746-436d-a45e-8b1701f17143", 2024-09-12T02:55:59.592 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:55:59.592 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:55:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 msgr2=0x7f8f5c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 0x7f8f5c0a5040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009fc0 tx=0x7f8f64005f90 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 msgr2=0x7f8f740fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c004af0 tx=0x7f8f6c02f3a0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 shutdown_connections 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f5c0a2b80 0x7f8f5c0a5040 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f74102ef0 0x7f8f740fd700 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f74102570 0x7f8f740fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 --2- 172.21.15.16:0/3463100643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f74100e20 0x7f8f740fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:55:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 >> 172.21.15.16:0/3463100643 conn(0x7f8f740fc7d0 msgr2=0x7f8f7406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:55:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 shutdown_connections 2024-09-12T02:55:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:55:59.586+0000 7f8f79ee1640 1 -- 172.21.15.16:0/3463100643 wait complete. 2024-09-12T02:55:59.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.84 MB/sec execute 301 sec latency 3978.960 ms 2024-09-12T02:55:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:55:59 smithi033 ceph-mon[35092]: pgmap v579: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 159 op/s 2024-09-12T02:55:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:55:59 smithi184 ceph-mon[35321]: pgmap v579: 97 pgs: 97 active+clean; 277 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 159 op/s 2024-09-12T02:56:00.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:56:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:00.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:56:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:56:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:56:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:00 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.728 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.80 MB/sec execute 302 sec latency 4979.008 ms 2024-09-12T02:56:00.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:56:00.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:56:00.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:56:00.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:00 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:00.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:00 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:01.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.028+0000 7f3beed4e640 1 -- 172.21.15.16:0/1035927581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8103740 msgr2=0x7f3be8107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.028+0000 7f3beed4e640 1 --2- 172.21.15.16:0/1035927581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8103740 0x7f3be8107b80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009860 tx=0x7f3bd402a340 comp rx=0 tx=0).stop 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 -- 172.21.15.16:0/1035927581 shutdown_connections 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 --2- 172.21.15.16:0/1035927581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8108290 0x7f3be810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 --2- 172.21.15.16:0/1035927581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8103740 0x7f3be8107b80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 --2- 172.21.15.16:0/1035927581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8102d90 0x7f3be8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 -- 172.21.15.16:0/1035927581 >> 172.21.15.16:0/1035927581 conn(0x7f3be80fc7d0 msgr2=0x7f3be80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 -- 172.21.15.16:0/1035927581 shutdown_connections 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 -- 172.21.15.16:0/1035927581 wait complete. 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 Processor -- start 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.029+0000 7f3beed4e640 1 -- start start 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 0x7f3be81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 0x7f3be8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8108290 0x7f3be819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8112090 con 0x7f3be8103740 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8111f10 con 0x7f3be8108290 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3beed4e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8112210 con 0x7f3be8102d90 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 0x7f3be8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 0x7f3be8198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35502/0 (socket says 172.21.15.16:35502) 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 0x7f3be81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3be7fff640 1 -- 172.21.15.16:0/2744798815 learned_addr learned my addr 172.21.15.16:0/2744798815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8108290 0x7f3be819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 -- 172.21.15.16:0/2744798815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8108290 msgr2=0x7f3be819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8108290 0x7f3be819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 -- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 msgr2=0x7f3be8198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 0x7f3be8198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.030+0000 7f3becaea640 1 -- 172.21.15.16:0/2744798815 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be819d580 con 0x7f3be8102d90 2024-09-12T02:56:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3becaea640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 0x7f3be81985d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00e9f0 tx=0x7f3bdc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3be5ffb640 1 -- 172.21.15.16:0/2744798815 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc00ccc0 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3be5ffb640 1 -- 172.21.15.16:0/2744798815 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bdc005020 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be819d870 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3be5ffb640 1 -- 172.21.15.16:0/2744798815 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0049e0 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.031+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a5150 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.033+0000 7f3bf1127740 1 Processor -- start 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3be5ffb640 1 -- 172.21.15.16:0/2744798815 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3bdc005560 con 0x7f3be8102d90 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 -- start start 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b29140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b2ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2a842e0 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2a2bce0 con 0x55d1a2b1edc0 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bf1127740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2a802a0 con 0x55d1a2b29680 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3becaea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b29140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35504/0 (socket says 172.21.15.16:35504) 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3bed2eb640 1 -- 172.21.15.16:0/783699192 learned_addr learned my addr 172.21.15.16:0/783699192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.034+0000 7f3be7fff640 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b2ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 -- 172.21.15.16:0/783699192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 msgr2=0x55d1a2b2ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b2ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 -- 172.21.15.16:0/783699192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1edc0 msgr2=0x55d1a2b29140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b29140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 -- 172.21.15.16:0/783699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1a2a10440 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3be5ffb640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 0x7f3bd80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3becaea640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 0x7f3bd80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3be5ffb640 1 -- 172.21.15.16:0/2744798815 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3bdc0cef00 con 0x7f3be8102d90 2024-09-12T02:56:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3be00099f0 tx=0x7f3be00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d45eb7b3de9eda server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bd2ffd640 1 -- 172.21.15.16:0/783699192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be001d070 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bd2ffd640 1 -- 172.21.15.16:0/783699192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3be0003e40 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.035+0000 7f3bd2ffd640 1 -- 172.21.15.16:0/783699192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be000dba0 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 -- 172.21.15.16:0/783699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 msgr2=0x55d1a2b1e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3becaea640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 0x7f3bd80a50c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0ccd80 tx=0x7f3bdc0d29e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3be00099f0 tx=0x7f3be00095c0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 -- 172.21.15.16:0/783699192 shutdown_connections 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b2ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b29140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 --2- 172.21.15.16:0/783699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b1e880 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 -- 172.21.15.16:0/783699192 >> 172.21.15.16:0/783699192 conn(0x55d1a2b1aa30 msgr2=0x55d1a2b1ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 -- 172.21.15.16:0/783699192 shutdown_connections 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.036+0000 7f3bf1127740 1 -- 172.21.15.16:0/783699192 wait complete. 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 -- start start 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 Processor -- start 2024-09-12T02:56:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b28fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b26290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2b2c250 con 0x55d1a2b29680 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2a3f050 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d1a2b2c3d0 con 0x55d1a2b1edc0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bf1127740 10 client.0 _ll_register_callbacks cb 0x55d1a2a7f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35508/0 (socket says 172.21.15.16:35508) 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bed2eb640 1 -- 172.21.15.16:0/1400756387 learned_addr learned my addr 172.21.15.16:0/1400756387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3be7fff640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b26290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.037+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b28fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 -- 172.21.15.16:0/1400756387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 msgr2=0x55d1a2b26290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b26290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 -- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 msgr2=0x55d1a2b28fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b28fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 -- 172.21.15.16:0/1400756387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1a27f53c0 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009e20 tx=0x7f3be000d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb91d734d94b21e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be001d070 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3be0004a40 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d1a2a91850 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3be0004050 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bf1127740 10 client.17433 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1a2b37c20 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.038+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d1a2a91850 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3be000b040 con 0x55d1a2b29680 2024-09-12T02:56:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 10 client.17433 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 20 client.17433 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2b39030 0x55d1a2b3b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 4 client.17433 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1a2a8bd60 con 0x55d1a2b39030 2024-09-12T02:56:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3be7fff640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2b39030 0x55d1a2b3b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3bc7fff640 10 client.17433 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.039+0000 7f3be7fff640 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2b39030 0x55d1a2b3b420 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f331625e920831 server_cookie=842dab981404b510 in_seq=0 out_seq=0 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.040+0000 7f3bc7fff640 1 -- 172.21.15.16:0/1400756387 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d1a2a8bd60 con 0x55d1a2b39030 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.040+0000 7f3bc7fff640 10 client.17433 handle_command_reply: tid=0 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.040+0000 7f3bf1127740 1 client.17433 shutdown 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.040+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 msgr2=0x55d1a2b267f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009e20 tx=0x7f3be000d780 comp rx=0 tx=0).stop 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 shutdown_connections 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2b39030 0x55d1a2b3b420 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2b267f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b26290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 --2- 172.21.15.16:0/1400756387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b28fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bc7fff640 0 client.17433 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 >> 172.21.15.16:0/1400756387 conn(0x55d1a2b1aa30 msgr2=0x55d1a2b19b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 shutdown_connections 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 1 -- 172.21.15.16:0/1400756387 wait complete. 2024-09-12T02:56:01.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.041+0000 7f3bf1127740 20 client.17433 trim_cache size 0 max 16384 2024-09-12T02:56:01.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 Processor -- start 2024-09-12T02:56:01.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 -- start start 2024-09-12T02:56:01.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b41010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ad600 con 0x55d1a2b1edc0 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81af140 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3bf1127740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81adcd0 con 0x55d1a2b29680 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.075+0000 7f3be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35514/0 (socket says 172.21.15.16:35514) 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 -- 172.21.15.16:0/3349714648 learned_addr learned my addr 172.21.15.16:0/3349714648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b41010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 -- 172.21.15.16:0/3349714648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 msgr2=0x55d1a2b41010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b41010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 -- 172.21.15.16:0/3349714648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 msgr2=0x55d1a2b354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 -- 172.21.15.16:0/3349714648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1a2b1e2d0 con 0x55d1a2b1edc0 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4035040 tx=0x7f3bd4009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9d5e790ad6923d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3bd27fc640 1 -- 172.21.15.16:0/3349714648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4037040 con 0x55d1a2b1edc0 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3bd27fc640 1 -- 172.21.15.16:0/3349714648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd4004040 con 0x55d1a2b1edc0 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.076+0000 7f3bd27fc640 1 -- 172.21.15.16:0/3349714648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4005280 con 0x55d1a2b1edc0 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 -- 172.21.15.16:0/3349714648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 msgr2=0x55d1a2b40ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4035040 tx=0x7f3bd4009530 comp rx=0 tx=0).stop 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 -- 172.21.15.16:0/3349714648 shutdown_connections 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b29680 0x55d1a2b41010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b40ad0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3349714648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 -- 172.21.15.16:0/3349714648 >> 172.21.15.16:0/3349714648 conn(0x55d1a2b1aa30 msgr2=0x55d1a2b21900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 -- 172.21.15.16:0/3349714648 shutdown_connections 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.077+0000 7f3bf1127740 1 -- 172.21.15.16:0/3349714648 wait complete. 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 -- start start 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 Processor -- start 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b2cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b2d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2ba1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81aea70 con 0x55d1a2b29680 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ae3a0 con 0x55d1a2b1e4a0 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ab5e0 con 0x55d1a2b1edc0 2024-09-12T02:56:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bf1127740 10 client.0 _ll_register_callbacks cb 0x7f3be001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2ba1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bed2eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2ba1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35518/0 (socket says 172.21.15.16:35518) 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bed2eb640 1 -- 172.21.15.16:0/3332473235 learned_addr learned my addr 172.21.15.16:0/3332473235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3be7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b2d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.078+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b2cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 -- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 msgr2=0x55d1a2b2cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b2cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 -- 172.21.15.16:0/3332473235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 msgr2=0x55d1a2ba1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2ba1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 -- 172.21.15.16:0/3332473235 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1a2ba2050 con 0x55d1a2b1edc0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3be7fff640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b2d0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55d1a2b16510 tx=0x7f3bd4005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b73e34a3ce9590 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4056060 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3bd40041f0 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd4004040 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3bd4037050 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bf1127740 10 client.26830 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.079+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1a2ba3340 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd4004040 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3bd4003b90 con 0x55d1a2b1edc0 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 10 client.26830 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 10 client.26830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 20 client.26830 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2ba39b0 0x55d1a2ba5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 4 client.26830 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d1a2a08210 con 0x55d1a2ba39b0 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2ba39b0 0x55d1a2ba5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bed2eb640 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2ba39b0 0x55d1a2ba5dd0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fadff7c320afed3 server_cookie=1a0b5d4cd807bba0 in_seq=0 out_seq=0 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.080+0000 7f3bc7fff640 10 client.26830 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bc7fff640 1 -- 172.21.15.16:0/3332473235 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d1a2a08210 con 0x55d1a2ba39b0 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bc7fff640 10 client.26830 handle_command_reply: tid=0 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 client.26830 shutdown 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 msgr2=0x55d1a2b2d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b2d0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55d1a2b16510 tx=0x7f3bd4005d20 comp rx=0 tx=0).stop 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 shutdown_connections 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d1a2ba39b0 0x55d1a2ba5dd0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1a2b29680 0x55d1a2ba1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d1a2b1edc0 0x55d1a2b2d0e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bc7fff640 0 client.26830 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 --2- 172.21.15.16:0/3332473235 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d1a2b1e4a0 0x55d1a2b2cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 >> 172.21.15.16:0/3332473235 conn(0x55d1a2b1aa30 msgr2=0x55d1a2b21030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 shutdown_connections 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 1 -- 172.21.15.16:0/3332473235 wait complete. 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.081+0000 7f3bf1127740 20 client.26830 trim_cache size 0 max 16384 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:56:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "dedcc50b-7746-436d-a45e-8b1701f17143": { 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "dedcc50b-7746-436d-a45e-8b1701f17143", 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 msgr2=0x7f3bd80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 0x7f3bd80a50c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0ccd80 tx=0x7f3bdc0d29e0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 msgr2=0x7f3be81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 0x7f3be81985d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00e9f0 tx=0x7f3bdc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 shutdown_connections 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3be8108290 0x7f3be819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3bd80a2c00 0x7f3bd80a50c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8103740 0x7f3be8198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 --2- 172.21.15.16:0/2744798815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3be8102d90 0x7f3be81985d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 >> 172.21.15.16:0/2744798815 conn(0x7f3be80fc7d0 msgr2=0x7f3be8105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 shutdown_connections 2024-09-12T02:56:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:01.082+0000 7f3beed4e640 1 -- 172.21.15.16:0/2744798815 wait complete. 2024-09-12T02:56:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:01 smithi016 ceph-mon[30925]: pgmap v580: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 194 op/s 2024-09-12T02:56:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:01 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:56:01.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.77 MB/sec execute 303 sec latency 5979.054 ms 2024-09-12T02:56:01.739 INFO:tasks.cephfs.filesystem:scrub status for tag:dedcc50b-7746-436d-a45e-8b1701f17143 - {'path': '/', 'tag': 'dedcc50b-7746-436d-a45e-8b1701f17143', 'options': 'recursive,force'} 2024-09-12T02:56:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:01 smithi033 ceph-mon[35092]: pgmap v580: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 194 op/s 2024-09-12T02:56:01.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:01 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:56:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:01 smithi184 ceph-mon[35321]: pgmap v580: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 194 op/s 2024-09-12T02:56:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:01 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:56:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:56:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:56:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:56:02.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.74 MB/sec execute 304 sec latency 6979.139 ms 2024-09-12T02:56:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:56:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:56:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:56:02.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:56:02.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:56:02.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:56:03.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.71 MB/sec execute 305 sec latency 7979.247 ms 2024-09-12T02:56:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:03 smithi033 ceph-mon[35092]: pgmap v581: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 185 op/s 2024-09-12T02:56:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:03 smithi184 ceph-mon[35321]: pgmap v581: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 185 op/s 2024-09-12T02:56:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:03 smithi016 ceph-mon[30925]: pgmap v581: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 185 op/s 2024-09-12T02:56:04.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.67 MB/sec execute 306 sec latency 8979.372 ms 2024-09-12T02:56:05.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.64 MB/sec execute 307 sec latency 9979.477 ms 2024-09-12T02:56:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:05 smithi033 ceph-mon[35092]: pgmap v582: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 169 op/s 2024-09-12T02:56:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:05 smithi184 ceph-mon[35321]: pgmap v582: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 169 op/s 2024-09-12T02:56:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:05 smithi016 ceph-mon[30925]: pgmap v582: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 169 op/s 2024-09-12T02:56:06.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.61 MB/sec execute 308 sec latency 10979.596 ms 2024-09-12T02:56:07.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.58 MB/sec execute 309 sec latency 11979.711 ms 2024-09-12T02:56:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:07 smithi033 ceph-mon[35092]: pgmap v583: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 159 op/s 2024-09-12T02:56:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:07 smithi184 ceph-mon[35321]: pgmap v583: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 159 op/s 2024-09-12T02:56:07.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:07 smithi016 ceph-mon[30925]: pgmap v583: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 159 op/s 2024-09-12T02:56:08.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.55 MB/sec execute 310 sec latency 12979.822 ms 2024-09-12T02:56:08.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:56:08.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:56:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:56:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:56:08.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:56:08.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:56:09.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.52 MB/sec execute 311 sec latency 13979.928 ms 2024-09-12T02:56:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:09 smithi033 ceph-mon[35092]: pgmap v584: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.5 MiB/s wr, 93 op/s 2024-09-12T02:56:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:09 smithi184 ceph-mon[35321]: pgmap v584: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.5 MiB/s wr, 93 op/s 2024-09-12T02:56:09.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:09 smithi016 ceph-mon[30925]: pgmap v584: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.5 MiB/s wr, 93 op/s 2024-09-12T02:56:10.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:56:10.729 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.49 MB/sec execute 312 sec latency 14979.985 ms 2024-09-12T02:56:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 -- 172.21.15.16:0/80037882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102ef0 msgr2=0x7fa14410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 --2- 172.21.15.16:0/80037882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102ef0 0x7fa14410da40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009ca0 tx=0x7fa13c02a950 comp rx=0 tx=0).stop 2024-09-12T02:56:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 -- 172.21.15.16:0/80037882 shutdown_connections 2024-09-12T02:56:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 --2- 172.21.15.16:0/80037882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102ef0 0x7fa14410da40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 --2- 172.21.15.16:0/80037882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102570 0x7fa1441029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 --2- 172.21.15.16:0/80037882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 0x7fa144101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 -- 172.21.15.16:0/80037882 >> 172.21.15.16:0/80037882 conn(0x7fa1440fc7d0 msgr2=0x7fa1440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 -- 172.21.15.16:0/80037882 shutdown_connections 2024-09-12T02:56:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.306+0000 7fa14b784640 1 -- 172.21.15.16:0/80037882 wait complete. 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 Processor -- start 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 -- start start 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 0x7fa1441993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 0x7fa1441a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa144110fa0 con 0x7fa144102570 2024-09-12T02:56:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa144110e20 con 0x7fa144102ef0 2024-09-12T02:56:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa14b784640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa144111120 con 0x7fa144100e20 2024-09-12T02:56:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56520/0 (socket says 172.21.15.16:56520) 2024-09-12T02:56:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 -- 172.21.15.16:0/81483140 learned_addr learned my addr 172.21.15.16:0/81483140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa149520640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 0x7fa1441993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa149d21640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 0x7fa1441a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 -- 172.21.15.16:0/81483140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 msgr2=0x7fa1441993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 0x7fa1441993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 -- 172.21.15.16:0/81483140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 msgr2=0x7fa1441a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 0x7fa1441a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa148d1f640 1 -- 172.21.15.16:0/81483140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1441a4970 con 0x7fa144102570 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.307+0000 7fa149d21640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 0x7fa1441a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa148d1f640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa13800c890 tx=0x7fa13800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138004220 con 0x7fa144102570 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1441a4b40 con 0x7fa144102570 2024-09-12T02:56:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1380043c0 con 0x7fa144102570 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138004a00 con 0x7fa144102570 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.308+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1441a5080 con 0x7fa144102570 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.309+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa144101200 con 0x7fa144102570 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.310+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa1380079b0 con 0x7fa144102570 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.311+0000 7fa1367fc640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 0x7fa1300a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.311+0000 7fa149520640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 0x7fa1300a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.311+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa1380cec80 con 0x7fa144102570 2024-09-12T02:56:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.311+0000 7fa149520640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 0x7fa1300a50c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa140009990 tx=0x7fa140009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.315+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa138014030 con 0x7fa144102570 2024-09-12T02:56:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.526+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1441a52b0 con 0x7fa144102570 2024-09-12T02:56:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.527+0000 7fa1367fc640 1 -- 172.21.15.16:0/81483140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fa13809bbf0 con 0x7fa144102570 2024-09-12T02:56:11.529 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:56:11.529 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:56:11.530 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:56:11.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 msgr2=0x7fa1300a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:11.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 0x7fa1300a50c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa140009990 tx=0x7fa140009110 comp rx=0 tx=0).stop 2024-09-12T02:56:11.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 msgr2=0x7fa1441998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa13800c890 tx=0x7fa13800cc90 comp rx=0 tx=0).stop 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 shutdown_connections 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa144102ef0 0x7fa1441a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa1300a2c00 0x7fa1300a50c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa144102570 0x7fa1441998f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 --2- 172.21.15.16:0/81483140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa144100e20 0x7fa1441993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 >> 172.21.15.16:0/81483140 conn(0x7fa1440fc7d0 msgr2=0x7fa144106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 shutdown_connections 2024-09-12T02:56:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:11.529+0000 7fa14b784640 1 -- 172.21.15.16:0/81483140 wait complete. 2024-09-12T02:56:11.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.46 MB/sec execute 313 sec latency 15980.068 ms 2024-09-12T02:56:11.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:11 smithi033 ceph-mon[35092]: pgmap v585: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 80 op/s 2024-09-12T02:56:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:11 smithi184 ceph-mon[35321]: pgmap v585: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 80 op/s 2024-09-12T02:56:11.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:11 smithi016 ceph-mon[30925]: pgmap v585: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 80 op/s 2024-09-12T02:56:12.289 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id ae63e4e6 --release --await' 2024-09-12T02:56:12.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id ae63e4e6 --release --await 2024-09-12T02:56:12.554 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:12.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:12 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/81483140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:12.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 767277 9.43 MB/sec execute 314 sec latency 16980.155 ms 2024-09-12T02:56:12.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:12 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/81483140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:12.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:12 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/81483140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 -- 172.21.15.16:0/2259145208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100480 msgr2=0x7f0720100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2259145208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100480 0x7f0720100860 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f071c009900 tx=0x7f071c02a410 comp rx=0 tx=0).stop 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 -- 172.21.15.16:0/2259145208 shutdown_connections 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2259145208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f07201055b0 0x7f072010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2259145208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0720100e30 0x7f0720104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2259145208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100480 0x7f0720100860 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 -- 172.21.15.16:0/2259145208 >> 172.21.15.16:0/2259145208 conn(0x7f0720077210 msgr2=0x7f07200fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 -- 172.21.15.16:0/2259145208 shutdown_connections 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.994+0000 7f0727c2d640 1 -- 172.21.15.16:0/2259145208 wait complete. 2024-09-12T02:56:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 Processor -- start 2024-09-12T02:56:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 -- start start 2024-09-12T02:56:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 0x7f0720198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07201055b0 0x7f072019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f072010fd60 con 0x7f0720100e30 2024-09-12T02:56:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f072010fbe0 con 0x7f0720100480 2024-09-12T02:56:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f0727c2d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f072010fee0 con 0x7f07201055b0 2024-09-12T02:56:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f07251c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.995+0000 7f07251c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56540/0 (socket says 172.21.15.16:56540) 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 -- 172.21.15.16:0/2620256771 learned_addr learned my addr 172.21.15.16:0/2620256771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07259c9640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 0x7f0720198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07261ca640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07201055b0 0x7f072019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 -- 172.21.15.16:0/2620256771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07201055b0 msgr2=0x7f072019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07201055b0 0x7f072019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 -- 172.21.15.16:0/2620256771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 msgr2=0x7f0720198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 0x7f0720198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 -- 172.21.15.16:0/2620256771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072019d4c0 con 0x7f0720100e30 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07259c9640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 0x7f0720198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f07251c8640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f071000ddc0 tx=0x7f071000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f0716ffd640 1 -- 172.21.15.16:0/2620256771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071000be60 con 0x7f0720100e30 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f0716ffd640 1 -- 172.21.15.16:0/2620256771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0710005500 con 0x7f0720100e30 2024-09-12T02:56:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072019d7b0 con 0x7f0720100e30 2024-09-12T02:56:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f0716ffd640 1 -- 172.21.15.16:0/2620256771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07100035f0 con 0x7f0720100e30 2024-09-12T02:56:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.996+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07201a5090 con 0x7f0720100e30 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f0716ffd640 1 -- 172.21.15.16:0/2620256771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f071001d020 con 0x7f0720100e30 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 Processor -- start 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 -- start start 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a03c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 0x55c04a049630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c049fa1620 con 0x55c04a03c980 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c049f48e50 con 0x55c04a047240 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f072a006740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c049f9d5e0 con 0x55c04a03c060 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56544/0 (socket says 172.21.15.16:56544) 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:12.999+0000 7f07259c9640 1 -- 172.21.15.16:0/3019878483 learned_addr learned my addr 172.21.15.16:0/3019878483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07261ca640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a03c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07251c8640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 0x55c04a049630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f0716ffd640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 0x7f070c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 -- 172.21.15.16:0/3019878483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 msgr2=0x55c04a03c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a03c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 -- 172.21.15.16:0/3019878483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 msgr2=0x55c04a049630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 0x55c04a049630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 -- 172.21.15.16:0/3019878483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c049f39d40 con 0x55c04a03c980 2024-09-12T02:56:13.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07261ca640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a03c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07251c8640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 0x55c04a049630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07261ca640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 0x7f070c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f0716ffd640 1 -- 172.21.15.16:0/2620256771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f07100cee50 con 0x7f0720100e30 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f07259c9640 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f071c031550 tx=0x7f071c032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a82eb7e085be60b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f070bfff640 1 -- 172.21.15.16:0/3019878483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c03f070 con 0x55c04a03c980 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f070bfff640 1 -- 172.21.15.16:0/3019878483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c0317b0 con 0x55c04a03c980 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.000+0000 7f070bfff640 1 -- 172.21.15.16:0/3019878483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c035070 con 0x55c04a03c980 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 -- 172.21.15.16:0/3019878483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 msgr2=0x55c04a046d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f071c031550 tx=0x7f071c032cd0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f07261ca640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 0x7f070c0a5040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0718005fd0 tx=0x7f0718004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 -- 172.21.15.16:0/3019878483 shutdown_connections 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a047240 0x55c04a049630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c980 0x55c04a046d00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 --2- 172.21.15.16:0/3019878483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a03c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 -- 172.21.15.16:0/3019878483 >> 172.21.15.16:0/3019878483 conn(0x55c04a0385f0 msgr2=0x55c04a0389d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 -- 172.21.15.16:0/3019878483 shutdown_connections 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.001+0000 7f072a006740 1 -- 172.21.15.16:0/3019878483 wait complete. 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 -- start start 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 Processor -- start 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a044650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a044b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c04a049e10 con 0x55c04a03c060 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c049f29f10 con 0x55c04a03c980 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c04a049f90 con 0x55c04a047240 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f072a006740 10 client.0 _ll_register_callbacks cb 0x55c049f4ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:13.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56548/0 (socket says 172.21.15.16:56548) 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f07259c9640 1 -- 172.21.15.16:0/1539431970 learned_addr learned my addr 172.21.15.16:0/1539431970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.002+0000 7f07251c8640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a044650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07261ca640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a044b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 -- 172.21.15.16:0/1539431970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 msgr2=0x55c04a044b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a044b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 -- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 msgr2=0x55c04a044650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a044650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 -- 172.21.15.16:0/1539431970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c049cb0580 con 0x55c04a03c060 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07251c8640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a044650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07259c9640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f071c000c00 tx=0x7f071c02ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c031528ccb8317be server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07097fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:13.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c03f070 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f071c0317b0 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f071c004240 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f071c035040 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f072a006740 10 client.17457 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.003+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c049d7a960 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c0317b0 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f071c0037b0 con 0x55c04a03c060 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 10 client.17457 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 10 client.17457 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 20 client.17457 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a05b030 0x55c04a05d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 4 client.17457 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c049faeb90 con 0x55c04a05b030 2024-09-12T02:56:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f07251c8640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a05b030 0x55c04a05d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f07251c8640 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a05b030 0x55c04a05d420 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bbe7a4afda418ca server_cookie=cc668e10105aa2c7 in_seq=0 out_seq=0 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.004+0000 7f07097fa640 10 client.17457 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f07097fa640 1 -- 172.21.15.16:0/1539431970 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c049faeb90 con 0x55c04a05b030 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f07097fa640 10 client.17457 handle_command_reply: tid=0 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 client.17457 shutdown 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 msgr2=0x55c04a044110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f071c000c00 tx=0x7f071c02ab10 comp rx=0 tx=0).stop 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 shutdown_connections 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a05b030 0x55c04a05d420 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a044b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a044650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f07097fa640 0 client.17457 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:13.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 --2- 172.21.15.16:0/1539431970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a044110 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.005+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 >> 172.21.15.16:0/1539431970 conn(0x55c04a0385f0 msgr2=0x55c04a03e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:13.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.006+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 shutdown_connections 2024-09-12T02:56:13.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.006+0000 7f072a006740 1 -- 172.21.15.16:0/1539431970 wait complete. 2024-09-12T02:56:13.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.006+0000 7f072a006740 20 client.17457 trim_cache size 0 max 16384 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.022+0000 7f072a006740 1 Processor -- start 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.022+0000 7f072a006740 1 -- start start 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a062a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a062f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07201ab4b0 con 0x55c04a047240 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f07201acff0 con 0x55c04a03c980 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f072a006740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f07201abb80 con 0x55c04a03c060 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56552/0 (socket says 172.21.15.16:56552) 2024-09-12T02:56:13.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 -- 172.21.15.16:0/2481530730 learned_addr learned my addr 172.21.15.16:0/2481530730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07259c9640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a062a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07251c8640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a062f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 -- 172.21.15.16:0/2481530730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 msgr2=0x55c04a062a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a062a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 -- 172.21.15.16:0/2481530730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 msgr2=0x55c04a062f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a062f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 -- 172.21.15.16:0/2481530730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c04a049990 con 0x55c04a047240 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07251c8640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a062f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.023+0000 7f07261ca640 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f071801e770 tx=0x7f0718014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1951ae841dec256 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f070b7fe640 1 -- 172.21.15.16:0/2481530730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718012730 con 0x55c04a047240 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f070b7fe640 1 -- 172.21.15.16:0/2481530730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718016070 con 0x55c04a047240 2024-09-12T02:56:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f070b7fe640 1 -- 172.21.15.16:0/2481530730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718027e50 con 0x55c04a047240 2024-09-12T02:56:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 -- 172.21.15.16:0/2481530730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 msgr2=0x55c04a041370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f071801e770 tx=0x7f0718014000 comp rx=0 tx=0).stop 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 -- 172.21.15.16:0/2481530730 shutdown_connections 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a047240 0x55c04a041370 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a062f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 --2- 172.21.15.16:0/2481530730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a03c060 0x55c04a062a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 -- 172.21.15.16:0/2481530730 >> 172.21.15.16:0/2481530730 conn(0x55c04a0385f0 msgr2=0x55c04a045970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.024+0000 7f072a006740 1 -- 172.21.15.16:0/2481530730 shutdown_connections 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 -- 172.21.15.16:0/2481530730 wait complete. 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 -- start start 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 Processor -- start 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a04fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a0500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07201ac920 con 0x55c04a03c060 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f07201ac250 con 0x55c04a03c980 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f07201a9490 con 0x55c04a047240 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f072a006740 10 client.0 _ll_register_callbacks cb 0x7f071c032f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.025+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56568/0 (socket says 172.21.15.16:56568) 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 -- 172.21.15.16:0/525906020 learned_addr learned my addr 172.21.15.16:0/525906020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07251c8640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a04fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07261ca640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a0500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 -- 172.21.15.16:0/525906020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 msgr2=0x55c04a0500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a0500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 -- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 msgr2=0x55c04a04fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a04fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 -- 172.21.15.16:0/525906020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c04a0c4c40 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07259c9640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f071c001eb0 tx=0x7f071c005190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6205399601fc1426 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07097fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c03f070 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f071c031d70 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0718016070 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f071c005320 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.026+0000 7f072a006740 10 client.17463 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c04a0c5f60 con 0x55c04a03c060 2024-09-12T02:56:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718016070 con 0x55c04a03c060 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f071c060030 con 0x55c04a03c060 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 10 client.17463 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 10 client.17463 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 20 client.17463 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a0c6670 0x55c04a0c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 4 client.17463 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ae63e4e6", "release": true, "await": true}] 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ae63e4e6", "release": true, "await": true}) -- 0x55c04a03fd30 con 0x55c04a0c6670 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.027+0000 7f07251c8640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a0c6670 0x55c04a0c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.028+0000 7f07251c8640 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a0c6670 0x55c04a0c8a60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d00996230aa1d60 server_cookie=75d9c4fe00c2ac8d in_seq=0 out_seq=0 2024-09-12T02:56:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.028+0000 7f07097fa640 10 client.17463 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f07097fa640 1 -- 172.21.15.16:0/525906020 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c04a03fd30 con 0x55c04a0c6670 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f07097fa640 10 client.17463 handle_command_reply: tid=0 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 client.17463 shutdown 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 msgr2=0x55c04a04f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f071c001eb0 tx=0x7f071c005190 comp rx=0 tx=0).stop 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 shutdown_connections 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c04a0c6670 0x55c04a0c8a60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c04a047240 0x55c04a0500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f07097fa640 0 client.17463 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c04a03c980 0x55c04a04fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 --2- 172.21.15.16:0/525906020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c04a03c060 0x55c04a04f620 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 >> 172.21.15.16:0/525906020 conn(0x55c04a0385f0 msgr2=0x55c04a036f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 shutdown_connections 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.040+0000 7f072a006740 1 -- 172.21.15.16:0/525906020 wait complete. 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f072a006740 20 client.17463 trim_cache size 0 max 16384 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 36, 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:56:13.043 INFO:teuthology.orchestra.run.smithi016.stdout: "ae63e4e6": { 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 36, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.4, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:56:13.044 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:13.045 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 msgr2=0x7f070c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 0x7f070c0a5040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0718005fd0 tx=0x7f0718004a40 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 msgr2=0x7f0720198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f071000ddc0 tx=0x7f071000a620 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 shutdown_connections 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07201055b0 0x7f072019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f070c0a2b80 0x7f070c0a5040 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0720100e30 0x7f0720198a50 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 --2- 172.21.15.16:0/2620256771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0720100480 0x7f0720198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 >> 172.21.15.16:0/2620256771 conn(0x7f0720077210 msgr2=0x7f0720103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 shutdown_connections 2024-09-12T02:56:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:13.041+0000 7f0727c2d640 1 -- 172.21.15.16:0/2620256771 wait complete. 2024-09-12T02:56:13.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 769317 9.42 MB/sec execute 315 sec latency 17292.413 ms 2024-09-12T02:56:13.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:13 smithi033 ceph-mon[35092]: pgmap v586: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 584 KiB/s wr, 35 op/s 2024-09-12T02:56:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:13 smithi184 ceph-mon[35321]: pgmap v586: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 584 KiB/s wr, 35 op/s 2024-09-12T02:56:13.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:13 smithi016 ceph-mon[30925]: pgmap v586: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 584 KiB/s wr, 35 op/s 2024-09-12T02:56:13.962 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ae63e4e6', total seconds elapsed: 22.8 2024-09-12T02:56:14.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 771642 9.44 MB/sec execute 316 sec latency 17.556 ms 2024-09-12T02:56:15.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 774713 9.46 MB/sec execute 317 sec latency 10.542 ms 2024-09-12T02:56:15.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:15 smithi033 ceph-mon[35092]: pgmap v587: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 307 KiB/s wr, 37 op/s 2024-09-12T02:56:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:15 smithi184 ceph-mon[35321]: pgmap v587: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 307 KiB/s wr, 37 op/s 2024-09-12T02:56:15.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:15 smithi016 ceph-mon[30925]: pgmap v587: 97 pgs: 97 active+clean; 250 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 307 KiB/s wr, 37 op/s 2024-09-12T02:56:16.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 777878 9.49 MB/sec execute 318 sec latency 11.991 ms 2024-09-12T02:56:17.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 780797 9.50 MB/sec execute 319 sec latency 12.421 ms 2024-09-12T02:56:17.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:17 smithi033 ceph-mon[35092]: pgmap v588: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 328 KiB/s wr, 35 op/s 2024-09-12T02:56:17.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:17 smithi184 ceph-mon[35321]: pgmap v588: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 328 KiB/s wr, 35 op/s 2024-09-12T02:56:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:17 smithi016 ceph-mon[30925]: pgmap v588: 97 pgs: 97 active+clean; 251 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 328 KiB/s wr, 35 op/s 2024-09-12T02:56:18.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 783054 9.51 MB/sec execute 320 sec latency 12.329 ms 2024-09-12T02:56:19.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 785472 9.52 MB/sec execute 321 sec latency 12.377 ms 2024-09-12T02:56:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:20 smithi016 ceph-mon[30925]: pgmap v589: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 308 KiB/s wr, 34 op/s 2024-09-12T02:56:20.448 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:20 smithi033 ceph-mon[35092]: pgmap v589: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 308 KiB/s wr, 34 op/s 2024-09-12T02:56:20.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:20 smithi184 ceph-mon[35321]: pgmap v589: 97 pgs: 97 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 308 KiB/s wr, 34 op/s 2024-09-12T02:56:20.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 788531 9.55 MB/sec execute 322 sec latency 10.343 ms 2024-09-12T02:56:21.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 791227 9.56 MB/sec execute 323 sec latency 11.460 ms 2024-09-12T02:56:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:22 smithi016 ceph-mon[30925]: pgmap v590: 97 pgs: 97 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 910 KiB/s wr, 125 op/s 2024-09-12T02:56:22.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:22 smithi033 ceph-mon[35092]: pgmap v590: 97 pgs: 97 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 910 KiB/s wr, 125 op/s 2024-09-12T02:56:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:22 smithi184 ceph-mon[35321]: pgmap v590: 97 pgs: 97 active+clean; 258 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 910 KiB/s wr, 125 op/s 2024-09-12T02:56:22.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 793476 9.57 MB/sec execute 324 sec latency 11.973 ms 2024-09-12T02:56:23.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 796121 9.58 MB/sec execute 325 sec latency 11.176 ms 2024-09-12T02:56:24.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:24 smithi016 ceph-mon[30925]: pgmap v591: 97 pgs: 97 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 950 KiB/s wr, 129 op/s 2024-09-12T02:56:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:24 smithi033 ceph-mon[35092]: pgmap v591: 97 pgs: 97 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 950 KiB/s wr, 129 op/s 2024-09-12T02:56:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:24 smithi184 ceph-mon[35321]: pgmap v591: 97 pgs: 97 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 950 KiB/s wr, 129 op/s 2024-09-12T02:56:24.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 799118 9.61 MB/sec execute 326 sec latency 45.588 ms 2024-09-12T02:56:25.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 801850 9.62 MB/sec execute 327 sec latency 17.535 ms 2024-09-12T02:56:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:26 smithi016 ceph-mon[30925]: pgmap v592: 97 pgs: 97 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1014 KiB/s wr, 138 op/s 2024-09-12T02:56:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:26 smithi033 ceph-mon[35092]: pgmap v592: 97 pgs: 97 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1014 KiB/s wr, 138 op/s 2024-09-12T02:56:26.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:26 smithi184 ceph-mon[35321]: pgmap v592: 97 pgs: 97 active+clean; 259 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1014 KiB/s wr, 138 op/s 2024-09-12T02:56:26.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 804187 9.63 MB/sec execute 328 sec latency 12.726 ms 2024-09-12T02:56:27.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 806621 9.64 MB/sec execute 329 sec latency 16.103 ms 2024-09-12T02:56:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:28 smithi016 ceph-mon[30925]: pgmap v593: 97 pgs: 97 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-12T02:56:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:28 smithi033 ceph-mon[35092]: pgmap v593: 97 pgs: 97 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-12T02:56:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:28 smithi184 ceph-mon[35321]: pgmap v593: 97 pgs: 97 active+clean; 265 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-12T02:56:28.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 809802 9.67 MB/sec execute 330 sec latency 15.891 ms 2024-09-12T02:56:29.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 812676 9.68 MB/sec execute 331 sec latency 19.959 ms 2024-09-12T02:56:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:30 smithi016 ceph-mon[30925]: pgmap v594: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:56:30.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:30 smithi033 ceph-mon[35092]: pgmap v594: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:56:30.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:30 smithi184 ceph-mon[35321]: pgmap v594: 97 pgs: 97 active+clean; 269 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:56:30.730 INFO:tasks.workunit.client.0.smithi016.stdout: 1 815023 9.68 MB/sec execute 332 sec latency 15.313 ms 2024-09-12T02:56:31.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 817653 9.70 MB/sec execute 333 sec latency 12.893 ms 2024-09-12T02:56:31.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:56:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:32.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:32 smithi016 ceph-mon[30925]: pgmap v595: 97 pgs: 97 active+clean; 267 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-12T02:56:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:32 smithi033 ceph-mon[35092]: pgmap v595: 97 pgs: 97 active+clean; 267 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-12T02:56:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 -- 172.21.15.16:0/3734240485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 msgr2=0x7fe27010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 --2- 172.21.15.16:0/3734240485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe27010d950 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe268009cd0 tx=0x7fe26802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 -- 172.21.15.16:0/3734240485 shutdown_connections 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 --2- 172.21.15.16:0/3734240485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe27010d950 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 --2- 172.21.15.16:0/3734240485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270100a80 0x7fe270100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 --2- 172.21.15.16:0/3734240485 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270106a80 0x7fe270106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 -- 172.21.15.16:0/3734240485 >> 172.21.15.16:0/3734240485 conn(0x7fe2700fc7a0 msgr2=0x7fe2700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 -- 172.21.15.16:0/3734240485 shutdown_connections 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.526+0000 7fe276bcc640 1 -- 172.21.15.16:0/3734240485 wait complete. 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 Processor -- start 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 -- start start 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270100a80 0x7fe270102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270106a80 0x7fe2701030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe270110e80 con 0x7fe270101420 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe270110d00 con 0x7fe270106a80 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe276bcc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe270111000 con 0x7fe270100a80 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38212/0 (socket says 172.21.15.16:38212) 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe26ffff640 1 -- 172.21.15.16:0/1192596839 learned_addr learned my addr 172.21.15.16:0/1192596839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe274968640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270100a80 0x7fe270102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.527+0000 7fe275169640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270106a80 0x7fe2701030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 -- 172.21.15.16:0/1192596839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270100a80 msgr2=0x7fe270102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270100a80 0x7fe270102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 -- 172.21.15.16:0/1192596839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270106a80 msgr2=0x7fe2701030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270106a80 0x7fe2701030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 -- 172.21.15.16:0/1192596839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2701a68d0 con 0x7fe270101420 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00e9f0 tx=0x7fe25c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26dffb640 1 -- 172.21.15.16:0/1192596839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00cd80 con 0x7fe270101420 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26dffb640 1 -- 172.21.15.16:0/1192596839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25c010910 con 0x7fe270101420 2024-09-12T02:56:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2701a6bc0 con 0x7fe270101420 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe26dffb640 1 -- 172.21.15.16:0/1192596839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c018990 con 0x7fe270101420 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.528+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2701a7100 con 0x7fe270101420 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe26dffb640 1 -- 172.21.15.16:0/1192596839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe25c010ac0 con 0x7fe270101420 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 Processor -- start 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 -- start start 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d176950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe26dffb640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d179280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d0c92e0 con 0x55dd0d16c5d0 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d070ce0 con 0x55dd0d16bcb0 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.531+0000 7fe278fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d0c52a0 con 0x55dd0d176e90 2024-09-12T02:56:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54294/0 (socket says 172.21.15.16:54294) 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe274968640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d176950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 -- 172.21.15.16:0/3309541292 learned_addr learned my addr 172.21.15.16:0/3309541292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d179280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe26dffb640 1 -- 172.21.15.16:0/1192596839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe25c0cf3d0 con 0x7fe270101420 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 -- 172.21.15.16:0/3309541292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 msgr2=0x55dd0d179280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d179280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 -- 172.21.15.16:0/3309541292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 msgr2=0x55dd0d176950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d176950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 -- 172.21.15.16:0/3309541292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd0d055440 con 0x55dd0d16bcb0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe275169640 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe268009a60 tx=0x7fe26802a8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32e8275a833f672d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cd250 tx=0x7fe25c013700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe25affd640 1 -- 172.21.15.16:0/3309541292 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268031670 con 0x55dd0d16bcb0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe25affd640 1 -- 172.21.15.16:0/3309541292 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe268036040 con 0x55dd0d16bcb0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.532+0000 7fe25affd640 1 -- 172.21.15.16:0/3309541292 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268004270 con 0x55dd0d16bcb0 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- 172.21.15.16:0/3309541292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 msgr2=0x55dd0d16c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe268009a60 tx=0x7fe26802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- 172.21.15.16:0/3309541292 shutdown_connections 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d179280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d176950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 --2- 172.21.15.16:0/3309541292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16c090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- 172.21.15.16:0/3309541292 >> 172.21.15.16:0/3309541292 conn(0x55dd0d168240 msgr2=0x55dd0d168620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- 172.21.15.16:0/3309541292 shutdown_connections 2024-09-12T02:56:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- 172.21.15.16:0/3309541292 wait complete. 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.533+0000 7fe278fa5740 1 -- start start 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 Processor -- start 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 0x55dd0d176770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16c5d0 0x55dd0d173a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d173f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d179a60 con 0x55dd0d16bcb0 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d084050 con 0x55dd0d16c5d0 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dd0d179be0 con 0x55dd0d176e90 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe278fa5740 10 client.0 _ll_register_callbacks cb 0x55dd0d0c4160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe275169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 0x55dd0d176770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe275169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 0x55dd0d176770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38234/0 (socket says 172.21.15.16:38234) 2024-09-12T02:56:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe275169640 1 -- 172.21.15.16:0/4230949292 learned_addr learned my addr 172.21.15.16:0/4230949292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.534+0000 7fe275169640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d173f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe274968640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16c5d0 0x55dd0d173a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 -- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16c5d0 msgr2=0x55dd0d173a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16c5d0 0x55dd0d173a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 -- 172.21.15.16:0/4230949292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 msgr2=0x55dd0d176770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 0x55dd0d176770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 -- 172.21.15.16:0/4230949292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0ce3a3c0 con 0x55dd0d176e90 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe275169640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d173f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe268003cf0 tx=0x7fe268033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a188df77bc1093d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.535+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2680039e0 con 0x55dd0d176e90 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe268036040 con 0x55dd0d176e90 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe268003f70 con 0x55dd0d176e90 2024-09-12T02:56:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe268031d00 con 0x55dd0d176e90 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 10 client.26860 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd0d179a60 con 0x55dd0d176e90 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe268036040 con 0x55dd0d176e90 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe268030070 con 0x55dd0d176e90 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 10 client.26860 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 20 client.26860 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d186580 0x55dd0d188970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 4 client.26860 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.536+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd0d0d6850 con 0x55dd0d186580 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.537+0000 7fe274968640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d186580 0x55dd0d188970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.537+0000 7fe274968640 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d186580 0x55dd0d188970 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=192aab989c2540b2 server_cookie=525c43a7054b5e5a in_seq=0 out_seq=0 2024-09-12T02:56:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.537+0000 7fe24ffff640 10 client.26860 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:32.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:32 smithi184 ceph-mon[35321]: pgmap v595: 97 pgs: 97 active+clean; 267 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe24ffff640 1 -- 172.21.15.16:0/4230949292 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd0d0d6850 con 0x55dd0d186580 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe24ffff640 10 client.26860 handle_command_reply: tid=0 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 client.26860 shutdown 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 msgr2=0x55dd0d173f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d173f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe268003cf0 tx=0x7fe268033000 comp rx=0 tx=0).stop 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 shutdown_connections 2024-09-12T02:56:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d186580 0x55dd0d188970 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d176e90 0x55dd0d173f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16c5d0 0x55dd0d173a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4230949292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16bcb0 0x55dd0d176770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe24ffff640 0 client.26860 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 >> 172.21.15.16:0/4230949292 conn(0x55dd0d168240 msgr2=0x55dd0d15eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 shutdown_connections 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 1 -- 172.21.15.16:0/4230949292 wait complete. 2024-09-12T02:56:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.538+0000 7fe278fa5740 20 client.26860 trim_cache size 0 max 16384 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 Processor -- start 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 -- start start 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d168240 0x55dd0d182cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d172a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ab380 con 0x55dd0d16c5d0 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701acec0 con 0x55dd0d16bcb0 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe278fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701aba50 con 0x55dd0d168240 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe275169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d172a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54312/0 (socket says 172.21.15.16:54312) 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 -- 172.21.15.16:0/1895847028 learned_addr learned my addr 172.21.15.16:0/1895847028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe275169640 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d168240 0x55dd0d182cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 -- 172.21.15.16:0/1895847028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d168240 msgr2=0x55dd0d182cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d168240 0x55dd0d182cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 -- 172.21.15.16:0/1895847028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 msgr2=0x55dd0d172a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d172a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.557+0000 7fe274968640 1 -- 172.21.15.16:0/1895847028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd0d176ae0 con 0x55dd0d16bcb0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe274968640 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55dd0d175060 tx=0x7fe264005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac3db6175e2b8fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1895847028 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264010430 con 0x55dd0d16bcb0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1895847028 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264010ab0 con 0x55dd0d16bcb0 2024-09-12T02:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1895847028 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264026be0 con 0x55dd0d16bcb0 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe278fa5740 1 -- 172.21.15.16:0/1895847028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 msgr2=0x55dd0d16f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.558+0000 7fe278fa5740 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55dd0d175060 tx=0x7fe264005f90 comp rx=0 tx=0).stop 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- 172.21.15.16:0/1895847028 shutdown_connections 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d16c5d0 0x55dd0d172a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d16f110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- 172.21.15.16:0/1895847028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d168240 0x55dd0d182cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- 172.21.15.16:0/1895847028 >> 172.21.15.16:0/1895847028 conn(0x55dd0d1740c0 msgr2=0x55dd0d1722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- 172.21.15.16:0/1895847028 shutdown_connections 2024-09-12T02:56:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- 172.21.15.16:0/1895847028 wait complete. 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- start start 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 Processor -- start 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d17a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d16c5d0 0x55dd0d17ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ac7f0 con 0x55dd0d168240 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ac120 con 0x55dd0d16bcb0 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.559+0000 7fe278fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701a93a0 con 0x55dd0d16c5d0 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe278fa5740 10 client.0 _ll_register_callbacks cb 0x7fe264005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38246/0 (socket says 172.21.15.16:38246) 2024-09-12T02:56:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 -- 172.21.15.16:0/4012321865 learned_addr learned my addr 172.21.15.16:0/4012321865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d16c5d0 0x55dd0d17ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe274968640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d17a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 -- 172.21.15.16:0/4012321865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d16c5d0 msgr2=0x55dd0d17ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d16c5d0 0x55dd0d17ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 -- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 msgr2=0x55dd0d17a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d17a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 -- 172.21.15.16:0/4012321865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0d17b060 con 0x55dd0d168240 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe274968640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d17a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe275169640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe268005e30 tx=0x7fe2680316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee23e0cf328e48 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268030b10 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe268053070 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.560+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe264010ab0 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe268030cb0 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 10 client.17481 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd0d1f0cb0 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264010ab0 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe26806b080 con 0x55dd0d168240 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 10 client.17481 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 10 client.17481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 20 client.17481 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d176e90 0x55dd0d1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 4 client.17481 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.561+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd0d0d0d60 con 0x55dd0d176e90 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe274968640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d176e90 0x55dd0d1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe274968640 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d176e90 0x55dd0d1f3200 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b190ce256d5b8aab server_cookie=89b5481c124f84e4 in_seq=0 out_seq=0 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe24ffff640 10 client.17481 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe24ffff640 1 -- 172.21.15.16:0/4012321865 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dd0d0d0d60 con 0x55dd0d176e90 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe24ffff640 10 client.17481 handle_command_reply: tid=0 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 client.17481 shutdown 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 msgr2=0x55dd0d17f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe268005e30 tx=0x7fe2680316f0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 shutdown_connections 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dd0d176e90 0x55dd0d1f3200 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd0d16c5d0 0x55dd0d17ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dd0d16bcb0 0x55dd0d17a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 --2- 172.21.15.16:0/4012321865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd0d168240 0x55dd0d17f260 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.562+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 >> 172.21.15.16:0/4012321865 conn(0x55dd0d1740c0 msgr2=0x55dd0d175280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe24ffff640 0 client.17481 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 shutdown_connections 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe278fa5740 1 -- 172.21.15.16:0/4012321865 wait complete. 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe278fa5740 20 client.17481 trim_cache size 0 max 16384 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:56:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:56:32.572 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:56:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 msgr2=0x7fe2600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cd250 tx=0x7fe25c013700 comp rx=0 tx=0).stop 2024-09-12T02:56:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 msgr2=0x7fe270102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.563+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00e9f0 tx=0x7fe25c00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 shutdown_connections 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe270106a80 0x7fe2701030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe2600a2b80 0x7fe2600a5040 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe270101420 0x7fe270102b90 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 --2- 172.21.15.16:0/1192596839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe270100a80 0x7fe270102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 >> 172.21.15.16:0/1192596839 conn(0x7fe2700fc7a0 msgr2=0x7fe270104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 shutdown_connections 2024-09-12T02:56:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:32.564+0000 7fe276bcc640 1 -- 172.21.15.16:0/1192596839 wait complete. 2024-09-12T02:56:32.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 820562 9.72 MB/sec execute 334 sec latency 12.254 ms 2024-09-12T02:56:33.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:33 smithi016 ceph-mon[30925]: pgmap v596: 97 pgs: 97 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-09-12T02:56:33.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:33 smithi033 ceph-mon[35092]: pgmap v596: 97 pgs: 97 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-09-12T02:56:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:33 smithi184 ceph-mon[35321]: pgmap v596: 97 pgs: 97 active+clean; 272 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-09-12T02:56:33.555 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:56:33.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:56:33.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 823743 9.75 MB/sec execute 335 sec latency 11.299 ms 2024-09-12T02:56:33.822 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:34.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 -- 172.21.15.16:0/3984919129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880100e20 msgr2=0x7fe880101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:34.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 --2- 172.21.15.16:0/3984919129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880100e20 0x7fe880101200 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009900 tx=0x7fe87c02a480 comp rx=0 tx=0).stop 2024-09-12T02:56:34.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 -- 172.21.15.16:0/3984919129 shutdown_connections 2024-09-12T02:56:34.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 --2- 172.21.15.16:0/3984919129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 --2- 172.21.15.16:0/3984919129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880102570 0x7fe8801029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 --2- 172.21.15.16:0/3984919129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880100e20 0x7fe880101200 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 -- 172.21.15.16:0/3984919129 >> 172.21.15.16:0/3984919129 conn(0x7fe8800fc7d0 msgr2=0x7fe8800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 -- 172.21.15.16:0/3984919129 shutdown_connections 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.293+0000 7fe887b3b640 1 -- 172.21.15.16:0/3984919129 wait complete. 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 Processor -- start 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 -- start start 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 0x7fe88019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 0x7fe88019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe880110f80 con 0x7fe880102570 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe880110e00 con 0x7fe880100e20 2024-09-12T02:56:34.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe887b3b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe880111100 con 0x7fe880102ef0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe8858d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 0x7fe88019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe8850d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 0x7fe88019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe8858d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 0x7fe88019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54344/0 (socket says 172.21.15.16:54344) 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe8858d7640 1 -- 172.21.15.16:0/2265845107 learned_addr learned my addr 172.21.15.16:0/2265845107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.294+0000 7fe8860d8640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 -- 172.21.15.16:0/2265845107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 msgr2=0x7fe88019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 0x7fe88019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 -- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 msgr2=0x7fe88019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 0x7fe88019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 -- 172.21.15.16:0/2265845107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88019f660 con 0x7fe880102ef0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8850d6640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 0x7fe88019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe8860d8640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88019ef80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe878009fd0 tx=0x7fe87800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87800ce20 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe878005020 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88019f950 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878010720 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.295+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8801a7230 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.296+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe880101200 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.298+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe878005540 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.298+0000 7fe876ffd640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 0x7fe86c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.298+0000 7fe8858d7640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 0x7fe86c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.298+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe8780cf760 con 0x7fe880102ef0 2024-09-12T02:56:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.299+0000 7fe8858d7640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 0x7fe86c0a5040 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe87c005770 tx=0x7fe87c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.301+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe87809c700 con 0x7fe880102ef0 2024-09-12T02:56:34.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.529+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe88019ba10 con 0x7fe880102ef0 2024-09-12T02:56:34.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.530+0000 7fe876ffd640 1 -- 172.21.15.16:0/2265845107 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fe88019ba10 con 0x7fe880102ef0 2024-09-12T02:56:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:56:34.532 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:56:34.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:56:34.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 msgr2=0x7fe86c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 0x7fe86c0a5040 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe87c005770 tx=0x7fe87c005a20 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 msgr2=0x7fe88019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88019ef80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe878009fd0 tx=0x7fe87800ef10 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 shutdown_connections 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe880102ef0 0x7fe88019ef80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe86c0a2b80 0x7fe86c0a5040 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe880102570 0x7fe88019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 --2- 172.21.15.16:0/2265845107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe880100e20 0x7fe88019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 >> 172.21.15.16:0/2265845107 conn(0x7fe8800fc7d0 msgr2=0x7fe88006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 shutdown_connections 2024-09-12T02:56:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:34.532+0000 7fe887b3b640 1 -- 172.21.15.16:0/2265845107 wait complete. 2024-09-12T02:56:34.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 826242 9.75 MB/sec execute 336 sec latency 37.306 ms 2024-09-12T02:56:35.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:56:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:35 smithi033 ceph-mon[35092]: pgmap v597: 97 pgs: 97 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 257 op/s 2024-09-12T02:56:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2265845107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:35 smithi184 ceph-mon[35321]: pgmap v597: 97 pgs: 97 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 257 op/s 2024-09-12T02:56:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2265845107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:35.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:35 smithi016 ceph-mon[30925]: pgmap v597: 97 pgs: 97 active+clean; 273 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 257 op/s 2024-09-12T02:56:35.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2265845107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:56:35.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 828448 9.76 MB/sec execute 337 sec latency 16.375 ms 2024-09-12T02:56:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 -- 172.21.15.16:0/3426436466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 msgr2=0x7f68e810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 --2- 172.21.15.16:0/3426436466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e810ec30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009be0 tx=0x7f68e002a6c0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 -- 172.21.15.16:0/3426436466 shutdown_connections 2024-09-12T02:56:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 --2- 172.21.15.16:0/3426436466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e810ec30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 --2- 172.21.15.16:0/3426436466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 0x7f68e8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 --2- 172.21.15.16:0/3426436466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 -- 172.21.15.16:0/3426436466 >> 172.21.15.16:0/3426436466 conn(0x7f68e80fc7d0 msgr2=0x7f68e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 -- 172.21.15.16:0/3426436466 shutdown_connections 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.034+0000 7f68f0755640 1 -- 172.21.15.16:0/3426436466 wait complete. 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 Processor -- start 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 -- start start 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 0x7f68e8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8112090 con 0x7f68e8102d90 2024-09-12T02:56:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8111f10 con 0x7f68e8103740 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68f0755640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8112210 con 0x7f68e8108290 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39014/0 (socket says 172.21.15.16:39014) 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3877232656 learned_addr learned my addr 172.21.15.16:0/3877232656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68edcf0640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 0x7f68e8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68eecf2640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3877232656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 msgr2=0x7f68e819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3877232656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 msgr2=0x7f68e8198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 0x7f68e8198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.035+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3877232656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e819d4a0 con 0x7f68e8102d90 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f68e400e9c0 tx=0x7f68e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68df7fe640 1 -- 172.21.15.16:0/3877232656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e400cde0 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e819d790 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68df7fe640 1 -- 172.21.15.16:0/3877232656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e40035f0 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68df7fe640 1 -- 172.21.15.16:0/3877232656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4010690 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.036+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e81a5070 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.038+0000 7f68df7fe640 1 -- 172.21.15.16:0/3877232656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f68e40037a0 con 0x7f68e8102d90 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.038+0000 7f68f2b2e740 1 Processor -- start 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 -- start start 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf12c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf0922f0 con 0x55a5cf12db40 2024-09-12T02:56:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf039cf0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf08e2b0 con 0x55a5cf137370 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68df7fe640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 0x7f68d40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68edcf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68edcf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55352/0 (socket says 172.21.15.16:55352) 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68eecf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf12c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46560/0 (socket says 172.21.15.16:46560) 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68edcf0640 1 -- 172.21.15.16:0/2298175498 learned_addr learned my addr 172.21.15.16:0/2298175498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.039+0000 7f68eecf2640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 0x7f68d40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68df7fe640 1 -- 172.21.15.16:0/3877232656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f68e40cf380 con 0x7f68e8102d90 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2298175498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 msgr2=0x55a5cf139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2298175498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 msgr2=0x55a5cf12c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf12c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2298175498 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5cf01e3a0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68e40ccdd0 tx=0x7f68e4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1569c129ce6aabc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68eecf2640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 0x7f68d40a5040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f68e0004af0 tx=0x7f68e0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68dcff9640 1 -- 172.21.15.16:0/2298175498 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e409b4f0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68dcff9640 1 -- 172.21.15.16:0/2298175498 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e409c290 con 0x55a5cf12c5d0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.040+0000 7f68dcff9640 1 -- 172.21.15.16:0/2298175498 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4098200 con 0x55a5cf12c5d0 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2298175498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 msgr2=0x55a5cf12ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68e40ccdd0 tx=0x7f68e4009560 comp rx=0 tx=0).stop 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2298175498 shutdown_connections 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf12ca10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2298175498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf12c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2298175498 >> 172.21.15.16:0/2298175498 conn(0x55a5cf0f5310 msgr2=0x55a5cf128c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2298175498 shutdown_connections 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.041+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2298175498 wait complete. 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 -- start start 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 Processor -- start 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf134020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf134610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf13a070 con 0x55a5cf12db40 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf04d060 con 0x55a5cf12c5d0 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a5cf13a1f0 con 0x55a5cf137370 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68f2b2e740 10 client.0 _ll_register_callbacks cb 0x55a5ceda08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:36.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68edcf0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46568/0 (socket says 172.21.15.16:46568) 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2396184369 learned_addr learned my addr 172.21.15.16:0/2396184369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68eecf2640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2396184369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 msgr2=0x55a5cf134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 msgr2=0x55a5cf134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf134020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.042+0000 7f68ee4f1640 1 -- 172.21.15.16:0/2396184369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5cee033c0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68edcf0640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68e409bb20 tx=0x7f68e409b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3acc6202153c5bd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40b8070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f68e409b1f0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68e409c290 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f68e40a4800 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68f2b2e740 10 client.26990 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a5cf145b10 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e409c290 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.043+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f68e4083020 con 0x55a5cf12c5d0 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 10 client.26990 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 10 client.26990 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 20 client.26990 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:36.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf146e60 0x55a5cf149250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 4 client.26990 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5cf09f860 con 0x55a5cf146e60 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68edcf0640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf146e60 0x55a5cf149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68edcf0640 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf146e60 0x55a5cf149250 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82fc6436043eeb83 server_cookie=77149682f4fa4d42 in_seq=0 out_seq=0 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.044+0000 7f68d1ffb640 10 client.26990 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68d1ffb640 1 -- 172.21.15.16:0/2396184369 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a5cf09f860 con 0x55a5cf146e60 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68d1ffb640 10 client.26990 handle_command_reply: tid=0 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 client.26990 shutdown 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 msgr2=0x55a5cf136d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68e409bb20 tx=0x7f68e409b4f0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 shutdown_connections 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12db40 0x55a5cf134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf146e60 0x55a5cf149250 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/2396184369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12c5d0 0x55a5cf136d70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68d1ffb640 0 client.26990 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 >> 172.21.15.16:0/2396184369 conn(0x55a5cf0f5310 msgr2=0x55a5cf12e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 shutdown_connections 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 1 -- 172.21.15.16:0/2396184369 wait complete. 2024-09-12T02:56:36.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.045+0000 7f68f2b2e740 20 client.26990 trim_cache size 0 max 16384 2024-09-12T02:56:36.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 Processor -- start 2024-09-12T02:56:36.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 -- start start 2024-09-12T02:56:36.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf14e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf14ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81ab3c0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81acf00 con 0x55a5cf12db40 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81aba90 con 0x55a5cf137370 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39030/0 (socket says 172.21.15.16:39030) 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 -- 172.21.15.16:0/1381509426 learned_addr learned my addr 172.21.15.16:0/1381509426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68eecf2640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf14ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68edcf0640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 -- 172.21.15.16:0/1381509426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 msgr2=0x55a5cf14ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf14ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 -- 172.21.15.16:0/1381509426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 msgr2=0x55a5cf14e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf14e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 -- 172.21.15.16:0/1381509426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5cf134940 con 0x55a5cf12c5d0 2024-09-12T02:56:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68edcf0640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf14e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.059+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c6a0 tx=0x7f68e4003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3618694020eae749 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68d3fff640 1 -- 172.21.15.16:0/1381509426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40b8070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68d3fff640 1 -- 172.21.15.16:0/1381509426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e4080070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68d3fff640 1 -- 172.21.15.16:0/1381509426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40cceb0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 -- 172.21.15.16:0/1381509426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 msgr2=0x55a5cf143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c6a0 tx=0x7f68e4003d20 comp rx=0 tx=0).stop 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 -- 172.21.15.16:0/1381509426 shutdown_connections 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf137370 0x55a5cf14ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf14e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/1381509426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf12c5d0 0x55a5cf143310 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 -- 172.21.15.16:0/1381509426 >> 172.21.15.16:0/1381509426 conn(0x55a5cf0f5310 msgr2=0x55a5cf123a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 -- 172.21.15.16:0/1381509426 shutdown_connections 2024-09-12T02:56:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.060+0000 7f68f2b2e740 1 -- 172.21.15.16:0/1381509426 wait complete. 2024-09-12T02:56:36.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 -- start start 2024-09-12T02:56:36.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 Processor -- start 2024-09-12T02:56:36.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf13f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81ac830 con 0x55a5cf137370 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81ac160 con 0x55a5cf12db40 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68e81a93a0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68f2b2e740 10 client.0 _ll_register_callbacks cb 0x7f68e4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68eecf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68edcf0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf13f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68ee4f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55388/0 (socket says 172.21.15.16:55388) 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68eecf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39044/0 (socket says 172.21.15.16:39044) 2024-09-12T02:56:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.061+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3037773420 learned_addr learned my addr 172.21.15.16:0/3037773420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3037773420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 msgr2=0x55a5cf13f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf13f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 msgr2=0x55a5cf130690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 -- 172.21.15.16:0/3037773420 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5cf130bd0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68eecf2640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68ee4f1640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f68e4097ff0 tx=0x7f68e4097e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec11d161db4f709d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40b8070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f68e409b1f0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68e4080070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f68e409b4f0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68f2b2e740 10 client.26890 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.062+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a5cf130bd0 con 0x55a5cf12c5d0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e4080070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f68e4083070 con 0x55a5cf12c5d0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 10 client.26890 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 10 client.26890 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 20 client.26890 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf1b12b0 0x55a5cf1b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 4 client.26890 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5cf099d70 con 0x55a5cf1b12b0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68edcf0640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf1b12b0 0x55a5cf1b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68edcf0640 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf1b12b0 0x55a5cf1b36a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c28e7d89a770fe4d server_cookie=b50c8ed6c2bc160d in_seq=0 out_seq=0 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.063+0000 7f68d1ffb640 10 client.26890 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:56:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68d1ffb640 1 -- 172.21.15.16:0/3037773420 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5cf099d70 con 0x55a5cf1b12b0 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68d1ffb640 10 client.26890 handle_command_reply: tid=0 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 client.26890 shutdown 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 msgr2=0x55a5cf13f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f68e4097ff0 tx=0x7f68e4097e30 comp rx=0 tx=0).stop 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 shutdown_connections 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5cf137370 0x55a5cf130690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55a5cf1b12b0 0x55a5cf1b36a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5cf12db40 0x55a5cf13f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 --2- 172.21.15.16:0/3037773420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5cf12c5d0 0x55a5cf13f480 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68d1ffb640 0 client.26890 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 >> 172.21.15.16:0/3037773420 conn(0x55a5cf0f5310 msgr2=0x55a5cf132910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 shutdown_connections 2024-09-12T02:56:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 1 -- 172.21.15.16:0/3037773420 wait complete. 2024-09-12T02:56:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f2b2e740 20 client.26890 trim_cache size 0 max 16384 2024-09-12T02:56:36.073 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:56:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 msgr2=0x7f68d40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 0x7f68d40a5040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f68e0004af0 tx=0x7f68e0005fd0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 msgr2=0x7f68e81984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.064+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f68e400e9c0 tx=0x7f68e400ee80 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 shutdown_connections 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e8108290 0x7f68e819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68e8103740 0x7f68e8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f68d40a2b80 0x7f68d40a5040 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 --2- 172.21.15.16:0/3877232656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e8102d90 0x7f68e81984f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 >> 172.21.15.16:0/3877232656 conn(0x7f68e80fc7d0 msgr2=0x7f68e8106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 shutdown_connections 2024-09-12T02:56:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:36.065+0000 7f68f0755640 1 -- 172.21.15.16:0/3877232656 wait complete. 2024-09-12T02:56:36.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 831183 9.78 MB/sec execute 338 sec latency 35.101 ms 2024-09-12T02:56:36.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:56:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:37.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:37 smithi016 ceph-mon[30925]: pgmap v598: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-09-12T02:56:37.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:37 smithi033 ceph-mon[35092]: pgmap v598: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-09-12T02:56:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:37 smithi184 ceph-mon[35321]: pgmap v598: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 -- 172.21.15.16:0/2598481962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 msgr2=0x7f4b60107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 --2- 172.21.15.16:0/2598481962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60107b80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009cd0 tx=0x7f4b5002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 -- 172.21.15.16:0/2598481962 shutdown_connections 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 --2- 172.21.15.16:0/2598481962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60108290 0x7f4b6010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 --2- 172.21.15.16:0/2598481962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60107b80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 --2- 172.21.15.16:0/2598481962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60102d90 0x7f4b60103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 -- 172.21.15.16:0/2598481962 >> 172.21.15.16:0/2598481962 conn(0x7f4b600fc7d0 msgr2=0x7f4b600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 -- 172.21.15.16:0/2598481962 shutdown_connections 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.552+0000 7f4b67119640 1 -- 172.21.15.16:0/2598481962 wait complete. 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 Processor -- start 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 -- start start 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 0x7f4b60198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60108290 0x7f4b6019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60112090 con 0x7f4b60103740 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60111f10 con 0x7f4b60102d90 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b67119640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60112210 con 0x7f4b60108290 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39056/0 (socket says 172.21.15.16:39056) 2024-09-12T02:56:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b64eb5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 0x7f4b60198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 -- 172.21.15.16:0/3299646383 learned_addr learned my addr 172.21.15.16:0/3299646383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b656b6640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60108290 0x7f4b6019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 -- 172.21.15.16:0/3299646383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60108290 msgr2=0x7f4b6019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60108290 0x7f4b6019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 -- 172.21.15.16:0/3299646383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 msgr2=0x7f4b60198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 0x7f4b60198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b5ffff640 1 -- 172.21.15.16:0/3299646383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6019d5c0 con 0x7f4b60103740 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.553+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 0x7f4b60198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4b5002ae00 tx=0x7f4b50031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b5dffb640 1 -- 172.21.15.16:0/3299646383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50003e00 con 0x7f4b60103740 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b5dffb640 1 -- 172.21.15.16:0/3299646383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50004520 con 0x7f4b60103740 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6019d850 con 0x7f4b60103740 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b5dffb640 1 -- 172.21.15.16:0/3299646383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50030de0 con 0x7f4b60103740 2024-09-12T02:56:37.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.554+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b601a5190 con 0x7f4b60103740 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b5dffb640 1 -- 172.21.15.16:0/3299646383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4b500046d0 con 0x7f4b60103740 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 Processor -- start 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 -- start start 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9e9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9fab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c57d94f2f0 con 0x55c57d9eab40 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c57d8f6cf0 con 0x55c57d9e95d0 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b694f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c57d94b2b0 con 0x55c57d9f8780 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b64eb5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b656b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b64eb5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46604/0 (socket says 172.21.15.16:46604) 2024-09-12T02:56:37.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b64eb5640 1 -- 172.21.15.16:0/3899058196 learned_addr learned my addr 172.21.15.16:0/3899058196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.557+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9fab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b5dffb640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 0x7f4b4c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 -- 172.21.15.16:0/3899058196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 msgr2=0x55c57d9fab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9fab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 -- 172.21.15.16:0/3899058196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 msgr2=0x55c57d9e9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9e9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 -- 172.21.15.16:0/3899058196 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c57d8db3a0 con 0x55c57d9e95d0 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b656b6640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 0x7f4b4c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b5dffb640 1 -- 172.21.15.16:0/3299646383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b5002e030 con 0x7f4b60103740 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b480115c0 tx=0x7f4b48011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d58b1bb3eb4e49 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b56ffd640 1 -- 172.21.15.16:0/3899058196 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4800f640 con 0x55c57d9e95d0 2024-09-12T02:56:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b56ffd640 1 -- 172.21.15.16:0/3899058196 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b4800fcc0 con 0x55c57d9e95d0 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.558+0000 7f4b56ffd640 1 -- 172.21.15.16:0/3899058196 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b48019d60 con 0x55c57d9e95d0 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- 172.21.15.16:0/3899058196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 msgr2=0x55c57d9e9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b656b6640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 0x7f4b4c0a5040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4b580058c0 tx=0x7f4b58004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b480115c0 tx=0x7f4b48011a80 comp rx=0 tx=0).stop 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- 172.21.15.16:0/3899058196 shutdown_connections 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9fab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e9a10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 --2- 172.21.15.16:0/3899058196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9e9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- 172.21.15.16:0/3899058196 >> 172.21.15.16:0/3899058196 conn(0x55c57d9b2310 msgr2=0x55c57d9e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- 172.21.15.16:0/3899058196 shutdown_connections 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- 172.21.15.16:0/3899058196 wait complete. 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 -- start start 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.559+0000 7f4b694f2740 1 Processor -- start 2024-09-12T02:56:37.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9f8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c57d9fb480 con 0x55c57d9eab40 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c57d90a060 con 0x55c57d9e95d0 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c57d9fb600 con 0x55c57d9f8780 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b694f2740 10 client.0 _ll_register_callbacks cb 0x55c57d65d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b64eb5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39074/0 (socket says 172.21.15.16:39074) 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 -- 172.21.15.16:0/2483624458 learned_addr learned my addr 172.21.15.16:0/2483624458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b656b6640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 -- 172.21.15.16:0/2483624458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 msgr2=0x55c57d9f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 -- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 msgr2=0x55c57d9f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 -- 172.21.15.16:0/2483624458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57d6c03c0 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009ca0 tx=0x7f4b500b9100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c1fb4bedacdcb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.560+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b500bc7c0 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b500305a0 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b4800fcc0 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b50009540 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 10 client.17517 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c57da06ed0 con 0x55c57d9eab40 2024-09-12T02:56:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b4800fcc0 con 0x55c57d9eab40 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b5004e070 con 0x55c57d9eab40 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 10 client.17517 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 10 client.17517 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 20 client.17517 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da08330 0x55c57da0a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 4 client.17517 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c57d95c860 con 0x55c57da08330 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.561+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da08330 0x55c57da0a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da08330 0x55c57da0a780 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a9144bb0c29bed24 server_cookie=6f4895aa4f933d2e in_seq=0 out_seq=0 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b3ffff640 10 client.17517 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b3ffff640 1 -- 172.21.15.16:0/2483624458 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c57d95c860 con 0x55c57da08330 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b3ffff640 10 client.17517 handle_command_reply: tid=0 2024-09-12T02:56:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b694f2740 1 client.17517 shutdown 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 msgr2=0x55c57d9f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.562+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009ca0 tx=0x7f4b500b9100 comp rx=0 tx=0).stop 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 shutdown_connections 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da08330 0x55c57da0a780 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57d9f53d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 --2- 172.21.15.16:0/2483624458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b3ffff640 0 client.17517 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 >> 172.21.15.16:0/2483624458 conn(0x55c57d9b2310 msgr2=0x55c57d9eb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 shutdown_connections 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 1 -- 172.21.15.16:0/2483624458 wait complete. 2024-09-12T02:56:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.563+0000 7f4b694f2740 20 client.17517 trim_cache size 0 max 16384 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 Processor -- start 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 -- start start 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ab300 con 0x55c57d9eab40 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ace40 con 0x55c57d9e95d0 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b694f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ab9d0 con 0x55c57d9f8780 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39078/0 (socket says 172.21.15.16:39078) 2024-09-12T02:56:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b5ffff640 1 -- 172.21.15.16:0/1093751038 learned_addr learned my addr 172.21.15.16:0/1093751038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.577+0000 7f4b656b6640 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 -- 172.21.15.16:0/1093751038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 msgr2=0x55c57d9f4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 -- 172.21.15.16:0/1093751038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 msgr2=0x55c57d9e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 -- 172.21.15.16:0/1093751038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c57d9ea300 con 0x55c57d9eab40 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4b500ee470 tx=0x7f4b5002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4aebed62652a17 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b567fc640 1 -- 172.21.15.16:0/1093751038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b500bcc30 con 0x55c57d9eab40 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b567fc640 1 -- 172.21.15.16:0/1093751038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50036070 con 0x55c57d9eab40 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.578+0000 7f4b567fc640 1 -- 172.21.15.16:0/1093751038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b500bce60 con 0x55c57d9eab40 2024-09-12T02:56:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- 172.21.15.16:0/1093751038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 msgr2=0x55c57da04720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4b500ee470 tx=0x7f4b5002a8e0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- 172.21.15.16:0/1093751038 shutdown_connections 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f8780 0x55c57d9f4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9eab40 0x55c57da04720 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 --2- 172.21.15.16:0/1093751038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57d9e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- 172.21.15.16:0/1093751038 >> 172.21.15.16:0/1093751038 conn(0x55c57d9b2310 msgr2=0x55c57d9e5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- 172.21.15.16:0/1093751038 shutdown_connections 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- 172.21.15.16:0/1093751038 wait complete. 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.579+0000 7f4b694f2740 1 -- start start 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 Processor -- start 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57da00f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 0x55c57d9f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ac770 con 0x55c57d9f8780 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ac0a0 con 0x55c57d9e95d0 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601a9300 con 0x55c57d9f2570 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b694f2740 10 client.0 _ll_register_callbacks cb 0x7f4b500f3bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39092/0 (socket says 172.21.15.16:39092) 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57da00f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b5ffff640 1 -- 172.21.15.16:0/4156476295 learned_addr learned my addr 172.21.15.16:0/4156476295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b656b6640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 0x55c57d9f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 -- 172.21.15.16:0/4156476295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 msgr2=0x55c57d9f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 0x55c57d9f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 -- 172.21.15.16:0/4156476295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 msgr2=0x55c57d9f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b64eb5640 1 -- 172.21.15.16:0/4156476295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57da72020 con 0x55c57d9e95d0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b656b6640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 0x55c57d9f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.580+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b64eb5640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57da00f20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009350 tx=0x7f4b48019800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6651903150f477 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b48019e50 con 0x55c57d9e95d0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b48005040 con 0x55c57d9e95d0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b50036070 con 0x55c57d9e95d0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b48005240 con 0x55c57d9e95d0 2024-09-12T02:56:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b694f2740 10 client.27014 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c57da732d0 con 0x55c57d9e95d0 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.581+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50036070 con 0x55c57d9e95d0 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b4802b020 con 0x55c57d9e95d0 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 10 client.27014 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 10 client.27014 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 20 client.27014 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da73830 0x55c57da75c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 4 client.27014 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c57d956d70 con 0x55c57da73830 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da73830 0x55c57da75c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b5ffff640 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da73830 0x55c57da75c50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=178080300cbdf7a2 server_cookie=cbd83bed132efb17 in_seq=0 out_seq=0 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b3ffff640 10 client.27014 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b3ffff640 1 -- 172.21.15.16:0/4156476295 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c57d956d70 con 0x55c57da73830 2024-09-12T02:56:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b3ffff640 10 client.27014 handle_command_reply: tid=0 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 1 client.27014 shutdown 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 msgr2=0x55c57da00f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.582+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57da00f20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009350 tx=0x7f4b48019800 comp rx=0 tx=0).stop 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 shutdown_connections 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55c57da73830 0x55c57da75c50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c57d9f2570 0x55c57d9f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c57d9f8780 0x55c57d9f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b3ffff640 0 client.27014 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 --2- 172.21.15.16:0/4156476295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c57d9e95d0 0x55c57da00f20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 >> 172.21.15.16:0/4156476295 conn(0x55c57d9b2310 msgr2=0x55c57d9e5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 shutdown_connections 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 1 -- 172.21.15.16:0/4156476295 wait complete. 2024-09-12T02:56:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b694f2740 20 client.27014 trim_cache size 0 max 16384 2024-09-12T02:56:37.598 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:56:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 msgr2=0x7f4b4c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 0x7f4b4c0a5040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4b580058c0 tx=0x7f4b58004730 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 msgr2=0x7f4b60198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4b5002ae00 tx=0x7f4b50031e60 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 shutdown_connections 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b60108290 0x7f4b6019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b4c0a2b80 0x7f4b4c0a5040 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b60103740 0x7f4b60198b50 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 --2- 172.21.15.16:0/3299646383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b60102d90 0x7f4b60198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.583+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 >> 172.21.15.16:0/3299646383 conn(0x7f4b600fc7d0 msgr2=0x7f4b60105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.584+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 shutdown_connections 2024-09-12T02:56:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:37.584+0000 7f4b67119640 1 -- 172.21.15.16:0/3299646383 wait complete. 2024-09-12T02:56:37.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 834125 9.79 MB/sec execute 339 sec latency 35.276 ms 2024-09-12T02:56:38.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:56:38.662 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:38.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 836775 9.80 MB/sec execute 340 sec latency 11.332 ms 2024-09-12T02:56:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 -- 172.21.15.16:0/2283739260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174102570 msgr2=0x7f91741029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 --2- 172.21.15.16:0/2283739260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174102570 0x7f91741029b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9168009900 tx=0x7f916802a480 comp rx=0 tx=0).stop 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 -- 172.21.15.16:0/2283739260 shutdown_connections 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 --2- 172.21.15.16:0/2283739260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102ef0 0x7f917410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 --2- 172.21.15.16:0/2283739260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174102570 0x7f91741029b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 --2- 172.21.15.16:0/2283739260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174100e20 0x7f9174101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 -- 172.21.15.16:0/2283739260 >> 172.21.15.16:0/2283739260 conn(0x7f91740fc7d0 msgr2=0x7f91740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 -- 172.21.15.16:0/2283739260 shutdown_connections 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.146+0000 7f917a632640 1 -- 172.21.15.16:0/2283739260 wait complete. 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 Processor -- start 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 -- start start 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 0x7f9174198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174102ef0 0x7f917419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9174110f80 con 0x7f9174100e20 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9174110e00 con 0x7f9174102570 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f917a632640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9174111100 con 0x7f9174102ef0 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.147+0000 7f9173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 0x7f9174198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46660/0 (socket says 172.21.15.16:46660) 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 -- 172.21.15.16:0/1888756319 learned_addr learned my addr 172.21.15.16:0/1888756319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f9178bcf640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174102ef0 0x7f917419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 -- 172.21.15.16:0/1888756319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174102ef0 msgr2=0x7f917419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174102ef0 0x7f917419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 -- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 msgr2=0x7f9174198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 0x7f9174198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 -- 172.21.15.16:0/1888756319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917419d3e0 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f9173fff640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 0x7f9174198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91737fe640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9168006010 tx=0x7f9168003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91717fa640 1 -- 172.21.15.16:0/1888756319 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916803f070 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91717fa640 1 -- 172.21.15.16:0/1888756319 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9168031c30 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917419d670 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.148+0000 7f91717fa640 1 -- 172.21.15.16:0/1888756319 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168035030 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.149+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91741a4fb0 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f91717fa640 1 -- 172.21.15.16:0/1888756319 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9168042070 con 0x7f9174102570 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 Processor -- start 2024-09-12T02:56:39.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 -- start start 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 0x55d339c93090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339ca4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d339bf92f0 con 0x55d339c94b40 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d339ba0cf0 con 0x55d339c935d0 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.151+0000 7f917ca0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d339bf52b0 con 0x55d339ca2780 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9178bcf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 0x55d339c93090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46664/0 (socket says 172.21.15.16:46664) 2024-09-12T02:56:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9178bcf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 0x55d339c93090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39122/0 (socket says 172.21.15.16:39122) 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 -- 172.21.15.16:0/2796202542 learned_addr learned my addr 172.21.15.16:0/2796202542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f91737fe640 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339ca4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f91717fa640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 0x7f915c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 -- 172.21.15.16:0/2796202542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 msgr2=0x55d339ca4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339ca4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 -- 172.21.15.16:0/2796202542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 msgr2=0x55d339c93090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 0x55d339c93090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 -- 172.21.15.16:0/2796202542 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d339b853a0 con 0x55d339c935d0 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9178bcf640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 0x7f915c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f91717fa640 1 -- 172.21.15.16:0/1888756319 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91680ef6c0 con 0x7f9174102570 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f9173fff640 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f91640115f0 tx=0x7f9164011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab711bcb6fede85f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f915a7fc640 1 -- 172.21.15.16:0/2796202542 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916401e070 con 0x55d339c935d0 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f915a7fc640 1 -- 172.21.15.16:0/2796202542 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916400fb20 con 0x55d339c935d0 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.152+0000 7f915a7fc640 1 -- 172.21.15.16:0/2796202542 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164019c40 con 0x55d339c935d0 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 -- 172.21.15.16:0/2796202542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 msgr2=0x55d339c93a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f9178bcf640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 0x7f915c0a5040 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f916c003a30 tx=0x7f916c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f91640115f0 tx=0x7f9164011ab0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 -- 172.21.15.16:0/2796202542 shutdown_connections 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339ca4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339c93a10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 --2- 172.21.15.16:0/2796202542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c94b40 0x55d339c93090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 -- 172.21.15.16:0/2796202542 >> 172.21.15.16:0/2796202542 conn(0x55d339c5c310 msgr2=0x55d339c8fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 -- 172.21.15.16:0/2796202542 shutdown_connections 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.153+0000 7f917ca0b740 1 -- 172.21.15.16:0/2796202542 wait complete. 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 -- start start 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 Processor -- start 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c94b40 0x55d339c9f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339c9fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d339ca5480 con 0x55d339c935d0 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d339bb4060 con 0x55d339c94b40 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d339ca5600 con 0x55d339ca2780 2024-09-12T02:56:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f917ca0b740 10 client.0 _ll_register_callbacks cb 0x55d3399078c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39132/0 (socket says 172.21.15.16:39132) 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9173fff640 1 -- 172.21.15.16:0/342030334 learned_addr learned my addr 172.21.15.16:0/342030334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9178bcf640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339c9fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f91737fe640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c94b40 0x55d339c9f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9173fff640 1 -- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 msgr2=0x55d339c9fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.154+0000 7f9173fff640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339c9fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f9173fff640 1 -- 172.21.15.16:0/342030334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c94b40 msgr2=0x55d339c9f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f9173fff640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c94b40 0x55d339c9f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f9173fff640 1 -- 172.21.15.16:0/342030334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d33996a3c0 con 0x55d339c935d0 2024-09-12T02:56:39.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f9173fff640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9164005e00 tx=0x7f916400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3427cc6b728dd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f91537fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164027060 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91640199c0 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f916400fb20 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f916401e040 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f917ca0b740 10 client.17535 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.155+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d339cb0f20 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916400fb20 con 0x55d339c935d0 2024-09-12T02:56:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9164005070 con 0x55d339c935d0 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 10 client.17535 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 10 client.17535 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 20 client.17535 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339cb2320 0x55d339cb4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 4 client.17535 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d339c06860 con 0x55d339cb2320 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f91737fe640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339cb2320 0x55d339cb4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.156+0000 7f91537fe640 10 client.17535 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:56:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.157+0000 7f91737fe640 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339cb2320 0x55d339cb4710 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4453fa25b948cc06 server_cookie=9e796c6b7196c7ac in_seq=0 out_seq=0 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.157+0000 7f91537fe640 1 -- 172.21.15.16:0/342030334 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f916809ae50 con 0x55d339cb2320 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.157+0000 7f91537fe640 10 client.17535 handle_command_reply: tid=0 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.157+0000 7f917ca0b740 1 client.17535 shutdown 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 msgr2=0x55d339ca2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9164005e00 tx=0x7f916400f850 comp rx=0 tx=0).stop 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 shutdown_connections 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339cb2320 0x55d339cb4710 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339ca2780 0x55d339c9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c94b40 0x55d339c9f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 --2- 172.21.15.16:0/342030334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339ca2170 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 >> 172.21.15.16:0/342030334 conn(0x55d339c5c310 msgr2=0x55d339c95930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f91537fe640 0 client.17535 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 shutdown_connections 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 1 -- 172.21.15.16:0/342030334 wait complete. 2024-09-12T02:56:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.158+0000 7f917ca0b740 20 client.17535 trim_cache size 0 max 16384 2024-09-12T02:56:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 Processor -- start 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 -- start start 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339cae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91741ab250 con 0x55d339ca2780 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91741acd90 con 0x55d339c935d0 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f917ca0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91741ab920 con 0x55d339c94b40 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f91737fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f9178bcf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f9178bcf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39136/0 (socket says 172.21.15.16:39136) 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f91737fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55472/0 (socket says 172.21.15.16:55472) 2024-09-12T02:56:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.172+0000 7f9178bcf640 1 -- 172.21.15.16:0/4223970434 learned_addr learned my addr 172.21.15.16:0/4223970434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9173fff640 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339cae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 -- 172.21.15.16:0/4223970434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 msgr2=0x55d339c9e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 -- 172.21.15.16:0/4223970434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 msgr2=0x55d339cae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339cae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 -- 172.21.15.16:0/4223970434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d339ca5360 con 0x55d339ca2780 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f91737fe640 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9178bcf640 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f916c00f4a0 tx=0x7f916c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f2367757e64bdc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9159ffb640 1 -- 172.21.15.16:0/4223970434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c010900 con 0x55d339ca2780 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9159ffb640 1 -- 172.21.15.16:0/4223970434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916c01f400 con 0x55d339ca2780 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.173+0000 7f9159ffb640 1 -- 172.21.15.16:0/4223970434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00e610 con 0x55d339ca2780 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- 172.21.15.16:0/4223970434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 msgr2=0x55d339cb9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f916c00f4a0 tx=0x7f916c015000 comp rx=0 tx=0).stop 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- 172.21.15.16:0/4223970434 shutdown_connections 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339ca2780 0x55d339cb9e00 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339c9e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 --2- 172.21.15.16:0/4223970434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339c935d0 0x55d339cae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- 172.21.15.16:0/4223970434 >> 172.21.15.16:0/4223970434 conn(0x55d339c5c310 msgr2=0x55d339c9dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- 172.21.15.16:0/4223970434 shutdown_connections 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- 172.21.15.16:0/4223970434 wait complete. 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.174+0000 7f917ca0b740 1 -- start start 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 Processor -- start 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339caafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339ca2780 0x55d339c9bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91741ac6c0 con 0x55d339c935d0 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91741abff0 con 0x55d339ca2780 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91741a9230 con 0x55d339c94b40 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39148/0 (socket says 172.21.15.16:39148) 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 -- 172.21.15.16:0/3034836338 learned_addr learned my addr 172.21.15.16:0/3034836338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f91737fe640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339caafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f917ca0b740 10 client.0 _ll_register_callbacks cb 0x7f916401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9178bcf640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339ca2780 0x55d339c9bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 -- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 msgr2=0x55d339caafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339caafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 -- 172.21.15.16:0/3034836338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339ca2780 msgr2=0x55d339c9bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339ca2780 0x55d339c9bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f9173fff640 1 -- 172.21.15.16:0/3034836338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d339c9c390 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.175+0000 7f91737fe640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339caafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f9173fff640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9164001eb0 tx=0x7f916400faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e5a1637df7f412 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164032070 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9164019ca0 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f916c01f400 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f916403e050 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 10 client.17541 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d339d1c7f0 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916c01f400 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9164004510 con 0x55d339c935d0 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 10 client.17541 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 10 client.17541 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 20 client.17541 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339d1ceb0 0x55d339d1f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 4 client.17541 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.176+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d339c00d70 con 0x55d339d1ceb0 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f91737fe640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339d1ceb0 0x55d339d1f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f91537fe640 10 client.17541 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f91737fe640 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339d1ceb0 0x55d339d1f2a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9a44ea7bdc1802d server_cookie=84def6ff80da7831 in_seq=0 out_seq=0 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f91537fe640 1 -- 172.21.15.16:0/3034836338 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d339c00d70 con 0x55d339d1ceb0 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f91537fe640 10 client.17541 handle_command_reply: tid=0 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f917ca0b740 1 client.17541 shutdown 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 msgr2=0x55d339caaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.177+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9164001eb0 tx=0x7f916400faf0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 shutdown_connections 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d339ca2780 0x55d339c9bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d339d1ceb0 0x55d339d1f2a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d339c94b40 0x55d339caafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 --2- 172.21.15.16:0/3034836338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d339c935d0 0x55d339caaa70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 >> 172.21.15.16:0/3034836338 conn(0x55d339c5c310 msgr2=0x55d339c9e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f91537fe640 0 client.17541 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 shutdown_connections 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 1 -- 172.21.15.16:0/3034836338 wait complete. 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917ca0b740 20 client.17541 trim_cache size 0 max 16384 2024-09-12T02:56:39.186 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:56:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 msgr2=0x7f915c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 0x7f915c0a5040 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f916c003a30 tx=0x7f916c004a40 comp rx=0 tx=0).stop 2024-09-12T02:56:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 msgr2=0x7f9174198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.178+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9168006010 tx=0x7f9168003c10 comp rx=0 tx=0).stop 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 shutdown_connections 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9174102ef0 0x7f917419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9174102570 0x7f9174198970 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f915c0a2b80 0x7f915c0a5040 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 --2- 172.21.15.16:0/1888756319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9174100e20 0x7f9174198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 >> 172.21.15.16:0/1888756319 conn(0x7f91740fc7d0 msgr2=0x7f9174069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 shutdown_connections 2024-09-12T02:56:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:39.179+0000 7f917a632640 1 -- 172.21.15.16:0/1888756319 wait complete. 2024-09-12T02:56:39.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:39 smithi016 ceph-mon[30925]: pgmap v599: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-09-12T02:56:39.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:39 smithi033 ceph-mon[35092]: pgmap v599: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-09-12T02:56:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:39 smithi184 ceph-mon[35321]: pgmap v599: 97 pgs: 97 active+clean; 274 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-09-12T02:56:39.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 838976 9.81 MB/sec execute 341 sec latency 34.459 ms 2024-09-12T02:56:39.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:56:40.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:40.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 -- 172.21.15.16:0/3685492723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20100e20 msgr2=0x7f4e20101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/3685492723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20100e20 0x7f4e20101200 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009900 tx=0x7f4e1002a480 comp rx=0 tx=0).stop 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 -- 172.21.15.16:0/3685492723 shutdown_connections 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/3685492723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20102ef0 0x7f4e2010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/3685492723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 0x7f4e201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/3685492723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20100e20 0x7f4e20101200 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 -- 172.21.15.16:0/3685492723 >> 172.21.15.16:0/3685492723 conn(0x7f4e200fc7d0 msgr2=0x7f4e200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 -- 172.21.15.16:0/3685492723 shutdown_connections 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.604+0000 7f4e25d8d640 1 -- 172.21.15.16:0/3685492723 wait complete. 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.605+0000 7f4e25d8d640 1 Processor -- start 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.605+0000 7f4e25d8d640 1 -- start start 2024-09-12T02:56:40.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 0x7f4e20199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 0x7f4e201a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20110f80 con 0x7f4e20102ef0 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20110e00 con 0x7f4e20102570 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e25d8d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20111100 con 0x7f4e20100e20 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 0x7f4e201a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 0x7f4e20199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55500/0 (socket says 172.21.15.16:55500) 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 0x7f4e201a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39160/0 (socket says 172.21.15.16:39160) 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/735731083 learned_addr learned my addr 172.21.15.16:0/735731083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/735731083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 msgr2=0x7f4e20199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 0x7f4e20199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 msgr2=0x7f4e201a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 0x7f4e201a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/735731083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e201a48f0 con 0x7f4e20100e20 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.606+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4e10006010 tx=0x7f4e10003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.607+0000 7f4e1cff9640 1 -- 172.21.15.16:0/735731083 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1003f070 con 0x7f4e20100e20 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.607+0000 7f4e1cff9640 1 -- 172.21.15.16:0/735731083 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e10031c30 con 0x7f4e20100e20 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.607+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e20197120 con 0x7f4e20100e20 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.607+0000 7f4e1cff9640 1 -- 172.21.15.16:0/735731083 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10035030 con 0x7f4e20100e20 2024-09-12T02:56:40.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.607+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e20197540 con 0x7f4e20100e20 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.609+0000 7f4e1cff9640 1 -- 172.21.15.16:0/735731083 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4e10042070 con 0x7f4e20100e20 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e1cff9640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 0x7f4e080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 Processor -- start 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 -- start start 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e1cff9640 1 -- 172.21.15.16:0/735731083 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4e100ef720 con 0x7f4e20100e20 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e1effd640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 0x7f4e080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac3d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3dd370 0x5594ac3df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594ac3382f0 con 0x5594ac3dd370 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5594ac2dfcf0 con 0x5594ac3d25d0 2024-09-12T02:56:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.610+0000 7f4e28166740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594ac3342b0 con 0x5594ac3d3b40 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac3d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55508/0 (socket says 172.21.15.16:55508) 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/1343389557 learned_addr learned my addr 172.21.15.16:0/1343389557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1ffff640 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3dd370 0x5594ac3df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/1343389557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 msgr2=0x5594ac3d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac3d2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/1343389557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3dd370 msgr2=0x5594ac3df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3dd370 0x5594ac3df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/1343389557 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594ac2c43a0 con 0x5594ac3d3b40 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1effd640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 0x7f4e080a5040 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400ffd0 tx=0x7f4e14009b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b8a10 tx=0x7f4e100b7ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b055f7a3769501d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e05ffb640 1 -- 172.21.15.16:0/1343389557 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10049050 con 0x5594ac3d3b40 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e05ffb640 1 -- 172.21.15.16:0/1343389557 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e100bc040 con 0x5594ac3d3b40 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.611+0000 7f4e05ffb640 1 -- 172.21.15.16:0/1343389557 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100b86b0 con 0x5594ac3d3b40 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- 172.21.15.16:0/1343389557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 msgr2=0x5594ac3d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b8a10 tx=0x7f4e100b7ce0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- 172.21.15.16:0/1343389557 shutdown_connections 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3dd370 0x5594ac3df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac3d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 --2- 172.21.15.16:0/1343389557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3d3b40 0x5594ac3d2090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- 172.21.15.16:0/1343389557 >> 172.21.15.16:0/1343389557 conn(0x5594ac39b310 msgr2=0x5594ac3cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- 172.21.15.16:0/1343389557 shutdown_connections 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- 172.21.15.16:0/1343389557 wait complete. 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.612+0000 7f4e28166740 1 -- start start 2024-09-12T02:56:40.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 Processor -- start 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3da0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3da840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594ac3dfe30 con 0x5594ac3d25d0 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5594ac2f3060 con 0x5594ac3d3b40 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594ac3dffb0 con 0x5594ac3dd370 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e28166740 10 client.0 _ll_register_callbacks cb 0x5594ac0468c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39184/0 (socket says 172.21.15.16:39184) 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/691225969 learned_addr learned my addr 172.21.15.16:0/691225969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1effd640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3da0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1ffff640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/691225969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 msgr2=0x5594ac3da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3da840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 msgr2=0x5594ac3da0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3da0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/691225969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594ac0a93c0 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.613+0000 7f4e1effd640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4e10005820 tx=0x7f4e100c4c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e975d748c94a653 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100b86b0 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4e100dabd0 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e100bc040 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e100b8850 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4e28166740 10 client.17553 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5594ac3dfe30 con 0x5594ac3d25d0 2024-09-12T02:56:40.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e100c4e50 con 0x5594ac3d25d0 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.614+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4e100bb490 con 0x5594ac3d25d0 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 10 client.17553 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 10 client.17553 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 20 client.17553 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac3ecdd0 0x5594ac3ef220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 4 client.17553 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:40.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594ac345860 con 0x5594ac3ecdd0 2024-09-12T02:56:40.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e1effd640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac3ecdd0 0x5594ac3ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4dfeffd640 10 client.17553 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.615+0000 7f4e1effd640 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac3ecdd0 0x5594ac3ef220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=826f2a5116cbffc0 server_cookie=b9a960d357850345 in_seq=0 out_seq=0 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4dfeffd640 1 -- 172.21.15.16:0/691225969 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5594ac345860 con 0x5594ac3ecdd0 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4dfeffd640 10 client.17553 handle_command_reply: tid=0 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4e28166740 1 client.17553 shutdown 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 msgr2=0x5594ac3dce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4e10005820 tx=0x7f4e100c4c00 comp rx=0 tx=0).stop 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 shutdown_connections 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.616+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac3ecdd0 0x5594ac3ef220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3da840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4dfeffd640 0 client.17553 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 --2- 172.21.15.16:0/691225969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3dce00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 >> 172.21.15.16:0/691225969 conn(0x5594ac39b310 msgr2=0x5594ac3d4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 shutdown_connections 2024-09-12T02:56:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 1 -- 172.21.15.16:0/691225969 wait complete. 2024-09-12T02:56:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.617+0000 7f4e28166740 20 client.17553 trim_cache size 0 max 16384 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.631+0000 7f4e28166740 1 Processor -- start 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.631+0000 7f4e28166740 1 -- start start 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.631+0000 7f4e28166740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.631+0000 7f4e28166740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e28166740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e28166740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201ab270 con 0x5594ac3d25d0 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e28166740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201acdb0 con 0x5594ac3d3b40 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e28166740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201ab940 con 0x5594ac3dd370 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39196/0 (socket says 172.21.15.16:39196) 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/3231121957 learned_addr learned my addr 172.21.15.16:0/3231121957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1effd640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1ffff640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/3231121957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 msgr2=0x5594ac3f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/3231121957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 msgr2=0x5594ac3d5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3d5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/3231121957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594ac3db040 con 0x5594ac3d25d0 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1effd640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4e1002a4c0 tx=0x7f4e100b7ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24ff6b0765c649d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e057fa640 1 -- 172.21.15.16:0/3231121957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100c8070 con 0x5594ac3d25d0 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e057fa640 1 -- 172.21.15.16:0/3231121957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e100bc040 con 0x5594ac3d25d0 2024-09-12T02:56:40.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.632+0000 7f4e057fa640 1 -- 172.21.15.16:0/3231121957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10049070 con 0x5594ac3d25d0 2024-09-12T02:56:40.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 -- 172.21.15.16:0/3231121957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 msgr2=0x5594ac3d5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4e1002a4c0 tx=0x7f4e100b7ce0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 -- 172.21.15.16:0/3231121957 shutdown_connections 2024-09-12T02:56:40.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d3b40 0x5594ac3d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 --2- 172.21.15.16:0/3231121957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d25d0 0x5594ac3d5120 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 -- 172.21.15.16:0/3231121957 >> 172.21.15.16:0/3231121957 conn(0x5594ac39b310 msgr2=0x5594ac3d91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 -- 172.21.15.16:0/3231121957 shutdown_connections 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.633+0000 7f4e28166740 1 -- 172.21.15.16:0/3231121957 wait complete. 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 -- start start 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 Processor -- start 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac44fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 0x5594ac450360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3e07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201ac6e0 con 0x5594ac3d3b40 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201ac010 con 0x5594ac3d25d0 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201a9250 con 0x5594ac3dd370 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 0x5594ac450360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e28166740 10 client.0 _ll_register_callbacks cb 0x7f4e100d3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3e07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac44fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3e07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55528/0 (socket says 172.21.15.16:55528) 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1ffff640 1 -- 172.21.15.16:0/2798530760 learned_addr learned my addr 172.21.15.16:0/2798530760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.634+0000 7f4e1effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 0x5594ac450360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39208/0 (socket says 172.21.15.16:39208) 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/2798530760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 msgr2=0x5594ac3e07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3e07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/2798530760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 msgr2=0x5594ac450360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 0x5594ac450360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 -- 172.21.15.16:0/2798530760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594ac3e0fa0 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e1f7fe640 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac44fe20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b8a10 tx=0x7f4e100ba550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86866a4fab43bf03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4dfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100c8060 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4e100c4770 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e100bc040 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e10049070 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e28166740 10 client.27047 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.635+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5594ac3e0fa0 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e100c4e50 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4e100dabd0 con 0x5594ac3d25d0 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 10 client.27047 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 10 client.27047 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 20 client.27047 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac457860 0x5594ac459c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 4 client.27047 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4e100bc040 con 0x5594ac457860 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4e1effd640 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac457860 0x5594ac459c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.636+0000 7f4dfeffd640 10 client.27047 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e1effd640 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac457860 0x5594ac459c50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74ebe2f781560238 server_cookie=c3f9d4267dca6024 in_seq=0 out_seq=0 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4dfeffd640 1 -- 172.21.15.16:0/2798530760 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4e100bc040 con 0x5594ac457860 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4dfeffd640 10 client.27047 handle_command_reply: tid=0 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 client.27047 shutdown 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 msgr2=0x5594ac44fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac44fe20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4e100b8a10 tx=0x7f4e100ba550 comp rx=0 tx=0).stop 2024-09-12T02:56:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 shutdown_connections 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5594ac457860 0x5594ac459c50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594ac3dd370 0x5594ac3e07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594ac3d3b40 0x5594ac450360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 --2- 172.21.15.16:0/2798530760 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594ac3d25d0 0x5594ac44fe20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 >> 172.21.15.16:0/2798530760 conn(0x5594ac39b310 msgr2=0x5594ac3e8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4dfeffd640 0 client.27047 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 shutdown_connections 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 1 -- 172.21.15.16:0/2798530760 wait complete. 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.637+0000 7f4e28166740 20 client.27047 trim_cache size 0 max 16384 2024-09-12T02:56:40.644 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 msgr2=0x7f4e080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 0x7f4e080a5040 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400ffd0 tx=0x7f4e14009b20 comp rx=0 tx=0).stop 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 msgr2=0x7f4e20199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4e10006010 tx=0x7f4e10003c10 comp rx=0 tx=0).stop 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 shutdown_connections 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4e080a2b80 0x7f4e080a5040 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e20102ef0 0x7f4e201a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e20102570 0x7f4e20199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 --2- 172.21.15.16:0/735731083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e20100e20 0x7f4e20199330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:40.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 >> 172.21.15.16:0/735731083 conn(0x7f4e200fc7d0 msgr2=0x7f4e200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:40.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 shutdown_connections 2024-09-12T02:56:40.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:40.638+0000 7f4e25d8d640 1 -- 172.21.15.16:0/735731083 wait complete. 2024-09-12T02:56:40.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 841863 9.83 MB/sec execute 342 sec latency 16.879 ms 2024-09-12T02:56:41.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:41 smithi016 ceph-mon[30925]: pgmap v600: 97 pgs: 97 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-09-12T02:56:41.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:56:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:41 smithi033 ceph-mon[35092]: pgmap v600: 97 pgs: 97 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-09-12T02:56:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:41 smithi184 ceph-mon[35321]: pgmap v600: 97 pgs: 97 active+clean; 280 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-09-12T02:56:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:41.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 845090 9.85 MB/sec execute 343 sec latency 11.806 ms 2024-09-12T02:56:42.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 -- 172.21.15.16:0/3313444962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4102d40 msgr2=0x7f27d4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3313444962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4102d40 0x7f27d4103120 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009900 tx=0x7f27c402a480 comp rx=0 tx=0).stop 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 -- 172.21.15.16:0/3313444962 shutdown_connections 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3313444962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d410c170 0x7f27d410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3313444962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d4103660 0x7f27d410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3313444962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4102d40 0x7f27d4103120 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 -- 172.21.15.16:0/3313444962 >> 172.21.15.16:0/3313444962 conn(0x7f27d40fc7a0 msgr2=0x7f27d40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 -- 172.21.15.16:0/3313444962 shutdown_connections 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.211+0000 7f27d94bc640 1 -- 172.21.15.16:0/3313444962 wait complete. 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 Processor -- start 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 -- start start 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d4102d40 0x7f27d41009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 0x7f27d4101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4111fc0 con 0x7f27d4103660 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4111e40 con 0x7f27d410c170 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.212+0000 7f27d94bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4112140 con 0x7f27d4102d40 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39232/0 (socket says 172.21.15.16:39232) 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 -- 172.21.15.16:0/3765483297 learned_addr learned my addr 172.21.15.16:0/3765483297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d37fe640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 0x7f27d4101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 -- 172.21.15.16:0/3765483297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d4102d40 msgr2=0x7f27d41009a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d4102d40 0x7f27d41009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d4102d40 0x7f27d41009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 -- 172.21.15.16:0/3765483297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 msgr2=0x7f27d4101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 0x7f27d4101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 -- 172.21.15.16:0/3765483297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d41a6910 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d37fe640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 0x7f27d4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.213+0000 7f27d27fc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f27c800e9f0 tx=0x7f27c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.214+0000 7f27bbfff640 1 -- 172.21.15.16:0/3765483297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c800ccc0 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.214+0000 7f27bbfff640 1 -- 172.21.15.16:0/3765483297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c8010910 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.214+0000 7f27bbfff640 1 -- 172.21.15.16:0/3765483297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8018920 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.214+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d41a6c00 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.214+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d41a70f0 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.216+0000 7f27bbfff640 1 -- 172.21.15.16:0/3765483297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f27c8010430 con 0x7f27d4103660 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.216+0000 7f27db895740 1 Processor -- start 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.216+0000 7f27db895740 1 -- start start 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56334e3412f0 con 0x56334e3dcb40 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56334e2e8cf0 con 0x56334e3e6370 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27db895740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56334e33d2b0 con 0x56334e3db5d0 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27bbfff640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 0x7f27bc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39240/0 (socket says 172.21.15.16:39240) 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d37fe640 1 -- 172.21.15.16:0/2927309106 learned_addr learned my addr 172.21.15.16:0/2927309106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55548/0 (socket says 172.21.15.16:55548) 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d37fe640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 0x7f27bc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d27fc640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27bbfff640 1 -- 172.21.15.16:0/3765483297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27c80cf330 con 0x7f27d4103660 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2927309106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 msgr2=0x56334e3e8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2927309106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 msgr2=0x56334e3db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2927309106 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56334e2cd3a0 con 0x56334e3db5d0 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.217+0000 7f27d37fe640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.218+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f27c4005ab0 tx=0x7f27c4005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=861dd77f09e89cd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.218+0000 7f27d37fe640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 0x7f27bc0a50c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f27cc014dd0 tx=0x7f27cc009b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.218+0000 7f27b97fa640 1 -- 172.21.15.16:0/2927309106 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c403f070 con 0x56334e3db5d0 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.218+0000 7f27b97fa640 1 -- 172.21.15.16:0/2927309106 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c4005520 con 0x56334e3db5d0 2024-09-12T02:56:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.218+0000 7f27b97fa640 1 -- 172.21.15.16:0/2927309106 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4035030 con 0x56334e3db5d0 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- 172.21.15.16:0/2927309106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 msgr2=0x56334e3dba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f27c4005ab0 tx=0x7f27c4005e00 comp rx=0 tx=0).stop 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- 172.21.15.16:0/2927309106 shutdown_connections 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3dba10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 --2- 172.21.15.16:0/2927309106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- 172.21.15.16:0/2927309106 >> 172.21.15.16:0/2927309106 conn(0x56334e3a4310 msgr2=0x56334e3d7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- 172.21.15.16:0/2927309106 shutdown_connections 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- 172.21.15.16:0/2927309106 wait complete. 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 -- start start 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.219+0000 7f27db895740 1 Processor -- start 2024-09-12T02:56:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56334e3e9010 con 0x56334e3dcb40 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56334e2fc060 con 0x56334e3e6370 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55562/0 (socket says 172.21.15.16:55562) 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39248/0 (socket says 172.21.15.16:39248) 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d2ffd640 1 -- 172.21.15.16:0/1915396793 learned_addr learned my addr 172.21.15.16:0/1915396793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56334e3e9190 con 0x56334e3db5d0 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27d37fe640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.220+0000 7f27db895740 10 client.0 _ll_register_callbacks cb 0x56334e04f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 -- 172.21.15.16:0/1915396793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 msgr2=0x56334e3e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 -- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 msgr2=0x56334e3e3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 -- 172.21.15.16:0/1915396793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56334e0b23c0 con 0x56334e3dcb40 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27d27fc640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f27c80cd1b0 tx=0x7f27c809a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5809e47c83066c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8026040 con 0x56334e3dcb40 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27c809a6d0 con 0x56334e3dcb40 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c4005520 con 0x56334e3dcb40 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.221+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f27c809ff50 con 0x56334e3dcb40 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 10 client.17571 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56334e3f4af0 con 0x56334e3dcb40 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c4005520 con 0x56334e3dcb40 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f27c809aff0 con 0x56334e3dcb40 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 10 client.17571 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 20 client.17571 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e3f6390 0x56334e3f8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 4 client.17571 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56334e34e860 con 0x56334e3f6390 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.222+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e3f6390 0x56334e3f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.223+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e3f6390 0x56334e3f8850 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fb7b101aaaf9ff4 server_cookie=db15dedb855fed9 in_seq=0 out_seq=0 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.223+0000 7f27b6ff5640 10 client.17571 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.223+0000 7f27b6ff5640 1 -- 172.21.15.16:0/1915396793 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56334e34e860 con 0x56334e3f6390 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.223+0000 7f27b6ff5640 10 client.17571 handle_command_reply: tid=0 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.223+0000 7f27db895740 1 client.17571 shutdown 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 msgr2=0x56334e3e3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f27c80cd1b0 tx=0x7f27c809a440 comp rx=0 tx=0).stop 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 shutdown_connections 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e3f6390 0x56334e3f8850 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3e3040 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 --2- 172.21.15.16:0/1915396793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 >> 172.21.15.16:0/1915396793 conn(0x56334e3a4310 msgr2=0x56334e3dd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27b6ff5640 0 client.17571 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 shutdown_connections 2024-09-12T02:56:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 1 -- 172.21.15.16:0/1915396793 wait complete. 2024-09-12T02:56:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.224+0000 7f27db895740 20 client.17571 trim_cache size 0 max 16384 2024-09-12T02:56:42.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 Processor -- start 2024-09-12T02:56:42.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 -- start start 2024-09-12T02:56:42.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3d7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41af6f0 con 0x56334e3dcb40 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41b1230 con 0x56334e3e6370 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27db895740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41afdc0 con 0x56334e3db5d0 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46772/0 (socket says 172.21.15.16:46772) 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55574/0 (socket says 172.21.15.16:55574) 2024-09-12T02:56:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d37fe640 1 -- 172.21.15.16:0/2068520651 learned_addr learned my addr 172.21.15.16:0/2068520651 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d27fc640 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3d7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.240+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2068520651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 msgr2=0x56334e3e02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2068520651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 msgr2=0x56334e3d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3d7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2068520651 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56334e3dc300 con 0x56334e3db5d0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d37fe640 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009310 tx=0x7f27c402ab00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575ab79d2c300c3b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27b8ff9640 1 -- 172.21.15.16:0/2068520651 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c403f040 con 0x56334e3db5d0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27b8ff9640 1 -- 172.21.15.16:0/2068520651 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c4004140 con 0x56334e3db5d0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.241+0000 7f27b8ff9640 1 -- 172.21.15.16:0/2068520651 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4035040 con 0x56334e3db5d0 2024-09-12T02:56:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.242+0000 7f27db895740 1 -- 172.21.15.16:0/2068520651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 msgr2=0x56334e3f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.242+0000 7f27db895740 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009310 tx=0x7f27c402ab00 comp rx=0 tx=0).stop 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 -- 172.21.15.16:0/2068520651 shutdown_connections 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e3d7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 --2- 172.21.15.16:0/2068520651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e3f2460 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 -- 172.21.15.16:0/2068520651 >> 172.21.15.16:0/2068520651 conn(0x56334e3a4310 msgr2=0x56334e3d7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 -- 172.21.15.16:0/2068520651 shutdown_connections 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 -- 172.21.15.16:0/2068520651 wait complete. 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 -- start start 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.243+0000 7f27db895740 1 Processor -- start 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e458460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e4589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e4589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27d27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e4589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39264/0 (socket says 172.21.15.16:39264) 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41b0b60 con 0x56334e3dcb40 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27d27fc640 1 -- 172.21.15.16:0/2979548465 learned_addr learned my addr 172.21.15.16:0/2979548465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:42.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41b0490 con 0x56334e3e6370 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41ad6d0 con 0x56334e3db5d0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27db895740 10 client.0 _ll_register_callbacks cb 0x7f27c8097e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27d37fe640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.244+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e458460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 msgr2=0x56334e3fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2979548465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 msgr2=0x56334e4589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e4589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56334e3fbf20 con 0x56334e3db5d0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27d2ffd640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e458460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f27c402aa70 tx=0x7f27c4009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d21acd4ff2d2b121 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4035070 con 0x56334e3db5d0 2024-09-12T02:56:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.245+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27c40042f0 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c4004140 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f27c40315b0 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 10 client.26959 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56334e460c40 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c4004140 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f27c4003c00 con 0x56334e3db5d0 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 10 client.26959 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 10 client.26959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 20 client.26959 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e460e70 0x56334e463260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 4 client.26959 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.246+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56334e348d70 con 0x56334e460e70 2024-09-12T02:56:42.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.247+0000 7f27d27fc640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e460e70 0x56334e463260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.247+0000 7f27b6ff5640 10 client.26959 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27d27fc640 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e460e70 0x56334e463260 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=774981d118af3c04 server_cookie=f81060429601ab4f in_seq=0 out_seq=0 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27b6ff5640 1 -- 172.21.15.16:0/2979548465 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56334e348d70 con 0x56334e460e70 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27b6ff5640 10 client.26959 handle_command_reply: tid=0 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 client.26959 shutdown 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 msgr2=0x56334e458460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e458460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f27c402aa70 tx=0x7f27c4009660 comp rx=0 tx=0).stop 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 shutdown_connections 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56334e460e70 0x56334e463260 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56334e3e6370 0x56334e3fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56334e3dcb40 0x56334e4589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 --2- 172.21.15.16:0/2979548465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56334e3db5d0 0x56334e458460 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 >> 172.21.15.16:0/2979548465 conn(0x56334e3a4310 msgr2=0x56334e3e48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.248+0000 7f27b6ff5640 0 client.26959 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:42.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.249+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 shutdown_connections 2024-09-12T02:56:42.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.249+0000 7f27db895740 1 -- 172.21.15.16:0/2979548465 wait complete. 2024-09-12T02:56:42.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.249+0000 7f27db895740 20 client.26959 trim_cache size 0 max 16384 2024-09-12T02:56:42.254 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:56:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.249+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 msgr2=0x7f27bc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 0x7f27bc0a50c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f27cc014dd0 tx=0x7f27cc009b20 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 msgr2=0x7f27d4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f27c800e9f0 tx=0x7f27c800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 shutdown_connections 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d410c170 0x7f27d4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27bc0a2c00 0x7f27bc0a50c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27d4103660 0x7f27d4100ee0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 --2- 172.21.15.16:0/3765483297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27d4102d40 0x7f27d41009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.250+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 >> 172.21.15.16:0/3765483297 conn(0x7f27d40fc7a0 msgr2=0x7f27d40fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.251+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 shutdown_connections 2024-09-12T02:56:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:42.251+0000 7f27d94bc640 1 -- 172.21.15.16:0/3765483297 wait complete. 2024-09-12T02:56:42.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 847462 9.85 MB/sec execute 344 sec latency 18.027 ms 2024-09-12T02:56:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:43 smithi033 ceph-mon[35092]: pgmap v601: 97 pgs: 97 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 242 op/s 2024-09-12T02:56:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:43 smithi184 ceph-mon[35321]: pgmap v601: 97 pgs: 97 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 242 op/s 2024-09-12T02:56:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:43 smithi016 ceph-mon[30925]: pgmap v601: 97 pgs: 97 active+clean; 281 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 242 op/s 2024-09-12T02:56:43.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 849750 9.86 MB/sec execute 345 sec latency 13.758 ms 2024-09-12T02:56:43.963 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:56:43.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:56:44.230 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 -- 172.21.15.16:0/2395323876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244106f10 msgr2=0x7f424410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 --2- 172.21.15.16:0/2395323876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244106f10 0x7f424410ea90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f423c009be0 tx=0x7f423c02a6c0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 -- 172.21.15.16:0/2395323876 shutdown_connections 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 --2- 172.21.15.16:0/2395323876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244106f10 0x7f424410ea90 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 --2- 172.21.15.16:0/2395323876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244102790 0x7f42441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 --2- 172.21.15.16:0/2395323876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244101de0 0x7f42441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 -- 172.21.15.16:0/2395323876 >> 172.21.15.16:0/2395323876 conn(0x7f42440fd8c0 msgr2=0x7f42440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 -- 172.21.15.16:0/2395323876 shutdown_connections 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.710+0000 7f424b5c9640 1 -- 172.21.15.16:0/2395323876 wait complete. 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 Processor -- start 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 -- start start 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244101de0 0x7f42441983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 0x7f424419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4244111f20 con 0x7f4244102790 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4244111da0 con 0x7f4244106f10 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f424b5c9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f42441120a0 con 0x7f4244101de0 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.711+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39282/0 (socket says 172.21.15.16:39282) 2024-09-12T02:56:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4249365640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244101de0 0x7f42441983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 -- 172.21.15.16:0/3286471601 learned_addr learned my addr 172.21.15.16:0/3286471601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4249b66640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 0x7f424419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 -- 172.21.15.16:0/3286471601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244101de0 msgr2=0x7f42441983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244101de0 0x7f42441983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 -- 172.21.15.16:0/3286471601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 msgr2=0x7f424419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 0x7f424419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 -- 172.21.15.16:0/3286471601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424419d350 con 0x7f4244102790 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4249b66640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 0x7f424419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f4248b64640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f423400ea60 tx=0x7f423400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f423a7fc640 1 -- 172.21.15.16:0/3286471601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423400cde0 con 0x7f4244102790 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424419d640 con 0x7f4244102790 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f423a7fc640 1 -- 172.21.15.16:0/3286471601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42340035f0 con 0x7f4244102790 2024-09-12T02:56:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42441a4f20 con 0x7f4244102790 2024-09-12T02:56:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.712+0000 7f423a7fc640 1 -- 172.21.15.16:0/3286471601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234010670 con 0x7f4244102790 2024-09-12T02:56:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 Processor -- start 2024-09-12T02:56:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f423a7fc640 1 -- 172.21.15.16:0/3286471601 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4234021430 con 0x7f4244102790 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 -- start start 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x5586725f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725fed40 0x558672601130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558672559280 con 0x5586725fed40 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558672500b10 con 0x5586725f4480 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f424d9a2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558672555240 con 0x5586725f3b60 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725fed40 0x558672601130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f4249365640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f4249365640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46818/0 (socket says 172.21.15.16:46818) 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f4249365640 1 -- 172.21.15.16:0/3099289579 learned_addr learned my addr 172.21.15.16:0/3099289579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:44.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.716+0000 7f4249b66640 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x5586725f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 -- 172.21.15.16:0/3099289579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 msgr2=0x5586725f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x5586725f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 -- 172.21.15.16:0/3099289579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725fed40 msgr2=0x558672601130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725fed40 0x558672601130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 -- 172.21.15.16:0/3099289579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55867250d1f0 con 0x5586725f4480 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249365640 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4240009cc0 tx=0x7f4240011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2aceb34625b4344 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f423a7fc640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 0x7f42300a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f4249b66640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 0x7f42300a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f422f7fe640 1 -- 172.21.15.16:0/3099289579 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240005220 con 0x5586725f4480 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f423a7fc640 1 -- 172.21.15.16:0/3286471601 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f42340cf0d0 con 0x7f4244102790 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f422f7fe640 1 -- 172.21.15.16:0/3099289579 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4240004c80 con 0x5586725f4480 2024-09-12T02:56:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.717+0000 7f422f7fe640 1 -- 172.21.15.16:0/3099289579 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240019970 con 0x5586725f4480 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 -- 172.21.15.16:0/3099289579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 msgr2=0x5586725fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f4249b66640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 0x7f42300a50c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f423c02abe0 tx=0x7f423c009880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4240009cc0 tx=0x7f4240011810 comp rx=0 tx=0).stop 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 -- 172.21.15.16:0/3099289579 shutdown_connections 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725fed40 0x558672601130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fe800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3099289579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x5586725f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 -- 172.21.15.16:0/3099289579 >> 172.21.15.16:0/3099289579 conn(0x5586725f00f0 msgr2=0x5586725f04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 -- 172.21.15.16:0/3099289579 shutdown_connections 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.718+0000 7f424d9a2740 1 -- 172.21.15.16:0/3099289579 wait complete. 2024-09-12T02:56:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 -- start start 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 Processor -- start 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586725fc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558672601910 con 0x5586725f3b60 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558672513f10 con 0x5586725f4480 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558672601a90 con 0x5586725fed40 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f424d9a2740 10 client.0 _ll_register_callbacks cb 0x5586724dca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f4249365640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f4249365640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39294/0 (socket says 172.21.15.16:39294) 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f4249365640 1 -- 172.21.15.16:0/772988176 learned_addr learned my addr 172.21.15.16:0/772988176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f4248b64640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.719+0000 7f4249b66640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586725fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 -- 172.21.15.16:0/772988176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 msgr2=0x5586725fc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586725fc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 -- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 msgr2=0x5586725fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 -- 172.21.15.16:0/772988176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586722ca6a0 con 0x5586725f3b60 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f4249365640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4240009530 tx=0x7f4240005580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e0d791034ede86 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f422cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240029070 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4240005240 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4240004c80 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4240009e70 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f424d9a2740 10 client.17589 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.720+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5586726118a0 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4240004c80 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f424002e7a0 con 0x5586725f3b60 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 10 client.17589 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 10 client.17589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 20 client.17589 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558672612ca0 0x558672615090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 4 client.17589 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586725667f0 con 0x558672612ca0 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.721+0000 7f4248b64640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558672612ca0 0x558672615090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f4248b64640 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558672612ca0 0x558672615090 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e5327afd159f03a server_cookie=957a3b9ec1d9fdd5 in_seq=0 out_seq=0 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f422cff9640 10 client.17589 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f422cff9640 1 -- 172.21.15.16:0/772988176 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5586725667f0 con 0x558672612ca0 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f422cff9640 10 client.17589 handle_command_reply: tid=0 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 client.17589 shutdown 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 msgr2=0x5586725fe7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4240009530 tx=0x7f4240005580 comp rx=0 tx=0).stop 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 shutdown_connections 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558672612ca0 0x558672615090 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586725fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f4480 0x5586725fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 --2- 172.21.15.16:0/772988176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f3b60 0x5586725fe7c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.722+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 >> 172.21.15.16:0/772988176 conn(0x5586725f00f0 msgr2=0x5586725f5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.723+0000 7f422cff9640 0 client.17589 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.723+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 shutdown_connections 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.723+0000 7f424d9a2740 1 -- 172.21.15.16:0/772988176 wait complete. 2024-09-12T02:56:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.723+0000 7f424d9a2740 20 client.17589 trim_cache size 0 max 16384 2024-09-12T02:56:44.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 852647 9.88 MB/sec execute 346 sec latency 35.552 ms 2024-09-12T02:56:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 Processor -- start 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 -- start start 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x5586725faaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 0x55867261a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f422800e400 con 0x5586725f4480 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f422800ff40 con 0x5586725fed40 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.740+0000 7f424d9a2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f422800ead0 con 0x5586725f3b60 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x5586725faaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249365640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249b66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 0x55867261a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249365640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55620/0 (socket says 172.21.15.16:55620) 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249b66640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 0x55867261a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46840/0 (socket says 172.21.15.16:46840) 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249365640 1 -- 172.21.15.16:0/3762647492 learned_addr learned my addr 172.21.15.16:0/3762647492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 -- 172.21.15.16:0/3762647492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 msgr2=0x55867260ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 -- 172.21.15.16:0/3762647492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 msgr2=0x55867261a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 0x55867261a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 -- 172.21.15.16:0/3762647492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586725f42b0 con 0x5586725f4480 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4249365640 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f4248b64640 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x5586725faaf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f42340ccc30 tx=0x7f42340ccab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1294b6397871c4c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f422effd640 1 -- 172.21.15.16:0/3762647492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423409b480 con 0x5586725f4480 2024-09-12T02:56:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f422effd640 1 -- 172.21.15.16:0/3762647492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423409c040 con 0x5586725f4480 2024-09-12T02:56:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.741+0000 7f422effd640 1 -- 172.21.15.16:0/3762647492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423409b2a0 con 0x5586725f4480 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 -- 172.21.15.16:0/3762647492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 msgr2=0x5586725faaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x5586725faaf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f42340ccc30 tx=0x7f42340ccab0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 -- 172.21.15.16:0/3762647492 shutdown_connections 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725fed40 0x55867261a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x5586725faaf0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3762647492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725f3b60 0x55867260ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 -- 172.21.15.16:0/3762647492 >> 172.21.15.16:0/3762647492 conn(0x5586725f00f0 msgr2=0x5586725fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 -- 172.21.15.16:0/3762647492 shutdown_connections 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.742+0000 7f424d9a2740 1 -- 172.21.15.16:0/3762647492 wait complete. 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 -- start start 2024-09-12T02:56:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 Processor -- start 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f3b60 0x558672602a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586726034e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f422800f870 con 0x5586725f4480 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f422800f1a0 con 0x5586725f3b60 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f422800c3e0 con 0x5586725fed40 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f424d9a2740 10 client.0 _ll_register_callbacks cb 0x7f42340d2a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f4248b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39306/0 (socket says 172.21.15.16:39306) 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f4249365640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f3b60 0x558672602a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f4248b64640 1 -- 172.21.15.16:0/3122765777 learned_addr learned my addr 172.21.15.16:0/3122765777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.743+0000 7f4249b66640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586726034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 -- 172.21.15.16:0/3122765777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 msgr2=0x5586726034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586726034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 -- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f3b60 msgr2=0x558672602a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f3b60 0x558672602a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 -- 172.21.15.16:0/3122765777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55867267c8e0 con 0x5586725f4480 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f4248b64640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f42340cf480 tx=0x7f423409fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b9228e6635ad35 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f422cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42340b3070 con 0x5586725f4480 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f42340860b0 con 0x5586725f4480 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f423409c040 con 0x5586725f4480 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f42340afe60 con 0x5586725f4480 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f424d9a2740 10 client.17595 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55867267dba0 con 0x5586725f4480 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.744+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423409c040 con 0x5586725f4480 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f42340ba1d0 con 0x5586725f4480 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 10 client.17595 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 10 client.17595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 20 client.17595 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55867267e1f0 0x558672680610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 4 client.17595 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558672560d00 con 0x55867267e1f0 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f4249365640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55867267e1f0 0x558672680610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f4249365640 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55867267e1f0 0x558672680610 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7341eec00e3d01f8 server_cookie=2c70f0ca23dda946 in_seq=0 out_seq=0 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.745+0000 7f422cff9640 10 client.17595 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f422cff9640 1 -- 172.21.15.16:0/3122765777 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558672560d00 con 0x55867267e1f0 2024-09-12T02:56:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f422cff9640 10 client.17595 handle_command_reply: tid=0 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 client.17595 shutdown 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 msgr2=0x558672602fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f42340cf480 tx=0x7f423409fcf0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 shutdown_connections 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55867267e1f0 0x558672680610 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5586725fed40 0x5586726034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586725f4480 0x558672602fa0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 --2- 172.21.15.16:0/3122765777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586725f3b60 0x558672602a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 >> 172.21.15.16:0/3122765777 conn(0x5586725f00f0 msgr2=0x5586725fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.746+0000 7f422cff9640 0 client.17595 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 shutdown_connections 2024-09-12T02:56:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424d9a2740 1 -- 172.21.15.16:0/3122765777 wait complete. 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424d9a2740 20 client.17595 trim_cache size 0 max 16384 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "1b64b891-2409-445e-9d78-48bacfcae1e9", 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:56:44.753 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 msgr2=0x7f42300a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 0x7f42300a50c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f423c02abe0 tx=0x7f423c009880 comp rx=0 tx=0).stop 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 msgr2=0x7f42441988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f423400ea60 tx=0x7f423400ef20 comp rx=0 tx=0).stop 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 shutdown_connections 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4244106f10 0x7f424419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f42300a2c00 0x7f42300a50c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4244102790 0x7f42441988e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 --2- 172.21.15.16:0/3286471601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4244101de0 0x7f42441983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.747+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 >> 172.21.15.16:0/3286471601 conn(0x7f42440fd8c0 msgr2=0x7f42440fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.748+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 shutdown_connections 2024-09-12T02:56:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:44.748+0000 7f424b5c9640 1 -- 172.21.15.16:0/3286471601 wait complete. 2024-09-12T02:56:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:45 smithi033 ceph-mon[35092]: pgmap v602: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-12T02:56:45.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:45 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:56:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:45 smithi184 ceph-mon[35321]: pgmap v602: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-12T02:56:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:45 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:56:45.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:45 smithi016 ceph-mon[30925]: pgmap v602: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-12T02:56:45.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:45 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:56:45.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:56:45.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 855652 9.91 MB/sec execute 347 sec latency 32.176 ms 2024-09-12T02:56:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- 172.21.15.16:0/3236501646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 msgr2=0x7f8f80103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 --2- 172.21.15.16:0/3236501646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80103120 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009900 tx=0x7f8f7002a480 comp rx=0 tx=0).stop 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- 172.21.15.16:0/3236501646 shutdown_connections 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 --2- 172.21.15.16:0/3236501646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8010c170 0x7f8f8010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 --2- 172.21.15.16:0/3236501646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f80103660 0x7f8f8010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 --2- 172.21.15.16:0/3236501646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80103120 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- 172.21.15.16:0/3236501646 >> 172.21.15.16:0/3236501646 conn(0x7f8f800fc7a0 msgr2=0x7f8f800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- 172.21.15.16:0/3236501646 shutdown_connections 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- 172.21.15.16:0/3236501646 wait complete. 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 Processor -- start 2024-09-12T02:56:46.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.347+0000 7f8f859df640 1 -- start start 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 0x7f8f80198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f8010c170 0x7f8f8019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80111f70 con 0x7f8f80102d40 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80111df0 con 0x7f8f80103660 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f859df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801120f0 con 0x7f8f8010c170 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52076/0 (socket says 172.21.15.16:52076) 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f7effd640 1 -- 172.21.15.16:0/3171733369 learned_addr learned my addr 172.21.15.16:0/3171733369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f8010c170 0x7f8f8019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 0x7f8f80198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 -- 172.21.15.16:0/3171733369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f8010c170 msgr2=0x7f8f8019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f8010c170 0x7f8f8019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 -- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 msgr2=0x7f8f80198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f77fff640 1 -- 172.21.15.16:0/3171733369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8019d3b0 con 0x7f8f80103660 2024-09-12T02:56:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.348+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 0x7f8f80198940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800e7a0 tx=0x7f8f6800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f849dd640 1 -- 172.21.15.16:0/3171733369 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68020070 con 0x7f8f80103660 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f849dd640 1 -- 172.21.15.16:0/3171733369 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6800ccc0 con 0x7f8f80103660 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8019d6a0 con 0x7f8f80103660 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f849dd640 1 -- 172.21.15.16:0/3171733369 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6801ba30 con 0x7f8f80103660 2024-09-12T02:56:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.349+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f801008d0 con 0x7f8f80103660 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.351+0000 7f8f849dd640 1 -- 172.21.15.16:0/3171733369 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8f68023070 con 0x7f8f80103660 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 Processor -- start 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 -- start start 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d171a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c8d0ca2e0 con 0x558c8d1644a0 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c8d071ce0 con 0x558c8d16f680 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f87db8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c8d0c62a0 con 0x558c8d164dc0 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f7f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f7f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52078/0 (socket says 172.21.15.16:52078) 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f7f7fe640 1 -- 172.21.15.16:0/3183270077 learned_addr learned my addr 172.21.15.16:0/3183270077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.352+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d171a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f849dd640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 0x7f8f6c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 -- 172.21.15.16:0/3183270077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 msgr2=0x558c8d171a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d171a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 -- 172.21.15.16:0/3183270077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 msgr2=0x558c8d164880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 -- 172.21.15.16:0/3183270077 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c8d056440 con 0x558c8d164dc0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f849dd640 1 -- 172.21.15.16:0/3171733369 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f680d3090 con 0x7f8f80103660 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 0x7f8f6c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16f140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009230 tx=0x7f8f70005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5edd8755484b471f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f75ffb640 1 -- 172.21.15.16:0/3183270077 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70040070 con 0x558c8d164dc0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 0x7f8f6c0a5040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8f780107d0 tx=0x7f8f78004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f75ffb640 1 -- 172.21.15.16:0/3183270077 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f70031970 con 0x558c8d164dc0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.353+0000 7f8f75ffb640 1 -- 172.21.15.16:0/3183270077 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70035040 con 0x558c8d164dc0 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- 172.21.15.16:0/3183270077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 msgr2=0x558c8d16f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16f140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009230 tx=0x7f8f70005dd0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- 172.21.15.16:0/3183270077 shutdown_connections 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d171a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16f140 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3183270077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- 172.21.15.16:0/3183270077 >> 172.21.15.16:0/3183270077 conn(0x558c8d160a30 msgr2=0x558c8d160e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- 172.21.15.16:0/3183270077 shutdown_connections 2024-09-12T02:56:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- 172.21.15.16:0/3183270077 wait complete. 2024-09-12T02:56:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.354+0000 7f8f87db8740 1 -- start start 2024-09-12T02:56:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 Processor -- start 2024-09-12T02:56:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d16c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558c8d172250 con 0x558c8d1644a0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c8d085050 con 0x558c8d16f680 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c8d1723d0 con 0x558c8d164dc0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f87db8740 10 client.0 _ll_register_callbacks cb 0x558c8d0c5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52086/0 (socket says 172.21.15.16:52086) 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f7f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d16c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f7effd640 1 -- 172.21.15.16:0/1833381287 learned_addr learned my addr 172.21.15.16:0/1833381287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.355+0000 7f8f77fff640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 -- 172.21.15.16:0/1833381287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 msgr2=0x558c8d16c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 -- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 msgr2=0x558c8d16c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d16c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 -- 172.21.15.16:0/1833381287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c8ce3b3c0 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d16c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8f7002a4c0 tx=0x7f8f700095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9524c97a6fbe5170 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f5effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7004b070 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f70031970 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f70004450 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f70057050 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f87db8740 10 client.17607 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.356+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c8d17dc00 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f70031970 con 0x558c8d1644a0 2024-09-12T02:56:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f7003c880 con 0x558c8d1644a0 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 10 client.17607 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 10 client.17607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 20 client.17607 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d17ef50 0x558c8d181340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 4 client.17607 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c8d0d7850 con 0x558c8d17ef50 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f77fff640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d17ef50 0x558c8d181340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f5effd640 10 client.17607 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.357+0000 7f8f77fff640 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d17ef50 0x558c8d181340 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ee6298bd604898 server_cookie=f035fa1f8e9c3423 in_seq=0 out_seq=0 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f5effd640 1 -- 172.21.15.16:0/1833381287 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c8d0d7850 con 0x558c8d17ef50 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f5effd640 10 client.17607 handle_command_reply: tid=0 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 client.17607 shutdown 2024-09-12T02:56:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 msgr2=0x558c8d16efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8f7002a4c0 tx=0x7f8f700095a0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 shutdown_connections 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d17ef50 0x558c8d181340 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d16c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d16c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1833381287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d16efc0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 >> 172.21.15.16:0/1833381287 conn(0x558c8d160a30 msgr2=0x558c8d166760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.358+0000 7f8f5effd640 0 client.17607 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.359+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 shutdown_connections 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.359+0000 7f8f87db8740 1 -- 172.21.15.16:0/1833381287 wait complete. 2024-09-12T02:56:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.359+0000 7f8f87db8740 20 client.17607 trim_cache size 0 max 16384 2024-09-12T02:56:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 Processor -- start 2024-09-12T02:56:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 -- start start 2024-09-12T02:56:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d1868e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d16f680 0x558c8d186e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ab340 con 0x558c8d16f680 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ace80 con 0x558c8d1644a0 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f87db8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801aba10 con 0x558c8d164dc0 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40634/0 (socket says 172.21.15.16:40634) 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f7f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d16f680 0x558c8d186e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f7effd640 1 -- 172.21.15.16:0/1103648678 learned_addr learned my addr 172.21.15.16:0/1103648678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.377+0000 7f8f77fff640 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d1868e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 -- 172.21.15.16:0/1103648678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 msgr2=0x558c8d1868e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d1868e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 -- 172.21.15.16:0/1103648678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d16f680 msgr2=0x558c8d186e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d16f680 0x558c8d186e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 -- 172.21.15.16:0/1103648678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c8d161bc0 con 0x558c8d1644a0 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f7effd640 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8f700055b0 tx=0x7f8f7002abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9eebf236ad536ac server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f757fa640 1 -- 172.21.15.16:0/1103648678 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7004b070 con 0x558c8d1644a0 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f757fa640 1 -- 172.21.15.16:0/1103648678 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f70004450 con 0x558c8d1644a0 2024-09-12T02:56:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.378+0000 7f8f757fa640 1 -- 172.21.15.16:0/1103648678 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70035070 con 0x558c8d1644a0 2024-09-12T02:56:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- 172.21.15.16:0/1103648678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 msgr2=0x558c8d17b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8f700055b0 tx=0x7f8f7002abe0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- 172.21.15.16:0/1103648678 shutdown_connections 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d16f680 0x558c8d186e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d1868e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 --2- 172.21.15.16:0/1103648678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d1644a0 0x558c8d17b4b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- 172.21.15.16:0/1103648678 >> 172.21.15.16:0/1103648678 conn(0x558c8d160a30 msgr2=0x558c8d16b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- 172.21.15.16:0/1103648678 shutdown_connections 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- 172.21.15.16:0/1103648678 wait complete. 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.379+0000 7f8f87db8740 1 -- start start 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 Processor -- start 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d177f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d174fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ac7b0 con 0x558c8d1644a0 2024-09-12T02:56:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ac0e0 con 0x558c8d16f680 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9320 con 0x558c8d164dc0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f87db8740 10 client.0 _ll_register_callbacks cb 0x7f8f78009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52106/0 (socket says 172.21.15.16:52106) 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 -- 172.21.15.16:0/3524182222 learned_addr learned my addr 172.21.15.16:0/3524182222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d177f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7f7fe640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d174fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 -- 172.21.15.16:0/3524182222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 msgr2=0x558c8d177f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d177f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 -- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 msgr2=0x558c8d174fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d174fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.380+0000 7f8f7effd640 1 -- 172.21.15.16:0/3524182222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c8d1757f0 con 0x558c8d1644a0 2024-09-12T02:56:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f7effd640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8f7003f040 tx=0x7f8f70003b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2128a8c5641e3d41 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7004b070 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8f7003c440 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f70004450 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f70035040 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 10 client.17613 fetch_fsmap learned FSMap version 32 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c8d1e9560 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f70004450 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f70050650 con 0x558c8d1644a0 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 10 client.17613 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 10 client.17613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:56:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 20 client.17613 populate_metadata read hostname 'smithi016' 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d1e9b10 0x558c8d1ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 4 client.17613 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.381+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c8d0d1d60 con 0x558c8d1e9b10 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d1e9b10 0x558c8d1ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f77fff640 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d1e9b10 0x558c8d1ebf00 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c157b96ac7406f18 server_cookie=c7c6127f1abbb31e in_seq=0 out_seq=0 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f5effd640 10 client.17613 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f5effd640 1 -- 172.21.15.16:0/3524182222 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558c8d0d1d60 con 0x558c8d1e9b10 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f5effd640 10 client.17613 handle_command_reply: tid=0 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.382+0000 7f8f87db8740 1 client.17613 shutdown 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 msgr2=0x558c8d1779f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8f7003f040 tx=0x7f8f70003b50 comp rx=0 tx=0).stop 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 shutdown_connections 2024-09-12T02:56:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558c8d1e9b10 0x558c8d1ebf00 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c8d16f680 0x558c8d174fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c8d164dc0 0x558c8d177f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 --2- 172.21.15.16:0/3524182222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558c8d1644a0 0x558c8d1779f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 >> 172.21.15.16:0/3524182222 conn(0x558c8d160a30 msgr2=0x558c8d15f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f5effd640 0 client.17613 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 shutdown_connections 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 1 -- 172.21.15.16:0/3524182222 wait complete. 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f87db8740 20 client.17613 trim_cache size 0 max 16384 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:56:46.388 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: "1b64b891-2409-445e-9d78-48bacfcae1e9": { 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "1b64b891-2409-445e-9d78-48bacfcae1e9", 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:56:46.389 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 msgr2=0x7f8f6c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 0x7f8f6c0a5040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8f780107d0 tx=0x7f8f78004a40 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 msgr2=0x7f8f80198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 0x7f8f80198940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800e7a0 tx=0x7f8f6800ec60 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 shutdown_connections 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f8010c170 0x7f8f8019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f80103660 0x7f8f80198940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8f6c0a2b80 0x7f8f6c0a5040 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 --2- 172.21.15.16:0/3171733369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102d40 0x7f8f80198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 >> 172.21.15.16:0/3171733369 conn(0x7f8f800fc7a0 msgr2=0x7f8f800fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:56:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 shutdown_connections 2024-09-12T02:56:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:56:46.383+0000 7f8f859df640 1 -- 172.21.15.16:0/3171733369 wait complete. 2024-09-12T02:56:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:46 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:56:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:46 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:46 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:56:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:46 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:46 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:46 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:56:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:46 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:46 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:56:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:46 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:46 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:46 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:56:46.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:46 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:46 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:56:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:46 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:46 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:56:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 858108 9.90 MB/sec execute 348 sec latency 35.427 ms 2024-09-12T02:56:47.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:47 smithi016 ceph-mon[30925]: pgmap v603: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-12T02:56:47.477 INFO:tasks.cephfs.filesystem:scrub status for tag:1b64b891-2409-445e-9d78-48bacfcae1e9 - {'path': '/', 'tag': '1b64b891-2409-445e-9d78-48bacfcae1e9', 'options': 'recursive,force'} 2024-09-12T02:56:47.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:47 smithi033 ceph-mon[35092]: pgmap v603: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-12T02:56:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:47 smithi184 ceph-mon[35321]: pgmap v603: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-09-12T02:56:47.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 860390 9.91 MB/sec execute 349 sec latency 21.235 ms 2024-09-12T02:56:48.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 863271 9.93 MB/sec execute 350 sec latency 12.547 ms 2024-09-12T02:56:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:49 smithi033 ceph-mon[35092]: pgmap v604: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-09-12T02:56:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:49 smithi184 ceph-mon[35321]: pgmap v604: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-09-12T02:56:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:49 smithi016 ceph-mon[30925]: pgmap v604: 97 pgs: 97 active+clean; 282 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-09-12T02:56:49.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 866379 9.96 MB/sec execute 351 sec latency 13.148 ms 2024-09-12T02:56:50.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 867942 9.95 MB/sec execute 352 sec latency 64.640 ms 2024-09-12T02:56:51.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:51 smithi033 ceph-mon[35092]: pgmap v605: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 295 op/s 2024-09-12T02:56:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:51 smithi184 ceph-mon[35321]: pgmap v605: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 295 op/s 2024-09-12T02:56:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:51 smithi016 ceph-mon[30925]: pgmap v605: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 295 op/s 2024-09-12T02:56:51.731 INFO:tasks.workunit.client.0.smithi016.stdout: 1 870626 9.96 MB/sec execute 353 sec latency 19.733 ms 2024-09-12T02:56:52.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 873677 9.98 MB/sec execute 354 sec latency 10.311 ms 2024-09-12T02:56:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:53 smithi033 ceph-mon[35092]: pgmap v606: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-09-12T02:56:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:56:53.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:56:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:53 smithi184 ceph-mon[35321]: pgmap v606: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-09-12T02:56:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:56:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:56:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:53 smithi016 ceph-mon[30925]: pgmap v606: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-09-12T02:56:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:56:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:56:53.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 876572 9.99 MB/sec execute 355 sec latency 13.187 ms 2024-09-12T02:56:54.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 879262 10.00 MB/sec execute 356 sec latency 11.665 ms 2024-09-12T02:56:55.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:55 smithi033 ceph-mon[35092]: pgmap v607: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.3 MiB/s wr, 231 op/s 2024-09-12T02:56:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:55 smithi184 ceph-mon[35321]: pgmap v607: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.3 MiB/s wr, 231 op/s 2024-09-12T02:56:55.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:55 smithi016 ceph-mon[30925]: pgmap v607: 97 pgs: 97 active+clean; 268 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.3 MiB/s wr, 231 op/s 2024-09-12T02:56:55.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 881159 10.01 MB/sec execute 357 sec latency 63.535 ms 2024-09-12T02:56:56.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 884258 10.03 MB/sec execute 358 sec latency 9.331 ms 2024-09-12T02:56:57.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:57 smithi033 ceph-mon[35092]: pgmap v608: 97 pgs: 97 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 306 op/s 2024-09-12T02:56:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:57 smithi184 ceph-mon[35321]: pgmap v608: 97 pgs: 97 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 306 op/s 2024-09-12T02:56:57.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:57 smithi016 ceph-mon[30925]: pgmap v608: 97 pgs: 97 active+clean; 273 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 306 op/s 2024-09-12T02:56:57.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 886895 10.04 MB/sec execute 359 sec latency 15.001 ms 2024-09-12T02:56:58.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 889054 10.04 MB/sec execute 360 sec latency 11.320 ms 2024-09-12T02:56:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:56:59 smithi184 ceph-mon[35321]: pgmap v609: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-09-12T02:56:59.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:56:59 smithi016 ceph-mon[30925]: pgmap v609: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-09-12T02:56:59.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 891826 10.06 MB/sec execute 361 sec latency 12.208 ms 2024-09-12T02:56:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:56:59 smithi033 ceph-mon[35092]: pgmap v609: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-09-12T02:57:00.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 894737 10.08 MB/sec execute 362 sec latency 48.013 ms 2024-09-12T02:57:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:01 smithi184 ceph-mon[35321]: pgmap v610: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 293 op/s 2024-09-12T02:57:01.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:01 smithi016 ceph-mon[30925]: pgmap v610: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 293 op/s 2024-09-12T02:57:01.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 897136 10.07 MB/sec execute 363 sec latency 20.630 ms 2024-09-12T02:57:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:01 smithi033 ceph-mon[35092]: pgmap v610: 97 pgs: 97 active+clean; 273 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 293 op/s 2024-09-12T02:57:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:57:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:57:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:57:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:57:02.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:57:02.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:57:02.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:57:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:57:02.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 899474 10.08 MB/sec execute 364 sec latency 20.246 ms 2024-09-12T02:57:02.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:57:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:57:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:57:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:57:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:03 smithi184 ceph-mon[35321]: pgmap v611: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-09-12T02:57:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:03 smithi016 ceph-mon[30925]: pgmap v611: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-09-12T02:57:03.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 902439 10.10 MB/sec execute 365 sec latency 11.300 ms 2024-09-12T02:57:03.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:03 smithi033 ceph-mon[35092]: pgmap v611: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-09-12T02:57:04.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 905662 10.13 MB/sec execute 366 sec latency 10.265 ms 2024-09-12T02:57:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:05 smithi184 ceph-mon[35321]: pgmap v612: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 264 op/s 2024-09-12T02:57:05.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:05 smithi016 ceph-mon[30925]: pgmap v612: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 264 op/s 2024-09-12T02:57:05.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 908186 10.13 MB/sec execute 367 sec latency 80.586 ms 2024-09-12T02:57:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:05 smithi033 ceph-mon[35092]: pgmap v612: 97 pgs: 97 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 264 op/s 2024-09-12T02:57:06.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 910077 10.13 MB/sec execute 368 sec latency 16.269 ms 2024-09-12T02:57:07.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:07 smithi184 ceph-mon[35321]: pgmap v613: 97 pgs: 97 active+clean; 282 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 335 op/s 2024-09-12T02:57:07.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:07 smithi016 ceph-mon[30925]: pgmap v613: 97 pgs: 97 active+clean; 282 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 335 op/s 2024-09-12T02:57:07.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 913043 10.15 MB/sec execute 369 sec latency 10.695 ms 2024-09-12T02:57:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:07 smithi033 ceph-mon[35092]: pgmap v613: 97 pgs: 97 active+clean; 282 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 335 op/s 2024-09-12T02:57:08.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 916072 10.17 MB/sec execute 370 sec latency 11.178 ms 2024-09-12T02:57:09.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:09 smithi184 ceph-mon[35321]: pgmap v614: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:57:09.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:09 smithi016 ceph-mon[30925]: pgmap v614: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:57:09.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 917816 10.21 MB/sec execute 371 sec latency 45.476 ms 2024-09-12T02:57:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:09 smithi033 ceph-mon[35092]: pgmap v614: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-09-12T02:57:10.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 920663 10.23 MB/sec execute 372 sec latency 58.870 ms 2024-09-12T02:57:11.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:11 smithi184 ceph-mon[35321]: pgmap v615: 97 pgs: 97 active+clean; 286 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-12T02:57:11.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:11 smithi016 ceph-mon[30925]: pgmap v615: 97 pgs: 97 active+clean; 286 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-12T02:57:11.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 923604 10.24 MB/sec execute 373 sec latency 18.952 ms 2024-09-12T02:57:11.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:11 smithi033 ceph-mon[35092]: pgmap v615: 97 pgs: 97 active+clean; 286 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-09-12T02:57:12.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 925882 10.24 MB/sec execute 374 sec latency 86.394 ms 2024-09-12T02:57:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:13 smithi184 ceph-mon[35321]: pgmap v616: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 293 op/s 2024-09-12T02:57:13.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:13 smithi016 ceph-mon[30925]: pgmap v616: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 293 op/s 2024-09-12T02:57:13.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 928280 10.25 MB/sec execute 375 sec latency 16.475 ms 2024-09-12T02:57:13.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:13 smithi033 ceph-mon[35092]: pgmap v616: 97 pgs: 97 active+clean; 285 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 293 op/s 2024-09-12T02:57:14.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 931431 10.27 MB/sec execute 376 sec latency 7.595 ms 2024-09-12T02:57:15.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:15 smithi184 ceph-mon[35321]: pgmap v617: 97 pgs: 97 active+clean; 284 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:57:15.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:15 smithi016 ceph-mon[30925]: pgmap v617: 97 pgs: 97 active+clean; 284 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:57:15.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 934267 10.28 MB/sec execute 377 sec latency 42.984 ms 2024-09-12T02:57:15.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:15 smithi033 ceph-mon[35092]: pgmap v617: 97 pgs: 97 active+clean; 284 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:57:16.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 936524 10.28 MB/sec execute 378 sec latency 30.334 ms 2024-09-12T02:57:17.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:57:17.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:17 smithi184 ceph-mon[35321]: pgmap v618: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:57:17.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:17 smithi016 ceph-mon[30925]: pgmap v618: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:57:17.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 938802 10.29 MB/sec execute 379 sec latency 82.272 ms 2024-09-12T02:57:17.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:17 smithi033 ceph-mon[35092]: pgmap v618: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T02:57:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.386+0000 7f98683e3640 1 -- 172.21.15.16:0/2467524401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860100a60 msgr2=0x7f9860100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.386+0000 7f98683e3640 1 --2- 172.21.15.16:0/2467524401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860100a60 0x7f9860100ec0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9850009930 tx=0x7f985002a420 comp rx=0 tx=0).stop 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 -- 172.21.15.16:0/2467524401 shutdown_connections 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 --2- 172.21.15.16:0/2467524401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860101400 0x7f986010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 --2- 172.21.15.16:0/2467524401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860100a60 0x7f9860100ec0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 --2- 172.21.15.16:0/2467524401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860106a60 0x7f9860106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 -- 172.21.15.16:0/2467524401 >> 172.21.15.16:0/2467524401 conn(0x7f98600fc780 msgr2=0x7f98600feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 -- 172.21.15.16:0/2467524401 shutdown_connections 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 -- 172.21.15.16:0/2467524401 wait complete. 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 Processor -- start 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.387+0000 7f98683e3640 1 -- start start 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860100a60 0x7f986019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 0x7f986019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9860110e60 con 0x7f9860101400 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9860110ce0 con 0x7f9860100a60 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f98683e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9860110fe0 con 0x7f9860106a60 2024-09-12T02:57:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50998/0 (socket says 172.21.15.16:50998) 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 -- 172.21.15.16:0/3286725282 learned_addr learned my addr 172.21.15.16:0/3286725282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f9866980640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 0x7f986019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986617f640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860100a60 0x7f986019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 -- 172.21.15.16:0/3286725282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 msgr2=0x7f986019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 0x7f986019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 -- 172.21.15.16:0/3286725282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860100a60 msgr2=0x7f986019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860100a60 0x7f986019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 -- 172.21.15.16:0/3286725282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986019f610 con 0x7f9860101400 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f9866980640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 0x7f986019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.388+0000 7f986597e640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9850009230 tx=0x7f9850004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.389+0000 7f98577fe640 1 -- 172.21.15.16:0/3286725282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985003f030 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.389+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986019f8a0 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.389+0000 7f98577fe640 1 -- 172.21.15.16:0/3286725282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9850004440 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.389+0000 7f98577fe640 1 -- 172.21.15.16:0/3286725282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850035070 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.389+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98601a71e0 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.391+0000 7f98577fe640 1 -- 172.21.15.16:0/3286725282 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9850042030 con 0x7f9860101400 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.391+0000 7f986a7bc740 1 Processor -- start 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.391+0000 7f986a7bc740 1 -- start start 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f1b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 0x561f26f26140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f28a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f26e812e0 con 0x561f26f1bdc0 2024-09-12T02:57:18.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f26e28ce0 con 0x561f26f1b4a0 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986a7bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f26e7d2a0 con 0x561f26f26680 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f98577fe640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 0x7f984c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f9866980640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f1b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f9866980640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f1b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57804/0 (socket says 172.21.15.16:57804) 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f9866980640 1 -- 172.21.15.16:0/607114644 learned_addr learned my addr 172.21.15.16:0/607114644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986597e640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f28a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f9866980640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 0x7f984c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f986617f640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 0x561f26f26140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.392+0000 7f98577fe640 1 -- 172.21.15.16:0/3286725282 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f98500ef2f0 con 0x7f9860101400 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 -- 172.21.15.16:0/607114644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 msgr2=0x561f26f1b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f1b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 -- 172.21.15.16:0/607114644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 msgr2=0x561f26f26140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 0x561f26f26140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 -- 172.21.15.16:0/607114644 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f26e0d440 con 0x561f26f26680 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986617f640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 0x561f26f26140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f986597e640 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f28a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f98500bb9f0 tx=0x7f9850009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f952edf781e5a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f9866980640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 0x7f984c0a5040 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9858005c40 tx=0x7f9858004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f9854dea640 1 -- 172.21.15.16:0/607114644 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850049070 con 0x561f26f26680 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f9854dea640 1 -- 172.21.15.16:0/607114644 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98500bff10 con 0x561f26f26680 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.393+0000 7f9854dea640 1 -- 172.21.15.16:0/607114644 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98500878e0 con 0x561f26f26680 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 -- 172.21.15.16:0/607114644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 msgr2=0x561f26f28a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f28a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f98500bb9f0 tx=0x7f9850009260 comp rx=0 tx=0).stop 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 -- 172.21.15.16:0/607114644 shutdown_connections 2024-09-12T02:57:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f28a70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1bdc0 0x561f26f26140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 --2- 172.21.15.16:0/607114644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f1b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 -- 172.21.15.16:0/607114644 >> 172.21.15.16:0/607114644 conn(0x561f26f17a30 msgr2=0x561f26f17e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 -- 172.21.15.16:0/607114644 shutdown_connections 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.394+0000 7f986a7bc740 1 -- 172.21.15.16:0/607114644 wait complete. 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 -- start start 2024-09-12T02:57:18.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 Processor -- start 2024-09-12T02:57:18.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f25fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1bdc0 0x561f26f23280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f26f29010 con 0x561f26f26680 2024-09-12T02:57:18.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f26e3c050 con 0x561f26f1b4a0 2024-09-12T02:57:18.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f26f29190 con 0x561f26f1bdc0 2024-09-12T02:57:18.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.395+0000 7f986a7bc740 10 client.0 _ll_register_callbacks cb 0x561f26e7c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9866980640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9866980640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51018/0 (socket says 172.21.15.16:51018) 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9866980640 1 -- 172.21.15.16:0/3506603881 learned_addr learned my addr 172.21.15.16:0/3506603881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f25fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986597e640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1bdc0 0x561f26f23280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 -- 172.21.15.16:0/3506603881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1bdc0 msgr2=0x561f26f23280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1bdc0 0x561f26f23280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 -- 172.21.15.16:0/3506603881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 msgr2=0x561f26f237e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 -- 172.21.15.16:0/3506603881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f26bf23c0 con 0x561f26f1b4a0 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9866980640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f986617f640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f25fd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f985c0126d0 tx=0x7f985c012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54ed94aa1fc154da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.396+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c00f920 con 0x561f26f1b4a0 2024-09-12T02:57:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98500bff10 con 0x561f26f1b4a0 2024-09-12T02:57:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f985c00fac0 con 0x561f26f1b4a0 2024-09-12T02:57:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f985c00fcc0 con 0x561f26f1b4a0 2024-09-12T02:57:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 10 client.27107 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561f26f349b0 con 0x561f26f1b4a0 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98500bff10 con 0x561f26f1b4a0 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f985c010550 con 0x561f26f1b4a0 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 10 client.27107 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 10 client.27107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 20 client.27107 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26f35e70 0x561f26f382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 4 client.27107 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.397+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f26e8e850 con 0x561f26f35e70 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.398+0000 7f986597e640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26f35e70 0x561f26f382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.398+0000 7f986597e640 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26f35e70 0x561f26f382c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f40f490eded197d1 server_cookie=3ea59ab9da0cab53 in_seq=0 out_seq=0 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.398+0000 7f9849ffb640 10 client.27107 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f9849ffb640 1 -- 172.21.15.16:0/3506603881 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f26e8e850 con 0x561f26f35e70 2024-09-12T02:57:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f9849ffb640 10 client.27107 handle_command_reply: tid=0 2024-09-12T02:57:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 client.27107 shutdown 2024-09-12T02:57:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 msgr2=0x561f26f25fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f25fd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f985c0126d0 tx=0x7f985c012b90 comp rx=0 tx=0).stop 2024-09-12T02:57:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 shutdown_connections 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26f35e70 0x561f26f382c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1bdc0 0x561f26f23280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f9849ffb640 0 client.27107 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3506603881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1b4a0 0x561f26f25fd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 >> 172.21.15.16:0/3506603881 conn(0x561f26f17a30 msgr2=0x561f26f1d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 shutdown_connections 2024-09-12T02:57:18.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 1 -- 172.21.15.16:0/3506603881 wait complete. 2024-09-12T02:57:18.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.399+0000 7f986a7bc740 20 client.27107 trim_cache size 0 max 16384 2024-09-12T02:57:18.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 Processor -- start 2024-09-12T02:57:18.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 -- start start 2024-09-12T02:57:18.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f22b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f1e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98601af7c0 con 0x561f26f1b4a0 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98601b1300 con 0x561f26f1bdc0 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986a7bc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98601afe90 con 0x561f26f26680 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.418+0000 7f986617f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986617f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51024/0 (socket says 172.21.15.16:51024) 2024-09-12T02:57:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986617f640 1 -- 172.21.15.16:0/2724118188 learned_addr learned my addr 172.21.15.16:0/2724118188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f22b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f9866980640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f1e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 -- 172.21.15.16:0/2724118188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 msgr2=0x561f26f1e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f1e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 -- 172.21.15.16:0/2724118188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 msgr2=0x561f26f32180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 -- 172.21.15.16:0/2724118188 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f26f38940 con 0x561f26f1bdc0 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986617f640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f9866980640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f1e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f986597e640 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f22b40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f98500a52b0 tx=0x7f9850046040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c1b653484dd6df server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f984bfff640 1 -- 172.21.15.16:0/2724118188 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850049040 con 0x561f26f1bdc0 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f984bfff640 1 -- 172.21.15.16:0/2724118188 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98500bff10 con 0x561f26f1bdc0 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.419+0000 7f984bfff640 1 -- 172.21.15.16:0/2724118188 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850031e30 con 0x561f26f1bdc0 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.420+0000 7f986a7bc740 1 -- 172.21.15.16:0/2724118188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 msgr2=0x561f26f22b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.420+0000 7f986a7bc740 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f22b40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f98500a52b0 tx=0x7f9850046040 comp rx=0 tx=0).stop 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 -- 172.21.15.16:0/2724118188 shutdown_connections 2024-09-12T02:57:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f26680 0x561f26f1e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f22b40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 --2- 172.21.15.16:0/2724118188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f1b4a0 0x561f26f32180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 -- 172.21.15.16:0/2724118188 >> 172.21.15.16:0/2724118188 conn(0x561f26f17a30 msgr2=0x561f26f1e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 -- 172.21.15.16:0/2724118188 shutdown_connections 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 -- 172.21.15.16:0/2724118188 wait complete. 2024-09-12T02:57:18.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 -- start start 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.421+0000 7f986a7bc740 1 Processor -- start 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f29f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f2e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34270/0 (socket says 172.21.15.16:34270) 2024-09-12T02:57:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 -- 172.21.15.16:0/3880525008 learned_addr learned my addr 172.21.15.16:0/3880525008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98601b0c30 con 0x561f26f26680 2024-09-12T02:57:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98601b0560 con 0x561f26f1bdc0 2024-09-12T02:57:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98601ad7a0 con 0x561f26f1b4a0 2024-09-12T02:57:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986597e640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f29f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f9866980640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f2e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 -- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 msgr2=0x561f26f29f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f29f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 -- 172.21.15.16:0/3880525008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 msgr2=0x561f26f2e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.422+0000 7f986617f640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f2e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986617f640 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f26f2e710 con 0x561f26f1b4a0 2024-09-12T02:57:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986a7bc740 10 client.0 _ll_register_callbacks cb 0x7f985c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98500bff10 con 0x561f26f1b4a0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986617f640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f985c009d20 tx=0x7f985c009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b96e774620fa5947 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f9849ffb640 10 client.27013 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f9849ffb640 1 -- 172.21.15.16:0/3880525008 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c036050 con 0x561f26f1b4a0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f9849ffb640 1 -- 172.21.15.16:0/3880525008 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f985c010430 con 0x561f26f1b4a0 2024-09-12T02:57:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f9849ffb640 1 -- 172.21.15.16:0/3880525008 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f985c0106b0 con 0x561f26f1b4a0 2024-09-12T02:57:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986a7bc740 10 client.27013 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.423+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=32}) -- 0x561f26fa0530 con 0x561f26f1b4a0 2024-09-12T02:57:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f9849ffb640 1 -- 172.21.15.16:0/3880525008 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f985c01c470 con 0x561f26f1b4a0 2024-09-12T02:57:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 10 client.27013 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 10 client.27013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 20 client.27013 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26fa0af0 0x561f26fa2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 4 client.27013 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561f26e88d60 con 0x561f26fa0af0 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986597e640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26fa0af0 0x561f26fa2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f986597e640 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26fa0af0 0x561f26fa2ee0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b81918030e8d43e5 server_cookie=4958e5e9315a022f in_seq=0 out_seq=0 2024-09-12T02:57:18.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f9849ffb640 10 client.27013 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:18.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f9849ffb640 1 -- 172.21.15.16:0/3880525008 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561f26e88d60 con 0x561f26fa0af0 2024-09-12T02:57:18.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.424+0000 7f9849ffb640 10 client.27013 handle_command_reply: tid=0 2024-09-12T02:57:18.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 client.27013 shutdown 2024-09-12T02:57:18.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 msgr2=0x561f26f29a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f985c009d20 tx=0x7f985c009e40 comp rx=0 tx=0).stop 2024-09-12T02:57:18.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 shutdown_connections 2024-09-12T02:57:18.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561f26fa0af0 0x561f26fa2ee0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f26f26680 0x561f26f2e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f26f1bdc0 0x561f26f29f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 --2- 172.21.15.16:0/3880525008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f26f1b4a0 0x561f26f29a20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f9849ffb640 0 client.27013 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 >> 172.21.15.16:0/3880525008 conn(0x561f26f17a30 msgr2=0x561f26f1e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 shutdown_connections 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 1 -- 172.21.15.16:0/3880525008 wait complete. 2024-09-12T02:57:18.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.425+0000 7f986a7bc740 20 client.27013 trim_cache size 0 max 16384 2024-09-12T02:57:18.444 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:18.444 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:57:18.444 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:57:18.445 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:18.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 msgr2=0x7f984c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 0x7f984c0a5040 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9858005c40 tx=0x7f9858004ab0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 msgr2=0x7f986019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:18.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9850009230 tx=0x7f9850004040 comp rx=0 tx=0).stop 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 shutdown_connections 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9860106a60 0x7f986019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f984c0a2b80 0x7f984c0a5040 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9860101400 0x7f986019aba0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 --2- 172.21.15.16:0/3286725282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9860100a60 0x7f986019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:18.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 >> 172.21.15.16:0/3286725282 conn(0x7f98600fc780 msgr2=0x7f986006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:18.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 shutdown_connections 2024-09-12T02:57:18.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:18.426+0000 7f98683e3640 1 -- 172.21.15.16:0/3286725282 wait complete. 2024-09-12T02:57:18.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 941804 10.31 MB/sec execute 380 sec latency 20.473 ms 2024-09-12T02:57:19.244 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:57:19.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:57:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:19.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 944280 10.31 MB/sec execute 381 sec latency 21.780 ms 2024-09-12T02:57:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 -- 172.21.15.16:0/760413965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178102b90 msgr2=0x7f2178102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 --2- 172.21.15.16:0/760413965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178102b90 0x7f2178102f70 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2168009900 tx=0x7f216802a480 comp rx=0 tx=0).stop 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 -- 172.21.15.16:0/760413965 shutdown_connections 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 --2- 172.21.15.16:0/760413965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f217810bfc0 0x7f217810e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 --2- 172.21.15.16:0/760413965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21781034b0 0x7f217810b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 --2- 172.21.15.16:0/760413965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178102b90 0x7f2178102f70 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 -- 172.21.15.16:0/760413965 >> 172.21.15.16:0/760413965 conn(0x7f21780fc610 msgr2=0x7f21780fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 -- 172.21.15.16:0/760413965 shutdown_connections 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.065+0000 7f217cde2640 1 -- 172.21.15.16:0/760413965 wait complete. 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.066+0000 7f217cde2640 1 Processor -- start 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 -- start start 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2178102b90 0x7f21781007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 0x7f2178101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2178111e10 con 0x7f217810bfc0 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2178111c90 con 0x7f2178102b90 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f217cde2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2178111f90 con 0x7f21781034b0 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f2176ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f2176ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34284/0 (socket says 172.21.15.16:34284) 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f2176ffd640 1 -- 172.21.15.16:0/2582474783 learned_addr learned my addr 172.21.15.16:0/2582474783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f2177fff640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 0x7f2178101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.067+0000 7f21777fe640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2178102b90 0x7f21781007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 -- 172.21.15.16:0/2582474783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2178102b90 msgr2=0x7f21781007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2178102b90 0x7f21781007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 -- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 msgr2=0x7f2178101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 0x7f2178101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 -- 172.21.15.16:0/2582474783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21781aac20 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2177fff640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 0x7f2178101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.068+0000 7f2176ffd640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f216c00e9f0 tx=0x7f216c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.070+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c00ccc0 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.070+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216c010910 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.070+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21781aae50 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.070+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c018920 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.071+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21781ab2b0 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.072+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217806a850 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.073+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f216c018ac0 con 0x7f21781034b0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.074+0000 7f2174ff9640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 0x7f21600a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.074+0000 7f21777fe640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 0x7f21600a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.074+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f216c0cf200 con 0x7f21781034b0 2024-09-12T02:57:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.075+0000 7f21777fe640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 0x7f21600a50c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2168000c00 tx=0x7f2168005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.079+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f216c09bb20 con 0x7f21781034b0 2024-09-12T02:57:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.325+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2178107090 con 0x7f21781034b0 2024-09-12T02:57:20.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:20 smithi016 ceph-mon[30925]: pgmap v619: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 259 op/s 2024-09-12T02:57:20.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.326+0000 7f2174ff9640 1 -- 172.21.15.16:0/2582474783 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f216c09fe20 con 0x7f21781034b0 2024-09-12T02:57:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:57:20.328 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:57:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:57:20.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 msgr2=0x7f21600a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 0x7f21600a50c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2168000c00 tx=0x7f2168005b50 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 msgr2=0x7f2178100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f216c00e9f0 tx=0x7f216c00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 shutdown_connections 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21600a2c00 0x7f21600a50c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f217810bfc0 0x7f2178101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21781034b0 0x7f2178100d30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 --2- 172.21.15.16:0/2582474783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2178102b90 0x7f21781007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 >> 172.21.15.16:0/2582474783 conn(0x7f21780fc610 msgr2=0x7f21780fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 shutdown_connections 2024-09-12T02:57:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:20.328+0000 7f217cde2640 1 -- 172.21.15.16:0/2582474783 wait complete. 2024-09-12T02:57:20.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:20 smithi033 ceph-mon[35092]: pgmap v619: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 259 op/s 2024-09-12T02:57:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:20 smithi184 ceph-mon[35321]: pgmap v619: 97 pgs: 97 active+clean; 279 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 259 op/s 2024-09-12T02:57:20.732 INFO:tasks.workunit.client.0.smithi016.stdout: 1 946288 10.31 MB/sec execute 382 sec latency 20.019 ms 2024-09-12T02:57:21.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:57:21.397 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:21 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2582474783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:21.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:21 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2582474783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:21.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:21 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2582474783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:21.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 948953 10.33 MB/sec execute 383 sec latency 47.143 ms 2024-09-12T02:57:21.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 -- 172.21.15.16:0/4172457182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8106a80 msgr2=0x7fb0d8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 --2- 172.21.15.16:0/4172457182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8106a80 0x7fb0d8106e60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009930 tx=0x7fb0d402a420 comp rx=0 tx=0).stop 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 -- 172.21.15.16:0/4172457182 shutdown_connections 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 --2- 172.21.15.16:0/4172457182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8101420 0x7fb0d810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 --2- 172.21.15.16:0/4172457182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 0x7fb0d8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 --2- 172.21.15.16:0/4172457182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8106a80 0x7fb0d8106e60 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 -- 172.21.15.16:0/4172457182 >> 172.21.15.16:0/4172457182 conn(0x7fb0d80fc7a0 msgr2=0x7fb0d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 -- 172.21.15.16:0/4172457182 shutdown_connections 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.920+0000 7fb0e0250640 1 -- 172.21.15.16:0/4172457182 wait complete. 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 Processor -- start 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 -- start start 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 0x7fb0d8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8106a80 0x7fb0d819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8110ed0 con 0x7fb0d8101420 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8110d50 con 0x7fb0d8100a80 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.921+0000 7fb0e0250640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8111050 con 0x7fb0d8106a80 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51076/0 (socket says 172.21.15.16:51076) 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/2181866066 learned_addr learned my addr 172.21.15.16:0/2181866066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8106a80 0x7fb0d819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 0x7fb0d8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/2181866066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8106a80 msgr2=0x7fb0d819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8106a80 0x7fb0d819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/2181866066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 msgr2=0x7fb0d8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 0x7fb0d8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/2181866066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d819d400 con 0x7fb0d8101420 2024-09-12T02:57:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e9f0 tx=0x7fb0c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0ceffd640 1 -- 172.21.15.16:0/2181866066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800cd80 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0ceffd640 1 -- 172.21.15.16:0/2181866066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c8010910 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0ceffd640 1 -- 172.21.15.16:0/2181866066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8018990 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d819d6f0 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.922+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d81a4fd0 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.925+0000 7fb0ceffd640 1 -- 172.21.15.16:0/2181866066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb0c8010ac0 con 0x7fb0d8101420 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.925+0000 7fb0e2629740 1 Processor -- start 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.925+0000 7fb0e2629740 1 -- start start 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57efb40 0x5615d57ee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57fb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5615d57542f0 con 0x5615d57ee5d0 2024-09-12T02:57:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5615d56fbcf0 con 0x5615d57efb40 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0e2629740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5615d57502b0 con 0x5615d57f9370 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51078/0 (socket says 172.21.15.16:51078) 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 -- 172.21.15.16:0/2667234829 learned_addr learned my addr 172.21.15.16:0/2667234829 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57fb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57efb40 0x5615d57ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ceffd640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 0x7fb0c40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ceffd640 1 -- 172.21.15.16:0/2181866066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0c80cf300 con 0x7fb0d8101420 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 -- 172.21.15.16:0/2667234829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 msgr2=0x5615d57fb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57fb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 -- 172.21.15.16:0/2667234829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57efb40 msgr2=0x5615d57ee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57efb40 0x5615d57ee090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 0x7fb0c40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0ddfec640 1 -- 172.21.15.16:0/2667234829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d56e03a0 con 0x5615d57ee5d0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.926+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57fb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.927+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009930 tx=0x7fb0d4005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5f107ba2eb9e38 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.927+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 0x7fb0c40a50c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0005fd0 tx=0x7fb0d0004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.927+0000 7fb0c3fff640 1 -- 172.21.15.16:0/2667234829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4035040 con 0x5615d57ee5d0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.927+0000 7fb0c3fff640 1 -- 172.21.15.16:0/2667234829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d4043070 con 0x5615d57ee5d0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.927+0000 7fb0c3fff640 1 -- 172.21.15.16:0/2667234829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d403fe30 con 0x5615d57ee5d0 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- 172.21.15.16:0/2667234829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 msgr2=0x5615d57eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009930 tx=0x7fb0d4005bf0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- 172.21.15.16:0/2667234829 shutdown_connections 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57fb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d57eea10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 --2- 172.21.15.16:0/2667234829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57efb40 0x5615d57ee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- 172.21.15.16:0/2667234829 >> 172.21.15.16:0/2667234829 conn(0x5615d57b7310 msgr2=0x5615d57eac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- 172.21.15.16:0/2667234829 shutdown_connections 2024-09-12T02:57:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- 172.21.15.16:0/2667234829 wait complete. 2024-09-12T02:57:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 -- start start 2024-09-12T02:57:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.928+0000 7fb0e2629740 1 Processor -- start 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5615d57fc010 con 0x5615d57efb40 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5615d570f060 con 0x5615d57f9370 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5615d57fc190 con 0x5615d57ee5d0 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0e2629740 10 client.0 _ll_register_callbacks cb 0x5615d54628c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51088/0 (socket says 172.21.15.16:51088) 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/3871327668 learned_addr learned my addr 172.21.15.16:0/3871327668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34328/0 (socket says 172.21.15.16:34328) 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 -- 172.21.15.16:0/3871327668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 msgr2=0x5615d57f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 -- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 msgr2=0x5615d57f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0ddfec640 1 -- 172.21.15.16:0/3871327668 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d54c53c0 con 0x5615d57ee5d0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.929+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb0d402a940 tx=0x7fb0d4005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b5fa110594b9ad server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4055060 con 0x5615d57ee5d0 2024-09-12T02:57:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d4043070 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0d4005040 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb0d4035070 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0e2629740 10 client.27031 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615d57fc060 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d4043070 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb0d4004b40 con 0x5615d57ee5d0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0e2629740 10 client.27031 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.930+0000 7fb0e2629740 10 client.27031 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0e2629740 20 client.27031 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d5808cb0 0x5615d580b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0e2629740 4 client.27031 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615d5761860 con 0x5615d5808cb0 2024-09-12T02:57:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d5808cb0 0x5615d580b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0c17fa640 10 client.27031 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.931+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d5808cb0 0x5615d580b0a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=834882e78df49fa0 server_cookie=6ea176001a0fd5b9 in_seq=0 out_seq=0 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0c17fa640 1 -- 172.21.15.16:0/3871327668 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5615d5761860 con 0x5615d5808cb0 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0c17fa640 10 client.27031 handle_command_reply: tid=0 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 client.27031 shutdown 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 msgr2=0x5615d57f8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb0d402a940 tx=0x7fb0d4005b50 comp rx=0 tx=0).stop 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 shutdown_connections 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d5808cb0 0x5615d580b0a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0c17fa640 0 client.27031 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:57:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3871327668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57ee5d0 0x5615d57f8d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 >> 172.21.15.16:0/3871327668 conn(0x5615d57b7310 msgr2=0x5615d57f0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 shutdown_connections 2024-09-12T02:57:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 1 -- 172.21.15.16:0/3871327668 wait complete. 2024-09-12T02:57:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.932+0000 7fb0e2629740 20 client.27031 trim_cache size 0 max 16384 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.946+0000 7fb0e2629740 1 Processor -- start 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.946+0000 7fb0e2629740 1 -- start start 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 0x5615d58052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57f1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ab2d0 con 0x5615d57efb40 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ace10 con 0x5615d57ee5d0 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0e2629740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ab9a0 con 0x5615d57f9370 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51102/0 (socket says 172.21.15.16:51102) 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/3414559316 learned_addr learned my addr 172.21.15.16:0/3414559316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 0x5615d58052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/3414559316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 msgr2=0x5615d57f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/3414559316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 msgr2=0x5615d58052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 0x5615d58052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 -- 172.21.15.16:0/3414559316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d57e5f70 con 0x5615d57efb40 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 0x5615d58052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8098540 tx=0x7fb0c80ccc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c75ba1f787d2b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0c37fe640 1 -- 172.21.15.16:0/3414559316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80a4780 con 0x5615d57efb40 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0c37fe640 1 -- 172.21.15.16:0/3414559316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c8099ec0 con 0x5615d57efb40 2024-09-12T02:57:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.947+0000 7fb0c37fe640 1 -- 172.21.15.16:0/3414559316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c809b300 con 0x5615d57efb40 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- 172.21.15.16:0/3414559316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 msgr2=0x5615d57e5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8098540 tx=0x7fb0c80ccc80 comp rx=0 tx=0).stop 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- 172.21.15.16:0/3414559316 shutdown_connections 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57f9370 0x5615d57f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57efb40 0x5615d57e5a30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 --2- 172.21.15.16:0/3414559316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57ee5d0 0x5615d58052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- 172.21.15.16:0/3414559316 >> 172.21.15.16:0/3414559316 conn(0x5615d57b7310 msgr2=0x5615d57f5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- 172.21.15.16:0/3414559316 shutdown_connections 2024-09-12T02:57:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- 172.21.15.16:0/3414559316 wait complete. 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.948+0000 7fb0e2629740 1 -- start start 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 Processor -- start 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d5801600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57efb40 0x5615d5801b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ac740 con 0x5615d57ee5d0 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ac070 con 0x5615d57f9370 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81a92d0 con 0x5615d57efb40 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0e2629740 10 client.0 _ll_register_callbacks cb 0x7fb0d0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d5801600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0de7ed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0ddfec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d5801600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51116/0 (socket says 172.21.15.16:51116) 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0ddfec640 1 -- 172.21.15.16:0/1688249524 learned_addr learned my addr 172.21.15.16:0/1688249524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0dd7eb640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57efb40 0x5615d5801b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.949+0000 7fb0de7ed640 1 -- 172.21.15.16:0/1688249524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57efb40 msgr2=0x5615d5801b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57efb40 0x5615d5801b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0de7ed640 1 -- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 msgr2=0x5615d5801600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d5801600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0de7ed640 1 -- 172.21.15.16:0/1688249524 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d57f2ec0 con 0x5615d57f9370 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0de7ed640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f2820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0012ed0 tx=0x7fb0d0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247cc65f65949d2f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000f900 con 0x5615d57f9370 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c8099ec0 con 0x5615d57f9370 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0d0019070 con 0x5615d57f9370 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb0d000faa0 con 0x5615d57f9370 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0e2629740 10 client.27134 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615d5873180 con 0x5615d57f9370 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.950+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c8099ec0 con 0x5615d57f9370 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb0d0010430 con 0x5615d57f9370 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 10 client.27134 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 10 client.27134 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 20 client.27134 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d58738e0 0x5615d5875d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 4 client.27134 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615d575bd70 con 0x5615d58738e0 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d58738e0 0x5615d5875d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0ddfec640 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d58738e0 0x5615d5875d00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=71f5a66ea52e6494 server_cookie=61880ed950ef0203 in_seq=0 out_seq=0 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0c17fa640 10 client.27134 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:57:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0c17fa640 1 -- 172.21.15.16:0/1688249524 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615d575bd70 con 0x5615d58738e0 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0c17fa640 10 client.27134 handle_command_reply: tid=0 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 1 client.27134 shutdown 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 msgr2=0x5615d57f2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.951+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f2820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0012ed0 tx=0x7fb0d0009500 comp rx=0 tx=0).stop 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 shutdown_connections 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5615d58738e0 0x5615d5875d00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5615d57f9370 0x5615d57f2820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0c17fa640 0 client.27134 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5615d57efb40 0x5615d5801b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 --2- 172.21.15.16:0/1688249524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d57ee5d0 0x5615d5801600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 >> 172.21.15.16:0/1688249524 conn(0x5615d57b7310 msgr2=0x5615d57e94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 shutdown_connections 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 1 -- 172.21.15.16:0/1688249524 wait complete. 2024-09-12T02:57:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e2629740 20 client.27134 trim_cache size 0 max 16384 2024-09-12T02:57:21.959 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:57:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 msgr2=0x7fb0c40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 0x7fb0c40a50c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0005fd0 tx=0x7fb0d0004ab0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 msgr2=0x7fb0d8198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e9f0 tx=0x7fb0c800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 shutdown_connections 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d8106a80 0x7fb0d819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c40a2c00 0x7fb0c40a50c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8101420 0x7fb0d8198990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 --2- 172.21.15.16:0/2181866066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8100a80 0x7fb0d8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.952+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 >> 172.21.15.16:0/2181866066 conn(0x7fb0d80fc7a0 msgr2=0x7fb0d806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.953+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 shutdown_connections 2024-09-12T02:57:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:21.953+0000 7fb0e0250640 1 -- 172.21.15.16:0/2181866066 wait complete. 2024-09-12T02:57:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:22 smithi016 ceph-mon[30925]: pgmap v620: 97 pgs: 97 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-09-12T02:57:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:22 smithi033 ceph-mon[35092]: pgmap v620: 97 pgs: 97 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-09-12T02:57:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:22 smithi184 ceph-mon[35321]: pgmap v620: 97 pgs: 97 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 318 op/s 2024-09-12T02:57:22.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:57:22.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 952149 10.35 MB/sec execute 384 sec latency 14.752 ms 2024-09-12T02:57:22.925 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:23.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:23 smithi016 ceph-mon[30925]: pgmap v621: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 277 op/s 2024-09-12T02:57:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- 172.21.15.16:0/2412307340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 msgr2=0x7fada410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 --2- 172.21.15.16:0/2412307340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada410ea90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009ca0 tx=0x7fad9c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- 172.21.15.16:0/2412307340 shutdown_connections 2024-09-12T02:57:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 --2- 172.21.15.16:0/2412307340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada410ea90 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 --2- 172.21.15.16:0/2412307340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 --2- 172.21.15.16:0/2412307340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 0x7fada41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- 172.21.15.16:0/2412307340 >> 172.21.15.16:0/2412307340 conn(0x7fada40fd880 msgr2=0x7fada40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- 172.21.15.16:0/2412307340 shutdown_connections 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- 172.21.15.16:0/2412307340 wait complete. 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 Processor -- start 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.448+0000 7fada965c640 1 -- start start 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 0x7fada4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fada4111f70 con 0x7fada4106f10 2024-09-12T02:57:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fada4111df0 con 0x7fada4102790 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada965c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fada41120f0 con 0x7fada4101de0 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51130/0 (socket says 172.21.15.16:51130) 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 -- 172.21.15.16:0/1256076742 learned_addr learned my addr 172.21.15.16:0/1256076742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 0x7fada4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 -- 172.21.15.16:0/1256076742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 msgr2=0x7fada4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 0x7fada4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 -- 172.21.15.16:0/1256076742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 msgr2=0x7fada41988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 -- 172.21.15.16:0/1256076742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada419d320 con 0x7fada4106f10 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.449+0000 7fada37fe640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fad9c031a20 tx=0x7fad9c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.450+0000 7fada0ff9640 1 -- 172.21.15.16:0/1256076742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c003b70 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.450+0000 7fada0ff9640 1 -- 172.21.15.16:0/1256076742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad9c003d10 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.450+0000 7fada0ff9640 1 -- 172.21.15.16:0/1256076742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c02fbd0 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.450+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada419d5b0 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.450+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada41a4ef0 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.452+0000 7fada0ff9640 1 -- 172.21.15.16:0/1256076742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fad9c02fd70 con 0x7fada4106f10 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.452+0000 7fadaba35740 1 Processor -- start 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.452+0000 7fadaba35740 1 -- start start 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55d4780 0x55caf55d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caf552b2f0 con 0x55caf55caf50 2024-09-12T02:57:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55caf54d2cf0 con 0x55caf55d4780 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fadaba35740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55caf55272b0 con 0x55caf55c99e0 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fada2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fada0ff9640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 0x7fad900a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51140/0 (socket says 172.21.15.16:51140) 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 -- 172.21.15.16:0/1563422252 learned_addr learned my addr 172.21.15.16:0/1563422252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55d4780 0x55caf55d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fada37fe640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 0x7fad900a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 -- 172.21.15.16:0/1563422252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 msgr2=0x55caf55c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 -- 172.21.15.16:0/1563422252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55d4780 msgr2=0x55caf55d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55d4780 0x55caf55d6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fad9bfff640 1 -- 172.21.15.16:0/1563422252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caf54b73a0 con 0x55caf55caf50 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.453+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.454+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00afd0 tx=0x7fad8c00aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c26e4726a5e600 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.454+0000 7fada0ff9640 1 -- 172.21.15.16:0/1256076742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad9c0f01b0 con 0x7fada4106f10 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.454+0000 7fad997fa640 1 -- 172.21.15.16:0/1563422252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00ddd0 con 0x55caf55caf50 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.454+0000 7fad997fa640 1 -- 172.21.15.16:0/1563422252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8c005500 con 0x55caf55caf50 2024-09-12T02:57:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.454+0000 7fad997fa640 1 -- 172.21.15.16:0/1563422252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c004ba0 con 0x55caf55caf50 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- 172.21.15.16:0/1563422252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 msgr2=0x55caf55c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fada37fe640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 0x7fad900a50c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0bc9e0 tx=0x7fad9c0b8d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00afd0 tx=0x7fad8c00aa60 comp rx=0 tx=0).stop 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- 172.21.15.16:0/1563422252 shutdown_connections 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55d4780 0x55caf55d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 --2- 172.21.15.16:0/1563422252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55caf50 0x55caf55c94a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- 172.21.15.16:0/1563422252 >> 172.21.15.16:0/1563422252 conn(0x55caf558e310 msgr2=0x55caf55c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- 172.21.15.16:0/1563422252 shutdown_connections 2024-09-12T02:57:23.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- 172.21.15.16:0/1563422252 wait complete. 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 -- start start 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.455+0000 7fadaba35740 1 Processor -- start 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caf55d7420 con 0x55caf55d4780 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55caf54e6060 con 0x55caf55caf50 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55caf55d75a0 con 0x55caf55c99e0 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fadaba35740 10 client.0 _ll_register_callbacks cb 0x55caf52398c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34370/0 (socket says 172.21.15.16:34370) 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad9bfff640 1 -- 172.21.15.16:0/1345345348 learned_addr learned my addr 172.21.15.16:0/1345345348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 -- 172.21.15.16:0/1345345348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 msgr2=0x55caf55d41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 -- 172.21.15.16:0/1345345348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 msgr2=0x55caf55d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 -- 172.21.15.16:0/1345345348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caf529c3c0 con 0x55caf55caf50 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fada2ffd640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d1450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad940126d0 tx=0x7fad94012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9177f0da6d20129 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.456+0000 7fad8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94011bb0 con 0x55caf55caf50 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad94011d50 con 0x55caf55caf50 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8c005500 con 0x55caf55caf50 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad94004030 con 0x55caf55caf50 2024-09-12T02:57:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 10 client.27143 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caf55e2f50 con 0x55caf55caf50 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8c005500 con 0x55caf55caf50 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fad9401c460 con 0x55caf55caf50 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 10 client.27143 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 10 client.27143 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.457+0000 7fadaba35740 20 client.27143 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf55e4460 0x55caf55e6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fadaba35740 4 client.27143 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caf5538860 con 0x55caf55e4460 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf55e4460 0x55caf55e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fad9bfff640 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf55e4460 0x55caf55e6880 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a8fed2d63f8f2e65 server_cookie=c4947bbc4e405279 in_seq=0 out_seq=0 2024-09-12T02:57:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.458+0000 7fad8a7fc640 10 client.27143 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fad8a7fc640 1 -- 172.21.15.16:0/1345345348 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55caf5538860 con 0x55caf55e4460 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fad8a7fc640 10 client.27143 handle_command_reply: tid=0 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 client.27143 shutdown 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 msgr2=0x55caf55d1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d1450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad940126d0 tx=0x7fad94012b90 comp rx=0 tx=0).stop 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 shutdown_connections 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf55e4460 0x55caf55e6880 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d1450 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fad8a7fc640 0 client.27143 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 --2- 172.21.15.16:0/1345345348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55d41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 >> 172.21.15.16:0/1345345348 conn(0x55caf558e310 msgr2=0x55caf55cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 shutdown_connections 2024-09-12T02:57:23.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 1 -- 172.21.15.16:0/1345345348 wait complete. 2024-09-12T02:57:23.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.459+0000 7fadaba35740 20 client.27143 trim_cache size 0 max 16384 2024-09-12T02:57:23.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 Processor -- start 2024-09-12T02:57:23.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 -- start start 2024-09-12T02:57:23.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 0x55caf55ec130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55ec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fada41ab240 con 0x55caf55d4780 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fada41acd80 con 0x55caf55c99e0 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fadaba35740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fada41ab910 con 0x55caf55caf50 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57926/0 (socket says 172.21.15.16:57926) 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fada2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 0x55caf55ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 -- 172.21.15.16:0/727312342 learned_addr learned my addr 172.21.15.16:0/727312342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 -- 172.21.15.16:0/727312342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 msgr2=0x55caf55ec130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 0x55caf55ec130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 -- 172.21.15.16:0/727312342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 msgr2=0x55caf55ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.473+0000 7fad9bfff640 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55ec670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fad9bfff640 1 -- 172.21.15.16:0/727312342 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caf55d2040 con 0x55caf55c99e0 2024-09-12T02:57:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fada2ffd640 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 0x55caf55ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fad9bfff640 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00af10 tx=0x7fad8c024040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7217cdcd2969c0b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fad98ff9640 1 -- 172.21.15.16:0/727312342 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c012450 con 0x55caf55c99e0 2024-09-12T02:57:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fad98ff9640 1 -- 172.21.15.16:0/727312342 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8c012ad0 con 0x55caf55c99e0 2024-09-12T02:57:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.474+0000 7fad98ff9640 1 -- 172.21.15.16:0/727312342 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00fab0 con 0x55caf55c99e0 2024-09-12T02:57:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- 172.21.15.16:0/727312342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 msgr2=0x55caf55e06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00af10 tx=0x7fad8c024040 comp rx=0 tx=0).stop 2024-09-12T02:57:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- 172.21.15.16:0/727312342 shutdown_connections 2024-09-12T02:57:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55caf50 0x55caf55ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- 172.21.15.16:0/727312342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55c99e0 0x55caf55e06c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- 172.21.15.16:0/727312342 >> 172.21.15.16:0/727312342 conn(0x55caf558e310 msgr2=0x55caf55d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- 172.21.15.16:0/727312342 shutdown_connections 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- 172.21.15.16:0/727312342 wait complete. 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- start start 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 Processor -- start 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55dd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fada41ac6b0 con 0x55caf55d4780 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fada41abfe0 con 0x55caf55caf50 2024-09-12T02:57:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fada41a9220 con 0x55caf55c99e0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.475+0000 7fadaba35740 10 client.0 _ll_register_callbacks cb 0x7fad940096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51168/0 (socket says 172.21.15.16:51168) 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 -- 172.21.15.16:0/3774694031 learned_addr learned my addr 172.21.15.16:0/3774694031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55dd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fada2ffd640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 -- 172.21.15.16:0/3774694031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 msgr2=0x55caf55dd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55dd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 -- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 msgr2=0x55caf55d8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 -- 172.21.15.16:0/3774694031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caf55d9110 con 0x55caf55d4780 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad9bfff640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005ee0 tx=0x7fad8c00f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6123cb99e58a12c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c012650 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8c012ad0 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad8c0127f0 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad8c00f830 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fadaba35740 10 client.17673 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.476+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caf564f1a0 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8c012ad0 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fad8c03d030 con 0x55caf55d4780 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 10 client.17673 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 10 client.17673 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 20 client.17673 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf564f620 0x55caf5651a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 4 client.17673 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55caf5532d70 con 0x55caf564f620 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.477+0000 7fada2ffd640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf564f620 0x55caf5651a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fada2ffd640 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf564f620 0x55caf5651a10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c76d31d0863496b server_cookie=50830525d183482b in_seq=0 out_seq=0 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fad8a7fc640 10 client.17673 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fad8a7fc640 1 -- 172.21.15.16:0/3774694031 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55caf5532d70 con 0x55caf564f620 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fad8a7fc640 10 client.17673 handle_command_reply: tid=0 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 client.17673 shutdown 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 msgr2=0x55caf55d88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005ee0 tx=0x7fad8c00f750 comp rx=0 tx=0).stop 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 shutdown_connections 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55caf564f620 0x55caf5651a10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caf55d4780 0x55caf55d88e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55caf55caf50 0x55caf55d8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fad8a7fc640 0 client.17673 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 --2- 172.21.15.16:0/3774694031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55caf55c99e0 0x55caf55dd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 >> 172.21.15.16:0/3774694031 conn(0x55caf558e310 msgr2=0x55caf55d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 shutdown_connections 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 1 -- 172.21.15.16:0/3774694031 wait complete. 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.478+0000 7fadaba35740 20 client.17673 trim_cache size 0 max 16384 2024-09-12T02:57:23.485 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 msgr2=0x7fad900a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 0x7fad900a50c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0bc9e0 tx=0x7fad9c0b8d90 comp rx=0 tx=0).stop 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 msgr2=0x7fada419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fad9c031a20 tx=0x7fad9c031a50 comp rx=0 tx=0).stop 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 shutdown_connections 2024-09-12T02:57:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad900a2c00 0x7fad900a50c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada4106f10 0x7fada419cc40 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fada4102790 0x7fada41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 --2- 172.21.15.16:0/1256076742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fada4101de0 0x7fada4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 >> 172.21.15.16:0/1256076742 conn(0x7fada40fd880 msgr2=0x7fada40ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 shutdown_connections 2024-09-12T02:57:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:23.479+0000 7fada965c640 1 -- 172.21.15.16:0/1256076742 wait complete. 2024-09-12T02:57:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:23 smithi033 ceph-mon[35092]: pgmap v621: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 277 op/s 2024-09-12T02:57:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:23 smithi184 ceph-mon[35321]: pgmap v621: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 277 op/s 2024-09-12T02:57:23.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 954693 10.35 MB/sec execute 385 sec latency 31.242 ms 2024-09-12T02:57:24.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:57:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:24.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 957179 10.36 MB/sec execute 386 sec latency 16.210 ms 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 -- 172.21.15.16:0/2550521639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244109140 msgr2=0x7f5244110cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 --2- 172.21.15.16:0/2550521639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244109140 0x7f5244110cc0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f523c009900 tx=0x7f523c02a410 comp rx=0 tx=0).stop 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 -- 172.21.15.16:0/2550521639 shutdown_connections 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 --2- 172.21.15.16:0/2550521639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244109140 0x7f5244110cc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 --2- 172.21.15.16:0/2550521639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52441049c0 0x7f5244108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 --2- 172.21.15.16:0/2550521639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5244104010 0x7f52441043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 -- 172.21.15.16:0/2550521639 >> 172.21.15.16:0/2550521639 conn(0x7f52440ffa50 msgr2=0x7f5244101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 -- 172.21.15.16:0/2550521639 shutdown_connections 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.078+0000 7f5248f52640 1 -- 172.21.15.16:0/2550521639 wait complete. 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 Processor -- start 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 -- start start 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 0x7f524419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5244109140 0x7f524419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5244114150 con 0x7f5244104010 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5244113fd0 con 0x7f5244109140 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.079+0000 7f5248f52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52441142d0 con 0x7f52441049c0 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51180/0 (socket says 172.21.15.16:51180) 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 -- 172.21.15.16:0/3200482255 learned_addr learned my addr 172.21.15.16:0/3200482255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f5241d9b640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 0x7f524419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f5242d9d640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5244109140 0x7f524419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 -- 172.21.15.16:0/3200482255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 msgr2=0x7f524419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 0x7f524419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 -- 172.21.15.16:0/3200482255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5244109140 msgr2=0x7f524419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5244109140 0x7f524419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 -- 172.21.15.16:0/3200482255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524419f5a0 con 0x7f5244104010 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f5241d9b640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 0x7f524419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f524259c640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f523400eab0 tx=0x7f523400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f52337fe640 1 -- 172.21.15.16:0/3200482255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523400ce40 con 0x7f5244104010 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524419f890 con 0x7f5244104010 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f52337fe640 1 -- 172.21.15.16:0/3200482255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5234011950 con 0x7f5244104010 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.080+0000 7f52337fe640 1 -- 172.21.15.16:0/3200482255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234010ae0 con 0x7f5244104010 2024-09-12T02:57:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.081+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52441a7170 con 0x7f5244104010 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.083+0000 7f52337fe640 1 -- 172.21.15.16:0/3200482255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5234011b00 con 0x7f5244104010 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.083+0000 7f524b32b740 1 Processor -- start 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 -- start start 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f52337fe640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 0x7f522c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859ddd9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55859dd3f2f0 con 0x55859dde4370 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55859dce6cf0 con 0x55859dddab40 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524b32b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55859dd3b2b0 con 0x55859ddd95d0 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f5242d9d640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 0x7f522c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f52337fe640 1 -- 172.21.15.16:0/3200482255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52340cf6d0 con 0x7f5244104010 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57972/0 (socket says 172.21.15.16:57972) 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 -- 172.21.15.16:0/1715117883 learned_addr learned my addr 172.21.15.16:0/1715117883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f5241d9b640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859ddd9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f5242d9d640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 -- 172.21.15.16:0/1715117883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 msgr2=0x55859ddd9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859ddd9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 -- 172.21.15.16:0/1715117883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 msgr2=0x55859dde6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f524259c640 1 -- 172.21.15.16:0/1715117883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859dccb3a0 con 0x55859dddab40 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.084+0000 7f5242d9d640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:57:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f524259c640 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5234098340 tx=0x7f523409bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=badcd8f506dee2af server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f5242d9d640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 0x7f522c0a5040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f523c0096c0 tx=0x7f523c005be0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f5230ff9640 1 -- 172.21.15.16:0/1715117883 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523409bee0 con 0x55859dddab40 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f5230ff9640 1 -- 172.21.15.16:0/1715117883 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52340a0290 con 0x55859dddab40 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f5230ff9640 1 -- 172.21.15.16:0/1715117883 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523409c140 con 0x55859dddab40 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f524b32b740 1 -- 172.21.15.16:0/1715117883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 msgr2=0x55859ddd9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.085+0000 7f524b32b740 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5234098340 tx=0x7f523409bde0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- 172.21.15.16:0/1715117883 shutdown_connections 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859ddd9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- 172.21.15.16:0/1715117883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dddab40 0x55859ddd9090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- 172.21.15.16:0/1715117883 >> 172.21.15.16:0/1715117883 conn(0x55859dda2310 msgr2=0x55859ddd5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- 172.21.15.16:0/1715117883 shutdown_connections 2024-09-12T02:57:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- 172.21.15.16:0/1715117883 wait complete. 2024-09-12T02:57:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- start start 2024-09-12T02:57:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 Processor -- start 2024-09-12T02:57:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859dde0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:25 smithi016 ceph-mon[30925]: pgmap v622: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:57:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55859dde7070 con 0x55859dde4370 2024-09-12T02:57:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55859dcfa060 con 0x55859ddd95d0 2024-09-12T02:57:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55859dde71f0 con 0x55859dddab40 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.086+0000 7f524b32b740 10 client.0 _ll_register_callbacks cb 0x55859da4d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57986/0 (socket says 172.21.15.16:57986) 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 -- 172.21.15.16:0/17617520 learned_addr learned my addr 172.21.15.16:0/17617520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5241d9b640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859dde0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5242d9d640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 -- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 msgr2=0x55859dde0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859dde0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 -- 172.21.15.16:0/17617520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 msgr2=0x55859dde1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 -- 172.21.15.16:0/17617520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859dab03c0 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524259c640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52340a4620 tx=0x7f5234005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2657c787807aae4c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52340b4070 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52340a48c0 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52340a0290 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.087+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f523409bee0 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 10 client.27158 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55859ddf2a10 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52340a0290 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f523400f040 con 0x55859ddd95d0 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 10 client.27158 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 10 client.27158 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 20 client.27158 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859ddf3d60 0x55859ddf6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 4 client.27158 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55859dd4c860 con 0x55859ddf3d60 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.088+0000 7f5241d9b640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859ddf3d60 0x55859ddf6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.089+0000 7f5241d9b640 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859ddf3d60 0x55859ddf6150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf167c28be0bb00d server_cookie=468048fd867020b1 in_seq=0 out_seq=0 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.089+0000 7f5229ffb640 10 client.27158 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.089+0000 7f5229ffb640 1 -- 172.21.15.16:0/17617520 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55859dd4c860 con 0x55859ddf3d60 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.089+0000 7f5229ffb640 10 client.27158 handle_command_reply: tid=0 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.089+0000 7f524b32b740 1 client.27158 shutdown 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 msgr2=0x55859dde3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52340a4620 tx=0x7f5234005ac0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 shutdown_connections 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dde4370 0x55859dde1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859ddf3d60 0x55859ddf6150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859dde0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f5229ffb640 0 client.27158 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 --2- 172.21.15.16:0/17617520 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859ddd95d0 0x55859dde3d00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 >> 172.21.15.16:0/17617520 conn(0x55859dda2310 msgr2=0x55859dddb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 shutdown_connections 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 1 -- 172.21.15.16:0/17617520 wait complete. 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.090+0000 7f524b32b740 20 client.27158 trim_cache size 0 max 16384 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.104+0000 7f524b32b740 1 Processor -- start 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.104+0000 7f524b32b740 1 -- start start 2024-09-12T02:57:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.104+0000 7f524b32b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859dddc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524b32b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524b32b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524b32b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52441af710 con 0x55859dddab40 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524b32b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b1250 con 0x55859dde4370 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524b32b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52441afde0 con 0x55859ddd95d0 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f5241d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f5241d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51202/0 (socket says 172.21.15.16:51202) 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859dddc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f5241d9b640 1 -- 172.21.15.16:0/2261318018 learned_addr learned my addr 172.21.15.16:0/2261318018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f5242d9d640 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 -- 172.21.15.16:0/2261318018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 msgr2=0x55859ddf9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 -- 172.21.15.16:0/2261318018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 msgr2=0x55859dddc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 -- 172.21.15.16:0/2261318018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859dddcba0 con 0x55859ddd95d0 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f5241d9b640 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f524259c640 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859dddc120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f523409c0a0 tx=0x7f5234000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=933ddc033779c617 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f522bfff640 1 -- 172.21.15.16:0/2261318018 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52340b4070 con 0x55859ddd95d0 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f522bfff640 1 -- 172.21.15.16:0/2261318018 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523408d020 con 0x55859ddd95d0 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.105+0000 7f522bfff640 1 -- 172.21.15.16:0/2261318018 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52340a4920 con 0x55859ddd95d0 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 -- 172.21.15.16:0/2261318018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 msgr2=0x55859dddc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859dddc120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f523409c0a0 tx=0x7f5234000f40 comp rx=0 tx=0).stop 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 -- 172.21.15.16:0/2261318018 shutdown_connections 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859dddab40 0x55859dddc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 --2- 172.21.15.16:0/2261318018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859ddd95d0 0x55859dddc120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 -- 172.21.15.16:0/2261318018 >> 172.21.15.16:0/2261318018 conn(0x55859dda2310 msgr2=0x55859dde00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 -- 172.21.15.16:0/2261318018 shutdown_connections 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.106+0000 7f524b32b740 1 -- 172.21.15.16:0/2261318018 wait complete. 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 -- start start 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 Processor -- start 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b0b80 con 0x55859ddd95d0 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b04b0 con 0x55859dde4370 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52441ad6f0 con 0x55859dddab40 2024-09-12T02:57:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524b32b740 10 client.0 _ll_register_callbacks cb 0x7f5234098550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f5241d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f5241d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34442/0 (socket says 172.21.15.16:34442) 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f5241d9b640 1 -- 172.21.15.16:0/4290349004 learned_addr learned my addr 172.21.15.16:0/4290349004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f524259c640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51214/0 (socket says 172.21.15.16:51214) 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.107+0000 7f5242d9d640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 -- 172.21.15.16:0/4290349004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 msgr2=0x55859ddf6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 -- 172.21.15.16:0/4290349004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 msgr2=0x55859ddf9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 -- 172.21.15.16:0/4290349004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859ddf7430 con 0x55859dddab40 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f524259c640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5241d9b640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f52380058c0 tx=0x7f5238010e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b229a3734eb01dd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238004020 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f523408d020 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52380041c0 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f52380043c0 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f524b32b740 10 client.27073 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55859de5e4e0 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.108+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523408d020 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f52380276d0 con 0x55859dddab40 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 10 client.27073 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 10 client.27073 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 20 client.27073 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859de5ec70 0x55859de61090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 4 client.27073 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55859dd46d70 con 0x55859de5ec70 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524259c640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859de5ec70 0x55859de61090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f5229ffb640 10 client.27073 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524259c640 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859de5ec70 0x55859de61090 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=219462b0a368d3c server_cookie=1209e3af465ef363 in_seq=0 out_seq=0 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f5229ffb640 1 -- 172.21.15.16:0/4290349004 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55859dd46d70 con 0x55859de5ec70 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f5229ffb640 10 client.27073 handle_command_reply: tid=0 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.109+0000 7f524b32b740 1 client.27073 shutdown 2024-09-12T02:57:25.283 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 msgr2=0x55859ddf6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f52380058c0 tx=0x7f5238010e90 comp rx=0 tx=0).stop 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 shutdown_connections 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55859dde4370 0x55859ddf6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55859de5ec70 0x55859de61090 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55859dddab40 0x55859ddf6980 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 --2- 172.21.15.16:0/4290349004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55859ddd95d0 0x55859ddf9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f5229ffb640 0 client.27073 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 >> 172.21.15.16:0/4290349004 conn(0x55859dda2310 msgr2=0x55859dde2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 shutdown_connections 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 1 -- 172.21.15.16:0/4290349004 wait complete. 2024-09-12T02:57:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f524b32b740 20 client.27073 trim_cache size 0 max 16384 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 msgr2=0x7f522c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 0x7f522c0a5040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f523c0096c0 tx=0x7f523c005be0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 msgr2=0x7f524419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.110+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f523400eab0 tx=0x7f523400ef70 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 shutdown_connections 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5244109140 0x7f524419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52441049c0 0x7f524419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f522c0a2b80 0x7f522c0a5040 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 --2- 172.21.15.16:0/3200482255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5244104010 0x7f524419a5f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 >> 172.21.15.16:0/3200482255 conn(0x7f52440ffa50 msgr2=0x7f5244100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 shutdown_connections 2024-09-12T02:57:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:25.111+0000 7f5248f52640 1 -- 172.21.15.16:0/3200482255 wait complete. 2024-09-12T02:57:25.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:25 smithi033 ceph-mon[35092]: pgmap v622: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:57:25.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:25 smithi184 ceph-mon[35321]: pgmap v622: 97 pgs: 97 active+clean; 256 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:57:25.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 960027 10.37 MB/sec execute 387 sec latency 55.700 ms 2024-09-12T02:57:25.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:57:26.020 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T02:57:26.020 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-09-12T02:57:26.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:57:26.263 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:26.303 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:26.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 963043 10.40 MB/sec execute 388 sec latency 59.898 ms 2024-09-12T02:57:26.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 -- 172.21.15.16:0/824932646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2708102db0 msgr2=0x7f2708103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 --2- 172.21.15.16:0/824932646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2708102db0 0x7f2708103190 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f2704009930 tx=0x7f270402a420 comp rx=0 tx=0).stop 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 -- 172.21.15.16:0/824932646 shutdown_connections 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 --2- 172.21.15.16:0/824932646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27081082e0 0x7f270810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 --2- 172.21.15.16:0/824932646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708103760 0x7f2708107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 --2- 172.21.15.16:0/824932646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2708102db0 0x7f2708103190 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 -- 172.21.15.16:0/824932646 >> 172.21.15.16:0/824932646 conn(0x7f27080fc7d0 msgr2=0x7f27080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 -- 172.21.15.16:0/824932646 shutdown_connections 2024-09-12T02:57:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.747+0000 7f270f4a4640 1 -- 172.21.15.16:0/824932646 wait complete. 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 Processor -- start 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 -- start start 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2708103760 0x7f2708198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 0x7f270819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2708112090 con 0x7f27081082e0 2024-09-12T02:57:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2708111f10 con 0x7f2708103760 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270f4a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2708112210 con 0x7f2708102db0 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270d240640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270da41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 0x7f270819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270d240640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34736/0 (socket says 172.21.15.16:34736) 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.748+0000 7f270d240640 1 -- 172.21.15.16:0/3573257161 learned_addr learned my addr 172.21.15.16:0/3573257161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270ca3f640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2708103760 0x7f2708198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 -- 172.21.15.16:0/3573257161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2708103760 msgr2=0x7f2708198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2708103760 0x7f2708198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 -- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 msgr2=0x7f270819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 0x7f270819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 -- 172.21.15.16:0/3573257161 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270819d550 con 0x7f2708102db0 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270da41640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 0x7f270819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270d240640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2704004770 tx=0x7f2704003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f26fe7fc640 1 -- 172.21.15.16:0/3573257161 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270403f070 con 0x7f2708102db0 2024-09-12T02:57:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270819d7e0 con 0x7f2708102db0 2024-09-12T02:57:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f26fe7fc640 1 -- 172.21.15.16:0/3573257161 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2704031c20 con 0x7f2708102db0 2024-09-12T02:57:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f26fe7fc640 1 -- 172.21.15.16:0/3573257161 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704035030 con 0x7f2708102db0 2024-09-12T02:57:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.749+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27081a5120 con 0x7f2708102db0 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 Processor -- start 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 -- start start 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f26fe7fc640 1 -- 172.21.15.16:0/3573257161 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2704042070 con 0x7f2708102db0 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f29a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f36760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d531e8f2f0 con 0x55d531f295d0 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d531e36cf0 con 0x55d531f34370 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f271187d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d531e8b2b0 con 0x55d531f2ab40 2024-09-12T02:57:26.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270d240640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f29a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270da41640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270da41640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34752/0 (socket says 172.21.15.16:34752) 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270d240640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f29a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41854/0 (socket says 172.21.15.16:41854) 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270da41640 1 -- 172.21.15.16:0/1050417598 learned_addr learned my addr 172.21.15.16:0/1050417598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.752+0000 7f270ca3f640 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f36760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 -- 172.21.15.16:0/1050417598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 msgr2=0x55d531f36760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f36760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 -- 172.21.15.16:0/1050417598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 msgr2=0x55d531f29a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f29a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 -- 172.21.15.16:0/1050417598 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d531e1b3a0 con 0x55d531f2ab40 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270da41640 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2700011790 tx=0x7f2700011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea50faba2bb96a40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f26fe7fc640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 0x7f26f40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f26f37fe640 1 -- 172.21.15.16:0/1050417598 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700003c30 con 0x55d531f2ab40 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f26f37fe640 1 -- 172.21.15.16:0/1050417598 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f270000fe80 con 0x55d531f2ab40 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f26f37fe640 1 -- 172.21.15.16:0/1050417598 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27000193e0 con 0x55d531f2ab40 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f270d240640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 0x7f26f40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.753+0000 7f26fe7fc640 1 -- 172.21.15.16:0/3573257161 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27040ef9c0 con 0x7f2708102db0 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 -- 172.21.15.16:0/1050417598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 msgr2=0x55d531f29090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f270d240640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 0x7f26f40a5040 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f27040ed840 tx=0x7f27040f5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2700011790 tx=0x7f2700011c50 comp rx=0 tx=0).stop 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 -- 172.21.15.16:0/1050417598 shutdown_connections 2024-09-12T02:57:26.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f36760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f29a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 --2- 172.21.15.16:0/1050417598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f29090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 -- 172.21.15.16:0/1050417598 >> 172.21.15.16:0/1050417598 conn(0x55d531ef2310 msgr2=0x55d531f25c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 -- 172.21.15.16:0/1050417598 shutdown_connections 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.754+0000 7f271187d740 1 -- 172.21.15.16:0/1050417598 wait complete. 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 -- start start 2024-09-12T02:57:26.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 Processor -- start 2024-09-12T02:57:26.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f34370 0x55d531f314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d531f37070 con 0x55d531f2ab40 2024-09-12T02:57:26.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d531e4a060 con 0x55d531f295d0 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d531f371f0 con 0x55d531f34370 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f271187d740 10 client.0 _ll_register_callbacks cb 0x55d531b9d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f270da41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f270ca3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f270da41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33994/0 (socket says 172.21.15.16:33994) 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f270ca3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41862/0 (socket says 172.21.15.16:41862) 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.755+0000 7f270da41640 1 -- 172.21.15.16:0/43099290 learned_addr learned my addr 172.21.15.16:0/43099290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270da41640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f34370 0x55d531f314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 -- 172.21.15.16:0/43099290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f34370 msgr2=0x55d531f314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f34370 0x55d531f314d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 -- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 msgr2=0x55d531f33a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 -- 172.21.15.16:0/43099290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d531c003c0 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270da41640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f270ca3f640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f27080fee40 tx=0x7f26f8012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f398ab1789b2474c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f26f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f800f8f0 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d531e9c860 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26f800fa90 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26f800fc90 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f271187d740 10 client.17700 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.756+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d531f4b3d0 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d531e9c860 con 0x55d531f2ab40 2024-09-12T02:57:26.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26f801c420 con 0x55d531f2ab40 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 10 client.17700 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 10 client.17700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 20 client.17700 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531f4c7e0 0x55d531f4ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 4 client.17700 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d531e96d70 con 0x55d531f4c7e0 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.757+0000 7f270da41640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531f4c7e0 0x55d531f4ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f26f0ff9640 10 client.17700 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f270da41640 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531f4c7e0 0x55d531f4ec20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c7f2f31f1ed7bf server_cookie=87f29d20728d712 in_seq=0 out_seq=0 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f26f0ff9640 1 -- 172.21.15.16:0/43099290 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d531e96d70 con 0x55d531f4c7e0 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f26f0ff9640 10 client.17700 handle_command_reply: tid=0 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f271187d740 1 client.17700 shutdown 2024-09-12T02:57:26.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 msgr2=0x55d531f30f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f27080fee40 tx=0x7f26f8012b60 comp rx=0 tx=0).stop 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 shutdown_connections 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.758+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531f4c7e0 0x55d531f4ec20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f34370 0x55d531f314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f2ab40 0x55d531f30f90 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 --2- 172.21.15.16:0/43099290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f295d0 0x55d531f33a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f26f0ff9640 0 client.17700 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 >> 172.21.15.16:0/43099290 conn(0x55d531ef2310 msgr2=0x55d531f2b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 shutdown_connections 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 1 -- 172.21.15.16:0/43099290 wait complete. 2024-09-12T02:57:26.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.759+0000 7f271187d740 20 client.17700 trim_cache size 0 max 16384 2024-09-12T02:57:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 Processor -- start 2024-09-12T02:57:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 -- start start 2024-09-12T02:57:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f2c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f2c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27081ab3e0 con 0x55d531f295d0 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27081acf20 con 0x55d531f34370 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f271187d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27081abab0 con 0x55d531f2ab40 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f270da41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f270da41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41878/0 (socket says 172.21.15.16:41878) 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.772+0000 7f270da41640 1 -- 172.21.15.16:0/801430678 learned_addr learned my addr 172.21.15.16:0/801430678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270ca3f640 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f2c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f2c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 -- 172.21.15.16:0/801430678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 msgr2=0x55d531f2c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f2c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 -- 172.21.15.16:0/801430678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 msgr2=0x55d531f2c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f2c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 -- 172.21.15.16:0/801430678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d531f2cba0 con 0x55d531f295d0 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f270da41640 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2700011790 tx=0x7f2700009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d07555e84d06505 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f26f2ffd640 1 -- 172.21.15.16:0/801430678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27000041b0 con 0x55d531f295d0 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f26f2ffd640 1 -- 172.21.15.16:0/801430678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27000049a0 con 0x55d531f295d0 2024-09-12T02:57:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.773+0000 7f26f2ffd640 1 -- 172.21.15.16:0/801430678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700005470 con 0x55d531f295d0 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- 172.21.15.16:0/801430678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 msgr2=0x55d531f24fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2700011790 tx=0x7f2700009500 comp rx=0 tx=0).stop 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- 172.21.15.16:0/801430678 shutdown_connections 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f34370 0x55d531f2c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f2ab40 0x55d531f2c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 --2- 172.21.15.16:0/801430678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f295d0 0x55d531f24fd0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- 172.21.15.16:0/801430678 >> 172.21.15.16:0/801430678 conn(0x55d531ef2310 msgr2=0x55d531f39ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- 172.21.15.16:0/801430678 shutdown_connections 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- 172.21.15.16:0/801430678 wait complete. 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.774+0000 7f271187d740 1 -- start start 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 Processor -- start 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f295d0 0x55d531f49770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f34370 0x55d531f2e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27081ac850 con 0x55d531f34370 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27081ac180 con 0x55d531f2ab40 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27081a93c0 con 0x55d531f295d0 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f271187d740 10 client.0 _ll_register_callbacks cb 0x55d531f342c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:26.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270da41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f34370 0x55d531f2e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34008/0 (socket says 172.21.15.16:34008) 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270da41640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f295d0 0x55d531f49770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 -- 172.21.15.16:0/334928602 learned_addr learned my addr 172.21.15.16:0/334928602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 -- 172.21.15.16:0/334928602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f295d0 msgr2=0x55d531f49770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f295d0 0x55d531f49770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 -- 172.21.15.16:0/334928602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f34370 msgr2=0x55d531f2e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f34370 0x55d531f2e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.775+0000 7f270ca3f640 1 -- 172.21.15.16:0/334928602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d531f2e620 con 0x55d531f2ab40 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f270ca3f640 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27080fee40 tx=0x7f26f800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=185f7c55683a0bde server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f801c420 con 0x55d531f2ab40 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26f8018040 con 0x55d531f2ab40 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27000049a0 con 0x55d531f2ab40 2024-09-12T02:57:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26f801c5c0 con 0x55d531f2ab40 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f271187d740 10 client.27182 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d531fb7210 con 0x55d531f2ab40 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27000049a0 con 0x55d531f2ab40 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26f8003610 con 0x55d531f2ab40 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.776+0000 7f271187d740 10 client.27182 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 10 client.27182 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 20 client.27182 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531fb7930 0x55d531fb9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 4 client.27182 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d531e13170 con 0x55d531fb7930 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f270da41640 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531fb7930 0x55d531fb9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f26f0ff9640 10 client.27182 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:57:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f270da41640 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531fb7930 0x55d531fb9d50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=658e963fd21b9d45 server_cookie=ffc8f20bc3f5f3fa in_seq=0 out_seq=0 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f26f0ff9640 1 -- 172.21.15.16:0/334928602 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d531e13170 con 0x55d531fb7930 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f26f0ff9640 10 client.27182 handle_command_reply: tid=0 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.777+0000 7f271187d740 1 client.27182 shutdown 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 msgr2=0x55d531f2dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27080fee40 tx=0x7f26f800fc00 comp rx=0 tx=0).stop 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 shutdown_connections 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d531fb7930 0x55d531fb9d50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d531f34370 0x55d531f2e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d531f2ab40 0x55d531f2dba0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f26f0ff9640 0 client.27182 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 --2- 172.21.15.16:0/334928602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d531f295d0 0x55d531f49770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 >> 172.21.15.16:0/334928602 conn(0x55d531ef2310 msgr2=0x55d531f39480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 shutdown_connections 2024-09-12T02:57:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 1 -- 172.21.15.16:0/334928602 wait complete. 2024-09-12T02:57:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f271187d740 20 client.27182 trim_cache size 0 max 16384 2024-09-12T02:57:26.785 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 msgr2=0x7f26f40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 0x7f26f40a5040 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f27040ed840 tx=0x7f27040f5550 comp rx=0 tx=0).stop 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.778+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 msgr2=0x7f27081985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2704004770 tx=0x7f2704003b90 comp rx=0 tx=0).stop 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 shutdown_connections 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26f40a2b80 0x7f26f40a5040 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27081082e0 0x7f270819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2708103760 0x7f2708198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 --2- 172.21.15.16:0/3573257161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2708102db0 0x7f27081985a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 >> 172.21.15.16:0/3573257161 conn(0x7f27080fc7d0 msgr2=0x7f2708105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 shutdown_connections 2024-09-12T02:57:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:26.779+0000 7f270f4a4640 1 -- 172.21.15.16:0/3573257161 wait complete. 2024-09-12T02:57:27.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:27 smithi016 ceph-mon[30925]: pgmap v623: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-09-12T02:57:27.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:27 smithi033 ceph-mon[35092]: pgmap v623: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-09-12T02:57:27.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:27 smithi184 ceph-mon[35321]: pgmap v623: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-09-12T02:57:27.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:57:27.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 965608 10.40 MB/sec execute 389 sec latency 35.567 ms 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 -- 172.21.15.16:0/1439179907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0102d30 msgr2=0x7fd1d0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/1439179907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0102d30 0x7fd1d0103110 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009930 tx=0x7fd1c002a420 comp rx=0 tx=0).stop 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 -- 172.21.15.16:0/1439179907 shutdown_connections 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/1439179907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d01082a0 0x7fd1d010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/1439179907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/1439179907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0102d30 0x7fd1d0103110 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 -- 172.21.15.16:0/1439179907 >> 172.21.15.16:0/1439179907 conn(0x7fd1d00fc7d0 msgr2=0x7fd1d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 -- 172.21.15.16:0/1439179907 shutdown_connections 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.887+0000 7fd1d5b92640 1 -- 172.21.15.16:0/1439179907 wait complete. 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 Processor -- start 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 -- start start 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d0102d30 0x7fd1d0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d01082a0 0x7fd1d019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01120e0 con 0x7fd1d01082a0 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d0111f60 con 0x7fd1d01036e0 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1d5b92640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d0112260 con 0x7fd1d0102d30 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d01082a0 0x7fd1d019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d0102d30 0x7fd1d0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34036/0 (socket says 172.21.15.16:34036) 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.888+0000 7fd1ceffd640 1 -- 172.21.15.16:0/2417976713 learned_addr learned my addr 172.21.15.16:0/2417976713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 -- 172.21.15.16:0/2417976713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d0102d30 msgr2=0x7fd1d0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d0102d30 0x7fd1d0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 -- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d01082a0 msgr2=0x7fd1d019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d01082a0 0x7fd1d019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 -- 172.21.15.16:0/2417976713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d019d4d0 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ceffd640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400d950 tx=0x7fd1c400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4017070 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c4009c30 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d019d7c0 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40047a0 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.889+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d01a50a0 con 0x7fd1d01036e0 2024-09-12T02:57:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.890+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d006a850 con 0x7fd1d01036e0 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.891+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd1c4004020 con 0x7fd1d01036e0 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.892+0000 7fd1ccff9640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 0x7fd1b80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.892+0000 7fd1cf7fe640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 0x7fd1b80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.892+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd1c40ced50 con 0x7fd1d01036e0 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.893+0000 7fd1cf7fe640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 0x7fd1b80a5040 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0006010 tx=0x7fd1c0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:27.896+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd1c4020020 con 0x7fd1d01036e0 2024-09-12T02:57:27.984 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.136+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1d01a5270 con 0x7fd1d01036e0 2024-09-12T02:57:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.137+0000 7fd1ccff9640 1 -- 172.21.15.16:0/2417976713 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fd1c409bcc0 con 0x7fd1d01036e0 2024-09-12T02:57:28.139 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:57:28.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:57:28.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:57:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 msgr2=0x7fd1b80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 0x7fd1b80a5040 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0006010 tx=0x7fd1c0000bc0 comp rx=0 tx=0).stop 2024-09-12T02:57:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 msgr2=0x7fd1d0198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400d950 tx=0x7fd1c400de10 comp rx=0 tx=0).stop 2024-09-12T02:57:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 shutdown_connections 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1b80a2b80 0x7fd1b80a5040 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d01082a0 0x7fd1d019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d01036e0 0x7fd1d0198a60 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400d950 tx=0x7fd1c400de10 comp rx=0 tx=0).stop 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 --2- 172.21.15.16:0/2417976713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d0102d30 0x7fd1d0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 >> 172.21.15.16:0/2417976713 conn(0x7fd1d00fc7d0 msgr2=0x7fd1d0105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 shutdown_connections 2024-09-12T02:57:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:28.140+0000 7fd1d5b92640 1 -- 172.21.15.16:0/2417976713 wait complete. 2024-09-12T02:57:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:28 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2417976713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:28 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2417976713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:28 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2417976713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:28.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 968046 10.41 MB/sec execute 390 sec latency 19.818 ms 2024-09-12T02:57:28.941 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-09-12T02:57:28.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-09-12T02:57:29.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.156+0000 7f9eb81ce640 1 -- 172.21.15.16:0/1755165978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 msgr2=0x7f9eb010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.156+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/1755165978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb010da40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009ca0 tx=0x7f9ea802a950 comp rx=0 tx=0).stop 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 -- 172.21.15.16:0/1755165978 shutdown_connections 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/1755165978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb010da40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/1755165978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0102570 0x7f9eb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/1755165978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0100e20 0x7f9eb0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 -- 172.21.15.16:0/1755165978 >> 172.21.15.16:0/1755165978 conn(0x7f9eb00fc7d0 msgr2=0x7f9eb00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 -- 172.21.15.16:0/1755165978 shutdown_connections 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.157+0000 7f9eb81ce640 1 -- 172.21.15.16:0/1755165978 wait complete. 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 Processor -- start 2024-09-12T02:57:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 -- start start 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0100e20 0x7f9eb019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb0110fd0 con 0x7f9eb0102ef0 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb0110e50 con 0x7f9eb0102570 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb81ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb0111150 con 0x7f9eb0100e20 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0100e20 0x7f9eb019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34046/0 (socket says 172.21.15.16:34046) 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 -- 172.21.15.16:0/3343299783 learned_addr learned my addr 172.21.15.16:0/3343299783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb676b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 -- 172.21.15.16:0/3343299783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0100e20 msgr2=0x7f9eb019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0100e20 0x7f9eb019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 -- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 msgr2=0x7f9eb019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.158+0000 7f9eb5769640 1 -- 172.21.15.16:0/3343299783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb019f660 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e9f0 tx=0x7f9ea000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9ea6ffd640 1 -- 172.21.15.16:0/3343299783 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000cd80 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb019f950 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9ea6ffd640 1 -- 172.21.15.16:0/3343299783 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea0005500 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9ea6ffd640 1 -- 172.21.15.16:0/3343299783 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0010690 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.159+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb01a7230 con 0x7f9eb0102570 2024-09-12T02:57:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.161+0000 7f9ea6ffd640 1 -- 172.21.15.16:0/3343299783 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9ea00046c0 con 0x7f9eb0102570 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 Processor -- start 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 -- start start 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9ea6ffd640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 0x7f9e9c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 0x55bd909d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd9093a2f0 con 0x55bd909d45d0 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd908e1cf0 con 0x55bd909d5b40 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bd909362b0 con 0x55bd909df370 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34060/0 (socket says 172.21.15.16:34060) 2024-09-12T02:57:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9ea6ffd640 1 -- 172.21.15.16:0/3343299783 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9ea00cf410 con 0x7f9eb0102570 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/3594744186 learned_addr learned my addr 172.21.15.16:0/3594744186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eb676b640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 0x7f9e9c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.162+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 0x55bd909d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb676b640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/3594744186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 msgr2=0x55bd909e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/3594744186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 msgr2=0x55bd909d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 0x55bd909d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/3594744186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd908c63a0 con 0x55bd909d5b40 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 0x55bd909d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb676b640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 0x7f9e9c0a5040 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006010 tx=0x7f9ea8009820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.163+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009cf0 tx=0x7f9eac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47bc86abb56a0041 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9e9bfff640 1 -- 172.21.15.16:0/3594744186 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac004230 con 0x55bd909d5b40 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9e9bfff640 1 -- 172.21.15.16:0/3594744186 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0043d0 con 0x55bd909d5b40 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9e9bfff640 1 -- 172.21.15.16:0/3594744186 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac019960 con 0x55bd909d5b40 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3594744186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 msgr2=0x55bd909d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009cf0 tx=0x7f9eac011810 comp rx=0 tx=0).stop 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3594744186 shutdown_connections 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d45d0 0x55bd909d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3594744186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d5b40 0x55bd909d4090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3594744186 >> 172.21.15.16:0/3594744186 conn(0x55bd9099d310 msgr2=0x55bd909d0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3594744186 shutdown_connections 2024-09-12T02:57:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.164+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3594744186 wait complete. 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 -- start start 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 Processor -- start 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909decd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd909e2070 con 0x55bd909df370 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd908f5060 con 0x55bd909d45d0 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.165+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bd909e21f0 con 0x55bd909d5b40 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eba5a7740 10 client.0 _ll_register_callbacks cb 0x55bd906488c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb676b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34840/0 (socket says 172.21.15.16:34840) 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb676b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41960/0 (socket says 172.21.15.16:41960) 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 -- 172.21.15.16:0/3922185345 learned_addr learned my addr 172.21.15.16:0/3922185345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909decd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 -- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 msgr2=0x55bd909decd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909decd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 -- 172.21.15.16:0/3922185345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 msgr2=0x55bd909dc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 -- 172.21.15.16:0/3922185345 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd906ab3c0 con 0x55bd909d5b40 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb676b640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eb5769640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00ccf70 tx=0x7f9ea009b6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb3c7dd813e4f755 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9e997fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea009bc80 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9ea00a4460 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.166+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac0043d0 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9ea00a0030 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 10 client.27106 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd909edb20 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0043d0 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9ea00a4b40 con 0x55bd909d5b40 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 10 client.27106 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 10 client.27106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 20 client.27106 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd909ef2e0 0x55bd909f1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 4 client.27106 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd90947860 con 0x55bd909ef2e0 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.167+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd909ef2e0 0x55bd909f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.168+0000 7f9e997fa640 10 client.27106 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.168+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd909ef2e0 0x55bd909f1710 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40b0410264381987 server_cookie=993db4c35f58b3be in_seq=0 out_seq=0 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.168+0000 7f9e997fa640 1 -- 172.21.15.16:0/3922185345 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd90947860 con 0x55bd909ef2e0 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.168+0000 7f9e997fa640 10 client.27106 handle_command_reply: tid=0 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 client.27106 shutdown 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 msgr2=0x55bd909dbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00ccf70 tx=0x7f9ea009b6a0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 shutdown_connections 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd909ef2e0 0x55bd909f1710 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909df370 0x55bd909dc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d5b40 0x55bd909dbfd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/3922185345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909decd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 >> 172.21.15.16:0/3922185345 conn(0x55bd9099d310 msgr2=0x55bd909d6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9e997fa640 0 client.27106 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 shutdown_connections 2024-09-12T02:57:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 1 -- 172.21.15.16:0/3922185345 wait complete. 2024-09-12T02:57:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.169+0000 7f9eba5a7740 20 client.27106 trim_cache size 0 max 16384 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 Processor -- start 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 -- start start 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 0x55bd909da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909df370 0x55bd909d7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01af670 con 0x55bd909d5b40 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b11b0 con 0x55bd909df370 2024-09-12T02:57:29.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.183+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01afd40 con 0x55bd909d45d0 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 0x55bd909da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41970/0 (socket says 172.21.15.16:41970) 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5769640 1 -- 172.21.15.16:0/1314277050 learned_addr learned my addr 172.21.15.16:0/1314277050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb676b640 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909df370 0x55bd909d7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/1314277050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909df370 msgr2=0x55bd909d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909df370 0x55bd909d7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/1314277050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 msgr2=0x55bd909ddf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/1314277050 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd909daf80 con 0x55bd909d45d0 2024-09-12T02:57:29.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5769640 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:29.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 0x55bd909da8a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9eac01b040 tx=0x7f9eac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=350e5ea4309642c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9e9b7fe640 1 -- 172.21.15.16:0/1314277050 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac031070 con 0x55bd909d45d0 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9e9b7fe640 1 -- 172.21.15.16:0/1314277050 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac005540 con 0x55bd909d45d0 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.184+0000 7f9e9b7fe640 1 -- 172.21.15.16:0/1314277050 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac019530 con 0x55bd909d45d0 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 -- 172.21.15.16:0/1314277050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 msgr2=0x55bd909da8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 0x55bd909da8a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9eac01b040 tx=0x7f9eac011810 comp rx=0 tx=0).stop 2024-09-12T02:57:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 -- 172.21.15.16:0/1314277050 shutdown_connections 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909df370 0x55bd909d7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909ddf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/1314277050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909d45d0 0x55bd909da8a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 -- 172.21.15.16:0/1314277050 >> 172.21.15.16:0/1314277050 conn(0x55bd9099d310 msgr2=0x55bd909dd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 -- 172.21.15.16:0/1314277050 shutdown_connections 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.185+0000 7f9eba5a7740 1 -- 172.21.15.16:0/1314277050 wait complete. 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 -- start start 2024-09-12T02:57:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 Processor -- start 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909e2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b0ae0 con 0x55bd909d5b40 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01b0410 con 0x55bd909d45d0 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb01ad650 con 0x55bd909df370 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eba5a7740 10 client.0 _ll_register_callbacks cb 0x7f9ea009b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:29.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb676b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb5f6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34082/0 (socket says 172.21.15.16:34082) 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/2270278749 learned_addr learned my addr 172.21.15.16:0/2270278749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb676b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34858/0 (socket says 172.21.15.16:34858) 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.186+0000 7f9eb5769640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909e2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/2270278749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 msgr2=0x55bd909e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/2270278749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 msgr2=0x55bd909e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909e2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 -- 172.21.15.16:0/2270278749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd909e3980 con 0x55bd909d45d0 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5769640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909e2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eb5f6a640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9eac011d10 tx=0x7f9eac011e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd51dc55bd6cd44f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:29.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9e997fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac031040 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac005540 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9eac003d00 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9eac005a70 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eba5a7740 10 client.27206 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.187+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd90a59e80 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac005540 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9eac041bc0 con 0x55bd909d45d0 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 10 client.27206 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 10 client.27206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 20 client.27206 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd90a5a350 0x55bd90a5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 4 client.27206 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:57:29.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd90941d70 con 0x55bd90a5a350 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.188+0000 7f9eb5769640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd90a5a350 0x55bd90a5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9e997fa640 10 client.27206 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eb5769640 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd90a5a350 0x55bd90a5c740 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=849a5ba5d18681e6 server_cookie=a8cf1d6747ecef1e in_seq=0 out_seq=0 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9e997fa640 1 -- 172.21.15.16:0/2270278749 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd90941d70 con 0x55bd90a5a350 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9e997fa640 10 client.27206 handle_command_reply: tid=0 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 client.27206 shutdown 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 msgr2=0x55bd909e7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9eac011d10 tx=0x7f9eac011e50 comp rx=0 tx=0).stop 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 shutdown_connections 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55bd90a5a350 0x55bd90a5c740 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd909df370 0x55bd909e32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd909d5b40 0x55bd909e2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 --2- 172.21.15.16:0/2270278749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd909d45d0 0x55bd909e7a70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 >> 172.21.15.16:0/2270278749 conn(0x55bd9099d310 msgr2=0x55bd909de050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9e997fa640 0 client.27206 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 shutdown_connections 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 1 -- 172.21.15.16:0/2270278749 wait complete. 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.189+0000 7f9eba5a7740 20 client.27206 trim_cache size 0 max 16384 2024-09-12T02:57:29.197 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:57:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 msgr2=0x7f9e9c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 0x7f9e9c0a5040 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006010 tx=0x7f9ea8009820 comp rx=0 tx=0).stop 2024-09-12T02:57:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 msgr2=0x7f9eb019abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e9f0 tx=0x7f9ea000edf0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 shutdown_connections 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9e9c0a2b80 0x7f9e9c0a5040 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9eb0102ef0 0x7f9eb019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9eb0102570 0x7f9eb019abf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 --2- 172.21.15.16:0/3343299783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9eb0100e20 0x7f9eb019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 >> 172.21.15.16:0/3343299783 conn(0x7f9eb00fc7d0 msgr2=0x7f9eb006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 shutdown_connections 2024-09-12T02:57:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:29.190+0000 7f9eb81ce640 1 -- 172.21.15.16:0/3343299783 wait complete. 2024-09-12T02:57:29.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:29 smithi033 ceph-mon[35092]: pgmap v624: 97 pgs: 97 active+clean; 260 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:57:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:29 smithi184 ceph-mon[35321]: pgmap v624: 97 pgs: 97 active+clean; 260 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:57:29.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:29 smithi016 ceph-mon[30925]: pgmap v624: 97 pgs: 97 active+clean; 260 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:57:29.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 970618 10.41 MB/sec execute 391 sec latency 17.699 ms 2024-09-12T02:57:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 -- 172.21.15.16:0/2471787803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0102ef0 msgr2=0x7f51f010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 --2- 172.21.15.16:0/2471787803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0102ef0 0x7f51f010da40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009ca0 tx=0x7f51e802a950 comp rx=0 tx=0).stop 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 -- 172.21.15.16:0/2471787803 shutdown_connections 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 --2- 172.21.15.16:0/2471787803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0102ef0 0x7f51f010da40 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 --2- 172.21.15.16:0/2471787803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 0x7f51f01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 --2- 172.21.15.16:0/2471787803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0100e20 0x7f51f0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.554+0000 7f51f6815640 1 -- 172.21.15.16:0/2471787803 >> 172.21.15.16:0/2471787803 conn(0x7f51f00fc7d0 msgr2=0x7f51f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.555+0000 7f51f6815640 1 -- 172.21.15.16:0/2471787803 shutdown_connections 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.555+0000 7f51f6815640 1 -- 172.21.15.16:0/2471787803 wait complete. 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.555+0000 7f51f6815640 1 Processor -- start 2024-09-12T02:57:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 -- start start 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 0x7f51f00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0102ef0 0x7f51f00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0110f80 con 0x7f51f0100e20 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0110e00 con 0x7f51f0102ef0 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f6815640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0111100 con 0x7f51f0102570 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41992/0 (socket says 172.21.15.16:41992) 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 -- 172.21.15.16:0/4186494653 learned_addr learned my addr 172.21.15.16:0/4186494653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51f4db2640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0102ef0 0x7f51f00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51e7fff640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 0x7f51f00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 -- 172.21.15.16:0/4186494653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 msgr2=0x7f51f00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 0x7f51f00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 -- 172.21.15.16:0/4186494653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0102ef0 msgr2=0x7f51f00fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0102ef0 0x7f51f00fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 -- 172.21.15.16:0/4186494653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f01a69c0 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.556+0000 7f51effff640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e9f0 tx=0x7f51e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.557+0000 7f51edffb640 1 -- 172.21.15.16:0/4186494653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e000cd80 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.557+0000 7f51edffb640 1 -- 172.21.15.16:0/4186494653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e0010910 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.557+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f01a6cb0 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.557+0000 7f51edffb640 1 -- 172.21.15.16:0/4186494653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0018990 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.557+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f01a71a0 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.559+0000 7f51f8bee740 1 Processor -- start 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.559+0000 7f51f8bee740 1 -- start start 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51edffb640 1 -- 172.21.15.16:0/4186494653 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f51e0010ac0 con 0x7f51f0100e20 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05422c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f05387160 con 0x560f05422850 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f0532e720 con 0x560f05423170 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f8bee740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f05383120 con 0x560f0542da30 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f4db2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05422c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f4db2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05422c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41998/0 (socket says 172.21.15.16:41998) 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51f4db2640 1 -- 172.21.15.16:0/872632118 learned_addr learned my addr 172.21.15.16:0/872632118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51e7fff640 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 -- 172.21.15.16:0/872632118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 msgr2=0x560f0542fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 -- 172.21.15.16:0/872632118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 msgr2=0x560f05422c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05422c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 -- 172.21.15.16:0/872632118 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f050f8ba0 con 0x560f05423170 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.560+0000 7f51effff640 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542d4f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f51e00cce00 tx=0x7f51e00d79b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c8a0b8ce3f818f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51edffb640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 0x7f51dc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51f4db2640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 0x7f51dc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51e67fc640 1 -- 172.21.15.16:0/872632118 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00e3070 con 0x560f05423170 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51e67fc640 1 -- 172.21.15.16:0/872632118 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e001c5b0 con 0x560f05423170 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51e67fc640 1 -- 172.21.15.16:0/872632118 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e009b5a0 con 0x560f05423170 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.561+0000 7f51edffb640 1 -- 172.21.15.16:0/4186494653 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51e00cf320 con 0x7f51f0100e20 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- 172.21.15.16:0/872632118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 msgr2=0x560f0542d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542d4f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f51e00cce00 tx=0x7f51e00d79b0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f4db2640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 0x7f51dc0a5040 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009ca0 tx=0x7f51e8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- 172.21.15.16:0/872632118 shutdown_connections 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542d4f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 --2- 172.21.15.16:0/872632118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05422c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- 172.21.15.16:0/872632118 >> 172.21.15.16:0/872632118 conn(0x560f0541ede0 msgr2=0x560f0541f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- 172.21.15.16:0/872632118 shutdown_connections 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- 172.21.15.16:0/872632118 wait complete. 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.562+0000 7f51f8bee740 1 -- start start 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 Processor -- start 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f05430600 con 0x560f05422850 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f0530f900 con 0x560f05423170 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f05430780 con 0x560f0542da30 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f8bee740 10 client.0 _ll_register_callbacks cb 0x560f0530d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42014/0 (socket says 172.21.15.16:42014) 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51effff640 1 -- 172.21.15.16:0/2363627336 learned_addr learned my addr 172.21.15.16:0/2363627336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.563+0000 7f51f4db2640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 -- 172.21.15.16:0/2363627336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 msgr2=0x560f0542b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 -- 172.21.15.16:0/2363627336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 msgr2=0x560f0542a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 -- 172.21.15.16:0/2363627336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0534fc50 con 0x560f05423170 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51effff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51e7fff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542ad90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f51f00fe710 tx=0x7f51d8012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb14f4384d43c03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:57:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d800f920 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51d800fac0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e001c5b0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.564+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f51d800fcc0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 10 client.27218 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f0543c2a0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e001c5b0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f51d80104c0 con 0x560f05423170 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 10 client.27218 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 10 client.27218 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 20 client.27218 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f0543d760 0x560f0543fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 4 client.27218 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f05394890 con 0x560f0543d760 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.565+0000 7f51effff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f0543d760 0x560f0543fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.566+0000 7f51cf7fe640 10 client.27218 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.566+0000 7f51effff640 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f0543d760 0x560f0543fc00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ff4c9fec021ac35 server_cookie=2e1a3621bd477cc5 in_seq=0 out_seq=0 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.566+0000 7f51cf7fe640 1 -- 172.21.15.16:0/2363627336 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f05394890 con 0x560f0543d760 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.566+0000 7f51cf7fe640 10 client.27218 handle_command_reply: tid=0 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 client.27218 shutdown 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 msgr2=0x560f0542ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542ad90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f51f00fe710 tx=0x7f51d8012b60 comp rx=0 tx=0).stop 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 shutdown_connections 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f0543d760 0x560f0543fc00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f0542da30 0x560f0542b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05423170 0x560f0542ad90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 --2- 172.21.15.16:0/2363627336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f0542a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51cf7fe640 0 client.27218 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 >> 172.21.15.16:0/2363627336 conn(0x560f0541ede0 msgr2=0x560f05424b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 shutdown_connections 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 1 -- 172.21.15.16:0/2363627336 wait complete. 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.567+0000 7f51f8bee740 20 client.27218 trim_cache size 0 max 16384 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.586+0000 7f51f8bee740 1 Processor -- start 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.586+0000 7f51f8bee740 1 -- start start 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.586+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05422850 0x560f0542c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.586+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f8bee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01af830 con 0x560f0542da30 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f8bee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01b1370 con 0x560f05422850 2024-09-12T02:57:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f8bee740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01aff00 con 0x560f05423170 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f4db2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34892/0 (socket says 172.21.15.16:34892) 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51effff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05422850 0x560f0542c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f4db2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42024/0 (socket says 172.21.15.16:42024) 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 -- 172.21.15.16:0/978413993 learned_addr learned my addr 172.21.15.16:0/978413993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 -- 172.21.15.16:0/978413993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05422850 msgr2=0x560f0542c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05422850 0x560f0542c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 -- 172.21.15.16:0/978413993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 msgr2=0x560f05445a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 -- 172.21.15.16:0/978413993 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f05422fa0 con 0x560f05423170 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51f4db2640 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e7fff640 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f51d80059f0 tx=0x7f51d800f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3bba6019a34df3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e5ffb640 1 -- 172.21.15.16:0/978413993 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8000e60 con 0x560f05423170 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e5ffb640 1 -- 172.21.15.16:0/978413993 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d80055b0 con 0x560f05423170 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.587+0000 7f51e5ffb640 1 -- 172.21.15.16:0/978413993 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8010db0 con 0x560f05423170 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- 172.21.15.16:0/978413993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 msgr2=0x560f05445510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f51d80059f0 tx=0x7f51d800f850 comp rx=0 tx=0).stop 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- 172.21.15.16:0/978413993 shutdown_connections 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f0542da30 0x560f05445a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f05445510 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 --2- 172.21.15.16:0/978413993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f05422850 0x560f0542c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- 172.21.15.16:0/978413993 >> 172.21.15.16:0/978413993 conn(0x560f0541ede0 msgr2=0x560f0542c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- 172.21.15.16:0/978413993 shutdown_connections 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- 172.21.15.16:0/978413993 wait complete. 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.588+0000 7f51f8bee740 1 -- start start 2024-09-12T02:57:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 Processor -- start 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f054276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 0x560f05427e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01b0ca0 con 0x560f05422850 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01b05d0 con 0x560f0542da30 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01ad810 con 0x560f05423170 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f8bee740 10 client.0 _ll_register_callbacks cb 0x7f51d8009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51e7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f054276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51e7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f054276e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34894/0 (socket says 172.21.15.16:34894) 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42038/0 (socket says 172.21.15.16:42038) 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51e7fff640 1 -- 172.21.15.16:0/706783592 learned_addr learned my addr 172.21.15.16:0/706783592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.589+0000 7f51f4db2640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 0x560f05427e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 -- 172.21.15.16:0/706783592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 msgr2=0x560f054276e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f054276e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 -- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 msgr2=0x560f05427e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 0x560f05427e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 -- 172.21.15.16:0/706783592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f054a74e0 con 0x560f05422850 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51f4db2640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 0x560f05427e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51effff640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e9c0 tx=0x7f51e00d5050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857f36902fb3ca1a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e000d340 con 0x560f05422850 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51e0099f40 con 0x560f05422850 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d80055b0 con 0x560f05422850 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f51e000d540 con 0x560f05422850 2024-09-12T02:57:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51f8bee740 10 client.17748 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f054a8680 con 0x560f05422850 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d80055b0 con 0x560f05422850 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.590+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f51e00ce040 con 0x560f05422850 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 10 client.17748 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 10 client.17748 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 20 client.17748 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f054a89c0 0x560f054aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 4 client.17748 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x560f0538ebe0 con 0x560f054a89c0 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51e7fff640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f054a89c0 0x560f054aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51cf7fe640 10 client.17748 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.591+0000 7f51e7fff640 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f054a89c0 0x560f054aadb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a870d5426dc166b1 server_cookie=361661921666a4fb in_seq=0 out_seq=0 2024-09-12T02:57:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.710+0000 7f51cf7fe640 1 -- 172.21.15.16:0/706783592 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560f0538ebe0 con 0x560f054a89c0 2024-09-12T02:57:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.710+0000 7f51cf7fe640 10 client.17748 handle_command_reply: tid=0 2024-09-12T02:57:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.710+0000 7f51f8bee740 1 client.17748 shutdown 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 msgr2=0x560f05436340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e9c0 tx=0x7f51e00d5050 comp rx=0 tx=0).stop 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 shutdown_connections 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f054a89c0 0x560f054aadb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f0542da30 0x560f05427e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f05423170 0x560f054276e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 --2- 172.21.15.16:0/706783592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f05422850 0x560f05436340 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51cf7fe640 0 client.17748 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 >> 172.21.15.16:0/706783592 conn(0x560f0541ede0 msgr2=0x560f0541dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 shutdown_connections 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 1 -- 172.21.15.16:0/706783592 wait complete. 2024-09-12T02:57:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.711+0000 7f51f8bee740 20 client.17748 trim_cache size 0 max 16384 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 38, 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "2049eb1e": { 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 38, 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 142.6, 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 msgr2=0x7f51dc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 0x7f51dc0a5040 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009ca0 tx=0x7f51e8005c50 comp rx=0 tx=0).stop 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 msgr2=0x7f51f00fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e9f0 tx=0x7f51e000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 shutdown_connections 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51f0102ef0 0x7f51f00fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51f0102570 0x7f51f00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51dc0a2b80 0x7f51dc0a5040 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 --2- 172.21.15.16:0/4186494653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0100e20 0x7f51f00fcc80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 >> 172.21.15.16:0/4186494653 conn(0x7f51f00fc7d0 msgr2=0x7f51f006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 shutdown_connections 2024-09-12T02:57:30.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:30.712+0000 7f51f6815640 1 -- 172.21.15.16:0/4186494653 wait complete. 2024-09-12T02:57:30.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.42 MB/sec execute 392 sec latency 108.188 ms 2024-09-12T02:57:30.968 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:57:30.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:57:31.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:31 smithi016 ceph-mon[30925]: pgmap v625: 97 pgs: 97 active+clean; 259 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 3.7 MiB/s wr, 254 op/s 2024-09-12T02:57:31.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:31 smithi033 ceph-mon[35092]: pgmap v625: 97 pgs: 97 active+clean; 259 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 3.7 MiB/s wr, 254 op/s 2024-09-12T02:57:31.525 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2049eb1e', quiesce took 5.5 seconds. Will release after: 17.05 2024-09-12T02:57:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:31 smithi184 ceph-mon[35321]: pgmap v625: 97 pgs: 97 active+clean; 259 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 3.7 MiB/s wr, 254 op/s 2024-09-12T02:57:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:31.733 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.40 MB/sec execute 393 sec latency 1022.362 ms 2024-09-12T02:57:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 -- 172.21.15.16:0/1505464945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc106a80 msgr2=0x7f0dbc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 --2- 172.21.15.16:0/1505464945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc106a80 0x7f0dbc106e60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009930 tx=0x7f0dac02a420 comp rx=0 tx=0).stop 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 -- 172.21.15.16:0/1505464945 shutdown_connections 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 --2- 172.21.15.16:0/1505464945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc101420 0x7f0dbc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 --2- 172.21.15.16:0/1505464945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 --2- 172.21.15.16:0/1505464945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc106a80 0x7f0dbc106e60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 -- 172.21.15.16:0/1505464945 >> 172.21.15.16:0/1505464945 conn(0x7f0dbc0fc7a0 msgr2=0x7f0dbc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 -- 172.21.15.16:0/1505464945 shutdown_connections 2024-09-12T02:57:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.078+0000 7f0dc2726640 1 -- 172.21.15.16:0/1505464945 wait complete. 2024-09-12T02:57:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.079+0000 7f0dc2726640 1 Processor -- start 2024-09-12T02:57:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.079+0000 7f0dc2726640 1 -- start start 2024-09-12T02:57:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc106a80 0x7f0dbc1a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc110e80 con 0x7f0dbc101420 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc110d00 con 0x7f0dbc100a80 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc2726640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc111000 con 0x7f0dbc106a80 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42048/0 (socket says 172.21.15.16:42048) 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 -- 172.21.15.16:0/663233871 learned_addr learned my addr 172.21.15.16:0/663233871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc106a80 0x7f0dbc1a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 -- 172.21.15.16:0/663233871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc106a80 msgr2=0x7f0dbc1a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc106a80 0x7f0dbc1a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 -- 172.21.15.16:0/663233871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 msgr2=0x7f0dbc1992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc1992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 -- 172.21.15.16:0/663233871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc1a4890 con 0x7f0dbc101420 2024-09-12T02:57:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.080+0000 7f0db3fff640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d950 tx=0x7f0da400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.081+0000 7f0db9ffb640 1 -- 172.21.15.16:0/663233871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4017070 con 0x7f0dbc101420 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.081+0000 7f0db9ffb640 1 -- 172.21.15.16:0/663233871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da4009c30 con 0x7f0dbc101420 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.081+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dbc1a4a60 con 0x7f0dbc101420 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.081+0000 7f0db9ffb640 1 -- 172.21.15.16:0/663233871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da400f6b0 con 0x7f0dbc101420 2024-09-12T02:57:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.081+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dbc1a4ff0 con 0x7f0dbc101420 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.083+0000 7f0db9ffb640 1 -- 172.21.15.16:0/663233871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0da40109d0 con 0x7f0dbc101420 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.083+0000 7f0dc4aff740 1 Processor -- start 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.083+0000 7f0dc4aff740 1 -- start start 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.083+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b248ede40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.083+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248fd150 0x561b248ff540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561b24853280 con 0x561b248ee380 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561b247fab10 con 0x561b248fd150 2024-09-12T02:57:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561b2484f240 con 0x561b248eda60 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42060/0 (socket says 172.21.15.16:42060) 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1743343202 learned_addr learned my addr 172.21.15.16:0/1743343202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b248ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0db3fff640 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248fd150 0x561b248ff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0db9ffb640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 0x7f0da80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1743343202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 msgr2=0x561b248ede40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b248ede40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1743343202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248fd150 msgr2=0x561b248ff540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248fd150 0x561b248ff540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1743343202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b248071f0 con 0x561b248ee380 2024-09-12T02:57:32.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 0x7f0da80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0db9ffb640 1 -- 172.21.15.16:0/663233871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0da40cecd0 con 0x7f0dbc101420 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.084+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009930 tx=0x7f0dac032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7500de75443f295 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0db27fc640 1 -- 172.21.15.16:0/1743343202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac03f070 con 0x561b248ee380 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 0x7f0da80a50c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008d40 tx=0x7f0db4004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0db27fc640 1 -- 172.21.15.16:0/1743343202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dac0317b0 con 0x561b248ee380 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0db27fc640 1 -- 172.21.15.16:0/1743343202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac035070 con 0x561b248ee380 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1743343202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 msgr2=0x561b248fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009930 tx=0x7f0dac032d60 comp rx=0 tx=0).stop 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1743343202 shutdown_connections 2024-09-12T02:57:32.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248fd150 0x561b248ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248fcc10 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1743343202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b248ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1743343202 >> 172.21.15.16:0/1743343202 conn(0x561b248ea0f0 msgr2=0x561b248ea4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1743343202 shutdown_connections 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.085+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1743343202 wait complete. 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 -- start start 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 Processor -- start 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b248fa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ee380 0x561b248fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561b248ffd20 con 0x561b248fd150 2024-09-12T02:57:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561b2480df10 con 0x561b248eda60 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561b248ffea0 con 0x561b248ee380 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc4aff740 10 client.0 _ll_register_callbacks cb 0x561b247d6a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc0cc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc0cc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42072/0 (socket says 172.21.15.16:42072) 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/2712600222 learned_addr learned my addr 172.21.15.16:0/2712600222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.086+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b248fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0db3fff640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ee380 0x561b248fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/2712600222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ee380 msgr2=0x561b248fa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ee380 0x561b248fa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 msgr2=0x561b248fa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b248fa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/2712600222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b245c46a0 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0db40086c0 tx=0x7f0db40083e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b91e32e9f6e1d1d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db401dd80 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0db401b450 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dac0317b0 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0db401b690 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc4aff740 10 client.17760 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.087+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b2490ba00 con 0x561b248fd150 2024-09-12T02:57:32.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dac0317b0 con 0x561b248fd150 2024-09-12T02:57:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0db4024070 con 0x561b248fd150 2024-09-12T02:57:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 10 client.17760 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 20 client.17760 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b2490ce10 0x561b2490f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 4 client.17760 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.088+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b248607f0 con 0x561b2490ce10 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.089+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b2490ce10 0x561b2490f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.089+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b2490ce10 0x561b2490f290 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e8976598b4e601f server_cookie=2b7adfd1720d96f5 in_seq=0 out_seq=0 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.089+0000 7f0d9b7fe640 10 client.17760 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/2712600222 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b248607f0 con 0x561b2490ce10 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0d9b7fe640 10 client.17760 handle_command_reply: tid=0 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 client.17760 shutdown 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 msgr2=0x561b248fab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0db40086c0 tx=0x7f0db40083e0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 shutdown_connections 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b2490ce10 0x561b2490f290 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248fd150 0x561b248fab00 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ee380 0x561b248fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0d9b7fe640 0 client.17760 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/2712600222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b248fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 >> 172.21.15.16:0/2712600222 conn(0x561b248ea0f0 msgr2=0x561b248efe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 shutdown_connections 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 1 -- 172.21.15.16:0/2712600222 wait complete. 2024-09-12T02:57:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.090+0000 7f0dc4aff740 20 client.17760 trim_cache size 0 max 16384 2024-09-12T02:57:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.107+0000 7f0dc4aff740 1 Processor -- start 2024-09-12T02:57:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 -- start start 2024-09-12T02:57:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ea0f0 0x561b24914bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b249150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1ab350 con 0x561b248ee380 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1ace90 con 0x561b248eda60 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1aba20 con 0x561b248ea0f0 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42088/0 (socket says 172.21.15.16:42088) 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ea0f0 0x561b24914bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/947880832 learned_addr learned my addr 172.21.15.16:0/947880832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0db3fff640 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b249150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/947880832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ea0f0 msgr2=0x561b24914bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ea0f0 0x561b24914bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/947880832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 msgr2=0x561b249150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b249150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0dc0cc3640 1 -- 172.21.15.16:0/947880832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b248fc950 con 0x561b248ee380 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.108+0000 7f0db3fff640 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b249150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008690 tx=0x7f0db4005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abb02688c4adad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0db1ffb640 1 -- 172.21.15.16:0/947880832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db401d0b0 con 0x561b248ee380 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0db1ffb640 1 -- 172.21.15.16:0/947880832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db402c030 con 0x561b248ee380 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0db1ffb640 1 -- 172.21.15.16:0/947880832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db401bb90 con 0x561b248ee380 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 -- 172.21.15.16:0/947880832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 msgr2=0x561b248f72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008690 tx=0x7f0db4005d90 comp rx=0 tx=0).stop 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 -- 172.21.15.16:0/947880832 shutdown_connections 2024-09-12T02:57:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ee380 0x561b248f72e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248eda60 0x561b249150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/947880832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248ea0f0 0x561b24914bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.109+0000 7f0dc4aff740 1 -- 172.21.15.16:0/947880832 >> 172.21.15.16:0/947880832 conn(0x561b248f5a70 msgr2=0x561b248e9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- 172.21.15.16:0/947880832 shutdown_connections 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- 172.21.15.16:0/947880832 wait complete. 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- start start 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 Processor -- start 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b24905a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248ee380 0x561b24901a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1ac7c0 con 0x561b248ea0f0 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1ac0f0 con 0x561b248ee380 2024-09-12T02:57:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1a9350 con 0x561b248eda60 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dc4aff740 10 client.0 _ll_register_callbacks cb 0x7f0db4008a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.110+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42102/0 (socket says 172.21.15.16:42102) 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1953786946 learned_addr learned my addr 172.21.15.16:0/1953786946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0db3fff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b24905a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dc0cc3640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248ee380 0x561b24901a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1953786946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 msgr2=0x561b24905a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b24905a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248ee380 msgr2=0x561b24901a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248ee380 0x561b24901a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 -- 172.21.15.16:0/1953786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b24902280 con 0x561b248ea0f0 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0db3fff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b24905a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dbbfff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005420 tx=0x7f0dac005450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf3675a2c42406 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac03f040 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0dac004040 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db402c030 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0dac05b050 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dc4aff740 10 client.17766 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.111+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b24978830 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db402c030 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0dac04c320 con 0x561b248ea0f0 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 10 client.17766 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 20 client.17766 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b248fd150 0x561b2497ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 4 client.17766 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561b2485ad00 con 0x561b248fd150 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.112+0000 7f0db3fff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b248fd150 0x561b2497ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.113+0000 7f0db3fff640 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b248fd150 0x561b2497ab80 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ff13ac54ccfc8a9 server_cookie=79dd33abeb63b8f7 in_seq=0 out_seq=0 2024-09-12T02:57:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.113+0000 7f0d9b7fe640 10 client.17766 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0d9b7fe640 1 -- 172.21.15.16:0/1953786946 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561b2485ad00 con 0x561b248fd150 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0d9b7fe640 10 client.17766 handle_command_reply: tid=0 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 client.17766 shutdown 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 msgr2=0x561b249054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005420 tx=0x7f0dac005450 comp rx=0 tx=0).stop 2024-09-12T02:57:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 shutdown_connections 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x561b248fd150 0x561b2497ab80 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561b248ee380 0x561b24901a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561b248eda60 0x561b24905a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 --2- 172.21.15.16:0/1953786946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b248ea0f0 0x561b249054c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 >> 172.21.15.16:0/1953786946 conn(0x561b248f5a70 msgr2=0x561b248ef5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0d9b7fe640 0 client.17766 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 shutdown_connections 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 1 -- 172.21.15.16:0/1953786946 wait complete. 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.114+0000 7f0dc4aff740 20 client.17766 trim_cache size 0 max 16384 2024-09-12T02:57:32.122 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "0ae6fc6f-8f5a-45d9-8ace-6942bd04f818", 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 msgr2=0x7f0da80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 0x7f0da80a50c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0db4008d40 tx=0x7f0db4004a40 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 msgr2=0x7f0dbc199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d950 tx=0x7f0da400de10 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 shutdown_connections 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dbc106a80 0x7f0dbc1a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0da80a2c00 0x7f0da80a50c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dbc101420 0x7f0dbc199810 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 --2- 172.21.15.16:0/663233871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dbc100a80 0x7f0dbc1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 >> 172.21.15.16:0/663233871 conn(0x7f0dbc0fc7a0 msgr2=0x7f0dbc0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 shutdown_connections 2024-09-12T02:57:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:32.115+0000 7f0dc2726640 1 -- 172.21.15.16:0/663233871 wait complete. 2024-09-12T02:57:32.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.37 MB/sec execute 394 sec latency 2022.725 ms 2024-09-12T02:57:32.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:57:33.035 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:33.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:57:33.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:57:33.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:57:33.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.478 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:33 smithi016 ceph-mon[30925]: pgmap v626: 97 pgs: 97 active+clean; 263 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- 172.21.15.16:0/2507197992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 msgr2=0x7f5cd410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/2507197992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd410eaa0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009ca0 tx=0x7f5ccc02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- 172.21.15.16:0/2507197992 shutdown_connections 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/2507197992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd410eaa0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/2507197992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 0x7f5cd41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/2507197992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 0x7f5cd41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- 172.21.15.16:0/2507197992 >> 172.21.15.16:0/2507197992 conn(0x7f5cd40fd8f0 msgr2=0x7f5cd40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- 172.21.15.16:0/2507197992 shutdown_connections 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- 172.21.15.16:0/2507197992 wait complete. 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 Processor -- start 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.476+0000 7f5cda3f0640 1 -- start start 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 0x7f5cd40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 0x7f5cd4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd4111f80 con 0x7f5cd4106f20 2024-09-12T02:57:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd4111e00 con 0x7f5cd41027a0 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cda3f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd4112100 con 0x7f5cd4101df0 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42128/0 (socket says 172.21.15.16:42128) 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 -- 172.21.15.16:0/564064687 learned_addr learned my addr 172.21.15.16:0/564064687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 0x7f5cd40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 0x7f5cd4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 -- 172.21.15.16:0/564064687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 msgr2=0x7f5cd40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 0x7f5cd40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 -- 172.21.15.16:0/564064687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 msgr2=0x7f5cd4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 0x7f5cd4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.477+0000 7f5cd898d640 1 -- 172.21.15.16:0/564064687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd41aad10 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cd898d640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc031a20 tx=0x7f5ccc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cd17fa640 1 -- 172.21.15.16:0/564064687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc003b70 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cd17fa640 1 -- 172.21.15.16:0/564064687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc003d10 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd41aaf40 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cd17fa640 1 -- 172.21.15.16:0/564064687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc02fc80 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.478+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd41ab3f0 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.480+0000 7f5cd17fa640 1 -- 172.21.15.16:0/564064687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5ccc0046a0 con 0x7f5cd4106f20 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 Processor -- start 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 -- start start 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca022140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca024a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ccc9f7d2e0 con 0x55ccca0174a0 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccc9f24ce0 con 0x55ccca022680 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ccc9f792a0 con 0x55ccca017dc0 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42144/0 (socket says 172.21.15.16:42144) 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2370571158 learned_addr learned my addr 172.21.15.16:0/2370571158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca022140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd17fa640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 0x7f5cbc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2370571158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 msgr2=0x55ccca022140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca022140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2370571158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 msgr2=0x55ccca024a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca024a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2370571158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc9f09440 con 0x55ccca0174a0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.481+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca022140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cd898d640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 0x7f5cbc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cd17fa640 1 -- 172.21.15.16:0/564064687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5ccc0f01c0 con 0x7f5cd4106f20 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800bc30 tx=0x7f5cc80127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98fbd221157ccf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cba7fc640 1 -- 172.21.15.16:0/2370571158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc80120d0 con 0x55ccca0174a0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cba7fc640 1 -- 172.21.15.16:0/2370571158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc8016070 con 0x55ccca0174a0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.482+0000 7f5cba7fc640 1 -- 172.21.15.16:0/2370571158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8003e60 con 0x55ccca0174a0 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.483+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/2370571158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 msgr2=0x55ccca017880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.483+0000 7f5cd898d640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 0x7f5cbc0a5040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0c5110 tx=0x7f5ccc0b1f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.483+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800bc30 tx=0x7f5cc80127c0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/2370571158 shutdown_connections 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca024a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca022140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/2370571158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca017880 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/2370571158 >> 172.21.15.16:0/2370571158 conn(0x55ccca013a30 msgr2=0x55ccca013e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/2370571158 shutdown_connections 2024-09-12T02:57:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/2370571158 wait complete. 2024-09-12T02:57:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.484+0000 7f5cdc7c9740 1 -- start start 2024-09-12T02:57:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 Processor -- start 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca01f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca01f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ccca025250 con 0x55ccca0174a0 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ccc9f38050 con 0x55ccca017dc0 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ccca0253d0 con 0x55ccca022680 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cdc7c9740 10 client.0 _ll_register_callbacks cb 0x55ccc9f78160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42152/0 (socket says 172.21.15.16:42152) 2024-09-12T02:57:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd37fe640 1 -- 172.21.15.16:0/103870595 learned_addr learned my addr 172.21.15.16:0/103870595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca01f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca01f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.485+0000 7f5cd37fe640 1 -- 172.21.15.16:0/103870595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 msgr2=0x55ccca01f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca01f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cd37fe640 1 -- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 msgr2=0x55ccca01f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca01f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cd37fe640 1 -- 172.21.15.16:0/103870595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc9cee3c0 con 0x55ccca0174a0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80057a0 tx=0x7f5cc80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b62a79e664aed0f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8003dc0 con 0x55ccca0174a0 2024-09-12T02:57:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5cc8003f60 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc8016070 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5cc8004160 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cdc7c9740 10 client.17778 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccca030c70 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.486+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc8016070 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5cc802b4f0 con 0x55ccca0174a0 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 10 client.17778 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 10 client.17778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 20 client.17778 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca032020 0x55ccca034410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 4 client.17778 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccc9f8a850 con 0x55ccca032020 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca032020 0x55ccca034410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca032020 0x55ccca034410 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79c074a4ad53ebd4 server_cookie=290c527cabc63ae4 in_seq=0 out_seq=0 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.487+0000 7f5cb37fe640 10 client.17778 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cb37fe640 1 -- 172.21.15.16:0/103870595 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ccc9f8a850 con 0x55ccca032020 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cb37fe640 10 client.17778 handle_command_reply: tid=0 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 client.17778 shutdown 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 msgr2=0x55ccca022040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80057a0 tx=0x7f5cc80056d0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 shutdown_connections 2024-09-12T02:57:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca032020 0x55ccca034410 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca01f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca01f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/103870595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca022040 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 >> 172.21.15.16:0/103870595 conn(0x55ccca013a30 msgr2=0x55ccca019760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cb37fe640 0 client.17778 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 shutdown_connections 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.488+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/103870595 wait complete. 2024-09-12T02:57:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.489+0000 7f5cdc7c9740 20 client.17778 trim_cache size 0 max 16384 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:33 smithi033 ceph-mon[35092]: pgmap v626: 97 pgs: 97 active+clean; 263 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 Processor -- start 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 -- start start 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca021260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca0399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b3ae0 con 0x55ccca0174a0 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b5620 con 0x55ccca017dc0 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b41b0 con 0x55ccca022680 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42162/0 (socket says 172.21.15.16:42162) 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 -- 172.21.15.16:0/1047471566 learned_addr learned my addr 172.21.15.16:0/1047471566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca0399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca021260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 -- 172.21.15.16:0/1047471566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 msgr2=0x55ccca0399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca0399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 -- 172.21.15.16:0/1047471566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 msgr2=0x55ccca021260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca021260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 -- 172.21.15.16:0/1047471566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccca017bf0 con 0x55ccca0174a0 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.506+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003810 tx=0x7f5cc80127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03431a524808aa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1047471566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc80053a0 con 0x55ccca0174a0 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1047471566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc8016040 con 0x55ccca0174a0 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cb9ffb640 1 -- 172.21.15.16:0/1047471566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8003920 con 0x55ccca0174a0 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/1047471566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 msgr2=0x55ccca02e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003810 tx=0x7f5cc80127c0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/1047471566 shutdown_connections 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca022680 0x55ccca0399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca017dc0 0x55ccca021260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/1047471566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02e4b0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/1047471566 >> 172.21.15.16:0/1047471566 conn(0x55ccca013a30 msgr2=0x55ccca020b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/1047471566 shutdown_connections 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.507+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/1047471566 wait complete. 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 -- start start 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 Processor -- start 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca026240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca026980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b4f50 con 0x55ccca0174a0 2024-09-12T02:57:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b4880 con 0x55ccca022680 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41b1ac0 con 0x55ccca017dc0 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cdc7c9740 10 client.0 _ll_register_callbacks cb 0x7f5cc800bb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42178/0 (socket says 172.21.15.16:42178) 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cd37fe640 1 -- 172.21.15.16:0/3353323197 learned_addr learned my addr 172.21.15.16:0/3353323197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.508+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca026240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 -- 172.21.15.16:0/3353323197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 msgr2=0x55ccca026240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca026240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 -- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 msgr2=0x55ccca026980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca026980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 -- 172.21.15.16:0/3353323197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccca09b9b0 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca026240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cd37fe640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c010 tx=0x7f5cc800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679b90808deb392a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc803a070 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5cc80055a0 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc8016040 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5cc8046050 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cdc7c9740 10 client.17784 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccca09cb60 con 0x55ccca0174a0 2024-09-12T02:57:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.509+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc8016040 con 0x55ccca0174a0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5cc8003e20 con 0x55ccca0174a0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 10 client.17784 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 10 client.17784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 20 client.17784 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca09d1c0 0x55ccca09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 4 client.17784 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ccc9f84d60 con 0x55ccca09d1c0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca09d1c0 0x55ccca09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cd3fff640 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca09d1c0 0x55ccca09f5e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=123ef908ffa3ebf8 server_cookie=3998e34a6281916a in_seq=0 out_seq=0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cb37fe640 10 client.17784 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cb37fe640 1 -- 172.21.15.16:0/3353323197 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ccc9f84d60 con 0x55ccca09d1c0 2024-09-12T02:57:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cb37fe640 10 client.17784 handle_command_reply: tid=0 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.510+0000 7f5cdc7c9740 1 client.17784 shutdown 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 msgr2=0x55ccca02af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c010 tx=0x7f5cc800b900 comp rx=0 tx=0).stop 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 shutdown_connections 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ccca09d1c0 0x55ccca09f5e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ccca022680 0x55ccca026980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ccca017dc0 0x55ccca026240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cb37fe640 0 client.17784 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 --2- 172.21.15.16:0/3353323197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ccca0174a0 0x55ccca02af10 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 >> 172.21.15.16:0/3353323197 conn(0x55ccca013a30 msgr2=0x55ccca012350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 shutdown_connections 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 1 -- 172.21.15.16:0/3353323197 wait complete. 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cdc7c9740 20 client.17784 trim_cache size 0 max 16384 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "0ae6fc6f-8f5a-45d9-8ace-6942bd04f818": { 2024-09-12T02:57:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "0ae6fc6f-8f5a-45d9-8ace-6942bd04f818", 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:33.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 msgr2=0x7f5cbc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 0x7f5cbc0a5040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0c5110 tx=0x7f5ccc0b1f40 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 msgr2=0x7f5cd4074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc031a20 tx=0x7f5ccc031a50 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 shutdown_connections 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5cbc0a2b80 0x7f5cbc0a5040 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cd4106f20 0x7f5cd4074f70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.511+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5cd41027a0 0x7f5cd4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.512+0000 7f5cda3f0640 1 --2- 172.21.15.16:0/564064687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cd4101df0 0x7f5cd40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.512+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 >> 172.21.15.16:0/564064687 conn(0x7f5cd40fd8f0 msgr2=0x7f5cd40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.512+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 shutdown_connections 2024-09-12T02:57:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:33.512+0000 7f5cda3f0640 1 -- 172.21.15.16:0/564064687 wait complete. 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:57:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:33 smithi184 ceph-mon[35321]: pgmap v626: 97 pgs: 97 active+clean; 263 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-09-12T02:57:33.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.35 MB/sec execute 395 sec latency 3022.781 ms 2024-09-12T02:57:34.211 INFO:tasks.cephfs.filesystem:scrub status for tag:0ae6fc6f-8f5a-45d9-8ace-6942bd04f818 - {'path': '/', 'tag': '0ae6fc6f-8f5a-45d9-8ace-6942bd04f818', 'options': 'recursive,force'} 2024-09-12T02:57:34.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.32 MB/sec execute 396 sec latency 4022.881 ms 2024-09-12T02:57:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:35 smithi184 ceph-mon[35321]: pgmap v627: 97 pgs: 97 active+clean; 267 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.6 MiB/s wr, 246 op/s 2024-09-12T02:57:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:35 smithi016 ceph-mon[30925]: pgmap v627: 97 pgs: 97 active+clean; 267 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.6 MiB/s wr, 246 op/s 2024-09-12T02:57:35.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.29 MB/sec execute 397 sec latency 5022.972 ms 2024-09-12T02:57:35.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:35 smithi033 ceph-mon[35092]: pgmap v627: 97 pgs: 97 active+clean; 267 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 3.6 MiB/s wr, 246 op/s 2024-09-12T02:57:36.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.27 MB/sec execute 398 sec latency 6023.027 ms 2024-09-12T02:57:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:37 smithi184 ceph-mon[35321]: pgmap v628: 97 pgs: 97 active+clean; 271 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-09-12T02:57:37.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:37 smithi016 ceph-mon[30925]: pgmap v628: 97 pgs: 97 active+clean; 271 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-09-12T02:57:37.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.24 MB/sec execute 399 sec latency 7023.153 ms 2024-09-12T02:57:37.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:37 smithi033 ceph-mon[35092]: pgmap v628: 97 pgs: 97 active+clean; 271 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-09-12T02:57:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:57:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:57:38.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:57:38.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:57:38.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.22 MB/sec execute 400 sec latency 8023.309 ms 2024-09-12T02:57:38.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:57:38.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:57:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:39 smithi184 ceph-mon[35321]: pgmap v629: 97 pgs: 97 active+clean; 268 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 3.6 MiB/s wr, 221 op/s 2024-09-12T02:57:39.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:39 smithi016 ceph-mon[30925]: pgmap v629: 97 pgs: 97 active+clean; 268 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 3.6 MiB/s wr, 221 op/s 2024-09-12T02:57:39.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.19 MB/sec execute 401 sec latency 9023.400 ms 2024-09-12T02:57:39.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:39 smithi033 ceph-mon[35092]: pgmap v629: 97 pgs: 97 active+clean; 268 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 3.6 MiB/s wr, 221 op/s 2024-09-12T02:57:40.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.17 MB/sec execute 402 sec latency 10023.513 ms 2024-09-12T02:57:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:41 smithi184 ceph-mon[35321]: pgmap v630: 97 pgs: 97 active+clean; 266 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-09-12T02:57:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:41 smithi016 ceph-mon[30925]: pgmap v630: 97 pgs: 97 active+clean; 266 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-09-12T02:57:41.734 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.14 MB/sec execute 403 sec latency 11023.627 ms 2024-09-12T02:57:41.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:41 smithi033 ceph-mon[35092]: pgmap v630: 97 pgs: 97 active+clean; 266 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-09-12T02:57:42.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.12 MB/sec execute 404 sec latency 12023.709 ms 2024-09-12T02:57:43.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:43 smithi184 ceph-mon[35321]: pgmap v631: 97 pgs: 97 active+clean; 272 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-09-12T02:57:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:43 smithi016 ceph-mon[30925]: pgmap v631: 97 pgs: 97 active+clean; 272 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-09-12T02:57:43.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.09 MB/sec execute 405 sec latency 13023.805 ms 2024-09-12T02:57:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:43 smithi033 ceph-mon[35092]: pgmap v631: 97 pgs: 97 active+clean; 272 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-09-12T02:57:44.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.07 MB/sec execute 406 sec latency 14023.907 ms 2024-09-12T02:57:45.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:45 smithi184 ceph-mon[35321]: pgmap v632: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 94 op/s 2024-09-12T02:57:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:45 smithi016 ceph-mon[30925]: pgmap v632: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 94 op/s 2024-09-12T02:57:45.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.04 MB/sec execute 407 sec latency 15023.996 ms 2024-09-12T02:57:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:45 smithi033 ceph-mon[35092]: pgmap v632: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 94 op/s 2024-09-12T02:57:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 10.02 MB/sec execute 408 sec latency 16024.096 ms 2024-09-12T02:57:47.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:47 smithi184 ceph-mon[35321]: pgmap v633: 97 pgs: 97 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.6 MiB/s wr, 99 op/s 2024-09-12T02:57:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:47 smithi016 ceph-mon[30925]: pgmap v633: 97 pgs: 97 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.6 MiB/s wr, 99 op/s 2024-09-12T02:57:47.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 9.99 MB/sec execute 409 sec latency 17024.205 ms 2024-09-12T02:57:47.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:47 smithi033 ceph-mon[35092]: pgmap v633: 97 pgs: 97 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.6 MiB/s wr, 99 op/s 2024-09-12T02:57:48.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:57:48.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 9.97 MB/sec execute 410 sec latency 18024.242 ms 2024-09-12T02:57:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 -- 172.21.15.16:0/957331279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 msgr2=0x7fb174101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 --2- 172.21.15.16:0/957331279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb174101200 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb164009900 tx=0x7fb16402a480 comp rx=0 tx=0).stop 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 -- 172.21.15.16:0/957331279 shutdown_connections 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 --2- 172.21.15.16:0/957331279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102ef0 0x7fb17410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 --2- 172.21.15.16:0/957331279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102570 0x7fb1741029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 --2- 172.21.15.16:0/957331279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb174101200 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 -- 172.21.15.16:0/957331279 >> 172.21.15.16:0/957331279 conn(0x7fb1740fc7d0 msgr2=0x7fb1740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 -- 172.21.15.16:0/957331279 shutdown_connections 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.334+0000 7fb17a8af640 1 -- 172.21.15.16:0/957331279 wait complete. 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 Processor -- start 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 -- start start 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102570 0x7fb174198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 0x7fb17419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb174110f50 con 0x7fb174100e20 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb174110dd0 con 0x7fb174102570 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb17a8af640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741110d0 con 0x7fb174102ef0 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48010/0 (socket says 172.21.15.16:48010) 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 -- 172.21.15.16:0/4026777779 learned_addr learned my addr 172.21.15.16:0/4026777779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb16bfff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102570 0x7fb174198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb178e4c640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 0x7fb17419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 -- 172.21.15.16:0/4026777779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 msgr2=0x7fb17419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 0x7fb17419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 -- 172.21.15.16:0/4026777779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102570 msgr2=0x7fb174198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102570 0x7fb174198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb173fff640 1 -- 172.21.15.16:0/4026777779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17419d470 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.335+0000 7fb178e4c640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 0x7fb17419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb173fff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb164006010 tx=0x7fb164003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16403f070 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb164031c30 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164035030 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17419d700 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.336+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1741a5040 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.337+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1741021b0 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.338+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb164042070 con 0x7fb174100e20 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.339+0000 7fb171ffb640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 0x7fb1600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.339+0000 7fb16bfff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 0x7fb1600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.339+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1640ef7f0 con 0x7fb174100e20 2024-09-12T02:57:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.339+0000 7fb16bfff640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 0x7fb1600a5040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb1740ff060 tx=0x7fb15c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.343+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb1640bc760 con 0x7fb174100e20 2024-09-12T02:57:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:49 smithi184 ceph-mon[35321]: pgmap v634: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.2 MiB/s wr, 91 op/s 2024-09-12T02:57:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.554+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb17410a0a0 con 0x7fb174100e20 2024-09-12T02:57:49.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:49 smithi016 ceph-mon[30925]: pgmap v634: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.2 MiB/s wr, 91 op/s 2024-09-12T02:57:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.555+0000 7fb171ffb640 1 -- 172.21.15.16:0/4026777779 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fb164031de0 con 0x7fb174100e20 2024-09-12T02:57:49.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:57:49.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:57:49.556 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:57:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 msgr2=0x7fb1600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 0x7fb1600a5040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb1740ff060 tx=0x7fb15c00a2d0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 msgr2=0x7fb1741984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb164006010 tx=0x7fb164003c10 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 shutdown_connections 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb174102ef0 0x7fb17419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb174102570 0x7fb174198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1600a2b80 0x7fb1600a5040 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 --2- 172.21.15.16:0/4026777779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174100e20 0x7fb1741984c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 >> 172.21.15.16:0/4026777779 conn(0x7fb1740fc7d0 msgr2=0x7fb17406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.558+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 shutdown_connections 2024-09-12T02:57:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:49.559+0000 7fb17a8af640 1 -- 172.21.15.16:0/4026777779 wait complete. 2024-09-12T02:57:49.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 9.94 MB/sec execute 411 sec latency 19024.335 ms 2024-09-12T02:57:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:49 smithi033 ceph-mon[35092]: pgmap v634: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.2 MiB/s wr, 91 op/s 2024-09-12T02:57:50.371 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 2049eb1e --release --await' 2024-09-12T02:57:50.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 2049eb1e --release --await 2024-09-12T02:57:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:50 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4026777779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:50.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:50 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4026777779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:50.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 973301 9.92 MB/sec execute 412 sec latency 20024.422 ms 2024-09-12T02:57:50.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:50 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4026777779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 -- 172.21.15.16:0/3672528379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff41002f0 msgr2=0x7f6ff41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3672528379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff41002f0 0x7f6ff41006d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009930 tx=0x7f6ff002a420 comp rx=0 tx=0).stop 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 -- 172.21.15.16:0/3672528379 shutdown_connections 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3672528379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4106f50 0x7f6ff410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3672528379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3672528379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff41002f0 0x7f6ff41006d0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.064+0000 7f6ffc332640 1 -- 172.21.15.16:0/3672528379 >> 172.21.15.16:0/3672528379 conn(0x7f6ff4077210 msgr2=0x7f6ff40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- 172.21.15.16:0/3672528379 shutdown_connections 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- 172.21.15.16:0/3672528379 wait complete. 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 Processor -- start 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- start start 2024-09-12T02:57:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff41002f0 0x7f6ff41984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4112000 con 0x7f6ff4106f50 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4111e80 con 0x7f6ff41002f0 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.065+0000 7f6ffc332640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4112180 con 0x7f6ff4100ca0 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48030/0 (socket says 172.21.15.16:48030) 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 -- 172.21.15.16:0/3855722805 learned_addr learned my addr 172.21.15.16:0/3855722805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff41002f0 0x7f6ff41984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 -- 172.21.15.16:0/3855722805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 msgr2=0x7f6ff4198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 -- 172.21.15.16:0/3855722805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff41002f0 msgr2=0x7f6ff41984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff41002f0 0x7f6ff41984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 -- 172.21.15.16:0/3855722805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff419d470 con 0x7f6ff4106f50 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffa8cf640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00cce0 tx=0x7f6fec00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6feb7fe640 1 -- 172.21.15.16:0/3855722805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00f450 con 0x7f6ff4106f50 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6feb7fe640 1 -- 172.21.15.16:0/3855722805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fec00fad0 con 0x7f6ff4106f50 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff419d760 con 0x7f6ff4106f50 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6feb7fe640 1 -- 172.21.15.16:0/3855722805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec018c10 con 0x7f6ff4106f50 2024-09-12T02:57:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.066+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff41a5040 con 0x7f6ff4106f50 2024-09-12T02:57:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6feb7fe640 1 -- 172.21.15.16:0/3855722805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6fec00f5f0 con 0x7f6ff4106f50 2024-09-12T02:57:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 Processor -- start 2024-09-12T02:57:51.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 -- start start 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c060 0x560cdc07c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc089630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560cdbfe1620 con 0x560cdc07c980 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560cdbf88e50 con 0x560cdc087240 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cdbfdd5e0 con 0x560cdc07c060 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.069+0000 7f6ffa0ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48038/0 (socket says 172.21.15.16:48038) 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/119145301 learned_addr learned my addr 172.21.15.16:0/119145301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc089630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c060 0x560cdc07c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6feb7fe640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 0x7f6fe00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/119145301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c060 msgr2=0x560cdc07c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c060 0x560cdc07c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/119145301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 msgr2=0x560cdc089630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc089630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/119145301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cdbf79d40 con 0x560cdc07c980 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc089630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa8cf640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 0x7f6fe00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6feb7fe640 1 -- 172.21.15.16:0/3855722805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fec0cf7a0 con 0x7f6ff4106f50 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ab0 tx=0x7f6ff0031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fed7d73c1f15e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6fe8dea640 1 -- 172.21.15.16:0/119145301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff003f070 con 0x560cdc07c980 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.070+0000 7f6fe8dea640 1 -- 172.21.15.16:0/119145301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff0031a20 con 0x560cdc07c980 2024-09-12T02:57:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6fe8dea640 1 -- 172.21.15.16:0/119145301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0035070 con 0x560cdc07c980 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 -- 172.21.15.16:0/119145301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 msgr2=0x560cdc086d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffa8cf640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 0x7f6fe00a50c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ab20 tx=0x7f6fec091490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ab0 tx=0x7f6ff0031630 comp rx=0 tx=0).stop 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 -- 172.21.15.16:0/119145301 shutdown_connections 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc089630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc086d00 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/119145301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c060 0x560cdc07c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 -- 172.21.15.16:0/119145301 >> 172.21.15.16:0/119145301 conn(0x560cdc0785f0 msgr2=0x560cdc0789d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 -- 172.21.15.16:0/119145301 shutdown_connections 2024-09-12T02:57:51.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.071+0000 7f6ffe70b740 1 -- 172.21.15.16:0/119145301 wait complete. 2024-09-12T02:57:51.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 -- start start 2024-09-12T02:57:51.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 Processor -- start 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c980 0x560cdc0845f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc084b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560cdc089e10 con 0x560cdc07c060 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560cdbf69f10 con 0x560cdc07c980 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cdc089f90 con 0x560cdc087240 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffe70b740 10 client.0 _ll_register_callbacks cb 0x560cdbf8ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ff98cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ff98cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48044/0 (socket says 172.21.15.16:48044) 2024-09-12T02:57:51.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ff98cd640 1 -- 172.21.15.16:0/4234096238 learned_addr learned my addr 172.21.15.16:0/4234096238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.072+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c980 0x560cdc0845f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 -- 172.21.15.16:0/4234096238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 msgr2=0x560cdc084b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc084b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 -- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c980 msgr2=0x560cdc0845f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c980 0x560cdc0845f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 -- 172.21.15.16:0/4234096238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cdbcf0580 con 0x560cdc07c060 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4015ce0 tx=0x7f6fe40146c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e8189b85b112b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400f040 con 0x560cdc07c060 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fe4014c70 con 0x560cdc07c060 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff0031a20 con 0x560cdc07c060 2024-09-12T02:57:51.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fe40059c0 con 0x560cdc07c060 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ffe70b740 10 client.17802 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560cdbdba960 con 0x560cdc07c060 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff0031a20 con 0x560cdc07c060 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.073+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe4015040 con 0x560cdc07c060 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 10 client.17802 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 10 client.17802 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 20 client.17802 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc099140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 4 client.17802 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cdbfeeb90 con 0x560cdc096d30 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc099140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9986e1e366971f49 server_cookie=81cd35d007dd4ed0 in_seq=0 out_seq=0 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.074+0000 7f6fddffb640 10 client.17802 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6fddffb640 1 -- 172.21.15.16:0/4234096238 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560cdbfeeb90 con 0x560cdc096d30 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6fddffb640 10 client.17802 handle_command_reply: tid=0 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 client.17802 shutdown 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 msgr2=0x560cdc0840b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4015ce0 tx=0x7f6fe40146c0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 shutdown_connections 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc099140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6fddffb640 0 client.17802 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc084b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c980 0x560cdc0845f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/4234096238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0840b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 >> 172.21.15.16:0/4234096238 conn(0x560cdc0785f0 msgr2=0x560cdc07e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 shutdown_connections 2024-09-12T02:57:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 1 -- 172.21.15.16:0/4234096238 wait complete. 2024-09-12T02:57:51.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.075+0000 7f6ffe70b740 20 client.17802 trim_cache size 0 max 16384 2024-09-12T02:57:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.092+0000 7f6ffe70b740 1 Processor -- start 2024-09-12T02:57:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.092+0000 7f6ffe70b740 1 -- start start 2024-09-12T02:57:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c060 0x560cdc07eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc08fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ad570 con 0x560cdc07c980 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41af0b0 con 0x560cdc07c060 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41adc40 con 0x560cdc087240 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48048/0 (socket says 172.21.15.16:48048) 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/206903193 learned_addr learned my addr 172.21.15.16:0/206903193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc08fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c060 0x560cdc07eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/206903193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 msgr2=0x560cdc08fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc08fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/206903193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c060 msgr2=0x560cdc07eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c060 0x560cdc07eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 -- 172.21.15.16:0/206903193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cdc076bb0 con 0x560cdc07c980 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ab0 tx=0x7f6ff00055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec6ec9b7095c1bc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6fdffff640 1 -- 172.21.15.16:0/206903193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0053070 con 0x560cdc07c980 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6fdffff640 1 -- 172.21.15.16:0/206903193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff003b670 con 0x560cdc07c980 2024-09-12T02:57:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.093+0000 7f6fdffff640 1 -- 172.21.15.16:0/206903193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0035040 con 0x560cdc07c980 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- 172.21.15.16:0/206903193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 msgr2=0x560cdc08f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ab0 tx=0x7f6ff00055f0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- 172.21.15.16:0/206903193 shutdown_connections 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc087240 0x560cdc08fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c980 0x560cdc08f5c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/206903193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc07c060 0x560cdc07eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- 172.21.15.16:0/206903193 >> 172.21.15.16:0/206903193 conn(0x560cdc0785f0 msgr2=0x560cdc073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- 172.21.15.16:0/206903193 shutdown_connections 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- 172.21.15.16:0/206903193 wait complete. 2024-09-12T02:57:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 -- start start 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.094+0000 7f6ffe70b740 1 Processor -- start 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c980 0x560cdc0fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc0fdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ae9e0 con 0x560cdc07c060 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ae310 con 0x560cdc087240 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ab550 con 0x560cdc07c980 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffe70b740 10 client.0 _ll_register_callbacks cb 0x7f6fe4015990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48060/0 (socket says 172.21.15.16:48060) 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 -- 172.21.15.16:0/2644565196 learned_addr learned my addr 172.21.15.16:0/2644565196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c980 0x560cdc0fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc0fdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 -- 172.21.15.16:0/2644565196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c980 msgr2=0x560cdc0fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c980 0x560cdc0fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 -- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 msgr2=0x560cdc0fdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc0fdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 -- 172.21.15.16:0/2644565196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cdc0fe420 con 0x560cdc07c060 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6ff98cd640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009040 tx=0x7f6fe4024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a675349e17d3db7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.095+0000 7f6fddffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4005b60 con 0x560cdc07c060 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fe4005d00 con 0x560cdc07c060 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff003b670 con 0x560cdc07c060 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fe4006150 con 0x560cdc07c060 2024-09-12T02:57:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 10 client.17808 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560cdc106960 con 0x560cdc07c060 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff003b670 con 0x560cdc07c060 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe4022be0 con 0x560cdc07c060 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 10 client.17808 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 10 client.17808 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 20 client.17808 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc1090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 4 client.17808 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2049eb1e", "release": true, "await": true}] 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.096+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2049eb1e", "release": true, "await": true}) -- 0x560cdbfe90a0 con 0x560cdc096d30 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.097+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc1090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.097+0000 7f6fddffb640 10 client.17808 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:51.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.097+0000 7f6ffa0ce640 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc1090f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e255d6f6d9d4010d server_cookie=9c527520e1c82a6e in_seq=0 out_seq=0 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.105+0000 7f6fddffb640 1 -- 172.21.15.16:0/2644565196 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560cdbfe90a0 con 0x560cdc096d30 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.105+0000 7f6fddffb640 10 client.17808 handle_command_reply: tid=0 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.105+0000 7f6ffe70b740 1 client.17808 shutdown 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 msgr2=0x560cdc0f9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009040 tx=0x7f6fe4024040 comp rx=0 tx=0).stop 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 shutdown_connections 2024-09-12T02:57:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560cdc096d30 0x560cdc1090f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cdc087240 0x560cdc0fdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cdc07c980 0x560cdc0fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 --2- 172.21.15.16:0/2644565196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cdc07c060 0x560cdc0f9b80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6fddffb640 0 client.17808 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 >> 172.21.15.16:0/2644565196 conn(0x560cdc0785f0 msgr2=0x560cdc076f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 shutdown_connections 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 1 -- 172.21.15.16:0/2644565196 wait complete. 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffe70b740 20 client.17808 trim_cache size 0 max 16384 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:51.107 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 40, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "2049eb1e": { 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 40, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 142.6, 2024-09-12T02:57:51.108 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:51.109 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:51.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 msgr2=0x7f6fe00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 0x7f6fe00a50c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ab20 tx=0x7f6fec091490 comp rx=0 tx=0).stop 2024-09-12T02:57:51.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 msgr2=0x7f6ff419cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:51.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.106+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00cce0 tx=0x7f6fec00a620 comp rx=0 tx=0).stop 2024-09-12T02:57:51.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 shutdown_connections 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fe00a2c00 0x7f6fe00a50c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff4106f50 0x7f6ff419cd90 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6ff4100ca0 0x7f6ff4198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 --2- 172.21.15.16:0/3855722805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ff41002f0 0x7f6ff41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 >> 172.21.15.16:0/3855722805 conn(0x7f6ff4077210 msgr2=0x7f6ff4105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 shutdown_connections 2024-09-12T02:57:51.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:51.107+0000 7f6ffc332640 1 -- 172.21.15.16:0/3855722805 wait complete. 2024-09-12T02:57:51.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:51 smithi016 ceph-mon[30925]: pgmap v635: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.5 MiB/s wr, 100 op/s 2024-09-12T02:57:51.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:51 smithi184 ceph-mon[35321]: pgmap v635: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.5 MiB/s wr, 100 op/s 2024-09-12T02:57:51.647 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:51 smithi033 ceph-mon[35092]: pgmap v635: 97 pgs: 97 active+clean; 280 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.5 MiB/s wr, 100 op/s 2024-09-12T02:57:51.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 974050 9.91 MB/sec execute 413 sec latency 20396.623 ms 2024-09-12T02:57:51.929 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2049eb1e', total seconds elapsed: 25.9 2024-09-12T02:57:51.929 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-09-12T02:57:51.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:57:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/1917272973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 msgr2=0x7f5dc01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:52.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/1917272973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc01021c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009900 tx=0x7f5db002a410 comp rx=0 tx=0).stop 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/1917272973 shutdown_connections 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/1917272973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 0x7f5dc010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/1917272973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 0x7f5dc01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/1917272973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc01021c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/1917272973 >> 172.21.15.16:0/1917272973 conn(0x7f5dc00fd8c0 msgr2=0x7f5dc00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/1917272973 shutdown_connections 2024-09-12T02:57:52.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.726+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/1917272973 wait complete. 2024-09-12T02:57:52.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 Processor -- start 2024-09-12T02:57:52.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 -- start start 2024-09-12T02:57:52.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 0x7f5dc0198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 0x7f5dc019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0111f70 con 0x7f5dc0101de0 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0111df0 con 0x7f5dc0106f10 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dc4bf7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc01120f0 con 0x7f5dc0102790 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dbe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dbe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48084/0 (socket says 172.21.15.16:48084) 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.727+0000 7f5dbe59c640 1 -- 172.21.15.16:0/451523446 learned_addr learned my addr 172.21.15.16:0/451523446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbed9d640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 0x7f5dc019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbdd9b640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 0x7f5dc0198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 -- 172.21.15.16:0/451523446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 msgr2=0x7f5dc0198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:52.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 0x7f5dc0198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 -- 172.21.15.16:0/451523446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 msgr2=0x7f5dc019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 0x7f5dc019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 -- 172.21.15.16:0/451523446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc019d400 con 0x7f5dc0101de0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dbe59c640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006010 tx=0x7f5db00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db003f030 con 0x7f5dc0101de0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db0003ce0 con 0x7f5dc0101de0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc019d690 con 0x7f5dc0101de0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0035070 con 0x7f5dc0101de0 2024-09-12T02:57:52.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.728+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc01a4fd0 con 0x7f5dc0101de0 2024-09-12T02:57:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.729+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc01032c0 con 0x7f5dc0101de0 2024-09-12T02:57:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.731+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5db0042030 con 0x7f5dc0101de0 2024-09-12T02:57:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.731+0000 7f5daf7fe640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 0x7f5da80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.732+0000 7f5dbdd9b640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 0x7f5da80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.732+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5db00ef7e0 con 0x7f5dc0101de0 2024-09-12T02:57:52.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.732+0000 7f5dbdd9b640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 0x7f5da80a5040 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00fdce0 tx=0x7f5db400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:52.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.735+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5db00bc1c0 con 0x7f5dc0101de0 2024-09-12T02:57:52.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 976829 9.92 MB/sec execute 414 sec latency 40.742 ms 2024-09-12T02:57:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.958+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dc010a0f0 con 0x7f5dc0101de0 2024-09-12T02:57:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.959+0000 7f5daf7fe640 1 -- 172.21.15.16:0/451523446 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f5db00c04c0 con 0x7f5dc0101de0 2024-09-12T02:57:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:57:52.959 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:57:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 msgr2=0x7f5da80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 0x7f5da80a5040 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00fdce0 tx=0x7f5db400b410 comp rx=0 tx=0).stop 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 msgr2=0x7f5dc0198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006010 tx=0x7f5db00042f0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 shutdown_connections 2024-09-12T02:57:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5dc0106f10 0x7f5dc019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc0102790 0x7f5dc0198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5da80a2b80 0x7f5da80a5040 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 --2- 172.21.15.16:0/451523446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dc0101de0 0x7f5dc0198450 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 >> 172.21.15.16:0/451523446 conn(0x7f5dc00fd8c0 msgr2=0x7f5dc00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 shutdown_connections 2024-09-12T02:57:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:52.961+0000 7f5dc4bf7640 1 -- 172.21.15.16:0/451523446 wait complete. 2024-09-12T02:57:53.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:53 smithi184 ceph-mon[35321]: pgmap v636: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.2 MiB/s wr, 85 op/s 2024-09-12T02:57:53.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/451523446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:53 smithi016 ceph-mon[30925]: pgmap v636: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.2 MiB/s wr, 85 op/s 2024-09-12T02:57:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/451523446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:53.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 978888 9.92 MB/sec execute 415 sec latency 12.680 ms 2024-09-12T02:57:53.741 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-09-12T02:57:53.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-09-12T02:57:53.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:53 smithi033 ceph-mon[35092]: pgmap v636: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 3.2 MiB/s wr, 85 op/s 2024-09-12T02:57:53.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/451523446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:57:54.008 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:57:54.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.637+0000 7f22896b9640 1 -- 172.21.15.16:0/1502694084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228410e230 msgr2=0x7f228410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.637+0000 7f22896b9640 1 --2- 172.21.15.16:0/1502694084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228410e230 0x7f228410e610 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2274009930 tx=0x7f227402a420 comp rx=0 tx=0).stop 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 -- 172.21.15.16:0/1502694084 shutdown_connections 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 --2- 172.21.15.16:0/1502694084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2284108b50 0x7f2284115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 --2- 172.21.15.16:0/1502694084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f2284108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 --2- 172.21.15.16:0/1502694084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228410e230 0x7f228410e610 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 -- 172.21.15.16:0/1502694084 >> 172.21.15.16:0/1502694084 conn(0x7f2284077bb0 msgr2=0x7f2284079fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 -- 172.21.15.16:0/1502694084 shutdown_connections 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 -- 172.21.15.16:0/1502694084 wait complete. 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 Processor -- start 2024-09-12T02:57:54.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.638+0000 7f22896b9640 1 -- start start 2024-09-12T02:57:54.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108b50 0x7f22841a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228410e230 0x7f228419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22841185b0 con 0x7f2284108b50 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2284118430 con 0x7f22841081f0 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22896b9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2284118730 con 0x7f228410e230 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57670/0 (socket says 172.21.15.16:57670) 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108b50 0x7f22841a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 -- 172.21.15.16:0/1457113051 learned_addr learned my addr 172.21.15.16:0/1457113051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f22837fe640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228410e230 0x7f228419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 -- 172.21.15.16:0/1457113051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228410e230 msgr2=0x7f228419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228410e230 0x7f228419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 -- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108b50 msgr2=0x7f22841a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108b50 0x7f22841a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 -- 172.21.15.16:0/1457113051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228419efb0 con 0x7f22841081f0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.639+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f227402a940 tx=0x7f2274003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.640+0000 7f226bfff640 1 -- 172.21.15.16:0/1457113051 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227403f070 con 0x7f22841081f0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.640+0000 7f226bfff640 1 -- 172.21.15.16:0/1457113051 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2274031c90 con 0x7f22841081f0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.640+0000 7f226bfff640 1 -- 172.21.15.16:0/1457113051 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274035030 con 0x7f22841081f0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.640+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228419f240 con 0x7f22841081f0 2024-09-12T02:57:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.640+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22841ab590 con 0x7f22841081f0 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f226bfff640 1 -- 172.21.15.16:0/1457113051 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2274042070 con 0x7f22841081f0 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f228ba92740 1 Processor -- start 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f228ba92740 1 -- start start 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f228ba92740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f226bfff640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 0x7f226c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.642+0000 7f228ba92740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f228ba92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8d2a30 0x55905a8d4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f228ba92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55905a82c160 con 0x55905a8d2a30 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f228ba92740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55905a7d3720 con 0x55905a8c8170 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f228ba92740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55905a828120 con 0x55905a8c7850 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f226bfff640 1 -- 172.21.15.16:0/1457113051 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22740ef6e0 con 0x7f22841081f0 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22837fe640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 0x7f226c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:57672/0 (socket says 172.21.15.16:57672) 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 -- 172.21.15.16:0/3494044840 learned_addr learned my addr 172.21.15.16:0/3494044840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22837fe640 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8d2a30 0x55905a8d4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 -- 172.21.15.16:0/3494044840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 msgr2=0x55905a8c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 -- 172.21.15.16:0/3494044840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8d2a30 msgr2=0x55905a8d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8d2a30 0x55905a8d4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 -- 172.21.15.16:0/3494044840 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905a59dba0 con 0x55905a8c8170 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22827fc640 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2278009c60 tx=0x7f2278011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=894959c83827a42d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22837fe640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 0x7f226c0a5040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f227c005fd0 tx=0x7f227c009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.643+0000 7f22697fa640 1 -- 172.21.15.16:0/3494044840 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278004230 con 0x55905a8c8170 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f22697fa640 1 -- 172.21.15.16:0/3494044840 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22780043d0 con 0x55905a8c8170 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f22697fa640 1 -- 172.21.15.16:0/3494044840 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278004d70 con 0x55905a8c8170 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 -- 172.21.15.16:0/3494044840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 msgr2=0x55905a8d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2278009c60 tx=0x7f2278011810 comp rx=0 tx=0).stop 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 -- 172.21.15.16:0/3494044840 shutdown_connections 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8d2a30 0x55905a8d4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c8170 0x55905a8d24f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 --2- 172.21.15.16:0/3494044840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 -- 172.21.15.16:0/3494044840 >> 172.21.15.16:0/3494044840 conn(0x55905a8c3de0 msgr2=0x55905a8c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 -- 172.21.15.16:0/3494044840 shutdown_connections 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.644+0000 7f228ba92740 1 -- 172.21.15.16:0/3494044840 wait complete. 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 -- start start 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 Processor -- start 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c8170 0x55905a8cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55905a8d5600 con 0x55905a8c7850 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55905a7b4900 con 0x55905a8d2a30 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55905a8d5780 con 0x55905a8c8170 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48116/0 (socket says 172.21.15.16:48116) 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 -- 172.21.15.16:0/1675970924 learned_addr learned my addr 172.21.15.16:0/1675970924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f228ba92740 10 client.0 _ll_register_callbacks cb 0x55905a7b2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f22827fc640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c8170 0x55905a8cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f22837fe640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 -- 172.21.15.16:0/1675970924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c8170 msgr2=0x55905a8cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c8170 0x55905a8cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 -- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 msgr2=0x55905a8d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.645+0000 7f2282ffd640 1 -- 172.21.15.16:0/1675970924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905a7f4c50 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2282ffd640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f227402a9e0 tx=0x7f2274009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2a18c6cedb7aaa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2266ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274049040 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f227403b460 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22780043d0 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f22740ba700 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f228ba92740 10 client.17826 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55905a8e12c0 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.646+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22780043d0 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f22740b85c0 con 0x55905a8c7850 2024-09-12T02:57:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 10 client.17826 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 10 client.17826 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 20 client.17826 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a8e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 4 client.17826 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55905a839890 con 0x55905a8e26b0 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f22827fc640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a8e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f22827fc640 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a8e4aa0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=582bafadd7883356 server_cookie=8bcf7ac395cde1f7 in_seq=0 out_seq=0 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.647+0000 7f2266ff5640 10 client.17826 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f2266ff5640 1 -- 172.21.15.16:0/1675970924 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55905a839890 con 0x55905a8e26b0 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f2266ff5640 10 client.17826 handle_command_reply: tid=0 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 client.17826 shutdown 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 msgr2=0x55905a8cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f227402a9e0 tx=0x7f2274009260 comp rx=0 tx=0).stop 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 shutdown_connections 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a8e4aa0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f2266ff5640 0 client.17826 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c8170 0x55905a8cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 --2- 172.21.15.16:0/1675970924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c7850 0x55905a8cf900 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 >> 172.21.15.16:0/1675970924 conn(0x55905a8c3de0 msgr2=0x55905a8c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 shutdown_connections 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 1 -- 172.21.15.16:0/1675970924 wait complete. 2024-09-12T02:57:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.648+0000 7f228ba92740 20 client.17826 trim_cache size 0 max 16384 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.665+0000 7f228ba92740 1 Processor -- start 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 -- start start 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8ea310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b3ac0 con 0x55905a8c8170 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b5600 con 0x55905a8d2a30 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f228ba92740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b4190 con 0x55905a8c7850 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48124/0 (socket says 172.21.15.16:48124) 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 -- 172.21.15.16:0/3761771978 learned_addr learned my addr 172.21.15.16:0/3761771978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22837fe640 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 -- 172.21.15.16:0/3761771978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 msgr2=0x55905a8ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 -- 172.21.15.16:0/3761771978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 msgr2=0x55905a8ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 -- 172.21.15.16:0/3761771978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905a8ce8b0 con 0x55905a8c8170 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.666+0000 7f22827fc640 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f22780099f0 tx=0x7f2278004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472b68acc91dd6f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.667+0000 7f2268ff9640 1 -- 172.21.15.16:0/3761771978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227801e040 con 0x55905a8c8170 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.667+0000 7f2268ff9640 1 -- 172.21.15.16:0/3761771978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f227801f040 con 0x55905a8c8170 2024-09-12T02:57:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.667+0000 7f2268ff9640 1 -- 172.21.15.16:0/3761771978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227802b2f0 con 0x55905a8c8170 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.667+0000 7f228ba92740 1 -- 172.21.15.16:0/3761771978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 msgr2=0x55905a8ea850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.667+0000 7f228ba92740 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f22780099f0 tx=0x7f2278004a10 comp rx=0 tx=0).stop 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- 172.21.15.16:0/3761771978 shutdown_connections 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8d2a30 0x55905a8ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8ea850 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- 172.21.15.16:0/3761771978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8c7850 0x55905a8ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- 172.21.15.16:0/3761771978 >> 172.21.15.16:0/3761771978 conn(0x55905a8c3de0 msgr2=0x55905a8c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- 172.21.15.16:0/3761771978 shutdown_connections 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- 172.21.15.16:0/3761771978 wait complete. 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- start start 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 Processor -- start 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c7850 0x55905a8d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8d2a30 0x55905a9462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b4f30 con 0x55905a8c8170 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b4860 con 0x55905a8c7850 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22841b1aa0 con 0x55905a8d2a30 2024-09-12T02:57:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f228ba92740 10 client.0 _ll_register_callbacks cb 0x7f2274009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.668+0000 7f22827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48132/0 (socket says 172.21.15.16:48132) 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f2282ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c7850 0x55905a8d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 -- 172.21.15.16:0/2516207002 learned_addr learned my addr 172.21.15.16:0/2516207002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22837fe640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8d2a30 0x55905a9462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 -- 172.21.15.16:0/2516207002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8d2a30 msgr2=0x55905a9462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8d2a30 0x55905a9462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 -- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c7850 msgr2=0x55905a8d6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c7850 0x55905a8d6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 -- 172.21.15.16:0/2516207002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905a9468a0 con 0x55905a8c8170 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f22827fc640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2278005d60 tx=0x7f2278005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1729032f2e7c93a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f2266ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:57:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278037070 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2278003bd0 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.669+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f227801f040 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2278043050 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 10 client.17832 fetch_fsmap learned FSMap version 32 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55905a94dc00 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f227801f040 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f227802e020 con 0x55905a8c8170 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 10 client.17832 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 10 client.17832 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 20 client.17832 populate_metadata read hostname 'smithi016' 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a950180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 4 client.17832 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-09-12T02:57:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x55905a833be0 con 0x55905a8e26b0 2024-09-12T02:57:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2282ffd640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a950180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:57:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2266ff5640 10 client.17832 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.670+0000 7f2282ffd640 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a950180 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5da47631ba07271b server_cookie=37420230c0347125 in_seq=0 out_seq=0 2024-09-12T02:57:54.735 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.93 MB/sec execute 416 sec latency 52.514 ms 2024-09-12T02:57:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.760+0000 7f2266ff5640 1 -- 172.21.15.16:0/2516207002 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55905a833be0 con 0x55905a8e26b0 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f2266ff5640 10 client.17832 handle_command_reply: tid=0 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 client.17832 shutdown 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 msgr2=0x55905a8d74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2278005d60 tx=0x7f2278005d90 comp rx=0 tx=0).stop 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 shutdown_connections 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55905a8e26b0 0x55905a950180 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55905a8d2a30 0x55905a9462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55905a8c8170 0x55905a8d74e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 --2- 172.21.15.16:0/2516207002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55905a8c7850 0x55905a8d6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 >> 172.21.15.16:0/2516207002 conn(0x55905a8c3de0 msgr2=0x55905a8c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f2266ff5640 0 client.17832 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 shutdown_connections 2024-09-12T02:57:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 1 -- 172.21.15.16:0/2516207002 wait complete. 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.761+0000 7f228ba92740 20 client.17832 trim_cache size 0 max 16384 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 42, 2024-09-12T02:57:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "38e0f169": { 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 42, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 142.6, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:57:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:57:54.765 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 msgr2=0x7f226c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 0x7f226c0a5040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f227c005fd0 tx=0x7f227c009e30 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 msgr2=0x7f22841a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f227402a940 tx=0x7f2274003b20 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 shutdown_connections 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f228410e230 0x7f228419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f226c0a2b80 0x7f226c0a5040 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108b50 0x7f22841a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 --2- 172.21.15.16:0/1457113051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22841081f0 0x7f22841a46c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 >> 172.21.15.16:0/1457113051 conn(0x7f2284077bb0 msgr2=0x7f2284079340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 shutdown_connections 2024-09-12T02:57:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:57:54.762+0000 7f22896b9640 1 -- 172.21.15.16:0/1457113051 wait complete. 2024-09-12T02:57:55.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:55 smithi016 ceph-mon[30925]: pgmap v637: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.5 MiB/s wr, 91 op/s 2024-09-12T02:57:55.458 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '38e0f169', quiesce took 3.5 seconds. Will release after: 19.05 2024-09-12T02:57:55.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:55 smithi184 ceph-mon[35321]: pgmap v637: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.5 MiB/s wr, 91 op/s 2024-09-12T02:57:55.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.90 MB/sec execute 417 sec latency 1052.617 ms 2024-09-12T02:57:55.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:55 smithi033 ceph-mon[35092]: pgmap v637: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.5 MiB/s wr, 91 op/s 2024-09-12T02:57:56.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.88 MB/sec execute 418 sec latency 2052.685 ms 2024-09-12T02:57:57.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:57 smithi184 ceph-mon[35321]: pgmap v638: 97 pgs: 97 active+clean; 275 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-09-12T02:57:57.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:57 smithi016 ceph-mon[30925]: pgmap v638: 97 pgs: 97 active+clean; 275 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-09-12T02:57:57.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.86 MB/sec execute 419 sec latency 3052.793 ms 2024-09-12T02:57:57.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:57 smithi033 ceph-mon[35092]: pgmap v638: 97 pgs: 97 active+clean; 275 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-09-12T02:57:58.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.83 MB/sec execute 420 sec latency 4052.906 ms 2024-09-12T02:57:59.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:57:59 smithi184 ceph-mon[35321]: pgmap v639: 97 pgs: 97 active+clean; 274 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-09-12T02:57:59.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:57:59 smithi016 ceph-mon[30925]: pgmap v639: 97 pgs: 97 active+clean; 274 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-09-12T02:57:59.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.81 MB/sec execute 421 sec latency 5053.000 ms 2024-09-12T02:57:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:57:59 smithi033 ceph-mon[35092]: pgmap v639: 97 pgs: 97 active+clean; 274 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-09-12T02:58:00.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.79 MB/sec execute 422 sec latency 6053.112 ms 2024-09-12T02:58:01.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:01 smithi184 ceph-mon[35321]: pgmap v640: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 155 op/s 2024-09-12T02:58:01.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:01 smithi016 ceph-mon[30925]: pgmap v640: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 155 op/s 2024-09-12T02:58:01.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.76 MB/sec execute 423 sec latency 7053.170 ms 2024-09-12T02:58:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:01 smithi033 ceph-mon[35092]: pgmap v640: 97 pgs: 97 active+clean; 278 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 155 op/s 2024-09-12T02:58:02.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:02 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:58:02.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.74 MB/sec execute 424 sec latency 8053.282 ms 2024-09-12T02:58:02.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:02 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:58:02.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:02 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:58:03.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:03 smithi016 ceph-mon[30925]: pgmap v641: 97 pgs: 97 active+clean; 277 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-09-12T02:58:03.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.72 MB/sec execute 425 sec latency 9053.323 ms 2024-09-12T02:58:03.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:03 smithi033 ceph-mon[35092]: pgmap v641: 97 pgs: 97 active+clean; 277 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-09-12T02:58:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:03 smithi184 ceph-mon[35321]: pgmap v641: 97 pgs: 97 active+clean; 277 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-09-12T02:58:04.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:58:04.467 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:04.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.69 MB/sec execute 426 sec latency 10053.359 ms 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:58:04.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:58:04.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 -- 172.21.15.16:0/3310761748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc100a80 msgr2=0x7f5fbc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/3310761748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc100a80 0x7f5fbc100ee0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009930 tx=0x7f5fb002a420 comp rx=0 tx=0).stop 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 -- 172.21.15.16:0/3310761748 shutdown_connections 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/3310761748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc101420 0x7f5fbc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/3310761748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc100a80 0x7f5fbc100ee0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/3310761748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc106a80 0x7f5fbc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 -- 172.21.15.16:0/3310761748 >> 172.21.15.16:0/3310761748 conn(0x7f5fbc0fc7a0 msgr2=0x7f5fbc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 -- 172.21.15.16:0/3310761748 shutdown_connections 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 -- 172.21.15.16:0/3310761748 wait complete. 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.906+0000 7f5fc19ca640 1 Processor -- start 2024-09-12T02:58:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 -- start start 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc100a80 0x7f5fbc1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc101420 0x7f5fbc198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc110ea0 con 0x7f5fbc106a80 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc110d20 con 0x7f5fbc101420 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fc19ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc111020 con 0x7f5fbc100a80 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51644/0 (socket says 172.21.15.16:51644) 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 -- 172.21.15.16:0/314198111 learned_addr learned my addr 172.21.15.16:0/314198111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc100a80 0x7f5fbc1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc101420 0x7f5fbc198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 -- 172.21.15.16:0/314198111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc100a80 msgr2=0x7f5fbc1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc100a80 0x7f5fbc1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 -- 172.21.15.16:0/314198111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc101420 msgr2=0x7f5fbc198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc101420 0x7f5fbc198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.907+0000 7f5fbb7fe640 1 -- 172.21.15.16:0/314198111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc19d480 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fbb7fe640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b970 tx=0x7f5fb400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fc09c8640 1 -- 172.21.15.16:0/314198111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4019070 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc19d770 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fc09c8640 1 -- 172.21.15.16:0/314198111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb400cb40 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fc09c8640 1 -- 172.21.15.16:0/314198111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4007610 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.908+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fbc1a5050 con 0x7f5fbc106a80 2024-09-12T02:58:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.910+0000 7f5fc3da3740 1 Processor -- start 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.910+0000 7f5fc09c8640 1 -- 172.21.15.16:0/314198111 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5fb401c070 con 0x7f5fbc106a80 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.910+0000 7f5fc3da3740 1 -- start start 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5397a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b548550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56029b49f2e0 con 0x56029b548a90 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56029b446ce0 con 0x56029b539ce0 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56029b49b2a0 con 0x56029b5393c0 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51656/0 (socket says 172.21.15.16:51656) 2024-09-12T02:58:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2065722139 learned_addr learned my addr 172.21.15.16:0/2065722139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b548550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc09c8640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 0x7f5fa40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2065722139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 msgr2=0x56029b5397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2065722139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 msgr2=0x56029b548550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b548550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2065722139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56029b42b440 con 0x56029b548a90 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b548550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fbb7fe640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 0x7f5fa40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fc09c8640 1 -- 172.21.15.16:0/314198111 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5fb40ce5c0 con 0x7f5fbc106a80 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.911+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002a940 tx=0x7f5fb0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4b1bdc08f7fe37 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fa1ffb640 1 -- 172.21.15.16:0/2065722139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0035070 con 0x56029b548a90 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fa1ffb640 1 -- 172.21.15.16:0/2065722139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb0031730 con 0x56029b548a90 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fa1ffb640 1 -- 172.21.15.16:0/2065722139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00047a0 con 0x56029b548a90 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2065722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 msgr2=0x56029b54ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fbb7fe640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 0x7f5fa40a5040 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40cc440 tx=0x7f5fb400f300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002a940 tx=0x7f5fb0005c00 comp rx=0 tx=0).stop 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2065722139 shutdown_connections 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b54ae80 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b548550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2065722139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2065722139 >> 172.21.15.16:0/2065722139 conn(0x56029b535a30 msgr2=0x56029b535e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2065722139 shutdown_connections 2024-09-12T02:58:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.912+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2065722139 wait complete. 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 -- start start 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 Processor -- start 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5484b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b545760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56029b54b5a0 con 0x56029b548a90 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56029b45a050 con 0x56029b539ce0 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56029b54b720 con 0x56029b5393c0 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fc3da3740 10 client.0 _ll_register_callbacks cb 0x56029b49a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51672/0 (socket says 172.21.15.16:51672) 2024-09-12T02:58:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.913+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2395026156 learned_addr learned my addr 172.21.15.16:0/2395026156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b545760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2395026156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 msgr2=0x56029b5484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5484b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 msgr2=0x56029b545760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b545760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 -- 172.21.15.16:0/2395026156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56029b2103c0 con 0x56029b548a90 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0005490 tx=0x7f5fb0005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efb13980f9ebc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5f9affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb004d030 con 0x56029b548a90 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fb0031730 con 0x56029b548a90 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5fb004e070 con 0x56029b548a90 2024-09-12T02:58:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5fb0035070 con 0x56029b548a90 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fc3da3740 10 client.17844 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.914+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56029b557070 con 0x56029b548a90 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb0031730 con 0x56029b548a90 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5fb003d900 con 0x56029b548a90 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 10 client.17844 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 10 client.17844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 20 client.17844 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5583c0 0x56029b55a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 4 client.17844 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56029b4ac850 con 0x56029b5583c0 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5583c0 0x56029b55a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5f9affd640 10 client.17844 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:04.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.915+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5583c0 0x56029b55a7b0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e176db57fe9d28d8 server_cookie=90196307a505d605 in_seq=0 out_seq=0 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5f9affd640 1 -- 172.21.15.16:0/2395026156 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56029b4ac850 con 0x56029b5583c0 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5f9affd640 10 client.17844 handle_command_reply: tid=0 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 client.17844 shutdown 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 msgr2=0x56029b545d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0005490 tx=0x7f5fb0005390 comp rx=0 tx=0).stop 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 shutdown_connections 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5583c0 0x56029b55a7b0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b548a90 0x56029b545d70 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b539ce0 0x56029b545760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2395026156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5f9affd640 0 client.17844 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.916+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 >> 172.21.15.16:0/2395026156 conn(0x56029b535a30 msgr2=0x56029b534b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.917+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 shutdown_connections 2024-09-12T02:58:04.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.917+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2395026156 wait complete. 2024-09-12T02:58:04.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.917+0000 7f5fc3da3740 20 client.17844 trim_cache size 0 max 16384 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 Processor -- start 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 -- start start 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5549c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b548a90 0x56029b55d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1ab3d0 con 0x56029b539ce0 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1acf10 con 0x56029b548a90 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1abaa0 con 0x56029b5393c0 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51684/0 (socket says 172.21.15.16:51684) 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 -- 172.21.15.16:0/435183375 learned_addr learned my addr 172.21.15.16:0/435183375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 -- 172.21.15.16:0/435183375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 msgr2=0x56029b5549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 -- 172.21.15.16:0/435183375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b548a90 msgr2=0x56029b55d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b548a90 0x56029b55d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fbaffd640 1 -- 172.21.15.16:0/435183375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56029b5366c0 con 0x56029b539ce0 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.934+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5fac005fd0 tx=0x7f5fac00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3812f8afeb54f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fa17fa640 1 -- 172.21.15.16:0/435183375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac004220 con 0x56029b539ce0 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fa17fa640 1 -- 172.21.15.16:0/435183375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fac0043c0 con 0x56029b539ce0 2024-09-12T02:58:04.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fa17fa640 1 -- 172.21.15.16:0/435183375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac01db10 con 0x56029b539ce0 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fc3da3740 1 -- 172.21.15.16:0/435183375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 msgr2=0x56029b540d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.935+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5fac005fd0 tx=0x7f5fac00b410 comp rx=0 tx=0).stop 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- 172.21.15.16:0/435183375 shutdown_connections 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b548a90 0x56029b55d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b540d10 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/435183375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b5393c0 0x56029b5549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- 172.21.15.16:0/435183375 >> 172.21.15.16:0/435183375 conn(0x56029b535a30 msgr2=0x56029b547860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- 172.21.15.16:0/435183375 shutdown_connections 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- 172.21.15.16:0/435183375 wait complete. 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- start start 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 Processor -- start 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b5393c0 0x56029b55b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 0x56029b55bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1ac840 con 0x56029b539ce0 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1ac170 con 0x56029b5393c0 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.936+0000 7f5fc3da3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1a93b0 con 0x56029b548a90 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fc3da3740 10 client.0 _ll_register_callbacks cb 0x7f5fac00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51700/0 (socket says 172.21.15.16:51700) 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 -- 172.21.15.16:0/2983614212 learned_addr learned my addr 172.21.15.16:0/2983614212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b5393c0 0x56029b55b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 0x56029b55bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 -- 172.21.15.16:0/2983614212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 msgr2=0x56029b55bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 0x56029b55bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 -- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b5393c0 msgr2=0x56029b55b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b5393c0 0x56029b55b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 -- 172.21.15.16:0/2983614212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56029b5466b0 con 0x56029b539ce0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 0x56029b55bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fbaffd640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5fac011fd0 tx=0x7f5fac01d580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946de91f261575f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5f9affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac029070 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5fac005590 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.937+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fac0043c0 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5fac01d890 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 10 client.17850 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56029b5460e0 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fac0043c0 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5fac036030 con 0x56029b539ce0 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 10 client.17850 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 20 client.17850 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5c3220 0x56029b5c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 4 client.17850 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56029b4a6d60 con 0x56029b5c3220 2024-09-12T02:58:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5c3220 0x56029b5c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5fba7fc640 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5c3220 0x56029b5c5640 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82e704e60e8627d server_cookie=80919d7eadb1917d in_seq=0 out_seq=0 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.938+0000 7f5f9affd640 10 client.17850 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5f9affd640 1 -- 172.21.15.16:0/2983614212 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56029b4a6d60 con 0x56029b5c3220 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5f9affd640 10 client.17850 handle_command_reply: tid=0 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 client.17850 shutdown 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 msgr2=0x56029b55b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5fac011fd0 tx=0x7f5fac01d580 comp rx=0 tx=0).stop 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 shutdown_connections 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56029b5c3220 0x56029b5c5640 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56029b548a90 0x56029b55bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56029b539ce0 0x56029b55b9c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5f9affd640 0 client.17850 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 --2- 172.21.15.16:0/2983614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56029b5393c0 0x56029b55b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 >> 172.21.15.16:0/2983614212 conn(0x56029b535a30 msgr2=0x56029b544740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 shutdown_connections 2024-09-12T02:58:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 1 -- 172.21.15.16:0/2983614212 wait complete. 2024-09-12T02:58:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.939+0000 7f5fc3da3740 20 client.17850 trim_cache size 0 max 16384 2024-09-12T02:58:04.945 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 msgr2=0x7f5fa40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 0x7f5fa40a5040 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40cc440 tx=0x7f5fb400f300 comp rx=0 tx=0).stop 2024-09-12T02:58:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 msgr2=0x7f5fbc19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b970 tx=0x7f5fb400be30 comp rx=0 tx=0).stop 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 shutdown_connections 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5fa40a2b80 0x7f5fa40a5040 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fbc106a80 0x7f5fbc19cda0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fbc101420 0x7f5fbc198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 --2- 172.21.15.16:0/314198111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fbc100a80 0x7f5fbc1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 >> 172.21.15.16:0/314198111 conn(0x7f5fbc0fc7a0 msgr2=0x7f5fbc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 shutdown_connections 2024-09-12T02:58:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:04.940+0000 7f5fc19ca640 1 -- 172.21.15.16:0/314198111 wait complete. 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:58:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:58:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:58:05.590 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:58:05.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:58:05.736 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.67 MB/sec execute 427 sec latency 11053.406 ms 2024-09-12T02:58:05.848 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:05 smithi016 ceph-mon[30925]: pgmap v642: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 145 op/s 2024-09-12T02:58:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:05 smithi033 ceph-mon[35092]: pgmap v642: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 145 op/s 2024-09-12T02:58:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:05 smithi184 ceph-mon[35321]: pgmap v642: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 145 op/s 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 -- 172.21.15.16:0/2823616598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528102da0 msgr2=0x7f5528103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 --2- 172.21.15.16:0/2823616598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528102da0 0x7f5528103180 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5518009900 tx=0x7f551802a480 comp rx=0 tx=0).stop 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 -- 172.21.15.16:0/2823616598 shutdown_connections 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 --2- 172.21.15.16:0/2823616598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f552810c180 0x7f552810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 --2- 172.21.15.16:0/2823616598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55281036c0 0x7f552810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 --2- 172.21.15.16:0/2823616598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528102da0 0x7f5528103180 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 -- 172.21.15.16:0/2823616598 >> 172.21.15.16:0/2823616598 conn(0x7f55280fc7a0 msgr2=0x7f55280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 -- 172.21.15.16:0/2823616598 shutdown_connections 2024-09-12T02:58:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.294+0000 7f552c98c640 1 -- 172.21.15.16:0/2823616598 wait complete. 2024-09-12T02:58:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.295+0000 7f552c98c640 1 Processor -- start 2024-09-12T02:58:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.295+0000 7f552c98c640 1 -- start start 2024-09-12T02:58:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.295+0000 7f552c98c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5528102da0 0x7f5528100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.295+0000 7f552c98c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552c98c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552810c180 0x7f5528101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552c98c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5528111f60 con 0x7f55281036c0 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552c98c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5528111de0 con 0x7f552810c180 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552c98c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55281120e0 con 0x7f5528102da0 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39074/0 (socket says 172.21.15.16:39074) 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 -- 172.21.15.16:0/4242085664 learned_addr learned my addr 172.21.15.16:0/4242085664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552659c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5528102da0 0x7f5528100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f5526d9d640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552810c180 0x7f5528101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 -- 172.21.15.16:0/4242085664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5528102da0 msgr2=0x7f5528100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5528102da0 0x7f5528100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 -- 172.21.15.16:0/4242085664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552810c180 msgr2=0x7f5528101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552810c180 0x7f5528101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 -- 172.21.15.16:0/4242085664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55281a6860 con 0x7f55281036c0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551ffff640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f551000c920 tx=0x7f551000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510007c70 con 0x7f55281036c0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55281a6b50 con 0x7f55281036c0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5510007e10 con 0x7f55281036c0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.296+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510018890 con 0x7f55281036c0 2024-09-12T02:58:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.297+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281a7090 con 0x7f55281036c0 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.298+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5528103180 con 0x7f55281036c0 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.299+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5510018a70 con 0x7f55281036c0 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.299+0000 7f551f7fe640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 0x7f55140a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.300+0000 7f552659c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 0x7f55140a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.300+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f55100cf180 con 0x7f55281036c0 2024-09-12T02:58:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.300+0000 7f552659c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 0x7f55140a5040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5518009230 tx=0x7f55180058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:06.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.304+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5510014030 con 0x7f55281036c0 2024-09-12T02:58:06.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.520+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5528106c90 con 0x7f55281036c0 2024-09-12T02:58:06.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.521+0000 7f551f7fe640 1 -- 172.21.15.16:0/4242085664 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f551009c0f0 con 0x7f55281036c0 2024-09-12T02:58:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:58:06.522 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:58:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 msgr2=0x7f55140a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 0x7f55140a5040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5518009230 tx=0x7f55180058d0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 msgr2=0x7f5528100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f551000c920 tx=0x7f551000cde0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 shutdown_connections 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552810c180 0x7f5528101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55140a2b80 0x7f55140a5040 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.523+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281036c0 0x7f5528100f40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.524+0000 7f552c98c640 1 --2- 172.21.15.16:0/4242085664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5528102da0 0x7f5528100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.524+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 >> 172.21.15.16:0/4242085664 conn(0x7f55280fc7a0 msgr2=0x7f55280fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.524+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 shutdown_connections 2024-09-12T02:58:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:06.524+0000 7f552c98c640 1 -- 172.21.15.16:0/4242085664 wait complete. 2024-09-12T02:58:06.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.65 MB/sec execute 428 sec latency 12053.474 ms 2024-09-12T02:58:06.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:06 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4242085664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:06.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:06 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4242085664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:07.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:06 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4242085664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:07.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:58:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:07.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.63 MB/sec execute 429 sec latency 13053.546 ms 2024-09-12T02:58:07.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:07 smithi016 ceph-mon[30925]: pgmap v643: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-12T02:58:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 -- 172.21.15.16:0/751300930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 msgr2=0x7fdb3010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 --2- 172.21.15.16:0/751300930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3010d950 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009cd0 tx=0x7fdb2802a8e0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 -- 172.21.15.16:0/751300930 shutdown_connections 2024-09-12T02:58:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 --2- 172.21.15.16:0/751300930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3010d950 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 --2- 172.21.15.16:0/751300930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb30100a80 0x7fdb30100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 --2- 172.21.15.16:0/751300930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 0x7fdb30106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 -- 172.21.15.16:0/751300930 >> 172.21.15.16:0/751300930 conn(0x7fdb300fc7a0 msgr2=0x7fdb300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 -- 172.21.15.16:0/751300930 shutdown_connections 2024-09-12T02:58:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 -- 172.21.15.16:0/751300930 wait complete. 2024-09-12T02:58:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.847+0000 7fdb36d29640 1 Processor -- start 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 -- start start 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb30100a80 0x7fdb3019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 0x7fdb3019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb30110e80 con 0x7fdb30101420 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb30110d00 con 0x7fdb30106a80 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb36d29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb30111000 con 0x7fdb30100a80 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39090/0 (socket says 172.21.15.16:39090) 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 -- 172.21.15.16:0/1073852417 learned_addr learned my addr 172.21.15.16:0/1073852417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb352c6640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 0x7fdb3019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 -- 172.21.15.16:0/1073852417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb30100a80 msgr2=0x7fdb3019a6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-12T02:58:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb30100a80 0x7fdb3019a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 -- 172.21.15.16:0/1073852417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 msgr2=0x7fdb3019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 0x7fdb3019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 -- 172.21.15.16:0/1073852417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3019f650 con 0x7fdb30101420 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.848+0000 7fdb2ffff640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400c970 tx=0x7fdb2400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.849+0000 7fdb2dffb640 1 -- 172.21.15.16:0/1073852417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24007bb0 con 0x7fdb30101420 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.849+0000 7fdb2dffb640 1 -- 172.21.15.16:0/1073852417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb24007d50 con 0x7fdb30101420 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.849+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3019f940 con 0x7fdb30101420 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.849+0000 7fdb2dffb640 1 -- 172.21.15.16:0/1073852417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400f450 con 0x7fdb30101420 2024-09-12T02:58:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.849+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb301a7220 con 0x7fdb30101420 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.851+0000 7fdb2dffb640 1 -- 172.21.15.16:0/1073852417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdb24005020 con 0x7fdb30101420 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.851+0000 7fdb39102740 1 Processor -- start 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 -- start start 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584292f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb2dffb640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 0x7fdb180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x55584293c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5558428952f0 con 0x555842930b40 2024-09-12T02:58:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55584283ccf0 con 0x55584292f5d0 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb39102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5558428912b0 con 0x55584293a370 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb2ffff640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 0x7fdb180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb352c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584292f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47116/0 (socket says 172.21.15.16:47116) 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 -- 172.21.15.16:0/2447281592 learned_addr learned my addr 172.21.15.16:0/2447281592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb2dffb640 1 -- 172.21.15.16:0/1073852417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdb240cef20 con 0x7fdb30101420 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x55584293c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 -- 172.21.15.16:0/2447281592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 msgr2=0x55584293c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x55584293c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 -- 172.21.15.16:0/2447281592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 msgr2=0x55584292f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584292f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb34ac5640 1 -- 172.21.15.16:0/2447281592 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558428213a0 con 0x55584292f5d0 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.852+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584292f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000a8f0 tx=0x7fdb2000adb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebbf0466e4aec6d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb2ffff640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 0x7fdb180a5040 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400c940 tx=0x7fdb24090c10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb16ffd640 1 -- 172.21.15.16:0/2447281592 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000ccc0 con 0x55584292f5d0 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb16ffd640 1 -- 172.21.15.16:0/2447281592 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb200035f0 con 0x55584292f5d0 2024-09-12T02:58:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb16ffd640 1 -- 172.21.15.16:0/2447281592 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000f650 con 0x55584292f5d0 2024-09-12T02:58:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb39102740 1 -- 172.21.15.16:0/2447281592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 msgr2=0x55584292fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.853+0000 7fdb39102740 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000a8f0 tx=0x7fdb2000adb0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- 172.21.15.16:0/2447281592 shutdown_connections 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x55584293c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x55584292fa10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- 172.21.15.16:0/2447281592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584292f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- 172.21.15.16:0/2447281592 >> 172.21.15.16:0/2447281592 conn(0x5558428f8310 msgr2=0x55584292bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- 172.21.15.16:0/2447281592 shutdown_connections 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- 172.21.15.16:0/2447281592 wait complete. 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- start start 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 Processor -- start 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x555842939cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x5558429374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55584293d070 con 0x555842930b40 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555842850060 con 0x55584292f5d0 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.854+0000 7fdb39102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55584293d1f0 con 0x55584293a370 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb39102740 10 client.0 _ll_register_callbacks cb 0x5558425a38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39112/0 (socket says 172.21.15.16:39112) 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 -- 172.21.15.16:0/1054530474 learned_addr learned my addr 172.21.15.16:0/1054530474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb352c6640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x555842939cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb352c6640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x5558429374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 -- 172.21.15.16:0/1054530474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 msgr2=0x5558429374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x5558429374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 -- 172.21.15.16:0/1054530474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 msgr2=0x555842939cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x555842939cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 -- 172.21.15.16:0/1054530474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558426063c0 con 0x555842930b40 2024-09-12T02:58:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000a4b0 tx=0x7fdb2000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7d34b4622b0132 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.855+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000a510 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdb200037a0 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb200035f0 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb200039e0 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 10 client.17868 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555842948a80 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb200035f0 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdb20004860 con 0x555842930b40 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 10 client.17868 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 10 client.17868 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 20 client.17868 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55584294a010 0x55584294c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 4 client.17868 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558428a2860 con 0x55584294a010 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.856+0000 7fdb352c6640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55584294a010 0x55584294c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.857+0000 7fdb352c6640 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55584294a010 0x55584294c400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ca77f1c9e769a290 server_cookie=4eea916b5839631f in_seq=0 out_seq=0 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.857+0000 7fdb0ffff640 10 client.17868 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.857+0000 7fdb0ffff640 1 -- 172.21.15.16:0/1054530474 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558428a2860 con 0x55584294a010 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.857+0000 7fdb0ffff640 10 client.17868 handle_command_reply: tid=0 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.857+0000 7fdb39102740 1 client.17868 shutdown 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 msgr2=0x555842936f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000a4b0 tx=0x7fdb2000ccd0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 shutdown_connections 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584293a370 0x5558429374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x555842936f80 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55584294a010 0x55584294c400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 --2- 172.21.15.16:0/1054530474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584292f5d0 0x555842939cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 >> 172.21.15.16:0/1054530474 conn(0x5558428f8310 msgr2=0x555842931960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb0ffff640 0 client.17868 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 shutdown_connections 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 1 -- 172.21.15.16:0/1054530474 wait complete. 2024-09-12T02:58:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.858+0000 7fdb39102740 20 client.17868 trim_cache size 0 max 16384 2024-09-12T02:58:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 Processor -- start 2024-09-12T02:58:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 -- start start 2024-09-12T02:58:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584292f5d0 0x55584292b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584293bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301af760 con 0x555842930b40 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301b12a0 con 0x55584293a370 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb39102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301afe30 con 0x55584292f5d0 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb34ac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39118/0 (socket says 172.21.15.16:39118) 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb34ac5640 1 -- 172.21.15.16:0/3899069640 learned_addr learned my addr 172.21.15.16:0/3899069640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.871+0000 7fdb352c6640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584292f5d0 0x55584292b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb352c6640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584293bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 -- 172.21.15.16:0/3899069640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584292f5d0 msgr2=0x55584292b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584292f5d0 0x55584292b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 -- 172.21.15.16:0/3899069640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 msgr2=0x55584293bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584293bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 -- 172.21.15.16:0/3899069640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55584292bed0 con 0x555842930b40 2024-09-12T02:58:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb352c6640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584293bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000d010 tx=0x7fdb2000a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53372f7c4b41cb72 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb167fc640 1 -- 172.21.15.16:0/3899069640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20032020 con 0x555842930b40 2024-09-12T02:58:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb167fc640 1 -- 172.21.15.16:0/3899069640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb2001c070 con 0x555842930b40 2024-09-12T02:58:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.872+0000 7fdb167fc640 1 -- 172.21.15.16:0/3899069640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20003de0 con 0x555842930b40 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- 172.21.15.16:0/3899069640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 msgr2=0x55584293b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000d010 tx=0x7fdb2000a4e0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- 172.21.15.16:0/3899069640 shutdown_connections 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584293bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555842930b40 0x55584293b760 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 --2- 172.21.15.16:0/3899069640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55584292f5d0 0x55584292b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- 172.21.15.16:0/3899069640 >> 172.21.15.16:0/3899069640 conn(0x5558428f8310 msgr2=0x555842936f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- 172.21.15.16:0/3899069640 shutdown_connections 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- 172.21.15.16:0/3899069640 wait complete. 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 -- start start 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.873+0000 7fdb39102740 1 Processor -- start 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555842930b40 0x55584294a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584294a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301b0bd0 con 0x55584292f5d0 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301b0500 con 0x55584293a370 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301ad740 con 0x555842930b40 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb39102740 10 client.0 _ll_register_callbacks cb 0x7fdb28009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39132/0 (socket says 172.21.15.16:39132) 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 -- 172.21.15.16:0/2848113609 learned_addr learned my addr 172.21.15.16:0/2848113609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584294a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555842930b40 0x55584294a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 -- 172.21.15.16:0/2848113609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555842930b40 msgr2=0x55584294a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555842930b40 0x55584294a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 -- 172.21.15.16:0/2848113609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 msgr2=0x55584294a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584294a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 -- 172.21.15.16:0/2848113609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558429b2b80 con 0x55584292f5d0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb352c6640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fdb2802ae00 tx=0x7fdb200043a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc984f7cf4dc722 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.874+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28003a30 con 0x55584292f5d0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdb2804ce80 con 0x55584292f5d0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2001c070 con 0x55584292f5d0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb28003bd0 con 0x55584292f5d0 2024-09-12T02:58:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 10 client.17874 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558429b3e50 con 0x55584292f5d0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb2001c070 con 0x55584292f5d0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdb28030070 con 0x55584292f5d0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 10 client.17874 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 10 client.17874 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 20 client.17874 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5558429b45a0 0x5558429b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 4 client.17874 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55584289cd70 con 0x5558429b45a0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.875+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5558429b45a0 0x5558429b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb34ac5640 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5558429b45a0 0x5558429b69c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9285af614365f886 server_cookie=bf7282cb700ac2e4 in_seq=0 out_seq=0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb0ffff640 10 client.17874 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb0ffff640 1 -- 172.21.15.16:0/2848113609 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55584289cd70 con 0x5558429b45a0 2024-09-12T02:58:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb0ffff640 10 client.17874 handle_command_reply: tid=0 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 client.17874 shutdown 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 msgr2=0x555842949e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fdb2802ae00 tx=0x7fdb200043a0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 shutdown_connections 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5558429b45a0 0x5558429b69c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb0ffff640 0 client.17874 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55584293a370 0x55584294a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555842930b40 0x55584294a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 --2- 172.21.15.16:0/2848113609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55584292f5d0 0x555842949e80 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.876+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 >> 172.21.15.16:0/2848113609 conn(0x5558428f8310 msgr2=0x5558429387e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 shutdown_connections 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb39102740 1 -- 172.21.15.16:0/2848113609 wait complete. 2024-09-12T02:58:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb39102740 20 client.17874 trim_cache size 0 max 16384 2024-09-12T02:58:07.883 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 msgr2=0x7fdb180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 0x7fdb180a5040 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400c940 tx=0x7fdb24090c10 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 msgr2=0x7fdb3019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400c970 tx=0x7fdb2400ce30 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 shutdown_connections 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb30106a80 0x7fdb3019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb180a2b80 0x7fdb180a5040 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb30101420 0x7fdb3019abe0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 --2- 172.21.15.16:0/1073852417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb30100a80 0x7fdb3019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 >> 172.21.15.16:0/1073852417 conn(0x7fdb300fc7a0 msgr2=0x7fdb3010c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 shutdown_connections 2024-09-12T02:58:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:07.877+0000 7fdb36d29640 1 -- 172.21.15.16:0/1073852417 wait complete. 2024-09-12T02:58:07.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:07 smithi033 ceph-mon[35092]: pgmap v643: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-12T02:58:08.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:07 smithi184 ceph-mon[35321]: pgmap v643: 97 pgs: 97 active+clean; 275 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-09-12T02:58:08.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:58:08.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.60 MB/sec execute 430 sec latency 14053.583 ms 2024-09-12T02:58:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 -- 172.21.15.16:0/204622362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa78410c160 msgr2=0x7fa78410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 --2- 172.21.15.16:0/204622362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa78410c160 0x7fa78410e600 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009ca0 tx=0x7fa77c02a950 comp rx=0 tx=0).stop 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 -- 172.21.15.16:0/204622362 shutdown_connections 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 --2- 172.21.15.16:0/204622362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa78410c160 0x7fa78410e600 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 --2- 172.21.15.16:0/204622362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa784103650 0x7fa78410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 --2- 172.21.15.16:0/204622362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784102d30 0x7fa784103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 -- 172.21.15.16:0/204622362 >> 172.21.15.16:0/204622362 conn(0x7fa7840fc7d0 msgr2=0x7fa7840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 -- 172.21.15.16:0/204622362 shutdown_connections 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 -- 172.21.15.16:0/204622362 wait complete. 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.215+0000 7fa78a39e640 1 Processor -- start 2024-09-12T02:58:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 -- start start 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784103650 0x7fa784100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa78410c160 0x7fa784101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa784111f60 con 0x7fa784102d30 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa784111de0 con 0x7fa78410c160 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78a39e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841120e0 con 0x7fa784103650 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39142/0 (socket says 172.21.15.16:39142) 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 -- 172.21.15.16:0/2550825016 learned_addr learned my addr 172.21.15.16:0/2550825016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa77bfff640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784103650 0x7fa784100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa78893b640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa78410c160 0x7fa784101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 -- 172.21.15.16:0/2550825016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784103650 msgr2=0x7fa784100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784103650 0x7fa784100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 -- 172.21.15.16:0/2550825016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa78410c160 msgr2=0x7fa784101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa78410c160 0x7fa784101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.216+0000 7fa783fff640 1 -- 172.21.15.16:0/2550825016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7841a6930 con 0x7fa784102d30 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa783fff640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa77400c970 tx=0x7fa77400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa781ffb640 1 -- 172.21.15.16:0/2550825016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774007bb0 con 0x7fa784102d30 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa781ffb640 1 -- 172.21.15.16:0/2550825016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa774007d50 con 0x7fa784102d30 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7841a6c20 con 0x7fa784102d30 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa781ffb640 1 -- 172.21.15.16:0/2550825016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77400f450 con 0x7fa784102d30 2024-09-12T02:58:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.217+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7841a7160 con 0x7fa784102d30 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.219+0000 7fa781ffb640 1 -- 172.21.15.16:0/2550825016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa774005020 con 0x7fa784102d30 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.219+0000 7fa78c777740 1 Processor -- start 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 -- start start 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129de760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e129372f0 con 0x556e129d15d0 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556e128decf0 con 0x556e129d2b40 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78c777740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556e129332b0 con 0x556e129dc370 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78893b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa781ffb640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 0x7fa7700a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78893b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47170/0 (socket says 172.21.15.16:47170) 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78893b640 1 -- 172.21.15.16:0/4176177007 learned_addr learned my addr 172.21.15.16:0/4176177007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa77bfff640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.220+0000 7fa78893b640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 0x7fa7700a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa781ffb640 1 -- 172.21.15.16:0/2550825016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa7740ceda0 con 0x7fa784102d30 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 -- 172.21.15.16:0/4176177007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 msgr2=0x556e129d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 -- 172.21.15.16:0/4176177007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 msgr2=0x556e129d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129d1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 -- 172.21.15.16:0/4176177007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e128c33a0 con 0x556e129dc370 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa78893b640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa783fff640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77bfff640 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129de760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0058c0 tx=0x7fa76c00af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ff2c728d85423 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa78893b640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 0x7fa7700a50c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009430 tx=0x7fa77c02a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77a7fc640 1 -- 172.21.15.16:0/4176177007 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c004020 con 0x556e129dc370 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77a7fc640 1 -- 172.21.15.16:0/4176177007 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c0041c0 con 0x556e129dc370 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.221+0000 7fa77a7fc640 1 -- 172.21.15.16:0/4176177007 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0187a0 con 0x556e129dc370 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- 172.21.15.16:0/4176177007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 msgr2=0x556e129de760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129de760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0058c0 tx=0x7fa76c00af40 comp rx=0 tx=0).stop 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- 172.21.15.16:0/4176177007 shutdown_connections 2024-09-12T02:58:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129de760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 --2- 172.21.15.16:0/4176177007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- 172.21.15.16:0/4176177007 >> 172.21.15.16:0/4176177007 conn(0x556e1299a310 msgr2=0x556e129cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- 172.21.15.16:0/4176177007 shutdown_connections 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- 172.21.15.16:0/4176177007 wait complete. 2024-09-12T02:58:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.222+0000 7fa78c777740 1 -- start start 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 Processor -- start 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 0x556e129dbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129d9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e129df070 con 0x556e129d2b40 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556e128f2060 con 0x556e129d15d0 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556e129df1f0 con 0x556e129dc370 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78c777740 10 client.0 _ll_register_callbacks cb 0x556e126458c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39152/0 (socket says 172.21.15.16:39152) 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 -- 172.21.15.16:0/3264905076 learned_addr learned my addr 172.21.15.16:0/3264905076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa78893b640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129d9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa783fff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 0x556e129dbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 -- 172.21.15.16:0/3264905076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 msgr2=0x556e129d9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129d9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 -- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 msgr2=0x556e129dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 0x556e129dbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa77bfff640 1 -- 172.21.15.16:0/3264905076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e126a83c0 con 0x556e129d2b40 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.223+0000 7fa783fff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 0x556e129dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa77bfff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa76c000c00 tx=0x7fa76c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348e1503e5d5625 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c027020 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa76c005040 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa76c0041c0 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa76c003a00 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 10 client.17886 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e129eaa20 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c0041c0 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa76c004c50 con 0x556e129d2b40 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 10 client.17886 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 10 client.17886 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 20 client.17886 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e129ebdd0 0x556e129ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 4 client.17886 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.224+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e12944860 con 0x556e129ebdd0 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.225+0000 7fa783fff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e129ebdd0 0x556e129ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.225+0000 7fa7637fe640 10 client.17886 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.225+0000 7fa783fff640 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e129ebdd0 0x556e129ee1c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=68062b0db9d20d59 server_cookie=72f40acd1c13c46a in_seq=0 out_seq=0 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa7637fe640 1 -- 172.21.15.16:0/3264905076 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556e12944860 con 0x556e129ebdd0 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa7637fe640 10 client.17886 handle_command_reply: tid=0 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 client.17886 shutdown 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 msgr2=0x556e129d8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa76c000c00 tx=0x7fa76c005e40 comp rx=0 tx=0).stop 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 shutdown_connections 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e129ebdd0 0x556e129ee1c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129d9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129d8f20 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 --2- 172.21.15.16:0/3264905076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d15d0 0x556e129dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 >> 172.21.15.16:0/3264905076 conn(0x556e1299a310 msgr2=0x556e129d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa7637fe640 0 client.17886 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 shutdown_connections 2024-09-12T02:58:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 1 -- 172.21.15.16:0/3264905076 wait complete. 2024-09-12T02:58:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.226+0000 7fa78c777740 20 client.17886 trim_cache size 0 max 16384 2024-09-12T02:58:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.239+0000 7fa78c777740 1 Processor -- start 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.239+0000 7fa78c777740 1 -- start start 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.239+0000 7fa78c777740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78c777740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129f3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78c777740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78c777740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841af760 con 0x556e129d15d0 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78c777740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841b12a0 con 0x556e129d2b40 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78c777740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841afe30 con 0x556e129dc370 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39160/0 (socket says 172.21.15.16:39160) 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 -- 172.21.15.16:0/807386861 learned_addr learned my addr 172.21.15.16:0/807386861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa77bfff640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78893b640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 -- 172.21.15.16:0/807386861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 msgr2=0x556e129f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 -- 172.21.15.16:0/807386861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 msgr2=0x556e129f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 -- 172.21.15.16:0/807386861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e129ce960 con 0x556e129d15d0 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa78893b640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa783fff640 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa7740cc8b0 tx=0x7fa7740cc780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea42ffbe00747b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa779ffb640 1 -- 172.21.15.16:0/807386861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77401d040 con 0x556e129d15d0 2024-09-12T02:58:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa779ffb640 1 -- 172.21.15.16:0/807386861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa774097040 con 0x556e129d15d0 2024-09-12T02:58:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.240+0000 7fa779ffb640 1 -- 172.21.15.16:0/807386861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7740999a0 con 0x556e129d15d0 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 -- 172.21.15.16:0/807386861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 msgr2=0x556e129e8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa7740cc8b0 tx=0x7fa7740cc780 comp rx=0 tx=0).stop 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 -- 172.21.15.16:0/807386861 shutdown_connections 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129dc370 0x556e129f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129d2b40 0x556e129f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 --2- 172.21.15.16:0/807386861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d15d0 0x556e129e8310 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 -- 172.21.15.16:0/807386861 >> 172.21.15.16:0/807386861 conn(0x556e1299a310 msgr2=0x556e129d8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 -- 172.21.15.16:0/807386861 shutdown_connections 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.241+0000 7fa78c777740 1 -- 172.21.15.16:0/807386861 wait complete. 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 -- start start 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 Processor -- start 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129d15d0 0x556e129e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129dc370 0x556e129e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841b0bd0 con 0x556e129d2b40 2024-09-12T02:58:09.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841b0500 con 0x556e129dc370 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841ad740 con 0x556e129d15d0 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78c777740 10 client.0 _ll_register_callbacks cb 0x7fa77409b3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa77bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39164/0 (socket says 172.21.15.16:39164) 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa77bfff640 1 -- 172.21.15.16:0/1264786506 learned_addr learned my addr 172.21.15.16:0/1264786506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa78893b640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129dc370 0x556e129e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.242+0000 7fa783fff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129d15d0 0x556e129e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 -- 172.21.15.16:0/1264786506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129d15d0 msgr2=0x556e129e4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129d15d0 0x556e129e4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 -- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129dc370 msgr2=0x556e129e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129dc370 0x556e129e1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 -- 172.21.15.16:0/1264786506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e129e1b60 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa77bfff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x556e129da4d0 tx=0x7fa76c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dac1f0727a3355a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa7637fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c03c070 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa76c003ea0 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa774097040 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa76c00a6f0 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa78c777740 10 client.17892 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e12a57170 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.243+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa774097040 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa76c0046d0 con 0x556e129d2b40 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 10 client.17892 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 10 client.17892 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 20 client.17892 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:09.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e12a578c0 0x556e12a59ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 4 client.17892 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e1293ed70 con 0x556e12a578c0 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa783fff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e12a578c0 0x556e12a59ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa783fff640 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e12a578c0 0x556e12a59ce0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6dec3183239e11dd server_cookie=4951101facb01968 in_seq=0 out_seq=0 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.244+0000 7fa7637fe640 10 client.17892 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa7637fe640 1 -- 172.21.15.16:0/1264786506 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e1293ed70 con 0x556e12a578c0 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa7637fe640 10 client.17892 handle_command_reply: tid=0 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 client.17892 shutdown 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 msgr2=0x556e129e4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x556e129da4d0 tx=0x7fa76c0058a0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 shutdown_connections 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556e12a578c0 0x556e12a59ce0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556e129dc370 0x556e129e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e129d2b40 0x556e129e4ba0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 --2- 172.21.15.16:0/1264786506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556e129d15d0 0x556e129e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 >> 172.21.15.16:0/1264786506 conn(0x556e1299a310 msgr2=0x556e129da780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa7637fe640 0 client.17892 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 shutdown_connections 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 1 -- 172.21.15.16:0/1264786506 wait complete. 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78c777740 20 client.17892 trim_cache size 0 max 16384 2024-09-12T02:58:09.251 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.245+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 msgr2=0x7fa7700a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 0x7fa7700a50c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009430 tx=0x7fa77c02a950 comp rx=0 tx=0).stop 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 msgr2=0x7fa784100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa77400c970 tx=0x7fa77400ce30 comp rx=0 tx=0).stop 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 shutdown_connections 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa78410c160 0x7fa784101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa784103650 0x7fa784100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7700a2c00 0x7fa7700a50c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 --2- 172.21.15.16:0/2550825016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784102d30 0x7fa784100990 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:09.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 >> 172.21.15.16:0/2550825016 conn(0x7fa7840fc7d0 msgr2=0x7fa7840fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:09.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 shutdown_connections 2024-09-12T02:58:09.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:09.246+0000 7fa78a39e640 1 -- 172.21.15.16:0/2550825016 wait complete. 2024-09-12T02:58:09.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.58 MB/sec execute 431 sec latency 15053.631 ms 2024-09-12T02:58:09.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:09 smithi016 ceph-mon[30925]: pgmap v644: 97 pgs: 97 active+clean; 273 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-09-12T02:58:09.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:58:09.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:09 smithi033 ceph-mon[35092]: pgmap v644: 97 pgs: 97 active+clean; 273 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-09-12T02:58:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:09 smithi184 ceph-mon[35321]: pgmap v644: 97 pgs: 97 active+clean; 273 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-09-12T02:58:10.220 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/3416451619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a010c170 msgr2=0x7fd1a010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/3416451619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a010c170 0x7fd1a010e610 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd198009ca0 tx=0x7fd19802a950 comp rx=0 tx=0).stop 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/3416451619 shutdown_connections 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/3416451619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a010c170 0x7fd1a010e610 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/3416451619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0103660 0x7fd1a010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/3416451619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a0102d40 0x7fd1a0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/3416451619 >> 172.21.15.16:0/3416451619 conn(0x7fd1a00fc7a0 msgr2=0x7fd1a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/3416451619 shutdown_connections 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.673+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/3416451619 wait complete. 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 Processor -- start 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 -- start start 2024-09-12T02:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 0x7fd1a01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a010c170 0x7fd1a0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0111f70 con 0x7fd1a0103660 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0111df0 con 0x7fd1a0102d40 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a7bfa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01120f0 con 0x7fd1a010c170 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a5195640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a5195640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39182/0 (socket says 172.21.15.16:39182) 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a5195640 1 -- 172.21.15.16:0/747800156 learned_addr learned my addr 172.21.15.16:0/747800156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a5996640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 0x7fd1a01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.674+0000 7fd1a6197640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a010c170 0x7fd1a0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 -- 172.21.15.16:0/747800156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a010c170 msgr2=0x7fd1a0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a010c170 0x7fd1a0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 -- 172.21.15.16:0/747800156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 msgr2=0x7fd1a01009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 0x7fd1a01009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 -- 172.21.15.16:0/747800156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a01a68d0 con 0x7fd1a0103660 2024-09-12T02:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5996640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 0x7fd1a01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a5195640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd19000d740 tx=0x7fd19000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd196ffd640 1 -- 172.21.15.16:0/747800156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190017070 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd196ffd640 1 -- 172.21.15.16:0/747800156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd19000de50 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a01a6bc0 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd196ffd640 1 -- 172.21.15.16:0/747800156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19000f620 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.675+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a01a7150 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.677+0000 7fd196ffd640 1 -- 172.21.15.16:0/747800156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd190009c30 con 0x7fd1a0103660 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 Processor -- start 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 -- start start 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70beea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc70b542f0 con 0x55fc70bf9370 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fc70afbcf0 con 0x55fc70bee5d0 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fc70b502b0 con 0x55fc70befb40 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd196ffd640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 0x7fd18c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a5195640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a5195640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39190/0 (socket says 172.21.15.16:39190) 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a5195640 1 -- 172.21.15.16:0/2271915784 learned_addr learned my addr 172.21.15.16:0/2271915784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a6197640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 0x7fd18c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.678+0000 7fd1a6197640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5996640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70beea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd196ffd640 1 -- 172.21.15.16:0/747800156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd1900cebb0 con 0x7fd1a0103660 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 -- 172.21.15.16:0/2271915784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 msgr2=0x55fc70bee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bee090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 -- 172.21.15.16:0/2271915784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 msgr2=0x55fc70beea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70beea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 -- 172.21.15.16:0/2271915784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc70ae03a0 con 0x55fc70bf9370 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5996640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70beea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a5195640 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd190097df0 tx=0x7fd190000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c514d57aa147014 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd1a6197640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 0x7fd18c0a50c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd198006010 tx=0x7fd1980095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd18bfff640 1 -- 172.21.15.16:0/2271915784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190020050 con 0x55fc70bf9370 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd18bfff640 1 -- 172.21.15.16:0/2271915784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd19009f7d0 con 0x55fc70bf9370 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.679+0000 7fd18bfff640 1 -- 172.21.15.16:0/2271915784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19009a870 con 0x55fc70bf9370 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/2271915784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 msgr2=0x55fc70bfb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd190097df0 tx=0x7fd190000bc0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/2271915784 shutdown_connections 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bfb760 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70beea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/2271915784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/2271915784 >> 172.21.15.16:0/2271915784 conn(0x55fc70bb7310 msgr2=0x55fc70beac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/2271915784 shutdown_connections 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/2271915784 wait complete. 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.680+0000 7fd1a9fd3740 1 -- start start 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 Processor -- start 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 0x55fc70bf6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70bf9370 0x55fc70bf6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc70bfc010 con 0x55fc70bee5d0 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fc70b0f060 con 0x55fc70befb40 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fc70bfc190 con 0x55fc70bf9370 2024-09-12T02:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a9fd3740 10 client.0 _ll_register_callbacks cb 0x55fc708628c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5996640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5996640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39202/0 (socket says 172.21.15.16:39202) 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5996640 1 -- 172.21.15.16:0/3131226305 learned_addr learned my addr 172.21.15.16:0/3131226305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5195640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 0x55fc70bf6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a6197640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70bf9370 0x55fc70bf6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5996640 1 -- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70bf9370 msgr2=0x55fc70bf6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.681+0000 7fd1a5996640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70bf9370 0x55fc70bf6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a5996640 1 -- 172.21.15.16:0/3131226305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 msgr2=0x55fc70bf6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a5996640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 0x55fc70bf6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a5996640 1 -- 172.21.15.16:0/3131226305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc708c53c0 con 0x55fc70bee5d0 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a5195640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 0x55fc70bf6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a5996640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd19c010670 tx=0x7fd19c010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c14660eee34c44 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ead0 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd19c00ec70 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd19009f7d0 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd19c010e70 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a9fd3740 10 client.17904 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc70c07bc0 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd19009f7d0 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd19c00f040 con 0x55fc70bee5d0 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.682+0000 7fd1a9fd3740 10 client.17904 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a9fd3740 10 client.17904 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a9fd3740 20 client.17904 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c09080 0x55fc70c0b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a9fd3740 4 client.17904 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc70b61860 con 0x55fc70c09080 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a5195640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c09080 0x55fc70c0b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1a5195640 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c09080 0x55fc70c0b490 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e23ce85e9099b7c server_cookie=7fcd8d6fa0ccfdb7 in_seq=0 out_seq=0 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.683+0000 7fd1897fa640 10 client.17904 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1897fa640 1 -- 172.21.15.16:0/3131226305 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fc70b61860 con 0x55fc70c09080 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1897fa640 10 client.17904 handle_command_reply: tid=0 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 client.17904 shutdown 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 msgr2=0x55fc70bf8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd19c010670 tx=0x7fd19c010b30 comp rx=0 tx=0).stop 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 shutdown_connections 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c09080 0x55fc70c0b490 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70bf9370 0x55fc70bf6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70befb40 0x55fc70bf6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1897fa640 0 client.17904 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/3131226305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70bf8d90 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.684+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 >> 172.21.15.16:0/3131226305 conn(0x55fc70bb7310 msgr2=0x55fc70bf0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.685+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 shutdown_connections 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.685+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/3131226305 wait complete. 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.685+0000 7fd1a9fd3740 20 client.17904 trim_cache size 0 max 16384 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 Processor -- start 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 -- start start 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70be9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01af730 con 0x55fc70bf9370 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01b1270 con 0x55fc70bee5d0 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01afe00 con 0x55fc70befb40 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a6197640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a6197640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39214/0 (socket says 172.21.15.16:39214) 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.698+0000 7fd1a6197640 1 -- 172.21.15.16:0/70152049 learned_addr learned my addr 172.21.15.16:0/70152049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a5996640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70be9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a5195640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 -- 172.21.15.16:0/70152049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 msgr2=0x55fc70bea130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bea130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 -- 172.21.15.16:0/70152049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 msgr2=0x55fc70be9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70be9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 -- 172.21.15.16:0/70152049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc70bf6fe0 con 0x55fc70bf9370 2024-09-12T02:58:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a5195640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bea130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a5996640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70be9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd1a6197640 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd19802ff60 tx=0x7fd198041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce534ad04360fe3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd18b7fe640 1 -- 172.21.15.16:0/70152049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198034040 con 0x55fc70bf9370 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd18b7fe640 1 -- 172.21.15.16:0/70152049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd198003e50 con 0x55fc70bf9370 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.699+0000 7fd18b7fe640 1 -- 172.21.15.16:0/70152049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1980556e0 con 0x55fc70bf9370 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/70152049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 msgr2=0x55fc70bf1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd19802ff60 tx=0x7fd198041000 comp rx=0 tx=0).stop 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/70152049 shutdown_connections 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bf9370 0x55fc70bf1180 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bea130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/70152049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bee5d0 0x55fc70be9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/70152049 >> 172.21.15.16:0/70152049 conn(0x55fc70bb7310 msgr2=0x55fc70be94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/70152049 shutdown_connections 2024-09-12T02:58:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/70152049 wait complete. 2024-09-12T02:58:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 -- start start 2024-09-12T02:58:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.700+0000 7fd1a9fd3740 1 Processor -- start 2024-09-12T02:58:10.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bfcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bf9370 0x55fc70bfd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01b0ba0 con 0x55fc70bee5d0 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01b04d0 con 0x55fc70bf9370 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ad710 con 0x55fc70befb40 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a9fd3740 10 client.0 _ll_register_callbacks cb 0x7fd19c0093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39230/0 (socket says 172.21.15.16:39230) 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 -- 172.21.15.16:0/163372217 learned_addr learned my addr 172.21.15.16:0/163372217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5195640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bfcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a6197640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bf9370 0x55fc70bfd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 -- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 msgr2=0x55fc70bfcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bfcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 -- 172.21.15.16:0/163372217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bf9370 msgr2=0x55fc70bfd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bf9370 0x55fc70bfd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 -- 172.21.15.16:0/163372217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc70bfd9c0 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1a5996640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd19c011010 tx=0x7fd19c010300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372f78263e352a89 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.701+0000 7fd1897fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ead0 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd19c00ec70 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd198003e50 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd19c01caf0 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 10 client.17910 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc70c73bf0 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd198003e50 con 0x55fc70bee5d0 2024-09-12T02:58:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd19c02f020 con 0x55fc70bee5d0 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 10 client.17910 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 10 client.17910 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 20 client.17910 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c74310 0x55fc70c76730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 4 client.17910 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.702+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc70b5bd70 con 0x55fc70c74310 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1a5195640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c74310 0x55fc70c76730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1897fa640 10 client.17910 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1a5195640 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c74310 0x55fc70c76730 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52252ea7c5a0cf6c server_cookie=5db0e29719166cff in_seq=0 out_seq=0 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1897fa640 1 -- 172.21.15.16:0/163372217 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc70b5bd70 con 0x55fc70c74310 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1897fa640 10 client.17910 handle_command_reply: tid=0 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.703+0000 7fd1a9fd3740 1 client.17910 shutdown 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 msgr2=0x55fc70c01a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd19c011010 tx=0x7fd19c010300 comp rx=0 tx=0).stop 2024-09-12T02:58:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 shutdown_connections 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fc70bf9370 0x55fc70bfd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55fc70c74310 0x55fc70c76730 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fc70befb40 0x55fc70bfcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 --2- 172.21.15.16:0/163372217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc70bee5d0 0x55fc70c01a60 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1897fa640 0 client.17910 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 >> 172.21.15.16:0/163372217 conn(0x55fc70bb7310 msgr2=0x55fc70be9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 shutdown_connections 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 1 -- 172.21.15.16:0/163372217 wait complete. 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a9fd3740 20 client.17910 trim_cache size 0 max 16384 2024-09-12T02:58:10.720 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 msgr2=0x7fd18c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 0x7fd18c0a50c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd198006010 tx=0x7fd1980095c0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 msgr2=0x7fd1a0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd19000d740 tx=0x7fd19000dc00 comp rx=0 tx=0).stop 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 shutdown_connections 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1a010c170 0x7fd1a0101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd18c0a2c00 0x7fd18c0a50c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a0103660 0x7fd1a0100ee0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.704+0000 7fd1a7bfa640 1 --2- 172.21.15.16:0/747800156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1a0102d40 0x7fd1a01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.705+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 >> 172.21.15.16:0/747800156 conn(0x7fd1a00fc7a0 msgr2=0x7fd1a00fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.705+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 shutdown_connections 2024-09-12T02:58:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:10.705+0000 7fd1a7bfa640 1 -- 172.21.15.16:0/747800156 wait complete. 2024-09-12T02:58:10.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.56 MB/sec execute 432 sec latency 16053.671 ms 2024-09-12T02:58:11.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:58:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:11.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:11 smithi016 ceph-mon[30925]: pgmap v645: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-12T02:58:11.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.54 MB/sec execute 433 sec latency 17053.788 ms 2024-09-12T02:58:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:11 smithi033 ceph-mon[35092]: pgmap v645: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-12T02:58:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:11 smithi184 ceph-mon[35321]: pgmap v645: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-12T02:58:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.098+0000 7f0f367b3640 1 -- 172.21.15.16:0/2963932308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30102790 msgr2=0x7f0f301067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.098+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2963932308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30102790 0x7f0f301067e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009900 tx=0x7f0f2402a410 comp rx=0 tx=0).stop 2024-09-12T02:58:12.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 -- 172.21.15.16:0/2963932308 shutdown_connections 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2963932308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30106f10 0x7f0f3010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2963932308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30102790 0x7f0f301067e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2963932308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 0x7f0f301021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 -- 172.21.15.16:0/2963932308 >> 172.21.15.16:0/2963932308 conn(0x7f0f300fd8c0 msgr2=0x7f0f300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 -- 172.21.15.16:0/2963932308 shutdown_connections 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 -- 172.21.15.16:0/2963932308 wait complete. 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 Processor -- start 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.099+0000 7f0f367b3640 1 -- start start 2024-09-12T02:58:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 0x7f0f30198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30102790 0x7f0f30198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30111f30 con 0x7f0f30106f10 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30111db0 con 0x7f0f30102790 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301120b0 con 0x7f0f30101de0 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39242/0 (socket says 172.21.15.16:39242) 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 -- 172.21.15.16:0/2708774048 learned_addr learned my addr 172.21.15.16:0/2708774048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 0x7f0f30198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30102790 0x7f0f30198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 -- 172.21.15.16:0/2708774048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 msgr2=0x7f0f30198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 0x7f0f30198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 -- 172.21.15.16:0/2708774048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30102790 msgr2=0x7f0f30198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30102790 0x7f0f30198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 -- 172.21.15.16:0/2708774048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3019d3c0 con 0x7f0f30106f10 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800cad0 tx=0x7f0f2800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f2d7fa640 1 -- 172.21.15.16:0/2708774048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2800ad60 con 0x7f0f30106f10 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f2d7fa640 1 -- 172.21.15.16:0/2708774048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f28005020 con 0x7f0f30106f10 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.100+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3019d6b0 con 0x7f0f30106f10 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.101+0000 7f0f2d7fa640 1 -- 172.21.15.16:0/2708774048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2800f6b0 con 0x7f0f30106f10 2024-09-12T02:58:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.101+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f301a4f90 con 0x7f0f30106f10 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 Processor -- start 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f2d7fa640 1 -- 172.21.15.16:0/2708774048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0f28005560 con 0x7f0f30106f10 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 -- start start 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 0x5556e2b22090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b33b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556e2a882f0 con 0x5556e2b225d0 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5556e2a2fcf0 con 0x5556e2b23b40 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5556e2a842b0 con 0x5556e2b31780 2024-09-12T02:58:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.103+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39248/0 (socket says 172.21.15.16:39248) 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 -- 172.21.15.16:0/4093980196 learned_addr learned my addr 172.21.15.16:0/4093980196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b33b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 0x5556e2b22090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 -- 172.21.15.16:0/4093980196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 msgr2=0x5556e2b33b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b33b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 -- 172.21.15.16:0/4093980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 msgr2=0x5556e2b22090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 0x5556e2b22090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 -- 172.21.15.16:0/4093980196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e2a143a0 con 0x5556e2b225d0 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 0x5556e2b22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2d7fa640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 0x7f0f180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000ad00 tx=0x7f0f2000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46ee7fc48c7b8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 0x7f0f180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f2d7fa640 1 -- 172.21.15.16:0/2708774048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0f280cefe0 con 0x7f0f30106f10 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f167fc640 1 -- 172.21.15.16:0/4093980196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001d070 con 0x5556e2b225d0 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f167fc640 1 -- 172.21.15.16:0/4093980196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f20018b20 con 0x5556e2b225d0 2024-09-12T02:58:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.104+0000 7f0f167fc640 1 -- 172.21.15.16:0/4093980196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20010bc0 con 0x5556e2b225d0 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- 172.21.15.16:0/4093980196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 msgr2=0x5556e2b22a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 0x7f0f180a5040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0f30108800 tx=0x7f0f2402a410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000ad00 tx=0x7f0f2000f790 comp rx=0 tx=0).stop 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- 172.21.15.16:0/4093980196 shutdown_connections 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b33b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b225d0 0x5556e2b22a10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/4093980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b23b40 0x5556e2b22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- 172.21.15.16:0/4093980196 >> 172.21.15.16:0/4093980196 conn(0x5556e2aeb310 msgr2=0x5556e2b1ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- 172.21.15.16:0/4093980196 shutdown_connections 2024-09-12T02:58:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- 172.21.15.16:0/4093980196 wait complete. 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.105+0000 7f0f38b8c740 1 -- start start 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 Processor -- start 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b31160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556e2b34480 con 0x5556e2b31780 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5556e2a43060 con 0x5556e2b225d0 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5556e2b34600 con 0x5556e2b23b40 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f38b8c740 10 client.0 _ll_register_callbacks cb 0x5556e27968c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39258/0 (socket says 172.21.15.16:39258) 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 -- 172.21.15.16:0/2080868979 learned_addr learned my addr 172.21.15.16:0/2080868979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b31160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 -- 172.21.15.16:0/2080868979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 msgr2=0x5556e2b2e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 -- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 msgr2=0x5556e2b31160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b31160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.106+0000 7f0f34d50640 1 -- 172.21.15.16:0/2080868979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e27f93c0 con 0x5556e2b31780 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0f280a4030 tx=0x7f0f2809b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18f02ec01d96622 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2801d040 con 0x5556e2b31780 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0f280a40d0 con 0x5556e2b31780 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f20018b20 con 0x5556e2b31780 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0f280a42d0 con 0x5556e2b31780 2024-09-12T02:58:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 10 client.17922 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5556e2b3fed0 con 0x5556e2b31780 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f20018b20 con 0x5556e2b31780 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0f2809ae60 con 0x5556e2b31780 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 10 client.17922 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 10 client.17922 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 20 client.17922 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2b412e0 0x5556e2b43760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 4 client.17922 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.107+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556e2a95860 con 0x5556e2b412e0 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.108+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2b412e0 0x5556e2b43760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.108+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2b412e0 0x5556e2b43760 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c3650ff3326e96b server_cookie=efc1330eaafcf5c1 in_seq=0 out_seq=0 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.108+0000 7f0f0f7fe640 10 client.17922 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/2080868979 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556e2a95860 con 0x5556e2b412e0 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f0f7fe640 10 client.17922 handle_command_reply: tid=0 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 client.17922 shutdown 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 msgr2=0x5556e2b2e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0f280a4030 tx=0x7f0f2809b8f0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 shutdown_connections 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2b412e0 0x5556e2b43760 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2e9e0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f0f7fe640 0 client.17922 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2080868979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b31160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 >> 172.21.15.16:0/2080868979 conn(0x5556e2aeb310 msgr2=0x5556e2b24960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 shutdown_connections 2024-09-12T02:58:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2080868979 wait complete. 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.109+0000 7f0f38b8c740 20 client.17922 trim_cache size 0 max 16384 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.122+0000 7f0f38b8c740 1 Processor -- start 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.122+0000 7f0f38b8c740 1 -- start start 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b3d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b1ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ab2f0 con 0x5556e2b23b40 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ace30 con 0x5556e2b225d0 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ab9c0 con 0x5556e2b31780 2024-09-12T02:58:12.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39270/0 (socket says 172.21.15.16:39270) 2024-09-12T02:58:12.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 -- 172.21.15.16:0/2268019479 learned_addr learned my addr 172.21.15.16:0/2268019479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:12.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f34d50640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b1ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.144 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:12.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b3d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 -- 172.21.15.16:0/2268019479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 msgr2=0x5556e2b1ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b1ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 -- 172.21.15.16:0/2268019479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 msgr2=0x5556e2b3d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b3d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 -- 172.21.15.16:0/2268019479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e2b25120 con 0x5556e2b23b40 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b3d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0f24033000 tx=0x7f0f2403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8202a8ab71169354 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.123+0000 7f0f15ffb640 1 -- 172.21.15.16:0/2268019479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2404e070 con 0x5556e2b23b40 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f15ffb640 1 -- 172.21.15.16:0/2268019479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f24004240 con 0x5556e2b23b40 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f15ffb640 1 -- 172.21.15.16:0/2268019479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24035040 con 0x5556e2b23b40 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2268019479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 msgr2=0x5556e2b1e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0f24033000 tx=0x7f0f2403e000 comp rx=0 tx=0).stop 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2268019479 shutdown_connections 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b31780 0x5556e2b1ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b23b40 0x5556e2b1e700 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/2268019479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b3d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2268019479 >> 172.21.15.16:0/2268019479 conn(0x5556e2aeb310 msgr2=0x5556e2b2e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2268019479 shutdown_connections 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.124+0000 7f0f38b8c740 1 -- 172.21.15.16:0/2268019479 wait complete. 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 -- start start 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 Processor -- start 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b2b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ac760 con 0x5556e2b31780 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ac090 con 0x5556e2b225d0 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301a92d0 con 0x5556e2b23b40 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f38b8c740 10 client.0 _ll_register_callbacks cb 0x7f0f28097b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f34d50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39274/0 (socket says 172.21.15.16:39274) 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f34d50640 1 -- 172.21.15.16:0/1804587556 learned_addr learned my addr 172.21.15.16:0/1804587556 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b2b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.125+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 -- 172.21.15.16:0/1804587556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 msgr2=0x5556e2b2b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 -- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 msgr2=0x5556e2b2b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b2b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 -- 172.21.15.16:0/1804587556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e2bab050 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f2f7fe640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f34d50640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0f28099ba0 tx=0x7f0f28097c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8d870b17076ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2801d040 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0f280a40d0 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f24004240 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0f28097940 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 10 client.17928 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5556e2bac1a0 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f24004240 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0f2809a830 con 0x5556e2b31780 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 10 client.17928 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 10 client.17928 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 20 client.17928 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.126+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2bac700 0x5556e2baeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f38b8c740 4 client.17928 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556e2a8fd70 con 0x5556e2bac700 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2bac700 0x5556e2baeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f0f7fe640 10 client.17928 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f2ffff640 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2bac700 0x5556e2baeb20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc9bd08be0c54ed6 server_cookie=18aac3d683943dd in_seq=0 out_seq=0 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f0f7fe640 1 -- 172.21.15.16:0/1804587556 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556e2a8fd70 con 0x5556e2bac700 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f0f7fe640 10 client.17928 handle_command_reply: tid=0 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f38b8c740 1 client.17928 shutdown 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 msgr2=0x5556e2b2bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.127+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0f28099ba0 tx=0x7f0f28097c40 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 shutdown_connections 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5556e2bac700 0x5556e2baeb20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556e2b31780 0x5556e2b2bd90 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5556e2b23b40 0x5556e2b2b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f0f7fe640 0 client.17928 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 --2- 172.21.15.16:0/1804587556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5556e2b225d0 0x5556e2b2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 >> 172.21.15.16:0/1804587556 conn(0x5556e2aeb310 msgr2=0x5556e2b2fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 shutdown_connections 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 1 -- 172.21.15.16:0/1804587556 wait complete. 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f38b8c740 20 client.17928 trim_cache size 0 max 16384 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 msgr2=0x7f0f180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 0x7f0f180a5040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0f30108800 tx=0x7f0f2402a410 comp rx=0 tx=0).stop 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 msgr2=0x7f0f3019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800cad0 tx=0x7f0f2800ced0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 shutdown_connections 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0f180a2b80 0x7f0f180a5040 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f30106f10 0x7f0f3019cce0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f30102790 0x7f0f30198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 --2- 172.21.15.16:0/2708774048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f30101de0 0x7f0f30198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 >> 172.21.15.16:0/2708774048 conn(0x7f0f300fd8c0 msgr2=0x7f0f300fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 shutdown_connections 2024-09-12T02:58:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:12.128+0000 7f0f367b3640 1 -- 172.21.15.16:0/2708774048 wait complete. 2024-09-12T02:58:12.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.51 MB/sec execute 434 sec latency 18053.820 ms 2024-09-12T02:58:12.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:58:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/177244925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc106ed0 msgr2=0x7f4bbc10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/177244925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc10eaa0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009ca0 tx=0x7f4bb402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/177244925 shutdown_connections 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/177244925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc10eaa0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/177244925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/177244925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc100500 0x7f4bbc1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/177244925 >> 172.21.15.16:0/177244925 conn(0x7f4bbc0fc5c0 msgr2=0x7f4bbc0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/177244925 shutdown_connections 2024-09-12T02:58:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.615+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/177244925 wait complete. 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 Processor -- start 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 -- start start 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100500 0x7f4bbc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc111fa0 con 0x7f4bbc100eb0 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc111e20 con 0x7f4bbc100500 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc2a2a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc112120 con 0x7f4bbc106ed0 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39300/0 (socket says 172.21.15.16:39300) 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100500 0x7f4bbc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1898446186 learned_addr learned my addr 172.21.15.16:0/1898446186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.616+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1898446186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc106ed0 msgr2=0x7f4bbc19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1898446186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100500 msgr2=0x7f4bbc198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100500 0x7f4bbc198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1898446186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bbc19d440 con 0x7f4bbc100eb0 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000b4a0 tx=0x7f4bb000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bb97fa640 1 -- 172.21.15.16:0/1898446186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0014070 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bb97fa640 1 -- 172.21.15.16:0/1898446186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb0010af0 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bbc19d730 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bb97fa640 1 -- 172.21.15.16:0/1898446186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0017d10 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.617+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bbc1a5010 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bb97fa640 1 -- 172.21.15.16:0/1898446186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4bb0017070 con 0x7f4bbc100eb0 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 Processor -- start 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 -- start start 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfae9fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cfae052f0 con 0x564cfaeaa370 2024-09-12T02:58:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564cfadaccf0 con 0x564cfae9f5d0 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564cfae012b0 con 0x564cfaea0b40 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bb97fa640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 0x7f4ba40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59338/0 (socket says 172.21.15.16:59338) 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfae9fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/2240092459 learned_addr learned my addr 172.21.15.16:0/2240092459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.620+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39308/0 (socket says 172.21.15.16:39308) 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 0x7f4ba40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bb97fa640 1 -- 172.21.15.16:0/1898446186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bb00ce2d0 con 0x7f4bbc100eb0 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/2240092459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 msgr2=0x564cfae9f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/2240092459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 msgr2=0x564cfae9fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfae9fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/2240092459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cfad913a0 con 0x564cfaeaa370 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfae9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0097560 tx=0x7f4bb0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5cf61e44d32c40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4ba27fc640 1 -- 172.21.15.16:0/2240092459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb001e070 con 0x564cfaeaa370 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4ba27fc640 1 -- 172.21.15.16:0/2240092459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb009ee90 con 0x564cfaeaa370 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.621+0000 7f4ba27fc640 1 -- 172.21.15.16:0/2240092459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0097200 con 0x564cfaeaa370 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2240092459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 msgr2=0x564cfaeac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0097560 tx=0x7f4bb0010000 comp rx=0 tx=0).stop 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 0x7f4ba40a50c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4031f80 tx=0x7f4bb402a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2240092459 shutdown_connections 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeac760 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfae9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2240092459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfae9f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2240092459 >> 172.21.15.16:0/2240092459 conn(0x564cfae68310 msgr2=0x564cfae9bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2240092459 shutdown_connections 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2240092459 wait complete. 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 -- start start 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.622+0000 7f4bc4e03740 1 Processor -- start 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaea7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564cfaead070 con 0x564cfaeaa370 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564cfadc0060 con 0x564cfae9f5d0 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564cfaead1f0 con 0x564cfaea0b40 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc4e03740 10 client.0 _ll_register_callbacks cb 0x564cfab138c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39312/0 (socket says 172.21.15.16:39312) 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/4166380281 learned_addr learned my addr 172.21.15.16:0/4166380281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaea7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/4166380281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 msgr2=0x564cfaea7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaea7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 msgr2=0x564cfaea9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/4166380281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cfab763c0 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009bf0 tx=0x7f4bb40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9544ec7e6af37cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.623+0000 7f4b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb404f070 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bb402f610 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bb009ee90 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bb4004b40 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 10 client.17940 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564cfaeb8b10 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb009ee90 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bb404a040 con 0x564cfaeaa370 2024-09-12T02:58:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 10 client.17940 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 10 client.17940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 20 client.17940 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaeb9f20 0x564cfaebc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 4 client.17940 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.624+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cfae12860 con 0x564cfaeb9f20 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.625+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaeb9f20 0x564cfaebc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.625+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaeb9f20 0x564cfaebc350 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5bab01b904460e server_cookie=8a558f1cc4f94d2d in_seq=0 out_seq=0 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.625+0000 7f4b9b7fe640 10 client.17940 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/4166380281 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564cfae12860 con 0x564cfaeb9f20 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4b9b7fe640 10 client.17940 handle_command_reply: tid=0 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 client.17940 shutdown 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 msgr2=0x564cfaea7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009bf0 tx=0x7f4bb40095c0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 shutdown_connections 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaeb9f20 0x564cfaebc350 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaea7610 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4b9b7fe640 0 client.17940 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaea7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/4166380281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 >> 172.21.15.16:0/4166380281 conn(0x564cfae68310 msgr2=0x564cfaea1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 shutdown_connections 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 1 -- 172.21.15.16:0/4166380281 wait complete. 2024-09-12T02:58:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.626+0000 7f4bc4e03740 20 client.17940 trim_cache size 0 max 16384 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 Processor -- start 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 -- start start 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaeaa370 0x564cfaea2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1ad5e0 con 0x564cfaea0b40 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1af120 con 0x564cfae9f5d0 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1adcb0 con 0x564cfaeaa370 2024-09-12T02:58:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39320/0 (socket says 172.21.15.16:39320) 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1266192414 learned_addr learned my addr 172.21.15.16:0/1266192414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaeaa370 0x564cfaea2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1266192414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaeaa370 msgr2=0x564cfaea2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaeaa370 0x564cfaea2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1266192414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 msgr2=0x564cfaea8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.640+0000 7f4bbb7fe640 1 -- 172.21.15.16:0/1266192414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cfaea5fd0 con 0x564cfaea0b40 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.641+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0098e40 tx=0x7f4bb00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3984ca4b94b4c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.641+0000 7f4ba1ffb640 1 -- 172.21.15.16:0/1266192414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb001e040 con 0x564cfaea0b40 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.641+0000 7f4ba1ffb640 1 -- 172.21.15.16:0/1266192414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb00a3750 con 0x564cfaea0b40 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.641+0000 7f4ba1ffb640 1 -- 172.21.15.16:0/1266192414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0099fe0 con 0x564cfaea0b40 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- 172.21.15.16:0/1266192414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 msgr2=0x564cfaea2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0098e40 tx=0x7f4bb00047d0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- 172.21.15.16:0/1266192414 shutdown_connections 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaeaa370 0x564cfaea2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaea0b40 0x564cfaea2120 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/1266192414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaea8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- 172.21.15.16:0/1266192414 >> 172.21.15.16:0/1266192414 conn(0x564cfae68310 msgr2=0x564cfaea8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- 172.21.15.16:0/1266192414 shutdown_connections 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- 172.21.15.16:0/1266192414 wait complete. 2024-09-12T02:58:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 -- start start 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.642+0000 7f4bc4e03740 1 Processor -- start 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaeb2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaeb29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1aea50 con 0x564cfaeaa370 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1ae380 con 0x564cfae9f5d0 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc1ab5c0 con 0x564cfaea0b40 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 10 client.0 _ll_register_callbacks cb 0x7f4bb4003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39330/0 (socket says 172.21.15.16:39330) 2024-09-12T02:58:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/2974959961 learned_addr learned my addr 172.21.15.16:0/2974959961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaeb2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bbb7fe640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaeb29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/2974959961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 msgr2=0x564cfaeb29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaeb29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 msgr2=0x564cfaeb2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaeb2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 -- 172.21.15.16:0/2974959961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cfaeb3100 con 0x564cfaeaa370 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc0fc7640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4bb402f540 tx=0x7f4bb4030fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c941aade476e5d19 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb404f040 con 0x564cfaeaa370 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.643+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bb00a3750 con 0x564cfaeaa370 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4bb402f810 con 0x564cfaeaa370 2024-09-12T02:58:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bb4004410 con 0x564cfaeaa370 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 10 client.17946 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564cfaf25200 con 0x564cfaeaa370 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb00a3750 con 0x564cfaeaa370 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bb404a040 con 0x564cfaeaa370 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 10 client.17946 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 20 client.17946 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaf25920 0x564cfaf27d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 4 client.17946 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cfae0cd70 con 0x564cfaf25920 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.644+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaf25920 0x564cfaf27d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bbbfff640 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaf25920 0x564cfaf27d10 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46130226fb99e2c0 server_cookie=77167e956f48da1d in_seq=0 out_seq=0 2024-09-12T02:58:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4b9b7fe640 10 client.17946 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4b9b7fe640 1 -- 172.21.15.16:0/2974959961 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cfae0cd70 con 0x564cfaf25920 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4b9b7fe640 10 client.17946 handle_command_reply: tid=0 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 client.17946 shutdown 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 msgr2=0x564cfaeaf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4bb402f540 tx=0x7f4bb4030fd0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 shutdown_connections 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564cfaf25920 0x564cfaf27d10 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564cfaeaa370 0x564cfaeaf450 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564cfaea0b40 0x564cfaeb29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 --2- 172.21.15.16:0/2974959961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cfae9f5d0 0x564cfaeb2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 >> 172.21.15.16:0/2974959961 conn(0x564cfae68310 msgr2=0x564cfaea90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4b9b7fe640 0 client.17946 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 shutdown_connections 2024-09-12T02:58:13.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 1 -- 172.21.15.16:0/2974959961 wait complete. 2024-09-12T02:58:13.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.645+0000 7f4bc4e03740 20 client.17946 trim_cache size 0 max 16384 2024-09-12T02:58:13.656 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 msgr2=0x7f4ba40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 0x7f4ba40a50c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4031f80 tx=0x7f4bb402a8c0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 msgr2=0x7f4bbc1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000b4a0 tx=0x7f4bb000b960 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 shutdown_connections 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bbc106ed0 0x7f4bbc19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ba40a2c00 0x7f4ba40a50c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bbc100eb0 0x7f4bbc1989d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 --2- 172.21.15.16:0/1898446186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4bbc100500 0x7f4bbc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 >> 172.21.15.16:0/1898446186 conn(0x7f4bbc0fc5c0 msgr2=0x7f4bbc105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 shutdown_connections 2024-09-12T02:58:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:13.646+0000 7f4bc2a2a640 1 -- 172.21.15.16:0/1898446186 wait complete. 2024-09-12T02:58:13.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.49 MB/sec execute 435 sec latency 19053.891 ms 2024-09-12T02:58:13.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:13 smithi016 ceph-mon[30925]: pgmap v646: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-09-12T02:58:13.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:13 smithi033 ceph-mon[35092]: pgmap v646: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-09-12T02:58:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:13 smithi184 ceph-mon[35321]: pgmap v646: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-09-12T02:58:14.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:58:15.068 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.47 MB/sec execute 436 sec latency 20054.011 ms 2024-09-12T02:58:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:15.347 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:58:15.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:58:15.623 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:15 smithi016 ceph-mon[30925]: pgmap v647: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-09-12T02:58:15.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.45 MB/sec execute 437 sec latency 21054.109 ms 2024-09-12T02:58:15.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 -- 172.21.15.16:0/985984304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac102570 msgr2=0x7f6eac1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:15.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/985984304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac102570 0x7f6eac1029b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009900 tx=0x7f6ea002a480 comp rx=0 tx=0).stop 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 -- 172.21.15.16:0/985984304 shutdown_connections 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/985984304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102ef0 0x7f6eac10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/985984304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac102570 0x7f6eac1029b0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/985984304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac100e20 0x7f6eac101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.847+0000 7f6eb10b4640 1 -- 172.21.15.16:0/985984304 >> 172.21.15.16:0/985984304 conn(0x7f6eac0fc7d0 msgr2=0x7f6eac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.848+0000 7f6eb10b4640 1 -- 172.21.15.16:0/985984304 shutdown_connections 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.848+0000 7f6eb10b4640 1 -- 172.21.15.16:0/985984304 wait complete. 2024-09-12T02:58:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.848+0000 7f6eb10b4640 1 Processor -- start 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 -- start start 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102570 0x7f6eac198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac102ef0 0x7f6eac19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110fd0 con 0x7f6eac100e20 2024-09-12T02:58:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110e50 con 0x7f6eac102ef0 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eb10b4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac111150 con 0x7f6eac102570 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eab59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac102ef0 0x7f6eac19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57974/0 (socket says 172.21.15.16:57974) 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 -- 172.21.15.16:0/1239607032 learned_addr learned my addr 172.21.15.16:0/1239607032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaa59c640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102570 0x7f6eac198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 -- 172.21.15.16:0/1239607032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102570 msgr2=0x7f6eac198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102570 0x7f6eac198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:15.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 -- 172.21.15.16:0/1239607032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac102ef0 msgr2=0x7f6eac19cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac102ef0 0x7f6eac19cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.849+0000 7f6eaad9d640 1 -- 172.21.15.16:0/1239607032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac19d4b0 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6eaad9d640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c017070 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac19d7a0 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c009c30 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00f6b0 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.850+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1a5080 con 0x7f6eac100e20 2024-09-12T02:58:15.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.851+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac101200 con 0x7f6eac100e20 2024-09-12T02:58:15.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.852+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6e9c0109d0 con 0x7f6eac100e20 2024-09-12T02:58:15.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.853+0000 7f6e93fff640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 0x7f6e940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:15.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.853+0000 7f6eaa59c640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 0x7f6e940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.853+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6e9c0ceb70 con 0x7f6eac100e20 2024-09-12T02:58:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.854+0000 7f6eaa59c640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 0x7f6e940a5040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea0005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:15.860+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6e9c009de0 con 0x7f6eac100e20 2024-09-12T02:58:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:15.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:15 smithi033 ceph-mon[35092]: pgmap v647: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-09-12T02:58:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:15 smithi184 ceph-mon[35321]: pgmap v647: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-09-12T02:58:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.081+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eac10a0a0 con 0x7f6eac100e20 2024-09-12T02:58:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.082+0000 7f6e93fff640 1 -- 172.21.15.16:0/1239607032 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f6e9c09b490 con 0x7f6eac100e20 2024-09-12T02:58:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:58:16.084 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:58:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:58:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 msgr2=0x7f6e940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 0x7f6e940a5040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea0005b50 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 msgr2=0x7f6eac198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 shutdown_connections 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eac102ef0 0x7f6eac19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac102570 0x7f6eac198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e940a2b80 0x7f6e940a5040 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 --2- 172.21.15.16:0/1239607032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eac100e20 0x7f6eac198500 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 >> 172.21.15.16:0/1239607032 conn(0x7f6eac0fc7d0 msgr2=0x7f6eac06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 shutdown_connections 2024-09-12T02:58:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.084+0000 7f6eb10b4640 1 -- 172.21.15.16:0/1239607032 wait complete. 2024-09-12T02:58:16.729 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 38e0f169 --release --await' 2024-09-12T02:58:16.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 38e0f169 --release --await 2024-09-12T02:58:16.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.43 MB/sec execute 438 sec latency 22054.140 ms 2024-09-12T02:58:16.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:16 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1239607032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 -- 172.21.15.16:0/2275231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102570 msgr2=0x7f12281029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 --2- 172.21.15.16:0/2275231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102570 0x7f12281029b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1218009900 tx=0x7f121802a480 comp rx=0 tx=0).stop 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 -- 172.21.15.16:0/2275231958 shutdown_connections 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 --2- 172.21.15.16:0/2275231958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102ef0 0x7f122810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 --2- 172.21.15.16:0/2275231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102570 0x7f12281029b0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 --2- 172.21.15.16:0/2275231958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 0x7f1228101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.866+0000 7f122f386640 1 -- 172.21.15.16:0/2275231958 >> 172.21.15.16:0/2275231958 conn(0x7f12280fc7d0 msgr2=0x7f12280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.867+0000 7f122f386640 1 -- 172.21.15.16:0/2275231958 shutdown_connections 2024-09-12T02:58:16.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.867+0000 7f122f386640 1 -- 172.21.15.16:0/2275231958 wait complete. 2024-09-12T02:58:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.867+0000 7f122f386640 1 Processor -- start 2024-09-12T02:58:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 -- start start 2024-09-12T02:58:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 0x7f12281992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102570 0x7f1228199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1228110f50 con 0x7f1228102ef0 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1228110dd0 con 0x7f1228100e20 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122f386640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f12281110d0 con 0x7f1228102570 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122d923640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122d923640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58002/0 (socket says 172.21.15.16:58002) 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122d923640 1 -- 172.21.15.16:0/3062763399 learned_addr learned my addr 172.21.15.16:0/3062763399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122d122640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 0x7f12281992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.868+0000 7f122c921640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102570 0x7f1228199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 -- 172.21.15.16:0/3062763399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102570 msgr2=0x7f1228199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102570 0x7f1228199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 -- 172.21.15.16:0/3062763399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 msgr2=0x7f12281992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 0x7f12281992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 -- 172.21.15.16:0/3062763399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12281a4880 con 0x7f1228102ef0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122d923640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f122000eaa0 tx=0x7f122000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f121e7fc640 1 -- 172.21.15.16:0/3062763399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220004d20 con 0x7f1228102ef0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f121e7fc640 1 -- 172.21.15.16:0/3062763399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1220005020 con 0x7f1228102ef0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12281a4a50 con 0x7f1228102ef0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f121e7fc640 1 -- 172.21.15.16:0/3062763399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200116d0 con 0x7f1228102ef0 2024-09-12T02:58:16.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.869+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281a4f60 con 0x7f1228102ef0 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 Processor -- start 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f121e7fc640 1 -- 172.21.15.16:0/3062763399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f12200051d0 con 0x7f1228102ef0 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 -- start start 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805631f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x55980563f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559805597280 con 0x55980563cd40 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55980553eb10 con 0x559805632480 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.872+0000 7f123175f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559805593240 con 0x559805631b60 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122c921640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805631f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38090/0 (socket says 172.21.15.16:38090) 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122c921640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805631f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 -- 172.21.15.16:0/1408304507 learned_addr learned my addr 172.21.15.16:0/1408304507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122c921640 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x55980563f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 -- 172.21.15.16:0/1408304507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 msgr2=0x559805631f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805631f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 -- 172.21.15.16:0/1408304507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 msgr2=0x55980563f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x55980563f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 -- 172.21.15.16:0/1408304507 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980554b1f0 con 0x559805632480 2024-09-12T02:58:16.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f121e7fc640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 0x7f12140a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122c921640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 0x7f12140a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f121e7fc640 1 -- 172.21.15.16:0/3062763399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f12200cf190 con 0x7f1228102ef0 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.873+0000 7f122d122640 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f122400ad00 tx=0x7f122400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=733083959f5d481d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f12137fe640 1 -- 172.21.15.16:0/1408304507 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122401d070 con 0x559805632480 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f12137fe640 1 -- 172.21.15.16:0/1408304507 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224018b20 con 0x559805632480 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f12137fe640 1 -- 172.21.15.16:0/1408304507 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224010b60 con 0x559805632480 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f123175f740 1 -- 172.21.15.16:0/1408304507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 msgr2=0x55980563c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f122c921640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 0x7f12140a50c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f122806aac0 tx=0x7f121802a480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.874+0000 7f123175f740 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f122400ad00 tx=0x7f122400f790 comp rx=0 tx=0).stop 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- 172.21.15.16:0/1408304507 shutdown_connections 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x55980563f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805632480 0x55980563c800 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- 172.21.15.16:0/1408304507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805631f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- 172.21.15.16:0/1408304507 >> 172.21.15.16:0/1408304507 conn(0x55980562e0f0 msgr2=0x55980562e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- 172.21.15.16:0/1408304507 shutdown_connections 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- 172.21.15.16:0/1408304507 wait complete. 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- start start 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 Processor -- start 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805639bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 0x55980563a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x55980563a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55980563f910 con 0x559805632480 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.875+0000 7f123175f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559805551f10 con 0x55980563cd40 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f123175f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55980563fa90 con 0x559805631b60 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f123175f740 10 client.0 _ll_register_callbacks cb 0x55980551aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:16.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122c921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 0x55980563a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122c921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 0x55980563a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58028/0 (socket says 172.21.15.16:58028) 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122c921640 1 -- 172.21.15.16:0/1011926889 learned_addr learned my addr 172.21.15.16:0/1011926889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x55980563a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d122640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805639bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 -- 172.21.15.16:0/1011926889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 msgr2=0x559805639bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805639bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 -- 172.21.15.16:0/1011926889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 msgr2=0x55980563a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 0x55980563a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 -- 172.21.15.16:0/1011926889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598053086a0 con 0x55980563cd40 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f122d923640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x55980563a630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f12200cccf0 tx=0x7f122009c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77e649c58084636 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f1210ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122001f040 con 0x55980563cd40 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1224018b20 con 0x55980563cd40 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.876+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f122009ae50 con 0x55980563cd40 2024-09-12T02:58:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f122009fdb0 con 0x55980563cd40 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 10 client.27395 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55980564b4f0 con 0x55980563cd40 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224018b20 con 0x55980563cd40 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f122000f040 con 0x55980563cd40 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 10 client.27395 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 10 client.27395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 20 client.27395 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55980564c900 0x55980564ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 4 client.27395 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598055a47f0 con 0x55980564c900 2024-09-12T02:58:16.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.877+0000 7f122d122640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55980564c900 0x55980564ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.878+0000 7f1210ff9640 10 client.27395 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:16.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.878+0000 7f122d122640 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55980564c900 0x55980564ed90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba2afeca50d21230 server_cookie=67c330c0e0604bd2 in_seq=0 out_seq=0 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f1210ff9640 1 -- 172.21.15.16:0/1011926889 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5598055a47f0 con 0x55980564c900 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f1210ff9640 10 client.27395 handle_command_reply: tid=0 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 client.27395 shutdown 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 msgr2=0x55980563a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x55980563a630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f12200cccf0 tx=0x7f122009c100 comp rx=0 tx=0).stop 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 shutdown_connections 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55980564c900 0x55980564ed90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x55980563a630 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805632480 0x55980563a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 --2- 172.21.15.16:0/1011926889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805631b60 0x559805639bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 >> 172.21.15.16:0/1011926889 conn(0x55980562e0f0 msgr2=0x55980562d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f1210ff9640 0 client.27395 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 shutdown_connections 2024-09-12T02:58:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 1 -- 172.21.15.16:0/1011926889 wait complete. 2024-09-12T02:58:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.879+0000 7f123175f740 20 client.27395 trim_cache size 0 max 16384 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 Processor -- start 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 -- start start 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805631b60 0x5598056345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x55980563ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ad530 con 0x55980563cd40 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12281af070 con 0x559805631b60 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f123175f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f12281adc00 con 0x559805632480 2024-09-12T02:58:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58040/0 (socket says 172.21.15.16:58040) 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 -- 172.21.15.16:0/3558121616 learned_addr learned my addr 172.21.15.16:0/3558121616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d122640 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805631b60 0x5598056345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122c921640 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x55980563ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 -- 172.21.15.16:0/3558121616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 msgr2=0x55980563ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x55980563ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 -- 172.21.15.16:0/3558121616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805631b60 msgr2=0x5598056345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805631b60 0x5598056345c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.897+0000 7f122d923640 1 -- 172.21.15.16:0/3558121616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980562df20 con 0x55980563cd40 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.898+0000 7f122d923640 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f12200cd010 tx=0x7f1220005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4331ea9e777e38d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.898+0000 7f1212ffd640 1 -- 172.21.15.16:0/3558121616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122001f070 con 0x55980563cd40 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.898+0000 7f1212ffd640 1 -- 172.21.15.16:0/3558121616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12200a4670 con 0x55980563cd40 2024-09-12T02:58:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.898+0000 7f1212ffd640 1 -- 172.21.15.16:0/3558121616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122000caf0 con 0x55980563cd40 2024-09-12T02:58:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- 172.21.15.16:0/3558121616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 msgr2=0x559805654660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f12200cd010 tx=0x7f1220005b10 comp rx=0 tx=0).stop 2024-09-12T02:58:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- 172.21.15.16:0/3558121616 shutdown_connections 2024-09-12T02:58:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55980563cd40 0x559805654660 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x55980563ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 --2- 172.21.15.16:0/3558121616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559805631b60 0x5598056345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- 172.21.15.16:0/3558121616 >> 172.21.15.16:0/3558121616 conn(0x55980562e0f0 msgr2=0x55980563b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- 172.21.15.16:0/3558121616 shutdown_connections 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- 172.21.15.16:0/3558121616 wait complete. 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 -- start start 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.899+0000 7f123175f740 1 Processor -- start 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x559805645220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x5598056b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ae9a0 con 0x559805631b60 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ae2d0 con 0x55980563cd40 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ab510 con 0x559805632480 2024-09-12T02:58:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f123175f740 10 client.0 _ll_register_callbacks cb 0x7f1220004f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58050/0 (socket says 172.21.15.16:58050) 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 -- 172.21.15.16:0/588629310 learned_addr learned my addr 172.21.15.16:0/588629310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122c921640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x559805645220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 -- 172.21.15.16:0/588629310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 msgr2=0x559805645220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x559805645220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 -- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 msgr2=0x5598056b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x5598056b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 -- 172.21.15.16:0/588629310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559805645810 con 0x559805631b60 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f122d122640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x55980562ef50 tx=0x7f1224010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1982af1b3f620c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f1210ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.900+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122401d040 con 0x559805631b60 2024-09-12T02:58:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1224005240 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12200a4670 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f122400a730 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 10 client.17970 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5598056bbf60 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12200a4670 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1224018440 con 0x559805631b60 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 10 client.17970 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 20 client.17970 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5598056bc400 0x5598056be840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 4 client.17970 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.901+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55980559ed00 con 0x5598056bc400 2024-09-12T02:58:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.902+0000 7f122c921640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5598056bc400 0x5598056be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.902+0000 7f1210ff9640 10 client.17970 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.902+0000 7f122c921640 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5598056bc400 0x5598056be840 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ede66315a9698c7 server_cookie=e303a3047b7c27cc in_seq=0 out_seq=0 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f1210ff9640 1 -- 172.21.15.16:0/588629310 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55980559ed00 con 0x5598056bc400 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f1210ff9640 10 client.17970 handle_command_reply: tid=0 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 client.17970 shutdown 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 msgr2=0x559805644ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x55980562ef50 tx=0x7f1224010660 comp rx=0 tx=0).stop 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 shutdown_connections 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5598056bc400 0x5598056be840 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55980563cd40 0x5598056b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559805632480 0x559805645220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f1210ff9640 0 client.17970 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 --2- 172.21.15.16:0/588629310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559805631b60 0x559805644ce0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 >> 172.21.15.16:0/588629310 conn(0x55980562e0f0 msgr2=0x55980562d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.903+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 shutdown_connections 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f123175f740 1 -- 172.21.15.16:0/588629310 wait complete. 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f123175f740 20 client.17970 trim_cache size 0 max 16384 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:58:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:58:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d805c2aa-e5cf-4cdc-96cb-24e6b245dfa9", 2024-09-12T02:58:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:58:16.910 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:58:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 msgr2=0x7f12140a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 0x7f12140a50c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f122806aac0 tx=0x7f121802a480 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 msgr2=0x7f12281a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f122000eaa0 tx=0x7f122000eea0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 shutdown_connections 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f12140a2c00 0x7f12140a50c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228102ef0 0x7f12281a4340 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1228102570 0x7f1228199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 --2- 172.21.15.16:0/3062763399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1228100e20 0x7f12281992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 >> 172.21.15.16:0/3062763399 conn(0x7f12280fc7d0 msgr2=0x7f1228106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 shutdown_connections 2024-09-12T02:58:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:16.904+0000 7f122f386640 1 -- 172.21.15.16:0/3062763399 wait complete. 2024-09-12T02:58:16.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:16 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1239607032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:16 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1239607032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:17.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:58:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:17.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 981157 9.41 MB/sec execute 439 sec latency 23054.172 ms 2024-09-12T02:58:17.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:17 smithi016 ceph-mon[30925]: pgmap v648: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 520 KiB/s wr, 35 op/s 2024-09-12T02:58:17.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:17 smithi033 ceph-mon[35092]: pgmap v648: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 520 KiB/s wr, 35 op/s 2024-09-12T02:58:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:17 smithi184 ceph-mon[35321]: pgmap v648: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 520 KiB/s wr, 35 op/s 2024-09-12T02:58:18.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.246+0000 7fe42e94f640 1 -- 172.21.15.16:0/876791222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 msgr2=0x7fe428074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.246+0000 7fe42e94f640 1 --2- 172.21.15.16:0/876791222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe428074a00 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe418009900 tx=0x7fe41802a480 comp rx=0 tx=0).stop 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 -- 172.21.15.16:0/876791222 shutdown_connections 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 --2- 172.21.15.16:0/876791222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 0x7fe428110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 --2- 172.21.15.16:0/876791222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 0x7fe42810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 --2- 172.21.15.16:0/876791222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe428074a00 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 -- 172.21.15.16:0/876791222 >> 172.21.15.16:0/876791222 conn(0x7fe4280fd8c0 msgr2=0x7fe4280ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 -- 172.21.15.16:0/876791222 shutdown_connections 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 -- 172.21.15.16:0/876791222 wait complete. 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 Processor -- start 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.247+0000 7fe42e94f640 1 -- start start 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 0x7fe42819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 0x7fe42819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe428114170 con 0x7fe428074620 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe428113ff0 con 0x7fe428074f40 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281142f0 con 0x7fe42810e3c0 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58076/0 (socket says 172.21.15.16:58076) 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 -- 172.21.15.16:0/2188415333 learned_addr learned my addr 172.21.15.16:0/2188415333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42ceec640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 0x7fe42819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe4277fe640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 0x7fe42819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 -- 172.21.15.16:0/2188415333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 msgr2=0x7fe42819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 0x7fe42819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 -- 172.21.15.16:0/2188415333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 msgr2=0x7fe42819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 0x7fe42819ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 -- 172.21.15.16:0/2188415333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42819f620 con 0x7fe428074620 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe427fff640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe418006010 tx=0x7fe4180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe4257fa640 1 -- 172.21.15.16:0/2188415333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41803f030 con 0x7fe428074620 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42819f8b0 con 0x7fe428074620 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe4257fa640 1 -- 172.21.15.16:0/2188415333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe418003ce0 con 0x7fe428074620 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe4257fa640 1 -- 172.21.15.16:0/2188415333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418035070 con 0x7fe428074620 2024-09-12T02:58:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.248+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4281a71f0 con 0x7fe428074620 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.251+0000 7fe4257fa640 1 -- 172.21.15.16:0/2188415333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe418042030 con 0x7fe428074620 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.251+0000 7fe430d28740 1 Processor -- start 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.251+0000 7fe430d28740 1 -- start start 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb39440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb43d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfa9e620 con 0x55cfbfb44240 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfa45e50 con 0x55cfbfb39980 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe430d28740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfa9a5e0 con 0x55cfbfb39060 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58086/0 (socket says 172.21.15.16:58086) 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 -- 172.21.15.16:0/2339738262 learned_addr learned my addr 172.21.15.16:0/2339738262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe427fff640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb43d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe42ceec640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb39440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 -- 172.21.15.16:0/2339738262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 msgr2=0x55cfbfb39440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb39440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 -- 172.21.15.16:0/2339738262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 msgr2=0x55cfbfb43d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb43d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.252+0000 7fe4277fe640 1 -- 172.21.15.16:0/2339738262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbfa36d40 con 0x55cfbfb44240 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe4257fa640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 0x7fe4100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe427fff640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb43d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe4277fe640 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009b60 tx=0x7fe41c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3dff3557878a02 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe42ceec640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 0x7fe4100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe4257fa640 1 -- 172.21.15.16:0/2188415333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe4180ef710 con 0x7fe428074620 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe40e7fc640 1 -- 172.21.15.16:0/2339738262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c016070 con 0x55cfbfb44240 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe40e7fc640 1 -- 172.21.15.16:0/2339738262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c009e50 con 0x55cfbfb44240 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.253+0000 7fe40e7fc640 1 -- 172.21.15.16:0/2339738262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c011470 con 0x55cfbfb44240 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- 172.21.15.16:0/2339738262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 msgr2=0x55cfbfb46630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe42ceec640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 0x7fe4100a5040 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe420005fd0 tx=0x7fe4200095a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009b60 tx=0x7fe41c00e6c0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- 172.21.15.16:0/2339738262 shutdown_connections 2024-09-12T02:58:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb46630 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb43d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 --2- 172.21.15.16:0/2339738262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb39440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- 172.21.15.16:0/2339738262 >> 172.21.15.16:0/2339738262 conn(0x55cfbfb355f0 msgr2=0x55cfbfb359d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- 172.21.15.16:0/2339738262 shutdown_connections 2024-09-12T02:58:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- 172.21.15.16:0/2339738262 wait complete. 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.254+0000 7fe430d28740 1 -- start start 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 Processor -- start 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39980 0x55cfbfb41640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb44240 0x55cfbfb41b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfb46e10 con 0x55cfbfb39060 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfa26f10 con 0x55cfbfb44240 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cfbfb46f90 con 0x55cfbfb39980 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe430d28740 10 client.0 _ll_register_callbacks cb 0x55cfbfa4adb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe427fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58100/0 (socket says 172.21.15.16:58100) 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe427fff640 1 -- 172.21.15.16:0/1968569844 learned_addr learned my addr 172.21.15.16:0/1968569844 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe4277fe640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39980 0x55cfbfb41640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.255+0000 7fe42ceec640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb44240 0x55cfbfb41b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 -- 172.21.15.16:0/1968569844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39980 msgr2=0x55cfbfb41640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39980 0x55cfbfb41640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 -- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb44240 msgr2=0x55cfbfb41b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb44240 0x55cfbfb41b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 -- 172.21.15.16:0/1968569844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbf7ad580 con 0x55cfbfb39060 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe427fff640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe418004650 tx=0x7fe4180095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff60d412de86de95 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe4077fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418049070 con 0x55cfbfb39060 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe4180c4820 con 0x55cfbfb39060 2024-09-12T02:58:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41c009e50 con 0x55cfbfb39060 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe4180bbe10 con 0x55cfbfb39060 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe430d28740 10 client.17982 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.256+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cfbf877960 con 0x55cfbfb39060 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c009e50 con 0x55cfbfb39060 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe4180efbc0 con 0x55cfbfb39060 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 10 client.17982 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 10 client.17982 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 20 client.17982 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfb560b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 4 client.17982 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfbfaabb90 con 0x55cfbfb53c60 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.257+0000 7fe4277fe640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfb560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe4277fe640 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfb560b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4ed1ef30390a07 server_cookie=574e6780dba24e71 in_seq=0 out_seq=0 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe4077fe640 10 client.17982 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe4077fe640 1 -- 172.21.15.16:0/1968569844 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cfbfaabb90 con 0x55cfbfb53c60 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe4077fe640 10 client.17982 handle_command_reply: tid=0 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe430d28740 1 client.17982 shutdown 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.258+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 msgr2=0x55cfbfb41100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe418004650 tx=0x7fe4180095e0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 shutdown_connections 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfb560b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe4077fe640 0 client.17982 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb44240 0x55cfbfb41b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39980 0x55cfbfb41640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 --2- 172.21.15.16:0/1968569844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39060 0x55cfbfb41100 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 >> 172.21.15.16:0/1968569844 conn(0x55cfbfb355f0 msgr2=0x55cfbfb3b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 shutdown_connections 2024-09-12T02:58:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 1 -- 172.21.15.16:0/1968569844 wait complete. 2024-09-12T02:58:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.259+0000 7fe430d28740 20 client.17982 trim_cache size 0 max 16384 2024-09-12T02:58:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.277+0000 7fe430d28740 1 Processor -- start 2024-09-12T02:58:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.277+0000 7fe430d28740 1 -- start start 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb3bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb34640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281af6c0 con 0x55cfbfb44240 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281b1200 con 0x55cfbfb39980 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe430d28740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281afd90 con 0x55cfbfb39060 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58106/0 (socket says 172.21.15.16:58106) 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 -- 172.21.15.16:0/3662349474 learned_addr learned my addr 172.21.15.16:0/3662349474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe4277fe640 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb34640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe427fff640 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb3bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 -- 172.21.15.16:0/3662349474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 msgr2=0x55cfbfb3bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb3bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 -- 172.21.15.16:0/3662349474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 msgr2=0x55cfbfb34640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb34640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 -- 172.21.15.16:0/3662349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbfb362c0 con 0x55cfbfb44240 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.278+0000 7fe42ceec640 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe420003b00 tx=0x7fe420015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78a967c3914a312 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.279+0000 7fe40dffb640 1 -- 172.21.15.16:0/3662349474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420029470 con 0x55cfbfb44240 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.279+0000 7fe40dffb640 1 -- 172.21.15.16:0/3662349474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe420024070 con 0x55cfbfb44240 2024-09-12T02:58:18.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.279+0000 7fe40dffb640 1 -- 172.21.15.16:0/3662349474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4200209d0 con 0x55cfbfb44240 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.279+0000 7fe430d28740 1 -- 172.21.15.16:0/3662349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 msgr2=0x55cfbfb5b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.279+0000 7fe430d28740 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe420003b00 tx=0x7fe420015000 comp rx=0 tx=0).stop 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- 172.21.15.16:0/3662349474 shutdown_connections 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb44240 0x55cfbfb5b990 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39980 0x55cfbfb34640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- 172.21.15.16:0/3662349474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb39060 0x55cfbfb3bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- 172.21.15.16:0/3662349474 >> 172.21.15.16:0/3662349474 conn(0x55cfbfb355f0 msgr2=0x55cfbfb33f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- 172.21.15.16:0/3662349474 shutdown_connections 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- 172.21.15.16:0/3662349474 wait complete. 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- start start 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 Processor -- start 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39060 0x55cfbfb4c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb44240 0x55cfbfb491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.280+0000 7fe430d28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281b0b30 con 0x55cfbfb39980 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe430d28740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281b0460 con 0x55cfbfb39060 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe430d28740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281ad6a0 con 0x55cfbfb44240 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe430d28740 10 client.0 _ll_register_callbacks cb 0x7fe4180f52f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58112/0 (socket says 172.21.15.16:58112) 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 -- 172.21.15.16:0/4205848091 learned_addr learned my addr 172.21.15.16:0/4205848091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe42ceec640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb44240 0x55cfbfb491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe427fff640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39060 0x55cfbfb4c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 -- 172.21.15.16:0/4205848091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb44240 msgr2=0x55cfbfb491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb44240 0x55cfbfb491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 -- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39060 msgr2=0x55cfbfb4c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39060 0x55cfbfb4c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 -- 172.21.15.16:0/4205848091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbfb49990 con 0x55cfbfb39980 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.281+0000 7fe4277fe640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x55cfbfb36970 tx=0x7fe41c0094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99554c95f8db4e2f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe4077fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c016040 con 0x55cfbfb39980 2024-09-12T02:58:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe41c031480 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe420024070 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe41c009e10 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe430d28740 10 client.17988 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cfbfbbf490 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.282+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe420024070 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe41c00f040 con 0x55cfbfb39980 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 10 client.17988 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 10 client.17988 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 20 client.17988 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfbc1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 4 client.17988 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "38e0f169", "release": true, "await": true}] 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "38e0f169", "release": true, "await": true}) -- 0x55cfbfaa60a0 con 0x55cfbfb53c60 2024-09-12T02:58:18.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe427fff640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfbc1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe427fff640 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfbc1af0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16c06b930aabaa7b server_cookie=2c65c5b0e9956901 in_seq=0 out_seq=0 2024-09-12T02:58:18.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.283+0000 7fe4077fe640 10 client.17988 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe4077fe640 1 -- 172.21.15.16:0/4205848091 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cfbfaa60a0 con 0x55cfbfb53c60 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe4077fe640 10 client.17988 handle_command_reply: tid=0 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 client.17988 shutdown 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 msgr2=0x55cfbfb4c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x55cfbfb36970 tx=0x7fe41c0094f0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 shutdown_connections 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cfbfb53c60 0x55cfbfbc1af0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe4077fe640 0 client.17988 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cfbfb44240 0x55cfbfb491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfbfb39980 0x55cfbfb4c900 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 --2- 172.21.15.16:0/4205848091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cfbfb39060 0x55cfbfb4c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 >> 172.21.15.16:0/4205848091 conn(0x55cfbfb355f0 msgr2=0x55cfbfb34750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 shutdown_connections 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.294+0000 7fe430d28740 1 -- 172.21.15.16:0/4205848091 wait complete. 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe430d28740 20 client.17988 trim_cache size 0 max 16384 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 44, 2024-09-12T02:58:18.297 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "38e0f169": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 44, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 142.6, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:58:18.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:18.299 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 msgr2=0x7fe4100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 0x7fe4100a5040 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe420005fd0 tx=0x7fe4200095a0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 msgr2=0x7fe42819a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe418006010 tx=0x7fe4180042f0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 shutdown_connections 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe42810e3c0 0x7fe42819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe428074f40 0x7fe42819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2b80 0x7fe4100a5040 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 --2- 172.21.15.16:0/2188415333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe428074620 0x7fe42819a6c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 >> 172.21.15.16:0/2188415333 conn(0x7fe4280fd8c0 msgr2=0x7fe4280fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.295+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 shutdown_connections 2024-09-12T02:58:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:18.296+0000 7fe42e94f640 1 -- 172.21.15.16:0/2188415333 wait complete. 2024-09-12T02:58:18.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 982649 9.40 MB/sec execute 440 sec latency 23614.054 ms 2024-09-12T02:58:18.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:58:18.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:18.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:58:18.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:58:18.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:18.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:18.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:58:18.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:18.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:58:18.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:58:18.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:18.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:58:19.053 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '38e0f169', total seconds elapsed: 27.1 2024-09-12T02:58:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 -- 172.21.15.16:0/227312181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4101de0 msgr2=0x7f44d41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 --2- 172.21.15.16:0/227312181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4101de0 0x7f44d41021c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009900 tx=0x7f44c802a410 comp rx=0 tx=0).stop 2024-09-12T02:58:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 -- 172.21.15.16:0/227312181 shutdown_connections 2024-09-12T02:58:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 --2- 172.21.15.16:0/227312181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4106f10 0x7f44d410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 --2- 172.21.15.16:0/227312181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4102790 0x7f44d41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 --2- 172.21.15.16:0/227312181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4101de0 0x7f44d41021c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 -- 172.21.15.16:0/227312181 >> 172.21.15.16:0/227312181 conn(0x7f44d40fd8c0 msgr2=0x7f44d40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 -- 172.21.15.16:0/227312181 shutdown_connections 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.489+0000 7f44daedc640 1 -- 172.21.15.16:0/227312181 wait complete. 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 Processor -- start 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 -- start start 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4101de0 0x7f44d4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 0x7f44d419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4111f30 con 0x7f44d4102790 2024-09-12T02:58:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4111db0 con 0x7f44d4106f10 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44daedc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41120b0 con 0x7f44d4101de0 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4101de0 0x7f44d4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58122/0 (socket says 172.21.15.16:58122) 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 -- 172.21.15.16:0/3231751491 learned_addr learned my addr 172.21.15.16:0/3231751491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d9479640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 0x7f44d419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 -- 172.21.15.16:0/3231751491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4101de0 msgr2=0x7f44d4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4101de0 0x7f44d4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 -- 172.21.15.16:0/3231751491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 msgr2=0x7f44d419cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 0x7f44d419cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 -- 172.21.15.16:0/3231751491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d419d4c0 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d9479640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 0x7f44d419cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.490+0000 7f44d3fff640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9f0 tx=0x7f44c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.491+0000 7f44d1ffb640 1 -- 172.21.15.16:0/3231751491 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c000cd80 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.491+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d419d7b0 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.491+0000 7f44d1ffb640 1 -- 172.21.15.16:0/3231751491 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c0010910 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.491+0000 7f44d1ffb640 1 -- 172.21.15.16:0/3231751491 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c00188f0 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.491+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d41a5090 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44d1ffb640 1 -- 172.21.15.16:0/3231751491 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f44c0010ac0 con 0x7f44d4102790 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 Processor -- start 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 -- start start 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 0x5606bb56d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb57ca90 0x5606bb57ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606bb4d32e0 con 0x5606bb56dca0 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5606bb47ace0 con 0x5606bb57ca90 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606bb4cf2a0 con 0x5606bb56d380 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.494+0000 7f44d9479640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 0x5606bb56d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58130/0 (socket says 172.21.15.16:58130) 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d9479640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 0x5606bb56d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47606/0 (socket says 172.21.15.16:47606) 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 -- 172.21.15.16:0/1692343088 learned_addr learned my addr 172.21.15.16:0/1692343088 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb57ca90 0x5606bb57ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 -- 172.21.15.16:0/1692343088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 msgr2=0x5606bb56d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 0x5606bb56d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 -- 172.21.15.16:0/1692343088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb57ca90 msgr2=0x5606bb57ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb57ca90 0x5606bb57ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 -- 172.21.15.16:0/1692343088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606bb45f440 con 0x5606bb56dca0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d1ffb640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 0x7f44c40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009900 tx=0x7f44c8005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913ecdac9f6919e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d9479640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 0x7f44c40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44d1ffb640 1 -- 172.21.15.16:0/3231751491 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44c00cf470 con 0x7f44d4102790 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44beffd640 1 -- 172.21.15.16:0/1692343088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8040070 con 0x5606bb56dca0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44beffd640 1 -- 172.21.15.16:0/1692343088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c8031870 con 0x5606bb56dca0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.495+0000 7f44beffd640 1 -- 172.21.15.16:0/1692343088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8035040 con 0x5606bb56dca0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1692343088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 msgr2=0x5606bb57c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44d9479640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 0x7f44c40a5040 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f44cc005fd0 tx=0x7f44cc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009900 tx=0x7f44c8005be0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1692343088 shutdown_connections 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb57ca90 0x5606bb57ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb57c550 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1692343088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb56d380 0x5606bb56d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1692343088 >> 172.21.15.16:0/1692343088 conn(0x5606bb569a30 msgr2=0x5606bb569e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1692343088 shutdown_connections 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.496+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1692343088 wait complete. 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 -- start start 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 Processor -- start 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb57c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb579c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606bb57f660 con 0x5606bb56dca0 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5606bb48e050 con 0x5606bb56d380 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606bb57f7e0 con 0x5606bb57ca90 2024-09-12T02:58:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44dd2b5740 10 client.0 _ll_register_callbacks cb 0x5606bb4ce160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58140/0 (socket says 172.21.15.16:58140) 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 -- 172.21.15.16:0/1904603205 learned_addr learned my addr 172.21.15.16:0/1904603205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d9479640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb579c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb57c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 -- 172.21.15.16:0/1904603205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 msgr2=0x5606bb579c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb579c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 -- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 msgr2=0x5606bb57c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb57c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.497+0000 7f44d3fff640 1 -- 172.21.15.16:0/1904603205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606bb2443c0 con 0x5606bb56dca0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9c0 tx=0x7f44c009bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589f01527ba4fad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c00cf2d0 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44c009b130 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c8031870 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f44c009c3e0 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 10 client.18000 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5606bb58b070 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c8031870 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f44c0097a30 con 0x5606bb56dca0 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 10 client.18000 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 10 client.18000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.498+0000 7f44dd2b5740 20 client.18000 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb58c480 0x5606bb58e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44dd2b5740 4 client.18000 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606bb4e0850 con 0x5606bb58c480 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb58c480 0x5606bb58e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb58c480 0x5606bb58e910 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ff0ae9804fffba server_cookie=36e2359b1ed28b66 in_seq=0 out_seq=0 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.499+0000 7f44b3fff640 10 client.18000 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44b3fff640 1 -- 172.21.15.16:0/1904603205 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5606bb4e0850 con 0x5606bb58c480 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44b3fff640 10 client.18000 handle_command_reply: tid=0 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 client.18000 shutdown 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 msgr2=0x5606bb5796a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9c0 tx=0x7f44c009bab0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 shutdown_connections 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb58c480 0x5606bb58e910 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb579c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb5796a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44b3fff640 0 client.18000 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1904603205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb57c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 >> 172.21.15.16:0/1904603205 conn(0x5606bb569a30 msgr2=0x5606bb56f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 shutdown_connections 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1904603205 wait complete. 2024-09-12T02:58:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.500+0000 7f44dd2b5740 20 client.18000 trim_cache size 0 max 16384 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 Processor -- start 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 -- start start 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 0x5606bb594230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb594770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41ab2c0 con 0x5606bb56d380 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41ace00 con 0x5606bb56dca0 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41ab990 con 0x5606bb57ca90 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58156/0 (socket says 172.21.15.16:58156) 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 -- 172.21.15.16:0/1743036676 learned_addr learned my addr 172.21.15.16:0/1743036676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d9479640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb594770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 0x5606bb594230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 -- 172.21.15.16:0/1743036676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 msgr2=0x5606bb594770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb594770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 -- 172.21.15.16:0/1743036676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 msgr2=0x5606bb594230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 0x5606bb594230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d8c78640 1 -- 172.21.15.16:0/1743036676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606bb58ef90 con 0x5606bb56d380 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.521+0000 7f44d3fff640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 0x5606bb594230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44d8c78640 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f44c80057c0 tx=0x7f44c8005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3e4f2f8c8890da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44be7fc640 1 -- 172.21.15.16:0/1743036676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8054070 con 0x5606bb56d380 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44be7fc640 1 -- 172.21.15.16:0/1743036676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c8004040 con 0x5606bb56d380 2024-09-12T02:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44be7fc640 1 -- 172.21.15.16:0/1743036676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8035040 con 0x5606bb56d380 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1743036676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 msgr2=0x5606bb588900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f44c80057c0 tx=0x7f44c8005420 comp rx=0 tx=0).stop 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1743036676 shutdown_connections 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb594770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56dca0 0x5606bb594230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/1743036676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56d380 0x5606bb588900 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.522+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1743036676 >> 172.21.15.16:0/1743036676 conn(0x5606bb569a30 msgr2=0x5606bb578790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1743036676 shutdown_connections 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- 172.21.15.16:0/1743036676 wait complete. 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- start start 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 Processor -- start 2024-09-12T02:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb580a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb5814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41ac730 con 0x5606bb56dca0 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41ac060 con 0x5606bb56d380 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41a92a0 con 0x5606bb57ca90 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.523+0000 7f44dd2b5740 10 client.0 _ll_register_callbacks cb 0x7f44c00d2c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb580a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58166/0 (socket says 172.21.15.16:58166) 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d3fff640 1 -- 172.21.15.16:0/4193397392 learned_addr learned my addr 172.21.15.16:0/4193397392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d9479640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb5814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 -- 172.21.15.16:0/4193397392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 msgr2=0x5606bb5814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb5814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 -- 172.21.15.16:0/4193397392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 msgr2=0x5606bb580f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 -- 172.21.15.16:0/4193397392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606bb5f64f0 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d3fff640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44d8c78640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb580a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f44c802a930 tx=0x7f44c80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416be2021dd7a448 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c805d060 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44c80043f0 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c8004040 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f44c8054070 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.524+0000 7f44dd2b5740 10 client.27428 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5606bb5f7780 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c8004040 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f44c804e4f0 con 0x5606bb56d380 2024-09-12T02:58:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 10 client.27428 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 20 client.27428 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb5f7a00 0x5606bb5f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 4 client.27428 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606bb4dad60 con 0x5606bb5f7a00 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.525+0000 7f44d3fff640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb5f7a00 0x5606bb5f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44b3fff640 10 client.27428 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44d3fff640 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb5f7a00 0x5606bb5f9df0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454490159b8df45d server_cookie=3dc0aaf5712dce1b in_seq=0 out_seq=0 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44b3fff640 1 -- 172.21.15.16:0/4193397392 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5606bb4dad60 con 0x5606bb5f7a00 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44b3fff640 10 client.27428 handle_command_reply: tid=0 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 client.27428 shutdown 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 msgr2=0x5606bb580a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb580a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f44c802a930 tx=0x7f44c80035f0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 shutdown_connections 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606bb5f7a00 0x5606bb5f9df0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606bb57ca90 0x5606bb5814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606bb56dca0 0x5606bb580f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 --2- 172.21.15.16:0/4193397392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606bb56d380 0x5606bb580a50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44b3fff640 0 client.27428 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 >> 172.21.15.16:0/4193397392 conn(0x5606bb569a30 msgr2=0x5606bb57af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 shutdown_connections 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.526+0000 7f44dd2b5740 1 -- 172.21.15.16:0/4193397392 wait complete. 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44dd2b5740 20 client.27428 trim_cache size 0 max 16384 2024-09-12T02:58:19.532 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "d805c2aa-e5cf-4cdc-96cb-24e6b245dfa9": { 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d805c2aa-e5cf-4cdc-96cb-24e6b245dfa9", 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:58:19.533 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 msgr2=0x7f44c40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 0x7f44c40a5040 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f44cc005fd0 tx=0x7f44cc009500 comp rx=0 tx=0).stop 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 msgr2=0x7f44d4198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9f0 tx=0x7f44c000eeb0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 shutdown_connections 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d4106f10 0x7f44d419cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44c40a2b80 0x7f44c40a5040 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d4102790 0x7f44d4198a50 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 --2- 172.21.15.16:0/3231751491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44d4101de0 0x7f44d4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 >> 172.21.15.16:0/3231751491 conn(0x7f44d40fd8c0 msgr2=0x7f44d40fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 shutdown_connections 2024-09-12T02:58:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:19.527+0000 7f44daedc640 1 -- 172.21.15.16:0/3231751491 wait complete. 2024-09-12T02:58:19.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:19 smithi016 ceph-mon[30925]: pgmap v649: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 104 KiB/s wr, 21 op/s 2024-09-12T02:58:19.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 985087 9.41 MB/sec execute 441 sec latency 33.729 ms 2024-09-12T02:58:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:19 smithi033 ceph-mon[35092]: pgmap v649: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 104 KiB/s wr, 21 op/s 2024-09-12T02:58:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:19 smithi184 ceph-mon[35321]: pgmap v649: 97 pgs: 97 active+clean; 270 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 104 KiB/s wr, 21 op/s 2024-09-12T02:58:20.237 INFO:tasks.cephfs.filesystem:scrub status for tag:d805c2aa-e5cf-4cdc-96cb-24e6b245dfa9 - {'path': '/', 'tag': 'd805c2aa-e5cf-4cdc-96cb-24e6b245dfa9', 'options': 'recursive,force'} 2024-09-12T02:58:20.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 988121 9.43 MB/sec execute 442 sec latency 9.154 ms 2024-09-12T02:58:21.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 991388 9.45 MB/sec execute 443 sec latency 13.195 ms 2024-09-12T02:58:21.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:21 smithi016 ceph-mon[30925]: pgmap v650: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 103 KiB/s wr, 32 op/s 2024-09-12T02:58:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:21 smithi033 ceph-mon[35092]: pgmap v650: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 103 KiB/s wr, 32 op/s 2024-09-12T02:58:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:21 smithi184 ceph-mon[35321]: pgmap v650: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 103 KiB/s wr, 32 op/s 2024-09-12T02:58:22.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 993686 9.45 MB/sec execute 444 sec latency 17.093 ms 2024-09-12T02:58:23.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 995677 9.45 MB/sec execute 445 sec latency 13.940 ms 2024-09-12T02:58:23.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:23 smithi016 ceph-mon[30925]: pgmap v651: 97 pgs: 97 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 202 KiB/s wr, 29 op/s 2024-09-12T02:58:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:58:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:58:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:23 smithi033 ceph-mon[35092]: pgmap v651: 97 pgs: 97 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 202 KiB/s wr, 29 op/s 2024-09-12T02:58:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:58:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:58:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:23 smithi184 ceph-mon[35321]: pgmap v651: 97 pgs: 97 active+clean; 272 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 202 KiB/s wr, 29 op/s 2024-09-12T02:58:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:58:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:58:24.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 998598 9.47 MB/sec execute 446 sec latency 11.754 ms 2024-09-12T02:58:25.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1001251 9.48 MB/sec execute 447 sec latency 11.793 ms 2024-09-12T02:58:25.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:25 smithi016 ceph-mon[30925]: pgmap v652: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 536 KiB/s wr, 77 op/s 2024-09-12T02:58:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:25 smithi033 ceph-mon[35092]: pgmap v652: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 536 KiB/s wr, 77 op/s 2024-09-12T02:58:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:25 smithi184 ceph-mon[35321]: pgmap v652: 97 pgs: 97 active+clean; 271 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 536 KiB/s wr, 77 op/s 2024-09-12T02:58:26.737 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1003360 9.48 MB/sec execute 448 sec latency 11.487 ms 2024-09-12T02:58:27.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1005805 9.49 MB/sec execute 449 sec latency 27.386 ms 2024-09-12T02:58:27.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:27 smithi016 ceph-mon[30925]: pgmap v653: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 KiB/s wr, 122 op/s 2024-09-12T02:58:27.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:27 smithi033 ceph-mon[35092]: pgmap v653: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 KiB/s wr, 122 op/s 2024-09-12T02:58:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:27 smithi184 ceph-mon[35321]: pgmap v653: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 KiB/s wr, 122 op/s 2024-09-12T02:58:28.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1008775 9.51 MB/sec execute 450 sec latency 14.945 ms 2024-09-12T02:58:29.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1011218 9.51 MB/sec execute 451 sec latency 12.324 ms 2024-09-12T02:58:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:29 smithi016 ceph-mon[30925]: pgmap v654: 97 pgs: 97 active+clean; 255 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-09-12T02:58:29.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:29 smithi033 ceph-mon[35092]: pgmap v654: 97 pgs: 97 active+clean; 255 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-09-12T02:58:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:29 smithi184 ceph-mon[35321]: pgmap v654: 97 pgs: 97 active+clean; 255 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-09-12T02:58:30.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1013295 9.52 MB/sec execute 452 sec latency 12.221 ms 2024-09-12T02:58:31.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1016294 9.54 MB/sec execute 453 sec latency 7.883 ms 2024-09-12T02:58:31.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:31 smithi016 ceph-mon[30925]: pgmap v655: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-12T02:58:31.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:31 smithi033 ceph-mon[35092]: pgmap v655: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-12T02:58:32.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:31 smithi184 ceph-mon[35321]: pgmap v655: 97 pgs: 97 active+clean; 258 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-09-12T02:58:32.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1018979 9.54 MB/sec execute 454 sec latency 11.791 ms 2024-09-12T02:58:33.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1021290 9.55 MB/sec execute 455 sec latency 11.534 ms 2024-09-12T02:58:33.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:33 smithi016 ceph-mon[30925]: pgmap v656: 97 pgs: 97 active+clean; 259 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-12T02:58:33.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:33 smithi033 ceph-mon[35092]: pgmap v656: 97 pgs: 97 active+clean; 259 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-12T02:58:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:33 smithi184 ceph-mon[35321]: pgmap v656: 97 pgs: 97 active+clean; 259 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-12T02:58:34.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1023893 9.56 MB/sec execute 456 sec latency 14.942 ms 2024-09-12T02:58:35.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1026941 9.58 MB/sec execute 457 sec latency 11.985 ms 2024-09-12T02:58:35.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:35 smithi016 ceph-mon[30925]: pgmap v657: 97 pgs: 97 active+clean; 262 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-09-12T02:58:35.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:35 smithi033 ceph-mon[35092]: pgmap v657: 97 pgs: 97 active+clean; 262 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-09-12T02:58:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:35 smithi184 ceph-mon[35321]: pgmap v657: 97 pgs: 97 active+clean; 262 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-09-12T02:58:36.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1029860 9.59 MB/sec execute 458 sec latency 14.832 ms 2024-09-12T02:58:37.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1032567 9.59 MB/sec execute 459 sec latency 13.879 ms 2024-09-12T02:58:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:37 smithi016 ceph-mon[30925]: pgmap v658: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:58:37.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:37 smithi033 ceph-mon[35092]: pgmap v658: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:58:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:37 smithi184 ceph-mon[35321]: pgmap v658: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-09-12T02:58:38.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1034933 9.60 MB/sec execute 460 sec latency 12.578 ms 2024-09-12T02:58:39.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1037869 9.62 MB/sec execute 461 sec latency 10.091 ms 2024-09-12T02:58:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:39 smithi016 ceph-mon[30925]: pgmap v659: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-09-12T02:58:39.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:39 smithi033 ceph-mon[35092]: pgmap v659: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-09-12T02:58:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:39 smithi184 ceph-mon[35321]: pgmap v659: 97 pgs: 97 active+clean; 265 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 197 op/s 2024-09-12T02:58:40.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1041097 9.64 MB/sec execute 462 sec latency 18.325 ms 2024-09-12T02:58:41.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1043846 9.65 MB/sec execute 463 sec latency 16.359 ms 2024-09-12T02:58:41.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:41 smithi016 ceph-mon[30925]: pgmap v660: 97 pgs: 97 active+clean; 271 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2024-09-12T02:58:41.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:41 smithi033 ceph-mon[35092]: pgmap v660: 97 pgs: 97 active+clean; 271 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2024-09-12T02:58:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:41 smithi184 ceph-mon[35321]: pgmap v660: 97 pgs: 97 active+clean; 271 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2024-09-12T02:58:42.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1046190 9.65 MB/sec execute 464 sec latency 13.267 ms 2024-09-12T02:58:43.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1048675 9.66 MB/sec execute 465 sec latency 53.786 ms 2024-09-12T02:58:43.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:43 smithi016 ceph-mon[30925]: pgmap v661: 97 pgs: 97 active+clean; 272 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-12T02:58:43.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:43 smithi033 ceph-mon[35092]: pgmap v661: 97 pgs: 97 active+clean; 272 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-12T02:58:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:43 smithi184 ceph-mon[35321]: pgmap v661: 97 pgs: 97 active+clean; 272 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-12T02:58:44.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1051794 9.68 MB/sec execute 466 sec latency 11.538 ms 2024-09-12T02:58:45.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1054880 9.69 MB/sec execute 467 sec latency 10.692 ms 2024-09-12T02:58:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:45 smithi016 ceph-mon[30925]: pgmap v662: 97 pgs: 97 active+clean; 273 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-09-12T02:58:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:45 smithi033 ceph-mon[35092]: pgmap v662: 97 pgs: 97 active+clean; 273 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-09-12T02:58:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:45 smithi184 ceph-mon[35321]: pgmap v662: 97 pgs: 97 active+clean; 273 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-09-12T02:58:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1057433 9.70 MB/sec execute 468 sec latency 11.687 ms 2024-09-12T02:58:47.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1059755 9.71 MB/sec execute 469 sec latency 17.222 ms 2024-09-12T02:58:47.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:47 smithi016 ceph-mon[30925]: pgmap v663: 97 pgs: 97 active+clean; 278 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 248 op/s 2024-09-12T02:58:47.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:47 smithi033 ceph-mon[35092]: pgmap v663: 97 pgs: 97 active+clean; 278 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 248 op/s 2024-09-12T02:58:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:47 smithi184 ceph-mon[35321]: pgmap v663: 97 pgs: 97 active+clean; 278 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 248 op/s 2024-09-12T02:58:48.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1062814 9.72 MB/sec execute 470 sec latency 13.050 ms 2024-09-12T02:58:49.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1065976 9.74 MB/sec execute 471 sec latency 35.945 ms 2024-09-12T02:58:49.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:49 smithi016 ceph-mon[30925]: pgmap v664: 97 pgs: 97 active+clean; 279 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-12T02:58:49.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:49 smithi033 ceph-mon[35092]: pgmap v664: 97 pgs: 97 active+clean; 279 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-12T02:58:50.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:49 smithi184 ceph-mon[35321]: pgmap v664: 97 pgs: 97 active+clean; 279 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-12T02:58:50.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:58:50.510 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:50.738 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1068712 9.75 MB/sec execute 472 sec latency 12.118 ms 2024-09-12T02:58:51.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 -- 172.21.15.16:0/3746240675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 msgr2=0x7efd44101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 --2- 172.21.15.16:0/3746240675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44101200 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7efd40009900 tx=0x7efd4002a480 comp rx=0 tx=0).stop 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 -- 172.21.15.16:0/3746240675 shutdown_connections 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 --2- 172.21.15.16:0/3746240675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102ef0 0x7efd4410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 --2- 172.21.15.16:0/3746240675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102570 0x7efd441029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 --2- 172.21.15.16:0/3746240675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44101200 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 -- 172.21.15.16:0/3746240675 >> 172.21.15.16:0/3746240675 conn(0x7efd440fc7d0 msgr2=0x7efd440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 -- 172.21.15.16:0/3746240675 shutdown_connections 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.000+0000 7efd4b99f640 1 -- 172.21.15.16:0/3746240675 wait complete. 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 Processor -- start 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 -- start start 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 0x7efd44198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102ef0 0x7efd4419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd44110f80 con 0x7efd44100e20 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd44110e00 con 0x7efd44102570 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4b99f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd44111100 con 0x7efd44102ef0 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4973b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4973b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54512/0 (socket says 172.21.15.16:54512) 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd4973b640 1 -- 172.21.15.16:0/2547452251 learned_addr learned my addr 172.21.15.16:0/2547452251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd48f3a640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 0x7efd44198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.001+0000 7efd49f3c640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102ef0 0x7efd4419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 -- 172.21.15.16:0/2547452251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102ef0 msgr2=0x7efd4419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102ef0 0x7efd4419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 -- 172.21.15.16:0/2547452251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 msgr2=0x7efd44198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 0x7efd44198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 -- 172.21.15.16:0/2547452251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4419d4b0 con 0x7efd44100e20 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd48f3a640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 0x7efd44198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4973b640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7efd40006010 tx=0x7efd40003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd3a7fc640 1 -- 172.21.15.16:0/2547452251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4003f070 con 0x7efd44100e20 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd3a7fc640 1 -- 172.21.15.16:0/2547452251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd40031c30 con 0x7efd44100e20 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4419d740 con 0x7efd44100e20 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd3a7fc640 1 -- 172.21.15.16:0/2547452251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40035030 con 0x7efd44100e20 2024-09-12T02:58:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.002+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd441a5080 con 0x7efd44100e20 2024-09-12T02:58:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 Processor -- start 2024-09-12T02:58:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd3a7fc640 1 -- 172.21.15.16:0/2547452251 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efd40042070 con 0x7efd44100e20 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 -- start start 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07c7a90 0x5639a07c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639a071e2e0 con 0x5639a07c7a90 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639a06c5ce0 con 0x5639a07b83a0 2024-09-12T02:58:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4dd78740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5639a071a2a0 con 0x5639a07b8cc0 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd4973b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07c7a90 0x5639a07c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40648/0 (socket says 172.21.15.16:40648) 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.005+0000 7efd49f3c640 1 -- 172.21.15.16:0/2591320004 learned_addr learned my addr 172.21.15.16:0/2591320004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 -- 172.21.15.16:0/2591320004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 msgr2=0x5639a07c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 -- 172.21.15.16:0/2591320004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07c7a90 msgr2=0x5639a07c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07c7a90 0x5639a07c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 -- 172.21.15.16:0/2591320004 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a06aa440 con 0x5639a07b83a0 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd3a7fc640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 0x7efd300a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd49f3c640 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fd0 tx=0x7efd3c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55288c935da97b16 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd4973b640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 0x7efd300a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd2f7fe640 1 -- 172.21.15.16:0/2591320004 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c004d20 con 0x5639a07b83a0 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd3a7fc640 1 -- 172.21.15.16:0/2547452251 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd400ef610 con 0x7efd44100e20 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd2f7fe640 1 -- 172.21.15.16:0/2591320004 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3c004050 con 0x5639a07b83a0 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.006+0000 7efd2f7fe640 1 -- 172.21.15.16:0/2591320004 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c005260 con 0x5639a07b83a0 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- 172.21.15.16:0/2591320004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 msgr2=0x5639a07b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fd0 tx=0x7efd3c0118d0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4973b640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 0x7efd300a50c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7efd400ed490 tx=0x7efd400f5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- 172.21.15.16:0/2591320004 shutdown_connections 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07c7a90 0x5639a07c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 --2- 172.21.15.16:0/2591320004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07b8780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- 172.21.15.16:0/2591320004 >> 172.21.15.16:0/2591320004 conn(0x5639a07b4a30 msgr2=0x5639a07b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- 172.21.15.16:0/2591320004 shutdown_connections 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- 172.21.15.16:0/2591320004 wait complete. 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 -- start start 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.007+0000 7efd4dd78740 1 Processor -- start 2024-09-12T02:58:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639a07ca6c0 con 0x5639a07b8cc0 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639a06d9050 con 0x5639a07c7a90 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5639a07ca840 con 0x5639a07b83a0 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd4dd78740 10 client.0 _ll_register_callbacks cb 0x5639a0719160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54542/0 (socket says 172.21.15.16:54542) 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd48f3a640 1 -- 172.21.15.16:0/4245149047 learned_addr learned my addr 172.21.15.16:0/4245149047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33158/0 (socket says 172.21.15.16:33158) 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.008+0000 7efd49f3c640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 -- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 msgr2=0x5639a07c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 -- 172.21.15.16:0/4245149047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 msgr2=0x5639a07c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 -- 172.21.15.16:0/4245149047 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a048f3c0 con 0x5639a07b83a0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd48f3a640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd49f3c640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fa0 tx=0x7efd3c005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a02dc67cb9cfc6fb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c004610 con 0x5639a07b83a0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd3c004050 con 0x5639a07b83a0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd3c00f4e0 con 0x5639a07b83a0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd3c00f6e0 con 0x5639a07b83a0 2024-09-12T02:58:51.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd4dd78740 10 client.27334 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.009+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639a07d60f0 con 0x5639a07b83a0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3c004050 con 0x5639a07b83a0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd3c025040 con 0x5639a07b83a0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 10 client.27334 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 10 client.27334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 20 client.27334 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a07d74f0 0x5639a07d98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 4 client.27334 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639a072b850 con 0x5639a07d74f0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd48f3a640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a07d74f0 0x5639a07d98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd48f3a640 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a07d74f0 0x5639a07d98e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d4e08e8271f557c server_cookie=a00d25fefe40e86d in_seq=0 out_seq=0 2024-09-12T02:58:51.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.010+0000 7efd2cff9640 10 client.27334 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd2cff9640 1 -- 172.21.15.16:0/4245149047 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5639a072b850 con 0x5639a07d74f0 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd2cff9640 10 client.27334 handle_command_reply: tid=0 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 client.27334 shutdown 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 msgr2=0x5639a07c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009fa0 tx=0x7efd3c005bd0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 shutdown_connections 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a07d74f0 0x5639a07d98e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 --2- 172.21.15.16:0/4245149047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b83a0 0x5639a07c7420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 >> 172.21.15.16:0/4245149047 conn(0x5639a07b4a30 msgr2=0x5639a07ba790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd2cff9640 0 client.27334 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.011+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 shutdown_connections 2024-09-12T02:58:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.012+0000 7efd4dd78740 1 -- 172.21.15.16:0/4245149047 wait complete. 2024-09-12T02:58:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.012+0000 7efd4dd78740 20 client.27334 trim_cache size 0 max 16384 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 Processor -- start 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 -- start start 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 0x5639a07d3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07df5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ab3a0 con 0x5639a07b83a0 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd441acee0 con 0x5639a07c7a90 2024-09-12T02:58:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.030+0000 7efd4dd78740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd441aba70 con 0x5639a07b8cc0 2024-09-12T02:58:51.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 0x5639a07d3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33164/0 (socket says 172.21.15.16:33164) 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 -- 172.21.15.16:0/3223862721 learned_addr learned my addr 172.21.15.16:0/3223862721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd49f3c640 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07df5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 -- 172.21.15.16:0/3223862721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 msgr2=0x5639a07df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07df5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 -- 172.21.15.16:0/3223862721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 msgr2=0x5639a07d3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 0x5639a07d3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 -- 172.21.15.16:0/3223862721 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a07b9230 con 0x5639a07b8cc0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd49f3c640 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 0x5639a07d3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd48f3a640 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5639a07c5d10 tx=0x7efd34005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d80dd5df8887dae server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.031+0000 7efd2effd640 1 -- 172.21.15.16:0/3223862721 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3400fc80 con 0x5639a07b8cc0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.032+0000 7efd2effd640 1 -- 172.21.15.16:0/3223862721 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3400fe20 con 0x5639a07b8cc0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.032+0000 7efd2effd640 1 -- 172.21.15.16:0/3223862721 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34004dd0 con 0x5639a07b8cc0 2024-09-12T02:58:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.032+0000 7efd4dd78740 1 -- 172.21.15.16:0/3223862721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 msgr2=0x5639a07df070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.032+0000 7efd4dd78740 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5639a07c5d10 tx=0x7efd34005f90 comp rx=0 tx=0).stop 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 -- 172.21.15.16:0/3223862721 shutdown_connections 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07c7a90 0x5639a07df5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07b8cc0 0x5639a07df070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- 172.21.15.16:0/3223862721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b83a0 0x5639a07d3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 -- 172.21.15.16:0/3223862721 >> 172.21.15.16:0/3223862721 conn(0x5639a07b4a30 msgr2=0x5639a07bfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 -- 172.21.15.16:0/3223862721 shutdown_connections 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 -- 172.21.15.16:0/3223862721 wait complete. 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 -- start start 2024-09-12T02:58:51.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 Processor -- start 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.033+0000 7efd4dd78740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07c7a90 0x5639a07cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd4dd78740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ac810 con 0x5639a07b8cc0 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd4dd78740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ac140 con 0x5639a07b83a0 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd4dd78740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd441a93a0 con 0x5639a07c7a90 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd4dd78740 10 client.0 _ll_register_callbacks cb 0x7efd34009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:51.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd49f3c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54558/0 (socket says 172.21.15.16:54558) 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 -- 172.21.15.16:0/1724019744 learned_addr learned my addr 172.21.15.16:0/1724019744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd49f3c640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07c7a90 0x5639a07cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 -- 172.21.15.16:0/1724019744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07c7a90 msgr2=0x5639a07cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07c7a90 0x5639a07cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 -- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 msgr2=0x5639a07d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 -- 172.21.15.16:0/1724019744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a07cb800 con 0x5639a07b8cc0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd49f3c640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd48f3a640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7efd34000c00 tx=0x7efd34012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde9e542b5f0b06d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34037060 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efd34032470 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.034+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd3400fe20 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd3400fc80 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 10 client.18024 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639a08410a0 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3400fe20 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd34020400 con 0x5639a07b8cc0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 10 client.18024 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 10 client.18024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 20 client.18024 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a08416e0 0x5639a0843b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 4 client.18024 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639a0725d60 con 0x5639a08416e0 2024-09-12T02:58:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.035+0000 7efd49f3c640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a08416e0 0x5639a0843b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd2cff9640 10 client.18024 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd49f3c640 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a08416e0 0x5639a0843b00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad57f5261e165e5 server_cookie=d567ed87de9aacb4 in_seq=0 out_seq=0 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd2cff9640 1 -- 172.21.15.16:0/1724019744 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5639a0725d60 con 0x5639a08416e0 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd2cff9640 10 client.18024 handle_command_reply: tid=0 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 client.18024 shutdown 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 msgr2=0x5639a07caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7efd34000c00 tx=0x7efd34012980 comp rx=0 tx=0).stop 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 shutdown_connections 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5639a08416e0 0x5639a0843b00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639a07c7a90 0x5639a07cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639a07b8cc0 0x5639a07caa90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd2cff9640 0 client.18024 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 --2- 172.21.15.16:0/1724019744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639a07b83a0 0x5639a07d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 >> 172.21.15.16:0/1724019744 conn(0x5639a07b4a30 msgr2=0x5639a07c2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 shutdown_connections 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.036+0000 7efd4dd78740 1 -- 172.21.15.16:0/1724019744 wait complete. 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4dd78740 20 client.18024 trim_cache size 0 max 16384 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:58:51.042 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:58:51.043 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:58:51.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 msgr2=0x7efd300a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 0x7efd300a50c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7efd400ed490 tx=0x7efd400f5550 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 msgr2=0x7efd44198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7efd40006010 tx=0x7efd40003c10 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 shutdown_connections 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd44102ef0 0x7efd4419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd44102570 0x7efd44198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efd300a2c00 0x7efd300a50c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 --2- 172.21.15.16:0/2547452251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd44100e20 0x7efd44198500 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 >> 172.21.15.16:0/2547452251 conn(0x7efd440fc7d0 msgr2=0x7efd4406a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 shutdown_connections 2024-09-12T02:58:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:51.037+0000 7efd4b99f640 1 -- 172.21.15.16:0/2547452251 wait complete. 2024-09-12T02:58:51.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1070897 9.75 MB/sec execute 473 sec latency 49.217 ms 2024-09-12T02:58:51.815 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:58:51.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:58:51.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:51 smithi016 ceph-mon[30925]: pgmap v665: 97 pgs: 97 active+clean; 282 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-12T02:58:51.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:51 smithi033 ceph-mon[35092]: pgmap v665: 97 pgs: 97 active+clean; 282 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-12T02:58:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:51 smithi184 ceph-mon[35321]: pgmap v665: 97 pgs: 97 active+clean; 282 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 261 op/s 2024-09-12T02:58:52.090 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:52.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 -- 172.21.15.16:0/2972514837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 msgr2=0x7fbc5c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/2972514837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c106810 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009900 tx=0x7fbc5002a3f0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 -- 172.21.15.16:0/2972514837 shutdown_connections 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/2972514837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 0x7fbc5c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/2972514837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c106810 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/2972514837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 0x7fbc5c1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 -- 172.21.15.16:0/2972514837 >> 172.21.15.16:0/2972514837 conn(0x7fbc5c0fd8d0 msgr2=0x7fbc5c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 -- 172.21.15.16:0/2972514837 shutdown_connections 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.527+0000 7fbc60ce7640 1 -- 172.21.15.16:0/2972514837 wait complete. 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 Processor -- start 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 -- start start 2024-09-12T02:58:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 0x7fbc5c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 0x7fbc5c19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c112040 con 0x7fbc5c1027c0 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c111ec0 con 0x7fbc5c106f40 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc60ce7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1121c0 con 0x7fbc5c101e10 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.528+0000 7fbc59d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54578/0 (socket says 172.21.15.16:54578) 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 -- 172.21.15.16:0/3928269714 learned_addr learned my addr 172.21.15.16:0/3928269714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc5a59c640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 0x7fbc5c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc5ad9d640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 0x7fbc5c19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 -- 172.21.15.16:0/3928269714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 msgr2=0x7fbc5c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 0x7fbc5c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 -- 172.21.15.16:0/3928269714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 msgr2=0x7fbc5c19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 0x7fbc5c19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 -- 172.21.15.16:0/3928269714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c19d570 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc59d9b640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fbc50005ab0 tx=0x7fbc500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5003f030 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c19d800 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc50003ce0 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50035070 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.529+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c1a5140 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.530+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5c1032f0 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.532+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbc50042030 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.532+0000 7fbc4b7fe640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 0x7fbc440a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.533+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbc500efb50 con 0x7fbc5c1027c0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.533+0000 7fbc5a59c640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 0x7fbc440a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.533+0000 7fbc5a59c640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 0x7fbc440a5040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009990 tx=0x7fbc4c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:52.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.536+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbc500bcac0 con 0x7fbc5c1027c0 2024-09-12T02:58:52.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1073400 9.76 MB/sec execute 474 sec latency 21.075 ms 2024-09-12T02:58:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.779+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc5c10a120 con 0x7fbc5c1027c0 2024-09-12T02:58:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.780+0000 7fbc4b7fe640 1 -- 172.21.15.16:0/3928269714 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7fbc500bc3c0 con 0x7fbc5c1027c0 2024-09-12T02:58:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:58:52.781 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:58:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 msgr2=0x7fbc440a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 0x7fbc440a5040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009990 tx=0x7fbc4c009110 comp rx=0 tx=0).stop 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 msgr2=0x7fbc5c198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fbc50005ab0 tx=0x7fbc500042f0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 shutdown_connections 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c106f40 0x7fbc5c19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbc440a2b80 0x7fbc440a5040 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc5c1027c0 0x7fbc5c198b00 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 --2- 172.21.15.16:0/3928269714 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc5c101e10 0x7fbc5c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 >> 172.21.15.16:0/3928269714 conn(0x7fbc5c0fd8d0 msgr2=0x7fbc5c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 shutdown_connections 2024-09-12T02:58:52.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:52.784+0000 7fbc60ce7640 1 -- 172.21.15.16:0/3928269714 wait complete. 2024-09-12T02:58:53.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:58:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:53.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1076157 9.77 MB/sec execute 475 sec latency 87.545 ms 2024-09-12T02:58:53.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:53 smithi016 ceph-mon[30925]: pgmap v666: 97 pgs: 97 active+clean; 285 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 199 op/s 2024-09-12T02:58:53.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3928269714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:53 smithi033 ceph-mon[35092]: pgmap v666: 97 pgs: 97 active+clean; 285 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 199 op/s 2024-09-12T02:58:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3928269714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:53 smithi184 ceph-mon[35321]: pgmap v666: 97 pgs: 97 active+clean; 285 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 199 op/s 2024-09-12T02:58:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3928269714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 -- 172.21.15.16:0/346202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 msgr2=0x7f7b481021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/346202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b481021d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009900 tx=0x7f7b3802a410 comp rx=0 tx=0).stop 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 -- 172.21.15.16:0/346202006 shutdown_connections 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/346202006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 0x7f7b4810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/346202006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 0x7f7b481067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/346202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b481021d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 -- 172.21.15.16:0/346202006 >> 172.21.15.16:0/346202006 conn(0x7f7b480fd8f0 msgr2=0x7f7b480ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 -- 172.21.15.16:0/346202006 shutdown_connections 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.175+0000 7f7b4ea29640 1 -- 172.21.15.16:0/346202006 wait complete. 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 Processor -- start 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 -- start start 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 0x7f7b48198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 0x7f7b4819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48111f80 con 0x7f7b48101df0 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48111e00 con 0x7f7b481027a0 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4ea29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112100 con 0x7f7b48106f20 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54590/0 (socket says 172.21.15.16:54590) 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 -- 172.21.15.16:0/1668565123 learned_addr learned my addr 172.21.15.16:0/1668565123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 0x7f7b48198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 0x7f7b4819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 -- 172.21.15.16:0/1668565123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 msgr2=0x7f7b4819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 0x7f7b4819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 -- 172.21.15.16:0/1668565123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 msgr2=0x7f7b48198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 0x7f7b48198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.176+0000 7f7b47fff640 1 -- 172.21.15.16:0/1668565123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4819d3d0 con 0x7f7b48101df0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b38004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b45ffb640 1 -- 172.21.15.16:0/1668565123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3803f030 con 0x7f7b48101df0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b45ffb640 1 -- 172.21.15.16:0/1668565123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b38004440 con 0x7f7b48101df0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4819d660 con 0x7f7b48101df0 2024-09-12T02:58:54.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b45ffb640 1 -- 172.21.15.16:0/1668565123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38035070 con 0x7f7b48101df0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.177+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b481a4fa0 con 0x7f7b48101df0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.179+0000 7f7b45ffb640 1 -- 172.21.15.16:0/1668565123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7b38042030 con 0x7f7b48101df0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.180+0000 7f7b50e02740 1 Processor -- start 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.180+0000 7f7b45ffb640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 0x7f7b340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.180+0000 7f7b50e02740 1 -- start start 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.180+0000 7f7b45ffb640 1 -- 172.21.15.16:0/1668565123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b380ef380 con 0x7f7b48101df0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.180+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 0x7f7b340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8c5d0 0x55b412a8ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 0x55b412a99760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4129f22f0 con 0x55b412a8db40 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b412999cf0 con 0x55b412a8c5d0 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b50e02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b4129ee2b0 con 0x55b412a97370 2024-09-12T02:58:54.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54600/0 (socket says 172.21.15.16:54600) 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 -- 172.21.15.16:0/2111785694 learned_addr learned my addr 172.21.15.16:0/2111785694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 0x55b412a99760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8c5d0 0x55b412a8ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 0x7f7b340a5040 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005fd0 tx=0x7f7b40009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 -- 172.21.15.16:0/2111785694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 msgr2=0x55b412a99760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 0x55b412a99760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 -- 172.21.15.16:0/2111785694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8c5d0 msgr2=0x55b412a8ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8c5d0 0x55b412a8ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 -- 172.21.15.16:0/2111785694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41297e3a0 con 0x55b412a8db40 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 0x55b412a99760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:54.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7b380b8670 tx=0x7f7b380bb6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf3ef4b47f7e12f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b3e7fc640 1 -- 172.21.15.16:0/2111785694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38049040 con 0x55b412a8db40 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.181+0000 7f7b3e7fc640 1 -- 172.21.15.16:0/2111785694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b380c4860 con 0x55b412a8db40 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b3e7fc640 1 -- 172.21.15.16:0/2111785694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380bc160 con 0x55b412a8db40 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 -- 172.21.15.16:0/2111785694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 msgr2=0x55b412a8c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7b380b8670 tx=0x7f7b380bb6c0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 -- 172.21.15.16:0/2111785694 shutdown_connections 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a97370 0x55b412a99760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8c5d0 0x55b412a8ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2111785694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8c090 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 -- 172.21.15.16:0/2111785694 >> 172.21.15.16:0/2111785694 conn(0x55b412a55310 msgr2=0x55b412a88c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 -- 172.21.15.16:0/2111785694 shutdown_connections 2024-09-12T02:58:54.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.182+0000 7f7b50e02740 1 -- 172.21.15.16:0/2111785694 wait complete. 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 -- start start 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 Processor -- start 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a96d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412a94650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b412a99e30 con 0x55b412a8db40 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b4129ad060 con 0x55b412a97370 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.183+0000 7f7b50e02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b412a99fb0 con 0x55b412a8c5d0 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b50e02740 10 client.0 _ll_register_callbacks cb 0x55b4127008c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a96d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54608/0 (socket says 172.21.15.16:54608) 2024-09-12T02:58:54.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b3ffff640 1 -- 172.21.15.16:0/2551425144 learned_addr learned my addr 172.21.15.16:0/2551425144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412a94650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 -- 172.21.15.16:0/2551425144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 msgr2=0x55b412a94650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412a94650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 -- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 msgr2=0x55b412a94040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 -- 172.21.15.16:0/2551425144 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4127633c0 con 0x55b412a8c5d0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a96d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b3803d040 tx=0x7f7b380058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c9f3bd221ce5f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b277fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380da070 con 0x55b412a8c5d0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b380bbf00 con 0x55b412a8c5d0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.184+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b380c4860 con 0x55b412a8c5d0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7b38049040 con 0x55b412a8c5d0 2024-09-12T02:58:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 10 client.27355 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b412a99e30 con 0x55b412a8c5d0 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b380c4860 con 0x55b412a8c5d0 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7b380c4460 con 0x55b412a8c5d0 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 10 client.27355 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 10 client.27355 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 20 client.27355 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412aa6b20 0x55b412aa8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 4 client.27355 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4129ff860 con 0x55b412aa6b20 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412aa6b20 0x55b412aa8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b277fe640 10 client.27355 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.185+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412aa6b20 0x55b412aa8f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3ce9f5a2d4542be6 server_cookie=5af1663ad11921f0 in_seq=0 out_seq=0 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b277fe640 1 -- 172.21.15.16:0/2551425144 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b4129ff860 con 0x55b412aa6b20 2024-09-12T02:58:54.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b277fe640 10 client.27355 handle_command_reply: tid=0 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 client.27355 shutdown 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 msgr2=0x55b412a96d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a96d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b3803d040 tx=0x7f7b380058b0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 shutdown_connections 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412aa6b20 0x55b412aa8f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412a94650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a94040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b277fe640 0 client.27355 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2551425144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a96d90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.186+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 >> 172.21.15.16:0/2551425144 conn(0x55b412a55310 msgr2=0x55b412a8e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.187+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 shutdown_connections 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.187+0000 7f7b50e02740 1 -- 172.21.15.16:0/2551425144 wait complete. 2024-09-12T02:58:54.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.187+0000 7f7b50e02740 20 client.27355 trim_cache size 0 max 16384 2024-09-12T02:58:54.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 Processor -- start 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 -- start start 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412aae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ad490 con 0x55b412a8db40 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481aefd0 con 0x55b412a97370 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b50e02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481adb60 con 0x55b412a8c5d0 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b4cfc6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412aae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b47fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33238/0 (socket says 172.21.15.16:33238) 2024-09-12T02:58:54.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.201+0000 7f7b47fff640 1 -- 172.21.15.16:0/2026766021 learned_addr learned my addr 172.21.15.16:0/2026766021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 -- 172.21.15.16:0/2026766021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 msgr2=0x55b412aae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412aae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 -- 172.21.15.16:0/2026766021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 msgr2=0x55b412a8f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 -- 172.21.15.16:0/2026766021 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b412a89960 con 0x55b412a8c5d0 2024-09-12T02:58:54.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b47fff640 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c4330 tx=0x7f7b380bbb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed4000cc55108cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b3dffb640 1 -- 172.21.15.16:0/2026766021 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380da040 con 0x55b412a8c5d0 2024-09-12T02:58:54.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b3dffb640 1 -- 172.21.15.16:0/2026766021 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b380d8070 con 0x55b412a8c5d0 2024-09-12T02:58:54.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.202+0000 7f7b3dffb640 1 -- 172.21.15.16:0/2026766021 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38003e10 con 0x55b412a8c5d0 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 -- 172.21.15.16:0/2026766021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 msgr2=0x55b412a8f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7b380c4330 tx=0x7f7b380bbb80 comp rx=0 tx=0).stop 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 -- 172.21.15.16:0/2026766021 shutdown_connections 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a97370 0x55b412aae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a8db40 0x55b412a8f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 --2- 172.21.15.16:0/2026766021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a8f120 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.203+0000 7f7b50e02740 1 -- 172.21.15.16:0/2026766021 >> 172.21.15.16:0/2026766021 conn(0x55b412a55310 msgr2=0x55b412a87d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.204+0000 7f7b50e02740 1 -- 172.21.15.16:0/2026766021 shutdown_connections 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.204+0000 7f7b50e02740 1 -- 172.21.15.16:0/2026766021 wait complete. 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.204+0000 7f7b50e02740 1 -- start start 2024-09-12T02:58:54.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.204+0000 7f7b50e02740 1 Processor -- start 2024-09-12T02:58:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a9f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 0x55b412a9ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ae900 con 0x55b412a97370 2024-09-12T02:58:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ae230 con 0x55b412a8db40 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ab470 con 0x55b412a8c5d0 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b50e02740 10 client.0 _ll_register_callbacks cb 0x7f7b300097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54616/0 (socket says 172.21.15.16:54616) 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 -- 172.21.15.16:0/1205494623 learned_addr learned my addr 172.21.15.16:0/1205494623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 0x55b412a9ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a9f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 -- 172.21.15.16:0/1205494623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 msgr2=0x55b412a9f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a9f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 -- 172.21.15.16:0/1205494623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 msgr2=0x55b412a9ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 0x55b412a9ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 -- 172.21.15.16:0/1205494623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b412a9b700 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b3ffff640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 0x55b412a9ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b4cfc6640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7b400044c0 tx=0x7f7b40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979b18f5309d3e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.205+0000 7f7b277fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4001da80 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b4001dc20 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b380d8070 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7b4001de20 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b50e02740 10 client.18048 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b412b11700 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.206+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b380d8070 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7b400327e0 con 0x55b412a97370 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 10 client.18048 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:54.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 10 client.18048 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 20 client.18048 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412b11ed0 0x55b412b142f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 4 client.18048 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4129f9d70 con 0x55b412b11ed0 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412b11ed0 0x55b412b142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b277fe640 10 client.18048 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.207+0000 7f7b47fff640 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412b11ed0 0x55b412b142f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b571ff75c8b3154 server_cookie=81a3324d7bec836c in_seq=0 out_seq=0 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b277fe640 1 -- 172.21.15.16:0/1205494623 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4129f9d70 con 0x55b412b11ed0 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b277fe640 10 client.18048 handle_command_reply: tid=0 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 client.18048 shutdown 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 msgr2=0x55b412a9b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7b400044c0 tx=0x7f7b40004ab0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 shutdown_connections 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b412a97370 0x55b412a9b060 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b412b11ed0 0x55b412b142f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b277fe640 0 client.18048 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b412a8db40 0x55b412a9ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 --2- 172.21.15.16:0/1205494623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b412a8c5d0 0x55b412a9f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 >> 172.21.15.16:0/1205494623 conn(0x55b412a55310 msgr2=0x55b412a958a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 shutdown_connections 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 1 -- 172.21.15.16:0/1205494623 wait complete. 2024-09-12T02:58:54.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.208+0000 7f7b50e02740 20 client.18048 trim_cache size 0 max 16384 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 msgr2=0x7f7b340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 0x7f7b340a5040 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005fd0 tx=0x7f7b40009500 comp rx=0 tx=0).stop 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 msgr2=0x7f7b48198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b38004040 comp rx=0 tx=0).stop 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 shutdown_connections 2024-09-12T02:58:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b48106f20 0x7f7b4819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b481027a0 0x7f7b48198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b340a2b80 0x7f7b340a5040 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 --2- 172.21.15.16:0/1668565123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b48101df0 0x7f7b48198420 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 >> 172.21.15.16:0/1668565123 conn(0x7f7b480fd8f0 msgr2=0x7f7b480fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 shutdown_connections 2024-09-12T02:58:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:54.209+0000 7f7b4ea29640 1 -- 172.21.15.16:0/1668565123 wait complete. 2024-09-12T02:58:54.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1077986 9.77 MB/sec execute 476 sec latency 60.993 ms 2024-09-12T02:58:54.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:58:55.176 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.669+0000 7f6f10508640 1 -- 172.21.15.16:0/1052050550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 msgr2=0x7f6f0810bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.669+0000 7f6f10508640 1 --2- 172.21.15.16:0/1052050550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f0810bbb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009900 tx=0x7f6ef802a480 comp rx=0 tx=0).stop 2024-09-12T02:58:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 -- 172.21.15.16:0/1052050550 shutdown_connections 2024-09-12T02:58:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 --2- 172.21.15.16:0/1052050550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 0x7f6f0810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 --2- 172.21.15.16:0/1052050550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f0810bbb0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 --2- 172.21.15.16:0/1052050550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 -- 172.21.15.16:0/1052050550 >> 172.21.15.16:0/1052050550 conn(0x7f6f080fc7a0 msgr2=0x7f6f080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 -- 172.21.15.16:0/1052050550 shutdown_connections 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 -- 172.21.15.16:0/1052050550 wait complete. 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 Processor -- start 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.670+0000 7f6f10508640 1 -- start start 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f08100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 0x7f6f08101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08111f60 con 0x7f6f081036c0 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f08100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f08100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52938/0 (socket says 172.21.15.16:52938) 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37912/0 (socket says 172.21.15.16:37912) 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0daa3640 1 -- 172.21.15.16:0/616330535 learned_addr learned my addr 172.21.15.16:0/616330535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 0x7f6f08101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08111de0 con 0x7f6f08102da0 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081120e0 con 0x7f6f0810c180 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/616330535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 msgr2=0x7f6f08101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 0x7f6f08101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 msgr2=0x7f6f08100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f08100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.671+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/616330535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f081a68e0 con 0x7f6f08102da0 2024-09-12T02:58:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.672+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400e9f0 tx=0x7f6f0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.672+0000 7f6eff7fe640 1 -- 172.21.15.16:0/616330535 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400cd80 con 0x7f6f08102da0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.672+0000 7f6eff7fe640 1 -- 172.21.15.16:0/616330535 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f040035f0 con 0x7f6f08102da0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.672+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f081a6bd0 con 0x7f6f08102da0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.673+0000 7f6eff7fe640 1 -- 172.21.15.16:0/616330535 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04010690 con 0x7f6f08102da0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.673+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081a7160 con 0x7f6f08102da0 2024-09-12T02:58:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.675+0000 7f6eff7fe640 1 -- 172.21.15.16:0/616330535 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6f04003b30 con 0x7f6f08102da0 2024-09-12T02:58:55.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 Processor -- start 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 -- start start 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6eff7fe640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 0x7f6ef40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151b4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55571510d2f0 con 0x5557151a8b40 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5557150b4cf0 con 0x5557151a75d0 2024-09-12T02:58:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f128e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5557151092b0 con 0x5557151b2370 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0eaa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0eaa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52944/0 (socket says 172.21.15.16:52944) 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37924/0 (socket says 172.21.15.16:37924) 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0eaa5640 1 -- 172.21.15.16:0/1488298490 learned_addr learned my addr 172.21.15.16:0/1488298490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6eff7fe640 1 -- 172.21.15.16:0/616330535 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6f040cf2e0 con 0x7f6f08102da0 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.676+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 0x7f6ef40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 -- 172.21.15.16:0/1488298490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 msgr2=0x5557151b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 -- 172.21.15.16:0/1488298490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 msgr2=0x5557151a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 -- 172.21.15.16:0/1488298490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557150993a0 con 0x5557151a8b40 2024-09-12T02:58:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x5557151a8070 tx=0x7f6f00005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4da37669170d18a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6efcff9640 1 -- 172.21.15.16:0/1488298490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004b20 con 0x5557151a8b40 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 0x7f6ef40a50c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8031550 tx=0x7f6ef8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6efcff9640 1 -- 172.21.15.16:0/1488298490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f00003e50 con 0x5557151a8b40 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.677+0000 7f6efcff9640 1 -- 172.21.15.16:0/1488298490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000fca0 con 0x5557151a8b40 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- 172.21.15.16:0/1488298490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 msgr2=0x5557151a7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x5557151a8070 tx=0x7f6f00005bd0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- 172.21.15.16:0/1488298490 shutdown_connections 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a75d0 0x5557151a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1488298490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a8b40 0x5557151a7090 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- 172.21.15.16:0/1488298490 >> 172.21.15.16:0/1488298490 conn(0x555715170310 msgr2=0x5557151a3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- 172.21.15.16:0/1488298490 shutdown_connections 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- 172.21.15.16:0/1488298490 wait complete. 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.678+0000 7f6f128e1740 1 -- start start 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 Processor -- start 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557151b5010 con 0x5557151a75d0 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5557150c8060 con 0x5557151a8b40 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5557151b5190 con 0x5557151b2370 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37936/0 (socket says 172.21.15.16:37936) 2024-09-12T02:58:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f0daa3640 1 -- 172.21.15.16:0/3703875870 learned_addr learned my addr 172.21.15.16:0/3703875870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f128e1740 10 client.0 _ll_register_callbacks cb 0x555714e1b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.679+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 -- 172.21.15.16:0/3703875870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 msgr2=0x5557151af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 -- 172.21.15.16:0/3703875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 msgr2=0x5557151b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 -- 172.21.15.16:0/3703875870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555714e7e3c0 con 0x5557151a8b40 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009a20 tx=0x7f6ef8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862deab84864565d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6ef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef804c020 con 0x5557151a8b40 2024-09-12T02:58:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6ef8004310 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f00003e50 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6ef8035040 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f128e1740 10 client.27479 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557151c0a00 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.680+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f00003e50 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6ef80035f0 con 0x5557151a8b40 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 10 client.27479 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 10 client.27479 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 20 client.27479 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x5557151c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 4 client.27479 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55571511a860 con 0x5557151c1f60 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x5557151c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x5557151c43e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=44ad95a1adc0323a server_cookie=2be5af6439258f3b in_seq=0 out_seq=0 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.681+0000 7f6ef1ffb640 10 client.27479 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/3703875870 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55571511a860 con 0x5557151c1f60 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6ef1ffb640 10 client.27479 handle_command_reply: tid=0 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 client.27479 shutdown 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 msgr2=0x5557151aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009a20 tx=0x7f6ef8009260 comp rx=0 tx=0).stop 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 shutdown_connections 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x5557151c43e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6ef1ffb640 0 client.27479 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151b2370 0x5557151af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151aef80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 --2- 172.21.15.16:0/3703875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.682+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 >> 172.21.15.16:0/3703875870 conn(0x555715170310 msgr2=0x5557151a9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.683+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 shutdown_connections 2024-09-12T02:58:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.683+0000 7f6f128e1740 1 -- 172.21.15.16:0/3703875870 wait complete. 2024-09-12T02:58:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.683+0000 7f6f128e1740 20 client.27479 trim_cache size 0 max 16384 2024-09-12T02:58:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 Processor -- start 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 -- start start 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151b2370 0x5557151c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081ab3e0 con 0x5557151b2370 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081acf20 con 0x5557151a8b40 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.698+0000 7f6f128e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081abab0 con 0x5557151a75d0 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0eaa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151b2370 0x5557151c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57682/0 (socket says 172.21.15.16:57682) 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/1817685731 learned_addr learned my addr 172.21.15.16:0/1817685731 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:55.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/1817685731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 msgr2=0x5557151ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/1817685731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151b2370 msgr2=0x5557151c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151b2370 0x5557151c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/1817685731 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557151aa120 con 0x5557151a75d0 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6f040984c0 tx=0x7f6f04003aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe8c8f6d023fedf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.699+0000 7f6ef3fff640 1 -- 172.21.15.16:0/1817685731 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0409fea0 con 0x5557151a75d0 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.700+0000 7f6ef3fff640 1 -- 172.21.15.16:0/1817685731 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f040a4760 con 0x5557151a75d0 2024-09-12T02:58:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.700+0000 7f6ef3fff640 1 -- 172.21.15.16:0/1817685731 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04098160 con 0x5557151a75d0 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.700+0000 7f6f128e1740 1 -- 172.21.15.16:0/1817685731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 msgr2=0x5557151be2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.700+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6f040984c0 tx=0x7f6f04003aa0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 -- 172.21.15.16:0/1817685731 shutdown_connections 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151b2370 0x5557151c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151a8b40 0x5557151ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 --2- 172.21.15.16:0/1817685731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a75d0 0x5557151be2b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 -- 172.21.15.16:0/1817685731 >> 172.21.15.16:0/1817685731 conn(0x555715170310 msgr2=0x5557151ae070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 -- 172.21.15.16:0/1817685731 shutdown_connections 2024-09-12T02:58:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 -- 172.21.15.16:0/1817685731 wait complete. 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 -- start start 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 Processor -- start 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.701+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f128e1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 0x55571522bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f128e1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081ac850 con 0x5557151a75d0 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f128e1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081ac180 con 0x5557151b2370 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f128e1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081a93c0 con 0x5557151a8b40 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0daa3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57690/0 (socket says 172.21.15.16:57690) 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0daa3640 1 -- 172.21.15.16:0/4096912245 learned_addr learned my addr 172.21.15.16:0/4096912245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 0x55571522bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f128e1740 10 client.0 _ll_register_callbacks cb 0x7f6ef8040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/4096912245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 msgr2=0x5557151b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 msgr2=0x55571522bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 0x55571522bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 -- 172.21.15.16:0/4096912245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571522c1c0 con 0x5557151a75d0 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0eaa5640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 0x55571522bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6f0e2a4640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b65a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6f04097f50 tx=0x7f6f04009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1918d5e2f70c48 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6ef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040ccea0 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.702+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6f04098160 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f040a4760 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6f040a9050 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 10 client.18066 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55571522d340 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f040a4760 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6f0409af40 con 0x5557151a75d0 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 10 client.18066 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 10 client.18066 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 20 client.18066 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x55571522fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 4 client.18066 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555715114d70 con 0x5557151c1f60 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.703+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x55571522fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f0daa3640 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x55571522fa40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=71b4f706f8b9b9e7 server_cookie=69e7c9cdb94632c7 in_seq=0 out_seq=0 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6ef1ffb640 10 client.18066 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6ef1ffb640 1 -- 172.21.15.16:0/4096912245 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555715114d70 con 0x5557151c1f60 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6ef1ffb640 10 client.18066 handle_command_reply: tid=0 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 client.18066 shutdown 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 msgr2=0x5557151b65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b65a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6f04097f50 tx=0x7f6f04009430 comp rx=0 tx=0).stop 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 shutdown_connections 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5557151c1f60 0x55571522fa40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5557151b2370 0x55571522bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5557151a8b40 0x5557151b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 --2- 172.21.15.16:0/4096912245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557151a75d0 0x5557151b65a0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 >> 172.21.15.16:0/4096912245 conn(0x555715170310 msgr2=0x5557151b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6ef1ffb640 0 client.18066 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 shutdown_connections 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 1 -- 172.21.15.16:0/4096912245 wait complete. 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.704+0000 7f6f128e1740 20 client.18066 trim_cache size 0 max 16384 2024-09-12T02:58:55.710 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 msgr2=0x7f6ef40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 0x7f6ef40a50c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8031550 tx=0x7f6ef8005a20 comp rx=0 tx=0).stop 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 msgr2=0x7f6f08100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400e9f0 tx=0x7f6f0400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 shutdown_connections 2024-09-12T02:58:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0810c180 0x7f6f08101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ef40a2c00 0x7f6ef40a50c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f081036c0 0x7f6f08100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 --2- 172.21.15.16:0/616330535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f08102da0 0x7f6f08100a00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 >> 172.21.15.16:0/616330535 conn(0x7f6f080fc7a0 msgr2=0x7f6f080fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 shutdown_connections 2024-09-12T02:58:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:55.705+0000 7f6f10508640 1 -- 172.21.15.16:0/616330535 wait complete. 2024-09-12T02:58:55.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1080381 9.78 MB/sec execute 477 sec latency 21.880 ms 2024-09-12T02:58:55.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:55 smithi016 ceph-mon[30925]: pgmap v667: 97 pgs: 97 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-09-12T02:58:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:55 smithi033 ceph-mon[35092]: pgmap v667: 97 pgs: 97 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-09-12T02:58:56.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:55 smithi184 ceph-mon[35321]: pgmap v667: 97 pgs: 97 active+clean; 286 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-09-12T02:58:56.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:58:56.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1083413 9.80 MB/sec execute 478 sec latency 19.545 ms 2024-09-12T02:58:56.802 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:57.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 -- 172.21.15.16:0/236603598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb88106ba0 msgr2=0x7fcb88106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/236603598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb88106ba0 0x7fcb88106f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009930 tx=0x7fcb7802a420 comp rx=0 tx=0).stop 2024-09-12T02:58:57.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 -- 172.21.15.16:0/236603598 shutdown_connections 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/236603598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb8810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/236603598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb881000d0 0x7fcb88100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/236603598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb88106ba0 0x7fcb88106f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 -- 172.21.15.16:0/236603598 >> 172.21.15.16:0/236603598 conn(0x7fcb88077220 msgr2=0x7fcb880fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 -- 172.21.15.16:0/236603598 shutdown_connections 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.307+0000 7fcb8d02b640 1 -- 172.21.15.16:0/236603598 wait complete. 2024-09-12T02:58:57.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.308+0000 7fcb8d02b640 1 Processor -- start 2024-09-12T02:58:57.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 -- start start 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 0x7fcb881012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb881017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb88106ba0 0x7fcb88101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881120a0 con 0x7fcb88100a70 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88111f20 con 0x7fcb88106ba0 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8d02b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb88112220 con 0x7fcb881000d0 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb881017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb88106ba0 0x7fcb88101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 0x7fcb881012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb881017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52992/0 (socket says 172.21.15.16:52992) 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.309+0000 7fcb8659c640 1 -- 172.21.15.16:0/164121303 learned_addr learned my addr 172.21.15.16:0/164121303 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 -- 172.21.15.16:0/164121303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb88106ba0 msgr2=0x7fcb88101d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb88106ba0 0x7fcb88101d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 -- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 msgr2=0x7fcb881017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb881017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 -- 172.21.15.16:0/164121303 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb881a6a00 con 0x7fcb881000d0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 0x7fcb881012b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb7802a940 tx=0x7fcb78003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb6ffff640 1 -- 172.21.15.16:0/164121303 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7803f070 con 0x7fcb881000d0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb881a6c90 con 0x7fcb881000d0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb6ffff640 1 -- 172.21.15.16:0/164121303 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb78031c90 con 0x7fcb881000d0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb6ffff640 1 -- 172.21.15.16:0/164121303 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb78035030 con 0x7fcb881000d0 2024-09-12T02:58:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.310+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb881a71c0 con 0x7fcb881000d0 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb6ffff640 1 -- 172.21.15.16:0/164121303 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcb78042070 con 0x7fcb881000d0 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 Processor -- start 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 -- start start 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b34090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b345d0 0x5605b3b34a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5605b3a9a2f0 con 0x5605b3b35b40 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5605b3a41cf0 con 0x5605b3b3f370 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8f404740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5605b3a962b0 con 0x5605b3b345d0 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b345d0 0x5605b3b34a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b34090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37990/0 (socket says 172.21.15.16:37990) 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.313+0000 7fcb8659c640 1 -- 172.21.15.16:0/3607185470 learned_addr learned my addr 172.21.15.16:0/3607185470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 -- 172.21.15.16:0/3607185470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b345d0 msgr2=0x5605b3b34a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb6ffff640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 0x7fcb700a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b345d0 0x5605b3b34a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 -- 172.21.15.16:0/3607185470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 msgr2=0x5605b3b34090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b34090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 -- 172.21.15.16:0/3607185470 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b3a263a0 con 0x5605b3b3f370 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb6ffff640 1 -- 172.21.15.16:0/164121303 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb780ef550 con 0x7fcb881000d0 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8659c640 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00bd70 tx=0x7fcb7c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34131859d2fbd0bb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb8759e640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 0x7fcb700a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb6d7fa640 1 -- 172.21.15.16:0/3607185470 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00be60 con 0x5605b3b3f370 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb6d7fa640 1 -- 172.21.15.16:0/3607185470 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c0047a0 con 0x5605b3b3f370 2024-09-12T02:58:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.314+0000 7fcb6d7fa640 1 -- 172.21.15.16:0/3607185470 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c011820 con 0x5605b3b3f370 2024-09-12T02:58:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 -- 172.21.15.16:0/3607185470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 msgr2=0x5605b3b41760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8759e640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 0x7fcb700a5040 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009b80 tx=0x7fcb80004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00bd70 tx=0x7fcb7c0107a0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 -- 172.21.15.16:0/3607185470 shutdown_connections 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b41760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b345d0 0x5605b3b34a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3607185470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b34090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 -- 172.21.15.16:0/3607185470 >> 172.21.15.16:0/3607185470 conn(0x5605b3afd310 msgr2=0x5605b3b30c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 -- 172.21.15.16:0/3607185470 shutdown_connections 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.315+0000 7fcb8f404740 1 -- 172.21.15.16:0/3607185470 wait complete. 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 -- start start 2024-09-12T02:58:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 Processor -- start 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b3ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b3c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 0x5605b3b3c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5605b3b42070 con 0x5605b3b3f370 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5605b3a55060 con 0x5605b3b345d0 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5605b3b421f0 con 0x5605b3b35b40 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8f404740 10 client.0 _ll_register_callbacks cb 0x5605b37a88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 0x5605b3b3c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b3ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 0x5605b3b3c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53014/0 (socket says 172.21.15.16:53014) 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8759e640 1 -- 172.21.15.16:0/449529529 learned_addr learned my addr 172.21.15.16:0/449529529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb8659c640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b3c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb86d9d640 1 -- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 msgr2=0x5605b3b3c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.316+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b3c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb86d9d640 1 -- 172.21.15.16:0/449529529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 msgr2=0x5605b3b3c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 0x5605b3b3c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb86d9d640 1 -- 172.21.15.16:0/449529529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b380b3c0 con 0x5605b3b345d0 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb8659c640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b3c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b3ed70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcb780ed0b0 tx=0x7fcb780046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132ecbdffddae414 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb78049070 con 0x5605b3b345d0 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb780b8430 con 0x5605b3b345d0 2024-09-12T02:58:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb7c0047a0 con 0x5605b3b345d0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb780b8060 con 0x5605b3b345d0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb8f404740 10 client.27497 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5605b3b56320 con 0x5605b3b345d0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.317+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c0047a0 con 0x5605b3b345d0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcb780bc020 con 0x5605b3b345d0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 10 client.27497 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 10 client.27497 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 20 client.27497 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3b57a50 0x5605b3b59ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 4 client.27497 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605b3aa7860 con 0x5605b3b57a50 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8659c640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3b57a50 0x5605b3b59ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb8659c640 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3b57a50 0x5605b3b59ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8f2d4ff1933cfcb server_cookie=55b4d176cd75cfe9 in_seq=0 out_seq=0 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.318+0000 7fcb6aff5640 10 client.27497 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb6aff5640 1 -- 172.21.15.16:0/449529529 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5605b3aa7860 con 0x5605b3b57a50 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb6aff5640 10 client.27497 handle_command_reply: tid=0 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 client.27497 shutdown 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 msgr2=0x5605b3b3ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b3ed70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcb780ed0b0 tx=0x7fcb780046e0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 shutdown_connections 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b3f370 0x5605b3b3c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3b57a50 0x5605b3b59ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b3c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 --2- 172.21.15.16:0/449529529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b3ed70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb6aff5640 0 client.27497 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 >> 172.21.15.16:0/449529529 conn(0x5605b3afd310 msgr2=0x5605b3b36960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 shutdown_connections 2024-09-12T02:58:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.319+0000 7fcb8f404740 1 -- 172.21.15.16:0/449529529 wait complete. 2024-09-12T02:58:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.320+0000 7fcb8f404740 20 client.27497 trim_cache size 0 max 16384 2024-09-12T02:58:57.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 Processor -- start 2024-09-12T02:58:57.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 -- start start 2024-09-12T02:58:57.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b53ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b5f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881ab330 con 0x5605b3b345d0 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881ace70 con 0x5605b3b3f370 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.335+0000 7fcb8f404740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881aba00 con 0x5605b3b35b40 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53030/0 (socket says 172.21.15.16:53030) 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb86d9d640 1 -- 172.21.15.16:0/3904554602 learned_addr learned my addr 172.21.15.16:0/3904554602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b5f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8659c640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b53ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 -- 172.21.15.16:0/3904554602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 msgr2=0x5605b3b53ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b53ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 -- 172.21.15.16:0/3904554602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 msgr2=0x5605b3b30e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 -- 172.21.15.16:0/3904554602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b3b31730 con 0x5605b3b3f370 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8659c640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b53ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b5f710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009fd0 tx=0x7fcb80018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f691345993c6bd6c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb6cff9640 1 -- 172.21.15.16:0/3904554602 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80021bb0 con 0x5605b3b3f370 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb6cff9640 1 -- 172.21.15.16:0/3904554602 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80021d50 con 0x5605b3b3f370 2024-09-12T02:58:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.336+0000 7fcb6cff9640 1 -- 172.21.15.16:0/3904554602 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80029620 con 0x5605b3b3f370 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 -- 172.21.15.16:0/3904554602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 msgr2=0x5605b3b5f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b5f710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009fd0 tx=0x7fcb80018000 comp rx=0 tx=0).stop 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 -- 172.21.15.16:0/3904554602 shutdown_connections 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b3f370 0x5605b3b5f710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b35b40 0x5605b3b53ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 --2- 172.21.15.16:0/3904554602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b345d0 0x5605b3b30e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 -- 172.21.15.16:0/3904554602 >> 172.21.15.16:0/3904554602 conn(0x5605b3afd310 msgr2=0x5605b3b30700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 -- 172.21.15.16:0/3904554602 shutdown_connections 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.337+0000 7fcb8f404740 1 -- 172.21.15.16:0/3904554602 wait complete. 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 -- start start 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 Processor -- start 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b50260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b38c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 0x5605b3b39140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881ac7a0 con 0x5605b3b35b40 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881ac0d0 con 0x5605b3b345d0 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb881a9310 con 0x5605b3b3f370 2024-09-12T02:58:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b50260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb86d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b50260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38014/0 (socket says 172.21.15.16:38014) 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8f404740 10 client.0 _ll_register_callbacks cb 0x7fcb780f2d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8659c640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b38c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb8759e640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 0x5605b3b39140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.338+0000 7fcb86d9d640 1 -- 172.21.15.16:0/73465827 learned_addr learned my addr 172.21.15.16:0/73465827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 -- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 msgr2=0x5605b3b39140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 0x5605b3b39140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 -- 172.21.15.16:0/73465827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 msgr2=0x5605b3b50260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b50260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 -- 172.21.15.16:0/73465827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b3b39780 con 0x5605b3b35b40 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8759e640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 0x5605b3b39140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8659c640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b38c00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00bd40 tx=0x7fcb7c005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7638d8a5434d11da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00b980 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb7c004660 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb80021d50 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.339+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb7c005570 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 10 client.18084 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5605b3bc2560 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c02e070 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcb7c0288d0 con 0x5605b3b35b40 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 10 client.18084 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 10 client.18084 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 20 client.18084 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3bc29b0 0x5605b3bc4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 4 client.18084 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcb80021d50 con 0x5605b3bc29b0 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.340+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3bc29b0 0x5605b3bc4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.341+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3bc29b0 0x5605b3bc4da0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1acef627a7c9fcb7 server_cookie=814474e7e59e918e in_seq=0 out_seq=0 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.341+0000 7fcb6aff5640 10 client.18084 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb6aff5640 1 -- 172.21.15.16:0/73465827 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcb80021d50 con 0x5605b3bc29b0 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb6aff5640 10 client.18084 handle_command_reply: tid=0 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 client.18084 shutdown 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 msgr2=0x5605b3b38c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b38c00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00bd40 tx=0x7fcb7c005730 comp rx=0 tx=0).stop 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 shutdown_connections 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5605b3bc29b0 0x5605b3bc4da0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb6aff5640 0 client.18084 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:58:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5605b3b3f370 0x5605b3b39140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605b3b35b40 0x5605b3b38c00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 --2- 172.21.15.16:0/73465827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b3b345d0 0x5605b3b50260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 >> 172.21.15.16:0/73465827 conn(0x5605b3afd310 msgr2=0x5605b3b36930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 shutdown_connections 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 1 -- 172.21.15.16:0/73465827 wait complete. 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.342+0000 7fcb8f404740 20 client.18084 trim_cache size 0 max 16384 2024-09-12T02:58:57.348 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 msgr2=0x7fcb700a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 0x7fcb700a5040 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009b80 tx=0x7fcb80004ab0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 msgr2=0x7fcb881012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 0x7fcb881012b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb7802a940 tx=0x7fcb78003b90 comp rx=0 tx=0).stop 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 shutdown_connections 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb88106ba0 0x7fcb88101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb700a2b80 0x7fcb700a5040 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb88100a70 0x7fcb881017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 --2- 172.21.15.16:0/164121303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb881000d0 0x7fcb881012b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:57.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 >> 172.21.15.16:0/164121303 conn(0x7fcb88077220 msgr2=0x7fcb88104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:57.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 shutdown_connections 2024-09-12T02:58:57.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:57.343+0000 7fcb8d02b640 1 -- 172.21.15.16:0/164121303 wait complete. 2024-09-12T02:58:57.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1085828 9.79 MB/sec execute 479 sec latency 11.606 ms 2024-09-12T02:58:57.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:57 smithi016 ceph-mon[30925]: pgmap v668: 97 pgs: 97 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-09-12T02:58:57.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:57 smithi033 ceph-mon[35092]: pgmap v668: 97 pgs: 97 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-09-12T02:58:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:57 smithi184 ceph-mon[35321]: pgmap v668: 97 pgs: 97 active+clean; 288 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-09-12T02:58:58.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:58:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:58:58.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1087637 9.80 MB/sec execute 480 sec latency 80.492 ms 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 -- 172.21.15.16:0/3180654883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424106f10 msgr2=0x7fe42410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 --2- 172.21.15.16:0/3180654883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424106f10 0x7fe42410ea90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009ca0 tx=0x7fe41c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 -- 172.21.15.16:0/3180654883 shutdown_connections 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 --2- 172.21.15.16:0/3180654883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424106f10 0x7fe42410ea90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 --2- 172.21.15.16:0/3180654883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424102790 0x7fe4241067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 --2- 172.21.15.16:0/3180654883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 -- 172.21.15.16:0/3180654883 >> 172.21.15.16:0/3180654883 conn(0x7fe4240fd8c0 msgr2=0x7fe4240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 -- 172.21.15.16:0/3180654883 shutdown_connections 2024-09-12T02:58:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.888+0000 7fe429750640 1 -- 172.21.15.16:0/3180654883 wait complete. 2024-09-12T02:58:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 Processor -- start 2024-09-12T02:58:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 -- start start 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4240744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 0x7fe424074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424106f10 0x7fe424074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe424111f70 con 0x7fe424102790 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe424111df0 con 0x7fe424106f10 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe429750640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241120f0 con 0x7fe424101de0 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 0x7fe424074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 0x7fe424074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53062/0 (socket says 172.21.15.16:53062) 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe4227fc640 1 -- 172.21.15.16:0/700811708 learned_addr learned my addr 172.21.15.16:0/700811708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe422ffd640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4240744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.889+0000 7fe4237fe640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424106f10 0x7fe424074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 -- 172.21.15.16:0/700811708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424106f10 msgr2=0x7fe424074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424106f10 0x7fe424074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 -- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 msgr2=0x7fe424074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 0x7fe424074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 -- 172.21.15.16:0/700811708 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241a6890 con 0x7fe424101de0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe422ffd640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4240744f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e9f0 tx=0x7fe41400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe40bfff640 1 -- 172.21.15.16:0/700811708 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41400ccc0 con 0x7fe424101de0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe40bfff640 1 -- 172.21.15.16:0/700811708 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe414005020 con 0x7fe424101de0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241a6b80 con 0x7fe424101de0 2024-09-12T02:58:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe40bfff640 1 -- 172.21.15.16:0/700811708 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4140049e0 con 0x7fe424101de0 2024-09-12T02:58:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.890+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a7070 con 0x7fe424101de0 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe40bfff640 1 -- 172.21.15.16:0/700811708 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe414005560 con 0x7fe424101de0 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 Processor -- start 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 -- start start 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb7fca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb809760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dcb7622f0 con 0x562dcb7fdb40 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562dcb709cf0 con 0x562dcb807370 2024-09-12T02:58:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe42bb29740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562dcb75e2b0 con 0x562dcb7fc5d0 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53068/0 (socket says 172.21.15.16:53068) 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe4237fe640 1 -- 172.21.15.16:0/4144604396 learned_addr learned my addr 172.21.15.16:0/4144604396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe422ffd640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb7fca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.893+0000 7fe4227fc640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb809760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 -- 172.21.15.16:0/4144604396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 msgr2=0x562dcb7fca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb7fca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 -- 172.21.15.16:0/4144604396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 msgr2=0x562dcb7fc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 -- 172.21.15.16:0/4144604396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dcb6ee3a0 con 0x562dcb807370 2024-09-12T02:58:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe422ffd640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb7fca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4237fe640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe40bfff640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 0x7fe40c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4237fe640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 0x7fe40c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4227fc640 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb809760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe418009c00 tx=0x7fe418011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=209cef4c3bd1b4b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe40bfff640 1 -- 172.21.15.16:0/700811708 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe4140cef80 con 0x7fe424101de0 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4097fa640 1 -- 172.21.15.16:0/4144604396 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418004230 con 0x562dcb807370 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4097fa640 1 -- 172.21.15.16:0/4144604396 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4180043d0 con 0x562dcb807370 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.894+0000 7fe4097fa640 1 -- 172.21.15.16:0/4144604396 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418004d40 con 0x562dcb807370 2024-09-12T02:58:58.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- 172.21.15.16:0/4144604396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 msgr2=0x562dcb809760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe4237fe640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 0x7fe40c0a5040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe41c004af0 tx=0x7fe41c004a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb809760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe418009c00 tx=0x7fe418011810 comp rx=0 tx=0).stop 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- 172.21.15.16:0/4144604396 shutdown_connections 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb809760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb7fca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 --2- 172.21.15.16:0/4144604396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb7fc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- 172.21.15.16:0/4144604396 >> 172.21.15.16:0/4144604396 conn(0x562dcb7c5310 msgr2=0x562dcb7f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- 172.21.15.16:0/4144604396 shutdown_connections 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- 172.21.15.16:0/4144604396 wait complete. 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.895+0000 7fe42bb29740 1 -- start start 2024-09-12T02:58:58.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 Processor -- start 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 0x562dcb806cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 0x562dcb804520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dcb80a070 con 0x562dcb7fdb40 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562dcb71d060 con 0x562dcb7fc5d0 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562dcb80a1f0 con 0x562dcb807370 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe42bb29740 10 client.0 _ll_register_callbacks cb 0x562dcb4708c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 0x562dcb806cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53078/0 (socket says 172.21.15.16:53078) 2024-09-12T02:58:58.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 0x562dcb806cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38058/0 (socket says 172.21.15.16:38058) 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4227fc640 1 -- 172.21.15.16:0/1500817353 learned_addr learned my addr 172.21.15.16:0/1500817353 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 0x562dcb804520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 -- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 msgr2=0x562dcb806cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 0x562dcb806cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 -- 172.21.15.16:0/1500817353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 msgr2=0x562dcb803fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4237fe640 1 -- 172.21.15.16:0/1500817353 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dcb4d33c0 con 0x562dcb807370 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.896+0000 7fe4227fc640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe4237fe640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 0x562dcb804520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe41c02f540 tx=0x7fe41c02a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f68b29677a6dbd72 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe406ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c031db0 con 0x562dcb807370 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe41c0318e0 con 0x562dcb807370 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4180043d0 con 0x562dcb807370 2024-09-12T02:58:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe41c004590 con 0x562dcb807370 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe42bb29740 10 client.27409 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562dcb815aa0 con 0x562dcb807370 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.897+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4180043d0 con 0x562dcb807370 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe41c0305c0 con 0x562dcb807370 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 10 client.27409 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 10 client.27409 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 20 client.27409 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb816f60 0x562dcb8193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 4 client.27409 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dcb76f860 con 0x562dcb816f60 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe422ffd640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb816f60 0x562dcb8193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe422ffd640 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb816f60 0x562dcb8193d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a0105ba423a21ac8 server_cookie=eb66099fc26f2a33 in_seq=0 out_seq=0 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.898+0000 7fe406ff5640 10 client.27409 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe406ff5640 1 -- 172.21.15.16:0/1500817353 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562dcb76f860 con 0x562dcb816f60 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe406ff5640 10 client.27409 handle_command_reply: tid=0 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 client.27409 shutdown 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 msgr2=0x562dcb804520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 0x562dcb804520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe41c02f540 tx=0x7fe41c02a8c0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 shutdown_connections 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb816f60 0x562dcb8193d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb807370 0x562dcb804520 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb803fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 --2- 172.21.15.16:0/1500817353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fc5d0 0x562dcb806cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 >> 172.21.15.16:0/1500817353 conn(0x562dcb7c5310 msgr2=0x562dcb7fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe406ff5640 0 client.27409 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 shutdown_connections 2024-09-12T02:58:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 1 -- 172.21.15.16:0/1500817353 wait complete. 2024-09-12T02:58:58.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.899+0000 7fe42bb29740 20 client.27409 trim_cache size 0 max 16384 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 Processor -- start 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 -- start start 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb813310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb807370 0x562dcb7f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241af6a0 con 0x562dcb807370 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b11e0 con 0x562dcb7fdb40 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe42bb29740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241afd70 con 0x562dcb7fc5d0 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb813310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38066/0 (socket says 172.21.15.16:38066) 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 -- 172.21.15.16:0/3847078037 learned_addr learned my addr 172.21.15.16:0/3847078037 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4237fe640 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb807370 0x562dcb7f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 -- 172.21.15.16:0/3847078037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 msgr2=0x562dcb813310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb813310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 -- 172.21.15.16:0/3847078037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb807370 msgr2=0x562dcb7f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb807370 0x562dcb7f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.914+0000 7fe4227fc640 1 -- 172.21.15.16:0/3847078037 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dcb7f9180 con 0x562dcb7fdb40 2024-09-12T02:58:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.915+0000 7fe4227fc640 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe4180048f0 tx=0x7fe4180097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3402ae2a6811207 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.915+0000 7fe408ff9640 1 -- 172.21.15.16:0/3847078037 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418028070 con 0x562dcb7fdb40 2024-09-12T02:58:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.915+0000 7fe408ff9640 1 -- 172.21.15.16:0/3847078037 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41801f040 con 0x562dcb7fdb40 2024-09-12T02:58:58.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.915+0000 7fe408ff9640 1 -- 172.21.15.16:0/3847078037 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418024680 con 0x562dcb7fdb40 2024-09-12T02:58:58.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- 172.21.15.16:0/3847078037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 msgr2=0x562dcb7f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe4180048f0 tx=0x7fe4180097b0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- 172.21.15.16:0/3847078037 shutdown_connections 2024-09-12T02:58:58.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb807370 0x562dcb7f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb7fdb40 0x562dcb7f8700 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- 172.21.15.16:0/3847078037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb813310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- 172.21.15.16:0/3847078037 >> 172.21.15.16:0/3847078037 conn(0x562dcb7c5310 msgr2=0x562dcb7ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- 172.21.15.16:0/3847078037 shutdown_connections 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- 172.21.15.16:0/3847078037 wait complete. 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 -- start start 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 Processor -- start 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb80fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.916+0000 7fe42bb29740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb8011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe42bb29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b0b10 con 0x562dcb7fdb40 2024-09-12T02:58:58.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe42bb29740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b0440 con 0x562dcb807370 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe42bb29740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241ad680 con 0x562dcb7fc5d0 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe42bb29740 10 client.0 _ll_register_callbacks cb 0x7fe41c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53104/0 (socket says 172.21.15.16:53104) 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb80fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 -- 172.21.15.16:0/741230643 learned_addr learned my addr 172.21.15.16:0/741230643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4237fe640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb8011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 -- 172.21.15.16:0/741230643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 msgr2=0x562dcb80fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb80fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 -- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 msgr2=0x562dcb8011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb8011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 -- 172.21.15.16:0/741230643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dcb8019d0 con 0x562dcb7fdb40 2024-09-12T02:58:58.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe422ffd640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb80fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe4227fc640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe418019d60 tx=0x7fe418004880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40d06cc96415214 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe406ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.917+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418031020 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41801f040 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe418019910 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe418028040 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 10 client.18102 fetch_fsmap learned FSMap version 32 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562dcb881c10 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41801f040 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe418018070 con 0x562dcb7fdb40 2024-09-12T02:58:58.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 10 client.18102 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 10 client.18102 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 20 client.18102 populate_metadata read hostname 'smithi016' 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb882030 0x562dcb884420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 4 client.18102 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562dcb769d70 con 0x562dcb882030 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.918+0000 7fe422ffd640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb882030 0x562dcb884420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe422ffd640 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb882030 0x562dcb884420 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f4b0776f0870ce server_cookie=24b47c73f7dbfe7b in_seq=0 out_seq=0 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe406ff5640 10 client.18102 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe406ff5640 1 -- 172.21.15.16:0/741230643 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562dcb769d70 con 0x562dcb882030 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe406ff5640 10 client.18102 handle_command_reply: tid=0 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 client.18102 shutdown 2024-09-12T02:58:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 msgr2=0x562dcb800c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe418019d60 tx=0x7fe418004880 comp rx=0 tx=0).stop 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 shutdown_connections 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x562dcb882030 0x562dcb884420 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe406ff5640 0 client.18102 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562dcb807370 0x562dcb8011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dcb7fdb40 0x562dcb800c90 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 --2- 172.21.15.16:0/741230643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562dcb7fc5d0 0x562dcb80fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 >> 172.21.15.16:0/741230643 conn(0x562dcb7c5310 msgr2=0x562dcb7ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 shutdown_connections 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 1 -- 172.21.15.16:0/741230643 wait complete. 2024-09-12T02:58:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.919+0000 7fe42bb29740 20 client.18102 trim_cache size 0 max 16384 2024-09-12T02:58:58.927 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:58:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 msgr2=0x7fe40c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 0x7fe40c0a5040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe41c004af0 tx=0x7fe41c004a80 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 msgr2=0x7fe4240744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4240744f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e9f0 tx=0x7fe41400eeb0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 shutdown_connections 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424106f10 0x7fe424074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe40c0a2b80 0x7fe40c0a5040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424102790 0x7fe424074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 --2- 172.21.15.16:0/700811708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424101de0 0x7fe4240744f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 >> 172.21.15.16:0/700811708 conn(0x7fe4240fd8c0 msgr2=0x7fe4240feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 shutdown_connections 2024-09-12T02:58:58.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:58:58.920+0000 7fe429750640 1 -- 172.21.15.16:0/700811708 wait complete. 2024-09-12T02:58:59.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1090581 9.81 MB/sec execute 481 sec latency 19.802 ms 2024-09-12T02:58:59.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:58:59 smithi016 ceph-mon[30925]: pgmap v669: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-09-12T02:58:59.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:58:59.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:58:59 smithi033 ceph-mon[35092]: pgmap v669: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-09-12T02:59:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:58:59 smithi184 ceph-mon[35321]: pgmap v669: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-09-12T02:59:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 -- 172.21.15.16:0/450154475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a8102d30 msgr2=0x7f66a8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 --2- 172.21.15.16:0/450154475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a8102d30 0x7f66a8103110 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009930 tx=0x7f66a402a420 comp rx=0 tx=0).stop 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 -- 172.21.15.16:0/450154475 shutdown_connections 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 --2- 172.21.15.16:0/450154475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 --2- 172.21.15.16:0/450154475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a81036e0 0x7f66a8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 --2- 172.21.15.16:0/450154475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a8102d30 0x7f66a8103110 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 -- 172.21.15.16:0/450154475 >> 172.21.15.16:0/450154475 conn(0x7f66a80fc7d0 msgr2=0x7f66a80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 -- 172.21.15.16:0/450154475 shutdown_connections 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.586+0000 7f66af826640 1 -- 172.21.15.16:0/450154475 wait complete. 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.587+0000 7f66af826640 1 Processor -- start 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.587+0000 7f66af826640 1 -- start start 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.587+0000 7f66af826640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a8101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81120a0 con 0x7f66a81036e0 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8111f20 con 0x7f66a81082a0 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8112220 con 0x7f66a8102d30 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53122/0 (socket says 172.21.15.16:53122) 2024-09-12T02:59:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57818/0 (socket says 172.21.15.16:57818) 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 -- 172.21.15.16:0/3835043300 learned_addr learned my addr 172.21.15.16:0/3835043300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66addc3640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a8101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 -- 172.21.15.16:0/3835043300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 msgr2=0x7f66a8100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 -- 172.21.15.16:0/3835043300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 msgr2=0x7f66a8101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a8101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 -- 172.21.15.16:0/3835043300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a81a6a60 con 0x7f66a81036e0 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66addc3640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9f0 tx=0x7f669800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f669e7fc640 1 -- 172.21.15.16:0/3835043300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669800ccc0 con 0x7f66a81036e0 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f669e7fc640 1 -- 172.21.15.16:0/3835043300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6698010910 con 0x7f66a81036e0 2024-09-12T02:59:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a81a6d50 con 0x7f66a81036e0 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f669e7fc640 1 -- 172.21.15.16:0/3835043300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698018920 con 0x7f66a81036e0 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.588+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a81a7230 con 0x7f66a81036e0 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.591+0000 7f669e7fc640 1 -- 172.21.15.16:0/3835043300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6698010ac0 con 0x7f66a81036e0 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.591+0000 7f66b1bff740 1 Processor -- start 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 -- start start 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f669e7fc640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 0x7f66940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c1b40 0x5589721c0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589721262f0 con 0x5589721cb370 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589720cdcf0 con 0x5589721c1b40 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66b1bff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5589721222b0 con 0x5589721c05d0 2024-09-12T02:59:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f669e7fc640 1 -- 172.21.15.16:0/3835043300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f66980cf2a0 con 0x7f66a81036e0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66addc3640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 0x7f66940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57830/0 (socket says 172.21.15.16:57830) 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 -- 172.21.15.16:0/895909579 learned_addr learned my addr 172.21.15.16:0/895909579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66addc3640 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c1b40 0x5589721c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 -- 172.21.15.16:0/895909579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c1b40 msgr2=0x5589721c0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c1b40 0x5589721c0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 -- 172.21.15.16:0/895909579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 msgr2=0x5589721cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.592+0000 7f66acdc1640 1 -- 172.21.15.16:0/895909579 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589720b23a0 con 0x5589721c05d0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.593+0000 7f66acdc1640 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6698098020 tx=0x7f669809bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=287add6512cf5b04 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.593+0000 7f66addc3640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 0x7f66940a5040 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009fd0 tx=0x7f66a00116b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.593+0000 7f66937fe640 1 -- 172.21.15.16:0/895909579 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698099c60 con 0x5589721c05d0 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.593+0000 7f66937fe640 1 -- 172.21.15.16:0/895909579 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6698099ec0 con 0x5589721c05d0 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.593+0000 7f66937fe640 1 -- 172.21.15.16:0/895909579 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698097910 con 0x5589721c05d0 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- 172.21.15.16:0/895909579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 msgr2=0x5589721c0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6698098020 tx=0x7f669809bbc0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- 172.21.15.16:0/895909579 shutdown_connections 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721c0a10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 --2- 172.21.15.16:0/895909579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c1b40 0x5589721c0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- 172.21.15.16:0/895909579 >> 172.21.15.16:0/895909579 conn(0x558972189310 msgr2=0x5589721bcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- 172.21.15.16:0/895909579 shutdown_connections 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- 172.21.15.16:0/895909579 wait complete. 2024-09-12T02:59:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 -- start start 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.594+0000 7f66b1bff740 1 Processor -- start 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721cb370 0x5589721c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589721ce010 con 0x5589721c1b40 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589720e1060 con 0x5589721c05d0 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5589721ce190 con 0x5589721cb370 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 10 client.0 _ll_register_callbacks cb 0x558971e348c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66addc3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721cb370 0x5589721c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53144/0 (socket says 172.21.15.16:53144) 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38112/0 (socket says 172.21.15.16:38112) 2024-09-12T02:59:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 -- 172.21.15.16:0/3252342438 learned_addr learned my addr 172.21.15.16:0/3252342438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 -- 172.21.15.16:0/3252342438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721cb370 msgr2=0x5589721c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721cb370 0x5589721c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 -- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 msgr2=0x5589721cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 -- 172.21.15.16:0/3252342438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558971e973c0 con 0x5589721c1b40 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f669800c6a0 tx=0x7f6698097f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c6df0a07a2509e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f6690ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698010630 con 0x5589721c1b40 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6698097910 con 0x5589721c1b40 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.595+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6698099ec0 con 0x5589721c1b40 2024-09-12T02:59:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6698097b30 con 0x5589721c1b40 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 10 client.18114 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5589721d9af0 con 0x5589721c1b40 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6698099ec0 con 0x5589721c1b40 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f669809aa80 con 0x5589721c1b40 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 10 client.18114 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 10 client.18114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 20 client.18114 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589721dae40 0x5589721dd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 4 client.18114 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558972133860 con 0x5589721dae40 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.596+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589721dae40 0x5589721dd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.597+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589721dae40 0x5589721dd230 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=423118ddc1ebd305 server_cookie=c9be456cae89f64a in_seq=0 out_seq=0 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.597+0000 7f6690ff9640 10 client.18114 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.597+0000 7f6690ff9640 1 -- 172.21.15.16:0/3252342438 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558972133860 con 0x5589721dae40 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.597+0000 7f6690ff9640 10 client.18114 handle_command_reply: tid=0 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.597+0000 7f66b1bff740 1 client.18114 shutdown 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 msgr2=0x5589721c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f669800c6a0 tx=0x7f6698097f90 comp rx=0 tx=0).stop 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 shutdown_connections 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589721dae40 0x5589721dd230 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721cb370 0x5589721c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721c8050 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3252342438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f6690ff9640 0 client.18114 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 >> 172.21.15.16:0/3252342438 conn(0x558972189310 msgr2=0x5589721c2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 shutdown_connections 2024-09-12T02:59:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 1 -- 172.21.15.16:0/3252342438 wait complete. 2024-09-12T02:59:00.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.598+0000 7f66b1bff740 20 client.18114 trim_cache size 0 max 16384 2024-09-12T02:59:00.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.613+0000 7f66b1bff740 1 Processor -- start 2024-09-12T02:59:00.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.613+0000 7f66b1bff740 1 -- start start 2024-09-12T02:59:00.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.613+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.613+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 0x5589721c3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 0x5589721c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 0x5589721c3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57858/0 (socket says 172.21.15.16:57858) 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53150/0 (socket says 172.21.15.16:53150) 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66b1bff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81af760 con 0x5589721cb370 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38122/0 (socket says 172.21.15.16:38122) 2024-09-12T02:59:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81b12a0 con 0x5589721c05d0 2024-09-12T02:59:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66ad5c2640 1 -- 172.21.15.16:0/472313152 learned_addr learned my addr 172.21.15.16:0/472313152 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81afe30 con 0x5589721c1b40 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 -- 172.21.15.16:0/472313152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 msgr2=0x5589721c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 0x5589721c3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 -- 172.21.15.16:0/472313152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 msgr2=0x5589721c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.614+0000 7f66addc3640 1 -- 172.21.15.16:0/472313152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589721bd960 con 0x5589721cb370 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.615+0000 7f66addc3640 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f66a0011910 tx=0x7f66a0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b379917a96988232 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.615+0000 7f6692ffd640 1 -- 172.21.15.16:0/472313152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a002e070 con 0x5589721cb370 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.615+0000 7f6692ffd640 1 -- 172.21.15.16:0/472313152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a0010de0 con 0x5589721cb370 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.615+0000 7f6692ffd640 1 -- 172.21.15.16:0/472313152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0029c40 con 0x5589721cb370 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 msgr2=0x5589721c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f66a0011910 tx=0x7f66a0021000 comp rx=0 tx=0).stop 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 shutdown_connections 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721cb370 0x5589721c3660 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c1b40 0x5589721c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 --2- 172.21.15.16:0/472313152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721c05d0 0x5589721c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 >> 172.21.15.16:0/472313152 conn(0x558972189310 msgr2=0x5589721c6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 shutdown_connections 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- 172.21.15.16:0/472313152 wait complete. 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.616+0000 7f66b1bff740 1 -- start start 2024-09-12T02:59:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 Processor -- start 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721dd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721cb370 0x55897223d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81b0bd0 con 0x5589721c1b40 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81b0500 con 0x5589721cb370 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ad740 con 0x5589721c05d0 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66b1bff740 10 client.0 _ll_register_callbacks cb 0x7f66a0011b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66ad5c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721dd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66acdc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53166/0 (socket says 172.21.15.16:53166) 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66acdc1640 1 -- 172.21.15.16:0/3373788118 learned_addr learned my addr 172.21.15.16:0/3373788118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.617+0000 7f66addc3640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721cb370 0x55897223d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 -- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721cb370 msgr2=0x55897223d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721cb370 0x55897223d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 -- 172.21.15.16:0/3373788118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 msgr2=0x5589721ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 -- 172.21.15.16:0/3373788118 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897223dd80 con 0x5589721c05d0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66ad5c2640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721dd5d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009900 tx=0x7f66a40044a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31cb9b1595378b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f6690ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4035040 con 0x5589721c05d0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f66a4005460 con 0x5589721c05d0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a0010de0 con 0x5589721c05d0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f66a4004770 con 0x5589721c05d0 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66b1bff740 10 client.27433 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:00.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.618+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558972245150 con 0x5589721c05d0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a0010de0 con 0x5589721c05d0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f66a4031bb0 con 0x5589721c05d0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 10 client.27433 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 10 client.27433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 20 client.27433 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558972245820 0x558972247c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 4 client.27433 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55897212dd70 con 0x558972245820 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558972245820 0x558972247c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f6690ff9640 10 client.27433 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.619+0000 7f66acdc1640 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558972245820 0x558972247c40 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64154a53d331e58a server_cookie=2583d1bfcc7049a2 in_seq=0 out_seq=0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f6690ff9640 1 -- 172.21.15.16:0/3373788118 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55897212dd70 con 0x558972245820 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f6690ff9640 10 client.27433 handle_command_reply: tid=0 2024-09-12T02:59:00.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 client.27433 shutdown 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 msgr2=0x5589721dd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721dd5d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009900 tx=0x7f66a40044a0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 shutdown_connections 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558972245820 0x558972247c40 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589721cb370 0x55897223d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f6690ff9640 0 client.27433 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589721c1b40 0x5589721ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 --2- 172.21.15.16:0/3373788118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589721c05d0 0x5589721dd5d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 >> 172.21.15.16:0/3373788118 conn(0x558972189310 msgr2=0x5589721c7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 shutdown_connections 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 1 -- 172.21.15.16:0/3373788118 wait complete. 2024-09-12T02:59:00.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.620+0000 7f66b1bff740 20 client.27433 trim_cache size 0 max 16384 2024-09-12T02:59:00.625 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:00.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 msgr2=0x7f66940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 0x7f66940a5040 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009fd0 tx=0x7f66a00116b0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 msgr2=0x7f66a8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:00.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9f0 tx=0x7f669800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 shutdown_connections 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81082a0 0x7f66a8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f66940a2b80 0x7f66940a5040 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a81036e0 0x7f66a8100ed0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 --2- 172.21.15.16:0/3835043300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66a8102d30 0x7f66a8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 >> 172.21.15.16:0/3835043300 conn(0x7f66a80fc7d0 msgr2=0x7f66a8105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 shutdown_connections 2024-09-12T02:59:00.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:00.621+0000 7f66af826640 1 -- 172.21.15.16:0/3835043300 wait complete. 2024-09-12T02:59:00.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1092912 9.81 MB/sec execute 482 sec latency 26.573 ms 2024-09-12T02:59:01.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1095125 9.82 MB/sec execute 483 sec latency 14.327 ms 2024-09-12T02:59:01.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:01 smithi016 ceph-mon[30925]: pgmap v670: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 252 op/s 2024-09-12T02:59:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:01 smithi033 ceph-mon[35092]: pgmap v670: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 252 op/s 2024-09-12T02:59:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:01 smithi184 ceph-mon[35321]: pgmap v670: 97 pgs: 97 active+clean; 291 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 252 op/s 2024-09-12T02:59:02.384 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:59:02.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:59:02.657 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:02.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1098010 9.84 MB/sec execute 484 sec latency 10.913 ms 2024-09-12T02:59:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 -- 172.21.15.16:0/343009400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 msgr2=0x7fb33c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 --2- 172.21.15.16:0/343009400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c103110 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009930 tx=0x7fb32c02a420 comp rx=0 tx=0).stop 2024-09-12T02:59:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 -- 172.21.15.16:0/343009400 shutdown_connections 2024-09-12T02:59:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 --2- 172.21.15.16:0/343009400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1082a0 0x7fb33c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 --2- 172.21.15.16:0/343009400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1036e0 0x7fb33c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 --2- 172.21.15.16:0/343009400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c103110 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 -- 172.21.15.16:0/343009400 >> 172.21.15.16:0/343009400 conn(0x7fb33c0fc7d0 msgr2=0x7fb33c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 -- 172.21.15.16:0/343009400 shutdown_connections 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.246+0000 7fb3422ae640 1 -- 172.21.15.16:0/343009400 wait complete. 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.247+0000 7fb3422ae640 1 Processor -- start 2024-09-12T02:59:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.247+0000 7fb3422ae640 1 -- start start 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1036e0 0x7fb33c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1082a0 0x7fb33c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1120a0 con 0x7fb33c102d30 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c111f20 con 0x7fb33c1036e0 2024-09-12T02:59:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb3422ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c112220 con 0x7fb33c1082a0 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53198/0 (socket says 172.21.15.16:53198) 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 -- 172.21.15.16:0/2963647359 learned_addr learned my addr 172.21.15.16:0/2963647359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1036e0 0x7fb33c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb34084b640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1082a0 0x7fb33c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 -- 172.21.15.16:0/2963647359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1082a0 msgr2=0x7fb33c19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1082a0 0x7fb33c19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 -- 172.21.15.16:0/2963647359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1036e0 msgr2=0x7fb33c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1036e0 0x7fb33c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.248+0000 7fb33bfff640 1 -- 172.21.15.16:0/2963647359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33c19d5c0 con 0x7fb33c102d30 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb33bfff640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009230 tx=0x7fb32c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb3397fa640 1 -- 172.21.15.16:0/2963647359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c03f030 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33c19d850 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb3397fa640 1 -- 172.21.15.16:0/2963647359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c003ad0 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb3397fa640 1 -- 172.21.15.16:0/2963647359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c035070 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.249+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb33c1a5190 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.251+0000 7fb3397fa640 1 -- 172.21.15.16:0/2963647359 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb32c042030 con 0x7fb33c102d30 2024-09-12T02:59:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 Processor -- start 2024-09-12T02:59:03.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 -- start start 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969b480 0x560f296a5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f296a5d40 0x560f296a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f29600280 con 0x560f2969b480 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f295a7b10 con 0x560f296a5d40 2024-09-12T02:59:03.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb344687740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f295fc240 con 0x560f2969ab60 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969b480 0x560f296a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb34084b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb34084b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57890/0 (socket says 172.21.15.16:57890) 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb34084b640 1 -- 172.21.15.16:0/3339273518 learned_addr learned my addr 172.21.15.16:0/3339273518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f296a5d40 0x560f296a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.252+0000 7fb3397fa640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 0x7fb3240a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 -- 172.21.15.16:0/3339273518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f296a5d40 msgr2=0x560f296a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f296a5d40 0x560f296a8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 -- 172.21.15.16:0/3339273518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969b480 msgr2=0x560f296a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969b480 0x560f296a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 -- 172.21.15.16:0/3339273518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f295b41f0 con 0x560f2969ab60 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 0x7fb3240a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb3397fa640 1 -- 172.21.15.16:0/2963647359 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb32c0ef3d0 con 0x7fb33c102d30 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3340104a0 tx=0x7fb334010960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0104c85fbe216a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb3227fc640 1 -- 172.21.15.16:0/3339273518 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33401d910 con 0x560f2969ab60 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb3227fc640 1 -- 172.21.15.16:0/3339273518 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb334011440 con 0x560f2969ab60 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb34084b640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 0x7fb3240a50c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb334021000 tx=0x7fb33401e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.253+0000 7fb3227fc640 1 -- 172.21.15.16:0/3339273518 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33401c640 con 0x560f2969ab60 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- 172.21.15.16:0/3339273518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 msgr2=0x560f2969af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3340104a0 tx=0x7fb334010960 comp rx=0 tx=0).stop 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- 172.21.15.16:0/3339273518 shutdown_connections 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f296a5d40 0x560f296a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969b480 0x560f296a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 --2- 172.21.15.16:0/3339273518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969af40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- 172.21.15.16:0/3339273518 >> 172.21.15.16:0/3339273518 conn(0x560f296970f0 msgr2=0x560f296974d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- 172.21.15.16:0/3339273518 shutdown_connections 2024-09-12T02:59:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- 172.21.15.16:0/3339273518 wait complete. 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.254+0000 7fb344687740 1 -- start start 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 Processor -- start 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f296a5d40 0x560f296a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f296acd20 con 0x560f2969ab60 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f295baf10 con 0x560f2969b480 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f296acea0 con 0x560f296a5d40 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb344687740 10 client.0 _ll_register_callbacks cb 0x560f29583a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53216/0 (socket says 172.21.15.16:53216) 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33bfff640 1 -- 172.21.15.16:0/618552425 learned_addr learned my addr 172.21.15.16:0/618552425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb34084b640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f296a5d40 0x560f296a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 -- 172.21.15.16:0/618552425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f296a5d40 msgr2=0x560f296a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f296a5d40 0x560f296a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 -- 172.21.15.16:0/618552425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 msgr2=0x560f296a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33b7fe640 1 -- 172.21.15.16:0/618552425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f293716a0 con 0x560f2969b480 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.255+0000 7fb33bfff640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a3150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0fee80 tx=0x7fb330012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e63d34abdfa8372 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330004020 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb3300041c0 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb334011440 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3300043c0 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb344687740 10 client.27551 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f296b8980 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb334011440 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.256+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb330003c30 con 0x560f2969b480 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 10 client.27551 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 10 client.27551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 20 client.27551 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f296b9e40 0x560f296bc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 4 client.27551 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f2960d7f0 con 0x560f296b9e40 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb33bfff640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f296b9e40 0x560f296bc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb33bfff640 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f296b9e40 0x560f296bc250 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460ca714b92917e0 server_cookie=5a9e5498b71022c7 in_seq=0 out_seq=0 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.257+0000 7fb31b7fe640 10 client.27551 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb31b7fe640 1 -- 172.21.15.16:0/618552425 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f2960d7f0 con 0x560f296b9e40 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb31b7fe640 10 client.27551 handle_command_reply: tid=0 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 client.27551 shutdown 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 msgr2=0x560f296a3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a3150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0fee80 tx=0x7fb330012b60 comp rx=0 tx=0).stop 2024-09-12T02:59:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 shutdown_connections 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f296b9e40 0x560f296bc250 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f296a5d40 0x560f296a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a3150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 --2- 172.21.15.16:0/618552425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f2969ab60 0x560f296a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 >> 172.21.15.16:0/618552425 conn(0x560f296970f0 msgr2=0x560f29696270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb31b7fe640 0 client.27551 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 shutdown_connections 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.258+0000 7fb344687740 1 -- 172.21.15.16:0/618552425 wait complete. 2024-09-12T02:59:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.259+0000 7fb344687740 20 client.27551 trim_cache size 0 max 16384 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 Processor -- start 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 -- start start 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1ab3c0 con 0x560f296a5d40 2024-09-12T02:59:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1acf00 con 0x560f2969b480 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb344687740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1aba90 con 0x560f2969ab60 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb34084b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb33b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb34084b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53222/0 (socket says 172.21.15.16:53222) 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.277+0000 7fb34084b640 1 -- 172.21.15.16:0/4027462739 learned_addr learned my addr 172.21.15.16:0/4027462739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 -- 172.21.15.16:0/4027462739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 msgr2=0x560f296a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 -- 172.21.15.16:0/4027462739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 msgr2=0x560f296c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 -- 172.21.15.16:0/4027462739 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f296981a0 con 0x560f2969ab60 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb33bfff640 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969d5c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009900 tx=0x7fb32c0b7fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e81beb29e16c385d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb321ffb640 1 -- 172.21.15.16:0/4027462739 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c049040 con 0x560f2969ab60 2024-09-12T02:59:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb321ffb640 1 -- 172.21.15.16:0/4027462739 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c03b460 con 0x560f2969ab60 2024-09-12T02:59:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.278+0000 7fb321ffb640 1 -- 172.21.15.16:0/4027462739 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0b8300 con 0x560f2969ab60 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- 172.21.15.16:0/4027462739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 msgr2=0x560f2969d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969d5c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009900 tx=0x7fb32c0b7fb0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- 172.21.15.16:0/4027462739 shutdown_connections 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969b480 0x560f296a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 --2- 172.21.15.16:0/4027462739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969ab60 0x560f2969d5c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- 172.21.15.16:0/4027462739 >> 172.21.15.16:0/4027462739 conn(0x560f296970f0 msgr2=0x560f296a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- 172.21.15.16:0/4027462739 shutdown_connections 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- 172.21.15.16:0/4027462739 wait complete. 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 -- start start 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.279+0000 7fb344687740 1 Processor -- start 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969ab60 0x560f296adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1ac830 con 0x560f296a5d40 2024-09-12T02:59:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1ac160 con 0x560f2969ab60 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a93a0 con 0x560f2969b480 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb344687740 10 client.0 _ll_register_callbacks cb 0x7fb330009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57924/0 (socket says 172.21.15.16:57924) 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 -- 172.21.15.16:0/1091152896 learned_addr learned my addr 172.21.15.16:0/1091152896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb34084b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33bfff640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969ab60 0x560f296adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 -- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969ab60 msgr2=0x560f296adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969ab60 0x560f296adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 -- 172.21.15.16:0/1091152896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 msgr2=0x560f296aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.280+0000 7fb33b7fe640 1 -- 172.21.15.16:0/1091152896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f29723cd0 con 0x560f2969b480 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb33b7fe640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb330000c00 tx=0x7fb330005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b0bdb3de2d61d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb31b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33000f8f0 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb33000fa90 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560f29607d00 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb33000fcd0 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb344687740 10 client.27451 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.281+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f29724fe0 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560f29607d00 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb33001cb90 con 0x560f2969b480 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 10 client.27451 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 10 client.27451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 20 client.27451 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f29725730 0x560f29727b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 4 client.27451 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:59:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560f29584110 con 0x560f29725730 2024-09-12T02:59:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.282+0000 7fb33bfff640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f29725730 0x560f29727b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.283+0000 7fb31b7fe640 10 client.27451 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.283+0000 7fb33bfff640 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f29725730 0x560f29727b50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2004c700b3376bc9 server_cookie=9053a38e84e19247 in_seq=0 out_seq=0 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb31b7fe640 1 -- 172.21.15.16:0/1091152896 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560f29584110 con 0x560f29725730 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb31b7fe640 10 client.27451 handle_command_reply: tid=0 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 client.27451 shutdown 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 msgr2=0x560f296ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb330000c00 tx=0x7fb330005b20 comp rx=0 tx=0).stop 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 shutdown_connections 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560f29725730 0x560f29727b50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f296a5d40 0x560f296aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f2969b480 0x560f296ae500 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 --2- 172.21.15.16:0/1091152896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f2969ab60 0x560f296adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb31b7fe640 0 client.27451 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 >> 172.21.15.16:0/1091152896 conn(0x560f296970f0 msgr2=0x560f296a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 shutdown_connections 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.284+0000 7fb344687740 1 -- 172.21.15.16:0/1091152896 wait complete. 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb344687740 20 client.27451 trim_cache size 0 max 16384 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:03.290 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "6156fe4d-7a33-4a3f-ac6e-14df46155f54", 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 msgr2=0x7fb3240a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 0x7fb3240a50c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb334021000 tx=0x7fb33401e000 comp rx=0 tx=0).stop 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 msgr2=0x7fb33c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009230 tx=0x7fb32c004270 comp rx=0 tx=0).stop 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 shutdown_connections 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb33c1082a0 0x7fb33c19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb33c1036e0 0x7fb33c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb3240a2c00 0x7fb3240a50c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 --2- 172.21.15.16:0/2963647359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb33c102d30 0x7fb33c198610 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 >> 172.21.15.16:0/2963647359 conn(0x7fb33c0fc7d0 msgr2=0x7fb33c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 shutdown_connections 2024-09-12T02:59:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:03.285+0000 7fb3422ae640 1 -- 172.21.15.16:0/2963647359 wait complete. 2024-09-12T02:59:03.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1100478 9.84 MB/sec execute 485 sec latency 52.046 ms 2024-09-12T02:59:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:03 smithi016 ceph-mon[30925]: pgmap v671: 97 pgs: 97 active+clean; 286 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:59:03.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:03 smithi033 ceph-mon[35092]: pgmap v671: 97 pgs: 97 active+clean; 286 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:59:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:03 smithi184 ceph-mon[35321]: pgmap v671: 97 pgs: 97 active+clean; 286 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 269 op/s 2024-09-12T02:59:04.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:59:04.388 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:04.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1102332 9.84 MB/sec execute 486 sec latency 16.903 ms 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:04.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:04 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 -- 172.21.15.16:0/9862943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90741082a0 msgr2=0x7f907410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 --2- 172.21.15.16:0/9862943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90741082a0 0x7f907410ec40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f906c009cd0 tx=0x7f906c02a8e0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 -- 172.21.15.16:0/9862943 shutdown_connections 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 --2- 172.21.15.16:0/9862943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90741082a0 0x7f907410ec40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 --2- 172.21.15.16:0/9862943 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741036e0 0x7f9074107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 --2- 172.21.15.16:0/9862943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9074102d30 0x7f9074103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 -- 172.21.15.16:0/9862943 >> 172.21.15.16:0/9862943 conn(0x7f90740fc7d0 msgr2=0x7f90740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 -- 172.21.15.16:0/9862943 shutdown_connections 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.905+0000 7f9079168640 1 -- 172.21.15.16:0/9862943 wait complete. 2024-09-12T02:59:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.906+0000 7f9079168640 1 Processor -- start 2024-09-12T02:59:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.907+0000 7f9079168640 1 -- start start 2024-09-12T02:59:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.907+0000 7f9079168640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 0x7f9074198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.907+0000 7f9079168640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 0x7f9074198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.907+0000 7f9079168640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741082a0 0x7f907419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9079168640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90741120f0 con 0x7f9074102d30 2024-09-12T02:59:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9079168640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9074111f70 con 0x7f90741036e0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9079168640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9074112270 con 0x7f90741082a0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 0x7f9074198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 0x7f9074198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 0x7f9074198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53254/0 (socket says 172.21.15.16:53254) 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f9072d9d640 1 -- 172.21.15.16:0/1004380102 learned_addr learned my addr 172.21.15.16:0/1004380102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907359e640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741082a0 0x7f907419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 -- 172.21.15.16:0/1004380102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741082a0 msgr2=0x7f907419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741082a0 0x7f907419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 -- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 msgr2=0x7f9074198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 0x7f9074198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 -- 172.21.15.16:0/1004380102 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907419d5e0 con 0x7f90741036e0 2024-09-12T02:59:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.908+0000 7f907259c640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 0x7f9074198b70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90740697d0 tx=0x7f906800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.909+0000 7f905bfff640 1 -- 172.21.15.16:0/1004380102 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906800cc40 con 0x7f90741036e0 2024-09-12T02:59:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.909+0000 7f905bfff640 1 -- 172.21.15.16:0/1004380102 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f906800cde0 con 0x7f90741036e0 2024-09-12T02:59:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.909+0000 7f905bfff640 1 -- 172.21.15.16:0/1004380102 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068004a00 con 0x7f90741036e0 2024-09-12T02:59:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.909+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907419d8d0 con 0x7f90741036e0 2024-09-12T02:59:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.909+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90741a51b0 con 0x7f90741036e0 2024-09-12T02:59:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.911+0000 7f905bfff640 1 -- 172.21.15.16:0/1004380102 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9068004ba0 con 0x7f90741036e0 2024-09-12T02:59:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 Processor -- start 2024-09-12T02:59:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 -- start start 2024-09-12T02:59:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680366880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5596802cc2e0 con 0x559680371680 2024-09-12T02:59:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559680273ce0 con 0x5596803664a0 2024-09-12T02:59:04.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907b541740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596802c82a0 con 0x559680366dc0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57964/0 (socket says 172.21.15.16:57964) 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53268/0 (socket says 172.21.15.16:53268) 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f9072d9d640 1 -- 172.21.15.16:0/2657399157 learned_addr learned my addr 172.21.15.16:0/2657399157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f905bfff640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 0x7f905c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f907359e640 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680366880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.912+0000 7f905bfff640 1 -- 172.21.15.16:0/1004380102 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f90680cef30 con 0x7f90741036e0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f907359e640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 0x7f905c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 -- 172.21.15.16:0/2657399157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 msgr2=0x559680366880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680366880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 -- 172.21.15.16:0/2657399157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 msgr2=0x559680373a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 -- 172.21.15.16:0/2657399157 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559680258440 con 0x559680366dc0 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f907259c640 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9064009dc0 tx=0x7f9064011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf18784c62a55088 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f907359e640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 0x7f905c0a5040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f906c005e30 tx=0x7f906c02a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f90597fa640 1 -- 172.21.15.16:0/2657399157 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064005220 con 0x559680366dc0 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f90597fa640 1 -- 172.21.15.16:0/2657399157 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9064004c80 con 0x559680366dc0 2024-09-12T02:59:04.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.913+0000 7f90597fa640 1 -- 172.21.15.16:0/2657399157 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064019930 con 0x559680366dc0 2024-09-12T02:59:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 -- 172.21.15.16:0/2657399157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 msgr2=0x559680371140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9064009dc0 tx=0x7f9064011810 comp rx=0 tx=0).stop 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 -- 172.21.15.16:0/2657399157 shutdown_connections 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x559680373a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x559680371140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 --2- 172.21.15.16:0/2657399157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680366880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 -- 172.21.15.16:0/2657399157 >> 172.21.15.16:0/2657399157 conn(0x559680362a30 msgr2=0x559680362e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 -- 172.21.15.16:0/2657399157 shutdown_connections 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.914+0000 7f907b541740 1 -- 172.21.15.16:0/2657399157 wait complete. 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 -- start start 2024-09-12T02:59:04.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 Processor -- start 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x55968036e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x55968036e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559680374250 con 0x559680371680 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559680287050 con 0x5596803664a0 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596803743d0 con 0x559680366dc0 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907b541740 10 client.0 _ll_register_callbacks cb 0x5596802c7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x55968036e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38216/0 (socket says 172.21.15.16:38216) 2024-09-12T02:59:04.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f9072d9d640 1 -- 172.21.15.16:0/815429456 learned_addr learned my addr 172.21.15.16:0/815429456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.915+0000 7f907259c640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x55968036e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 -- 172.21.15.16:0/815429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 msgr2=0x55968036e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x55968036e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 -- 172.21.15.16:0/815429456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 msgr2=0x55968036e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x55968036e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 -- 172.21.15.16:0/815429456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968003d3c0 con 0x5596803664a0 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f907359e640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x55968036e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9072d9d640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f906400a010 tx=0x7f9064004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4d81c8c1d5a80 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9056ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064027070 con 0x5596803664a0 2024-09-12T02:59:04.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9064004450 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.916+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9064004c80 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9064033050 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f907b541740 10 client.27569 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5596803883e0 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9064004c80 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9064019be0 con 0x5596803664a0 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f907b541740 10 client.27569 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.917+0000 7f907b541740 10 client.27569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907b541740 20 client.27569 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559680389790 0x55968038bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907b541740 4 client.27569 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596802d9850 con 0x559680389790 2024-09-12T02:59:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907259c640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559680389790 0x55968038bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f907259c640 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559680389790 0x55968038bb80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f282544ccad39026 server_cookie=233793e81ec1207c in_seq=0 out_seq=0 2024-09-12T02:59:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.918+0000 7f9056ff5640 10 client.27569 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f9056ff5640 1 -- 172.21.15.16:0/815429456 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5596802d9850 con 0x559680389790 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f9056ff5640 10 client.27569 handle_command_reply: tid=0 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f907b541740 1 client.27569 shutdown 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 msgr2=0x559680370fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f906400a010 tx=0x7f9064004970 comp rx=0 tx=0).stop 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 shutdown_connections 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.919+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559680389790 0x55968038bb80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680371680 0x55968036e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680366dc0 0x55968036e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f9056ff5640 0 client.27569 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 --2- 172.21.15.16:0/815429456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596803664a0 0x559680370fa0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 >> 172.21.15.16:0/815429456 conn(0x559680362a30 msgr2=0x559680368710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 shutdown_connections 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 1 -- 172.21.15.16:0/815429456 wait complete. 2024-09-12T02:59:04.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.920+0000 7f907b541740 20 client.27569 trim_cache size 0 max 16384 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.947+0000 7f907b541740 1 Processor -- start 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.947+0000 7f907b541740 1 -- start start 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 0x55968036e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 0x5596803912f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90741ab4a0 con 0x559680366dc0 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90741acfe0 con 0x559680371680 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907b541740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90741abb70 con 0x5596803664a0 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 0x5596803912f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 0x55968036e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53296/0 (socket says 172.21.15.16:53296) 2024-09-12T02:59:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907259c640 1 -- 172.21.15.16:0/536282858 learned_addr learned my addr 172.21.15.16:0/536282858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 -- 172.21.15.16:0/536282858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 msgr2=0x55968036e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 0x55968036e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 -- 172.21.15.16:0/536282858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 msgr2=0x559680369740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 -- 172.21.15.16:0/536282858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559680367a20 con 0x559680371680 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907259c640 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f9072d9d640 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 0x55968036e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.948+0000 7f907359e640 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 0x5596803912f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f906c009ca0 tx=0x7f906c005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94650e131b6c2e40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f9058ff9640 1 -- 172.21.15.16:0/536282858 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c031d70 con 0x559680371680 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f9058ff9640 1 -- 172.21.15.16:0/536282858 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f906c036040 con 0x559680371680 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f9058ff9640 1 -- 172.21.15.16:0/536282858 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c030e80 con 0x559680371680 2024-09-12T02:59:04.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 -- 172.21.15.16:0/536282858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 msgr2=0x5596803912f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 0x5596803912f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f906c009ca0 tx=0x7f906c005f70 comp rx=0 tx=0).stop 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 -- 172.21.15.16:0/536282858 shutdown_connections 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680371680 0x5596803912f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559680366dc0 0x559680369740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 --2- 172.21.15.16:0/536282858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5596803664a0 0x55968036e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.949+0000 7f907b541740 1 -- 172.21.15.16:0/536282858 >> 172.21.15.16:0/536282858 conn(0x559680362a30 msgr2=0x559680369030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- 172.21.15.16:0/536282858 shutdown_connections 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- 172.21.15.16:0/536282858 wait complete. 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- start start 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 Processor -- start 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 0x5596803821c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680366dc0 0x559680382700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90741ac910 con 0x5596803664a0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90741ac240 con 0x559680366dc0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.950+0000 7f907b541740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90741a9480 con 0x559680371680 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907b541740 10 client.0 _ll_register_callbacks cb 0x7f90680053b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 0x5596803821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58002/0 (socket says 172.21.15.16:58002) 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907359e640 1 -- 172.21.15.16:0/2422981629 learned_addr learned my addr 172.21.15.16:0/2422981629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907259c640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680366dc0 0x559680382700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 -- 172.21.15.16:0/2422981629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 msgr2=0x55968037d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 -- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680366dc0 msgr2=0x559680382700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680366dc0 0x559680382700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 -- 172.21.15.16:0/2422981629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968037ddb0 con 0x5596803664a0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907359e640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9072d9d640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 0x5596803821c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f90640036e0 tx=0x7f9064005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9212d497dc3d9d51 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9056ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064027070 con 0x5596803664a0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9064003e00 con 0x5596803664a0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.951+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f906c036040 con 0x5596803664a0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f90640099d0 con 0x5596803664a0 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 10 client.18156 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5596803f3e90 con 0x5596803664a0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f906c036040 con 0x5596803664a0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9064019430 con 0x5596803664a0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 10 client.18156 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 10 client.18156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 20 client.18156 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5596803f45a0 0x5596803f6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 4 client.18156 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.952+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5596802d3d60 con 0x5596803f45a0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907259c640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5596803f45a0 0x5596803f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f9056ff5640 10 client.18156 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907259c640 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5596803f45a0 0x5596803f6990 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb0adc1fdfff3f7 server_cookie=6350dfa697d487bf in_seq=0 out_seq=0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f9056ff5640 1 -- 172.21.15.16:0/2422981629 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5596802d3d60 con 0x5596803f45a0 2024-09-12T02:59:04.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f9056ff5640 10 client.18156 handle_command_reply: tid=0 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 client.18156 shutdown 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 msgr2=0x5596803821c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 0x5596803821c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f90640036e0 tx=0x7f9064005330 comp rx=0 tx=0).stop 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 shutdown_connections 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5596803f45a0 0x5596803f6990 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559680371680 0x55968037d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559680366dc0 0x559680382700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 --2- 172.21.15.16:0/2422981629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596803664a0 0x5596803821c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 >> 172.21.15.16:0/2422981629 conn(0x559680362a30 msgr2=0x559680361b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f9056ff5640 0 client.18156 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 shutdown_connections 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 1 -- 172.21.15.16:0/2422981629 wait complete. 2024-09-12T02:59:04.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.953+0000 7f907b541740 20 client.18156 trim_cache size 0 max 16384 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "6156fe4d-7a33-4a3f-ac6e-14df46155f54": { 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "6156fe4d-7a33-4a3f-ac6e-14df46155f54", 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:59:04.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:04.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:04.961 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:04.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 msgr2=0x7f905c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 0x7f905c0a5040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f906c005e30 tx=0x7f906c02a8e0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 msgr2=0x7f9074198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 0x7f9074198b70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90740697d0 tx=0x7f906800ee50 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 shutdown_connections 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90741082a0 0x7f907419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90741036e0 0x7f9074198b70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f905c0a2b80 0x7f905c0a5040 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 --2- 172.21.15.16:0/1004380102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9074102d30 0x7f9074198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 >> 172.21.15.16:0/1004380102 conn(0x7f90740fc7d0 msgr2=0x7f90740fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 shutdown_connections 2024-09-12T02:59:04.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:04.954+0000 7f9079168640 1 -- 172.21.15.16:0/1004380102 wait complete. 2024-09-12T02:59:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:59:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:59:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:59:04.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:04.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:59:04.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:04.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:04 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T02:59:05.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:05.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:04 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:05.676 INFO:tasks.cephfs.filesystem:scrub status for tag:6156fe4d-7a33-4a3f-ac6e-14df46155f54 - {'path': '/', 'tag': '6156fe4d-7a33-4a3f-ac6e-14df46155f54', 'options': 'recursive,force'} 2024-09-12T02:59:05.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1105099 9.85 MB/sec execute 487 sec latency 16.524 ms 2024-09-12T02:59:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:05 smithi016 ceph-mon[30925]: pgmap v672: 97 pgs: 97 active+clean; 283 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-12T02:59:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:59:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:59:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:05 smithi033 ceph-mon[35092]: pgmap v672: 97 pgs: 97 active+clean; 283 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-12T02:59:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:59:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:59:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:05 smithi184 ceph-mon[35321]: pgmap v672: 97 pgs: 97 active+clean; 283 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 255 op/s 2024-09-12T02:59:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T02:59:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T02:59:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T02:59:06.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1108042 9.86 MB/sec execute 488 sec latency 17.086 ms 2024-09-12T02:59:07.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1110638 9.87 MB/sec execute 489 sec latency 12.739 ms 2024-09-12T02:59:07.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:07 smithi016 ceph-mon[30925]: pgmap v673: 97 pgs: 97 active+clean; 268 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2024-09-12T02:59:07.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:07 smithi033 ceph-mon[35092]: pgmap v673: 97 pgs: 97 active+clean; 268 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2024-09-12T02:59:08.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:07 smithi184 ceph-mon[35321]: pgmap v673: 97 pgs: 97 active+clean; 268 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2024-09-12T02:59:08.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1112539 9.87 MB/sec execute 490 sec latency 37.925 ms 2024-09-12T02:59:09.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1115620 9.89 MB/sec execute 491 sec latency 18.661 ms 2024-09-12T02:59:09.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:09 smithi016 ceph-mon[30925]: pgmap v674: 97 pgs: 97 active+clean; 271 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-09-12T02:59:09.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:09 smithi033 ceph-mon[35092]: pgmap v674: 97 pgs: 97 active+clean; 271 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-09-12T02:59:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:09 smithi184 ceph-mon[35321]: pgmap v674: 97 pgs: 97 active+clean; 271 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-09-12T02:59:10.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1118663 9.90 MB/sec execute 492 sec latency 10.875 ms 2024-09-12T02:59:11.739 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1121326 9.90 MB/sec execute 493 sec latency 12.374 ms 2024-09-12T02:59:11.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:11 smithi033 ceph-mon[35092]: pgmap v675: 97 pgs: 97 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:59:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:11 smithi184 ceph-mon[35321]: pgmap v675: 97 pgs: 97 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:59:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:11 smithi016 ceph-mon[30925]: pgmap v675: 97 pgs: 97 active+clean; 274 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 299 op/s 2024-09-12T02:59:12.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1123587 9.91 MB/sec execute 494 sec latency 12.107 ms 2024-09-12T02:59:12.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:12 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:59:13.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:12 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:59:13.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:12 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:59:13.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1126281 9.93 MB/sec execute 495 sec latency 91.997 ms 2024-09-12T02:59:13.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:13 smithi033 ceph-mon[35092]: pgmap v676: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-12T02:59:13.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:13 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:59:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:13 smithi184 ceph-mon[35321]: pgmap v676: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-12T02:59:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:13 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:59:14.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:13 smithi016 ceph-mon[30925]: pgmap v676: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 298 op/s 2024-09-12T02:59:14.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:13 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:59:14.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1128895 9.93 MB/sec execute 496 sec latency 17.058 ms 2024-09-12T02:59:15.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1131148 9.93 MB/sec execute 497 sec latency 14.591 ms 2024-09-12T02:59:15.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:15 smithi033 ceph-mon[35092]: pgmap v677: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-09-12T02:59:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:15 smithi184 ceph-mon[35321]: pgmap v677: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-09-12T02:59:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:15 smithi016 ceph-mon[30925]: pgmap v677: 97 pgs: 97 active+clean; 275 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 259 op/s 2024-09-12T02:59:16.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1133722 9.94 MB/sec execute 498 sec latency 9.078 ms 2024-09-12T02:59:17.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1136761 9.96 MB/sec execute 499 sec latency 12.605 ms 2024-09-12T02:59:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:17 smithi033 ceph-mon[35092]: pgmap v678: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 305 op/s 2024-09-12T02:59:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:17 smithi184 ceph-mon[35321]: pgmap v678: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 305 op/s 2024-09-12T02:59:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:17 smithi016 ceph-mon[30925]: pgmap v678: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 305 op/s 2024-09-12T02:59:18.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1138569 9.95 MB/sec execute 500 sec latency 67.775 ms 2024-09-12T02:59:19.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1140800 9.96 MB/sec execute 501 sec latency 26.398 ms 2024-09-12T02:59:19.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:19 smithi033 ceph-mon[35092]: pgmap v679: 97 pgs: 97 active+clean; 276 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-12T02:59:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:19 smithi184 ceph-mon[35321]: pgmap v679: 97 pgs: 97 active+clean; 276 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-12T02:59:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:19 smithi016 ceph-mon[30925]: pgmap v679: 97 pgs: 97 active+clean; 276 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-09-12T02:59:20.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1143862 9.98 MB/sec execute 502 sec latency 11.467 ms 2024-09-12T02:59:21.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1146194 9.97 MB/sec execute 503 sec latency 12.333 ms 2024-09-12T02:59:21.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:21 smithi033 ceph-mon[35092]: pgmap v680: 97 pgs: 97 active+clean; 282 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:59:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:21 smithi184 ceph-mon[35321]: pgmap v680: 97 pgs: 97 active+clean; 282 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:59:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:21 smithi016 ceph-mon[30925]: pgmap v680: 97 pgs: 97 active+clean; 282 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2024-09-12T02:59:22.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1148510 9.98 MB/sec execute 504 sec latency 12.104 ms 2024-09-12T02:59:23.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1151289 9.99 MB/sec execute 505 sec latency 45.557 ms 2024-09-12T02:59:23.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:23 smithi033 ceph-mon[35092]: pgmap v681: 97 pgs: 97 active+clean; 281 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-09-12T02:59:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:23 smithi184 ceph-mon[35321]: pgmap v681: 97 pgs: 97 active+clean; 281 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-09-12T02:59:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:23 smithi016 ceph-mon[30925]: pgmap v681: 97 pgs: 97 active+clean; 281 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-09-12T02:59:24.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1154042 10.00 MB/sec execute 506 sec latency 40.503 ms 2024-09-12T02:59:25.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1156076 10.00 MB/sec execute 507 sec latency 67.745 ms 2024-09-12T02:59:25.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:25 smithi033 ceph-mon[35092]: pgmap v682: 97 pgs: 97 active+clean; 283 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:59:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:25 smithi184 ceph-mon[35321]: pgmap v682: 97 pgs: 97 active+clean; 283 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:59:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:25 smithi016 ceph-mon[30925]: pgmap v682: 97 pgs: 97 active+clean; 283 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-09-12T02:59:26.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1158718 10.01 MB/sec execute 508 sec latency 15.826 ms 2024-09-12T02:59:27.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1161867 10.03 MB/sec execute 509 sec latency 10.102 ms 2024-09-12T02:59:27.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:27 smithi033 ceph-mon[35092]: pgmap v683: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 312 op/s 2024-09-12T02:59:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:27 smithi184 ceph-mon[35321]: pgmap v683: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 312 op/s 2024-09-12T02:59:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:27 smithi016 ceph-mon[30925]: pgmap v683: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 312 op/s 2024-09-12T02:59:28.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1165027 10.04 MB/sec execute 510 sec latency 14.486 ms 2024-09-12T02:59:29.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1167158 10.04 MB/sec execute 511 sec latency 64.712 ms 2024-09-12T02:59:29.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:29 smithi033 ceph-mon[35092]: pgmap v684: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-09-12T02:59:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:29 smithi184 ceph-mon[35321]: pgmap v684: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-09-12T02:59:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:29 smithi016 ceph-mon[30925]: pgmap v684: 97 pgs: 97 active+clean; 287 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-09-12T02:59:30.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1169511 10.05 MB/sec execute 512 sec latency 16.131 ms 2024-09-12T02:59:31.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1172500 10.06 MB/sec execute 513 sec latency 15.022 ms 2024-09-12T02:59:31.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:31 smithi033 ceph-mon[35092]: pgmap v685: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-12T02:59:32.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:31 smithi184 ceph-mon[35321]: pgmap v685: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-12T02:59:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:31 smithi016 ceph-mon[30925]: pgmap v685: 97 pgs: 97 active+clean; 277 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 303 op/s 2024-09-12T02:59:32.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1175283 10.07 MB/sec execute 514 sec latency 15.672 ms 2024-09-12T02:59:33.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1177349 10.07 MB/sec execute 515 sec latency 12.437 ms 2024-09-12T02:59:33.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:33 smithi033 ceph-mon[35092]: pgmap v686: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 267 op/s 2024-09-12T02:59:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:33 smithi184 ceph-mon[35321]: pgmap v686: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 267 op/s 2024-09-12T02:59:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:33 smithi016 ceph-mon[30925]: pgmap v686: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 267 op/s 2024-09-12T02:59:34.740 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1179618 10.08 MB/sec execute 516 sec latency 61.030 ms 2024-09-12T02:59:35.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:59:35.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1182179 10.08 MB/sec execute 517 sec latency 32.503 ms 2024-09-12T02:59:35.944 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:35.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:35 smithi033 ceph-mon[35092]: pgmap v687: 97 pgs: 97 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 290 op/s 2024-09-12T02:59:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:35 smithi184 ceph-mon[35321]: pgmap v687: 97 pgs: 97 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 290 op/s 2024-09-12T02:59:36.065 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:35 smithi016 ceph-mon[30925]: pgmap v687: 97 pgs: 97 active+clean; 275 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 290 op/s 2024-09-12T02:59:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 -- 172.21.15.16:0/1528133248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102570 msgr2=0x7f30781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 --2- 172.21.15.16:0/1528133248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102570 0x7f30781029b0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f306c009900 tx=0x7f306c02a480 comp rx=0 tx=0).stop 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 -- 172.21.15.16:0/1528133248 shutdown_connections 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 --2- 172.21.15.16:0/1528133248 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102ef0 0x7f307810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 --2- 172.21.15.16:0/1528133248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102570 0x7f30781029b0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 --2- 172.21.15.16:0/1528133248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f3078101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 -- 172.21.15.16:0/1528133248 >> 172.21.15.16:0/1528133248 conn(0x7f30780fc7d0 msgr2=0x7f30780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 -- 172.21.15.16:0/1528133248 shutdown_connections 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.439+0000 7f307e22a640 1 -- 172.21.15.16:0/1528133248 wait complete. 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 Processor -- start 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 -- start start 2024-09-12T02:59:36.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102570 0x7f30780fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 0x7f30780fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3078110fd0 con 0x7f3078102ef0 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3078110e50 con 0x7f3078100e20 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f307e22a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3078111150 con 0x7f3078102570 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f30777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102570 0x7f30780fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f3077fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 0x7f30780fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f30777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43924/0 (socket says 172.21.15.16:43924) 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.440+0000 7f30777fe640 1 -- 172.21.15.16:0/4177213580 learned_addr learned my addr 172.21.15.16:0/4177213580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 -- 172.21.15.16:0/4177213580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102570 msgr2=0x7f30780fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102570 0x7f30780fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 -- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 msgr2=0x7f30780fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 0x7f30780fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 -- 172.21.15.16:0/4177213580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30781a69e0 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f3077fff640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 0x7f30780fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f30777fe640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f306800e9f0 tx=0x7f306800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f3074ff9640 1 -- 172.21.15.16:0/4177213580 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306800cd80 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f3074ff9640 1 -- 172.21.15.16:0/4177213580 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3068010910 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30781a6cd0 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f3074ff9640 1 -- 172.21.15.16:0/4177213580 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068018990 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.441+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30781a7190 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3074ff9640 1 -- 172.21.15.16:0/4177213580 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3068010ac0 con 0x7f3078100e20 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 Processor -- start 2024-09-12T02:59:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 -- start start 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b48880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4aae2e0 con 0x55a2a4b484a0 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4a55ce0 con 0x55a2a4b53680 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3080603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4aaa2a0 con 0x55a2a4b48dc0 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3074ff9640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 0x7f30600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f30777fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f3077fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b48880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43926/0 (socket says 172.21.15.16:43926) 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 -- 172.21.15.16:0/3848168193 learned_addr learned my addr 172.21.15.16:0/3848168193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.444+0000 7f30777fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42174/0 (socket says 172.21.15.16:42174) 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3074ff9640 1 -- 172.21.15.16:0/4177213580 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f30680cf380 con 0x7f3078100e20 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3077fff640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 0x7f30600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 -- 172.21.15.16:0/3848168193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 msgr2=0x55a2a4b53140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 -- 172.21.15.16:0/3848168193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 msgr2=0x55a2a4b48880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b48880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 -- 172.21.15.16:0/3848168193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2a4a3a440 con 0x55a2a4b53680 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3077fff640 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b48880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f30777fe640 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3076ffd640 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f306c009230 tx=0x7f306c005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1756e183b48415c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f305dffb640 1 -- 172.21.15.16:0/3848168193 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c035040 con 0x55a2a4b53680 2024-09-12T02:59:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f3077fff640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 0x7f30600a5040 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f30700099e0 tx=0x7f3070013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f305dffb640 1 -- 172.21.15.16:0/3848168193 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f306c004400 con 0x55a2a4b53680 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.445+0000 7f305dffb640 1 -- 172.21.15.16:0/3848168193 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c0035f0 con 0x55a2a4b53680 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 -- 172.21.15.16:0/3848168193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 msgr2=0x55a2a4b55a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f306c009230 tx=0x7f306c005e00 comp rx=0 tx=0).stop 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 -- 172.21.15.16:0/3848168193 shutdown_connections 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b55a70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b53140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 --2- 172.21.15.16:0/3848168193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b48880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 -- 172.21.15.16:0/3848168193 >> 172.21.15.16:0/3848168193 conn(0x55a2a4b44a30 msgr2=0x55a2a4b44e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 -- 172.21.15.16:0/3848168193 shutdown_connections 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.446+0000 7f3080603740 1 -- 172.21.15.16:0/3848168193 wait complete. 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 -- start start 2024-09-12T02:59:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 Processor -- start 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b50210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b50750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4b56250 con 0x55a2a4b48dc0 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4a69050 con 0x55a2a4b484a0 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a2a4b563d0 con 0x55a2a4b53680 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3080603740 10 client.0 _ll_register_callbacks cb 0x55a2a4aa9160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b50210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f30777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f30777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43938/0 (socket says 172.21.15.16:43938) 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b50210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39504/0 (socket says 172.21.15.16:39504) 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f30777fe640 1 -- 172.21.15.16:0/2049379847 learned_addr learned my addr 172.21.15.16:0/2049379847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.447+0000 7f3077fff640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b50750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 -- 172.21.15.16:0/2049379847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 msgr2=0x55a2a4b50750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b50750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 -- 172.21.15.16:0/2049379847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 msgr2=0x55a2a4b50210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b50210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 -- 172.21.15.16:0/2049379847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2a481f3c0 con 0x55a2a4b484a0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3077fff640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b50750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f30777fe640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f306800c6a0 tx=0x7f306809b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b6d1ae00255ea4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068026040 con 0x55a2a4b484a0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f306807b070 con 0x55a2a4b484a0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f306c004400 con 0x55a2a4b484a0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f30680a4800 con 0x55a2a4b484a0 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3080603740 10 client.27587 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.448+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a2a4b61c40 con 0x55a2a4b484a0 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f306c004400 con 0x55a2a4b484a0 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f30680ce030 con 0x55a2a4b484a0 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 10 client.27587 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 10 client.27587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 20 client.27587 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4b63100 0x55a2a4b65550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 4 client.27587 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2a4abb850 con 0x55a2a4b63100 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3076ffd640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4b63100 0x55a2a4b65550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3056ffd640 10 client.27587 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.449+0000 7f3076ffd640 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4b63100 0x55a2a4b65550 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84800cb9d388ed93 server_cookie=92ad4db894e042d4 in_seq=0 out_seq=0 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3056ffd640 1 -- 172.21.15.16:0/2049379847 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a2a4abb850 con 0x55a2a4b63100 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3056ffd640 10 client.27587 handle_command_reply: tid=0 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 client.27587 shutdown 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 msgr2=0x55a2a4b52f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f306800c6a0 tx=0x7f306809b5a0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 shutdown_connections 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4b63100 0x55a2a4b65550 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b50750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b50210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 --2- 172.21.15.16:0/2049379847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b484a0 0x55a2a4b52f60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3056ffd640 0 client.27587 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 >> 172.21.15.16:0/2049379847 conn(0x55a2a4b44a30 msgr2=0x55a2a4b4a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 shutdown_connections 2024-09-12T02:59:36.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.450+0000 7f3080603740 1 -- 172.21.15.16:0/2049379847 wait complete. 2024-09-12T02:59:36.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.451+0000 7f3080603740 20 client.27587 trim_cache size 0 max 16384 2024-09-12T02:59:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 Processor -- start 2024-09-12T02:59:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 -- start start 2024-09-12T02:59:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b4b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b43a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30781af710 con 0x55a2a4b484a0 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f30781b1250 con 0x55a2a4b53680 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3080603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f30781afde0 con 0x55a2a4b48dc0 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f30777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f30777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39508/0 (socket says 172.21.15.16:39508) 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f30777fe640 1 -- 172.21.15.16:0/76499191 learned_addr learned my addr 172.21.15.16:0/76499191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3076ffd640 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b4b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.469+0000 7f3077fff640 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b43a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 -- 172.21.15.16:0/76499191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 msgr2=0x55a2a4b4b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b4b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 -- 172.21.15.16:0/76499191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 msgr2=0x55a2a4b43a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b43a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 -- 172.21.15.16:0/76499191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2a4b444d0 con 0x55a2a4b484a0 2024-09-12T02:59:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f30777fe640 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3068018da0 tx=0x7f3068097840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecf99854395368a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f305d7fa640 1 -- 172.21.15.16:0/76499191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306809bc40 con 0x55a2a4b484a0 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f305d7fa640 1 -- 172.21.15.16:0/76499191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f306807b0b0 con 0x55a2a4b484a0 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.470+0000 7f305d7fa640 1 -- 172.21.15.16:0/76499191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068097bf0 con 0x55a2a4b484a0 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- 172.21.15.16:0/76499191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 msgr2=0x55a2a4b5f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3068018da0 tx=0x7f3068097840 comp rx=0 tx=0).stop 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- 172.21.15.16:0/76499191 shutdown_connections 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b53680 0x55a2a4b43a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b4b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 --2- 172.21.15.16:0/76499191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b5f4e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- 172.21.15.16:0/76499191 >> 172.21.15.16:0/76499191 conn(0x55a2a4b44a30 msgr2=0x55a2a4b43350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- 172.21.15.16:0/76499191 shutdown_connections 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- 172.21.15.16:0/76499191 wait complete. 2024-09-12T02:59:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.471+0000 7f3080603740 1 -- start start 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 Processor -- start 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b57f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30781b0b80 con 0x55a2a4b484a0 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f30781b04b0 con 0x55a2a4b48dc0 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f30781ad6f0 con 0x55a2a4b53680 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3080603740 10 client.0 _ll_register_callbacks cb 0x7f306800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f30777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f30777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39520/0 (socket says 172.21.15.16:39520) 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3076ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f30777fe640 1 -- 172.21.15.16:0/2124934413 learned_addr learned my addr 172.21.15.16:0/2124934413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f3077fff640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b57f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f30777fe640 1 -- 172.21.15.16:0/2124934413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 msgr2=0x55a2a4b57f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.472+0000 7f30777fe640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b57f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f30777fe640 1 -- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 msgr2=0x55a2a4b579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f30777fe640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f30777fe640 1 -- 172.21.15.16:0/2124934413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2a4bcce70 con 0x55a2a4b484a0 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3076ffd640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f30777fe640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f3068000c00 tx=0x7f306809b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c063831c08de8fed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068082060 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3068010430 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f306807b0b0 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f30680a9050 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3080603740 10 client.18174 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.473+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a2a4bcce70 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f306807b0b0 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f30680ce070 con 0x55a2a4b484a0 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 10 client.18174 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 20 client.18174 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4bce3e0 0x55a2a4bd07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 4 client.18174 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2a4ab5d60 con 0x55a2a4bce3e0 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.474+0000 7f3076ffd640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4bce3e0 0x55a2a4bd07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3056ffd640 10 client.18174 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3076ffd640 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4bce3e0 0x55a2a4bd07d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aba6310b29258b70 server_cookie=c559a879ed247c30 in_seq=0 out_seq=0 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3056ffd640 1 -- 172.21.15.16:0/2124934413 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a2a4ab5d60 con 0x55a2a4bce3e0 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3056ffd640 10 client.18174 handle_command_reply: tid=0 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 client.18174 shutdown 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 msgr2=0x55a2a4b57490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f3068000c00 tx=0x7f306809b440 comp rx=0 tx=0).stop 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 shutdown_connections 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a2a4bce3e0 0x55a2a4bd07d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a2a4b53680 0x55a2a4b57f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a2a4b48dc0 0x55a2a4b579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 --2- 172.21.15.16:0/2124934413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2a4b484a0 0x55a2a4b57490 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 >> 172.21.15.16:0/2124934413 conn(0x55a2a4b44a30 msgr2=0x55a2a4b43b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.475+0000 7f3056ffd640 0 client.18174 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 shutdown_connections 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f3080603740 1 -- 172.21.15.16:0/2124934413 wait complete. 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f3080603740 20 client.18174 trim_cache size 0 max 16384 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T02:59:36.481 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T02:59:36.482 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 msgr2=0x7f30600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 0x7f30600a5040 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f30700099e0 tx=0x7f3070013040 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 msgr2=0x7f30780fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f306800e9f0 tx=0x7f306800eeb0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 shutdown_connections 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f30600a2b80 0x7f30600a5040 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3078102ef0 0x7f30780fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102570 0x7f30780fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 --2- 172.21.15.16:0/4177213580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3078100e20 0x7f30780fcc80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.476+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 >> 172.21.15.16:0/4177213580 conn(0x7f30780fc7d0 msgr2=0x7f307806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.477+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 shutdown_connections 2024-09-12T02:59:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:36.477+0000 7f307e22a640 1 -- 172.21.15.16:0/4177213580 wait complete. 2024-09-12T02:59:36.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1184331 10.09 MB/sec execute 518 sec latency 12.908 ms 2024-09-12T02:59:37.284 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T02:59:37.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:59:37.549 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:37.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1186957 10.09 MB/sec execute 519 sec latency 17.757 ms 2024-09-12T02:59:37.968 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:37 smithi016 ceph-mon[30925]: pgmap v688: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 303 op/s 2024-09-12T02:59:37.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:37 smithi033 ceph-mon[35092]: pgmap v688: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 303 op/s 2024-09-12T02:59:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:37 smithi184 ceph-mon[35321]: pgmap v688: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 303 op/s 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 -- 172.21.15.16:0/55572690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc108290 msgr2=0x7f33bc10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 --2- 172.21.15.16:0/55572690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc108290 0x7f33bc10ec30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009cd0 tx=0x7f33b402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 -- 172.21.15.16:0/55572690 shutdown_connections 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 --2- 172.21.15.16:0/55572690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc108290 0x7f33bc10ec30 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 --2- 172.21.15.16:0/55572690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc103740 0x7f33bc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 --2- 172.21.15.16:0/55572690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc102d90 0x7f33bc103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 -- 172.21.15.16:0/55572690 >> 172.21.15.16:0/55572690 conn(0x7f33bc0fc7d0 msgr2=0x7f33bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 -- 172.21.15.16:0/55572690 shutdown_connections 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.056+0000 7f33c1097640 1 -- 172.21.15.16:0/55572690 wait complete. 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 Processor -- start 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 -- start start 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc103740 0x7f33bc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc108290 0x7f33bc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc112090 con 0x7f33bc102d90 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc111f10 con 0x7f33bc103740 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33c1097640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc112210 con 0x7f33bc108290 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.057+0000 7f33bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39530/0 (socket says 172.21.15.16:39530) 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 -- 172.21.15.16:0/3290474960 learned_addr learned my addr 172.21.15.16:0/3290474960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bb59e640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc108290 0x7f33bc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33ba59c640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc103740 0x7f33bc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 -- 172.21.15.16:0/3290474960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc108290 msgr2=0x7f33bc19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc108290 0x7f33bc19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 -- 172.21.15.16:0/3290474960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc103740 msgr2=0x7f33bc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc103740 0x7f33bc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 -- 172.21.15.16:0/3290474960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc19d500 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33bad9d640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00ea60 tx=0x7f33ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac00cde0 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc19d7f0 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33ac0035f0 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac010670 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.058+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33bc1a50d0 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.059+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33bc103170 con 0x7f33bc102d90 2024-09-12T02:59:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.061+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f33ac0037a0 con 0x7f33bc102d90 2024-09-12T02:59:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.061+0000 7f33a3fff640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 0x7f33a40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.061+0000 7f33ba59c640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 0x7f33a40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.061+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f33ac0cf2e0 con 0x7f33bc102d90 2024-09-12T02:59:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.062+0000 7f33ba59c640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 0x7f33a40a5040 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0feda0 tx=0x7f33b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.066+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f33ac09c250 con 0x7f33bc102d90 2024-09-12T02:59:38.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.299+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33bc10a2a0 con 0x7f33bc102d90 2024-09-12T02:59:38.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.300+0000 7f33a3fff640 1 -- 172.21.15.16:0/3290474960 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f33ac09bb50 con 0x7f33bc102d90 2024-09-12T02:59:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:59:38.302 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:59:38.302 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 msgr2=0x7f33a40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 0x7f33a40a5040 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0feda0 tx=0x7f33b0005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 msgr2=0x7f33bc198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00ea60 tx=0x7f33ac00ef20 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 shutdown_connections 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.302+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33bc108290 0x7f33bc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33bc103740 0x7f33bc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f33a40a2b80 0x7f33a40a5040 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 --2- 172.21.15.16:0/3290474960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33bc102d90 0x7f33bc198550 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:38.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 >> 172.21.15.16:0/3290474960 conn(0x7f33bc0fc7d0 msgr2=0x7f33bc106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 shutdown_connections 2024-09-12T02:59:38.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:38.303+0000 7f33c1097640 1 -- 172.21.15.16:0/3290474960 wait complete. 2024-09-12T02:59:38.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1190050 10.11 MB/sec execute 520 sec latency 11.550 ms 2024-09-12T02:59:38.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:38 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3290474960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:38 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3290474960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:38 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3290474960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:39.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T02:59:39.605 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:39.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1192133 10.10 MB/sec execute 521 sec latency 114.159 ms 2024-09-12T02:59:39.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:39 smithi033 ceph-mon[35092]: pgmap v689: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-09-12T02:59:40.038 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:39 smithi016 ceph-mon[30925]: pgmap v689: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-09-12T02:59:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:39 smithi184 ceph-mon[35321]: pgmap v689: 97 pgs: 97 active+clean; 272 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 -- 172.21.15.16:0/2616483208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 msgr2=0x7fabf01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 --2- 172.21.15.16:0/2616483208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01067e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009900 tx=0x7fabe402a410 comp rx=0 tx=0).stop 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 -- 172.21.15.16:0/2616483208 shutdown_connections 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 --2- 172.21.15.16:0/2616483208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0106f10 0x7fabf010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 --2- 172.21.15.16:0/2616483208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01067e0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 --2- 172.21.15.16:0/2616483208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0101de0 0x7fabf01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 -- 172.21.15.16:0/2616483208 >> 172.21.15.16:0/2616483208 conn(0x7fabf00fd8c0 msgr2=0x7fabf00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 -- 172.21.15.16:0/2616483208 shutdown_connections 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.125+0000 7fabf7a88640 1 -- 172.21.15.16:0/2616483208 wait complete. 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 Processor -- start 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 -- start start 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 0x7fabf019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0111f70 con 0x7fabf0102790 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0111df0 con 0x7fabf0101de0 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.126+0000 7fabf7a88640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01120f0 con 0x7fabf0106f10 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5824640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5824640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43988/0 (socket says 172.21.15.16:43988) 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5824640 1 -- 172.21.15.16:0/1905942199 learned_addr learned my addr 172.21.15.16:0/1905942199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39546/0 (socket says 172.21.15.16:39546) 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf6025640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 0x7fabf019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 -- 172.21.15.16:0/1905942199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 msgr2=0x7fabf019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 0x7fabf019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 -- 172.21.15.16:0/1905942199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 msgr2=0x7fabf01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 -- 172.21.15.16:0/1905942199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf019d460 con 0x7fabf0102790 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5824640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf6025640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 0x7fabf019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf5023640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fabe4005940 tx=0x7fabe40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabe2ffd640 1 -- 172.21.15.16:0/1905942199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe403f030 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf019d6f0 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabe2ffd640 1 -- 172.21.15.16:0/1905942199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe40044c0 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabe2ffd640 1 -- 172.21.15.16:0/1905942199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4035070 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.127+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf01a5030 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.130+0000 7fabf9e61740 1 Processor -- start 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.130+0000 7fabf9e61740 1 -- start start 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.130+0000 7fabe2ffd640 1 -- 172.21.15.16:0/1905942199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fabe4042030 con 0x7fabf0102790 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f561fca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f56209760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f561622f0 con 0x560f561fdb40 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f56109cf0 con 0x560f56207370 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf9e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f5615e2b0 con 0x560f561fc5d0 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf5824640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f561fca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39560/0 (socket says 172.21.15.16:39560) 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 -- 172.21.15.16:0/3626106548 learned_addr learned my addr 172.21.15.16:0/3626106548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf5023640 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f56209760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf5824640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f561fca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42220/0 (socket says 172.21.15.16:42220) 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 -- 172.21.15.16:0/3626106548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 msgr2=0x560f561fca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f561fca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 -- 172.21.15.16:0/3626106548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 msgr2=0x560f56209760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f56209760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 -- 172.21.15.16:0/3626106548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f560ee3a0 con 0x560f561fdb40 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf5023640 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f56209760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabe2ffd640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 0x7fabdc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf6025640 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fabe8011790 tx=0x7fabe8011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382d653d77ee7933 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.131+0000 7fabf5824640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 0x7fabdc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabe2ffd640 1 -- 172.21.15.16:0/1905942199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fabe40ef300 con 0x7fabf0102790 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabdbfff640 1 -- 172.21.15.16:0/3626106548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8003c30 con 0x560f561fdb40 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabdbfff640 1 -- 172.21.15.16:0/3626106548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe800fe80 con 0x560f561fdb40 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabdbfff640 1 -- 172.21.15.16:0/3626106548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe80193e0 con 0x560f561fdb40 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabf9e61740 1 -- 172.21.15.16:0/3626106548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 msgr2=0x560f561fc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabf5824640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 0x7fabdc0a5040 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fabec009fd0 tx=0x7fabec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.132+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fabe8011790 tx=0x7fabe8011c50 comp rx=0 tx=0).stop 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 -- 172.21.15.16:0/3626106548 shutdown_connections 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f56209760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f561fca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3626106548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f561fc090 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 -- 172.21.15.16:0/3626106548 >> 172.21.15.16:0/3626106548 conn(0x560f561c5310 msgr2=0x560f561f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 -- 172.21.15.16:0/3626106548 shutdown_connections 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 -- 172.21.15.16:0/3626106548 wait complete. 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 -- start start 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.133+0000 7fabf9e61740 1 Processor -- start 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56204000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f562045d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560f5620a070 con 0x560f561fdb40 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560f5611d060 con 0x560f56207370 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560f5620a1f0 con 0x560f561fc5d0 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf9e61740 10 client.0 _ll_register_callbacks cb 0x560f55e708c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5023640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56204000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42222/0 (socket says 172.21.15.16:42222) 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5023640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56204000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39566/0 (socket says 172.21.15.16:39566) 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5023640 1 -- 172.21.15.16:0/3157267282 learned_addr learned my addr 172.21.15.16:0/3157267282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf6025640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f562045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 -- 172.21.15.16:0/3157267282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 msgr2=0x560f562045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f562045d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 -- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 msgr2=0x560f56204000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56204000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.134+0000 7fabf5824640 1 -- 172.21.15.16:0/3157267282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f55ed33c0 con 0x560f561fc5d0 2024-09-12T02:59:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf5824640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fabec009640 tx=0x7fabec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacb626a322a1125 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec011940 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fabec011ae0 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560f5616f860 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fabec011ce0 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 10 client.27505 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f56215b50 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560f5616f860 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fabec027070 con 0x560f561fc5d0 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 10 client.27505 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:40.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 10 client.27505 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 20 client.27505 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56217010 0x560f56219440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 4 client.27505 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.135+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f56169d70 con 0x560f56217010 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.136+0000 7fabf5023640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56217010 0x560f56219440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.136+0000 7fabf5023640 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56217010 0x560f56219440 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba0f89c6bc017491 server_cookie=6869812eab9bd649 in_seq=0 out_seq=0 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.136+0000 7fabd97fa640 10 client.27505 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.137+0000 7fabd97fa640 1 -- 172.21.15.16:0/3157267282 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f56169d70 con 0x560f56217010 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.137+0000 7fabd97fa640 10 client.27505 handle_command_reply: tid=0 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.137+0000 7fabf9e61740 1 client.27505 shutdown 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.137+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 msgr2=0x560f56206d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.137+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fabec009640 tx=0x7fabec004730 comp rx=0 tx=0).stop 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 shutdown_connections 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56217010 0x560f56219440 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f562045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56204000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3157267282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f56206d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 >> 172.21.15.16:0/3157267282 conn(0x560f561c5310 msgr2=0x560f561fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabd97fa640 0 client.27505 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 shutdown_connections 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 1 -- 172.21.15.16:0/3157267282 wait complete. 2024-09-12T02:59:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.138+0000 7fabf9e61740 20 client.27505 trim_cache size 0 max 16384 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 Processor -- start 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 -- start start 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 0x560f56213310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56205f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 0x560f562064b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01ad590 con 0x560f561fdb40 2024-09-12T02:59:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01af0d0 con 0x560f561fc5d0 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf9e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01adc60 con 0x560f56207370 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf6025640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 0x560f562064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf6025640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 0x560f562064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42226/0 (socket says 172.21.15.16:42226) 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf5824640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 0x560f56213310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf6025640 1 -- 172.21.15.16:0/3301470500 learned_addr learned my addr 172.21.15.16:0/3301470500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.153+0000 7fabf5023640 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56205f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 -- 172.21.15.16:0/3301470500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 msgr2=0x560f562064b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 0x560f562064b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 -- 172.21.15.16:0/3301470500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 msgr2=0x560f56205f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56205f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 -- 172.21.15.16:0/3301470500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f56202550 con 0x560f561fc5d0 2024-09-12T02:59:40.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabf5824640 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 0x560f56213310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fabec034640 tx=0x7fabec0092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb8c662f8f0d377 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabdb7fe640 1 -- 172.21.15.16:0/3301470500 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec038030 con 0x560f561fc5d0 2024-09-12T02:59:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabdb7fe640 1 -- 172.21.15.16:0/3301470500 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabec02f030 con 0x560f561fc5d0 2024-09-12T02:59:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.154+0000 7fabdb7fe640 1 -- 172.21.15.16:0/3301470500 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec029440 con 0x560f561fc5d0 2024-09-12T02:59:40.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- 172.21.15.16:0/3301470500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 msgr2=0x560f56213310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 0x560f56213310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fabec034640 tx=0x7fabec0092a0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- 172.21.15.16:0/3301470500 shutdown_connections 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f56207370 0x560f562064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f56205f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3301470500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f561fc5d0 0x560f56213310 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- 172.21.15.16:0/3301470500 >> 172.21.15.16:0/3301470500 conn(0x560f561c5310 msgr2=0x560f56205860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- 172.21.15.16:0/3301470500 shutdown_connections 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- 172.21.15.16:0/3301470500 wait complete. 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.155+0000 7fabf9e61740 1 -- start start 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 Processor -- start 2024-09-12T02:59:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f5620bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f5620c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01aea00 con 0x560f561fdb40 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01ae330 con 0x560f56207370 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabf01ab570 con 0x560f561fc5d0 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf9e61740 10 client.0 _ll_register_callbacks cb 0x7fabec00f760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf6025640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf5824640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f5620bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf6025640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44028/0 (socket says 172.21.15.16:44028) 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf6025640 1 -- 172.21.15.16:0/3871126140 learned_addr learned my addr 172.21.15.16:0/3871126140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.156+0000 7fabf5023640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f5620c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 -- 172.21.15.16:0/3871126140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 msgr2=0x560f5627ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 -- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 msgr2=0x560f5620c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f5620c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 -- 172.21.15.16:0/3871126140 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5627b3a0 con 0x560f561fc5d0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf6025640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf5824640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f5620bb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabec019040 tx=0x7fabec005530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e98ae07d931bd88 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec042060 con 0x560f561fc5d0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fabec034a30 con 0x560f561fc5d0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabec02f030 con 0x560f561fc5d0 2024-09-12T02:59:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fabec023070 con 0x560f561fc5d0 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf9e61740 10 client.27511 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.157+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f56282750 con 0x560f561fc5d0 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabec02f030 con 0x560f561fc5d0 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fabec027070 con 0x560f561fc5d0 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 10 client.27511 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 10 client.27511 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 20 client.27511 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56282d00 0x560f56285120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 4 client.27511 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f560e6170 con 0x560f56282d00 2024-09-12T02:59:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.158+0000 7fabf5023640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56282d00 0x560f56285120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.159+0000 7fabd97fa640 10 client.27511 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.159+0000 7fabf5023640 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56282d00 0x560f56285120 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=887243e469c7e50e server_cookie=6d276340b52e6a5a in_seq=0 out_seq=0 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabd97fa640 1 -- 172.21.15.16:0/3871126140 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f560e6170 con 0x560f56282d00 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabd97fa640 10 client.27511 handle_command_reply: tid=0 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 client.27511 shutdown 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 msgr2=0x560f5620bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f5620bb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabec019040 tx=0x7fabec005530 comp rx=0 tx=0).stop 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 shutdown_connections 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560f56282d00 0x560f56285120 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560f56207370 0x560f5627ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560f561fdb40 0x560f5620c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabd97fa640 0 client.27511 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 --2- 172.21.15.16:0/3871126140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560f561fc5d0 0x560f5620bb60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 >> 172.21.15.16:0/3871126140 conn(0x560f561c5310 msgr2=0x560f56206080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 shutdown_connections 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 1 -- 172.21.15.16:0/3871126140 wait complete. 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.160+0000 7fabf9e61740 20 client.27511 trim_cache size 0 max 16384 2024-09-12T02:59:40.166 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 msgr2=0x7fabdc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 0x7fabdc0a5040 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fabec009fd0 tx=0x7fabec005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 msgr2=0x7fabf01989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fabe4005940 tx=0x7fabe40040c0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 shutdown_connections 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabf0106f10 0x7fabf019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabdc0a2b80 0x7fabdc0a5040 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0102790 0x7fabf01989f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 --2- 172.21.15.16:0/1905942199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabf0101de0 0x7fabf01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 >> 172.21.15.16:0/1905942199 conn(0x7fabf00fd8c0 msgr2=0x7fabf00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 shutdown_connections 2024-09-12T02:59:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:40.161+0000 7fabf7a88640 1 -- 172.21.15.16:0/1905942199 wait complete. 2024-09-12T02:59:40.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1193942 10.11 MB/sec execute 522 sec latency 88.349 ms 2024-09-12T02:59:40.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T02:59:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:41.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1196556 10.11 MB/sec execute 523 sec latency 100.649 ms 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 -- 172.21.15.16:0/3695076047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 msgr2=0x7fc2d4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/3695076047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d4100ed0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009cd0 tx=0x7fc2c402a8e0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 -- 172.21.15.16:0/3695076047 shutdown_connections 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/3695076047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/3695076047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d4100ed0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/3695076047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 0x7fc2d4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 -- 172.21.15.16:0/3695076047 >> 172.21.15.16:0/3695076047 conn(0x7fc2d40fc790 msgr2=0x7fc2d40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 -- 172.21.15.16:0/3695076047 shutdown_connections 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.736+0000 7fc2dabdb640 1 -- 172.21.15.16:0/3695076047 wait complete. 2024-09-12T02:59:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 Processor -- start 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 -- start start 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 0x7fc2d419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4110e70 con 0x7fc2d4100a70 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4110cf0 con 0x7fc2d4101410 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2dabdb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4110ff0 con 0x7fc2d4106a70 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39580/0 (socket says 172.21.15.16:39580) 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2d8977640 1 -- 172.21.15.16:0/1122133821 learned_addr learned my addr 172.21.15.16:0/1122133821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 0x7fc2d419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.737+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 -- 172.21.15.16:0/1122133821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 msgr2=0x7fc2d419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 0x7fc2d419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 -- 172.21.15.16:0/1122133821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 msgr2=0x7fc2d419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 -- 172.21.15.16:0/1122133821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d419f5f0 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d8977640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00e9f0 tx=0x7fc2bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d1ffb640 1 -- 172.21.15.16:0/1122133821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00cd80 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d1ffb640 1 -- 172.21.15.16:0/1122133821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc010910 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d419f8e0 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2d1ffb640 1 -- 172.21.15.16:0/1122133821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc018990 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.738+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d41a71c0 con 0x7fc2d4100a70 2024-09-12T02:59:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.741+0000 7fc2d1ffb640 1 -- 172.21.15.16:0/1122133821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc2bc010ac0 con 0x7fc2d4100a70 2024-09-12T02:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.741+0000 7fc2dcfb4740 1 Processor -- start 2024-09-12T02:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 -- start start 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888a96090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d1ffb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 0x7fc2c00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888aa3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5588889fc2f0 con 0x558888a97b40 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5588889a3cf0 con 0x558888a965d0 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5588889f82b0 con 0x558888aa1370 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44068/0 (socket says 172.21.15.16:44068) 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d3fff640 1 -- 172.21.15.16:0/149293495 learned_addr learned my addr 172.21.15.16:0/149293495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 0x7fc2c00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d1ffb640 1 -- 172.21.15.16:0/1122133821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc2bc0cf2b0 con 0x7fc2d4100a70 2024-09-12T02:59:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d8977640 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888a96090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.742+0000 7fc2d9178640 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888aa3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 -- 172.21.15.16:0/149293495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 msgr2=0x558888aa3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888aa3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 -- 172.21.15.16:0/149293495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 msgr2=0x558888a96090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888a96090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 -- 172.21.15.16:0/149293495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588889883a0 con 0x558888a965d0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 0x7fc2c00a50c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x558888a97070 tx=0x7fc2cc0041c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4000f80 tx=0x7fc2c4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a748e7fa29cab50c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2caffd640 1 -- 172.21.15.16:0/149293495 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40318c0 con 0x558888a965d0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2caffd640 1 -- 172.21.15.16:0/149293495 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c4036040 con 0x558888a965d0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.743+0000 7fc2caffd640 1 -- 172.21.15.16:0/149293495 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4004540 con 0x558888a965d0 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/149293495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 msgr2=0x558888a96a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4000f80 tx=0x7fc2c4005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/149293495 shutdown_connections 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888aa3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a965d0 0x558888a96a10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/149293495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888a96090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/149293495 >> 172.21.15.16:0/149293495 conn(0x558888a5f310 msgr2=0x558888a92c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/149293495 shutdown_connections 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/149293495 wait complete. 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.744+0000 7fc2dcfb4740 1 -- start start 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 Processor -- start 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a965d0 0x558888aa0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888a9e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558888aa4010 con 0x558888a965d0 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5588889b7060 con 0x558888a97b40 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558888aa4190 con 0x558888aa1370 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2dcfb4740 10 client.0 _ll_register_callbacks cb 0x55888870a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2d9178640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888a9e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44082/0 (socket says 172.21.15.16:44082) 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a965d0 0x558888aa0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.745+0000 7fc2d3fff640 1 -- 172.21.15.16:0/1780742885 learned_addr learned my addr 172.21.15.16:0/1780742885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 -- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 msgr2=0x558888a9dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 -- 172.21.15.16:0/1780742885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a965d0 msgr2=0x558888aa0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a965d0 0x558888aa0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 -- 172.21.15.16:0/1780742885 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55888876d3c0 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888a9e5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003e50 tx=0x7fc2cc00b2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c04cb564d70cb5e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc005020 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc2cc0051c0 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c4036040 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc2cc0053c0 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2dcfb4740 10 client.27523 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558888aafa70 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.746+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c4036040 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc2cc012960 con 0x558888aa1370 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 10 client.27523 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 10 client.27523 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 20 client.27523 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888ab0fd0 0x558888ab3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 4 client.27523 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558888a09860 con 0x558888ab0fd0 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2d8977640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888ab0fd0 0x558888ab3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2d8977640 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888ab0fd0 0x558888ab3430 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=297e98bac6c04345 server_cookie=c38f1e2d43d31917 in_seq=0 out_seq=0 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.747+0000 7fc2b3fff640 10 client.27523 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2b3fff640 1 -- 172.21.15.16:0/1780742885 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558888a09860 con 0x558888ab0fd0 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2b3fff640 10 client.27523 handle_command_reply: tid=0 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 client.27523 shutdown 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 msgr2=0x558888a9e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888a9e5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003e50 tx=0x7fc2cc00b2e0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 shutdown_connections 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888ab0fd0 0x558888ab3430 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888aa1370 0x558888a9e5d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888a97b40 0x558888a9dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1780742885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a965d0 0x558888aa0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2b3fff640 0 client.27523 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 >> 172.21.15.16:0/1780742885 conn(0x558888a5f310 msgr2=0x558888a98930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 shutdown_connections 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1780742885 wait complete. 2024-09-12T02:59:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.748+0000 7fc2dcfb4740 20 client.27523 trim_cache size 0 max 16384 2024-09-12T02:59:41.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.761+0000 7fc2dcfb4740 1 Processor -- start 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 -- start start 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888a92e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41af7c0 con 0x558888a97b40 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41b1300 con 0x558888aa1370 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41afe90 con 0x558888a965d0 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888a92e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44086/0 (socket says 172.21.15.16:44086) 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888a92e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42300/0 (socket says 172.21.15.16:42300) 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aad2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39600/0 (socket says 172.21.15.16:39600) 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 -- 172.21.15.16:0/1837660517 learned_addr learned my addr 172.21.15.16:0/1837660517 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 -- 172.21.15.16:0/1837660517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 msgr2=0x558888a92e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888a92e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 -- 172.21.15.16:0/1837660517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 msgr2=0x558888aad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.762+0000 7fc2d9178640 1 -- 172.21.15.16:0/1837660517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558888a9cf90 con 0x558888aa1370 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2d9178640 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003ef0 tx=0x7fc2cc004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc18a29fba5a5481 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2ca7fc640 1 -- 172.21.15.16:0/1837660517 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc039020 con 0x558888aa1370 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2ca7fc640 1 -- 172.21.15.16:0/1837660517 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2cc00bc40 con 0x558888aa1370 2024-09-12T02:59:41.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2ca7fc640 1 -- 172.21.15.16:0/1837660517 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc034a80 con 0x558888aa1370 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1837660517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 msgr2=0x558888a9d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003ef0 tx=0x7fc2cc004060 comp rx=0 tx=0).stop 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1837660517 shutdown_connections 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888a9d8a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/1837660517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888a92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.763+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1837660517 >> 172.21.15.16:0/1837660517 conn(0x558888a5f310 msgr2=0x558888a92700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1837660517 shutdown_connections 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/1837660517 wait complete. 2024-09-12T02:59:41.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- start start 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 Processor -- start 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888aa5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41b0c30 con 0x558888a97b40 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41b0560 con 0x558888aa1370 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41ad7a0 con 0x558888a965d0 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.764+0000 7fc2dcfb4740 10 client.0 _ll_register_callbacks cb 0x7fc2cc0043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42304/0 (socket says 172.21.15.16:42304) 2024-09-12T02:59:41.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39606/0 (socket says 172.21.15.16:39606) 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d9178640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888aa5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 -- 172.21.15.16:0/3696287070 learned_addr learned my addr 172.21.15.16:0/3696287070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 -- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 msgr2=0x558888aa5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888aa5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 -- 172.21.15.16:0/3696287070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 msgr2=0x558888aa4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 -- 172.21.15.16:0/3696287070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558888aa5b90 con 0x558888a965d0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2d8977640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc09b820 tx=0x7fc2bc009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d56c9d58460509d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc09fed0 con 0x558888a965d0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc2bc099e20 con 0x558888a965d0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2cc00bc40 con 0x558888a965d0 2024-09-12T02:59:41.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.765+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc0a4790 con 0x558888a965d0 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 10 client.27529 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558888b1bc70 con 0x558888a965d0 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc09ce50 con 0x558888a965d0 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc2bc09af70 con 0x558888a965d0 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 10 client.27529 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 10 client.27529 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 20 client.27529 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888b1c390 0x558888b1e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 4 client.27529 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc2cc00bc40 con 0x558888b1c390 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888b1c390 0x558888b1e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.766+0000 7fc2b3fff640 10 client.27529 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T02:59:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2d3fff640 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888b1c390 0x558888b1e780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3808468e6eb6aca server_cookie=55051093aaf3a2f1 in_seq=0 out_seq=0 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2b3fff640 1 -- 172.21.15.16:0/3696287070 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc2cc00bc40 con 0x558888b1c390 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2b3fff640 10 client.27529 handle_command_reply: tid=0 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 client.27529 shutdown 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 msgr2=0x558888aa9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc09b820 tx=0x7fc2bc009700 comp rx=0 tx=0).stop 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 shutdown_connections 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558888b1c390 0x558888b1e780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558888aa1370 0x558888aa5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558888a97b40 0x558888aa4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2b3fff640 0 client.27529 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 --2- 172.21.15.16:0/3696287070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558888a965d0 0x558888aa9aa0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 >> 172.21.15.16:0/3696287070 conn(0x558888a5f310 msgr2=0x558888a9f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 shutdown_connections 2024-09-12T02:59:41.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 1 -- 172.21.15.16:0/3696287070 wait complete. 2024-09-12T02:59:41.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.767+0000 7fc2dcfb4740 20 client.27529 trim_cache size 0 max 16384 2024-09-12T02:59:41.774 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 msgr2=0x7fc2c00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 0x7fc2c00a50c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x558888a97070 tx=0x7fc2cc0041c0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 msgr2=0x7fc2d419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00e9f0 tx=0x7fc2bc00eeb0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 shutdown_connections 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d4106a70 0x7fc2d419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc2d4101410 0x7fc2d419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc2c00a2c00 0x7fc2c00a50c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 --2- 172.21.15.16:0/1122133821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2d4100a70 0x7fc2d419a640 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 >> 172.21.15.16:0/1122133821 conn(0x7fc2d40fc790 msgr2=0x7fc2d406a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:41.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 shutdown_connections 2024-09-12T02:59:41.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:41.768+0000 7fc2dabdb640 1 -- 172.21.15.16:0/1122133821 wait complete. 2024-09-12T02:59:41.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:41 smithi016 ceph-mon[30925]: pgmap v690: 97 pgs: 97 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-09-12T02:59:41.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:41 smithi033 ceph-mon[35092]: pgmap v690: 97 pgs: 97 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-09-12T02:59:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:41 smithi184 ceph-mon[35321]: pgmap v690: 97 pgs: 97 active+clean; 274 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-09-12T02:59:42.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T02:59:42.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1198909 10.12 MB/sec execute 524 sec latency 11.999 ms 2024-09-12T02:59:42.790 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:42.811 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.2 2024-09-12T02:59:42.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 -- 172.21.15.16:0/1568230357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700100e20 msgr2=0x7f1700101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 --2- 172.21.15.16:0/1568230357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700100e20 0x7f1700101200 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009930 tx=0x7f16f002a4d0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 -- 172.21.15.16:0/1568230357 shutdown_connections 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 --2- 172.21.15.16:0/1568230357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102ef0 0x7f170010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 --2- 172.21.15.16:0/1568230357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700102570 0x7f17001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 --2- 172.21.15.16:0/1568230357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700100e20 0x7f1700101200 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 -- 172.21.15.16:0/1568230357 >> 172.21.15.16:0/1568230357 conn(0x7f17000fc7d0 msgr2=0x7f17000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 -- 172.21.15.16:0/1568230357 shutdown_connections 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.283+0000 7f1705e4e640 1 -- 172.21.15.16:0/1568230357 wait complete. 2024-09-12T02:59:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 Processor -- start 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 -- start start 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102570 0x7f1700198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1700110f80 con 0x7f1700102570 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.284+0000 7f1705e4e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1700110e00 con 0x7f1700102ef0 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f1705e4e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1700111100 con 0x7f1700100e20 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102570 0x7f1700198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42318/0 (socket says 172.21.15.16:42318) 2024-09-12T02:59:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44116/0 (socket says 172.21.15.16:44116) 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 -- 172.21.15.16:0/4005826756 learned_addr learned my addr 172.21.15.16:0/4005826756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 -- 172.21.15.16:0/4005826756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 msgr2=0x7f170019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 -- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102570 msgr2=0x7f1700198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102570 0x7f1700198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 -- 172.21.15.16:0/4005826756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170019d580 con 0x7f1700100e20 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16fffff640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16ff7fe640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009230 tx=0x7f16f0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f16fcff9640 1 -- 172.21.15.16:0/4005826756 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f003f030 con 0x7f1700100e20 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.285+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170019d810 con 0x7f1700100e20 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.286+0000 7f16fcff9640 1 -- 172.21.15.16:0/4005826756 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f0004440 con 0x7f1700100e20 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.286+0000 7f16fcff9640 1 -- 172.21.15.16:0/4005826756 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f0035070 con 0x7f1700100e20 2024-09-12T02:59:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.286+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17001a5150 con 0x7f1700100e20 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f16fcff9640 1 -- 172.21.15.16:0/4005826756 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f16f0042030 con 0x7f1700100e20 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 Processor -- start 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 -- start start 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e85d0 0x55a10d4e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a10d44e2f0 con 0x55a10d4e85d0 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a10d3f5cf0 con 0x55a10d4e9b40 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.288+0000 7f1708227740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a10d44a2b0 con 0x55a10d4f7780 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e85d0 0x55a10d4e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16feffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16feffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4f9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42324/0 (socket says 172.21.15.16:42324) 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16feffd640 1 -- 172.21.15.16:0/1466099043 learned_addr learned my addr 172.21.15.16:0/1466099043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 -- 172.21.15.16:0/1466099043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 msgr2=0x55a10d4f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 -- 172.21.15.16:0/1466099043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e85d0 msgr2=0x55a10d4e8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e85d0 0x55a10d4e8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 -- 172.21.15.16:0/1466099043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a10d3da3a0 con 0x55a10d4e9b40 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fcff9640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 0x7f16e80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16fffff640 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4e8090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f16f8011760 tx=0x7f16f8011c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673a47bba98f60d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16ff7fe640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 0x7f16e80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16e5ffb640 1 -- 172.21.15.16:0/1466099043 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8004d20 con 0x55a10d4e9b40 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16e5ffb640 1 -- 172.21.15.16:0/1466099043 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f8003970 con 0x55a10d4e9b40 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.289+0000 7f16e5ffb640 1 -- 172.21.15.16:0/1466099043 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8004480 con 0x55a10d4e9b40 2024-09-12T02:59:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f16fcff9640 1 -- 172.21.15.16:0/4005826756 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16f00ef300 con 0x7f1700100e20 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 -- 172.21.15.16:0/1466099043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 msgr2=0x55a10d4e8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4e8090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f16f8011760 tx=0x7f16f8011c20 comp rx=0 tx=0).stop 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f16ff7fe640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 0x7f16e80a5040 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f16f00c42b0 tx=0x7f16f00f4040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 -- 172.21.15.16:0/1466099043 shutdown_connections 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e85d0 0x55a10d4e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 --2- 172.21.15.16:0/1466099043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4e8090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 -- 172.21.15.16:0/1466099043 >> 172.21.15.16:0/1466099043 conn(0x55a10d4b1310 msgr2=0x55a10d4e4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 -- 172.21.15.16:0/1466099043 shutdown_connections 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.290+0000 7f1708227740 1 -- 172.21.15.16:0/1466099043 wait complete. 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 -- start start 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 Processor -- start 2024-09-12T02:59:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d4f7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 0x55a10d4f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a10d4fa480 con 0x55a10d4f7780 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a10d409060 con 0x55a10d4e9b40 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a10d4fa600 con 0x55a10d4e85d0 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f1708227740 10 client.0 _ll_register_callbacks cb 0x55a10d15c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 0x55a10d4f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16feffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 0x55a10d4f4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39656/0 (socket says 172.21.15.16:39656) 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16feffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44140/0 (socket says 172.21.15.16:44140) 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16fffff640 1 -- 172.21.15.16:0/2624682309 learned_addr learned my addr 172.21.15.16:0/2624682309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.291+0000 7f16fffff640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d4f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 -- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 msgr2=0x55a10d4f7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d4f7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 -- 172.21.15.16:0/2624682309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 msgr2=0x55a10d4f4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 0x55a10d4f4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 -- 172.21.15.16:0/2624682309 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a10d1bf3c0 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16feffd640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f16f40126d0 tx=0x7f16f4012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628c835763d5297a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16deffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4004050 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16f8003970 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16f40041f0 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16f40043f0 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f1708227740 10 client.27647 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a10d505f50 con 0x55a10d4e9b40 2024-09-12T02:59:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.292+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f8003970 con 0x55a10d4e9b40 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f16f4003bf0 con 0x55a10d4e9b40 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 10 client.27647 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 10 client.27647 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 20 client.27647 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d507460 0x55a10d509880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 4 client.27647 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a10d45b860 con 0x55a10d507460 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f16fffff640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d507460 0x55a10d509880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f16fffff640 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d507460 0x55a10d509880 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=951df1566d296b01 server_cookie=e73ed54aff52c804 in_seq=0 out_seq=0 2024-09-12T02:59:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.293+0000 7f16deffd640 10 client.27647 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:59:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f16deffd640 1 -- 172.21.15.16:0/2624682309 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a10d45b860 con 0x55a10d507460 2024-09-12T02:59:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f16deffd640 10 client.27647 handle_command_reply: tid=0 2024-09-12T02:59:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 client.27647 shutdown 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 msgr2=0x55a10d4f4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f16f40126d0 tx=0x7f16f4012b90 comp rx=0 tx=0).stop 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 shutdown_connections 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4f7780 0x55a10d4f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e9b40 0x55a10d4f4430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d507460 0x55a10d509880 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 --2- 172.21.15.16:0/2624682309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d4f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f16deffd640 0 client.27647 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 >> 172.21.15.16:0/2624682309 conn(0x55a10d4b1310 msgr2=0x55a10d4ea930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 shutdown_connections 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.294+0000 7f1708227740 1 -- 172.21.15.16:0/2624682309 wait complete. 2024-09-12T02:59:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.295+0000 7f1708227740 20 client.27647 trim_cache size 0 max 16384 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 Processor -- start 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 -- start start 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4f3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4f7780 0x55a10d50f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17001ab390 con 0x55a10d4e9b40 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17001aced0 con 0x55a10d4f7780 2024-09-12T02:59:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f1708227740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f17001aba60 con 0x55a10d4e85d0 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42340/0 (socket says 172.21.15.16:42340) 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 -- 172.21.15.16:0/467972826 learned_addr learned my addr 172.21.15.16:0/467972826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4f7780 0x55a10d50f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 -- 172.21.15.16:0/467972826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4f7780 msgr2=0x55a10d50f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4f7780 0x55a10d50f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 -- 172.21.15.16:0/467972826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 msgr2=0x55a10d4f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.309+0000 7f16fffff640 1 -- 172.21.15.16:0/467972826 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a10d4e5730 con 0x55a10d4e85d0 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.310+0000 7f16fffff640 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f16f8005f10 tx=0x7f16f8009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fdf1c9ca2236133 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.310+0000 7f16e57fa640 1 -- 172.21.15.16:0/467972826 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f801d040 con 0x55a10d4e85d0 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.310+0000 7f16e57fa640 1 -- 172.21.15.16:0/467972826 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f801e040 con 0x55a10d4e85d0 2024-09-12T02:59:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.310+0000 7f16e57fa640 1 -- 172.21.15.16:0/467972826 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80052c0 con 0x55a10d4e85d0 2024-09-12T02:59:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- 172.21.15.16:0/467972826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 msgr2=0x55a10d503720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f16f8005f10 tx=0x7f16f8009500 comp rx=0 tx=0).stop 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- 172.21.15.16:0/467972826 shutdown_connections 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4f7780 0x55a10d50f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 --2- 172.21.15.16:0/467972826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4e85d0 0x55a10d503720 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- 172.21.15.16:0/467972826 >> 172.21.15.16:0/467972826 conn(0x55a10d4b1310 msgr2=0x55a10d4f2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- 172.21.15.16:0/467972826 shutdown_connections 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- 172.21.15.16:0/467972826 wait complete. 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 -- start start 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.311+0000 7f1708227740 1 Processor -- start 2024-09-12T02:59:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4fb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4fb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17001ac800 con 0x55a10d4e9b40 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4fb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44170/0 (socket says 172.21.15.16:44170) 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 -- 172.21.15.16:0/933335682 learned_addr learned my addr 172.21.15.16:0/933335682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17001ac130 con 0x55a10d4e85d0 2024-09-12T02:59:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f17001a9390 con 0x55a10d4f7780 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f1708227740 10 client.0 _ll_register_callbacks cb 0x7f16f4009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4fb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 -- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 msgr2=0x55a10d4fb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4fb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 -- 172.21.15.16:0/933335682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 msgr2=0x55a10d4fb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4fb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.312+0000 7f16fffff640 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a10d4fba80 con 0x55a10d4e85d0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16fffff640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f16f8011c60 tx=0x7f16f800ff10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265d2152bef983ff server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16deffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f801d040 con 0x55a10d4e85d0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16f800fc40 con 0x55a10d4e85d0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16f801e040 con 0x55a10d4e85d0 2024-09-12T02:59:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16f8011040 con 0x55a10d4e85d0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f1708227740 10 client.27653 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a10d571e10 con 0x55a10d4e85d0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.313+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f801e040 con 0x55a10d4e85d0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f16f8036020 con 0x55a10d4e85d0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 10 client.27653 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 10 client.27653 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 20 client.27653 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d5722d0 0x55a10d5746c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 4 client.27653 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a10d455d70 con 0x55a10d5722d0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f16feffd640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d5722d0 0x55a10d5746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f16feffd640 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d5722d0 0x55a10d5746c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18c9f251015fd97e server_cookie=96db5b4edb24381f in_seq=0 out_seq=0 2024-09-12T02:59:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.314+0000 7f16deffd640 10 client.27653 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T02:59:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f16deffd640 1 -- 172.21.15.16:0/933335682 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a10d455d70 con 0x55a10d5722d0 2024-09-12T02:59:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f16deffd640 10 client.27653 handle_command_reply: tid=0 2024-09-12T02:59:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 client.27653 shutdown 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 msgr2=0x55a10d4ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f16f8011c60 tx=0x7f16f800ff10 comp rx=0 tx=0).stop 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 shutdown_connections 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a10d5722d0 0x55a10d5746c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a10d4f7780 0x55a10d4fb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a10d4e9b40 0x55a10d4fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 --2- 172.21.15.16:0/933335682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a10d4e85d0 0x55a10d4ffcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f16deffd640 0 client.27653 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 >> 172.21.15.16:0/933335682 conn(0x55a10d4b1310 msgr2=0x55a10d4f3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 shutdown_connections 2024-09-12T02:59:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 1 -- 172.21.15.16:0/933335682 wait complete. 2024-09-12T02:59:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.315+0000 7f1708227740 20 client.27653 trim_cache size 0 max 16384 2024-09-12T02:59:43.322 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 msgr2=0x7f16e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 0x7f16e80a5040 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f16f00c42b0 tx=0x7f16f00f4040 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 msgr2=0x7f17001985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009230 tx=0x7f16f0004040 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 shutdown_connections 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1700102ef0 0x7f170019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16e80a2b80 0x7f16e80a5040 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1700102570 0x7f1700198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 --2- 172.21.15.16:0/4005826756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1700100e20 0x7f17001985d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 >> 172.21.15.16:0/4005826756 conn(0x7f17000fc7d0 msgr2=0x7f1700105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 shutdown_connections 2024-09-12T02:59:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:43.316+0000 7f1705e4e640 1 -- 172.21.15.16:0/4005826756 wait complete. 2024-09-12T02:59:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:43.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1201262 10.12 MB/sec execute 525 sec latency 14.286 ms 2024-09-12T02:59:43.944 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:43 smithi016 ceph-mon[30925]: pgmap v691: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 299 op/s 2024-09-12T02:59:43.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:43 smithi033 ceph-mon[35092]: pgmap v691: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 299 op/s 2024-09-12T02:59:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:43 smithi184 ceph-mon[35321]: pgmap v691: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 299 op/s 2024-09-12T02:59:44.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 -- 172.21.15.16:0/2894345118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 msgr2=0x7f311c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 --2- 172.21.15.16:0/2894345118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c10ec40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3114009930 tx=0x7f311402a420 comp rx=0 tx=0).stop 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 -- 172.21.15.16:0/2894345118 shutdown_connections 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 --2- 172.21.15.16:0/2894345118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c10ec40 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 --2- 172.21.15.16:0/2894345118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 --2- 172.21.15.16:0/2894345118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 0x7f311c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 -- 172.21.15.16:0/2894345118 >> 172.21.15.16:0/2894345118 conn(0x7f311c0fc7d0 msgr2=0x7f311c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.655+0000 7f312178e640 1 -- 172.21.15.16:0/2894345118 shutdown_connections 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.656+0000 7f312178e640 1 -- 172.21.15.16:0/2894345118 wait complete. 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.656+0000 7f312178e640 1 Processor -- start 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.656+0000 7f312178e640 1 -- start start 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 0x7f311c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1120a0 con 0x7f311c1082a0 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f311c111f20 con 0x7f311c1036e0 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f312178e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f311c112220 con 0x7f311c102d30 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 0x7f311c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:44.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39682/0 (socket says 172.21.15.16:39682) 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44184/0 (socket says 172.21.15.16:44184) 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311b7fe640 1 -- 172.21.15.16:0/2449793970 learned_addr learned my addr 172.21.15.16:0/2449793970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 -- 172.21.15.16:0/2449793970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 msgr2=0x7f311c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 0x7f311c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 -- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 msgr2=0x7f311c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 -- 172.21.15.16:0/2449793970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311c19f6e0 con 0x7f311c1036e0 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311b7fe640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f311a7fc640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f311c0697d0 tx=0x7f310400db30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.657+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104017070 con 0x7f311c1036e0 2024-09-12T02:59:44.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.658+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3104009820 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.658+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311c19f9d0 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.658+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104004ca0 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.658+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311c1a72b0 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.659+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311c103110 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.660+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3104005220 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.660+0000 7f3113fff640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.661+0000 7f311affd640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.661+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31040ce820 con 0x7f311c1036e0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.661+0000 7f311affd640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f310c005fd0 tx=0x7f310c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:44.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.664+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3104004e40 con 0x7f311c1036e0 2024-09-12T02:59:44.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1204091 10.14 MB/sec execute 526 sec latency 58.961 ms 2024-09-12T02:59:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.906+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f311c10c2c0 con 0x7f311c1036e0 2024-09-12T02:59:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.907+0000 7f3113fff640 1 -- 172.21.15.16:0/2449793970 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f310409b0e0 con 0x7f311c1036e0 2024-09-12T02:59:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T02:59:44.909 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T02:59:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.909+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 msgr2=0x7f31080a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.909+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f310c005fd0 tx=0x7f310c009500 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.909+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 msgr2=0x7f311c19ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.909+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f311c0697d0 tx=0x7f310400db30 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 shutdown_connections 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31080a2c00 0x7f31080a50c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c1082a0 0x7f311c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f311c1036e0 0x7f311c19ac70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 --2- 172.21.15.16:0/2449793970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f311c102d30 0x7f311c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 >> 172.21.15.16:0/2449793970 conn(0x7f311c0fc7d0 msgr2=0x7f311c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 shutdown_connections 2024-09-12T02:59:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:44.910+0000 7f312178e640 1 -- 172.21.15.16:0/2449793970 wait complete. 2024-09-12T02:59:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 -- 172.21.15.16:0/3514595321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa304102d30 msgr2=0x7fa304103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 --2- 172.21.15.16:0/3514595321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa304102d30 0x7fa304103110 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009930 tx=0x7fa2f402a420 comp rx=0 tx=0).stop 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 -- 172.21.15.16:0/3514595321 shutdown_connections 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 --2- 172.21.15.16:0/3514595321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3041082a0 0x7fa30410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 --2- 172.21.15.16:0/3514595321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 --2- 172.21.15.16:0/3514595321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa304102d30 0x7fa304103110 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 -- 172.21.15.16:0/3514595321 >> 172.21.15.16:0/3514595321 conn(0x7fa3040fc7d0 msgr2=0x7fa3040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 -- 172.21.15.16:0/3514595321 shutdown_connections 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.570+0000 7fa309be1640 1 -- 172.21.15.16:0/3514595321 wait complete. 2024-09-12T02:59:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 Processor -- start 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 -- start start 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa304102d30 0x7fa3041985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041120a0 con 0x7fa3041082a0 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa304111f20 con 0x7fa3041036e0 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa309be1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa304112220 con 0x7fa304102d30 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa303fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa302ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa304102d30 0x7fa3041985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa303fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51364/0 (socket says 172.21.15.16:51364) 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.571+0000 7fa303fff640 1 -- 172.21.15.16:0/1808390332 learned_addr learned my addr 172.21.15.16:0/1808390332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:45.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 -- 172.21.15.16:0/1808390332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa304102d30 msgr2=0x7fa3041985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa304102d30 0x7fa3041985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 -- 172.21.15.16:0/1808390332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 msgr2=0x7fa304198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 -- 172.21.15.16:0/1808390332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30419d550 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa302ffd640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa303fff640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb10 tx=0x7fa2fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa300ff9640 1 -- 172.21.15.16:0/1808390332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc004b20 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30419d840 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa300ff9640 1 -- 172.21.15.16:0/1808390332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2fc004cc0 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa300ff9640 1 -- 172.21.15.16:0/1808390332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc010760 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.572+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3041008c0 con 0x7fa3041082a0 2024-09-12T02:59:45.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa300ff9640 1 -- 172.21.15.16:0/1808390332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa2fc005020 con 0x7fa3041082a0 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 Processor -- start 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 -- start start 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c529e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 0x55d9c529ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52ab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52042f0 con 0x55d9c529fb40 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d9c51abcf0 con 0x55d9c529e5d0 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa30bfba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52002b0 con 0x55d9c52a9370 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c529e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 0x55d9c529ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c529e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51376/0 (socket says 172.21.15.16:51376) 2024-09-12T02:59:45.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa302ffd640 1 -- 172.21.15.16:0/852254332 learned_addr learned my addr 172.21.15.16:0/852254332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.575+0000 7fa302ffd640 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 -- 172.21.15.16:0/852254332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 msgr2=0x55d9c52ab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52ab760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 -- 172.21.15.16:0/852254332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 msgr2=0x55d9c529e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c529e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 -- 172.21.15.16:0/852254332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c51903a0 con 0x55d9c529e5d0 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa302ffd640 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa300ff9640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 0x7fa2ec0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa300ff9640 1 -- 172.21.15.16:0/1808390332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa2fc0cf520 con 0x7fa3041082a0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa3037fe640 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 0x55d9c529ea10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005940 tx=0x7fa2f4005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d327d805ed12ecd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa302ffd640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 0x7fa2ec0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa2e9ffb640 1 -- 172.21.15.16:0/852254332 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4040070 con 0x55d9c529e5d0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa2e9ffb640 1 -- 172.21.15.16:0/852254332 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4031870 con 0x55d9c529e5d0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.576+0000 7fa2e9ffb640 1 -- 172.21.15.16:0/852254332 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4035040 con 0x55d9c529e5d0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- 172.21.15.16:0/852254332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 msgr2=0x55d9c529ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa302ffd640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 0x7fa2ec0a5040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8012010 tx=0x7fa2f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 0x55d9c529ea10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005940 tx=0x7fa2f4005bd0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- 172.21.15.16:0/852254332 shutdown_connections 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529e5d0 0x55d9c529ea10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 --2- 172.21.15.16:0/852254332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c529e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- 172.21.15.16:0/852254332 >> 172.21.15.16:0/852254332 conn(0x55d9c5267310 msgr2=0x55d9c529ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- 172.21.15.16:0/852254332 shutdown_connections 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- 172.21.15.16:0/852254332 wait complete. 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.577+0000 7fa30bfba740 1 -- start start 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 Processor -- start 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c52a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c52a9370 0x55d9c52a6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52ac010 con 0x55d9c529fb40 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d9c51bf060 con 0x55d9c52a9370 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52ac190 con 0x55d9c529e5d0 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41106/0 (socket says 172.21.15.16:41106) 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa3037fe640 1 -- 172.21.15.16:0/509573293 learned_addr learned my addr 172.21.15.16:0/509573293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa303fff640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c52a9370 0x55d9c52a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa302ffd640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c52a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.578+0000 7fa30bfba740 10 client.0 _ll_register_callbacks cb 0x55d9c4f128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 -- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c52a9370 msgr2=0x55d9c52a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c52a9370 0x55d9c52a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 -- 172.21.15.16:0/509573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 msgr2=0x55d9c52a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c52a5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 -- 172.21.15.16:0/509573293 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c4f753c0 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa302ffd640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c52a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa3037fe640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005280 tx=0x7fa2f40052b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b5058d5a2af9df1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa2e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4040070 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2f4031870 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa2f4031670 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa2f4035030 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa30bfba740 10 client.27565 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d9c52ac060 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.579+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4031870 con 0x55d9c529e5d0 2024-09-12T02:59:45.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa2f40035f0 con 0x55d9c529e5d0 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 10 client.27565 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 10 client.27565 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 20 client.27565 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c52baf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 4 client.27565 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c5211860 con 0x55d9c52b8ba0 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa302ffd640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c52baf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa302ffd640 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c52baf90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65a019564d3fc3c server_cookie=53fb241cca14a420 in_seq=0 out_seq=0 2024-09-12T02:59:45.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.580+0000 7fa2e2ffd640 10 client.27565 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:59:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/509573293 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9c5211860 con 0x55d9c52b8ba0 2024-09-12T02:59:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa2e2ffd640 10 client.27565 handle_command_reply: tid=0 2024-09-12T02:59:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 client.27565 shutdown 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 msgr2=0x55d9c52a8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005280 tx=0x7fa2f40052b0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 shutdown_connections 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c52baf90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c52a9370 0x55d9c52a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529fb40 0x55d9c52a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa2e2ffd640 0 client.27565 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 --2- 172.21.15.16:0/509573293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c529e5d0 0x55d9c52a8cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 >> 172.21.15.16:0/509573293 conn(0x55d9c5267310 msgr2=0x55d9c52a0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 shutdown_connections 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.581+0000 7fa30bfba740 1 -- 172.21.15.16:0/509573293 wait complete. 2024-09-12T02:59:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.582+0000 7fa30bfba740 20 client.27565 trim_cache size 0 max 16384 2024-09-12T02:59:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 Processor -- start 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 -- start start 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041ab2a0 con 0x55d9c529e5d0 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041acde0 con 0x55d9c529fb40 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa30bfba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041ab970 con 0x55d9c52a9370 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51404/0 (socket says 172.21.15.16:51404) 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 -- 172.21.15.16:0/3614635574 learned_addr learned my addr 172.21.15.16:0/3614635574 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa302ffd640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa303fff640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 -- 172.21.15.16:0/3614635574 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 msgr2=0x55d9c52c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 -- 172.21.15.16:0/3614635574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 msgr2=0x55d9c52b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa3037fe640 1 -- 172.21.15.16:0/3614635574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c529b950 con 0x55d9c529e5d0 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.597+0000 7fa302ffd640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa3037fe640 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa2f402a460 tx=0x7fa2f4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eef5cee8dc5403 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa2e97fa640 1 -- 172.21.15.16:0/3614635574 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4040070 con 0x55d9c529e5d0 2024-09-12T02:59:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa2e97fa640 1 -- 172.21.15.16:0/3614635574 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4005550 con 0x55d9c529e5d0 2024-09-12T02:59:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa2e97fa640 1 -- 172.21.15.16:0/3614635574 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40044a0 con 0x55d9c529e5d0 2024-09-12T02:59:45.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa30bfba740 1 -- 172.21.15.16:0/3614635574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 msgr2=0x55d9c529ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.598+0000 7fa30bfba740 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa2f402a460 tx=0x7fa2f4005fd0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 -- 172.21.15.16:0/3614635574 shutdown_connections 2024-09-12T02:59:45.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 --2- 172.21.15.16:0/3614635574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c529ae10 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 -- 172.21.15.16:0/3614635574 >> 172.21.15.16:0/3614635574 conn(0x55d9c5267310 msgr2=0x55d9c529a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 -- 172.21.15.16:0/3614635574 shutdown_connections 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 -- 172.21.15.16:0/3614635574 wait complete. 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 -- start start 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.599+0000 7fa30bfba740 1 Processor -- start 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041ac710 con 0x55d9c529e5d0 2024-09-12T02:59:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041ac040 con 0x55d9c529fb40 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041a9280 con 0x55d9c52a9370 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa30bfba740 10 client.0 _ll_register_callbacks cb 0x7fa2f8009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51418/0 (socket says 172.21.15.16:51418) 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 -- 172.21.15.16:0/902402085 learned_addr learned my addr 172.21.15.16:0/902402085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa302ffd640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa303fff640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 -- 172.21.15.16:0/902402085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 msgr2=0x55d9c52a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 -- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 msgr2=0x55d9c52a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.600+0000 7fa3037fe640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa3037fe640 1 -- 172.21.15.16:0/902402085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c52a3ae0 con 0x55d9c529e5d0 2024-09-12T02:59:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa303fff640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa3037fe640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004720 tx=0x7fa2f4005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e53d409a9f6ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa2e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4040070 con 0x55d9c529e5d0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa2f405e070 con 0x55d9c529e5d0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2f4005550 con 0x55d9c529e5d0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.601+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa2f4009520 con 0x55d9c529e5d0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.602+0000 7fa30bfba740 10 client.18258 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.602+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d9c532bfc0 con 0x55d9c529e5d0 2024-09-12T02:59:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.602+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4005550 con 0x55d9c529e5d0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.602+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa2f4062460 con 0x55d9c529e5d0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 10 client.18258 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 10 client.18258 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 20 client.18258 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c532e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 4 client.18258 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9c520bd70 con 0x55d9c52b8ba0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa302ffd640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c532e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa302ffd640 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c532e5d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae11e6e2fa55a65c server_cookie=9dc44fcf0f98addb in_seq=0 out_seq=0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.603+0000 7fa2e2ffd640 10 client.18258 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa2e2ffd640 1 -- 172.21.15.16:0/902402085 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9c520bd70 con 0x55d9c52b8ba0 2024-09-12T02:59:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa2e2ffd640 10 client.18258 handle_command_reply: tid=0 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 client.18258 shutdown 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 msgr2=0x55d9c52b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004720 tx=0x7fa2f4005c90 comp rx=0 tx=0).stop 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 shutdown_connections 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d9c52b8ba0 0x55d9c532e5d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d9c52a9370 0x55d9c52a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d9c529fb40 0x55d9c52a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 --2- 172.21.15.16:0/902402085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9c529e5d0 0x55d9c52b1c00 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 >> 172.21.15.16:0/902402085 conn(0x55d9c5267310 msgr2=0x55d9c52a0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa2e2ffd640 0 client.18258 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T02:59:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 shutdown_connections 2024-09-12T02:59:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 1 -- 172.21.15.16:0/902402085 wait complete. 2024-09-12T02:59:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.604+0000 7fa30bfba740 20 client.18258 trim_cache size 0 max 16384 2024-09-12T02:59:45.610 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 msgr2=0x7fa2ec0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 0x7fa2ec0a5040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8012010 tx=0x7fa2f8005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 msgr2=0x7fa30419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb10 tx=0x7fa2fc00c2d0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 shutdown_connections 2024-09-12T02:59:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2ec0a2b80 0x7fa2ec0a5040 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3041082a0 0x7fa30419ce70 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3041036e0 0x7fa304198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 --2- 172.21.15.16:0/1808390332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa304102d30 0x7fa3041985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 >> 172.21.15.16:0/1808390332 conn(0x7fa3040fc7d0 msgr2=0x7fa304105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 shutdown_connections 2024-09-12T02:59:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:45.605+0000 7fa309be1640 1 -- 172.21.15.16:0/1808390332 wait complete. 2024-09-12T02:59:45.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1206488 10.14 MB/sec execute 527 sec latency 14.826 ms 2024-09-12T02:59:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:45 smithi016 ceph-mon[30925]: pgmap v692: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-09-12T02:59:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:45 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2449793970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:45.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:45 smithi033 ceph-mon[35092]: pgmap v692: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-09-12T02:59:45.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:45 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2449793970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:45 smithi184 ceph-mon[35321]: pgmap v692: 97 pgs: 97 active+clean; 278 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-09-12T02:59:46.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:45 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2449793970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T02:59:46.091 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0' 2024-09-12T02:59:46.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 147.2 --await-for 45.0 2024-09-12T02:59:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1208785 10.14 MB/sec execute 528 sec latency 11.597 ms 2024-09-12T02:59:46.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T02:59:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3635543307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 msgr2=0x7f3cc41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3635543307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc41029b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009900 tx=0x7f3cb802a480 comp rx=0 tx=0).stop 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3635543307 shutdown_connections 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3635543307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3635543307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc41029b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3635543307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3635543307 >> 172.21.15.16:0/3635543307 conn(0x7f3cc40fc7d0 msgr2=0x7f3cc40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.475+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3635543307 shutdown_connections 2024-09-12T02:59:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.476+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3635543307 wait complete. 2024-09-12T02:59:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.476+0000 7f3cc8d3a640 1 Processor -- start 2024-09-12T02:59:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 -- start start 2024-09-12T02:59:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4110f80 con 0x7f3cc4102570 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4110e00 con 0x7f3cc4100e20 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc8d3a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4111100 con 0x7f3cc4102ef0 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35312/0 (socket says 172.21.15.16:35312) 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc259c640 1 -- 172.21.15.16:0/3281511896 learned_addr learned my addr 172.21.15.16:0/3281511896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.477+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3281511896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 msgr2=0x7f3cc40fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 msgr2=0x7f3cc40fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc40fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3281511896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc41a6940 con 0x7f3cc4102ef0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc40fd700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc009fd0 tx=0x7f3cbc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cb37fe640 1 -- 172.21.15.16:0/3281511896 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc00ce20 con 0x7f3cc4102ef0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cb37fe640 1 -- 172.21.15.16:0/3281511896 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cbc005020 con 0x7f3cc4102ef0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc41a6c30 con 0x7f3cc4102ef0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cb37fe640 1 -- 172.21.15.16:0/3281511896 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc010720 con 0x7f3cc4102ef0 2024-09-12T02:59:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.478+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc41a7190 con 0x7f3cc4102ef0 2024-09-12T02:59:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.481+0000 7f3ccb113740 1 Processor -- start 2024-09-12T02:59:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.481+0000 7f3cb37fe640 1 -- 172.21.15.16:0/3281511896 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3cbc005540 con 0x7f3cc4102ef0 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.481+0000 7f3ccb113740 1 -- start start 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d01f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d021d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55940cf7a2a0 con 0x55940d01f940 2024-09-12T02:59:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55940cf21b10 con 0x55940d015080 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3ccb113740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55940cf76260 con 0x55940d014760 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41140/0 (socket says 172.21.15.16:41140) 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/2993792883 learned_addr learned my addr 172.21.15.16:0/2993792883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d021d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc259c640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d01f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/2993792883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 msgr2=0x55940d01f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d01f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/2993792883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 msgr2=0x55940d021d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d021d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/2993792883 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55940ccebb90 con 0x55940d014760 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cc259c640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d01f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.482+0000 7f3cb37fe640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 0x7f3cac0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8031470 tx=0x7f3cb8031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aa653b0f0e979ce server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 0x7f3cac0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cb37fe640 1 -- 172.21.15.16:0/3281511896 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3cbc0cf550 con 0x7f3cc4102ef0 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cb0ff9640 1 -- 172.21.15.16:0/2993792883 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8035040 con 0x55940d014760 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cb0ff9640 1 -- 172.21.15.16:0/2993792883 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb8003ad0 con 0x55940d014760 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.483+0000 7f3cb0ff9640 1 -- 172.21.15.16:0/2993792883 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8031860 con 0x55940d014760 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 -- 172.21.15.16:0/2993792883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 msgr2=0x55940d014b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 0x7f3cac0a50c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009620 tx=0x7f3cb4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8031470 tx=0x7f3cb8031400 comp rx=0 tx=0).stop 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 -- 172.21.15.16:0/2993792883 shutdown_connections 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d021d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d01f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2993792883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d014760 0x55940d014b40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 -- 172.21.15.16:0/2993792883 >> 172.21.15.16:0/2993792883 conn(0x55940d010cf0 msgr2=0x55940d0110d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 -- 172.21.15.16:0/2993792883 shutdown_connections 2024-09-12T02:59:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.484+0000 7f3ccb113740 1 -- 172.21.15.16:0/2993792883 wait complete. 2024-09-12T02:59:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 -- start start 2024-09-12T02:59:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 Processor -- start 2024-09-12T02:59:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d01c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d01f940 0x55940d01d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55940d022510 con 0x55940d014760 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55940cf026e0 con 0x55940d01f940 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55940d022690 con 0x55940d015080 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3ccb113740 10 client.0 _ll_register_callbacks cb 0x55940cf26a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d01c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41152/0 (socket says 172.21.15.16:41152) 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/3602610952 learned_addr learned my addr 172.21.15.16:0/3602610952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d01f940 0x55940d01d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d01f940 msgr2=0x55940d01d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d01f940 0x55940d01d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/3602610952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 msgr2=0x55940d01c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.485+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d01c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3cc1d9b640 1 -- 172.21.15.16:0/3602610952 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55940d02ba50 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8003f90 tx=0x7f3cb8009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ef071b67bb3b0d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8035070 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3cb8031620 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cb8003ad0 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3cb8031820 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ccb113740 10 client.27583 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55940d022510 con 0x55940d015080 2024-09-12T02:59:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.486+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb8003ad0 con 0x55940d015080 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3cb803d780 con 0x55940d015080 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 10 client.27583 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 10 client.27583 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 20 client.27583 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d02f560 0x55940d031950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 4 client.27583 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55940cf879d0 con 0x55940d02f560 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.487+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d02f560 0x55940d031950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.488+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d02f560 0x55940d031950 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba6c958373e84eed server_cookie=746da23fc1b9662c in_seq=0 out_seq=0 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.488+0000 7f3ca9ffb640 10 client.27583 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.488+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/3602610952 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55940cf879d0 con 0x55940d02f560 2024-09-12T02:59:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.488+0000 7f3ca9ffb640 10 client.27583 handle_command_reply: tid=0 2024-09-12T02:59:47.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 client.27583 shutdown 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 msgr2=0x55940d01cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8003f90 tx=0x7f3cb8009680 comp rx=0 tx=0).stop 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 shutdown_connections 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d02f560 0x55940d031950 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d01f940 0x55940d01d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d01cdb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3602610952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d01c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ca9ffb640 0 client.27583 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 >> 172.21.15.16:0/3602610952 conn(0x55940d010cf0 msgr2=0x55940d0169d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 shutdown_connections 2024-09-12T02:59:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 1 -- 172.21.15.16:0/3602610952 wait complete. 2024-09-12T02:59:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.489+0000 7f3ccb113740 20 client.27583 trim_cache size 0 max 16384 2024-09-12T02:59:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.510+0000 7f3ccb113740 1 Processor -- start 2024-09-12T02:59:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.510+0000 7f3ccb113740 1 -- start start 2024-09-12T02:59:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.510+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.510+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d00fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.510+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 0x55940d037040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3ccb113740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41af5d0 con 0x55940d014760 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3ccb113740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41b1110 con 0x55940d015080 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3ccb113740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41afca0 con 0x55940d01f940 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51440/0 (socket says 172.21.15.16:51440) 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 0x55940d037040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc259c640 1 -- 172.21.15.16:0/3959077605 learned_addr learned my addr 172.21.15.16:0/3959077605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d00fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3959077605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 msgr2=0x55940d00fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d00fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3959077605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 msgr2=0x55940d0171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/3959077605 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55940d01adc0 con 0x55940d01f940 2024-09-12T02:59:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc259c640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d00fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:59:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 0x55940d037040 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c760 tx=0x7f3cbc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aefd1aaa19d173d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.511+0000 7f3cabfff640 1 -- 172.21.15.16:0/3959077605 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc09a0c0 con 0x55940d01f940 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3cabfff640 1 -- 172.21.15.16:0/3959077605 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cbc0051d0 con 0x55940d01f940 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3cabfff640 1 -- 172.21.15.16:0/3959077605 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0a4800 con 0x55940d01f940 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 -- 172.21.15.16:0/3959077605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 msgr2=0x55940d037040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 0x55940d037040 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c760 tx=0x7f3cbc005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 -- 172.21.15.16:0/3959077605 shutdown_connections 2024-09-12T02:59:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d01f940 0x55940d037040 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d015080 0x55940d00fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 --2- 172.21.15.16:0/3959077605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d014760 0x55940d0171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 -- 172.21.15.16:0/3959077605 >> 172.21.15.16:0/3959077605 conn(0x55940d010cf0 msgr2=0x55940d00f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 -- 172.21.15.16:0/3959077605 shutdown_connections 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.512+0000 7f3ccb113740 1 -- 172.21.15.16:0/3959077605 wait complete. 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 -- start start 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 Processor -- start 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d014760 0x55940d023f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d024450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41b0a40 con 0x55940d01f940 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41b0370 con 0x55940d014760 2024-09-12T02:59:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41ad5b0 con 0x55940d015080 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3ccb113740 10 client.0 _ll_register_callbacks cb 0x7f3cb8004450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3cc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3cc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51442/0 (socket says 172.21.15.16:51442) 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.513+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/2264198378 learned_addr learned my addr 172.21.15.16:0/2264198378 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc1d9b640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d024450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/2264198378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 msgr2=0x55940d024450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d024450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d014760 msgr2=0x55940d023f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc259c640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d014760 0x55940d023f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d014760 0x55940d023f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 -- 172.21.15.16:0/2264198378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55940d093690 con 0x55940d01f940 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3cc2d9d640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0a4500 tx=0x7f3cbc005380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd81c305ce51d506 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc07e070 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3cbc0a45a0 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.514+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cbc0051d0 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3cbc09c020 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 10 client.18276 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55940d09a930 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cbc0051d0 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3cbc09b210 con 0x55940d01f940 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 10 client.18276 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 10 client.18276 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 20 client.18276 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d09b010 0x55940d09d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 4 client.18276 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}] 2024-09-12T02:59:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.2, "await_for": 45.0}) -- 0x55940cf81d20 con 0x55940d09b010 2024-09-12T02:59:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.515+0000 7f3cc259c640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d09b010 0x55940d09d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.516+0000 7f3ca9ffb640 10 client.18276 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.516+0000 7f3cc259c640 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d09b010 0x55940d09d400 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1828f14c6f50400c server_cookie=5f760404fc2d20f6 in_seq=0 out_seq=0 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ca9ffb640 1 -- 172.21.15.16:0/2264198378 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55940cf81d20 con 0x55940d09b010 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ca9ffb640 10 client.18276 handle_command_reply: tid=0 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 client.18276 shutdown 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 msgr2=0x55940d093050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0a4500 tx=0x7f3cbc005380 comp rx=0 tx=0).stop 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 shutdown_connections 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55940d09b010 0x55940d09d400 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55940d01f940 0x55940d093050 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55940d015080 0x55940d024450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 --2- 172.21.15.16:0/2264198378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55940d014760 0x55940d023f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ca9ffb640 0 client.18276 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 >> 172.21.15.16:0/2264198378 conn(0x55940d010cf0 msgr2=0x55940d01cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.616+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 shutdown_connections 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3ccb113740 1 -- 172.21.15.16:0/2264198378 wait complete. 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3ccb113740 20 client.18276 trim_cache size 0 max 16384 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T02:59:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 46, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "1157782c": { 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 46, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.2, 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T02:59:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:47.621 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 msgr2=0x7f3cac0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 0x7f3cac0a50c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009620 tx=0x7f3cb4005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 msgr2=0x7f3cc40fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc40fd700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc009fd0 tx=0x7f3cbc00ef10 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 shutdown_connections 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cc4102ef0 0x7f3cc40fd700 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3cac0a2c00 0x7f3cac0a50c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cc4102570 0x7f3cc40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 --2- 172.21.15.16:0/3281511896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3cc4100e20 0x7f3cc40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 >> 172.21.15.16:0/3281511896 conn(0x7f3cc40fc7d0 msgr2=0x7f3cc4069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 shutdown_connections 2024-09-12T02:59:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:47.617+0000 7f3cc8d3a640 1 -- 172.21.15.16:0/3281511896 wait complete. 2024-09-12T02:59:47.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:47 smithi016 ceph-mon[30925]: pgmap v693: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-09-12T02:59:47.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.15 MB/sec execute 529 sec latency 186.005 ms 2024-09-12T02:59:47.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:47 smithi033 ceph-mon[35092]: pgmap v693: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-09-12T02:59:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:47 smithi184 ceph-mon[35321]: pgmap v693: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-09-12T02:59:48.385 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1157782c', quiesce took 5.6 seconds. Will release after: 21.6 2024-09-12T02:59:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 -- 172.21.15.16:0/597665335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100460 msgr2=0x7f55a4100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/597665335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100460 0x7f55a4100840 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f55a000aa10 tx=0x7f55a002b720 comp rx=0 tx=0).stop 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 -- 172.21.15.16:0/597665335 shutdown_connections 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/597665335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4105590 0x7f55a410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/597665335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4100e10 0x7f55a4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/597665335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100460 0x7f55a4100840 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 -- 172.21.15.16:0/597665335 >> 172.21.15.16:0/597665335 conn(0x7f55a40fbee0 msgr2=0x7f55a40fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 -- 172.21.15.16:0/597665335 shutdown_connections 2024-09-12T02:59:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.575+0000 7f55ab3d4640 1 -- 172.21.15.16:0/597665335 wait complete. 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 Processor -- start 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 -- start start 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4100460 0x7f55a4101f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 0x7f55a4102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410fd70 con 0x7f55a4100e10 2024-09-12T02:59:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410fbf0 con 0x7f55a4100460 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.576+0000 7f55ab3d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410fef0 con 0x7f55a4105590 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51466/0 (socket says 172.21.15.16:51466) 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 -- 172.21.15.16:0/1353051097 learned_addr learned my addr 172.21.15.16:0/1353051097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a9170640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4100460 0x7f55a4101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a9971640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 0x7f55a4102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 -- 172.21.15.16:0/1353051097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 msgr2=0x7f55a4102980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 0x7f55a4102980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 -- 172.21.15.16:0/1353051097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4100460 msgr2=0x7f55a4101f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4100460 0x7f55a4101f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 -- 172.21.15.16:0/1353051097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a41a6900 con 0x7f55a4100e10 2024-09-12T02:59:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a9971640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 0x7f55a4102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55a896f640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f559400d770 tx=0x7f559400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f559a7fc640 1 -- 172.21.15.16:0/1353051097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594017070 con 0x7f55a4100e10 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f559a7fc640 1 -- 172.21.15.16:0/1353051097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5594005500 con 0x7f55a4100e10 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a41a6bf0 con 0x7f55a4100e10 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f559a7fc640 1 -- 172.21.15.16:0/1353051097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594004e00 con 0x7f55a4100e10 2024-09-12T02:59:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.577+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a41a70e0 con 0x7f55a4100e10 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.580+0000 7f559a7fc640 1 -- 172.21.15.16:0/1353051097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5594004fa0 con 0x7f55a4100e10 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.580+0000 7f55ad7ad740 1 Processor -- start 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.580+0000 7f55ad7ad740 1 -- start start 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c5ab40 0x558850c59090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c59a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558850bbf2f0 con 0x558850c64370 2024-09-12T02:59:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f559a7fc640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 0x7f55900a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558850b66cf0 con 0x558850c595d0 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558850bbb2b0 con 0x558850c5ab40 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a9971640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 0x7f55900a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51478/0 (socket says 172.21.15.16:51478) 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 -- 172.21.15.16:0/3425098527 learned_addr learned my addr 172.21.15.16:0/3425098527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a9971640 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c5ab40 0x558850c59090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a9170640 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c59a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 -- 172.21.15.16:0/3425098527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c5ab40 msgr2=0x558850c59090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c5ab40 0x558850c59090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 -- 172.21.15.16:0/3425098527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 msgr2=0x558850c59a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c59a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a896f640 1 -- 172.21.15.16:0/3425098527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558850b4b3a0 con 0x558850c64370 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f559a7fc640 1 -- 172.21.15.16:0/1353051097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f55940cead0 con 0x7f55a4100e10 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.581+0000 7f55a9971640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 0x7f55900a5040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f559c013ab0 tx=0x7f559c00ac90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55a896f640 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f559401d000 tx=0x7f559406d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f05f7cfc58adcaa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f558f7fe640 1 -- 172.21.15.16:0/3425098527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594020070 con 0x558850c64370 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f558f7fe640 1 -- 172.21.15.16:0/3425098527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55940046c0 con 0x558850c64370 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f558f7fe640 1 -- 172.21.15.16:0/3425098527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559409b2e0 con 0x558850c64370 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 -- 172.21.15.16:0/3425098527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 msgr2=0x558850c66760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f559401d000 tx=0x7f559406d2f0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 -- 172.21.15.16:0/3425098527 shutdown_connections 2024-09-12T02:59:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c64370 0x558850c66760 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c59a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/3425098527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c5ab40 0x558850c59090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.582+0000 7f55ad7ad740 1 -- 172.21.15.16:0/3425098527 >> 172.21.15.16:0/3425098527 conn(0x558850c22310 msgr2=0x558850c55c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- 172.21.15.16:0/3425098527 shutdown_connections 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- 172.21.15.16:0/3425098527 wait complete. 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- start start 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 Processor -- start 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 0x558850c61030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c61640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558850c67010 con 0x558850c595d0 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558850b7a060 con 0x558850c5ab40 2024-09-12T02:59:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.583+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558850c67190 con 0x558850c64370 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55ad7ad740 10 client.0 _ll_register_callbacks cb 0x5588508cd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9170640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9170640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51492/0 (socket says 172.21.15.16:51492) 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9170640 1 -- 172.21.15.16:0/1523615783 learned_addr learned my addr 172.21.15.16:0/1523615783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 0x558850c61030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9971640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c61640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 -- 172.21.15.16:0/1523615783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 msgr2=0x558850c61640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c61640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 -- 172.21.15.16:0/1523615783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 msgr2=0x558850c63d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 -- 172.21.15.16:0/1523615783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588509303c0 con 0x558850c5ab40 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9971640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c61640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a9170640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f55a896f640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 0x558850c61030 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5594099590 tx=0x7f559409bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8271f90a07646c3d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.584+0000 7f558cff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594020070 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558850bcc860 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f55940b4210 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f55940056b0 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 10 client.27698 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558850c72ae0 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558850bcc860 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f55940cc600 con 0x558850c5ab40 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 10 client.27698 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 10 client.27698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 20 client.27698 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850c73e30 0x558850c76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 4 client.27698 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.585+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558850bc6d70 con 0x558850c73e30 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.586+0000 7f55a9170640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850c73e30 0x558850c76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.586+0000 7f558cff9640 10 client.27698 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.586+0000 7f55a9170640 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850c73e30 0x558850c76220 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b349d5251b6c50ca server_cookie=2b01d2b9dd5cc72e in_seq=0 out_seq=0 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f558cff9640 1 -- 172.21.15.16:0/1523615783 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558850bc6d70 con 0x558850c73e30 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f558cff9640 10 client.27698 handle_command_reply: tid=0 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 client.27698 shutdown 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 msgr2=0x558850c61030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 0x558850c61030 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5594099590 tx=0x7f559409bac0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 shutdown_connections 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850c73e30 0x558850c76220 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c61640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c5ab40 0x558850c61030 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1523615783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c595d0 0x558850c63d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 >> 172.21.15.16:0/1523615783 conn(0x558850c22310 msgr2=0x558850c5b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f558cff9640 0 client.27698 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 shutdown_connections 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1523615783 wait complete. 2024-09-12T02:59:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.587+0000 7f55ad7ad740 20 client.27698 trim_cache size 0 max 16384 2024-09-12T02:59:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.600+0000 7f55ad7ad740 1 Processor -- start 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.600+0000 7f55ad7ad740 1 -- start start 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41af6a0 con 0x558850c5ab40 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41b11e0 con 0x558850c595d0 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41afd70 con 0x558850c64370 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51500/0 (socket says 172.21.15.16:51500) 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 -- 172.21.15.16:0/603599669 learned_addr learned my addr 172.21.15.16:0/603599669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a9971640 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a9170640 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 -- 172.21.15.16:0/603599669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 msgr2=0x558850c5c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 -- 172.21.15.16:0/603599669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 msgr2=0x558850c702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 -- 172.21.15.16:0/603599669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558850c634e0 con 0x558850c5ab40 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f55a896f640 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f559409b1c0 tx=0x7f5594009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913f3aff577a2ce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.601+0000 7f558effd640 1 -- 172.21.15.16:0/603599669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594020070 con 0x558850c5ab40 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f558effd640 1 -- 172.21.15.16:0/603599669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5594095e50 con 0x558850c5ab40 2024-09-12T02:59:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f558effd640 1 -- 172.21.15.16:0/603599669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55940a0a60 con 0x558850c5ab40 2024-09-12T02:59:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 -- 172.21.15.16:0/603599669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 msgr2=0x558850c62fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f559409b1c0 tx=0x7f5594009740 comp rx=0 tx=0).stop 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 -- 172.21.15.16:0/603599669 shutdown_connections 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c62fa0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/603599669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 -- 172.21.15.16:0/603599669 >> 172.21.15.16:0/603599669 conn(0x558850c22310 msgr2=0x558850c62890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 -- 172.21.15.16:0/603599669 shutdown_connections 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.602+0000 7f55ad7ad740 1 -- 172.21.15.16:0/603599669 wait complete. 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 -- start start 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 Processor -- start 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c6c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41b0b10 con 0x558850c5ab40 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41b0440 con 0x558850c595d0 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41ad680 con 0x558850c64370 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55ad7ad740 10 client.0 _ll_register_callbacks cb 0x7f55a000a670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55a896f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51506/0 (socket says 172.21.15.16:51506) 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55a896f640 1 -- 172.21.15.16:0/1799437703 learned_addr learned my addr 172.21.15.16:0/1799437703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55a9971640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.603+0000 7f55a9170640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c6c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 -- 172.21.15.16:0/1799437703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 msgr2=0x558850c5d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 -- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 msgr2=0x558850c6c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c6c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 -- 172.21.15.16:0/1799437703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558850c5dba0 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55a896f640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5594099590 tx=0x7f559409bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a786447521da61d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f558cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594020070 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5594097950 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5594095e50 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5594009cc0 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55ad7ad740 10 client.18294 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558850c5dba0 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.604+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5594095e50 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5594084020 con 0x558850c5ab40 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 10 client.18294 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 20 client.18294 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850cde3e0 0x558850ce0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 4 client.18294 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558850b43170 con 0x558850cde3e0 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55a9170640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850cde3e0 0x558850ce0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f55a9170640 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850cde3e0 0x558850ce0800 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=936b328c77dce4df server_cookie=124d593d5f0698b4 in_seq=0 out_seq=0 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.605+0000 7f558cff9640 10 client.18294 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f558cff9640 1 -- 172.21.15.16:0/1799437703 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558850b43170 con 0x558850cde3e0 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f558cff9640 10 client.18294 handle_command_reply: tid=0 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 client.18294 shutdown 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 msgr2=0x558850c6c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5594099590 tx=0x7f559409bac0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 shutdown_connections 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558850cde3e0 0x558850ce0800 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558850c64370 0x558850c5d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558850c5ab40 0x558850c6c960 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 --2- 172.21.15.16:0/1799437703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558850c595d0 0x558850c6c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 >> 172.21.15.16:0/1799437703 conn(0x558850c22310 msgr2=0x558850c630b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f558cff9640 0 client.18294 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 shutdown_connections 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 1 -- 172.21.15.16:0/1799437703 wait complete. 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.606+0000 7f55ad7ad740 20 client.18294 trim_cache size 0 max 16384 2024-09-12T02:59:48.612 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 msgr2=0x7f55900a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 0x7f55900a5040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f559c013ab0 tx=0x7f559c00ac90 comp rx=0 tx=0).stop 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 msgr2=0x7f55a4102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f559400d770 tx=0x7f559400dc30 comp rx=0 tx=0).stop 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 shutdown_connections 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55a4105590 0x7f55a4102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f55900a2b80 0x7f55900a5040 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4100e10 0x7f55a4102440 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 --2- 172.21.15.16:0/1353051097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55a4100460 0x7f55a4101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 >> 172.21.15.16:0/1353051097 conn(0x7f55a40fbee0 msgr2=0x7f55a4104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 shutdown_connections 2024-09-12T02:59:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:48.607+0000 7f55ab3d4640 1 -- 172.21.15.16:0/1353051097 wait complete. 2024-09-12T02:59:48.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.14 MB/sec execute 530 sec latency 1186.058 ms 2024-09-12T02:59:49.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.12 MB/sec execute 531 sec latency 2186.118 ms 2024-09-12T02:59:49.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:49 smithi033 ceph-mon[35092]: pgmap v694: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:59:50.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:49 smithi184 ceph-mon[35321]: pgmap v694: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:59:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:49 smithi016 ceph-mon[30925]: pgmap v694: 97 pgs: 97 active+clean; 280 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 263 op/s 2024-09-12T02:59:50.312 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T02:59:50.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T02:59:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:50.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.10 MB/sec execute 532 sec latency 3186.214 ms 2024-09-12T02:59:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- 172.21.15.16:0/4192381279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 msgr2=0x7f1030106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 --2- 172.21.15.16:0/4192381279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030106840 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1018009930 tx=0x7f101802a420 comp rx=0 tx=0).stop 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- 172.21.15.16:0/4192381279 shutdown_connections 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 --2- 172.21.15.16:0/4192381279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 0x7f103010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 --2- 172.21.15.16:0/4192381279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030106840 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 --2- 172.21.15.16:0/4192381279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 0x7f1030100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- 172.21.15.16:0/4192381279 >> 172.21.15.16:0/4192381279 conn(0x7f1030077210 msgr2=0x7f10300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- 172.21.15.16:0/4192381279 shutdown_connections 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- 172.21.15.16:0/4192381279 wait complete. 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 Processor -- start 2024-09-12T02:59:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.166+0000 7f1036658640 1 -- start start 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 0x7f10301012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 0x7f1030101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1030111fa0 con 0x7f1030100d00 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1030111e20 con 0x7f1030100350 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1030112120 con 0x7f1030106f50 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 -- 172.21.15.16:0/2278048941 learned_addr learned my addr 172.21.15.16:0/2278048941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f102ffff640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 0x7f10301012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1034bf5640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 0x7f1030101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 -- 172.21.15.16:0/2278048941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 msgr2=0x7f1030101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 0x7f1030101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 -- 172.21.15.16:0/2278048941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 msgr2=0x7f10301012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 0x7f10301012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 -- 172.21.15.16:0/2278048941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10301a6980 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1027fff640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f1018005940 tx=0x7f10180040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f102dffb640 1 -- 172.21.15.16:0/2278048941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101803f030 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f102dffb640 1 -- 172.21.15.16:0/2278048941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10180044c0 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10301a6c10 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.167+0000 7f102dffb640 1 -- 172.21.15.16:0/2278048941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018035070 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.168+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10301a70f0 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.170+0000 7f102dffb640 1 -- 172.21.15.16:0/2278048941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1018042030 con 0x7f1030100d00 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.170+0000 7f1038a31740 1 Processor -- start 2024-09-12T02:59:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.170+0000 7f1038a31740 1 -- start start 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7ccd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7c25280 con 0x55c2f7cbfb60 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7bccb10 con 0x55c2f7ccad40 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1038a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7c21240 con 0x55c2f7cc0480 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51526/0 (socket says 172.21.15.16:51526) 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 -- 172.21.15.16:0/4238563794 learned_addr learned my addr 172.21.15.16:0/4238563794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f102ffff640 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1027fff640 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7ccd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f102dffb640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 0x7f101c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 -- 172.21.15.16:0/4238563794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 msgr2=0x55c2f7cca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 -- 172.21.15.16:0/4238563794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 msgr2=0x55c2f7ccd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7ccd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 -- 172.21.15.16:0/4238563794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2f7bd91f0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f1034bf5640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 0x7f101c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.171+0000 7f102dffb640 1 -- 172.21.15.16:0/2278048941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f10180ef4c0 con 0x7f1030100d00 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f1034bf5640 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f1028010520 tx=0x7f10280109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e6288713457218 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f10267fc640 1 -- 172.21.15.16:0/4238563794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028019070 con 0x55c2f7cbfb60 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f10267fc640 1 -- 172.21.15.16:0/4238563794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1028003e10 con 0x55c2f7cbfb60 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f10267fc640 1 -- 172.21.15.16:0/4238563794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028011770 con 0x55c2f7cbfb60 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f1038a31740 1 -- 172.21.15.16:0/4238563794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 msgr2=0x55c2f7cbff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f1034bf5640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 0x7f101c0a5040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1028021000 tx=0x7f102801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.172+0000 7f1038a31740 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f1028010520 tx=0x7f10280109e0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- 172.21.15.16:0/4238563794 shutdown_connections 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7ccd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- 172.21.15.16:0/4238563794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cbff40 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- 172.21.15.16:0/4238563794 >> 172.21.15.16:0/4238563794 conn(0x55c2f7cbc0f0 msgr2=0x55c2f7cbc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- 172.21.15.16:0/4238563794 shutdown_connections 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- 172.21.15.16:0/4238563794 wait complete. 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- start start 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 Processor -- start 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7ccd910 con 0x55c2f7ccad40 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7bdff10 con 0x55c2f7cc0480 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.173+0000 7f1038a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c2f7ccda90 con 0x55c2f7cbfb60 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1038a31740 10 client.0 _ll_register_callbacks cb 0x55c2f7ba8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51540/0 (socket says 172.21.15.16:51540) 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 -- 172.21.15.16:0/3999655693 learned_addr learned my addr 172.21.15.16:0/3999655693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f102ffff640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1027fff640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 -- 172.21.15.16:0/3999655693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cbfb60 msgr2=0x55c2f7cc7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 -- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 msgr2=0x55c2f7cc80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 -- 172.21.15.16:0/3999655693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2f79966a0 con 0x55c2f7ccad40 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1027fff640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1034bf5640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f10280116f0 tx=0x7f1028011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2744a9537c8123d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f100f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028019070 con 0x55c2f7ccad40 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1028011d60 con 0x55c2f7ccad40 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1028003e10 con 0x55c2f7ccad40 2024-09-12T02:59:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1028013520 con 0x55c2f7ccad40 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.174+0000 7f1038a31740 10 client.18306 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c2f7cd94c0 con 0x55c2f7ccad40 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1028003e10 con 0x55c2f7ccad40 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f102802e070 con 0x55c2f7ccad40 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 10 client.18306 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 10 client.18306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 20 client.18306 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7cdcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 4 client.18306 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2f7c327f0 con 0x55c2f7cda810 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.175+0000 7f102ffff640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7cdcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f102ffff640 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7cdcc00 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=891cad864f1812eb server_cookie=6cb0d6bae9017aca in_seq=0 out_seq=0 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f100f7fe640 10 client.18306 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f100f7fe640 1 -- 172.21.15.16:0/3999655693 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2f7c327f0 con 0x55c2f7cda810 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f100f7fe640 10 client.18306 handle_command_reply: tid=0 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 client.18306 shutdown 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 msgr2=0x55c2f7cc85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f10280116f0 tx=0x7f1028011720 comp rx=0 tx=0).stop 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 shutdown_connections 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7cdcc00 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f100f7fe640 0 client.18306 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.176+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc85f0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 1 --2- 172.21.15.16:0/3999655693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 >> 172.21.15.16:0/3999655693 conn(0x55c2f7cbc0f0 msgr2=0x55c2f7cbb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 shutdown_connections 2024-09-12T02:59:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 1 -- 172.21.15.16:0/3999655693 wait complete. 2024-09-12T02:59:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.177+0000 7f1038a31740 20 client.18306 trim_cache size 0 max 16384 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 Processor -- start 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 -- start start 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10301af7c0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10301b1300 con 0x55c2f7ccad40 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1038a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10301afe90 con 0x55c2f7cc0480 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f102ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f102ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51556/0 (socket says 172.21.15.16:51556) 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f102ffff640 1 -- 172.21.15.16:0/1097793725 learned_addr learned my addr 172.21.15.16:0/1097793725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1027fff640 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.195+0000 7f1034bf5640 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 -- 172.21.15.16:0/1097793725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 msgr2=0x55c2f7cc32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 -- 172.21.15.16:0/1097793725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 msgr2=0x55c2f7cc99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 -- 172.21.15.16:0/1097793725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2f7cc02b0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f102ffff640 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1020000c00 tx=0x7f1020005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36d480f6979da48 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f1025ffb640 1 -- 172.21.15.16:0/1097793725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020022070 con 0x55c2f7cbfb60 2024-09-12T02:59:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f1025ffb640 1 -- 172.21.15.16:0/1097793725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f102001f910 con 0x55c2f7cbfb60 2024-09-12T02:59:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.196+0000 7f1025ffb640 1 -- 172.21.15.16:0/1097793725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020010a90 con 0x55c2f7cbfb60 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- 172.21.15.16:0/1097793725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 msgr2=0x55c2f7cc25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1020000c00 tx=0x7f1020005f90 comp rx=0 tx=0).stop 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- 172.21.15.16:0/1097793725 shutdown_connections 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cc99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cc32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 --2- 172.21.15.16:0/1097793725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cc25c0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- 172.21.15.16:0/1097793725 >> 172.21.15.16:0/1097793725 conn(0x55c2f7cbc0f0 msgr2=0x55c2f7cc92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- 172.21.15.16:0/1097793725 shutdown_connections 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- 172.21.15.16:0/1097793725 wait complete. 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.197+0000 7f1038a31740 1 -- start start 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 Processor -- start 2024-09-12T02:59:51.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10301b0c30 con 0x55c2f7cbfb60 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10301b0560 con 0x55c2f7ccad40 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10301ad7a0 con 0x55c2f7cc0480 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1038a31740 10 client.0 _ll_register_callbacks cb 0x7f1028020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51572/0 (socket says 172.21.15.16:51572) 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 -- 172.21.15.16:0/972441646 learned_addr learned my addr 172.21.15.16:0/972441646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1027fff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f1034bf5640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 -- 172.21.15.16:0/972441646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 msgr2=0x55c2f7cce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 -- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 msgr2=0x55c2f7cce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.198+0000 7f102ffff640 1 -- 172.21.15.16:0/972441646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2f7ccef20 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f102ffff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1020009fa0 tx=0x7f1020009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8803b687377c2f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020022030 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f102001fac0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f102001f910 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f102001fcc0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 10 client.18312 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c2f7d450e0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f102001f910 con 0x55c2f7cbfb60 2024-09-12T02:59:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f10200108a0 con 0x55c2f7cbfb60 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 10 client.18312 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 20 client.18312 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7d47780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 4 client.18312 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.199+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c2f7c2cd00 con 0x55c2f7cda810 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.200+0000 7f1027fff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7d47780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.200+0000 7f1027fff640 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7d47780 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8cd7b8a71a04fe0 server_cookie=15c4714850c13070 in_seq=0 out_seq=0 2024-09-12T02:59:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.200+0000 7f100f7fe640 10 client.18312 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f100f7fe640 1 -- 172.21.15.16:0/972441646 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c2f7c2cd00 con 0x55c2f7cda810 2024-09-12T02:59:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f100f7fe640 10 client.18312 handle_command_reply: tid=0 2024-09-12T02:59:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 client.18312 shutdown 2024-09-12T02:59:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 msgr2=0x55c2f7cd3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1020009fa0 tx=0x7f1020009310 comp rx=0 tx=0).stop 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 shutdown_connections 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c2f7cda810 0x55c2f7d47780 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2f7ccad40 0x55c2f7cce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f100f7fe640 0 client.18312 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c2f7cc0480 0x55c2f7cce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 --2- 172.21.15.16:0/972441646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c2f7cbfb60 0x55c2f7cd3170 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 >> 172.21.15.16:0/972441646 conn(0x55c2f7cbc0f0 msgr2=0x55c2f7cbb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 shutdown_connections 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.201+0000 7f1038a31740 1 -- 172.21.15.16:0/972441646 wait complete. 2024-09-12T02:59:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1038a31740 20 client.18312 trim_cache size 0 max 16384 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "248d6a2a-e783-4432-b5d6-0cf846565d17", 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 msgr2=0x7f101c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 0x7f101c0a5040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1028021000 tx=0x7f102801e000 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 msgr2=0x7f1030101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f1018005940 tx=0x7f10180040c0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 shutdown_connections 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1030106f50 0x7f1030101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f101c0a2b80 0x7f101c0a5040 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1030100d00 0x7f1030101810 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 --2- 172.21.15.16:0/2278048941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1030100350 0x7f10301012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 >> 172.21.15.16:0/2278048941 conn(0x7f1030077210 msgr2=0x7f10300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 shutdown_connections 2024-09-12T02:59:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:51.202+0000 7f1036658640 1 -- 172.21.15.16:0/2278048941 wait complete. 2024-09-12T02:59:51.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.08 MB/sec execute 533 sec latency 4186.355 ms 2024-09-12T02:59:51.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:51 smithi016 ceph-mon[30925]: pgmap v695: 97 pgs: 97 active+clean; 281 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-09-12T02:59:51.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:51 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T02:59:51.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:51 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:51.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:51 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:59:51.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:51 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:51.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T02:59:51.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:51 smithi033 ceph-mon[35092]: pgmap v695: 97 pgs: 97 active+clean; 281 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-09-12T02:59:51.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:51 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T02:59:51.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:51 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:51.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:51 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:59:51.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:51 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:51 smithi184 ceph-mon[35321]: pgmap v695: 97 pgs: 97 active+clean; 281 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-09-12T02:59:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:51 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T02:59:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:51 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:51 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:59:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:51 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T02:59:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 -- 172.21.15.16:0/1838445839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef010e810 msgr2=0x7f0ef0110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/1838445839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef010e810 0x7f0ef0110c00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ca0 tx=0x7f0ee802a8b0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 -- 172.21.15.16:0/1838445839 shutdown_connections 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/1838445839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef010e810 0x7f0ef0110c00 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/1838445839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef0102fc0 0x7f0ef01005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/1838445839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef0102b50 0x7f0ef0100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 -- 172.21.15.16:0/1838445839 >> 172.21.15.16:0/1838445839 conn(0x7f0ef0077210 msgr2=0x7f0ef00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 -- 172.21.15.16:0/1838445839 shutdown_connections 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.581+0000 7f0ef61e0640 1 -- 172.21.15.16:0/1838445839 wait complete. 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 Processor -- start 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 -- start start 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 0x7f0ef019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 0x7f0ef019f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01141d0 con 0x7f0ef0102b50 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef0114050 con 0x7f0ef019eea0 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0ef61e0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef0114350 con 0x7f0ef010e810 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0eef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.582+0000 7f0eef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51588/0 (socket says 172.21.15.16:51588) 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 -- 172.21.15.16:0/2109003430 learned_addr learned my addr 172.21.15.16:0/2109003430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 0x7f0ef019f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 0x7f0ef019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 -- 172.21.15.16:0/2109003430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 msgr2=0x7f0ef019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 0x7f0ef019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 -- 172.21.15.16:0/2109003430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 msgr2=0x7f0ef019f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 0x7f0ef019f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 -- 172.21.15.16:0/2109003430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef019f960 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 0x7f0ef019f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 0x7f0ef019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000d770 tx=0x7f0ee000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eecff9640 1 -- 172.21.15.16:0/2109003430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0017070 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef019fc50 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eecff9640 1 -- 172.21.15.16:0/2109003430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee0005500 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0eecff9640 1 -- 172.21.15.16:0/2109003430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0004e00 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.583+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef01a7430 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0eecff9640 1 -- 172.21.15.16:0/2109003430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0ee0004fa0 con 0x7f0ef0102b50 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 Processor -- start 2024-09-12T02:59:52.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 -- start start 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb40880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb4da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c48eaa62e0 con 0x55c48eb40dc0 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c48ea4dce0 con 0x55c48eb4b680 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.586+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c48eaa22a0 con 0x55c48eb404a0 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51594/0 (socket says 172.21.15.16:51594) 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 -- 172.21.15.16:0/248097718 learned_addr learned my addr 172.21.15.16:0/248097718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eecff9640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 0x7f0ed80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eeffff640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb40880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb4da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 0x7f0ed80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 -- 172.21.15.16:0/248097718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 msgr2=0x55c48eb40880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb40880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 -- 172.21.15.16:0/248097718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 msgr2=0x55c48eb4da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb4da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eecff9640 1 -- 172.21.15.16:0/2109003430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0ee00cebc0 con 0x7f0ef0102b50 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 -- 172.21.15.16:0/248097718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c48ea32440 con 0x55c48eb40dc0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb4da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00cc670 tx=0x7f0ee009b2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83186d12e4e250dc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0ed5ffb640 1 -- 172.21.15.16:0/248097718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0020030 con 0x55c48eb40dc0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 0x7f0ed80a5040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8005e30 tx=0x7f0ee802a8b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.587+0000 7f0ed5ffb640 1 -- 172.21.15.16:0/248097718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee00a4040 con 0x55c48eb40dc0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ed5ffb640 1 -- 172.21.15.16:0/248097718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0009cc0 con 0x55c48eb40dc0 2024-09-12T02:59:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 -- 172.21.15.16:0/248097718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 msgr2=0x55c48eb4b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00cc670 tx=0x7f0ee009b2d0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 -- 172.21.15.16:0/248097718 shutdown_connections 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb4da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb4b140 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/248097718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb40880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 -- 172.21.15.16:0/248097718 >> 172.21.15.16:0/248097718 conn(0x55c48eb3ca30 msgr2=0x55c48eb3ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 -- 172.21.15.16:0/248097718 shutdown_connections 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.588+0000 7f0ef85b9740 1 -- 172.21.15.16:0/248097718 wait complete. 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 -- start start 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 Processor -- start 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb4afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c48eb4e1f0 con 0x55c48eb40dc0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c48ea61050 con 0x55c48eb4b680 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c48eb4e370 con 0x55c48eb404a0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0ef85b9740 10 client.0 _ll_register_callbacks cb 0x55c48eaa1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0eeeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0eeeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51608/0 (socket says 172.21.15.16:51608) 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.589+0000 7f0eeeffd640 1 -- 172.21.15.16:0/3509056684 learned_addr learned my addr 172.21.15.16:0/3509056684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeffff640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb4afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 -- 172.21.15.16:0/3509056684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 msgr2=0x55c48eb4afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb4afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 -- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 msgr2=0x55c48eb487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb487e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 -- 172.21.15.16:0/3509056684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c48e8173c0 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0ef006a1c0 tx=0x7f0ee4010b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bd61e589d710b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eceffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4005220 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0ee40053c0 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee00a4040 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0ee40055c0 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0ef85b9740 10 client.18324 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c48eb59c20 con 0x55c48eb40dc0 2024-09-12T02:59:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.590+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee00a4040 con 0x55c48eb40dc0 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee400f040 con 0x55c48eb40dc0 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 10 client.18324 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 10 client.18324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 20 client.18324 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48eb5b180 0x55c48eb5d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 4 client.18324 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c48eab3850 con 0x55c48eb5b180 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48eb5b180 0x55c48eb5d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0eceffd640 10 client.18324 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.591+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48eb5b180 0x55c48eb5d5a0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66c7af67647c1a71 server_cookie=1b42a415fb3b0df9 in_seq=0 out_seq=0 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0eceffd640 1 -- 172.21.15.16:0/3509056684 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c48eab3850 con 0x55c48eb5b180 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0eceffd640 10 client.18324 handle_command_reply: tid=0 2024-09-12T02:59:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 client.18324 shutdown 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 msgr2=0x55c48eb48280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0ef006a1c0 tx=0x7f0ee4010b00 comp rx=0 tx=0).stop 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 shutdown_connections 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48eb5b180 0x55c48eb5d5a0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb4b680 0x55c48eb487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0eceffd640 0 client.18324 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb40dc0 0x55c48eb48280 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/3509056684 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb404a0 0x55c48eb4afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 >> 172.21.15.16:0/3509056684 conn(0x55c48eb3ca30 msgr2=0x55c48eb42760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 shutdown_connections 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 1 -- 172.21.15.16:0/3509056684 wait complete. 2024-09-12T02:59:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.592+0000 7f0ef85b9740 20 client.18324 trim_cache size 0 max 16384 2024-09-12T02:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 Processor -- start 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 -- start start 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb57450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01af7b0 con 0x55c48eb4b680 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01b12f0 con 0x55c48eb404a0 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.610+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01afe80 con 0x55c48eb40dc0 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeeffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51622/0 (socket says 172.21.15.16:51622) 2024-09-12T02:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 -- 172.21.15.16:0/2201638135 learned_addr learned my addr 172.21.15.16:0/2201638135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb57450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 -- 172.21.15.16:0/2201638135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 msgr2=0x55c48eb62e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb62e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 -- 172.21.15.16:0/2201638135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 msgr2=0x55c48eb57450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb57450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 -- 172.21.15.16:0/2201638135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c48eb48b50 con 0x55c48eb4b680 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb62e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0eeffff640 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80099e0 tx=0x7f0ee8042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfde544dbb6b1a8c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0ed57fa640 1 -- 172.21.15.16:0/2201638135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8043070 con 0x55c48eb4b680 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.611+0000 7f0ed57fa640 1 -- 172.21.15.16:0/2201638135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee8036040 con 0x55c48eb4b680 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ed57fa640 1 -- 172.21.15.16:0/2201638135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8031440 con 0x55c48eb4b680 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 -- 172.21.15.16:0/2201638135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 msgr2=0x55c48eb633b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80099e0 tx=0x7f0ee8042000 comp rx=0 tx=0).stop 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 -- 172.21.15.16:0/2201638135 shutdown_connections 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48eb633b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb62e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/2201638135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb57450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 -- 172.21.15.16:0/2201638135 >> 172.21.15.16:0/2201638135 conn(0x55c48eb3ca30 msgr2=0x55c48eb43900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 -- 172.21.15.16:0/2201638135 shutdown_connections 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.612+0000 7f0ef85b9740 1 -- 172.21.15.16:0/2201638135 wait complete. 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 -- start start 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 Processor -- start 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb4f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb4fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01b0c20 con 0x55c48eb4b680 2024-09-12T02:59:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01b0550 con 0x55c48eb404a0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01ad790 con 0x55c48eb40dc0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0ef85b9740 10 client.0 _ll_register_callbacks cb 0x7f0ee4009420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51638/0 (socket says 172.21.15.16:51638) 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0eeffff640 1 -- 172.21.15.16:0/1664455629 learned_addr learned my addr 172.21.15.16:0/1664455629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0eeeffd640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb4fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.613+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb4f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 -- 172.21.15.16:0/1664455629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 msgr2=0x55c48eb4fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb4fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 -- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 msgr2=0x55c48eb4f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb4f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 -- 172.21.15.16:0/1664455629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c48eb50470 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eeffff640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f0ee802add0 tx=0x7f0ee802afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b293ebe15091316 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eceffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8043070 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0ee8004650 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee8036040 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0ee8067480 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0ef85b9740 10 client.18330 fetch_fsmap learned FSMap version 32 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c48ebcec70 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.614+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee8036040 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee8003970 con 0x55c48eb4b680 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 10 client.18330 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 10 client.18330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T02:59:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 20 client.18330 populate_metadata read hostname 'smithi016' 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48ebcf2d0 0x55c48ebd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 4 client.18330 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c48eaadd60 con 0x55c48ebcf2d0 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48ebcf2d0 0x55c48ebd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48ebcf2d0 0x55c48ebd16c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42b260a18f03de32 server_cookie=726ace363a673cdd in_seq=0 out_seq=0 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.615+0000 7f0eceffd640 10 client.18330 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0eceffd640 1 -- 172.21.15.16:0/1664455629 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c48eaadd60 con 0x55c48ebcf2d0 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0eceffd640 10 client.18330 handle_command_reply: tid=0 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 client.18330 shutdown 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 msgr2=0x55c48ebcd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f0ee802add0 tx=0x7f0ee802afc0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 shutdown_connections 2024-09-12T02:59:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c48ebcf2d0 0x55c48ebd16c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c48eb4b680 0x55c48ebcd650 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c48eb40dc0 0x55c48eb4fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0eceffd640 0 client.18330 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 --2- 172.21.15.16:0/1664455629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c48eb404a0 0x55c48eb4f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 >> 172.21.15.16:0/1664455629 conn(0x55c48eb3ca30 msgr2=0x55c48eb41ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 shutdown_connections 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 1 -- 172.21.15.16:0/1664455629 wait complete. 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.616+0000 7f0ef85b9740 20 client.18330 trim_cache size 0 max 16384 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T02:59:52.626 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: "248d6a2a-e783-4432-b5d6-0cf846565d17": { 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "248d6a2a-e783-4432-b5d6-0cf846565d17", 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T02:59:52.627 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 msgr2=0x7f0ed80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 0x7f0ed80a5040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8005e30 tx=0x7f0ee802a8b0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 msgr2=0x7f0ef019a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000d770 tx=0x7f0ee000dc30 comp rx=0 tx=0).stop 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 shutdown_connections 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ef019eea0 0x7f0ef019f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef010e810 0x7f0ef019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0ed80a2b80 0x7f0ed80a5040 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 --2- 172.21.15.16:0/2109003430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef0102b50 0x7f0ef019a5d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T02:59:52.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 >> 172.21.15.16:0/2109003430 conn(0x7f0ef0077210 msgr2=0x7f0ef010fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T02:59:52.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 shutdown_connections 2024-09-12T02:59:52.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T02:59:52.617+0000 7f0ef61e0640 1 -- 172.21.15.16:0/2109003430 wait complete. 2024-09-12T02:59:52.741 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.06 MB/sec execute 534 sec latency 5186.466 ms 2024-09-12T02:59:52.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:52 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T02:59:52.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:52 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:52.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:52 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T02:59:52.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:52 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:53.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:52 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T02:59:53.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:52 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T02:59:53.561 INFO:tasks.cephfs.filesystem:scrub status for tag:248d6a2a-e783-4432-b5d6-0cf846565d17 - {'path': '/', 'tag': '248d6a2a-e783-4432-b5d6-0cf846565d17', 'options': 'recursive,force'} 2024-09-12T02:59:53.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.04 MB/sec execute 535 sec latency 6186.560 ms 2024-09-12T02:59:53.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:53 smithi016 ceph-mon[30925]: pgmap v696: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 255 op/s 2024-09-12T02:59:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:53 smithi033 ceph-mon[35092]: pgmap v696: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 255 op/s 2024-09-12T02:59:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:53 smithi184 ceph-mon[35321]: pgmap v696: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 255 op/s 2024-09-12T02:59:54.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.02 MB/sec execute 536 sec latency 7186.684 ms 2024-09-12T02:59:55.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 10.00 MB/sec execute 537 sec latency 8186.801 ms 2024-09-12T02:59:55.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:55 smithi033 ceph-mon[35092]: pgmap v697: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-09-12T02:59:56.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:55 smithi184 ceph-mon[35321]: pgmap v697: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-09-12T02:59:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:55 smithi016 ceph-mon[30925]: pgmap v697: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-09-12T02:59:56.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.99 MB/sec execute 538 sec latency 9186.878 ms 2024-09-12T02:59:57.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.97 MB/sec execute 539 sec latency 10186.984 ms 2024-09-12T02:59:57.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:57 smithi033 ceph-mon[35092]: pgmap v698: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-12T02:59:58.120 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:57 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:59:58.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:57 smithi184 ceph-mon[35321]: pgmap v698: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-12T02:59:58.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:57 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:59:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:57 smithi016 ceph-mon[30925]: pgmap v698: 97 pgs: 97 active+clean; 288 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-09-12T02:59:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:57 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:59:58.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.95 MB/sec execute 540 sec latency 11187.085 ms 2024-09-12T02:59:58.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T02:59:59.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T02:59:59.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T02:59:59.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.93 MB/sec execute 541 sec latency 12187.166 ms 2024-09-12T02:59:59.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 02:59:59 smithi033 ceph-mon[35092]: pgmap v699: 97 pgs: 97 active+clean; 285 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 02:59:59 smithi184 ceph-mon[35321]: pgmap v699: 97 pgs: 97 active+clean; 285 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 02:59:59 smithi016 ceph-mon[30925]: pgmap v699: 97 pgs: 97 active+clean; 285 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:00.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.91 MB/sec execute 542 sec latency 13187.276 ms 2024-09-12T03:00:00.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:00 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T03:00:01.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:00 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T03:00:01.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:00 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T03:00:01.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.89 MB/sec execute 543 sec latency 14187.388 ms 2024-09-12T03:00:01.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:01 smithi033 ceph-mon[35092]: pgmap v700: 97 pgs: 97 active+clean; 286 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:02.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:01 smithi184 ceph-mon[35321]: pgmap v700: 97 pgs: 97 active+clean; 286 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:01 smithi016 ceph-mon[30925]: pgmap v700: 97 pgs: 97 active+clean; 286 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.6 MiB/s wr, 124 op/s 2024-09-12T03:00:02.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.87 MB/sec execute 544 sec latency 15187.461 ms 2024-09-12T03:00:03.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.86 MB/sec execute 545 sec latency 16187.521 ms 2024-09-12T03:00:03.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:03 smithi033 ceph-mon[35092]: pgmap v701: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-09-12T03:00:04.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:03 smithi184 ceph-mon[35321]: pgmap v701: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-09-12T03:00:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:03 smithi016 ceph-mon[30925]: pgmap v701: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-09-12T03:00:04.742 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.84 MB/sec execute 546 sec latency 17187.575 ms 2024-09-12T03:00:05.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.82 MB/sec execute 547 sec latency 18187.604 ms 2024-09-12T03:00:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:05 smithi033 ceph-mon[35092]: pgmap v702: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-09-12T03:00:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:00:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:05.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:05 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:05 smithi184 ceph-mon[35321]: pgmap v702: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-09-12T03:00:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:00:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:05 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:05 smithi016 ceph-mon[30925]: pgmap v702: 97 pgs: 97 active+clean; 287 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-09-12T03:00:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:00:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:05 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:06.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.80 MB/sec execute 548 sec latency 19187.642 ms 2024-09-12T03:00:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:00:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:00:07.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:06 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:00:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:00:07.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:06 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:00:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:00:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:06 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:00:07.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.78 MB/sec execute 549 sec latency 20187.713 ms 2024-09-12T03:00:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:07 smithi033 ceph-mon[35092]: pgmap v703: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-09-12T03:00:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:07 smithi184 ceph-mon[35321]: pgmap v703: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-09-12T03:00:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:07 smithi016 ceph-mon[30925]: pgmap v703: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-09-12T03:00:08.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.77 MB/sec execute 550 sec latency 21187.782 ms 2024-09-12T03:00:09.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.75 MB/sec execute 551 sec latency 22187.828 ms 2024-09-12T03:00:09.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:00:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:09 smithi033 ceph-mon[35092]: pgmap v704: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.0 MiB/s wr, 77 op/s 2024-09-12T03:00:10.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:09 smithi184 ceph-mon[35321]: pgmap v704: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.0 MiB/s wr, 77 op/s 2024-09-12T03:00:10.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:09 smithi016 ceph-mon[30925]: pgmap v704: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 2.0 MiB/s wr, 77 op/s 2024-09-12T03:00:10.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.73 MB/sec execute 552 sec latency 23187.885 ms 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 -- 172.21.15.16:0/4168932963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0101420 msgr2=0x7f3eb010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 --2- 172.21.15.16:0/4168932963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0101420 0x7f3eb010d950 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009cd0 tx=0x7f3ea802a8e0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 -- 172.21.15.16:0/4168932963 shutdown_connections 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 --2- 172.21.15.16:0/4168932963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0101420 0x7f3eb010d950 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 --2- 172.21.15.16:0/4168932963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0100a80 0x7f3eb0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 --2- 172.21.15.16:0/4168932963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0106a80 0x7f3eb0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 -- 172.21.15.16:0/4168932963 >> 172.21.15.16:0/4168932963 conn(0x7f3eb00fc7a0 msgr2=0x7f3eb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 -- 172.21.15.16:0/4168932963 shutdown_connections 2024-09-12T03:00:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.822+0000 7f3eb6383640 1 -- 172.21.15.16:0/4168932963 wait complete. 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.823+0000 7f3eb6383640 1 Processor -- start 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.823+0000 7f3eb6383640 1 -- start start 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.823+0000 7f3eb6383640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0100a80 0x7f3eb0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.823+0000 7f3eb6383640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0101420 0x7f3eb0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.823+0000 7f3eb6383640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb6383640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0110e80 con 0x7f3eb0106a80 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb6383640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0110d00 con 0x7f3eb0101420 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb6383640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0111000 con 0x7f3eb0100a80 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40254/0 (socket says 172.21.15.16:40254) 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 -- 172.21.15.16:0/1347383603 learned_addr learned my addr 172.21.15.16:0/1347383603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eaffff640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0100a80 0x7f3eb0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eaf7fe640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0101420 0x7f3eb0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 -- 172.21.15.16:0/1347383603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0100a80 msgr2=0x7f3eb0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0100a80 0x7f3eb0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 -- 172.21.15.16:0/1347383603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0101420 msgr2=0x7f3eb0198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0101420 0x7f3eb0198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 -- 172.21.15.16:0/1347383603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb019d3d0 con 0x7f3eb0106a80 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb4920640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8005cc0 tx=0x7f3ea8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8004040 con 0x7f3eb0106a80 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea80046c0 con 0x7f3eb0106a80 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb019d660 con 0x7f3eb0106a80 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8046480 con 0x7f3eb0106a80 2024-09-12T03:00:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.824+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb01a4fa0 con 0x7f3eb0106a80 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.826+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb0100ee0 con 0x7f3eb0106a80 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.827+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3ea80305a0 con 0x7f3eb0106a80 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.827+0000 7f3ead7fa640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 0x7f3e980a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.828+0000 7f3eaffff640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 0x7f3e980a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.828+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ea802e030 con 0x7f3eb0106a80 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.828+0000 7f3eaffff640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 0x7f3e980a5040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0005d00 tx=0x7f3ea000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:10.832+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ea804b060 con 0x7f3eb0106a80 2024-09-12T03:00:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.043+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb01a5170 con 0x7f3eb0106a80 2024-09-12T03:00:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.044+0000 7f3ead7fa640 1 -- 172.21.15.16:0/1347383603 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f3ea80bd7d0 con 0x7f3eb0106a80 2024-09-12T03:00:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:00:11.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T03:00:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 msgr2=0x7f3e980a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 0x7f3e980a5040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0005d00 tx=0x7f3ea000b410 comp rx=0 tx=0).stop 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 msgr2=0x7f3eb019ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8005cc0 tx=0x7f3ea8003970 comp rx=0 tx=0).stop 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 shutdown_connections 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3e980a2b80 0x7f3e980a5040 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb0106a80 0x7f3eb019ccf0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3eb0101420 0x7f3eb0198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 --2- 172.21.15.16:0/1347383603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb0100a80 0x7f3eb0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 >> 172.21.15.16:0/1347383603 conn(0x7f3eb00fc7a0 msgr2=0x7f3eb0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 shutdown_connections 2024-09-12T03:00:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:11.046+0000 7f3eb6383640 1 -- 172.21.15.16:0/1347383603 wait complete. 2024-09-12T03:00:11.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211332 9.71 MB/sec execute 553 sec latency 24187.934 ms 2024-09-12T03:00:11.819 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 1157782c --release --await' 2024-09-12T03:00:11.819 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 1157782c --release --await 2024-09-12T03:00:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:11 smithi016 ceph-mon[30925]: pgmap v705: 97 pgs: 97 active+clean; 284 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-09-12T03:00:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:11 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1347383603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:12.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:11 smithi033 ceph-mon[35092]: pgmap v705: 97 pgs: 97 active+clean; 284 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-09-12T03:00:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:11 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1347383603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:11 smithi184 ceph-mon[35321]: pgmap v705: 97 pgs: 97 active+clean; 284 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-09-12T03:00:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:11 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1347383603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.655+0000 7f64ad6c7640 1 -- 172.21.15.16:0/2945810431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8106f10 msgr2=0x7f64a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.655+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/2945810431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8106f10 0x7f64a810ea90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ca0 tx=0x7f64a002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 -- 172.21.15.16:0/2945810431 shutdown_connections 2024-09-12T03:00:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/2945810431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8106f10 0x7f64a810ea90 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/2945810431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8102790 0x7f64a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/2945810431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8101de0 0x7f64a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 -- 172.21.15.16:0/2945810431 >> 172.21.15.16:0/2945810431 conn(0x7f64a80fd8c0 msgr2=0x7f64a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 -- 172.21.15.16:0/2945810431 shutdown_connections 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 -- 172.21.15.16:0/2945810431 wait complete. 2024-09-12T03:00:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 Processor -- start 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.656+0000 7f64ad6c7640 1 -- start start 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8102790 0x7f64a81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8106f10 0x7f64a819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8111f70 con 0x7f64a8101de0 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8111df0 con 0x7f64a8102790 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64ad6c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81120f0 con 0x7f64a8106f10 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40274/0 (socket says 172.21.15.16:40274) 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 -- 172.21.15.16:0/3381059964 learned_addr learned my addr 172.21.15.16:0/3381059964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8102790 0x7f64a81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a77fe640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8106f10 0x7f64a819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 -- 172.21.15.16:0/3381059964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8106f10 msgr2=0x7f64a819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8106f10 0x7f64a819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 -- 172.21.15.16:0/3381059964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8102790 msgr2=0x7f64a81988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8102790 0x7f64a81988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 -- 172.21.15.16:0/3381059964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a819d360 con 0x7f64a8101de0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.657+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f649800d950 tx=0x7f649800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.658+0000 7f648ffff640 1 -- 172.21.15.16:0/3381059964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498017070 con 0x7f64a8101de0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.658+0000 7f648ffff640 1 -- 172.21.15.16:0/3381059964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6498009c30 con 0x7f64a8101de0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.658+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a819d650 con 0x7f64a8101de0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.658+0000 7f648ffff640 1 -- 172.21.15.16:0/3381059964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649800f6b0 con 0x7f64a8101de0 2024-09-12T03:00:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.658+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a81a4f30 con 0x7f64a8101de0 2024-09-12T03:00:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.660+0000 7f648ffff640 1 -- 172.21.15.16:0/3381059964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f64980109d0 con 0x7f64a8101de0 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.660+0000 7f64afaa0740 1 Processor -- start 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.660+0000 7f64afaa0740 1 -- start start 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ffc506620 con 0x559ffc5a1980 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ffc4ade50 con 0x559ffc5ac240 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64afaa0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ffc5025e0 con 0x559ffc5a1060 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40280/0 (socket says 172.21.15.16:40280) 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 -- 172.21.15.16:0/1586423674 learned_addr learned my addr 172.21.15.16:0/1586423674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a67fc640 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f648ffff640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 0x7f64900a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a77fe640 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 -- 172.21.15.16:0/1586423674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 msgr2=0x559ffc5a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 -- 172.21.15.16:0/1586423674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 msgr2=0x559ffc5ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a6ffd640 1 -- 172.21.15.16:0/1586423674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ffc49ed40 con 0x559ffc5a1980 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f64a77fe640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 0x7f64900a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.661+0000 7f648ffff640 1 -- 172.21.15.16:0/3381059964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f64980ceb70 con 0x7f64a8101de0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6498097e60 tx=0x7f6498005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6a525454df74f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f648d7fa640 1 -- 172.21.15.16:0/1586423674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649809afd0 con 0x559ffc5a1980 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f64a77fe640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 0x7f64900a50c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f64a0005e30 tx=0x7f64a003f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f648d7fa640 1 -- 172.21.15.16:0/1586423674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6498009de0 con 0x559ffc5a1980 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f648d7fa640 1 -- 172.21.15.16:0/1586423674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649809adf0 con 0x559ffc5a1980 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f64afaa0740 1 -- 172.21.15.16:0/1586423674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 msgr2=0x559ffc5abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.662+0000 7f64afaa0740 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6498097e60 tx=0x7f6498005f90 comp rx=0 tx=0).stop 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- 172.21.15.16:0/1586423674 shutdown_connections 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5abd00 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- 172.21.15.16:0/1586423674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- 172.21.15.16:0/1586423674 >> 172.21.15.16:0/1586423674 conn(0x559ffc59d5f0 msgr2=0x559ffc59d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- 172.21.15.16:0/1586423674 shutdown_connections 2024-09-12T03:00:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- 172.21.15.16:0/1586423674 wait complete. 2024-09-12T03:00:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- start start 2024-09-12T03:00:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 Processor -- start 2024-09-12T03:00:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ffc5aedb0 con 0x559ffc5a1980 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ffc48ef10 con 0x559ffc5ac240 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ffc5aef30 con 0x559ffc5a1060 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64afaa0740 10 client.0 _ll_register_callbacks cb 0x559ffc4b2db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.663+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40288/0 (socket says 172.21.15.16:40288) 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 -- 172.21.15.16:0/3467407331 learned_addr learned my addr 172.21.15.16:0/3467407331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a77fe640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 -- 172.21.15.16:0/3467407331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 msgr2=0x559ffc5a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 -- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 msgr2=0x559ffc5a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 -- 172.21.15.16:0/3467407331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ffc215580 con 0x559ffc5a1980 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f649c00b990 tx=0x7f649c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81973829d775c8f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f648aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c003e00 con 0x559ffc5a1980 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f649c005020 con 0x559ffc5a1980 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6498009de0 con 0x559ffc5a1980 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f649c005220 con 0x559ffc5a1980 2024-09-12T03:00:12.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64afaa0740 10 client.18348 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.664+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ffc2df960 con 0x559ffc5a1980 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6498009de0 con 0x559ffc5a1980 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f649c01b570 con 0x559ffc5a1980 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 10 client.18348 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 10 client.18348 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 20 client.18348 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5bbe10 0x559ffc5be230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 4 client.18348 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ffc513b90 con 0x559ffc5bbe10 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.665+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5bbe10 0x559ffc5be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5bbe10 0x559ffc5be230 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a546d93507b8a64 server_cookie=e04a6429e6c5e8b2 in_seq=0 out_seq=0 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f648aff5640 10 client.18348 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f648aff5640 1 -- 172.21.15.16:0/3467407331 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ffc513b90 con 0x559ffc5bbe10 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f648aff5640 10 client.18348 handle_command_reply: tid=0 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 client.18348 shutdown 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 msgr2=0x559ffc5a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f649c00b990 tx=0x7f649c00be50 comp rx=0 tx=0).stop 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 shutdown_connections 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5bbe10 0x559ffc5be230 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc5ac240 0x559ffc5a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc5a9680 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3467407331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 >> 172.21.15.16:0/3467407331 conn(0x559ffc59d5f0 msgr2=0x559ffc5a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.666+0000 7f648aff5640 0 client.18348 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.667+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 shutdown_connections 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.667+0000 7f64afaa0740 1 -- 172.21.15.16:0/3467407331 wait complete. 2024-09-12T03:00:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.667+0000 7f64afaa0740 20 client.18348 trim_cache size 0 max 16384 2024-09-12T03:00:12.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.683+0000 7f64afaa0740 1 Processor -- start 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.683+0000 7f64afaa0740 1 -- start start 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1980 0x559ffc5a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ad590 con 0x559ffc5a1060 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81af0d0 con 0x559ffc59d5f0 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64afaa0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81adc60 con 0x559ffc5a1980 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40304/0 (socket says 172.21.15.16:40304) 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 -- 172.21.15.16:0/3075175257 learned_addr learned my addr 172.21.15.16:0/3075175257 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a77fe640 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1980 0x559ffc5a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 -- 172.21.15.16:0/3075175257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1980 msgr2=0x559ffc5a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1980 0x559ffc5a63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 -- 172.21.15.16:0/3075175257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 msgr2=0x559ffc5c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 -- 172.21.15.16:0/3075175257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ffc5a17b0 con 0x559ffc5a1060 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f64a67fc640 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f649c003a40 tx=0x7f649c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d683a03d5d33e29 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f648cff9640 1 -- 172.21.15.16:0/3075175257 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c003e80 con 0x559ffc5a1060 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.684+0000 7f648cff9640 1 -- 172.21.15.16:0/3075175257 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c01bc00 con 0x559ffc5a1060 2024-09-12T03:00:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f648cff9640 1 -- 172.21.15.16:0/3075175257 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c026b90 con 0x559ffc5a1060 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- 172.21.15.16:0/3075175257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 msgr2=0x559ffc5c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f649c003a40 tx=0x7f649c005f70 comp rx=0 tx=0).stop 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- 172.21.15.16:0/3075175257 shutdown_connections 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1980 0x559ffc5a63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1060 0x559ffc5c4040 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 --2- 172.21.15.16:0/3075175257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- 172.21.15.16:0/3075175257 >> 172.21.15.16:0/3075175257 conn(0x559ffc59e070 msgr2=0x559ffc5a7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- 172.21.15.16:0/3075175257 shutdown_connections 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- 172.21.15.16:0/3075175257 wait complete. 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.685+0000 7f64afaa0740 1 -- start start 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 Processor -- start 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81aea00 con 0x559ffc5a1980 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ae330 con 0x559ffc59d5f0 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ab570 con 0x559ffc5a1060 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64afaa0740 10 client.0 _ll_register_callbacks cb 0x7f649c004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40310/0 (socket says 172.21.15.16:40310) 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 -- 172.21.15.16:0/2921583794 learned_addr learned my addr 172.21.15.16:0/2921583794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a67fc640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 -- 172.21.15.16:0/2921583794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 msgr2=0x559ffc5b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 -- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 msgr2=0x559ffc5b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.686+0000 7f64a77fe640 1 -- 172.21.15.16:0/2921583794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ffc6260c0 con 0x559ffc5a1980 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64a77fe640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ca0 tx=0x7f64a0004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbde63265b42819 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0003970 con 0x559ffc5a1980 2024-09-12T03:00:12.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f64a0003b10 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f649c01bc00 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f64a0003d10 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 10 client.18354 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ffc6273e0 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f649c01bc00 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f64a005c020 con 0x559ffc5a1980 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 10 client.18354 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 10 client.18354 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 20 client.18354 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5ac240 0x559ffc629ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 4 client.18354 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1157782c", "release": true, "await": true}] 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.687+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1157782c", "release": true, "await": true}) -- 0x559ffc50e0a0 con 0x559ffc5ac240 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.688+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5ac240 0x559ffc629ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.688+0000 7f648aff5640 10 client.18354 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.688+0000 7f64a6ffd640 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5ac240 0x559ffc629ba0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=768f3a1b9024a04e server_cookie=41712434d03c9e66 in_seq=0 out_seq=0 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f648aff5640 1 -- 172.21.15.16:0/2921583794 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559ffc50e0a0 con 0x559ffc5ac240 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f648aff5640 10 client.18354 handle_command_reply: tid=0 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 client.18354 shutdown 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 msgr2=0x559ffc625ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ca0 tx=0x7f64a0004a80 comp rx=0 tx=0).stop 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 shutdown_connections 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559ffc5ac240 0x559ffc629ba0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f648aff5640 0 client.18354 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ffc5a1980 0x559ffc625ad0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ffc5a1060 0x559ffc5b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 --2- 172.21.15.16:0/2921583794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ffc59d5f0 0x559ffc5b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 >> 172.21.15.16:0/2921583794 conn(0x559ffc59e070 msgr2=0x559ffc59bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 shutdown_connections 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 1 -- 172.21.15.16:0/2921583794 wait complete. 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.698+0000 7f64afaa0740 20 client.18354 trim_cache size 0 max 16384 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 32, 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 48, 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:00:12.700 INFO:teuthology.orchestra.run.smithi016.stdout: "1157782c": { 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 48, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.2, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:00:12.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:00:12.702 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:00:12.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:12.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:12.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:12.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:12.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:12.705 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:00:12.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 msgr2=0x7f64900a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 0x7f64900a50c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f64a0005e30 tx=0x7f64a003f040 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 msgr2=0x7f64a81983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f649800d950 tx=0x7f649800de10 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 shutdown_connections 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64a8106f10 0x7f64a819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64a8102790 0x7f64a81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f64900a2c00 0x7f64900a50c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 --2- 172.21.15.16:0/3381059964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64a8101de0 0x7f64a81983b0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 >> 172.21.15.16:0/3381059964 conn(0x7f64a80fd8c0 msgr2=0x7f64a80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 shutdown_connections 2024-09-12T03:00:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:12.699+0000 7f64ad6c7640 1 -- 172.21.15.16:0/3381059964 wait complete. 2024-09-12T03:00:12.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1211335 9.70 MB/sec execute 554 sec latency 25144.668 ms 2024-09-12T03:00:13.402 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1157782c', total seconds elapsed: 30.6 2024-09-12T03:00:13.759 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1213588 9.69 MB/sec execute 555 sec latency 42.962 ms 2024-09-12T03:00:14.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:13 smithi033 ceph-mon[35092]: pgmap v706: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T03:00:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:13 smithi184 ceph-mon[35321]: pgmap v706: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T03:00:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:13 smithi016 ceph-mon[30925]: pgmap v706: 97 pgs: 97 active+clean; 283 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-12T03:00:14.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1216047 9.71 MB/sec execute 556 sec latency 12.094 ms 2024-09-12T03:00:15.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1218673 9.71 MB/sec execute 557 sec latency 51.295 ms 2024-09-12T03:00:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:15 smithi033 ceph-mon[35092]: pgmap v707: 97 pgs: 97 active+clean; 275 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-12T03:00:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:15 smithi184 ceph-mon[35321]: pgmap v707: 97 pgs: 97 active+clean; 275 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-12T03:00:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:15 smithi016 ceph-mon[30925]: pgmap v707: 97 pgs: 97 active+clean; 275 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-12T03:00:16.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1221597 9.72 MB/sec execute 558 sec latency 14.652 ms 2024-09-12T03:00:17.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1224113 9.73 MB/sec execute 559 sec latency 12.050 ms 2024-09-12T03:00:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:17 smithi033 ceph-mon[35092]: pgmap v708: 97 pgs: 97 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-09-12T03:00:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:17 smithi184 ceph-mon[35321]: pgmap v708: 97 pgs: 97 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-09-12T03:00:18.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:17 smithi016 ceph-mon[30925]: pgmap v708: 97 pgs: 97 active+clean; 278 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-09-12T03:00:18.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1226091 9.73 MB/sec execute 560 sec latency 81.527 ms 2024-09-12T03:00:19.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1229138 9.75 MB/sec execute 561 sec latency 8.977 ms 2024-09-12T03:00:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:19 smithi033 ceph-mon[35092]: pgmap v709: 97 pgs: 97 active+clean; 273 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-09-12T03:00:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:19 smithi184 ceph-mon[35321]: pgmap v709: 97 pgs: 97 active+clean; 273 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-09-12T03:00:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:19 smithi016 ceph-mon[30925]: pgmap v709: 97 pgs: 97 active+clean; 273 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-09-12T03:00:20.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1231341 9.74 MB/sec execute 562 sec latency 99.633 ms 2024-09-12T03:00:21.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1233556 9.75 MB/sec execute 563 sec latency 11.637 ms 2024-09-12T03:00:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:21 smithi033 ceph-mon[35092]: pgmap v710: 97 pgs: 97 active+clean; 276 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-09-12T03:00:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:21 smithi184 ceph-mon[35321]: pgmap v710: 97 pgs: 97 active+clean; 276 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-09-12T03:00:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:21 smithi016 ceph-mon[30925]: pgmap v710: 97 pgs: 97 active+clean; 276 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 207 op/s 2024-09-12T03:00:22.743 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1236341 9.76 MB/sec execute 564 sec latency 10.438 ms 2024-09-12T03:00:23.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:00:23.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1239456 9.78 MB/sec execute 565 sec latency 12.154 ms 2024-09-12T03:00:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:23 smithi033 ceph-mon[35092]: pgmap v711: 97 pgs: 97 active+clean; 275 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 203 op/s 2024-09-12T03:00:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:23 smithi184 ceph-mon[35321]: pgmap v711: 97 pgs: 97 active+clean; 275 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 203 op/s 2024-09-12T03:00:24.325 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:23 smithi016 ceph-mon[30925]: pgmap v711: 97 pgs: 97 active+clean; 275 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 203 op/s 2024-09-12T03:00:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 -- 172.21.15.16:0/3990328477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c102790 msgr2=0x7fa02c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3990328477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c102790 0x7fa02c1067e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa020009900 tx=0x7fa02002a410 comp rx=0 tx=0).stop 2024-09-12T03:00:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 -- 172.21.15.16:0/3990328477 shutdown_connections 2024-09-12T03:00:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3990328477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c106f10 0x7fa02c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3990328477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c102790 0x7fa02c1067e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3990328477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c101de0 0x7fa02c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 -- 172.21.15.16:0/3990328477 >> 172.21.15.16:0/3990328477 conn(0x7fa02c0fd8c0 msgr2=0x7fa02c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 -- 172.21.15.16:0/3990328477 shutdown_connections 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.411+0000 7fa0308f4640 1 -- 172.21.15.16:0/3990328477 wait complete. 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 Processor -- start 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 -- start start 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c102790 0x7fa02c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 0x7fa02c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c111f70 con 0x7fa02c106f10 2024-09-12T03:00:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c111df0 con 0x7fa02c102790 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa0308f4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1120f0 con 0x7fa02c101de0 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa02a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa02ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 0x7fa02c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa02a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55010/0 (socket says 172.21.15.16:55010) 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa02a59c640 1 -- 172.21.15.16:0/3232349334 learned_addr learned my addr 172.21.15.16:0/3232349334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.412+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c102790 0x7fa02c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 -- 172.21.15.16:0/3232349334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c102790 msgr2=0x7fa02c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c102790 0x7fa02c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 -- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 msgr2=0x7fa02c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 0x7fa02c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 -- 172.21.15.16:0/3232349334 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c19d440 con 0x7fa02c101de0 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 0x7fa02c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00e9f0 tx=0x7fa01c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa01b7fe640 1 -- 172.21.15.16:0/3232349334 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c00cd80 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02c19d730 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa01b7fe640 1 -- 172.21.15.16:0/3232349334 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa01c010910 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa01b7fe640 1 -- 172.21.15.16:0/3232349334 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0188f0 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.413+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02c074420 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa01b7fe640 1 -- 172.21.15.16:0/3232349334 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa01c010ac0 con 0x7fa02c101de0 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 Processor -- start 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 -- start start 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a040880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a040dc0 0x55fe4a04b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fe49fa62e0 con 0x55fe4a04b680 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fe49f4dce0 con 0x55fe4a0404a0 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa032ccd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fe49fa22a0 con 0x55fe4a040dc0 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa02ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a040880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa02a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a040dc0 0x55fe4a04b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34844/0 (socket says 172.21.15.16:34844) 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.416+0000 7fa029d9b640 1 -- 172.21.15.16:0/3348744570 learned_addr learned my addr 172.21.15.16:0/3348744570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 -- 172.21.15.16:0/3348744570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a040dc0 msgr2=0x55fe4a04b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a040dc0 0x55fe4a04b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 -- 172.21.15.16:0/3348744570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 msgr2=0x55fe4a040880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a040880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 -- 172.21.15.16:0/3348744570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe49f32440 con 0x55fe4a04b680 2024-09-12T03:00:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a040880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fa020009900 tx=0x7fa0200058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4d461b8537a943 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa01b7fe640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 0x7fa0140a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa018ff9640 1 -- 172.21.15.16:0/3348744570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020048020 con 0x55fe4a04b680 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 0x7fa0140a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa018ff9640 1 -- 172.21.15.16:0/3348744570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0200317b0 con 0x55fe4a04b680 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa01b7fe640 1 -- 172.21.15.16:0/3232349334 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa01c0cf3c0 con 0x7fa02c101de0 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.417+0000 7fa018ff9640 1 -- 172.21.15.16:0/3348744570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020035050 con 0x55fe4a04b680 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 -- 172.21.15.16:0/3348744570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 msgr2=0x55fe4a04da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 0x7fa0140a5040 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa024009fd0 tx=0x7fa024009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fa020009900 tx=0x7fa0200058d0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 -- 172.21.15.16:0/3348744570 shutdown_connections 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a04da70 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a040dc0 0x55fe4a04b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3348744570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a040880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 -- 172.21.15.16:0/3348744570 >> 172.21.15.16:0/3348744570 conn(0x55fe4a03ca30 msgr2=0x55fe4a03ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 -- 172.21.15.16:0/3348744570 shutdown_connections 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.418+0000 7fa032ccd740 1 -- 172.21.15.16:0/3348744570 wait complete. 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 -- start start 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 Processor -- start 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a0404a0 0x55fe4a04af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 0x55fe4a048710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fe4a04e250 con 0x55fe4a040dc0 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fe49f61050 con 0x55fe4a04b680 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fe4a04e3d0 con 0x55fe4a0404a0 2024-09-12T03:00:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa032ccd740 10 client.0 _ll_register_callbacks cb 0x55fe49fa1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34854/0 (socket says 172.21.15.16:34854) 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa029d9b640 1 -- 172.21.15.16:0/1591922576 learned_addr learned my addr 172.21.15.16:0/1591922576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 0x55fe4a048710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.419+0000 7fa02a59c640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a0404a0 0x55fe4a04af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 -- 172.21.15.16:0/1591922576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a0404a0 msgr2=0x55fe4a04af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a0404a0 0x55fe4a04af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 -- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 msgr2=0x55fe4a048710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 0x55fe4a048710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 -- 172.21.15.16:0/1591922576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe49d173c0 con 0x55fe4a040dc0 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 0x55fe4a048710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa029d9b640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa020005940 tx=0x7fa02002ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8909c7fd1e2c9e3e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020048040 con 0x55fe4a040dc0 2024-09-12T03:00:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa020031960 con 0x55fe4a040dc0 2024-09-12T03:00:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0200317b0 con 0x55fe4a040dc0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa02005c050 con 0x55fe4a040dc0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa032ccd740 10 client.18366 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fe4a059b70 con 0x55fe4a040dc0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0200317b0 con 0x55fe4a040dc0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.420+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa02003a940 con 0x55fe4a040dc0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 10 client.18366 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 20 client.18366 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a05b050 0x55fe4a05d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 4 client.18366 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe49fb3850 con 0x55fe4a05b050 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa02a59c640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a05b050 0x55fe4a05d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa02a59c640 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a05b050 0x55fe4a05d440 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb7680c9742bb7f0 server_cookie=a7a35f870f1e6b2b in_seq=0 out_seq=0 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.421+0000 7fa011ffb640 10 client.18366 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa011ffb640 1 -- 172.21.15.16:0/1591922576 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe49fb3850 con 0x55fe4a05b050 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa011ffb640 10 client.18366 handle_command_reply: tid=0 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 client.18366 shutdown 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 msgr2=0x55fe4a0481d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa020005940 tx=0x7fa02002ab90 comp rx=0 tx=0).stop 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 shutdown_connections 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a05b050 0x55fe4a05d440 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a04b680 0x55fe4a048710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a040dc0 0x55fe4a0481d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 --2- 172.21.15.16:0/1591922576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a0404a0 0x55fe4a04af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 >> 172.21.15.16:0/1591922576 conn(0x55fe4a03ca30 msgr2=0x55fe4a042710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa011ffb640 0 client.18366 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 shutdown_connections 2024-09-12T03:00:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 1 -- 172.21.15.16:0/1591922576 wait complete. 2024-09-12T03:00:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.422+0000 7fa032ccd740 20 client.18366 trim_cache size 0 max 16384 2024-09-12T03:00:24.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 Processor -- start 2024-09-12T03:00:24.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 -- start start 2024-09-12T03:00:24.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a043900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a042b40 0x55fe4a05fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1ab250 con 0x55fe4a04b680 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1acd90 con 0x55fe4a0404a0 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa032ccd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1ab920 con 0x55fe4a042b40 2024-09-12T03:00:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34864/0 (socket says 172.21.15.16:34864) 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa029d9b640 1 -- 172.21.15.16:0/2989241507 learned_addr learned my addr 172.21.15.16:0/2989241507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa02a59c640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a043900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.441+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a042b40 0x55fe4a05fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 -- 172.21.15.16:0/2989241507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a042b40 msgr2=0x55fe4a05fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a042b40 0x55fe4a05fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 -- 172.21.15.16:0/2989241507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 msgr2=0x55fe4a043900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a043900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 -- 172.21.15.16:0/2989241507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe4a047180 con 0x55fe4a04b680 2024-09-12T03:00:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa02a59c640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a043900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa029d9b640 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa020005390 tx=0x7fa0200045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff927d2fd6740e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa013fff640 1 -- 172.21.15.16:0/2989241507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020035070 con 0x55fe4a04b680 2024-09-12T03:00:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa013fff640 1 -- 172.21.15.16:0/2989241507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02003a870 con 0x55fe4a04b680 2024-09-12T03:00:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.442+0000 7fa013fff640 1 -- 172.21.15.16:0/2989241507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200039e0 con 0x55fe4a04b680 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- 172.21.15.16:0/2989241507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 msgr2=0x55fe4a042600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa020005390 tx=0x7fa0200045a0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- 172.21.15.16:0/2989241507 shutdown_connections 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a042b40 0x55fe4a05fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a04b680 0x55fe4a042600 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 --2- 172.21.15.16:0/2989241507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a043900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- 172.21.15.16:0/2989241507 >> 172.21.15.16:0/2989241507 conn(0x55fe4a03ca30 msgr2=0x55fe4a041ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- 172.21.15.16:0/2989241507 shutdown_connections 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- 172.21.15.16:0/2989241507 wait complete. 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 -- start start 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.443+0000 7fa032ccd740 1 Processor -- start 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 0x55fe4a05dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a04b680 0x55fe4a05e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1ac6c0 con 0x55fe4a042b40 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1abff0 con 0x55fe4a0404a0 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 0x55fe4a05dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1a9230 con 0x55fe4a04b680 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa029d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 0x55fe4a05dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34868/0 (socket says 172.21.15.16:34868) 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34602/0 (socket says 172.21.15.16:34602) 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 -- 172.21.15.16:0/3755273972 learned_addr learned my addr 172.21.15.16:0/3755273972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02ad9d640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a04b680 0x55fe4a05e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa032ccd740 10 client.0 _ll_register_callbacks cb 0x7fa024009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 -- 172.21.15.16:0/3755273972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a04b680 msgr2=0x55fe4a05e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a04b680 0x55fe4a05e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 -- 172.21.15.16:0/3755273972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 msgr2=0x55fe4a05dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 0x55fe4a05dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 -- 172.21.15.16:0/3755273972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4a04b290 con 0x55fe4a0404a0 2024-09-12T03:00:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa02a59c640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00c6a0 tx=0x7fa01c0cf7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=111d5ed77a060400 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa011ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.444+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c09ffe0 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02003a870 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa01c098350 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa01c0a48a0 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 10 client.27779 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fe4a0c58a0 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02003a870 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa01c097c30 con 0x55fe4a0404a0 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 10 client.27779 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 10 client.27779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 20 client.27779 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a0c5fc0 0x55fe4a0c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 4 client.27779 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe49fadd60 con 0x55fe4a0c5fc0 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.445+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a0c5fc0 0x55fe4a0c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa011ffb640 10 client.27779 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa029d9b640 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a0c5fc0 0x55fe4a0c83b0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d38b935612c10257 server_cookie=392dc93775344e7e in_seq=0 out_seq=0 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa011ffb640 1 -- 172.21.15.16:0/3755273972 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fe49fadd60 con 0x55fe4a0c5fc0 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa011ffb640 10 client.27779 handle_command_reply: tid=0 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 client.27779 shutdown 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 msgr2=0x55fe4a060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00c6a0 tx=0x7fa01c0cf7b0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 shutdown_connections 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fe4a0c5fc0 0x55fe4a0c83b0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe4a04b680 0x55fe4a05e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe4a042b40 0x55fe4a05dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 --2- 172.21.15.16:0/3755273972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fe4a0404a0 0x55fe4a060780 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa011ffb640 0 client.27779 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 >> 172.21.15.16:0/3755273972 conn(0x55fe4a03ca30 msgr2=0x55fe4a049a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 shutdown_connections 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 1 -- 172.21.15.16:0/3755273972 wait complete. 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.446+0000 7fa032ccd740 20 client.27779 trim_cache size 0 max 16384 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:00:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:00:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:00:24.456 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:00:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 msgr2=0x7fa0140a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 0x7fa0140a5040 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa024009fd0 tx=0x7fa024009500 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 msgr2=0x7fa02c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00e9f0 tx=0x7fa01c00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 shutdown_connections 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0140a2b80 0x7fa0140a5040 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02c106f10 0x7fa02c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa02c102790 0x7fa02c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 --2- 172.21.15.16:0/3232349334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa02c101de0 0x7fa02c198490 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 >> 172.21.15.16:0/3232349334 conn(0x7fa02c0fd8c0 msgr2=0x7fa02c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 shutdown_connections 2024-09-12T03:00:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:24.447+0000 7fa0308f4640 1 -- 172.21.15.16:0/3232349334 wait complete. 2024-09-12T03:00:24.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1241950 9.78 MB/sec execute 566 sec latency 37.654 ms 2024-09-12T03:00:25.192 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:00:25.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:00:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:25.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1244191 9.78 MB/sec execute 567 sec latency 16.759 ms 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 -- 172.21.15.16:0/2268343353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 msgr2=0x7f384410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 --2- 172.21.15.16:0/2268343353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f384410ea90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f383c009ca0 tx=0x7f383c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 -- 172.21.15.16:0/2268343353 shutdown_connections 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 --2- 172.21.15.16:0/2268343353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f384410ea90 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 --2- 172.21.15.16:0/2268343353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844102790 0x7f38441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 --2- 172.21.15.16:0/2268343353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844101de0 0x7f38441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 -- 172.21.15.16:0/2268343353 >> 172.21.15.16:0/2268343353 conn(0x7f38440fd8e0 msgr2=0x7f38440ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 -- 172.21.15.16:0/2268343353 shutdown_connections 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.942+0000 7f384bdd1640 1 -- 172.21.15.16:0/2268343353 wait complete. 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 Processor -- start 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 -- start start 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 0x7f38440744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 0x7f3844074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3844111f70 con 0x7f3844106f10 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3844111df0 con 0x7f3844102790 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384bdd1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38441120f0 con 0x7f3844101de0 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384a36e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384a36e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47160/0 (socket says 172.21.15.16:47160) 2024-09-12T03:00:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384a36e640 1 -- 172.21.15.16:0/3299552302 learned_addr learned my addr 172.21.15.16:0/3299552302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f384936c640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 0x7f3844074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.943+0000 7f3849b6d640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 0x7f38440744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 -- 172.21.15.16:0/3299552302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 msgr2=0x7f38440744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 0x7f38440744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 -- 172.21.15.16:0/3299552302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 msgr2=0x7f3844074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 0x7f3844074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 -- 172.21.15.16:0/3299552302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38441a6870 con 0x7f3844106f10 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384936c640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 0x7f3844074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f3849b6d640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 0x7f38440744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384a36e640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f383c031a20 tx=0x7f383c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c003b70 con 0x7f3844106f10 2024-09-12T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f383c003d10 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38441a6b00 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c02fc40 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.944+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38441a7090 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.945+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38441021c0 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.946+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f383c004050 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.947+0000 7f383affd640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 0x7f38300a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.947+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f383c0effd0 con 0x7f3844106f10 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.947+0000 7f3849b6d640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 0x7f38300a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.947+0000 7f3849b6d640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 0x7f38300a50c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3840009fc0 tx=0x7f3840005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:25.952+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f383c0bd190 con 0x7f3844106f10 2024-09-12T03:00:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:25 smithi016 ceph-mon[30925]: pgmap v712: 97 pgs: 97 active+clean; 271 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-09-12T03:00:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.190+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f384410a0f0 con 0x7f3844106f10 2024-09-12T03:00:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.190+0000 7f383affd640 1 -- 172.21.15.16:0/3299552302 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11969 (secure 0 0 0) 0x7f383c0bca90 con 0x7f3844106f10 2024-09-12T03:00:26.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:00:26.192 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"btime":"2024-09-12T02:51:27:817141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T02:51:27.567786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T03:00:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-09-12T03:00:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 msgr2=0x7f38300a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 0x7f38300a50c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3840009fc0 tx=0x7f3840005f90 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 msgr2=0x7f3844074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f383c031a20 tx=0x7f383c031a50 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 shutdown_connections 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38300a2c00 0x7f38300a50c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3844106f10 0x7f3844074f70 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3844102790 0x7f3844074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 --2- 172.21.15.16:0/3299552302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3844101de0 0x7f38440744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.192+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 >> 172.21.15.16:0/3299552302 conn(0x7f38440fd8e0 msgr2=0x7f38440feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.193+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 shutdown_connections 2024-09-12T03:00:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:26.193+0000 7f384bdd1640 1 -- 172.21.15.16:0/3299552302 wait complete. 2024-09-12T03:00:26.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:25 smithi033 ceph-mon[35092]: pgmap v712: 97 pgs: 97 active+clean; 271 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-09-12T03:00:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:25 smithi184 ceph-mon[35321]: pgmap v712: 97 pgs: 97 active+clean; 271 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-09-12T03:00:26.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1246834 9.79 MB/sec execute 568 sec latency 74.737 ms 2024-09-12T03:00:26.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:00:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:27 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3299552302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:27.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3299552302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:27.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:27 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3299552302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:00:27.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1249583 9.80 MB/sec execute 569 sec latency 18.406 ms 2024-09-12T03:00:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 -- 172.21.15.16:0/3514275254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c101de0 msgr2=0x7f134c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 --2- 172.21.15.16:0/3514275254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c101de0 0x7f134c1021c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f133c009900 tx=0x7f133c02a410 comp rx=0 tx=0).stop 2024-09-12T03:00:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 -- 172.21.15.16:0/3514275254 shutdown_connections 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 --2- 172.21.15.16:0/3514275254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c106f10 0x7f134c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 --2- 172.21.15.16:0/3514275254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c102790 0x7f134c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 --2- 172.21.15.16:0/3514275254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c101de0 0x7f134c1021c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 -- 172.21.15.16:0/3514275254 >> 172.21.15.16:0/3514275254 conn(0x7f134c0fd8e0 msgr2=0x7f134c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 -- 172.21.15.16:0/3514275254 shutdown_connections 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.766+0000 7f1351f92640 1 -- 172.21.15.16:0/3514275254 wait complete. 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 Processor -- start 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 -- start start 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 0x7f134c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 0x7f134c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f134c111f70 con 0x7f134c106f10 2024-09-12T03:00:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f134c111df0 con 0x7f134c102790 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f1351f92640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1120f0 con 0x7f134c101de0 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 0x7f134c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 0x7f134c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53880/0 (socket says 172.21.15.16:53880) 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 0x7f134c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47170/0 (socket says 172.21.15.16:47170) 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134affd640 1 -- 172.21.15.16:0/2620228169 learned_addr learned my addr 172.21.15.16:0/2620228169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.767+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 0x7f134c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59512/0 (socket says 172.21.15.16:59512) 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 -- 172.21.15.16:0/2620228169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 msgr2=0x7f134c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 0x7f134c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 -- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 msgr2=0x7f134c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 0x7f134c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 -- 172.21.15.16:0/2620228169 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c1a68f0 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f134affd640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9f0 tx=0x7f134000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f1348ff9640 1 -- 172.21.15.16:0/2620228169 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134000cd80 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c1a6be0 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f1348ff9640 1 -- 172.21.15.16:0/2620228169 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1340005500 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f1348ff9640 1 -- 172.21.15.16:0/2620228169 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340010690 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.768+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c1a7120 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.770+0000 7f1348ff9640 1 -- 172.21.15.16:0/2620228169 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f13400046c0 con 0x7f134c102790 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.770+0000 7f135436b740 1 Processor -- start 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 -- start start 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 0x5641b3ce5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3cf2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5641b3c4b2f0 con 0x5641b3cf0370 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5641b3bf2cf0 con 0x5641b3ce6b40 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f135436b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5641b3c472b0 con 0x5641b3ce55d0 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3cf2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59526/0 (socket says 172.21.15.16:59526) 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 -- 172.21.15.16:0/4191821982 learned_addr learned my addr 172.21.15.16:0/4191821982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134bfff640 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 0x5641b3ce5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 -- 172.21.15.16:0/4191821982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 msgr2=0x5641b3ce5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 0x5641b3ce5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 -- 172.21.15.16:0/4191821982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 msgr2=0x5641b3cf2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3cf2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 -- 172.21.15.16:0/4191821982 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b3bd73a0 con 0x5641b3ce55d0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134bfff640 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 0x5641b3ce5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f1348ff9640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 0x7f13340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.771+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f133c009900 tx=0x7f133c005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2533648e0d506481 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f134bfff640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 0x7f13340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f1348ff9640 1 -- 172.21.15.16:0/2620228169 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f13400cf4b0 con 0x7f134c102790 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f1331ffb640 1 -- 172.21.15.16:0/4191821982 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c040070 con 0x5641b3ce55d0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f1331ffb640 1 -- 172.21.15.16:0/4191821982 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c031870 con 0x5641b3ce55d0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f1331ffb640 1 -- 172.21.15.16:0/4191821982 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c035040 con 0x5641b3ce55d0 2024-09-12T03:00:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 -- 172.21.15.16:0/4191821982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 msgr2=0x5641b3ce5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f133c009900 tx=0x7f133c005be0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f134bfff640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 0x7f13340a5040 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 -- 172.21.15.16:0/4191821982 shutdown_connections 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3cf2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce55d0 0x5641b3ce5a10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.772+0000 7f135436b740 1 --2- 172.21.15.16:0/4191821982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce6b40 0x5641b3ce5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- 172.21.15.16:0/4191821982 >> 172.21.15.16:0/4191821982 conn(0x5641b3cae310 msgr2=0x5641b3ce1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- 172.21.15.16:0/4191821982 shutdown_connections 2024-09-12T03:00:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- 172.21.15.16:0/4191821982 wait complete. 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- start start 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 Processor -- start 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3cecf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3ced4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5641b3cf3010 con 0x5641b3cf0370 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5641b3c06060 con 0x5641b3ce55d0 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.773+0000 7f135436b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5641b3cf3190 con 0x5641b3ce6b40 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f135436b740 10 client.0 _ll_register_callbacks cb 0x5641b39598c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3ced4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53890/0 (socket says 172.21.15.16:53890) 2024-09-12T03:00:27.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3cecf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 -- 172.21.15.16:0/2156278710 learned_addr learned my addr 172.21.15.16:0/2156278710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 -- 172.21.15.16:0/2156278710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 msgr2=0x5641b3cecf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3cecf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 -- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 msgr2=0x5641b3ced4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3ced4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 -- 172.21.15.16:0/2156278710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b39bc3c0 con 0x5641b3ce55d0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f134b7fe640 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a930 tx=0x7f133c02ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db4791af02401b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f132affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c040070 con 0x5641b3ce55d0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f133c004240 con 0x5641b3ce55d0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.774+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c031870 con 0x5641b3ce55d0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f133c035040 con 0x5641b3ce55d0 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 10 client.27797 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:27.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641b3cfea80 con 0x5641b3ce55d0 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c031870 con 0x5641b3ce55d0 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f133c0035f0 con 0x5641b3ce55d0 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 10 client.27797 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 10 client.27797 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 20 client.27797 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3cffdd0 0x5641b3d021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 4 client.27797 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641b3c58860 con 0x5641b3cffdd0 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f134affd640 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3cffdd0 0x5641b3d021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.775+0000 7f132affd640 10 client.27797 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f134affd640 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3cffdd0 0x5641b3d021c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dd31b0965dbd8262 server_cookie=e47353ec8d4c0455 in_seq=0 out_seq=0 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f132affd640 1 -- 172.21.15.16:0/2156278710 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5641b3c58860 con 0x5641b3cffdd0 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f132affd640 10 client.27797 handle_command_reply: tid=0 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f135436b740 1 client.27797 shutdown 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 msgr2=0x5641b3cefce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.776+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a930 tx=0x7f133c02ab10 comp rx=0 tx=0).stop 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 shutdown_connections 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3cf0370 0x5641b3ced4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3cecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3cffdd0 0x5641b3d021c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 --2- 172.21.15.16:0/2156278710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3ce55d0 0x5641b3cefce0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f132affd640 0 client.27797 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:00:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 >> 172.21.15.16:0/2156278710 conn(0x5641b3cae310 msgr2=0x5641b3ce7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 shutdown_connections 2024-09-12T03:00:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 1 -- 172.21.15.16:0/2156278710 wait complete. 2024-09-12T03:00:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.777+0000 7f135436b740 20 client.27797 trim_cache size 0 max 16384 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 Processor -- start 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 -- start start 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3ce8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3ce8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1af670 con 0x5641b3ce55d0 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1b11b0 con 0x5641b3cf0370 2024-09-12T03:00:27.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f135436b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1afd40 con 0x5641b3ce6b40 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f134affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3ce8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47186/0 (socket says 172.21.15.16:47186) 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f134b7fe640 1 -- 172.21.15.16:0/1885938791 learned_addr learned my addr 172.21.15.16:0/1885938791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.791+0000 7f134bfff640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3ce8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 -- 172.21.15.16:0/1885938791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 msgr2=0x5641b3ce8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3ce8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 -- 172.21.15.16:0/1885938791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 msgr2=0x5641b3ce8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3ce8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 -- 172.21.15.16:0/1885938791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b3ce8ba0 con 0x5641b3ce55d0 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134bfff640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3ce8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134affd640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3ce8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f133c0039c0 tx=0x7f133c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3882a0733145dd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f13317fa640 1 -- 172.21.15.16:0/1885938791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c040040 con 0x5641b3ce55d0 2024-09-12T03:00:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f13317fa640 1 -- 172.21.15.16:0/1885938791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c04d070 con 0x5641b3ce55d0 2024-09-12T03:00:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.792+0000 7f13317fa640 1 -- 172.21.15.16:0/1885938791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0044a0 con 0x5641b3ce55d0 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- 172.21.15.16:0/1885938791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 msgr2=0x5641b3ce6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f133c0039c0 tx=0x7f133c005720 comp rx=0 tx=0).stop 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- 172.21.15.16:0/1885938791 shutdown_connections 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3ce8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3ce8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 --2- 172.21.15.16:0/1885938791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3ce6410 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- 172.21.15.16:0/1885938791 >> 172.21.15.16:0/1885938791 conn(0x5641b3cae310 msgr2=0x5641b3cec080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- 172.21.15.16:0/1885938791 shutdown_connections 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- 172.21.15.16:0/1885938791 wait complete. 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 -- start start 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.793+0000 7f135436b740 1 Processor -- start 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3d05790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3d02780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1b0ae0 con 0x5641b3ce55d0 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1b0410 con 0x5641b3cf0370 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1ad650 con 0x5641b3ce6b40 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f135436b740 10 client.0 _ll_register_callbacks cb 0x7f13400d2dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47200/0 (socket says 172.21.15.16:47200) 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 -- 172.21.15.16:0/4037582217 learned_addr learned my addr 172.21.15.16:0/4037582217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134bfff640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3d02780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134affd640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3d05790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 -- 172.21.15.16:0/4037582217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 msgr2=0x5641b3d05790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3d05790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 -- 172.21.15.16:0/4037582217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 msgr2=0x5641b3d02780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3d02780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 -- 172.21.15.16:0/4037582217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b3d02e70 con 0x5641b3ce55d0 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134bfff640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3d02780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f134b7fe640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a930 tx=0x7f133c02aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e58568379d0ba7a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.794+0000 7f132affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c048040 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f133c004440 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c04d070 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f133c06d050 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 10 client.18396 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641b3d6a1b0 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c04d070 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f133c004ae0 con 0x5641b3ce55d0 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 10 client.18396 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 10 client.18396 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 20 client.18396 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3d6a870 0x5641b3d6cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 4 client.18396 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:00:27.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641b3c52d70 con 0x5641b3d6a870 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.795+0000 7f134affd640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3d6a870 0x5641b3d6cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f134affd640 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3d6a870 0x5641b3d6cc90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c0e422b67fbc181 server_cookie=90c51074b064bf37 in_seq=0 out_seq=0 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f132affd640 10 client.18396 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f132affd640 1 -- 172.21.15.16:0/4037582217 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641b3c52d70 con 0x5641b3d6a870 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f132affd640 10 client.18396 handle_command_reply: tid=0 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 client.18396 shutdown 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 msgr2=0x5641b3d05250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f133c02a930 tx=0x7f133c02aba0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 shutdown_connections 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5641b3d6a870 0x5641b3d6cc90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5641b3cf0370 0x5641b3d02780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641b3ce6b40 0x5641b3d05790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 --2- 172.21.15.16:0/4037582217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641b3ce55d0 0x5641b3d05250 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f132affd640 0 client.18396 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 >> 172.21.15.16:0/4037582217 conn(0x5641b3cae310 msgr2=0x5641b3ce04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 shutdown_connections 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 1 -- 172.21.15.16:0/4037582217 wait complete. 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.796+0000 7f135436b740 20 client.18396 trim_cache size 0 max 16384 2024-09-12T03:00:27.803 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 msgr2=0x7f13340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 0x7f13340a5040 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344009500 comp rx=0 tx=0).stop 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 msgr2=0x7f134c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9f0 tx=0x7f134000edf0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 shutdown_connections 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f13340a2b80 0x7f13340a5040 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c106f10 0x7f134c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f134c102790 0x7f134c074a30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 --2- 172.21.15.16:0/2620228169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f134c101de0 0x7f134c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 >> 172.21.15.16:0/2620228169 conn(0x7f134c0fd8e0 msgr2=0x7f134c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 shutdown_connections 2024-09-12T03:00:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:27.797+0000 7f1351f92640 1 -- 172.21.15.16:0/2620228169 wait complete. 2024-09-12T03:00:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:28 smithi184 ceph-mon[35321]: pgmap v713: 97 pgs: 97 active+clean; 258 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-09-12T03:00:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:28 smithi016 ceph-mon[30925]: pgmap v713: 97 pgs: 97 active+clean; 258 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-09-12T03:00:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:28 smithi033 ceph-mon[35092]: pgmap v713: 97 pgs: 97 active+clean; 258 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-09-12T03:00:28.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:00:28.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1251880 9.80 MB/sec execute 570 sec latency 11.410 ms 2024-09-12T03:00:28.899 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 -- 172.21.15.16:0/3108202902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc102790 msgr2=0x7f48cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 --2- 172.21.15.16:0/3108202902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc102790 0x7f48cc1067e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009900 tx=0x7f48bc02a410 comp rx=0 tx=0).stop 2024-09-12T03:00:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 -- 172.21.15.16:0/3108202902 shutdown_connections 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 --2- 172.21.15.16:0/3108202902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f10 0x7f48cc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 --2- 172.21.15.16:0/3108202902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc102790 0x7f48cc1067e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 --2- 172.21.15.16:0/3108202902 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 0x7f48cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 -- 172.21.15.16:0/3108202902 >> 172.21.15.16:0/3108202902 conn(0x7f48cc0fd8c0 msgr2=0x7f48cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 -- 172.21.15.16:0/3108202902 shutdown_connections 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.415+0000 7f48d33ab640 1 -- 172.21.15.16:0/3108202902 wait complete. 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 Processor -- start 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 -- start start 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 0x7f48cc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 0x7f48cc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc111f70 con 0x7f48cc106f10 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc111df0 con 0x7f48cc101de0 2024-09-12T03:00:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d33ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1120f0 con 0x7f48cc102790 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d1948640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d0946640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 0x7f48cc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d1948640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47230/0 (socket says 172.21.15.16:47230) 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d1948640 1 -- 172.21.15.16:0/2032445786 learned_addr learned my addr 172.21.15.16:0/2032445786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.416+0000 7f48d1147640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 0x7f48cc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 -- 172.21.15.16:0/2032445786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 msgr2=0x7f48cc198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 0x7f48cc198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 -- 172.21.15.16:0/2032445786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 msgr2=0x7f48cc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 0x7f48cc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 -- 172.21.15.16:0/2032445786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc19d400 con 0x7f48cc106f10 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d0946640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 0x7f48cc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d1948640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eb10 tx=0x7f48c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48c27fc640 1 -- 172.21.15.16:0/2032445786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4004b20 con 0x7f48cc106f10 2024-09-12T03:00:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48c27fc640 1 -- 172.21.15.16:0/2032445786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c4004cc0 con 0x7f48cc106f10 2024-09-12T03:00:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48c27fc640 1 -- 172.21.15.16:0/2032445786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4010760 con 0x7f48cc106f10 2024-09-12T03:00:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc19d6f0 con 0x7f48cc106f10 2024-09-12T03:00:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.417+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc074420 con 0x7f48cc106f10 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.419+0000 7f48c27fc640 1 -- 172.21.15.16:0/2032445786 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f48c4005020 con 0x7f48cc106f10 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.420+0000 7f48d5784740 1 Processor -- start 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.420+0000 7f48d5784740 1 -- start start 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48c27fc640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 0x7f48b80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965936a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x561965943760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56196589c2f0 con 0x561965937b40 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561965843cf0 con 0x561965941370 2024-09-12T03:00:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d5784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5619658982b0 con 0x5619659365d0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47240/0 (socket says 172.21.15.16:47240) 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48c27fc640 1 -- 172.21.15.16:0/2032445786 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48c40cf640 con 0x7f48cc106f10 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d0946640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 0x7f48b80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 -- 172.21.15.16:0/3193734314 learned_addr learned my addr 172.21.15.16:0/3193734314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d0946640 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965936a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1948640 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x561965943760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 -- 172.21.15.16:0/3193734314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 msgr2=0x561965936a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965936a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 -- 172.21.15.16:0/3193734314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 msgr2=0x561965943760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x561965943760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.421+0000 7f48d1147640 1 -- 172.21.15.16:0/3193734314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619658283a0 con 0x561965937b40 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48d0946640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 0x7f48b80a50c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009230 tx=0x7f48bc005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48d1147640 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009cf0 tx=0x7f48c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fef06256b3f5686 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48b77fe640 1 -- 172.21.15.16:0/3193734314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8019420 con 0x561965937b40 2024-09-12T03:00:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48b77fe640 1 -- 172.21.15.16:0/3193734314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c8019aa0 con 0x561965937b40 2024-09-12T03:00:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48b77fe640 1 -- 172.21.15.16:0/3193734314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8018b80 con 0x561965937b40 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48d5784740 1 -- 172.21.15.16:0/3193734314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 msgr2=0x561965936090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.422+0000 7f48d5784740 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009cf0 tx=0x7f48c8011810 comp rx=0 tx=0).stop 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 -- 172.21.15.16:0/3193734314 shutdown_connections 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x561965943760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965936a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 --2- 172.21.15.16:0/3193734314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965937b40 0x561965936090 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 -- 172.21.15.16:0/3193734314 >> 172.21.15.16:0/3193734314 conn(0x5619658ff310 msgr2=0x561965932c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 -- 172.21.15.16:0/3193734314 shutdown_connections 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 -- 172.21.15.16:0/3193734314 wait complete. 2024-09-12T03:00:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 -- start start 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 Processor -- start 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.423+0000 7f48d5784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x56196593e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965941370 0x56196593e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561965944070 con 0x5619659365d0 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561965857060 con 0x561965937b40 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5619659441f0 con 0x561965941370 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 10 client.0 _ll_register_callbacks cb 0x5619655aa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47254/0 (socket says 172.21.15.16:47254) 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d0946640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x56196593e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 -- 172.21.15.16:0/2627634 learned_addr learned my addr 172.21.15.16:0/2627634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1948640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965941370 0x56196593e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 -- 172.21.15.16:0/2627634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965941370 msgr2=0x56196593e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965941370 0x56196593e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 -- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 msgr2=0x56196593e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x56196593e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 -- 172.21.15.16:0/2627634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56196560d3c0 con 0x5619659365d0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d0946640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x56196593e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d1147640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000c00 tx=0x7f48c80093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e75b895cb7de752 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8004240 con 0x5619659365d0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48c80043e0 con 0x5619659365d0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.424+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48c8019aa0 con 0x5619659365d0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48c8019420 con 0x5619659365d0 2024-09-12T03:00:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 10 client.18408 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56196594fb10 con 0x5619659365d0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c8019aa0 con 0x5619659365d0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f48c8018a90 con 0x5619659365d0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 10 client.18408 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 10 client.18408 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 20 client.18408 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659512e0 0x561965953730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 4 client.18408 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619658a9860 con 0x5619659512e0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.425+0000 7f48d0946640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659512e0 0x561965953730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.426+0000 7f48b4ff9640 10 client.18408 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.426+0000 7f48d0946640 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659512e0 0x561965953730 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8bf461108251db5f server_cookie=e38bcd2fd06c59c5 in_seq=0 out_seq=0 2024-09-12T03:00:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.426+0000 7f48b4ff9640 1 -- 172.21.15.16:0/2627634 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5619658a9860 con 0x5619659512e0 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.426+0000 7f48b4ff9640 10 client.18408 handle_command_reply: tid=0 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 client.18408 shutdown 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 msgr2=0x561965940d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000c00 tx=0x7f48c80093a0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 shutdown_connections 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659512e0 0x561965953730 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965941370 0x56196593e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x56196593e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 --2- 172.21.15.16:0/2627634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x561965940d70 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 >> 172.21.15.16:0/2627634 conn(0x5619658ff310 msgr2=0x561965938960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48b4ff9640 0 client.18408 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 shutdown_connections 2024-09-12T03:00:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 1 -- 172.21.15.16:0/2627634 wait complete. 2024-09-12T03:00:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.427+0000 7f48d5784740 20 client.18408 trim_cache size 0 max 16384 2024-09-12T03:00:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.442+0000 7f48d5784740 1 Processor -- start 2024-09-12T03:00:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.442+0000 7f48d5784740 1 -- start start 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965937b40 0x561965932700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x56196593c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1af7e0 con 0x5619659365d0 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1b1320 con 0x561965941370 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d5784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1afeb0 con 0x561965937b40 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47270/0 (socket says 172.21.15.16:47270) 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1147640 1 -- 172.21.15.16:0/3129214430 learned_addr learned my addr 172.21.15.16:0/3129214430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x56196593c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d0946640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965937b40 0x561965932700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 -- 172.21.15.16:0/3129214430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965937b40 msgr2=0x561965932700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965937b40 0x561965932700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 -- 172.21.15.16:0/3129214430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 msgr2=0x56196594d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 -- 172.21.15.16:0/3129214430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561965933960 con 0x561965941370 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1147640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48d1948640 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x56196593c8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f48c4005bf0 tx=0x7f48c4004f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff593f7dfb4dac7b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48b6ffd640 1 -- 172.21.15.16:0/3129214430 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c409c550 con 0x561965941370 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48b6ffd640 1 -- 172.21.15.16:0/3129214430 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c409e030 con 0x561965941370 2024-09-12T03:00:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.443+0000 7f48b6ffd640 1 -- 172.21.15.16:0/3129214430 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c40984c0 con 0x561965941370 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 -- 172.21.15.16:0/3129214430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 msgr2=0x56196593c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x56196593c8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f48c4005bf0 tx=0x7f48c4004f50 comp rx=0 tx=0).stop 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 -- 172.21.15.16:0/3129214430 shutdown_connections 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965941370 0x56196593c8f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561965937b40 0x561965932700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 --2- 172.21.15.16:0/3129214430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5619659365d0 0x56196594d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 -- 172.21.15.16:0/3129214430 >> 172.21.15.16:0/3129214430 conn(0x5619658ff310 msgr2=0x5619659392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 -- 172.21.15.16:0/3129214430 shutdown_connections 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.444+0000 7f48d5784740 1 -- 172.21.15.16:0/3129214430 wait complete. 2024-09-12T03:00:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 -- start start 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 Processor -- start 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x561965944440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965941370 0x561965944980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1b0c50 con 0x561965941370 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1b0580 con 0x561965937b40 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1ad7c0 con 0x5619659365d0 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.445+0000 7f48d5784740 10 client.0 _ll_register_callbacks cb 0x7f48c40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1948640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965941370 0x561965944980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59608/0 (socket says 172.21.15.16:59608) 2024-09-12T03:00:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 -- 172.21.15.16:0/3238001825 learned_addr learned my addr 172.21.15.16:0/3238001825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d0946640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x561965944440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 -- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 msgr2=0x561965944440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x561965944440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 -- 172.21.15.16:0/3238001825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965941370 msgr2=0x561965944980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965941370 0x561965944980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 -- 172.21.15.16:0/3238001825 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561965945180 con 0x5619659365d0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d1147640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48c8018ee0 tx=0x7f48c8005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9441248a3089d327 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8004450 con 0x5619659365d0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48c80195c0 con 0x5619659365d0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.446+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48c409e030 con 0x5619659365d0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48c8045050 con 0x5619659365d0 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 10 client.27721 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5619659baf80 con 0x5619659365d0 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c409e030 con 0x5619659365d0 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f48c80035f0 con 0x5619659365d0 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 10 client.27721 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 10 client.27721 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 20 client.27721 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659bb670 0x5619659bda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 4 client.27721 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5619658a3d70 con 0x5619659bb670 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.447+0000 7f48d0946640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659bb670 0x5619659bda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d0946640 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659bb670 0x5619659bda90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bbd62b0de3f369e9 server_cookie=8c32ecf695fcfe1b in_seq=0 out_seq=0 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48b4ff9640 10 client.27721 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48b4ff9640 1 -- 172.21.15.16:0/3238001825 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5619658a3d70 con 0x5619659bb670 2024-09-12T03:00:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48b4ff9640 10 client.27721 handle_command_reply: tid=0 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 client.27721 shutdown 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 msgr2=0x561965949ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48c8018ee0 tx=0x7f48c8005b90 comp rx=0 tx=0).stop 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 shutdown_connections 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5619659bb670 0x5619659bda90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561965941370 0x561965944980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561965937b40 0x561965944440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 --2- 172.21.15.16:0/3238001825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5619659365d0 0x561965949ba0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 >> 172.21.15.16:0/3238001825 conn(0x5619658ff310 msgr2=0x56196593c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48b4ff9640 0 client.27721 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 shutdown_connections 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.448+0000 7f48d5784740 1 -- 172.21.15.16:0/3238001825 wait complete. 2024-09-12T03:00:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d5784740 20 client.27721 trim_cache size 0 max 16384 2024-09-12T03:00:29.455 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:00:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 msgr2=0x7f48b80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 0x7f48b80a50c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009230 tx=0x7f48bc005d50 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 msgr2=0x7f48cc19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eb10 tx=0x7f48c400c2d0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 shutdown_connections 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2c00 0x7f48b80a50c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc106f10 0x7f48cc19cd20 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc102790 0x7f48cc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 --2- 172.21.15.16:0/2032445786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc101de0 0x7f48cc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 >> 172.21.15.16:0/2032445786 conn(0x7f48cc0fd8c0 msgr2=0x7f48cc0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 shutdown_connections 2024-09-12T03:00:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:29.449+0000 7f48d33ab640 1 -- 172.21.15.16:0/2032445786 wait complete. 2024-09-12T03:00:29.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1254386 9.81 MB/sec execute 571 sec latency 29.719 ms 2024-09-12T03:00:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:30 smithi016 ceph-mon[30925]: pgmap v714: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 220 op/s 2024-09-12T03:00:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:30 smithi184 ceph-mon[35321]: pgmap v714: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 220 op/s 2024-09-12T03:00:30.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:00:30.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:30 smithi033 ceph-mon[35092]: pgmap v714: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 220 op/s 2024-09-12T03:00:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:30.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1257347 9.82 MB/sec execute 572 sec latency 17.661 ms 2024-09-12T03:00:31.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 -- 172.21.15.16:0/3945035162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 msgr2=0x7f21a010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 --2- 172.21.15.16:0/3945035162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a010ea90 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f2198009ca0 tx=0x7f219802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 -- 172.21.15.16:0/3945035162 shutdown_connections 2024-09-12T03:00:31.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 --2- 172.21.15.16:0/3945035162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a010ea90 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 --2- 172.21.15.16:0/3945035162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 --2- 172.21.15.16:0/3945035162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 -- 172.21.15.16:0/3945035162 >> 172.21.15.16:0/3945035162 conn(0x7f21a00fd880 msgr2=0x7f21a00ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.283+0000 7f21a554e640 1 -- 172.21.15.16:0/3945035162 shutdown_connections 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 -- 172.21.15.16:0/3945035162 wait complete. 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 Processor -- start 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 -- start start 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21a0111fc0 con 0x7f21a0106f10 2024-09-12T03:00:31.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21a0111e40 con 0x7f21a0101de0 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.284+0000 7f21a554e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21a0112140 con 0x7f21a0102790 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a019ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47302/0 (socket says 172.21.15.16:47302) 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219f7fe640 1 -- 172.21.15.16:0/3358793177 learned_addr learned my addr 172.21.15.16:0/3358793177 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59614/0 (socket says 172.21.15.16:59614) 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219effd640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 -- 172.21.15.16:0/3358793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 msgr2=0x7f21a019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 -- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 msgr2=0x7f21a019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 -- 172.21.15.16:0/3358793177 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a019f570 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219effd640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f219400e9f0 tx=0x7f219400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f2187fff640 1 -- 172.21.15.16:0/3358793177 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219400ccc0 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f2187fff640 1 -- 172.21.15.16:0/3358793177 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2194005020 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a019f860 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.285+0000 7f2187fff640 1 -- 172.21.15.16:0/3358793177 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21940049e0 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.286+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a01a7140 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.288+0000 7f2187fff640 1 -- 172.21.15.16:0/3358793177 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2194005560 con 0x7f21a0102790 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.288+0000 7f21a7927740 1 Processor -- start 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.288+0000 7f21a7927740 1 -- start start 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c24b40 0x55b216c23090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 0x55b216c30760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b216b892f0 con 0x55b216c2e370 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b216b30cf0 con 0x55b216c235d0 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f21a7927740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b216b852b0 con 0x55b216c24b40 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f2187fff640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 0x7f21880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 0x55b216c30760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 0x55b216c30760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47308/0 (socket says 172.21.15.16:47308) 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53980/0 (socket says 172.21.15.16:53980) 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219e7fc640 1 -- 172.21.15.16:0/3292421947 learned_addr learned my addr 172.21.15.16:0/3292421947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f2187fff640 1 -- 172.21.15.16:0/3358793177 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f21940cefd0 con 0x7f21a0102790 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219f7fe640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 0x7f21880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 -- 172.21.15.16:0/3292421947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c24b40 msgr2=0x55b216c23090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c24b40 0x55b216c23090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 -- 172.21.15.16:0/3292421947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 msgr2=0x55b216c30760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 0x55b216c30760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.289+0000 7f219effd640 1 -- 172.21.15.16:0/3292421947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b216b153a0 con 0x55b216c235d0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.290+0000 7f219effd640 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f219000bc80 tx=0x7f219000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7217f5ffbdd0d947 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.290+0000 7f21857fa640 1 -- 172.21.15.16:0/3292421947 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190015070 con 0x55b216c235d0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.290+0000 7f21857fa640 1 -- 172.21.15.16:0/3292421947 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219001a940 con 0x55b216c235d0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.290+0000 7f21857fa640 1 -- 172.21.15.16:0/3292421947 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900119a0 con 0x55b216c235d0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- 172.21.15.16:0/3292421947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 msgr2=0x55b216c23a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f219f7fe640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 0x7f21880a5040 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2198009a60 tx=0x7f219803f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f219000bc80 tx=0x7f219000f730 comp rx=0 tx=0).stop 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- 172.21.15.16:0/3292421947 shutdown_connections 2024-09-12T03:00:31.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c2e370 0x55b216c30760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c23a10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- 172.21.15.16:0/3292421947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c24b40 0x55b216c23090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- 172.21.15.16:0/3292421947 >> 172.21.15.16:0/3292421947 conn(0x55b216bec310 msgr2=0x55b216c1fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- 172.21.15.16:0/3292421947 shutdown_connections 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- 172.21.15.16:0/3292421947 wait complete. 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- start start 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 Processor -- start 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c2dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c2b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b216c31070 con 0x55b216c24b40 2024-09-12T03:00:31.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b216b44060 con 0x55b216c2e370 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b216c311f0 con 0x55b216c235d0 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f21a7927740 10 client.0 _ll_register_callbacks cb 0x55b2168978c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.291+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47322/0 (socket says 172.21.15.16:47322) 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c2b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 -- 172.21.15.16:0/1758095360 learned_addr learned my addr 172.21.15.16:0/1758095360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219effd640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c2dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 -- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 msgr2=0x55b216c2dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c2dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 -- 172.21.15.16:0/1758095360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 msgr2=0x55b216c2b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c2b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 -- 172.21.15.16:0/1758095360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2168fa3c0 con 0x55b216c24b40 2024-09-12T03:00:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219effd640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c2dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219f7fe640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c2b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f219e7fc640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f21940a3f80 tx=0x7f21940054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43847385cb6800f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f2182ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219401c040 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f21940d1110 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f219001a940 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f21940051d0 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f21a7927740 10 client.18426 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.292+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b216c3cb20 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219001a940 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2194004be0 con 0x55b216c24b40 2024-09-12T03:00:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 10 client.18426 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 10 client.18426 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 20 client.18426 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216c3e350 0x55b216c40740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 4 client.18426 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b216b96860 con 0x55b216c3e350 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.293+0000 7f219effd640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216c3e350 0x55b216c40740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f219effd640 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216c3e350 0x55b216c40740 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4a4234a15850fc server_cookie=2b4254bef28aebe2 in_seq=0 out_seq=0 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f2182ff5640 10 client.18426 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f2182ff5640 1 -- 172.21.15.16:0/1758095360 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b216b96860 con 0x55b216c3e350 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f2182ff5640 10 client.18426 handle_command_reply: tid=0 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 client.18426 shutdown 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 msgr2=0x55b216c2afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f21940a3f80 tx=0x7f21940054d0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 shutdown_connections 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c2b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2afd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216c3e350 0x55b216c40740 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 --2- 172.21.15.16:0/1758095360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c2dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 >> 172.21.15.16:0/1758095360 conn(0x55b216bec310 msgr2=0x55b216c25960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.294+0000 7f2182ff5640 0 client.18426 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.295+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 shutdown_connections 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.295+0000 7f21a7927740 1 -- 172.21.15.16:0/1758095360 wait complete. 2024-09-12T03:00:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.295+0000 7f21a7927740 20 client.18426 trim_cache size 0 max 16384 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 Processor -- start 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 -- start start 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c2e370 0x55b216c26120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01af720 con 0x55b216c24b40 2024-09-12T03:00:31.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01b1260 con 0x55b216c235d0 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f21a7927740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01afdf0 con 0x55b216c2e370 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c2e370 0x55b216c26120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47336/0 (socket says 172.21.15.16:47336) 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53994/0 (socket says 172.21.15.16:53994) 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219e7fc640 1 -- 172.21.15.16:0/3073440565 learned_addr learned my addr 172.21.15.16:0/3073440565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 -- 172.21.15.16:0/3073440565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c2e370 msgr2=0x55b216c26120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c2e370 0x55b216c26120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 -- 172.21.15.16:0/3073440565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 msgr2=0x55b216c2cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219effd640 1 -- 172.21.15.16:0/3073440565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b216c232d0 con 0x55b216c235d0 2024-09-12T03:00:31.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.314+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:31.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.315+0000 7f219effd640 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2190000c00 tx=0x7f2190011620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82c3ddc7071d4233 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.315+0000 7f2184ff9640 1 -- 172.21.15.16:0/3073440565 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190015040 con 0x55b216c235d0 2024-09-12T03:00:31.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.315+0000 7f2184ff9640 1 -- 172.21.15.16:0/3073440565 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2190005520 con 0x55b216c235d0 2024-09-12T03:00:31.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.315+0000 7f2184ff9640 1 -- 172.21.15.16:0/3073440565 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190011ab0 con 0x55b216c235d0 2024-09-12T03:00:31.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- 172.21.15.16:0/3073440565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 msgr2=0x55b216c2a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2190000c00 tx=0x7f2190011620 comp rx=0 tx=0).stop 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- 172.21.15.16:0/3073440565 shutdown_connections 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c2e370 0x55b216c26120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c2cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 --2- 172.21.15.16:0/3073440565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c235d0 0x55b216c2a890 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- 172.21.15.16:0/3073440565 >> 172.21.15.16:0/3073440565 conn(0x55b216bec310 msgr2=0x55b216c2c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- 172.21.15.16:0/3073440565 shutdown_connections 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- 172.21.15.16:0/3073440565 wait complete. 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 -- start start 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.316+0000 7f21a7927740 1 Processor -- start 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c36480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01b0b90 con 0x55b216c24b40 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01b04c0 con 0x55b216c2e370 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01ad700 con 0x55b216c235d0 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f21a7927740 10 client.0 _ll_register_callbacks cb 0x7f21940046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47340/0 (socket says 172.21.15.16:47340) 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219e7fc640 1 -- 172.21.15.16:0/3084164148 learned_addr learned my addr 172.21.15.16:0/3084164148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c36480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 -- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 msgr2=0x55b216c36480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c36480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 -- 172.21.15.16:0/3084164148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 msgr2=0x55b216c369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 -- 172.21.15.16:0/3084164148 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b216c27b70 con 0x55b216c2e370 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219e7fc640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219effd640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c36480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f219f7fe640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c27600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f219802f540 tx=0x7f219802a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4affb6c808748157 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.317+0000 7f2182ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198004310 con 0x55b216c2e370 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2190005520 con 0x55b216c2e370 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f21980044b0 con 0x55b216c2e370 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f21980046b0 con 0x55b216c2e370 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 10 client.27839 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:31.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b216ca84d0 con 0x55b216c2e370 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2190005520 con 0x55b216c2e370 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f219805c080 con 0x55b216c2e370 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 10 client.27839 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 10 client.27839 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 20 client.27839 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216ca8ca0 0x55b216cab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 4 client.27839 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b216b90d70 con 0x55b216ca8ca0 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.318+0000 7f219effd640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216ca8ca0 0x55b216cab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f2182ff5640 10 client.27839 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f219effd640 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216ca8ca0 0x55b216cab0c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e944906183b76218 server_cookie=b21aa98aa079a56a in_seq=0 out_seq=0 2024-09-12T03:00:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f2182ff5640 1 -- 172.21.15.16:0/3084164148 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b216b90d70 con 0x55b216ca8ca0 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f2182ff5640 10 client.27839 handle_command_reply: tid=0 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 client.27839 shutdown 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 msgr2=0x55b216c27600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c27600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f219802f540 tx=0x7f219802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 shutdown_connections 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b216c2e370 0x55b216c27600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b216c24b40 0x55b216c369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55b216ca8ca0 0x55b216cab0c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 --2- 172.21.15.16:0/3084164148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b216c235d0 0x55b216c36480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f2182ff5640 0 client.27839 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.319+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 >> 172.21.15.16:0/3084164148 conn(0x55b216bec310 msgr2=0x55b216c2d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 shutdown_connections 2024-09-12T03:00:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a7927740 1 -- 172.21.15.16:0/3084164148 wait complete. 2024-09-12T03:00:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a7927740 20 client.27839 trim_cache size 0 max 16384 2024-09-12T03:00:31.327 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:00:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 msgr2=0x7f21880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 0x7f21880a5040 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2198009a60 tx=0x7f219803f040 comp rx=0 tx=0).stop 2024-09-12T03:00:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 msgr2=0x7f21a019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f219400e9f0 tx=0x7f219400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 shutdown_connections 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21880a2b80 0x7f21880a5040 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a0106f10 0x7f21a019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a0102790 0x7f21a019ab00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 --2- 172.21.15.16:0/3358793177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21a0101de0 0x7f21a019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 >> 172.21.15.16:0/3358793177 conn(0x7f21a00fd880 msgr2=0x7f21a00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.320+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 shutdown_connections 2024-09-12T03:00:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:31.321+0000 7f21a554e640 1 -- 172.21.15.16:0/3358793177 wait complete. 2024-09-12T03:00:31.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1259967 9.83 MB/sec execute 573 sec latency 13.541 ms 2024-09-12T03:00:32.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:00:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:32 smithi016 ceph-mon[30925]: pgmap v715: 97 pgs: 97 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-09-12T03:00:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:32 smithi184 ceph-mon[35321]: pgmap v715: 97 pgs: 97 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-09-12T03:00:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:32 smithi033 ceph-mon[35092]: pgmap v715: 97 pgs: 97 active+clean; 261 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-09-12T03:00:32.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1261976 9.83 MB/sec execute 574 sec latency 43.156 ms 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 -- 172.21.15.16:0/402106610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc101de0 msgr2=0x7f7bcc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/402106610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc101de0 0x7f7bcc1021c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009900 tx=0x7f7bc802a410 comp rx=0 tx=0).stop 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 -- 172.21.15.16:0/402106610 shutdown_connections 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/402106610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc106f10 0x7f7bcc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/402106610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc102790 0x7f7bcc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/402106610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc101de0 0x7f7bcc1021c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 -- 172.21.15.16:0/402106610 >> 172.21.15.16:0/402106610 conn(0x7f7bcc0fd8c0 msgr2=0x7f7bcc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.747+0000 7f7bd47dd640 1 -- 172.21.15.16:0/402106610 shutdown_connections 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.748+0000 7f7bd47dd640 1 -- 172.21.15.16:0/402106610 wait complete. 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.748+0000 7f7bd47dd640 1 Processor -- start 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.748+0000 7f7bd47dd640 1 -- start start 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.748+0000 7f7bd47dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 0x7f7bcc0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.748+0000 7f7bd47dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 0x7f7bcc074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd47dd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 0x7f7bcc074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd47dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc111fc0 con 0x7f7bcc102790 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd47dd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc111e40 con 0x7f7bcc106f10 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd47dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc112140 con 0x7f7bcc101de0 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2579640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 0x7f7bcc0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2579640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 0x7f7bcc0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59650/0 (socket says 172.21.15.16:59650) 2024-09-12T03:00:32.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2579640 1 -- 172.21.15.16:0/679818842 learned_addr learned my addr 172.21.15.16:0/679818842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:32.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 0x7f7bcc074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 0x7f7bcc074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/679818842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 msgr2=0x7f7bcc0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 0x7f7bcc0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 msgr2=0x7f7bcc074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 0x7f7bcc074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/679818842 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bcc1a68e0 con 0x7f7bcc106f10 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.749+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 0x7f7bcc074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 0x7f7bcc074f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009fd0 tx=0x7f7bc400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bc37fe640 1 -- 172.21.15.16:0/679818842 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400ce20 con 0x7f7bcc106f10 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bc37fe640 1 -- 172.21.15.16:0/679818842 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc4005020 con 0x7f7bcc106f10 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bcc1a6bd0 con 0x7f7bcc106f10 2024-09-12T03:00:32.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bc37fe640 1 -- 172.21.15.16:0/679818842 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4010720 con 0x7f7bcc106f10 2024-09-12T03:00:32.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.750+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bcc1a7110 con 0x7f7bcc106f10 2024-09-12T03:00:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.752+0000 7f7bc37fe640 1 -- 172.21.15.16:0/679818842 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7bc4005540 con 0x7f7bcc106f10 2024-09-12T03:00:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.753+0000 7f7bc37fe640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 0x7f7bb80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.753+0000 7f7bd6bb6740 1 Processor -- start 2024-09-12T03:00:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.753+0000 7f7bd2579640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 0x7f7bb80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.753+0000 7f7bc37fe640 1 -- 172.21.15.16:0/679818842 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bc40cf6d0 con 0x7f7bcc106f10 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.753+0000 7f7bd6bb6740 1 -- start start 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 0x55ef82e97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef82df02f0 con 0x55ef82e8bb40 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ef82d97cf0 con 0x55ef82e95370 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef82dec2b0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd2579640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd2d7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 0x55ef82e97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59662/0 (socket says 172.21.15.16:59662) 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2852502801 learned_addr learned my addr 172.21.15.16:0/2852502801 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:32.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd2579640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 0x7f7bb80a5040 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80096c0 tx=0x7f7bc80046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2852502801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 msgr2=0x55ef82e97760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 0x55ef82e97760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2852502801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 msgr2=0x55ef82e8a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2852502801 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef82d7c3a0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.754+0000 7f7bd2579640 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.755+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 0x55ef82e97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.755+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009cf0 tx=0x7f7bbc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73e8fc12ba5855a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.755+0000 7f7bc0ff9640 1 -- 172.21.15.16:0/2852502801 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc012450 con 0x55ef82e8a5d0 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.755+0000 7f7bc0ff9640 1 -- 172.21.15.16:0/2852502801 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bbc012ad0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.755+0000 7f7bc0ff9640 1 -- 172.21.15.16:0/2852502801 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00fbf0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2852502801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 msgr2=0x55ef82e8aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009cf0 tx=0x7f7bbc0117b0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2852502801 shutdown_connections 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e95370 0x55ef82e97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8a5d0 0x55ef82e8aa10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2852502801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2852502801 >> 172.21.15.16:0/2852502801 conn(0x55ef82e53310 msgr2=0x55ef82e86c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2852502801 shutdown_connections 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2852502801 wait complete. 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 -- start start 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.756+0000 7f7bd6bb6740 1 Processor -- start 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e94d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e91fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e92570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef82e98070 con 0x55ef82e8bb40 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ef82dab060 con 0x55ef82e8a5d0 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef82e981f0 con 0x55ef82e95370 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd6bb6740 10 client.0 _ll_register_callbacks cb 0x55ef82afe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e91fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e94d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e91fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47376/0 (socket says 172.21.15.16:47376) 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd1d78640 1 -- 172.21.15.16:0/930644771 learned_addr learned my addr 172.21.15.16:0/930644771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e92570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 -- 172.21.15.16:0/930644771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 msgr2=0x55ef82e92570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e92570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 -- 172.21.15.16:0/930644771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 msgr2=0x55ef82e91fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e91fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.757+0000 7f7bd2579640 1 -- 172.21.15.16:0/930644771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef82b613c0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd2579640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e94d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8000c00 tx=0x7f7bc8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88368519461ad1b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8035040 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bc8003cc0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bbc012ad0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7bc80315b0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 10 client.27851 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef82ea3ac0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bbc012ad0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7bc8052860 con 0x55ef82e8a5d0 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 10 client.27851 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 10 client.27851 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 20 client.27851 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82ea5250 0x55ef82ea7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 4 client.27851 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.758+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef82dfd860 con 0x55ef82ea5250 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.759+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82ea5250 0x55ef82ea7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.759+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82ea5250 0x55ef82ea7640 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90916bf6344a76f1 server_cookie=9a30efdf1ce71ed0 in_seq=0 out_seq=0 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.759+0000 7f7bb5ffb640 10 client.27851 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/930644771 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ef82dfd860 con 0x55ef82ea5250 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bb5ffb640 10 client.27851 handle_command_reply: tid=0 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 client.27851 shutdown 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 msgr2=0x55ef82e94d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e94d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8000c00 tx=0x7f7bc8009260 comp rx=0 tx=0).stop 2024-09-12T03:00:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 shutdown_connections 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82ea5250 0x55ef82ea7640 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e92570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e91fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/930644771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e94d20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 >> 172.21.15.16:0/930644771 conn(0x55ef82e53310 msgr2=0x55ef82e8c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bb5ffb640 0 client.27851 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 shutdown_connections 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/930644771 wait complete. 2024-09-12T03:00:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.760+0000 7f7bd6bb6740 20 client.27851 trim_cache size 0 max 16384 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 Processor -- start 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 -- start start 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e85bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e8d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1af730 con 0x55ef82e8bb40 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1b1270 con 0x55ef82e8a5d0 2024-09-12T03:00:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1afe00 con 0x55ef82e95370 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47388/0 (socket says 172.21.15.16:47388) 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2663158931 learned_addr learned my addr 172.21.15.16:0/2663158931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd2579640 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e85bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.774+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e8d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2663158931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 msgr2=0x55ef82e8d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e8d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2663158931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 msgr2=0x55ef82e85bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e85bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 -- 172.21.15.16:0/2663158931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef82e8dba0 con 0x55ef82e8bb40 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00f7e0 tx=0x7f7bbc0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9668cb091a32b16d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bb7fff640 1 -- 172.21.15.16:0/2663158931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00f440 con 0x55ef82e8bb40 2024-09-12T03:00:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bb7fff640 1 -- 172.21.15.16:0/2663158931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bbc009870 con 0x55ef82e8bb40 2024-09-12T03:00:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.775+0000 7f7bb7fff640 1 -- 172.21.15.16:0/2663158931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc005040 con 0x55ef82e8bb40 2024-09-12T03:00:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2663158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 msgr2=0x55ef82e8d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00f7e0 tx=0x7f7bbc0058d0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2663158931 shutdown_connections 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e95370 0x55ef82e8d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e8bb40 0x55ef82e8d120 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/2663158931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e85bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2663158931 >> 172.21.15.16:0/2663158931 conn(0x55ef82e53310 msgr2=0x55ef82e854e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2663158931 shutdown_connections 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/2663158931 wait complete. 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 -- start start 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.776+0000 7f7bd6bb6740 1 Processor -- start 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e99340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 0x55ef82e99880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1b0ba0 con 0x55ef82e95370 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1b04d0 con 0x55ef82e8a5d0 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc1ad710 con 0x55ef82e8bb40 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd6bb6740 10 client.0 _ll_register_callbacks cb 0x7f7bc40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2579640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e99340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd1d78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 0x55ef82e99880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47396/0 (socket says 172.21.15.16:47396) 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/3968591225 learned_addr learned my addr 172.21.15.16:0/3968591225 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/3968591225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 msgr2=0x55ef82e99880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 0x55ef82e99880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 msgr2=0x55ef82e99340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e99340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 -- 172.21.15.16:0/3968591225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef82f0ede0 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd1d78640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 0x55ef82e99880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bd2d7a640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005ba0 tx=0x7f7bc4005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea207e544e3271a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.777+0000 7f7bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc409a300 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bc4060820 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bbc009870 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7bc40a4720 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bd6bb6740 10 client.18450 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ef82f10100 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bbc009870 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.778+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7bc409b390 con 0x55ef82e95370 2024-09-12T03:00:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 10 client.18450 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 10 client.18450 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 20 client.18450 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82f10820 0x55ef82f12c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 4 client.18450 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef82df7d70 con 0x55ef82f10820 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd2579640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82f10820 0x55ef82f12c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd2579640 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82f10820 0x55ef82f12c10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=214fb378e9e12fcc server_cookie=246ed1833d3f4189 in_seq=0 out_seq=0 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bb5ffb640 10 client.18450 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bb5ffb640 1 -- 172.21.15.16:0/3968591225 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef82df7d70 con 0x55ef82f10820 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bb5ffb640 10 client.18450 handle_command_reply: tid=0 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 client.18450 shutdown 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 msgr2=0x55ef82e99dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005ba0 tx=0x7f7bc4005940 comp rx=0 tx=0).stop 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 shutdown_connections 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55ef82f10820 0x55ef82f12c10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef82e95370 0x55ef82e99dc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef82e8bb40 0x55ef82e99880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.779+0000 7f7bd6bb6740 1 --2- 172.21.15.16:0/3968591225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ef82e8a5d0 0x55ef82e99340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 >> 172.21.15.16:0/3968591225 conn(0x55ef82e53310 msgr2=0x55ef82e85d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bb5ffb640 0 client.18450 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 shutdown_connections 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd6bb6740 1 -- 172.21.15.16:0/3968591225 wait complete. 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd6bb6740 20 client.18450 trim_cache size 0 max 16384 2024-09-12T03:00:32.786 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:00:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 msgr2=0x7f7bb80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 0x7f7bb80a5040 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80096c0 tx=0x7f7bc80046c0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 msgr2=0x7f7bcc074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 0x7f7bcc074f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009fd0 tx=0x7f7bc400ef10 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 shutdown_connections 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bcc106f10 0x7f7bcc074f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bb80a2b80 0x7f7bb80a5040 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bcc102790 0x7f7bcc074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 --2- 172.21.15.16:0/679818842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bcc101de0 0x7f7bcc0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 >> 172.21.15.16:0/679818842 conn(0x7f7bcc0fd8c0 msgr2=0x7f7bcc0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 shutdown_connections 2024-09-12T03:00:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:32.780+0000 7f7bd47dd640 1 -- 172.21.15.16:0/679818842 wait complete. 2024-09-12T03:00:33.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:00:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1264769 9.84 MB/sec execute 575 sec latency 10.450 ms 2024-09-12T03:00:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:34.237 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:34 smithi016 ceph-mon[30925]: pgmap v716: 97 pgs: 97 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-09-12T03:00:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:34 smithi184 ceph-mon[35321]: pgmap v716: 97 pgs: 97 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-09-12T03:00:34.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 -- 172.21.15.16:0/2517806192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 msgr2=0x7fa6641029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 --2- 172.21.15.16:0/2517806192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6641029b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fa650009900 tx=0x7fa65002a480 comp rx=0 tx=0).stop 2024-09-12T03:00:34.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 -- 172.21.15.16:0/2517806192 shutdown_connections 2024-09-12T03:00:34.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 --2- 172.21.15.16:0/2517806192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664102ef0 0x7fa66410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 --2- 172.21.15.16:0/2517806192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6641029b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 --2- 172.21.15.16:0/2517806192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664100e20 0x7fa664101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.327+0000 7fa66abd4640 1 -- 172.21.15.16:0/2517806192 >> 172.21.15.16:0/2517806192 conn(0x7fa6640fc7d0 msgr2=0x7fa6640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 -- 172.21.15.16:0/2517806192 shutdown_connections 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 -- 172.21.15.16:0/2517806192 wait complete. 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 Processor -- start 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 -- start start 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664100e20 0x7fa6640fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.328+0000 7fa66abd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa66abd4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664102ef0 0x7fa6640fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa66abd4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa664110f80 con 0x7fa664102570 2024-09-12T03:00:34.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa66abd4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa664110e00 con 0x7fa664100e20 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa66abd4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa664111100 con 0x7fa664102ef0 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa668970640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664100e20 0x7fa6640fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47414/0 (socket says 172.21.15.16:47414) 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 -- 172.21.15.16:0/4001934196 learned_addr learned my addr 172.21.15.16:0/4001934196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa669171640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664102ef0 0x7fa6640fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 -- 172.21.15.16:0/4001934196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664102ef0 msgr2=0x7fa6640fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664102ef0 0x7fa6640fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 -- 172.21.15.16:0/4001934196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664100e20 msgr2=0x7fa6640fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664100e20 0x7fa6640fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 -- 172.21.15.16:0/4001934196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6641a6990 con 0x7fa664102570 2024-09-12T03:00:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa663fff640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa650006010 tx=0x7fa650003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa661ffb640 1 -- 172.21.15.16:0/4001934196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65003f070 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa661ffb640 1 -- 172.21.15.16:0/4001934196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa650031c30 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.329+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6641a6c20 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.330+0000 7fa661ffb640 1 -- 172.21.15.16:0/4001934196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650035030 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.330+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a7100 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.332+0000 7fa661ffb640 1 -- 172.21.15.16:0/4001934196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa650042070 con 0x7fa664102570 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 Processor -- start 2024-09-12T03:00:34.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 -- start start 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4712090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc471f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ebc46782f0 con 0x55ebc471d370 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebc461fcf0 con 0x55ebc47125d0 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa66cfad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebc46742b0 con 0x55ebc4713b40 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa661ffb640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 0x7fa6540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa669171640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4712090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54094/0 (socket says 172.21.15.16:54094) 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 -- 172.21.15.16:0/1745447465 learned_addr learned my addr 172.21.15.16:0/1745447465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa663fff640 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc471f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa669171640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 0x7fa6540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 -- 172.21.15.16:0/1745447465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 msgr2=0x55ebc4712090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4712090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.333+0000 7fa668970640 1 -- 172.21.15.16:0/1745447465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 msgr2=0x55ebc471f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa668970640 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc471f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa668970640 1 -- 172.21.15.16:0/1745447465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc46043a0 con 0x55ebc47125d0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa669171640 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4712090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa661ffb640 1 -- 172.21.15.16:0/4001934196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa6500ef690 con 0x7fa664102570 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa668970640 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa658009d20 tx=0x7fa658011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e00a7ab21feb182 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa64effd640 1 -- 172.21.15.16:0/1745447465 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65801e070 con 0x55ebc47125d0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa669171640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 0x7fa6540a5040 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009fd0 tx=0x7fa65c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa64effd640 1 -- 172.21.15.16:0/1745447465 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa658019b10 con 0x55ebc47125d0 2024-09-12T03:00:34.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.334+0000 7fa64effd640 1 -- 172.21.15.16:0/1745447465 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65800eb90 con 0x55ebc47125d0 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- 172.21.15.16:0/1745447465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 msgr2=0x55ebc4712a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa658009d20 tx=0x7fa658011810 comp rx=0 tx=0).stop 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- 172.21.15.16:0/1745447465 shutdown_connections 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc471f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4712a10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1745447465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4712090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- 172.21.15.16:0/1745447465 >> 172.21.15.16:0/1745447465 conn(0x55ebc46db310 msgr2=0x55ebc470ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- 172.21.15.16:0/1745447465 shutdown_connections 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- 172.21.15.16:0/1745447465 wait complete. 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 -- start start 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.335+0000 7fa66cfad740 1 Processor -- start 2024-09-12T03:00:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4719fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc471a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ebc4720070 con 0x55ebc47125d0 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebc4633060 con 0x55ebc4713b40 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebc47201f0 con 0x55ebc471d370 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa66cfad740 10 client.0 _ll_register_callbacks cb 0x55ebc43868c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47438/0 (socket says 172.21.15.16:47438) 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 -- 172.21.15.16:0/1357632831 learned_addr learned my addr 172.21.15.16:0/1357632831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa669171640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc471a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa663fff640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4719fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 -- 172.21.15.16:0/1357632831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 msgr2=0x55ebc471a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc471a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 -- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 msgr2=0x55ebc4719fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4719fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 -- 172.21.15.16:0/1357632831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc43e93c0 con 0x55ebc47125d0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa668970640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa658011d10 tx=0x7fa658019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6c3e0edb0c5e9d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.336+0000 7fa643fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658027020 con 0x55ebc47125d0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa65800eaf0 con 0x55ebc47125d0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa658019b10 con 0x55ebc47125d0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa65801e070 con 0x55ebc47125d0 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 10 client.18462 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ebc472bac0 con 0x55ebc47125d0 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa658019b10 con 0x55ebc47125d0 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa6580046c0 con 0x55ebc47125d0 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 10 client.18462 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 10 client.18462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 20 client.18462 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc472ce10 0x55ebc472f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 4 client.18462 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.337+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc4685860 con 0x55ebc472ce10 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.338+0000 7fa663fff640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc472ce10 0x55ebc472f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.338+0000 7fa663fff640 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc472ce10 0x55ebc472f200 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7172e53a99c971 server_cookie=e85e9a9094745fc2 in_seq=0 out_seq=0 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.338+0000 7fa643fff640 10 client.18462 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:34.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.338+0000 7fa643fff640 1 -- 172.21.15.16:0/1357632831 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ebc4685860 con 0x55ebc472ce10 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.338+0000 7fa643fff640 10 client.18462 handle_command_reply: tid=0 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 client.18462 shutdown 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 msgr2=0x55ebc471cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa658011d10 tx=0x7fa658019660 comp rx=0 tx=0).stop 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 shutdown_connections 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc472ce10 0x55ebc472f200 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc471a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4719fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 --2- 172.21.15.16:0/1357632831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc471cd20 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 >> 172.21.15.16:0/1357632831 conn(0x55ebc46db310 msgr2=0x55ebc4714960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa643fff640 0 client.18462 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:34.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 shutdown_connections 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 1 -- 172.21.15.16:0/1357632831 wait complete. 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.339+0000 7fa66cfad740 20 client.18462 trim_cache size 0 max 16384 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 Processor -- start 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 -- start start 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4715120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc4715660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ab300 con 0x55ebc47125d0 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ace40 con 0x55ebc4713b40 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa66cfad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ab9d0 con 0x55ebc471d370 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa668970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4715120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa668970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47454/0 (socket says 172.21.15.16:47454) 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa668970640 1 -- 172.21.15.16:0/3008781014 learned_addr learned my addr 172.21.15.16:0/3008781014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa669171640 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc4715660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 -- 172.21.15.16:0/3008781014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 msgr2=0x55ebc4715660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc4715660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 -- 172.21.15.16:0/3008781014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 msgr2=0x55ebc4729310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa663fff640 1 -- 172.21.15.16:0/3008781014 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc470f960 con 0x55ebc4713b40 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.354+0000 7fa668970640 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa663fff640 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4715120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ed1f0 tx=0x7fa6500e6a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2cea1fd6ca6ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa64e7fc640 1 -- 172.21.15.16:0/3008781014 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650049040 con 0x55ebc4713b40 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa64e7fc640 1 -- 172.21.15.16:0/3008781014 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa650031de0 con 0x55ebc4713b40 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa64e7fc640 1 -- 172.21.15.16:0/3008781014 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500b8620 con 0x55ebc4713b40 2024-09-12T03:00:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 -- 172.21.15.16:0/3008781014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 msgr2=0x55ebc4715120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4715120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ed1f0 tx=0x7fa6500e6a90 comp rx=0 tx=0).stop 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 -- 172.21.15.16:0/3008781014 shutdown_connections 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc471d370 0x55ebc4715660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc4713b40 0x55ebc4715120 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3008781014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc47125d0 0x55ebc4729310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 -- 172.21.15.16:0/3008781014 >> 172.21.15.16:0/3008781014 conn(0x55ebc46db310 msgr2=0x55ebc47190c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.355+0000 7fa66cfad740 1 -- 172.21.15.16:0/3008781014 shutdown_connections 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 -- 172.21.15.16:0/3008781014 wait complete. 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 -- start start 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 Processor -- start 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4725760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4725ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ac770 con 0x55ebc471d370 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ac0a0 con 0x55ebc47125d0 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641a9300 con 0x55ebc4713b40 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa66cfad740 10 client.0 _ll_register_callbacks cb 0x7fa65801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa669171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.356+0000 7fa663fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4725ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4725760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa669171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47462/0 (socket says 172.21.15.16:47462) 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa669171640 1 -- 172.21.15.16:0/3168284616 learned_addr learned my addr 172.21.15.16:0/3168284616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa663fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4725ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:59722/0 (socket says 172.21.15.16:59722) 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 -- 172.21.15.16:0/3168284616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 msgr2=0x55ebc4725ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4725ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 -- 172.21.15.16:0/3168284616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 msgr2=0x55ebc4716920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 -- 172.21.15.16:0/3168284616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc4717150 con 0x55ebc47125d0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa669171640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa668970640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4725760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa658004bb0 tx=0x7fa658019cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f4b3ad08af23bb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa643fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658027040 con 0x55ebc47125d0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa658005350 con 0x55ebc47125d0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.357+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa650031de0 con 0x55ebc47125d0 2024-09-12T03:00:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa6580196d0 con 0x55ebc47125d0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 10 client.27875 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ebc47974c0 con 0x55ebc47125d0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa650031de0 con 0x55ebc47125d0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa658035020 con 0x55ebc47125d0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 10 client.27875 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 10 client.27875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 20 client.27875 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc4797b20 0x55ebc4799f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 4 client.27875 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebc467fd70 con 0x55ebc4797b20 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa663fff640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc4797b20 0x55ebc4799f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa663fff640 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc4797b20 0x55ebc4799f40 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb2cde22b0a73e5e server_cookie=f8dc361a1f5b0491 in_seq=0 out_seq=0 2024-09-12T03:00:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.358+0000 7fa643fff640 10 client.27875 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa643fff640 1 -- 172.21.15.16:0/3168284616 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebc467fd70 con 0x55ebc4797b20 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa643fff640 10 client.27875 handle_command_reply: tid=0 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 client.27875 shutdown 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 msgr2=0x55ebc4725760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4725760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa658004bb0 tx=0x7fa658019cc0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 shutdown_connections 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebc4797b20 0x55ebc4799f40 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebc471d370 0x55ebc4716920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebc4713b40 0x55ebc4725ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 --2- 172.21.15.16:0/3168284616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebc47125d0 0x55ebc4725760 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa643fff640 0 client.27875 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 >> 172.21.15.16:0/3168284616 conn(0x55ebc46db310 msgr2=0x55ebc470d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 shutdown_connections 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 1 -- 172.21.15.16:0/3168284616 wait complete. 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.359+0000 7fa66cfad740 20 client.27875 trim_cache size 0 max 16384 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 msgr2=0x7fa6540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 0x7fa6540a5040 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009fd0 tx=0x7fa65c004ab0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 msgr2=0x7fa6640fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa650006010 tx=0x7fa650003c10 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 shutdown_connections 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa664102ef0 0x7fa6640fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6540a2b80 0x7fa6540a5040 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa664102570 0x7fa6640fd1c0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 --2- 172.21.15.16:0/4001934196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa664100e20 0x7fa6640fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 >> 172.21.15.16:0/4001934196 conn(0x7fa6640fc7d0 msgr2=0x7fa66406a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 shutdown_connections 2024-09-12T03:00:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:34.360+0000 7fa66abd4640 1 -- 172.21.15.16:0/4001934196 wait complete. 2024-09-12T03:00:34.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:34 smithi033 ceph-mon[35092]: pgmap v716: 97 pgs: 97 active+clean; 262 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-09-12T03:00:34.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1267810 9.85 MB/sec execute 576 sec latency 12.695 ms 2024-09-12T03:00:35.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1270242 9.85 MB/sec execute 577 sec latency 19.847 ms 2024-09-12T03:00:36.155 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:00:36.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:00:36.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:36 smithi184 ceph-mon[35321]: pgmap v717: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-12T03:00:36.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:36 smithi016 ceph-mon[30925]: pgmap v717: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-12T03:00:36.434 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:36.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:36 smithi033 ceph-mon[35092]: pgmap v717: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-09-12T03:00:36.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1272432 9.86 MB/sec execute 578 sec latency 17.912 ms 2024-09-12T03:00:36.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 -- 172.21.15.16:0/3307284580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590102570 msgr2=0x7fc5901029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 --2- 172.21.15.16:0/3307284580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590102570 0x7fc5901029b0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fc578009900 tx=0x7fc57802a480 comp rx=0 tx=0).stop 2024-09-12T03:00:36.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 -- 172.21.15.16:0/3307284580 shutdown_connections 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 --2- 172.21.15.16:0/3307284580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 --2- 172.21.15.16:0/3307284580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590102570 0x7fc5901029b0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 --2- 172.21.15.16:0/3307284580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590100e20 0x7fc590101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 -- 172.21.15.16:0/3307284580 >> 172.21.15.16:0/3307284580 conn(0x7fc5900fc7d0 msgr2=0x7fc5900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 -- 172.21.15.16:0/3307284580 shutdown_connections 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.939+0000 7fc594f7c640 1 -- 172.21.15.16:0/3307284580 wait complete. 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 Processor -- start 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 -- start start 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 0x7fc590198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc590110f80 con 0x7fc590100e20 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc590110e00 con 0x7fc590102570 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc594f7c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc590111100 con 0x7fc590102ef0 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50664/0 (socket says 172.21.15.16:50664) 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58e59c640 1 -- 172.21.15.16:0/2470367490 learned_addr learned my addr 172.21.15.16:0/2470367490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42124/0 (socket says 172.21.15.16:42124) 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc587fff640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 0x7fc590198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 -- 172.21.15.16:0/2470367490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 msgr2=0x7fc590198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 0x7fc590198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 -- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 msgr2=0x7fc590198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.940+0000 7fc58ed9d640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc58ed9d640 1 -- 172.21.15.16:0/2470367490 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59019d4c0 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc58e59c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc587fff640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 0x7fc590198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc58ed9d640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc588009fd0 tx=0x7fc58800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc5877fe640 1 -- 172.21.15.16:0/2470367490 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58800ce20 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59019d7b0 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc5877fe640 1 -- 172.21.15.16:0/2470367490 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc588005020 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc5877fe640 1 -- 172.21.15.16:0/2470367490 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588010720 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.941+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5901a5090 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc5877fe640 1 -- 172.21.15.16:0/2470367490 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc588005540 con 0x7fc590102ef0 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 Processor -- start 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 -- start start 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaf340 0x56124bfbdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfc0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56124bf14280 con 0x56124bfaf340 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56124bebbb10 con 0x56124bfaea20 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.944+0000 7fc597355740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56124bf10240 con 0x56124bfbe150 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaf340 0x56124bfbdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39972/0 (socket says 172.21.15.16:39972) 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 -- 172.21.15.16:0/1612325346 learned_addr learned my addr 172.21.15.16:0/1612325346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfc0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 -- 172.21.15.16:0/1612325346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 msgr2=0x56124bfc0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfc0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 -- 172.21.15.16:0/1612325346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaf340 msgr2=0x56124bfbdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc5877fe640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 0x7fc57c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaf340 0x56124bfbdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 -- 172.21.15.16:0/1612325346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124bec81f0 con 0x56124bfaea20 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc5877fe640 1 -- 172.21.15.16:0/2470367490 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc5880cf770 con 0x7fc590102ef0 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc587fff640 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc57802a9a0 tx=0x7fc5780056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dccab7c20ea518b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.945+0000 7fc58ed9d640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 0x7fc57c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.946+0000 7fc584ff9640 1 -- 172.21.15.16:0/1612325346 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578035040 con 0x56124bfaea20 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.946+0000 7fc58ed9d640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 0x7fc57c0a5040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc588005ba0 tx=0x7fc5880054b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.946+0000 7fc584ff9640 1 -- 172.21.15.16:0/1612325346 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc578003ad0 con 0x56124bfaea20 2024-09-12T03:00:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.946+0000 7fc584ff9640 1 -- 172.21.15.16:0/1612325346 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57803d840 con 0x56124bfaea20 2024-09-12T03:00:36.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 -- 172.21.15.16:0/1612325346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 msgr2=0x56124bfaee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc57802a9a0 tx=0x7fc5780056f0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 -- 172.21.15.16:0/1612325346 shutdown_connections 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfc0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaf340 0x56124bfbdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 --2- 172.21.15.16:0/1612325346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaea20 0x56124bfaee00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 -- 172.21.15.16:0/1612325346 >> 172.21.15.16:0/1612325346 conn(0x56124bfab0f0 msgr2=0x56124bfab4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 -- 172.21.15.16:0/1612325346 shutdown_connections 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.948+0000 7fc597355740 1 -- 172.21.15.16:0/1612325346 wait complete. 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 -- start start 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 Processor -- start 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaf340 0x56124bfbb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfbe150 0x56124bfbbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56124bfc0cc0 con 0x56124bfaea20 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56124becef10 con 0x56124bfbe150 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56124bfc0e40 con 0x56124bfaf340 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc597355740 10 client.0 _ll_register_callbacks cb 0x56124be97a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50682/0 (socket says 172.21.15.16:50682) 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc587fff640 1 -- 172.21.15.16:0/78034805 learned_addr learned my addr 172.21.15.16:0/78034805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc58e59c640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaf340 0x56124bfbb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.949+0000 7fc587fff640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfbe150 0x56124bfbbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 -- 172.21.15.16:0/78034805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaf340 msgr2=0x56124bfbb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaf340 0x56124bfbb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 -- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfbe150 msgr2=0x56124bfbbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfbe150 0x56124bfbbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 -- 172.21.15.16:0/78034805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124bc856a0 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc587fff640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc57803d500 tx=0x7fc578009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f122d3d48743cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc575ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578054060 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc57803d5c0 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc578003ad0 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc578041070 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc597355740 10 client.18480 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.950+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56124bfcc980 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc578003ad0 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc57803a040 con 0x56124bfaea20 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 10 client.18480 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 10 client.18480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 20 client.18480 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124bfcdcd0 0x56124bfd00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 4 client.18480 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56124bf217f0 con 0x56124bfcdcd0 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc58e59c640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124bfcdcd0 0x56124bfd00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc575ffb640 10 client.18480 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.951+0000 7fc58e59c640 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124bfcdcd0 0x56124bfd00c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=147d11bf6c3c7686 server_cookie=13f1a49beb15bf34 in_seq=0 out_seq=0 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc575ffb640 1 -- 172.21.15.16:0/78034805 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56124bf217f0 con 0x56124bfcdcd0 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc575ffb640 10 client.18480 handle_command_reply: tid=0 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 client.18480 shutdown 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 msgr2=0x56124bfbb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc57803d500 tx=0x7fc578009660 comp rx=0 tx=0).stop 2024-09-12T03:00:36.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 shutdown_connections 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124bfcdcd0 0x56124bfd00c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfbe150 0x56124bfbbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaf340 0x56124bfbb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc575ffb640 0 client.18480 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 --2- 172.21.15.16:0/78034805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb0a0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 >> 172.21.15.16:0/78034805 conn(0x56124bfab0f0 msgr2=0x56124bfb0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.952+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 shutdown_connections 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.953+0000 7fc597355740 1 -- 172.21.15.16:0/78034805 wait complete. 2024-09-12T03:00:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.953+0000 7fc597355740 20 client.18480 trim_cache size 0 max 16384 2024-09-12T03:00:36.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.971+0000 7fc597355740 1 Processor -- start 2024-09-12T03:00:36.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.971+0000 7fc597355740 1 -- start start 2024-09-12T03:00:36.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfbbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfb8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ab290 con 0x56124bfaea20 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901acdd0 con 0x56124bfaf340 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc597355740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ab960 con 0x56124bfbe150 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50692/0 (socket says 172.21.15.16:50692) 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfbbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 -- 172.21.15.16:0/3045987306 learned_addr learned my addr 172.21.15.16:0/3045987306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfb8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 -- 172.21.15.16:0/3045987306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 msgr2=0x56124bfb8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfb8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 -- 172.21.15.16:0/3045987306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 msgr2=0x56124bfbbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfbbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 -- 172.21.15.16:0/3045987306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124bfaf170 con 0x56124bfaea20 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc58e59c640 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfbbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc587fff640 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc578004100 tx=0x7fc578005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fa04ae96abf244 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.972+0000 7fc577fff640 1 -- 172.21.15.16:0/3045987306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578054070 con 0x56124bfaea20 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc577fff640 1 -- 172.21.15.16:0/3045987306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc578009780 con 0x56124bfaea20 2024-09-12T03:00:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc577fff640 1 -- 172.21.15.16:0/3045987306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578035070 con 0x56124bfaea20 2024-09-12T03:00:36.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 -- 172.21.15.16:0/3045987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 msgr2=0x56124bfbb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc578004100 tx=0x7fc578005dd0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 -- 172.21.15.16:0/3045987306 shutdown_connections 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfbe150 0x56124bfb8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfbbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 --2- 172.21.15.16:0/3045987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfaea20 0x56124bfbb770 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 -- 172.21.15.16:0/3045987306 >> 172.21.15.16:0/3045987306 conn(0x56124bfab0f0 msgr2=0x56124bfbb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 -- 172.21.15.16:0/3045987306 shutdown_connections 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.973+0000 7fc597355740 1 -- 172.21.15.16:0/3045987306 wait complete. 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 -- start start 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 Processor -- start 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 0x56124bfd1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfbe150 0x56124c037610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ac700 con 0x56124bfbe150 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ac030 con 0x56124bfaf340 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901a9270 con 0x56124bfaea20 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc597355740 10 client.0 _ll_register_callbacks cb 0x7fc580009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfbe150 0x56124c037610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.974+0000 7fc58e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39998/0 (socket says 172.21.15.16:39998) 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 -- 172.21.15.16:0/2616664450 learned_addr learned my addr 172.21.15.16:0/2616664450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc587fff640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 0x56124bfd1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 -- 172.21.15.16:0/2616664450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 msgr2=0x56124bfd1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 0x56124bfd1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 -- 172.21.15.16:0/2616664450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfbe150 msgr2=0x56124c037610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfbe150 0x56124c037610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 -- 172.21.15.16:0/2616664450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124c037b50 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc587fff640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 0x56124bfd1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc58e59c640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc580005d00 tx=0x7fc580004560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2808b39608669b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc575ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580005220 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc5800053c0 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc578009780 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.975+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc5800055c0 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 10 client.27893 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56124c038e80 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc578009780 con 0x56124bfaf340 2024-09-12T03:00:36.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc580027070 con 0x56124bfaf340 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 10 client.27893 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 10 client.27893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 20 client.27893 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124c039630 0x56124c03ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 4 client.27893 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56124bf1bd00 con 0x56124c039630 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.976+0000 7fc587fff640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124c039630 0x56124c03ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.977+0000 7fc587fff640 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124c039630 0x56124c03ba50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc574bcb1d08e04d server_cookie=ed5f087ebed4f5af in_seq=0 out_seq=0 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.977+0000 7fc575ffb640 10 client.27893 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.977+0000 7fc575ffb640 1 -- 172.21.15.16:0/2616664450 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56124bf1bd00 con 0x56124c039630 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.977+0000 7fc575ffb640 10 client.27893 handle_command_reply: tid=0 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.977+0000 7fc597355740 1 client.27893 shutdown 2024-09-12T03:00:36.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 msgr2=0x56124bfd1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc580005d00 tx=0x7fc580004560 comp rx=0 tx=0).stop 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 shutdown_connections 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56124c039630 0x56124c03ba50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56124bfbe150 0x56124c037610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56124bfaf340 0x56124bfd1a50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 --2- 172.21.15.16:0/2616664450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56124bfaea20 0x56124bfd1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 >> 172.21.15.16:0/2616664450 conn(0x56124bfab0f0 msgr2=0x56124bfbb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc575ffb640 0 client.27893 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 shutdown_connections 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 1 -- 172.21.15.16:0/2616664450 wait complete. 2024-09-12T03:00:36.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.978+0000 7fc597355740 20 client.27893 trim_cache size 0 max 16384 2024-09-12T03:00:36.985 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:00:36.985 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:00:36.985 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "133348c8-a0e2-41a8-98a4-c0cabbdeb602", 2024-09-12T03:00:36.985 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:00:36.985 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 msgr2=0x7fc57c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 0x7fc57c0a5040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc588005ba0 tx=0x7fc5880054b0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 msgr2=0x7fc59019cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc588009fd0 tx=0x7fc58800ef10 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 shutdown_connections 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590102ef0 0x7fc59019cde0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc590102570 0x7fc590198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc57c0a2b80 0x7fc57c0a5040 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 --2- 172.21.15.16:0/2470367490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc590100e20 0x7fc590198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 >> 172.21.15.16:0/2470367490 conn(0x7fc5900fc7d0 msgr2=0x7fc59006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:36.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 shutdown_connections 2024-09-12T03:00:36.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:36.979+0000 7fc594f7c640 1 -- 172.21.15.16:0/2470367490 wait complete. 2024-09-12T03:00:37.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:00:37.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1275297 9.87 MB/sec execute 579 sec latency 7.767 ms 2024-09-12T03:00:37.959 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: pgmap v718: 97 pgs: 97 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.7 MiB/s wr, 228 op/s 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:38 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: pgmap v718: 97 pgs: 97 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.7 MiB/s wr, 228 op/s 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:38 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 -- 172.21.15.16:0/2535405475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 msgr2=0x7faebc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 --2- 172.21.15.16:0/2535405475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc101200 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7faeac009900 tx=0x7faeac02a480 comp rx=0 tx=0).stop 2024-09-12T03:00:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 -- 172.21.15.16:0/2535405475 shutdown_connections 2024-09-12T03:00:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 --2- 172.21.15.16:0/2535405475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102ef0 0x7faebc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 --2- 172.21.15.16:0/2535405475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102570 0x7faebc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 --2- 172.21.15.16:0/2535405475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc101200 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 -- 172.21.15.16:0/2535405475 >> 172.21.15.16:0/2535405475 conn(0x7faebc0fc7d0 msgr2=0x7faebc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 -- 172.21.15.16:0/2535405475 shutdown_connections 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.446+0000 7faec1c64640 1 -- 172.21.15.16:0/2535405475 wait complete. 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 Processor -- start 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 -- start start 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102570 0x7faebc197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 0x7faebc19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.447+0000 7faec1c64640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faebc110f80 con 0x7faebc100e20 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faec1c64640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faebc110e00 con 0x7faebc102ef0 2024-09-12T03:00:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faec1c64640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faebc111100 con 0x7faebc102570 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50714/0 (socket says 172.21.15.16:50714) 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 -- 172.21.15.16:0/593506590 learned_addr learned my addr 172.21.15.16:0/593506590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faeb3fff640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102570 0x7faebc197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebbfff640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 0x7faebc19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 -- 172.21.15.16:0/593506590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102570 msgr2=0x7faebc197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102570 0x7faebc197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 -- 172.21.15.16:0/593506590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 msgr2=0x7faebc19bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 0x7faebc19bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 -- 172.21.15.16:0/593506590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc19c370 con 0x7faebc100e20 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebbfff640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 0x7faebc19bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faebb7fe640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7faeac006010 tx=0x7faeac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faeb97fa640 1 -- 172.21.15.16:0/593506590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac03f070 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faeb97fa640 1 -- 172.21.15.16:0/593506590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeac031c30 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc19c600 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.448+0000 7faeb97fa640 1 -- 172.21.15.16:0/593506590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac035030 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.449+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc1a3f40 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faeb97fa640 1 -- 172.21.15.16:0/593506590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7faeac042070 con 0x7faebc100e20 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 Processor -- start 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 -- start start 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0009fd0 0x5653a0014350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0016c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56539ff6d2e0 con 0x5653a0014890 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56539ff14ce0 con 0x5653a0009fd0 2024-09-12T03:00:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faec403d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56539ff692a0 con 0x5653a00096b0 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.451+0000 7faeb97fa640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 0x7faea80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0009fd0 0x5653a0014350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0016c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42188/0 (socket says 172.21.15.16:42188) 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 -- 172.21.15.16:0/56734376 learned_addr learned my addr 172.21.15.16:0/56734376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 0x7faea80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faeb97fa640 1 -- 172.21.15.16:0/593506590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faeac0ef710 con 0x7faebc100e20 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 -- 172.21.15.16:0/56734376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0009fd0 msgr2=0x5653a0014350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0009fd0 0x5653a0014350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 -- 172.21.15.16:0/56734376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 msgr2=0x5653a0016c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0016c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 -- 172.21.15.16:0/56734376 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539fef9440 con 0x5653a00096b0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faeb4012e10 tx=0x7faeb40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f3b03c73a1ac53 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faebbfff640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 0x7faea80a5040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faeb4010ca0 tx=0x7faeb401d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faeb1ffb640 1 -- 172.21.15.16:0/56734376 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4012600 con 0x5653a00096b0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faeb1ffb640 1 -- 172.21.15.16:0/56734376 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeb4009d80 con 0x5653a00096b0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.452+0000 7faeb1ffb640 1 -- 172.21.15.16:0/56734376 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4005430 con 0x5653a00096b0 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 -- 172.21.15.16:0/56734376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 msgr2=0x5653a0009a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faeb4012e10 tx=0x7faeb40107a0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 -- 172.21.15.16:0/56734376 shutdown_connections 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0016c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0009fd0 0x5653a0014350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 --2- 172.21.15.16:0/56734376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0009a90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 -- 172.21.15.16:0/56734376 >> 172.21.15.16:0/56734376 conn(0x5653a0005c40 msgr2=0x5653a0006020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 -- 172.21.15.16:0/56734376 shutdown_connections 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.453+0000 7faec403d740 1 -- 172.21.15.16:0/56734376 wait complete. 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 -- start start 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 Processor -- start 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 0x5653a000fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0010310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5653a00035a0 con 0x5653a0014890 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56539ff28050 con 0x5653a00096b0 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5653a0003720 con 0x5653a0009fd0 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faec403d740 10 client.0 _ll_register_callbacks cb 0x56539ff68160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faebbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0010310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 0x5653a000fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40040/0 (socket says 172.21.15.16:40040) 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.454+0000 7faebb7fe640 1 -- 172.21.15.16:0/4034133185 learned_addr learned my addr 172.21.15.16:0/4034133185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 -- 172.21.15.16:0/4034133185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 msgr2=0x5653a000fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 0x5653a000fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 -- 172.21.15.16:0/4034133185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 msgr2=0x5653a0010310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0010310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 -- 172.21.15.16:0/4034133185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539fcde3c0 con 0x5653a00096b0 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faeb3fff640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 0x5653a000fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faebb7fe640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faeac000c00 tx=0x7faeac08af00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe09c1726cd35950 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac049070 con 0x5653a00096b0 2024-09-12T03:00:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faeac03b460 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb4009d80 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faeac0bc680 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faec403d740 10 client.27905 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653a0020bc0 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeb4009d80 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.455+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faeac0b85f0 con 0x5653a00096b0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 10 client.27905 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 10 client.27905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 20 client.27905 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a0021fb0 0x5653a0024420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 4 client.27905 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56539ff7a850 con 0x5653a0021fb0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faeb3fff640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a0021fb0 0x5653a0024420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7faeb3fff640 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a0021fb0 0x5653a0024420 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=551c1e977dc9bc31 server_cookie=29135e3e48d3a55a in_seq=0 out_seq=0 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.456+0000 7fae9affd640 10 client.27905 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7fae9affd640 1 -- 172.21.15.16:0/4034133185 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56539ff7a850 con 0x5653a0021fb0 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7fae9affd640 10 client.27905 handle_command_reply: tid=0 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 client.27905 shutdown 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 msgr2=0x5653a000f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faeac000c00 tx=0x7faeac08af00 comp rx=0 tx=0).stop 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 shutdown_connections 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a0021fb0 0x5653a0024420 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0014890 0x5653a0010310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a0009fd0 0x5653a000fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 --2- 172.21.15.16:0/4034133185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a00096b0 0x5653a000f890 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7fae9affd640 0 client.27905 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 >> 172.21.15.16:0/4034133185 conn(0x5653a0005c40 msgr2=0x5653a0002b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 shutdown_connections 2024-09-12T03:00:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 1 -- 172.21.15.16:0/4034133185 wait complete. 2024-09-12T03:00:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.457+0000 7faec403d740 20 client.27905 trim_cache size 0 max 16384 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.476+0000 7faec403d740 1 Processor -- start 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.476+0000 7faec403d740 1 -- start start 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a000e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a0015c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1ab3b0 con 0x5653a0009fd0 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1acef0 con 0x5653a0014890 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faec403d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1aba80 con 0x5653a00096b0 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a0015c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a0015c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50742/0 (socket says 172.21.15.16:50742) 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faeb3fff640 1 -- 172.21.15.16:0/1589918151 learned_addr learned my addr 172.21.15.16:0/1589918151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a000e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 -- 172.21.15.16:0/1589918151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 msgr2=0x5653a000e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a000e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 -- 172.21.15.16:0/1589918151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 msgr2=0x5653a0015c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a0015c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 -- 172.21.15.16:0/1589918151 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a0009e00 con 0x5653a0014890 2024-09-12T03:00:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.477+0000 7faebbfff640 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0004940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faeb4028040 tx=0x7faeb4012d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc84de1f7c15222b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faeb17fa640 1 -- 172.21.15.16:0/1589918151 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4009500 con 0x5653a0014890 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faeb17fa640 1 -- 172.21.15.16:0/1589918151 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeb4034070 con 0x5653a0014890 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faeb17fa640 1 -- 172.21.15.16:0/1589918151 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb40122a0 con 0x5653a0014890 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 -- 172.21.15.16:0/1589918151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 msgr2=0x5653a0004940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0004940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faeb4028040 tx=0x7faeb4012d10 comp rx=0 tx=0).stop 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 -- 172.21.15.16:0/1589918151 shutdown_connections 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0004940 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a0015c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 --2- 172.21.15.16:0/1589918151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a000e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 -- 172.21.15.16:0/1589918151 >> 172.21.15.16:0/1589918151 conn(0x5653a0005c40 msgr2=0x5653a0002b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 -- 172.21.15.16:0/1589918151 shutdown_connections 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.478+0000 7faec403d740 1 -- 172.21.15.16:0/1589918151 wait complete. 2024-09-12T03:00:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 -- start start 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 Processor -- start 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0019c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1ac820 con 0x5653a0009fd0 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1ac150 con 0x5653a0014890 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1a93b0 con 0x5653a00096b0 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faec403d740 10 client.0 _ll_register_callbacks cb 0x7faeac009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50746/0 (socket says 172.21.15.16:50746) 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42218/0 (socket says 172.21.15.16:42218) 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.479+0000 7faeb3fff640 1 -- 172.21.15.16:0/416683275 learned_addr learned my addr 172.21.15.16:0/416683275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:00:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebbfff640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0019c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 -- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 msgr2=0x5653a0019c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0019c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 -- 172.21.15.16:0/416683275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 msgr2=0x5653a00196d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 -- 172.21.15.16:0/416683275 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a008bcc0 con 0x5653a00096b0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faeb3fff640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faebb7fe640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faeac005ec0 tx=0x7faeac0ed290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a109bda6897640 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7fae9affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac049070 con 0x5653a00096b0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faeac0b85f0 con 0x5653a00096b0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb4034070 con 0x5653a00096b0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faeac0bbf80 con 0x5653a00096b0 2024-09-12T03:00:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faec403d740 10 client.27808 fetch_fsmap learned FSMap version 32 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.480+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653a008cf80 con 0x5653a00096b0 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeac0d6030 con 0x5653a00096b0 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faeac0bb460 con 0x5653a00096b0 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 10 client.27808 fetch_fsmap finished waiting for FSMap version 32 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 10 client.27808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 20 client.27808 populate_metadata read hostname 'smithi016' 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a008d570 0x5653a008f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 4 client.27808 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56539ff74d60 con 0x5653a008d570 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faeb3fff640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a008d570 0x5653a008f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7fae9affd640 10 client.27808 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.481+0000 7faeb3fff640 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a008d570 0x5653a008f990 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e321923dd5a17a0 server_cookie=32cfaf3d76b5d7b2 in_seq=0 out_seq=0 2024-09-12T03:00:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7fae9affd640 1 -- 172.21.15.16:0/416683275 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56539ff74d60 con 0x5653a008d570 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7fae9affd640 10 client.27808 handle_command_reply: tid=0 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 client.27808 shutdown 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 msgr2=0x5653a0019190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faeac005ec0 tx=0x7faeac0ed290 comp rx=0 tx=0).stop 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 shutdown_connections 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5653a008d570 0x5653a008f990 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5653a0014890 0x5653a0019c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a0009fd0 0x5653a00196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 --2- 172.21.15.16:0/416683275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653a00096b0 0x5653a0019190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7fae9affd640 0 client.27808 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 >> 172.21.15.16:0/416683275 conn(0x5653a0005c40 msgr2=0x5653a000f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 shutdown_connections 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 1 -- 172.21.15.16:0/416683275 wait complete. 2024-09-12T03:00:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.482+0000 7faec403d740 20 client.27808 trim_cache size 0 max 16384 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "133348c8-a0e2-41a8-98a4-c0cabbdeb602": { 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "133348c8-a0e2-41a8-98a4-c0cabbdeb602", 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:00:38.490 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:00:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 msgr2=0x7faea80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 0x7faea80a5040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faeb4010ca0 tx=0x7faeb401d000 comp rx=0 tx=0).stop 2024-09-12T03:00:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 msgr2=0x7faebc1973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:00:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7faeac006010 tx=0x7faeac003c10 comp rx=0 tx=0).stop 2024-09-12T03:00:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 shutdown_connections 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faebc102ef0 0x7faebc19bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faebc102570 0x7faebc197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faea80a2b80 0x7faea80a5040 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 --2- 172.21.15.16:0/593506590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faebc100e20 0x7faebc1973c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 >> 172.21.15.16:0/593506590 conn(0x7faebc0fc7d0 msgr2=0x7faebc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 shutdown_connections 2024-09-12T03:00:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:00:38.483+0000 7faec1c64640 1 -- 172.21.15.16:0/593506590 wait complete. 2024-09-12T03:00:38.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: pgmap v718: 97 pgs: 97 active+clean; 268 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.7 MiB/s wr, 228 op/s 2024-09-12T03:00:38.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:00:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:00:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:00:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:38 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:00:38.744 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1277533 9.87 MB/sec execute 580 sec latency 63.176 ms 2024-09-12T03:00:39.380 INFO:tasks.cephfs.filesystem:scrub status for tag:133348c8-a0e2-41a8-98a4-c0cabbdeb602 - {'path': '/', 'tag': '133348c8-a0e2-41a8-98a4-c0cabbdeb602', 'options': 'recursive,force'} 2024-09-12T03:00:39.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1279731 9.88 MB/sec execute 581 sec latency 11.886 ms 2024-09-12T03:00:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:40 smithi184 ceph-mon[35321]: pgmap v719: 97 pgs: 97 active+clean; 269 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-09-12T03:00:40.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:40 smithi016 ceph-mon[30925]: pgmap v719: 97 pgs: 97 active+clean; 269 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-09-12T03:00:40.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:40 smithi033 ceph-mon[35092]: pgmap v719: 97 pgs: 97 active+clean; 269 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-09-12T03:00:40.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1282482 9.89 MB/sec execute 582 sec latency 15.686 ms 2024-09-12T03:00:41.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1285390 9.90 MB/sec execute 583 sec latency 11.838 ms 2024-09-12T03:00:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:42 smithi184 ceph-mon[35321]: pgmap v720: 97 pgs: 97 active+clean; 270 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-12T03:00:42.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:42 smithi016 ceph-mon[30925]: pgmap v720: 97 pgs: 97 active+clean; 270 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-12T03:00:42.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:42 smithi033 ceph-mon[35092]: pgmap v720: 97 pgs: 97 active+clean; 270 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-12T03:00:42.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1287437 9.90 MB/sec execute 584 sec latency 13.235 ms 2024-09-12T03:00:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:00:43.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:00:43.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:00:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:00:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:00:43.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:00:43.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1289836 9.90 MB/sec execute 585 sec latency 12.657 ms 2024-09-12T03:00:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:44 smithi184 ceph-mon[35321]: pgmap v721: 97 pgs: 97 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:44.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:44 smithi016 ceph-mon[30925]: pgmap v721: 97 pgs: 97 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:44.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:44 smithi033 ceph-mon[35092]: pgmap v721: 97 pgs: 97 active+clean; 273 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:44.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1292762 9.92 MB/sec execute 586 sec latency 18.117 ms 2024-09-12T03:00:45.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1294903 9.91 MB/sec execute 587 sec latency 90.635 ms 2024-09-12T03:00:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:46 smithi184 ceph-mon[35321]: pgmap v722: 97 pgs: 97 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:46.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:46 smithi016 ceph-mon[30925]: pgmap v722: 97 pgs: 97 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:46.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:46 smithi033 ceph-mon[35092]: pgmap v722: 97 pgs: 97 active+clean; 274 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.3 MiB/s wr, 182 op/s 2024-09-12T03:00:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1297118 9.92 MB/sec execute 588 sec latency 17.221 ms 2024-09-12T03:00:47.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1300104 9.93 MB/sec execute 589 sec latency 11.009 ms 2024-09-12T03:00:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:48 smithi184 ceph-mon[35321]: pgmap v723: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-09-12T03:00:48.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:48 smithi016 ceph-mon[30925]: pgmap v723: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-09-12T03:00:48.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:48 smithi033 ceph-mon[35092]: pgmap v723: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-09-12T03:00:48.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1302789 9.94 MB/sec execute 590 sec latency 11.838 ms 2024-09-12T03:00:49.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1304812 9.94 MB/sec execute 591 sec latency 11.995 ms 2024-09-12T03:00:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:50 smithi184 ceph-mon[35321]: pgmap v724: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-12T03:00:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:50 smithi016 ceph-mon[30925]: pgmap v724: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-12T03:00:50.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:50 smithi033 ceph-mon[35092]: pgmap v724: 97 pgs: 97 active+clean; 278 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-12T03:00:50.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1307574 9.95 MB/sec execute 592 sec latency 54.332 ms 2024-09-12T03:00:51.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1310591 9.96 MB/sec execute 593 sec latency 97.988 ms 2024-09-12T03:00:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:52 smithi184 ceph-mon[35321]: pgmap v725: 97 pgs: 97 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-09-12T03:00:52.298 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:52 smithi033 ceph-mon[35092]: pgmap v725: 97 pgs: 97 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-09-12T03:00:52.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:52 smithi016 ceph-mon[30925]: pgmap v725: 97 pgs: 97 active+clean; 282 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 253 op/s 2024-09-12T03:00:52.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1313375 9.97 MB/sec execute 594 sec latency 15.191 ms 2024-09-12T03:00:53.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1315529 9.97 MB/sec execute 595 sec latency 11.109 ms 2024-09-12T03:00:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:54 smithi184 ceph-mon[35321]: pgmap v726: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 249 op/s 2024-09-12T03:00:54.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:54 smithi016 ceph-mon[30925]: pgmap v726: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 249 op/s 2024-09-12T03:00:54.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:54 smithi033 ceph-mon[35092]: pgmap v726: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.4 MiB/s wr, 249 op/s 2024-09-12T03:00:54.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1318269 9.98 MB/sec execute 596 sec latency 12.723 ms 2024-09-12T03:00:55.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1321100 9.99 MB/sec execute 597 sec latency 35.123 ms 2024-09-12T03:00:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:56 smithi184 ceph-mon[35321]: pgmap v727: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-09-12T03:00:56.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:56 smithi016 ceph-mon[30925]: pgmap v727: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-09-12T03:00:56.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:56 smithi033 ceph-mon[35092]: pgmap v727: 97 pgs: 97 active+clean; 284 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-09-12T03:00:56.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1322320 9.98 MB/sec execute 598 sec latency 132.179 ms 2024-09-12T03:00:57.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1325040 9.99 MB/sec execute 599 sec latency 12.829 ms 2024-09-12T03:00:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:00:58 smithi184 ceph-mon[35321]: pgmap v728: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 305 op/s 2024-09-12T03:00:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:00:58 smithi016 ceph-mon[30925]: pgmap v728: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 305 op/s 2024-09-12T03:00:58.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:00:58 smithi033 ceph-mon[35092]: pgmap v728: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 305 op/s 2024-09-12T03:00:58.745 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 600 sec 2024-09-12T03:00:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: 0 cleanup 600 sec 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Operation Count AvgLat MaxLat 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: ---------------------------------------- 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: NTCreateX 191221 1.303 25144.663 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Close 140476 0.002 0.282 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Rename 8097 10.704 20828.874 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Unlink 38598 2.197 23614.047 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Deltree 4 72.887 98.685 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Mkdir 2 0.463 0.922 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Qpathinfo 173379 0.585 203.752 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Qfileinfo 30370 0.001 0.052 2024-09-12T03:00:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: Qfsinfo 31764 0.176 37.791 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: Sfileinfo 15574 1.348 20688.267 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: Find 67005 0.070 40.981 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: WriteX 95205 0.014 6.460 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: ReadX 299813 0.017 29.257 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: LockX 624 0.484 4.257 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: UnlockX 624 0.474 1.419 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: Flush 13396 2.728 90.516 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T03:00:58.900 INFO:tasks.workunit.client.0.smithi016.stdout:Throughput 9.99027 MB/sec 1 clients 1 procs max_latency=25144.668 ms 2024-09-12T03:00:58.901 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 10 2024-09-12T03:00:58.901 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-12T03:00:58.901 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T03:00:58.901 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-12T03:00:58.901 INFO:tasks.workunit.client.0.smithi016.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-12T03:00:59.000 INFO:tasks.workunit.client.0.smithi016.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-12T03:00:59.001 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-09-12T03:01:00.000 INFO:tasks.workunit.client.0.smithi016.stdout: 10 391 139.03 MB/sec warmup 1 sec latency 46.373 ms 2024-09-12T03:01:00.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:00 smithi016 ceph-mon[30925]: pgmap v729: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 244 op/s 2024-09-12T03:01:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:00 smithi033 ceph-mon[35092]: pgmap v729: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 244 op/s 2024-09-12T03:01:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:00 smithi184 ceph-mon[35321]: pgmap v729: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 244 op/s 2024-09-12T03:01:01.000 INFO:tasks.workunit.client.0.smithi016.stdout: 10 832 107.66 MB/sec warmup 2 sec latency 51.414 ms 2024-09-12T03:01:02.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 1287 78.94 MB/sec warmup 3 sec latency 154.098 ms 2024-09-12T03:01:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:02 smithi016 ceph-mon[30925]: pgmap v730: 97 pgs: 97 active+clean; 268 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 317 op/s 2024-09-12T03:01:02.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:02 smithi033 ceph-mon[35092]: pgmap v730: 97 pgs: 97 active+clean; 268 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 317 op/s 2024-09-12T03:01:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:02 smithi184 ceph-mon[35321]: pgmap v730: 97 pgs: 97 active+clean; 268 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 317 op/s 2024-09-12T03:01:03.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2149 67.46 MB/sec warmup 4 sec latency 53.992 ms 2024-09-12T03:01:03.917 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T03:01:03.917 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-09-12T03:01:03.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:01:04.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2782 63.80 MB/sec warmup 5 sec latency 37.422 ms 2024-09-12T03:01:04.215 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:04.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:04 smithi016 ceph-mon[30925]: pgmap v731: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 272 op/s 2024-09-12T03:01:04.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:04 smithi033 ceph-mon[35092]: pgmap v731: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 272 op/s 2024-09-12T03:01:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:04 smithi184 ceph-mon[35321]: pgmap v731: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 272 op/s 2024-09-12T03:01:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 -- 172.21.15.16:0/1854378638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 msgr2=0x7fa758107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1854378638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758107bc0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009930 tx=0x7fa74c02a420 comp rx=0 tx=0).stop 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 -- 172.21.15.16:0/1854378638 shutdown_connections 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1854378638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1854378638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758107bc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.875+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1854378638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.876+0000 7fa75cf00640 1 -- 172.21.15.16:0/1854378638 >> 172.21.15.16:0/1854378638 conn(0x7fa7580fc7d0 msgr2=0x7fa7580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.876+0000 7fa75cf00640 1 -- 172.21.15.16:0/1854378638 shutdown_connections 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.876+0000 7fa75cf00640 1 -- 172.21.15.16:0/1854378638 wait complete. 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 Processor -- start 2024-09-12T03:01:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 -- start start 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa758112050 con 0x7fa758103780 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa758111ed0 con 0x7fa758102dd0 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75cf00640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7581121d0 con 0x7fa758108300 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49474/0 (socket says 172.21.15.16:49474) 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 -- 172.21.15.16:0/1463560938 learned_addr learned my addr 172.21.15.16:0/1463560938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75659c640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa756d9d640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 -- 172.21.15.16:0/1463560938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 msgr2=0x7fa75819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 -- 172.21.15.16:0/1463560938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 msgr2=0x7fa7581985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa755d9b640 1 -- 172.21.15.16:0/1463560938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75819d580 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa75659c640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.877+0000 7fa756d9d640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa755d9b640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005940 tx=0x7fa74c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c03f030 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75819d810 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa758100960 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c003ce0 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.878+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c049060 con 0x7fa758103780 2024-09-12T03:01:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.879+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa738005290 con 0x7fa758103780 2024-09-12T03:01:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.881+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa74c042030 con 0x7fa758103780 2024-09-12T03:01:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.882+0000 7fa7477fe640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 0x7fa7400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.883+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa74c0ef540 con 0x7fa758103780 2024-09-12T03:01:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.883+0000 7fa75659c640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 0x7fa7400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.883+0000 7fa75659c640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 0x7fa7400a50c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa748009690 tx=0x7fa74800f7a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:04.886+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa74c0bc4b0 con 0x7fa758103780 2024-09-12T03:01:05.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3525 63.07 MB/sec warmup 6 sec latency 201.025 ms 2024-09-12T03:01:05.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.126+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa738005580 con 0x7fa758103780 2024-09-12T03:01:05.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.127+0000 7fa7477fe640 1 -- 172.21.15.16:0/1463560938 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11976 (secure 0 0 0) 0x7fa74c0bbdb0 con 0x7fa758103780 2024-09-12T03:01:05.127 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:01:05.127 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"btime":"2024-09-12T03:01:04:048707+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:03.760889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:01:05.127 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 msgr2=0x7fa7400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 0x7fa7400a50c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa748009690 tx=0x7fa74800f7a0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 msgr2=0x7fa758198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005940 tx=0x7fa74c0042f0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 shutdown_connections 2024-09-12T03:01:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa758108300 0x7fa75819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa7400a2c00 0x7fa7400a50c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa758103780 0x7fa758198b10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 --2- 172.21.15.16:0/1463560938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa758102dd0 0x7fa7581985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 >> 172.21.15.16:0/1463560938 conn(0x7fa7580fc7d0 msgr2=0x7fa758105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 shutdown_connections 2024-09-12T03:01:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:05.129+0000 7fa75cf00640 1 -- 172.21.15.16:0/1463560938 wait complete. 2024-09-12T03:01:05.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:05 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:05 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:05 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:06.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4003 57.39 MB/sec warmup 7 sec latency 1152.913 ms 2024-09-12T03:01:06.035 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-09-12T03:01:06.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-09-12T03:01:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:06 smithi016 ceph-mon[30925]: pgmap v732: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T03:01:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:06 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1463560938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:06.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:06 smithi033 ceph-mon[35092]: pgmap v732: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T03:01:06.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:06 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1463560938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:06.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:06 smithi184 ceph-mon[35321]: pgmap v732: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-09-12T03:01:06.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:06 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1463560938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 -- 172.21.15.16:0/4073837327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 msgr2=0x7f8e2010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 --2- 172.21.15.16:0/4073837327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2010d950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009be0 tx=0x7f8e1802a6c0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 -- 172.21.15.16:0/4073837327 shutdown_connections 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 --2- 172.21.15.16:0/4073837327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2010d950 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 --2- 172.21.15.16:0/4073837327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e20100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 --2- 172.21.15.16:0/4073837327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e20106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 -- 172.21.15.16:0/4073837327 >> 172.21.15.16:0/4073837327 conn(0x7f8e200fc7a0 msgr2=0x7f8e200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 -- 172.21.15.16:0/4073837327 shutdown_connections 2024-09-12T03:01:06.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 -- 172.21.15.16:0/4073837327 wait complete. 2024-09-12T03:01:06.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.894+0000 7f8e27aae640 1 Processor -- start 2024-09-12T03:01:06.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 -- start start 2024-09-12T03:01:06.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e2019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e2019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e20110e30 con 0x7f8e20106a80 2024-09-12T03:01:06.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e25049640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e25049640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39240/0 (socket says 172.21.15.16:39240) 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e2019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e25049640 1 -- 172.21.15.16:0/1559780495 learned_addr learned my addr 172.21.15.16:0/1559780495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e2019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e20110cb0 con 0x7f8e20101420 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.895+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e20110fb0 con 0x7f8e20100a80 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 -- 172.21.15.16:0/1559780495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 msgr2=0x7f8e2019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e2019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 -- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 msgr2=0x7f8e2019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e2019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 -- 172.21.15.16:0/1559780495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2019f540 con 0x7f8e20101420 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e2019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:06.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e2584a640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e2019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e25049640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e9f0 tx=0x7f8e1000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e16ffd640 1 -- 172.21.15.16:0/1559780495 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000cd80 con 0x7f8e20101420 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e16ffd640 1 -- 172.21.15.16:0/1559780495 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e10010910 con 0x7f8e20101420 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e16ffd640 1 -- 172.21.15.16:0/1559780495 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10018990 con 0x7f8e20101420 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.896+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2019f830 con 0x7f8e20101420 2024-09-12T03:01:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.897+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e201a7110 con 0x7f8e20101420 2024-09-12T03:01:06.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.900+0000 7f8e16ffd640 1 -- 172.21.15.16:0/1559780495 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8e10018b30 con 0x7f8e20101420 2024-09-12T03:01:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.900+0000 7f8e29e87740 1 Processor -- start 2024-09-12T03:01:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 -- start start 2024-09-12T03:01:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698bbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603698142a0 con 0x5603698af080 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5603697bbb10 con 0x5603698ae760 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e29e87740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560369810260 con 0x5603698b9940 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50566/0 (socket says 172.21.15.16:50566) 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e2584a640 1 -- 172.21.15.16:0/501424516 learned_addr learned my addr 172.21.15.16:0/501424516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e2604b640 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.901+0000 7f8e25049640 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698bbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 -- 172.21.15.16:0/501424516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 msgr2=0x5603698bbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698bbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 -- 172.21.15.16:0/501424516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 msgr2=0x5603698aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 -- 172.21.15.16:0/501424516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560369585b90 con 0x5603698af080 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e16ffd640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 0x7f8e0c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 0x7f8e0c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e2584a640 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c005ab0 tx=0x7f8e1c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4291e351e57fb4f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.902+0000 7f8e16ffd640 1 -- 172.21.15.16:0/1559780495 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e100cf280 con 0x7f8e20101420 2024-09-12T03:01:06.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e0bfff640 1 -- 172.21.15.16:0/501424516 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c005020 con 0x5603698af080 2024-09-12T03:01:06.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e0bfff640 1 -- 172.21.15.16:0/501424516 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e1c0047a0 con 0x5603698af080 2024-09-12T03:01:06.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e0bfff640 1 -- 172.21.15.16:0/501424516 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0116a0 con 0x5603698af080 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e29e87740 1 -- 172.21.15.16:0/501424516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 msgr2=0x5603698b9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e29e87740 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c005ab0 tx=0x7f8e1c00f730 comp rx=0 tx=0).stop 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.903+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 0x7f8e0c0a50c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005cc0 tx=0x7f8e180303a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 -- 172.21.15.16:0/501424516 shutdown_connections 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698bbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698b9400 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 --2- 172.21.15.16:0/501424516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 -- 172.21.15.16:0/501424516 >> 172.21.15.16:0/501424516 conn(0x5603698aacf0 msgr2=0x5603698ab0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 -- 172.21.15.16:0/501424516 shutdown_connections 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 -- 172.21.15.16:0/501424516 wait complete. 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 -- start start 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.904+0000 7f8e29e87740 1 Processor -- start 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698ae760 0x5603698b6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 0x5603698b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698b7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5603698bc510 con 0x5603698ae760 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56036979c6e0 con 0x5603698af080 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5603698bc690 con 0x5603698b9940 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e29e87740 10 client.0 _ll_register_callbacks cb 0x5603697c0a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e25049640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 0x5603698b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698ae760 0x5603698b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e25049640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 0x5603698b6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39254/0 (socket says 172.21.15.16:39254) 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e25049640 1 -- 172.21.15.16:0/1347088165 learned_addr learned my addr 172.21.15.16:0/1347088165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.905+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 -- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 msgr2=0x5603698b6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 0x5603698b6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 -- 172.21.15.16:0/1347088165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698ae760 msgr2=0x5603698b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698ae760 0x5603698b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 -- 172.21.15.16:0/1347088165 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603698c5a10 con 0x5603698b9940 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e2604b640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698b7200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8e1802abe0 tx=0x7f8e1802a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83127a6357fa5b9b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e097fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e18044070 con 0x5603698b9940 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e18004040 con 0x5603698b9940 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e1c0047a0 con 0x5603698b9940 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.906+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e18036040 con 0x5603698b9940 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 10 client.27823 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5603698c8370 con 0x5603698b9940 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e1c0047a0 con 0x5603698b9940 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f8e1803c040 con 0x5603698b9940 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 10 client.27823 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 10 client.27823 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 20 client.27823 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5603698c9880 0x5603698cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 4 client.27823 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603698219d0 con 0x5603698c9880 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.907+0000 7f8e2584a640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5603698c9880 0x5603698cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.908+0000 7f8e097fa640 10 client.27823 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.908+0000 7f8e2584a640 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5603698c9880 0x5603698cbd20 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2172e8f7789f9e9 server_cookie=77dbb6f014217490 in_seq=0 out_seq=0 2024-09-12T03:01:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.909+0000 7f8e097fa640 1 -- 172.21.15.16:0/1347088165 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5603698219d0 con 0x5603698c9880 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.909+0000 7f8e097fa640 10 client.27823 handle_command_reply: tid=0 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.909+0000 7f8e29e87740 1 client.27823 shutdown 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 msgr2=0x5603698b7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698b7200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8e1802abe0 tx=0x7f8e1802a6c0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 shutdown_connections 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5603698c9880 0x5603698cbd20 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603698b7200 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698af080 0x5603698b6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 --2- 172.21.15.16:0/1347088165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698ae760 0x5603698b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 >> 172.21.15.16:0/1347088165 conn(0x5603698aacf0 msgr2=0x5603698b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e097fa640 0 client.27823 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:06.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 shutdown_connections 2024-09-12T03:01:06.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 1 -- 172.21.15.16:0/1347088165 wait complete. 2024-09-12T03:01:06.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.910+0000 7f8e29e87740 20 client.27823 trim_cache size 0 max 16384 2024-09-12T03:01:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 Processor -- start 2024-09-12T03:01:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 -- start start 2024-09-12T03:01:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698af080 0x5603698d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 0x5603698d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0400e400 con 0x5603698b9940 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e04010000 con 0x5603698ae760 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e29e87740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0400ead0 con 0x5603698af080 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e2604b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 0x5603698d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39268/0 (socket says 172.21.15.16:39268) 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e2584a640 1 -- 172.21.15.16:0/3020025915 learned_addr learned my addr 172.21.15.16:0/3020025915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.932+0000 7f8e25049640 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698af080 0x5603698d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 -- 172.21.15.16:0/3020025915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698af080 msgr2=0x5603698d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698af080 0x5603698d15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 -- 172.21.15.16:0/3020025915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 msgr2=0x5603698b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 -- 172.21.15.16:0/3020025915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603698aab20 con 0x5603698b9940 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2584a640 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e2604b640 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 0x5603698d1b30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009b30 tx=0x7f8e18031440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c4f8ba023a7d7a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e0b7fe640 1 -- 172.21.15.16:0/3020025915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e18044070 con 0x5603698b9940 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e0b7fe640 1 -- 172.21.15.16:0/3020025915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e18050070 con 0x5603698b9940 2024-09-12T03:01:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.933+0000 7f8e0b7fe640 1 -- 172.21.15.16:0/3020025915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1804b470 con 0x5603698b9940 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 -- 172.21.15.16:0/3020025915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 msgr2=0x5603698d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 0x5603698d1b30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009b30 tx=0x7f8e18031440 comp rx=0 tx=0).stop 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 -- 172.21.15.16:0/3020025915 shutdown_connections 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698b9940 0x5603698d1b30 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698af080 0x5603698d15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 --2- 172.21.15.16:0/3020025915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 -- 172.21.15.16:0/3020025915 >> 172.21.15.16:0/3020025915 conn(0x5603698aacf0 msgr2=0x5603698a9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:06.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 -- 172.21.15.16:0/3020025915 shutdown_connections 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.934+0000 7f8e29e87740 1 -- 172.21.15.16:0/3020025915 wait complete. 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 -- start start 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 Processor -- start 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698be3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603699335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0400f930 con 0x5603698af080 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0400f260 con 0x5603698ae760 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.935+0000 7f8e29e87740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0400c3e0 con 0x5603698b9940 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e29e87740 10 client.0 _ll_register_callbacks cb 0x7f8e18009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:06.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39270/0 (socket says 172.21.15.16:39270) 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 -- 172.21.15.16:0/444196680 learned_addr learned my addr 172.21.15.16:0/444196680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2604b640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603699335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e25049640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698be3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 -- 172.21.15.16:0/444196680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 msgr2=0x5603699335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603699335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 -- 172.21.15.16:0/444196680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 msgr2=0x5603698be3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698be3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 -- 172.21.15.16:0/444196680 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603698a6260 con 0x5603698ae760 2024-09-12T03:01:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2604b640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603699335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e2584a640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c019000 tx=0x7f8e1c00ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff2afb32c106948 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.936+0000 7f8e097fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c011640 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8e1c0117e0 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e18050070 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e1c037050 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e29e87740 10 client.27935 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.937+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560369934ce0 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e18050070 con 0x5603698ae760 2024-09-12T03:01:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f8e1c031990 con 0x5603698ae760 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 10 client.27935 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 10 client.27935 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 20 client.27935 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560369935360 0x560369937780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 4 client.27935 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x56036981bd20 con 0x560369935360 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e25049640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560369935360 0x560369937780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e097fa640 10 client.27935 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:06.938+0000 7f8e25049640 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560369935360 0x560369937780 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8716af65a87d923 server_cookie=a1348bec7ccce8d in_seq=0 out_seq=0 2024-09-12T03:01:07.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4459 52.33 MB/sec warmup 8 sec latency 1392.553 ms 2024-09-12T03:01:07.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:01:07.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:01:07.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:01:08.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 46.51 MB/sec warmup 9 sec latency 1001.350 ms 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e097fa640 1 -- 172.21.15.16:0/444196680 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56036981bd20 con 0x560369935360 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e097fa640 10 client.27935 handle_command_reply: tid=0 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 client.27935 shutdown 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 msgr2=0x5603698bde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c019000 tx=0x7f8e1c00ff60 comp rx=0 tx=0).stop 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 shutdown_connections 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560369935360 0x560369937780 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5603698b9940 0x5603699335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5603698af080 0x5603698be3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 --2- 172.21.15.16:0/444196680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5603698ae760 0x5603698bde80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e097fa640 0 client.27935 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 >> 172.21.15.16:0/444196680 conn(0x5603698aacf0 msgr2=0x5603698b7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 shutdown_connections 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 1 -- 172.21.15.16:0/444196680 wait complete. 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.082+0000 7f8e29e87740 20 client.27935 trim_cache size 0 max 16384 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 33, 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:01:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 50, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "8c2a116": { 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 50, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:01:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:08.086 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 msgr2=0x7f8e0c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 0x7f8e0c0a50c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005cc0 tx=0x7f8e180303a0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 msgr2=0x7f8e2019aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e9f0 tx=0x7f8e1000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 shutdown_connections 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8e0c0a2c00 0x7f8e0c0a50c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e20106a80 0x7f8e2019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e20101420 0x7f8e2019aad0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 --2- 172.21.15.16:0/1559780495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e20100a80 0x7f8e2019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 >> 172.21.15.16:0/1559780495 conn(0x7f8e200fc7a0 msgr2=0x7f8e2006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 shutdown_connections 2024-09-12T03:01:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:08.083+0000 7f8e27aae640 1 -- 172.21.15.16:0/1559780495 wait complete. 2024-09-12T03:01:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:08 smithi016 ceph-mon[30925]: pgmap v733: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 341 op/s 2024-09-12T03:01:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:01:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:01:08.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:01:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:08 smithi033 ceph-mon[35092]: pgmap v733: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 341 op/s 2024-09-12T03:01:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:01:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:01:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:01:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:08 smithi184 ceph-mon[35321]: pgmap v733: 97 pgs: 97 active+clean; 269 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 341 op/s 2024-09-12T03:01:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:01:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:01:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:01:08.984 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8c2a116', quiesce took 5.1 seconds. Will release after: 24.9 2024-09-12T03:01:09.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 41.86 MB/sec warmup 10 sec latency 2001.430 ms 2024-09-12T03:01:09.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:01:09.651 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:10.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 38.06 MB/sec warmup 11 sec latency 3001.528 ms 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 -- 172.21.15.16:0/1172240378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 msgr2=0x7ff03c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 --2- 172.21.15.16:0/1172240378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c103120 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff034009ca0 tx=0x7ff03402a950 comp rx=0 tx=0).stop 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 -- 172.21.15.16:0/1172240378 shutdown_connections 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 --2- 172.21.15.16:0/1172240378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c10c170 0x7ff03c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 --2- 172.21.15.16:0/1172240378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c103660 0x7ff03c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.112+0000 7ff043d8b640 1 --2- 172.21.15.16:0/1172240378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c103120 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- 172.21.15.16:0/1172240378 >> 172.21.15.16:0/1172240378 conn(0x7ff03c0fc7a0 msgr2=0x7ff03c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- 172.21.15.16:0/1172240378 shutdown_connections 2024-09-12T03:01:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- 172.21.15.16:0/1172240378 wait complete. 2024-09-12T03:01:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 Processor -- start 2024-09-12T03:01:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- start start 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c103660 0x7ff03c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c10c170 0x7ff03c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c111f70 con 0x7ff03c102d40 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.113+0000 7ff043d8b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c111df0 con 0x7ff03c10c170 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff043d8b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1120f0 con 0x7ff03c103660 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50624/0 (socket says 172.21.15.16:50624) 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 -- 172.21.15.16:0/3716630475 learned_addr learned my addr 172.21.15.16:0/3716630475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041326640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c103660 0x7ff03c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff042328640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c10c170 0x7ff03c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 -- 172.21.15.16:0/3716630475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c103660 msgr2=0x7ff03c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c103660 0x7ff03c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 -- 172.21.15.16:0/3716630475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c10c170 msgr2=0x7ff03c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c10c170 0x7ff03c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 -- 172.21.15.16:0/3716630475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c19d3e0 con 0x7ff03c102d40 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff041b27640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff034005cc0 tx=0x7ff03402f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff03affd640 1 -- 172.21.15.16:0/3716630475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034034040 con 0x7ff03c102d40 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff03affd640 1 -- 172.21.15.16:0/3716630475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff034030920 con 0x7ff03c102d40 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c19d670 con 0x7ff03c102d40 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.114+0000 7ff03affd640 1 -- 172.21.15.16:0/3716630475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034004830 con 0x7ff03c102d40 2024-09-12T03:01:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.115+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c1a4fb0 con 0x7ff03c102d40 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff03affd640 1 -- 172.21.15.16:0/3716630475 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff034037020 con 0x7ff03c102d40 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 Processor -- start 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 -- start start 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b14550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b14a90 0x563050b16e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563050a6b2e0 con 0x563050b05380 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563050a12ce0 con 0x563050b05ca0 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.117+0000 7ff046164740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563050a672a0 con 0x563050b14a90 2024-09-12T03:01:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50626/0 (socket says 172.21.15.16:50626) 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 -- 172.21.15.16:0/1426330313 learned_addr learned my addr 172.21.15.16:0/1426330313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff041b27640 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b14550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff03affd640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 0x7ff0240a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff041326640 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b14a90 0x563050b16e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 0x7ff0240a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 -- 172.21.15.16:0/1426330313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b14a90 msgr2=0x563050b16e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b14a90 0x563050b16e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 -- 172.21.15.16:0/1426330313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 msgr2=0x563050b14550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b14550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 -- 172.21.15.16:0/1426330313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630509f7440 con 0x563050b05380 2024-09-12T03:01:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff03affd640 1 -- 172.21.15.16:0/3716630475 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff0340efff0 con 0x7ff03c102d40 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff042328640 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff02c012620 tx=0x7ff02c012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa038f07773aea40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.118+0000 7ff023fff640 1 -- 172.21.15.16:0/1426330313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c00cc30 con 0x563050b05380 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff023fff640 1 -- 172.21.15.16:0/1426330313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02c010d80 con 0x563050b05380 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff023fff640 1 -- 172.21.15.16:0/1426330313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c004dc0 con 0x563050b05380 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff042328640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 0x7ff0240a5040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01f000 tx=0x7ff02c01d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 -- 172.21.15.16:0/1426330313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 msgr2=0x563050b05760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff02c012620 tx=0x7ff02c012ae0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 -- 172.21.15.16:0/1426330313 shutdown_connections 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b14a90 0x563050b16e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b14550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 --2- 172.21.15.16:0/1426330313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05380 0x563050b05760 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 -- 172.21.15.16:0/1426330313 >> 172.21.15.16:0/1426330313 conn(0x563050b01a30 msgr2=0x563050b01e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 -- 172.21.15.16:0/1426330313 shutdown_connections 2024-09-12T03:01:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.119+0000 7ff046164740 1 -- 172.21.15.16:0/1426330313 wait complete. 2024-09-12T03:01:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 -- start start 2024-09-12T03:01:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 Processor -- start 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 0x563050b11750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563050b17660 con 0x563050b14a90 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563050a26050 con 0x563050b05380 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563050b177e0 con 0x563050b05ca0 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff046164740 10 client.0 _ll_register_callbacks cb 0x563050a66160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.120+0000 7ff041b27640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39308/0 (socket says 172.21.15.16:39308) 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 -- 172.21.15.16:0/357711208 learned_addr learned my addr 172.21.15.16:0/357711208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50634/0 (socket says 172.21.15.16:50634) 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041326640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 0x563050b11750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 -- 172.21.15.16:0/357711208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 msgr2=0x563050b11750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 0x563050b11750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 -- 172.21.15.16:0/357711208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 msgr2=0x563050b11d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 -- 172.21.15.16:0/357711208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630507dc3c0 con 0x563050b05380 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041326640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 0x563050b11750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff042328640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff041b27640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b8cc0 tx=0x7ff0340bc3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d22decde4188325 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0340bcf60 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff02c010d80 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff034004040 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff0340bc860 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff046164740 10 client.27947 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.121+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563050b17660 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02c010d80 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7ff03404d030 con 0x563050b05380 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 10 client.27947 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 10 client.27947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 20 client.27947 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b245e0 0x563050b269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 4 client.27947 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563050a78850 con 0x563050b245e0 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.122+0000 7ff041326640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b245e0 0x563050b269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff0217fa640 10 client.27947 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff041326640 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b245e0 0x563050b269f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81289ddf1e890f7d server_cookie=a75ee99df2fe1a8b in_seq=0 out_seq=0 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff0217fa640 1 -- 172.21.15.16:0/357711208 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563050a78850 con 0x563050b245e0 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff0217fa640 10 client.27947 handle_command_reply: tid=0 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 client.27947 shutdown 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 msgr2=0x563050b144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b8cc0 tx=0x7ff0340bc3a0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 shutdown_connections 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b245e0 0x563050b269f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b11d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05ca0 0x563050b11750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 --2- 172.21.15.16:0/357711208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05380 0x563050b144a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff0217fa640 0 client.27947 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.123+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 >> 172.21.15.16:0/357711208 conn(0x563050b01a30 msgr2=0x563050b07790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.124+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 shutdown_connections 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.124+0000 7ff046164740 1 -- 172.21.15.16:0/357711208 wait complete. 2024-09-12T03:01:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.124+0000 7ff046164740 20 client.27947 trim_cache size 0 max 16384 2024-09-12T03:01:10.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.141+0000 7ff046164740 1 Processor -- start 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.141+0000 7ff046164740 1 -- start start 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b20900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b2c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1ab2d0 con 0x563050b14a90 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1aced0 con 0x563050b05ca0 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff046164740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1ab9a0 con 0x563050b05380 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50644/0 (socket says 172.21.15.16:50644) 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 -- 172.21.15.16:0/762026780 learned_addr learned my addr 172.21.15.16:0/762026780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff041326640 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b2c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff041b27640 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b20900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 -- 172.21.15.16:0/762026780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 msgr2=0x563050b20900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b20900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 -- 172.21.15.16:0/762026780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 msgr2=0x563050b2c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b2c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 -- 172.21.15.16:0/762026780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563050b120d0 con 0x563050b14a90 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff042328640 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00c730 tx=0x7ff02c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc22e1083b88ce server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.142+0000 7ff0237fe640 1 -- 172.21.15.16:0/762026780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0049b0 con 0x563050b14a90 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff0237fe640 1 -- 172.21.15.16:0/762026780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02c037070 con 0x563050b14a90 2024-09-12T03:01:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff0237fe640 1 -- 172.21.15.16:0/762026780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0052b0 con 0x563050b14a90 2024-09-12T03:01:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 -- 172.21.15.16:0/762026780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 msgr2=0x563050b2c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00c730 tx=0x7ff02c00c760 comp rx=0 tx=0).stop 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 -- 172.21.15.16:0/762026780 shutdown_connections 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b14a90 0x563050b2c810 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b05ca0 0x563050b2c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 --2- 172.21.15.16:0/762026780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b20900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 -- 172.21.15.16:0/762026780 >> 172.21.15.16:0/762026780 conn(0x563050b01a30 msgr2=0x563050b0cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 -- 172.21.15.16:0/762026780 shutdown_connections 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.143+0000 7ff046164740 1 -- 172.21.15.16:0/762026780 wait complete. 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 -- start start 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 Processor -- start 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b18700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b14a90 0x563050b19180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1ac800 con 0x563050b05ca0 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1ac130 con 0x563050b14a90 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1a92b0 con 0x563050b05380 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff046164740 10 client.0 _ll_register_callbacks cb 0x7ff0340b84c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff041326640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff041326640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50646/0 (socket says 172.21.15.16:50646) 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff041326640 1 -- 172.21.15.16:0/2897291255 learned_addr learned my addr 172.21.15.16:0/2897291255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff042328640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b14a90 0x563050b19180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.144+0000 7ff041b27640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b18700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 -- 172.21.15.16:0/2897291255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 msgr2=0x563050b18700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b18700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 -- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b14a90 msgr2=0x563050b19180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b14a90 0x563050b19180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 -- 172.21.15.16:0/2897291255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563050b8e2a0 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff041326640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x563050b02db0 tx=0x7ff03000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5b93dc65255033 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03001d410 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff03001d5b0 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff02c037070 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff03001d7b0 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff046164740 10 client.18543 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563050b8f530 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.145+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02c037070 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7ff030024030 con 0x563050b05ca0 2024-09-12T03:01:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 10 client.18543 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 10 client.18543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 20 client.18543 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b8fc30 0x563050b92020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 4 client.18543 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563050a72d60 con 0x563050b8fc30 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff041b27640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b8fc30 0x563050b92020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff0217fa640 10 client.18543 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.146+0000 7ff041b27640 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b8fc30 0x563050b92020 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bb6633ad78144ba server_cookie=65b23d3e9ade6020 in_seq=0 out_seq=0 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff0217fa640 1 -- 172.21.15.16:0/2897291255 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563050a72d60 con 0x563050b8fc30 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff0217fa640 10 client.18543 handle_command_reply: tid=0 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 client.18543 shutdown 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 msgr2=0x563050b18c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x563050b02db0 tx=0x7ff03000be10 comp rx=0 tx=0).stop 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 shutdown_connections 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563050b8fc30 0x563050b92020 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563050b14a90 0x563050b19180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563050b05ca0 0x563050b18c40 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff0217fa640 0 client.18543 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 --2- 172.21.15.16:0/2897291255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563050b05380 0x563050b18700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 >> 172.21.15.16:0/2897291255 conn(0x563050b01a30 msgr2=0x563050b06ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 shutdown_connections 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 1 -- 172.21.15.16:0/2897291255 wait complete. 2024-09-12T03:01:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff046164740 20 client.18543 trim_cache size 0 max 16384 2024-09-12T03:01:10.153 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:01:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:01:10.153 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 msgr2=0x7ff0240a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.147+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 0x7ff0240a5040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01f000 tx=0x7ff02c01d000 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 msgr2=0x7ff03c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff034005cc0 tx=0x7ff03402f630 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 shutdown_connections 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff03c10c170 0x7ff03c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff03c103660 0x7ff03c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0240a2b80 0x7ff0240a5040 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 --2- 172.21.15.16:0/3716630475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03c102d40 0x7ff03c198430 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 >> 172.21.15.16:0/3716630475 conn(0x7ff03c0fc7a0 msgr2=0x7ff03c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 shutdown_connections 2024-09-12T03:01:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:10.148+0000 7ff043d8b640 1 -- 172.21.15.16:0/3716630475 wait complete. 2024-09-12T03:01:10.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:10 smithi016 ceph-mon[30925]: pgmap v734: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 268 op/s 2024-09-12T03:01:10.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:10 smithi033 ceph-mon[35092]: pgmap v734: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 268 op/s 2024-09-12T03:01:10.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:10 smithi184 ceph-mon[35321]: pgmap v734: 97 pgs: 97 active+clean; 273 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 268 op/s 2024-09-12T03:01:11.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 34.88 MB/sec warmup 12 sec latency 4001.559 ms 2024-09-12T03:01:11.452 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:01:11.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:01:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:12.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 32.20 MB/sec warmup 13 sec latency 5001.594 ms 2024-09-12T03:01:12.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:12 smithi016 ceph-mon[30925]: pgmap v735: 97 pgs: 97 active+clean; 310 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.6 MiB/s wr, 378 op/s 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 -- 172.21.15.16:0/2888366021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 msgr2=0x7fdb2c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 --2- 172.21.15.16:0/2888366021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c1006d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009930 tx=0x7fdb1c02a420 comp rx=0 tx=0).stop 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 -- 172.21.15.16:0/2888366021 shutdown_connections 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 --2- 172.21.15.16:0/2888366021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c106f50 0x7fdb2c10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 --2- 172.21.15.16:0/2888366021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 --2- 172.21.15.16:0/2888366021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c1006d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 -- 172.21.15.16:0/2888366021 >> 172.21.15.16:0/2888366021 conn(0x7fdb2c077210 msgr2=0x7fdb2c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 -- 172.21.15.16:0/2888366021 shutdown_connections 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.143+0000 7fdb311b1640 1 -- 172.21.15.16:0/2888366021 wait complete. 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 Processor -- start 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 -- start start 2024-09-12T03:01:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c106f50 0x7fdb2c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c112080 con 0x7fdb2c1002f0 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c111f00 con 0x7fdb2c100ca0 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb311b1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c112200 con 0x7fdb2c106f50 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50678/0 (socket says 172.21.15.16:50678) 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.144+0000 7fdb2ad9d640 1 -- 172.21.15.16:0/3539214419 learned_addr learned my addr 172.21.15.16:0/3539214419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2b59e640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c106f50 0x7fdb2c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2a59c640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 -- 172.21.15.16:0/3539214419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c106f50 msgr2=0x7fdb2c19cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c106f50 0x7fdb2c19cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 -- 172.21.15.16:0/3539214419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 msgr2=0x7fdb2c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 -- 172.21.15.16:0/3539214419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c19d5f0 con 0x7fdb2c1002f0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2a59c640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb2ad9d640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009a50 tx=0x7fdb1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c03f030 con 0x7fdb2c1002f0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb1c003ce0 con 0x7fdb2c1002f0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c19d880 con 0x7fdb2c1002f0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.145+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c035070 con 0x7fdb2c1002f0 2024-09-12T03:01:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.146+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c1a51c0 con 0x7fdb2c1002f0 2024-09-12T03:01:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.147+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c100750 con 0x7fdb2c1002f0 2024-09-12T03:01:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.148+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdb1c042030 con 0x7fdb2c1002f0 2024-09-12T03:01:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.148+0000 7fdb13fff640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 0x7fdb140a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.149+0000 7fdb2a59c640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 0x7fdb140a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.149+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdb1c0ef5f0 con 0x7fdb2c1002f0 2024-09-12T03:01:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.149+0000 7fdb2a59c640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 0x7fdb140a5040 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:12.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.153+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdb1c0bc560 con 0x7fdb2c1002f0 2024-09-12T03:01:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.367+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb2c10a120 con 0x7fdb2c1002f0 2024-09-12T03:01:12.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.368+0000 7fdb13fff640 1 -- 172.21.15.16:0/3539214419 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11976 (secure 0 0 0) 0x7fdb1c0bbe60 con 0x7fdb2c1002f0 2024-09-12T03:01:12.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:01:12.368 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"btime":"2024-09-12T03:01:04:048707+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:03.760889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:01:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-09-12T03:01:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 msgr2=0x7fdb140a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 0x7fdb140a5040 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20008040 comp rx=0 tx=0).stop 2024-09-12T03:01:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 msgr2=0x7fdb2c198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009a50 tx=0x7fdb1c004270 comp rx=0 tx=0).stop 2024-09-12T03:01:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 shutdown_connections 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb2c106f50 0x7fdb2c19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2c100ca0 0x7fdb2c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdb140a2b80 0x7fdb140a5040 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 --2- 172.21.15.16:0/3539214419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1002f0 0x7fdb2c198640 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 >> 172.21.15.16:0/3539214419 conn(0x7fdb2c077210 msgr2=0x7fdb2c105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 shutdown_connections 2024-09-12T03:01:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:12.370+0000 7fdb311b1640 1 -- 172.21.15.16:0/3539214419 wait complete. 2024-09-12T03:01:12.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:12 smithi033 ceph-mon[35092]: pgmap v735: 97 pgs: 97 active+clean; 310 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.6 MiB/s wr, 378 op/s 2024-09-12T03:01:12.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:12 smithi184 ceph-mon[35321]: pgmap v735: 97 pgs: 97 active+clean; 310 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.6 MiB/s wr, 378 op/s 2024-09-12T03:01:13.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 29.90 MB/sec warmup 14 sec latency 6001.694 ms 2024-09-12T03:01:13.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:01:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:13.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:13 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3539214419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:13.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:13 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3539214419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:13 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3539214419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- 172.21.15.16:0/3425585153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe170100a70 msgr2=0x7fe17010ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 --2- 172.21.15.16:0/3425585153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe170100a70 0x7fe17010ea60 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe168009ca0 tx=0x7fe16802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- 172.21.15.16:0/3425585153 shutdown_connections 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 --2- 172.21.15.16:0/3425585153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe170100a70 0x7fe17010ea60 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 --2- 172.21.15.16:0/3425585153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1701000d0 0x7fe170100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 --2- 172.21.15.16:0/3425585153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170106ba0 0x7fe170106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- 172.21.15.16:0/3425585153 >> 172.21.15.16:0/3425585153 conn(0x7fe170077210 msgr2=0x7fe1700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- 172.21.15.16:0/3425585153 shutdown_connections 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- 172.21.15.16:0/3425585153 wait complete. 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 Processor -- start 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.773+0000 7fe176e21640 1 -- start start 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170100a70 0x7fe1701989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 0x7fe17019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe170112090 con 0x7fe1701000d0 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe170111f10 con 0x7fe170106ba0 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe176e21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe170112210 con 0x7fe170100a70 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50702/0 (socket says 172.21.15.16:50702) 2024-09-12T03:01:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 -- 172.21.15.16:0/1412796942 learned_addr learned my addr 172.21.15.16:0/1412796942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe16ffff640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170100a70 0x7fe1701989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 -- 172.21.15.16:0/1412796942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170100a70 msgr2=0x7fe1701989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170100a70 0x7fe1701989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe1753be640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 0x7fe17019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 -- 172.21.15.16:0/1412796942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 msgr2=0x7fe17019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 0x7fe17019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 -- 172.21.15.16:0/1412796942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17019d440 con 0x7fe1701000d0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe1753be640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 0x7fe17019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.774+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e9f0 tx=0x7fe16000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.775+0000 7fe16dffb640 1 -- 172.21.15.16:0/1412796942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16000cd80 con 0x7fe1701000d0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.775+0000 7fe16dffb640 1 -- 172.21.15.16:0/1412796942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe160005500 con 0x7fe1701000d0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.775+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17019d730 con 0x7fe1701000d0 2024-09-12T03:01:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.775+0000 7fe16dffb640 1 -- 172.21.15.16:0/1412796942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160010690 con 0x7fe1701000d0 2024-09-12T03:01:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.775+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1701a5010 con 0x7fe1701000d0 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.777+0000 7fe16dffb640 1 -- 172.21.15.16:0/1412796942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe1600046c0 con 0x7fe1701000d0 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.777+0000 7fe1791fa740 1 Processor -- start 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.777+0000 7fe1791fa740 1 -- start start 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16dffb640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 0x7fe1580a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da75090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da80370 0x55cf5da82760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf5d9db2f0 con 0x55cf5da755d0 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf5d982cf0 con 0x55cf5da80370 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1791fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cf5d9d72b0 con 0x55cf5da76b40 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16dffb640 1 -- 172.21.15.16:0/1412796942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe1600cf490 con 0x7fe1701000d0 2024-09-12T03:01:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50706/0 (socket says 172.21.15.16:50706) 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 -- 172.21.15.16:0/3557020308 learned_addr learned my addr 172.21.15.16:0/3557020308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe174bbd640 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da75090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe1753be640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 0x7fe1580a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 -- 172.21.15.16:0/3557020308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 msgr2=0x55cf5da75090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da75090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 -- 172.21.15.16:0/3557020308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da80370 msgr2=0x55cf5da82760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da80370 0x55cf5da82760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.778+0000 7fe16ffff640 1 -- 172.21.15.16:0/3557020308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5d9673a0 con 0x55cf5da755d0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe16ffff640 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe16400bd10 tx=0x7fe1640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4238f5c24dddfe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe1753be640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 0x7fe1580a50c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x55cf5da76070 tx=0x7fe168005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe156ffd640 1 -- 172.21.15.16:0/3557020308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164005220 con 0x55cf5da755d0 2024-09-12T03:01:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe156ffd640 1 -- 172.21.15.16:0/3557020308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe164004c80 con 0x55cf5da755d0 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe156ffd640 1 -- 172.21.15.16:0/3557020308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164011840 con 0x55cf5da755d0 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe1791fa740 1 -- 172.21.15.16:0/3557020308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 msgr2=0x55cf5da75a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.779+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe16400bd10 tx=0x7fe1640107a0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- 172.21.15.16:0/3557020308 shutdown_connections 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da80370 0x55cf5da82760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da755d0 0x55cf5da75a10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3557020308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da75090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- 172.21.15.16:0/3557020308 >> 172.21.15.16:0/3557020308 conn(0x55cf5da3e310 msgr2=0x55cf5da71c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- 172.21.15.16:0/3557020308 shutdown_connections 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- 172.21.15.16:0/3557020308 wait complete. 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- start start 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 Processor -- start 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da7d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf5da87480 con 0x55cf5da80370 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf5d996060 con 0x55cf5da755d0 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.780+0000 7fe1791fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cf5da87600 con 0x55cf5da76b40 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1791fa740 10 client.0 _ll_register_callbacks cb 0x55cf5d6e98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50716/0 (socket says 172.21.15.16:50716) 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 -- 172.21.15.16:0/1173678192 learned_addr learned my addr 172.21.15.16:0/1173678192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe16ffff640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da7d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 -- 172.21.15.16:0/1173678192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 msgr2=0x55cf5da7d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da7d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 -- 172.21.15.16:0/1173678192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 msgr2=0x55cf5da7fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 -- 172.21.15.16:0/1173678192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5d74c3c0 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1753be640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe168004af0 tx=0x7fe16802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6161a9439fda2425 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe14ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168004510 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.781+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe164004c80 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe1680046b0 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe1680305c0 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 10 client.18561 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf5da92f60 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe164004c80 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fe16802fa50 con 0x55cf5da80370 2024-09-12T03:01:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 10 client.18561 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 10 client.18561 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 20 client.18561 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5da94420 0x55cf5da96850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 4 client.18561 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf5d9e8860 con 0x55cf5da94420 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.782+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5da94420 0x55cf5da96850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe174bbd640 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5da94420 0x55cf5da96850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11ae08012a6a7c9 server_cookie=f8356ed4972c97a9 in_seq=0 out_seq=0 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe14ffff640 10 client.18561 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe14ffff640 1 -- 172.21.15.16:0/1173678192 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cf5d9e8860 con 0x55cf5da94420 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe14ffff640 10 client.18561 handle_command_reply: tid=0 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 client.18561 shutdown 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 msgr2=0x55cf5da7d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe168004af0 tx=0x7fe16802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 shutdown_connections 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da7d620 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da7d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5da94420 0x55cf5da96850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 --2- 172.21.15.16:0/1173678192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 >> 172.21.15.16:0/1173678192 conn(0x55cf5da3e310 msgr2=0x55cf5da77960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.783+0000 7fe14ffff640 0 client.18561 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.784+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 shutdown_connections 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.784+0000 7fe1791fa740 1 -- 172.21.15.16:0/1173678192 wait complete. 2024-09-12T03:01:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.784+0000 7fe1791fa740 20 client.18561 trim_cache size 0 max 16384 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 Processor -- start 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 -- start start 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da78120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701ab2d0 con 0x55cf5da80370 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701aced0 con 0x55cf5da755d0 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1791fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701ab9a0 con 0x55cf5da76b40 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50728/0 (socket says 172.21.15.16:50728) 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe1753be640 1 -- 172.21.15.16:0/2378951314 learned_addr learned my addr 172.21.15.16:0/2378951314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe16ffff640 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da78120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.797+0000 7fe174bbd640 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 -- 172.21.15.16:0/2378951314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 msgr2=0x55cf5da78120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da78120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 -- 172.21.15.16:0/2378951314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 msgr2=0x55cf5da7ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 -- 172.21.15.16:0/2378951314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5da76780 con 0x55cf5da80370 2024-09-12T03:01:13.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1753be640 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe16802fdf0 tx=0x7fe16802b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f93477ebe2a72d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1567fc640 1 -- 172.21.15.16:0/2378951314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168048420 con 0x55cf5da80370 2024-09-12T03:01:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1567fc640 1 -- 172.21.15.16:0/2378951314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe168036050 con 0x55cf5da80370 2024-09-12T03:01:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.798+0000 7fe1567fc640 1 -- 172.21.15.16:0/2378951314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168048860 con 0x55cf5da80370 2024-09-12T03:01:13.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- 172.21.15.16:0/2378951314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 msgr2=0x55cf5da78660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe16802fdf0 tx=0x7fe16802b070 comp rx=0 tx=0).stop 2024-09-12T03:01:13.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- 172.21.15.16:0/2378951314 shutdown_connections 2024-09-12T03:01:13.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da78660 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da78120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 --2- 172.21.15.16:0/2378951314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da7ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- 172.21.15.16:0/2378951314 >> 172.21.15.16:0/2378951314 conn(0x55cf5da3e310 msgr2=0x55cf5da7e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- 172.21.15.16:0/2378951314 shutdown_connections 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- 172.21.15.16:0/2378951314 wait complete. 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 -- start start 2024-09-12T03:01:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.799+0000 7fe1791fa740 1 Processor -- start 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da8d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701ac800 con 0x55cf5da80370 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701ac130 con 0x55cf5da755d0 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701a92b0 con 0x55cf5da76b40 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1791fa740 10 client.0 _ll_register_callbacks cb 0x7fe160005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50742/0 (socket says 172.21.15.16:50742) 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 -- 172.21.15.16:0/3270173401 learned_addr learned my addr 172.21.15.16:0/3270173401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe174bbd640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe16ffff640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da8d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 -- 172.21.15.16:0/3270173401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 msgr2=0x55cf5da8d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da8d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 -- 172.21.15.16:0/3270173401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 msgr2=0x55cf5da8cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da8cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 -- 172.21.15.16:0/3270173401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5da7a430 con 0x55cf5da80370 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe16ffff640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da8d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe1753be640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe168004af0 tx=0x7fe16802f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ec30bf713b7471 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.800+0000 7fe14ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168004d40 con 0x55cf5da80370 2024-09-12T03:01:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe168004310 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe168036050 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe168004510 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 10 client.18567 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf5dafedc0 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe168036050 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fe16805a510 con 0x55cf5da80370 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 10 client.18567 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 10 client.18567 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 20 client.18567 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5daff380 0x55cf5db017a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 4 client.18567 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf5d9e2d70 con 0x55cf5daff380 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.801+0000 7fe174bbd640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5daff380 0x55cf5db017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe14ffff640 10 client.18567 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe174bbd640 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5daff380 0x55cf5db017a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3efa0164a2fbda18 server_cookie=c527ad79cb40ac57 in_seq=0 out_seq=0 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe14ffff640 1 -- 172.21.15.16:0/3270173401 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf5d9e2d70 con 0x55cf5daff380 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe14ffff640 10 client.18567 handle_command_reply: tid=0 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 client.18567 shutdown 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 msgr2=0x55cf5da79c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe168004af0 tx=0x7fe16802f720 comp rx=0 tx=0).stop 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 shutdown_connections 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf5da80370 0x55cf5da79c30 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf5da76b40 0x55cf5da8d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55cf5daff380 0x55cf5db017a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 --2- 172.21.15.16:0/3270173401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf5da755d0 0x55cf5da8cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe14ffff640 0 client.18567 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 >> 172.21.15.16:0/3270173401 conn(0x55cf5da3e310 msgr2=0x55cf5da70d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 shutdown_connections 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 1 -- 172.21.15.16:0/3270173401 wait complete. 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.802+0000 7fe1791fa740 20 client.18567 trim_cache size 0 max 16384 2024-09-12T03:01:13.809 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 msgr2=0x7fe1580a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 0x7fe1580a50c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x55cf5da76070 tx=0x7fe168005ef0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 msgr2=0x7fe170198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e9f0 tx=0x7fe16000edf0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 shutdown_connections 2024-09-12T03:01:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe170106ba0 0x7fe17019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe170100a70 0x7fe1701989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe1580a2c00 0x7fe1580a50c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 --2- 172.21.15.16:0/1412796942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1701000d0 0x7fe170198490 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 >> 172.21.15.16:0/1412796942 conn(0x7fe170077210 msgr2=0x7fe170104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 shutdown_connections 2024-09-12T03:01:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:13.803+0000 7fe176e21640 1 -- 172.21.15.16:0/1412796942 wait complete. 2024-09-12T03:01:14.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 27.91 MB/sec warmup 15 sec latency 7001.759 ms 2024-09-12T03:01:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:14 smithi016 ceph-mon[30925]: pgmap v736: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 327 op/s 2024-09-12T03:01:14.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:14 smithi033 ceph-mon[35092]: pgmap v736: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 327 op/s 2024-09-12T03:01:14.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:14 smithi184 ceph-mon[35321]: pgmap v736: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 327 op/s 2024-09-12T03:01:14.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:01:14.837 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:15.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 26.16 MB/sec warmup 16 sec latency 8001.821 ms 2024-09-12T03:01:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 -- 172.21.15.16:0/4146400950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 msgr2=0x7fc8501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 --2- 172.21.15.16:0/4146400950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501067e0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc844009900 tx=0x7fc84402a3f0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 -- 172.21.15.16:0/4146400950 shutdown_connections 2024-09-12T03:01:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 --2- 172.21.15.16:0/4146400950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 0x7fc85010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 --2- 172.21.15.16:0/4146400950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501067e0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 --2- 172.21.15.16:0/4146400950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 -- 172.21.15.16:0/4146400950 >> 172.21.15.16:0/4146400950 conn(0x7fc8500fd8c0 msgr2=0x7fc8500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.267+0000 7fc8560f5640 1 -- 172.21.15.16:0/4146400950 shutdown_connections 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.268+0000 7fc8560f5640 1 -- 172.21.15.16:0/4146400950 wait complete. 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.268+0000 7fc8560f5640 1 Processor -- start 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.268+0000 7fc8560f5640 1 -- start start 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.268+0000 7fc8560f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 0x7fc85019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc850111f70 con 0x7fc850102790 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc850111df0 con 0x7fc850101de0 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501120f0 con 0x7fc850106f10 2024-09-12T03:01:15.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50764/0 (socket says 172.21.15.16:50764) 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 -- 172.21.15.16:0/3666351580 learned_addr learned my addr 172.21.15.16:0/3666351580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84ffff640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 0x7fc85019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 -- 172.21.15.16:0/3666351580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 msgr2=0x7fc85019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 0x7fc85019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 -- 172.21.15.16:0/3666351580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 msgr2=0x7fc8501983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 -- 172.21.15.16:0/3666351580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85019d360 con 0x7fc850102790 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84effd640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc844006010 tx=0x7fc8440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84cff9640 1 -- 172.21.15.16:0/3666351580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84403f030 con 0x7fc850102790 2024-09-12T03:01:15.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85019d5f0 con 0x7fc850102790 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84cff9640 1 -- 172.21.15.16:0/3666351580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844003ce0 con 0x7fc850102790 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc84cff9640 1 -- 172.21.15.16:0/3666351580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844035070 con 0x7fc850102790 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.269+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8501a4f30 con 0x7fc850102790 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.272+0000 7fc84cff9640 1 -- 172.21.15.16:0/3666351580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc844042030 con 0x7fc850102790 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.272+0000 7fc8584ce740 1 Processor -- start 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.272+0000 7fc8584ce740 1 -- start start 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 0x5580d7c47090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c47a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5580d7bad2f0 con 0x5580d7c52370 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5580d7b54cf0 con 0x5580d7c475d0 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc8584ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5580d7ba92b0 con 0x5580d7c48b40 2024-09-12T03:01:15.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c47a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38282/0 (socket says 172.21.15.16:38282) 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84effd640 1 -- 172.21.15.16:0/37183321 learned_addr learned my addr 172.21.15.16:0/37183321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84cff9640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 0x7fc8380a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84ffff640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 0x5580d7c47090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84ffff640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 0x7fc8380a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 -- 172.21.15.16:0/37183321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 msgr2=0x5580d7c47090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 0x5580d7c47090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 -- 172.21.15.16:0/37183321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 msgr2=0x5580d7c54760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84cff9640 1 -- 172.21.15.16:0/3666351580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc8440ef800 con 0x7fc850102790 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 -- 172.21.15.16:0/37183321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580d7b393a0 con 0x5580d7c475d0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84ffff640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 0x5580d7c47090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84effd640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.273+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c47a10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc84000bc80 tx=0x7fc84000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f3d1ee92eb87d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc84ffff640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 0x7fc8380a5040 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bfd0 tx=0x7fc848004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc835ffb640 1 -- 172.21.15.16:0/37183321 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840015070 con 0x5580d7c475d0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc835ffb640 1 -- 172.21.15.16:0/37183321 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84001a940 con 0x5580d7c475d0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc835ffb640 1 -- 172.21.15.16:0/37183321 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8400119a0 con 0x5580d7c475d0 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 -- 172.21.15.16:0/37183321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 msgr2=0x5580d7c47a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c47a10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc84000bc80 tx=0x7fc84000f730 comp rx=0 tx=0).stop 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 -- 172.21.15.16:0/37183321 shutdown_connections 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c52370 0x5580d7c54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c47a10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 --2- 172.21.15.16:0/37183321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c48b40 0x5580d7c47090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 -- 172.21.15.16:0/37183321 >> 172.21.15.16:0/37183321 conn(0x5580d7c10310 msgr2=0x5580d7c43c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 -- 172.21.15.16:0/37183321 shutdown_connections 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.274+0000 7fc8584ce740 1 -- 172.21.15.16:0/37183321 wait complete. 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 -- start start 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 Processor -- start 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5580d7c55070 con 0x5580d7c475d0 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5580d7b68060 con 0x5580d7c48b40 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5580d7c551f0 con 0x5580d7c52370 2024-09-12T03:01:15.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc8584ce740 10 client.0 _ll_register_callbacks cb 0x5580d78bb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.275+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38290/0 (socket says 172.21.15.16:38290) 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 -- 172.21.15.16:0/953565311 learned_addr learned my addr 172.21.15.16:0/953565311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84effd640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84ffff640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 -- 172.21.15.16:0/953565311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 msgr2=0x5580d7c4f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 -- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 msgr2=0x5580d7c4efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 -- 172.21.15.16:0/953565311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580d791e3c0 con 0x5580d7c475d0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84effd640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc840005890 tx=0x7fc84000bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da047d6951d089 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc82effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840015070 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc84001aaf0 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc84001a940 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc84001acf0 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc8584ce740 10 client.18579 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.276+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5580d7c60ac0 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84001a940 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fc840011720 con 0x5580d7c475d0 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 10 client.18579 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 10 client.18579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 20 client.18579 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7c61e10 0x5580d7c64200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 4 client.18579 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580d7bba860 con 0x5580d7c61e10 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.277+0000 7fc84effd640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7c61e10 0x5580d7c64200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.278+0000 7fc84effd640 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7c61e10 0x5580d7c64200 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa4d0d8f483a2cdd server_cookie=ea24ad2473377c5c in_seq=0 out_seq=0 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.278+0000 7fc82effd640 10 client.18579 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.278+0000 7fc82effd640 1 -- 172.21.15.16:0/953565311 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5580d7bba860 con 0x5580d7c61e10 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.278+0000 7fc82effd640 10 client.18579 handle_command_reply: tid=0 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.278+0000 7fc8584ce740 1 client.18579 shutdown 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 msgr2=0x5580d7c51d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc840005890 tx=0x7fc84000bfa0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 shutdown_connections 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7c61e10 0x5580d7c64200 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc82effd640 0 client.18579 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 --2- 172.21.15.16:0/953565311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c51d10 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 >> 172.21.15.16:0/953565311 conn(0x5580d7c10310 msgr2=0x5580d7c49960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 shutdown_connections 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 1 -- 172.21.15.16:0/953565311 wait complete. 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.279+0000 7fc8584ce740 20 client.18579 trim_cache size 0 max 16384 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 Processor -- start 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 -- start start 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c69970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501ad4f0 con 0x5580d7c475d0 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501af0f0 con 0x5580d7c48b40 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc8584ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501adbc0 con 0x5580d7c52370 2024-09-12T03:01:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38306/0 (socket says 172.21.15.16:38306) 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 -- 172.21.15.16:0/1590219197 learned_addr learned my addr 172.21.15.16:0/1590219197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84effd640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84ffff640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c69970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 -- 172.21.15.16:0/1590219197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 msgr2=0x5580d7c69970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c69970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 -- 172.21.15.16:0/1590219197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 msgr2=0x5580d7c4a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84f7fe640 1 -- 172.21.15.16:0/1590219197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580d7c4f8d0 con 0x5580d7c475d0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.292+0000 7fc84ffff640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c69970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc840005d80 tx=0x7fc840011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdefed932df07a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8357fa640 1 -- 172.21.15.16:0/1590219197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840033030 con 0x5580d7c475d0 2024-09-12T03:01:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8357fa640 1 -- 172.21.15.16:0/1590219197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84002a030 con 0x5580d7c475d0 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8357fa640 1 -- 172.21.15.16:0/1590219197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840015040 con 0x5580d7c475d0 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 -- 172.21.15.16:0/1590219197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 msgr2=0x5580d7c4a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc840005d80 tx=0x7fc840011440 comp rx=0 tx=0).stop 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 -- 172.21.15.16:0/1590219197 shutdown_connections 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c69970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c48b40 0x5580d7c4a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 --2- 172.21.15.16:0/1590219197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c475d0 0x5580d7c4a120 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 -- 172.21.15.16:0/1590219197 >> 172.21.15.16:0/1590219197 conn(0x5580d7c10310 msgr2=0x5580d7c42d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 -- 172.21.15.16:0/1590219197 shutdown_connections 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.293+0000 7fc8584ce740 1 -- 172.21.15.16:0/1590219197 wait complete. 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 -- start start 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 Processor -- start 2024-09-12T03:01:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c4c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501aea20 con 0x5580d7c48b40 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501ae350 con 0x5580d7c475d0 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501ab4d0 con 0x5580d7c52370 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc8584ce740 10 client.0 _ll_register_callbacks cb 0x7fc8440b81a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc84effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38320/0 (socket says 172.21.15.16:38320) 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.294+0000 7fc84effd640 1 -- 172.21.15.16:0/2574191604 learned_addr learned my addr 172.21.15.16:0/2574191604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c4c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84ffff640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 -- 172.21.15.16:0/2574191604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 msgr2=0x5580d7c4cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 -- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 msgr2=0x5580d7c4c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c4c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 -- 172.21.15.16:0/2574191604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580d7ccbae0 con 0x5580d7c48b40 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c4c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84ffff640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc84effd640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc8440c4810 tx=0x7fc844009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0fc0ca80382a2e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc82effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844049070 con 0x5580d7c48b40 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc8440b8740 con 0x5580d7c48b40 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc84002a030 con 0x5580d7c48b40 2024-09-12T03:01:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc8440bbc10 con 0x5580d7c48b40 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc8584ce740 10 client.18585 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.295+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5580d7cccd70 con 0x5580d7c48b40 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84002a030 con 0x5580d7c48b40 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fc8440c00d0 con 0x5580d7c48b40 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 10 client.18585 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 10 client.18585 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 20 client.18585 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7ccd480 0x5580d7ccf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 4 client.18585 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580d7bb4d70 con 0x5580d7ccd480 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7ccd480 0x5580d7ccf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc82effd640 10 client.18585 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.296+0000 7fc84f7fe640 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7ccd480 0x5580d7ccf8a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=90b8adbad64b490c server_cookie=2831630a63f5ad10 in_seq=0 out_seq=0 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc82effd640 1 -- 172.21.15.16:0/2574191604 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580d7bb4d70 con 0x5580d7ccd480 2024-09-12T03:01:15.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc82effd640 10 client.18585 handle_command_reply: tid=0 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 client.18585 shutdown 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 msgr2=0x5580d7c4c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc8440c4810 tx=0x7fc844009880 comp rx=0 tx=0).stop 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 shutdown_connections 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5580d7ccd480 0x5580d7ccf8a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580d7c52370 0x5580d7c4cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5580d7c48b40 0x5580d7c4c680 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc82effd640 0 client.18585 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 --2- 172.21.15.16:0/2574191604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5580d7c475d0 0x5580d7c4c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 >> 172.21.15.16:0/2574191604 conn(0x5580d7c10310 msgr2=0x5580d7c50820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 shutdown_connections 2024-09-12T03:01:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 1 -- 172.21.15.16:0/2574191604 wait complete. 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.297+0000 7fc8584ce740 20 client.18585 trim_cache size 0 max 16384 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 msgr2=0x7fc8380a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 0x7fc8380a5040 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bfd0 tx=0x7fc848004a40 comp rx=0 tx=0).stop 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 msgr2=0x7fc8501988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc844006010 tx=0x7fc8440042f0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 shutdown_connections 2024-09-12T03:01:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc850106f10 0x7fc85019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc8380a2b80 0x7fc8380a5040 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc850102790 0x7fc8501988f0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 --2- 172.21.15.16:0/3666351580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc850101de0 0x7fc8501983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 >> 172.21.15.16:0/3666351580 conn(0x7fc8500fd8c0 msgr2=0x7fc8500fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 shutdown_connections 2024-09-12T03:01:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:15.298+0000 7fc8560f5640 1 -- 172.21.15.16:0/3666351580 wait complete. 2024-09-12T03:01:16.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 24.62 MB/sec warmup 17 sec latency 9001.851 ms 2024-09-12T03:01:16.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:16 smithi016 ceph-mon[30925]: pgmap v737: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 MiB/s wr, 293 op/s 2024-09-12T03:01:16.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:01:16.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:16 smithi033 ceph-mon[35092]: pgmap v737: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 MiB/s wr, 293 op/s 2024-09-12T03:01:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:16 smithi184 ceph-mon[35321]: pgmap v737: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 MiB/s wr, 293 op/s 2024-09-12T03:01:17.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 23.26 MB/sec warmup 18 sec latency 10001.878 ms 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.009+0000 7f54357fc640 1 -- 172.21.15.16:0/3216320145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430101de0 msgr2=0x7f54301021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.009+0000 7f54357fc640 1 --2- 172.21.15.16:0/3216320145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430101de0 0x7f54301021c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f5420009ca0 tx=0x7f542002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 -- 172.21.15.16:0/3216320145 shutdown_connections 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 --2- 172.21.15.16:0/3216320145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430106f10 0x7f543010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 --2- 172.21.15.16:0/3216320145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430102790 0x7f54301067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 --2- 172.21.15.16:0/3216320145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430101de0 0x7f54301021c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 -- 172.21.15.16:0/3216320145 >> 172.21.15.16:0/3216320145 conn(0x7f54300fd8c0 msgr2=0x7f54300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 -- 172.21.15.16:0/3216320145 shutdown_connections 2024-09-12T03:01:17.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 -- 172.21.15.16:0/3216320145 wait complete. 2024-09-12T03:01:17.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 Processor -- start 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.010+0000 7f54357fc640 1 -- start start 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430101de0 0x7f5430198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 0x7f543019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111f70 con 0x7f5430102790 2024-09-12T03:01:17.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111df0 con 0x7f5430106f10 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54301120f0 con 0x7f5430101de0 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38344/0 (socket says 172.21.15.16:38344) 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 -- 172.21.15.16:0/1861066201 learned_addr learned my addr 172.21.15.16:0/1861066201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542effd640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430101de0 0x7f5430198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542f7fe640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 0x7f543019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 -- 172.21.15.16:0/1861066201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430101de0 msgr2=0x7f5430198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430101de0 0x7f5430198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 -- 172.21.15.16:0/1861066201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 msgr2=0x7f543019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 0x7f543019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 -- 172.21.15.16:0/1861066201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543019d3c0 con 0x7f5430102790 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542f7fe640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 0x7f543019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f542e7fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f541800c890 tx=0x7f541800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f5427fff640 1 -- 172.21.15.16:0/1861066201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418004220 con 0x7f5430102790 2024-09-12T03:01:17.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543019d6b0 con 0x7f5430102790 2024-09-12T03:01:17.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f5427fff640 1 -- 172.21.15.16:0/1861066201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54180043c0 con 0x7f5430102790 2024-09-12T03:01:17.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.011+0000 7f5427fff640 1 -- 172.21.15.16:0/1861066201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418004a00 con 0x7f5430102790 2024-09-12T03:01:17.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.012+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54301a4f90 con 0x7f5430102790 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5427fff640 1 -- 172.21.15.16:0/1861066201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5418004c60 con 0x7f5430102790 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 Processor -- start 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 -- start start 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b4da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b5eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639e6ab32f0 con 0x5639e6b4eb40 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.014+0000 7f5437bd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639e6a5acf0 con 0x5639e6b5c780 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f5437bd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5639e6aaf2b0 con 0x5639e6b4d5d0 2024-09-12T03:01:17.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38348/0 (socket says 172.21.15.16:38348) 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 -- 172.21.15.16:0/801473202 learned_addr learned my addr 172.21.15.16:0/801473202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542effd640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b4da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542e7fc640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b5eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 -- 172.21.15.16:0/801473202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 msgr2=0x5639e6b4da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b4da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 -- 172.21.15.16:0/801473202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 msgr2=0x5639e6b5eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b5eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 -- 172.21.15.16:0/801473202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e6a3f3a0 con 0x5639e6b4eb40 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542e7fc640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b5eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f5427fff640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 0x7f541c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542effd640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 0x7f541c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f542f7fe640 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f542800bca0 tx=0x7f542800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99823cdb04cc114f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f5427fff640 1 -- 172.21.15.16:0/1861066201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f54180ced40 con 0x7f5430102790 2024-09-12T03:01:17.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.015+0000 7f54257fa640 1 -- 172.21.15.16:0/801473202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428015070 con 0x5639e6b4eb40 2024-09-12T03:01:17.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f54257fa640 1 -- 172.21.15.16:0/801473202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5428005500 con 0x5639e6b4eb40 2024-09-12T03:01:17.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f54257fa640 1 -- 172.21.15.16:0/801473202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54280116a0 con 0x5639e6b4eb40 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 -- 172.21.15.16:0/801473202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 msgr2=0x5639e6b4d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f542effd640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 0x7f541c0a5040 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5420006010 tx=0x7f542003d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f542800bca0 tx=0x7f542800f730 comp rx=0 tx=0).stop 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 -- 172.21.15.16:0/801473202 shutdown_connections 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b5eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b4da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 --2- 172.21.15.16:0/801473202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b4d090 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 -- 172.21.15.16:0/801473202 >> 172.21.15.16:0/801473202 conn(0x5639e6b16310 msgr2=0x5639e6b49c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 -- 172.21.15.16:0/801473202 shutdown_connections 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.016+0000 7f5437bd5740 1 -- 172.21.15.16:0/801473202 wait complete. 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 -- start start 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 Processor -- start 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b5c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 0x5639e6b59400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5639e6b5f480 con 0x5639e6b5c780 2024-09-12T03:01:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639e6a6e060 con 0x5639e6b4eb40 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5639e6b5f600 con 0x5639e6b4d5d0 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f5437bd5740 10 client.0 _ll_register_callbacks cb 0x5639e67c18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 0x5639e6b59400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38350/0 (socket says 172.21.15.16:38350) 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 0x5639e6b59400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54688/0 (socket says 172.21.15.16:54688) 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b5c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.017+0000 7f542f7fe640 1 -- 172.21.15.16:0/3637440266 learned_addr learned my addr 172.21.15.16:0/3637440266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 -- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 msgr2=0x5639e6b5c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b5c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 -- 172.21.15.16:0/3637440266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 msgr2=0x5639e6b59400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 0x5639e6b59400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 -- 172.21.15.16:0/3637440266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e68243c0 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f542f7fe640 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f54280058b0 tx=0x7f5428004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba68bc9bfacd257e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f54167fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428028060 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f54280048d0 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5428005500 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5428015070 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f5437bd5740 10 client.18597 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.018+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5639e6b6ae60 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5428005500 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f5428011bc0 con 0x5639e6b5c780 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 10 client.18597 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 10 client.18597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:01:17.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 20 client.18597 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6b6c260 0x5639e6b6e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 4 client.18597 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639e6ac0860 con 0x5639e6b6c260 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.019+0000 7f542effd640 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6b6c260 0x5639e6b6e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.020+0000 7f54167fc640 10 client.18597 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.020+0000 7f542effd640 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6b6c260 0x5639e6b6e650 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86a64ba104c79533 server_cookie=cdf6899031abd761 in_seq=0 out_seq=0 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.020+0000 7f54167fc640 1 -- 172.21.15.16:0/3637440266 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5639e6ac0860 con 0x5639e6b6c260 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.020+0000 7f54167fc640 10 client.18597 handle_command_reply: tid=0 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.020+0000 7f5437bd5740 1 client.18597 shutdown 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 msgr2=0x5639e6b59a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f54280058b0 tx=0x7f5428004b50 comp rx=0 tx=0).stop 2024-09-12T03:01:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 shutdown_connections 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b5c780 0x5639e6b59a10 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4eb40 0x5639e6b59400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6b6c260 0x5639e6b6e650 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 --2- 172.21.15.16:0/3637440266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b5c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 >> 172.21.15.16:0/3637440266 conn(0x5639e6b16310 msgr2=0x5639e6b4f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f54167fc640 0 client.18597 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 shutdown_connections 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 1 -- 172.21.15.16:0/3637440266 wait complete. 2024-09-12T03:01:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.021+0000 7f5437bd5740 20 client.18597 trim_cache size 0 max 16384 2024-09-12T03:01:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.034+0000 7f5437bd5740 1 Processor -- start 2024-09-12T03:01:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.034+0000 7f5437bd5740 1 -- start start 2024-09-12T03:01:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4d5d0 0x5639e6b68720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b5c780 0x5639e6b74190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ab310 con 0x5639e6b4eb40 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f54301acf10 con 0x5639e6b4d5d0 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5437bd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ab9e0 con 0x5639e6b5c780 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38358/0 (socket says 172.21.15.16:38358) 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 -- 172.21.15.16:0/4057243874 learned_addr learned my addr 172.21.15.16:0/4057243874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542effd640 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4d5d0 0x5639e6b68720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542f7fe640 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b5c780 0x5639e6b74190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 -- 172.21.15.16:0/4057243874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b5c780 msgr2=0x5639e6b74190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b5c780 0x5639e6b74190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 -- 172.21.15.16:0/4057243874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4d5d0 msgr2=0x5639e6b68720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4d5d0 0x5639e6b68720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 -- 172.21.15.16:0/4057243874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e6b5a380 con 0x5639e6b4eb40 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f542e7fc640 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f5418004610 tx=0x7f5418005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb399231d49ec7b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.035+0000 7f5424ff9640 1 -- 172.21.15.16:0/4057243874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541801d040 con 0x5639e6b4eb40 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5424ff9640 1 -- 172.21.15.16:0/4057243874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541809b5b0 con 0x5639e6b4eb40 2024-09-12T03:01:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5424ff9640 1 -- 172.21.15.16:0/4057243874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418004f50 con 0x5639e6b4eb40 2024-09-12T03:01:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 -- 172.21.15.16:0/4057243874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 msgr2=0x5639e6b73c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f5418004610 tx=0x7f5418005300 comp rx=0 tx=0).stop 2024-09-12T03:01:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 -- 172.21.15.16:0/4057243874 shutdown_connections 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b5c780 0x5639e6b74190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b73c50 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 --2- 172.21.15.16:0/4057243874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b4d5d0 0x5639e6b68720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 -- 172.21.15.16:0/4057243874 >> 172.21.15.16:0/4057243874 conn(0x5639e6b16310 msgr2=0x5639e6b4f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 -- 172.21.15.16:0/4057243874 shutdown_connections 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.036+0000 7f5437bd5740 1 -- 172.21.15.16:0/4057243874 wait complete. 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 -- start start 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 Processor -- start 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b56790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ac840 con 0x5639e6b4eb40 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ac170 con 0x5639e6b5c780 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54301a92f0 con 0x5639e6b4d5d0 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f5437bd5740 10 client.0 _ll_register_callbacks cb 0x7f5420005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:53000/0 (socket says 172.21.15.16:53000) 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38362/0 (socket says 172.21.15.16:38362) 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542effd640 1 -- 172.21.15.16:0/651793425 learned_addr learned my addr 172.21.15.16:0/651793425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542f7fe640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b56790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 -- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 msgr2=0x5639e6b65020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 -- 172.21.15.16:0/651793425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 msgr2=0x5639e6b56790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b56790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.037+0000 7f542e7fc640 1 -- 172.21.15.16:0/651793425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e6b56fc0 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f542effd640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f542e7fc640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5418004610 tx=0x7f54180053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ad73442621d4ca server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54180b8020 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5418084b40 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541809b5b0 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f541801d040 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 10 client.18603 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5639e6bd6c90 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541809b5b0 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f541809ab90 con 0x5639e6b4eb40 2024-09-12T03:01:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 10 client.18603 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 10 client.18603 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 20 client.18603 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6bd73c0 0x5639e6bd97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 4 client.18603 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.038+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639e6abad70 con 0x5639e6bd73c0 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f542effd640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6bd73c0 0x5639e6bd97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f542effd640 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6bd73c0 0x5639e6bd97e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1be90defd38035a8 server_cookie=cf8e2f10c13271a8 in_seq=0 out_seq=0 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f54167fc640 10 client.18603 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f54167fc640 1 -- 172.21.15.16:0/651793425 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639e6abad70 con 0x5639e6bd73c0 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f54167fc640 10 client.18603 handle_command_reply: tid=0 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f5437bd5740 1 client.18603 shutdown 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 msgr2=0x5639e6b561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.039+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5418004610 tx=0x7f54180053a0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 shutdown_connections 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639e6b5c780 0x5639e6b56790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5639e6b4eb40 0x5639e6b561f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5639e6bd73c0 0x5639e6bd97e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 --2- 172.21.15.16:0/651793425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5639e6b4d5d0 0x5639e6b65020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54167fc640 0 client.18603 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 >> 172.21.15.16:0/651793425 conn(0x5639e6b16310 msgr2=0x5639e6b484e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 shutdown_connections 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 1 -- 172.21.15.16:0/651793425 wait complete. 2024-09-12T03:01:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f5437bd5740 20 client.18603 trim_cache size 0 max 16384 2024-09-12T03:01:17.048 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 msgr2=0x7f541c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 0x7f541c0a5040 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5420006010 tx=0x7f542003d040 comp rx=0 tx=0).stop 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 msgr2=0x7f5430198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f541800c890 tx=0x7f541800cd50 comp rx=0 tx=0).stop 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 shutdown_connections 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5430106f10 0x7f543019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f541c0a2b80 0x7f541c0a5040 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5430102790 0x7f5430198950 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 --2- 172.21.15.16:0/1861066201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5430101de0 0x7f5430198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 >> 172.21.15.16:0/1861066201 conn(0x7f54300fd8c0 msgr2=0x7f54300fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.040+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 shutdown_connections 2024-09-12T03:01:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:17.041+0000 7f54357fc640 1 -- 172.21.15.16:0/1861066201 wait complete. 2024-09-12T03:01:17.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:01:18.001 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 22.03 MB/sec warmup 19 sec latency 11001.932 ms 2024-09-12T03:01:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:18.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:18 smithi016 ceph-mon[30925]: pgmap v738: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.5 MiB/s wr, 297 op/s 2024-09-12T03:01:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:18 smithi033 ceph-mon[35092]: pgmap v738: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.5 MiB/s wr, 297 op/s 2024-09-12T03:01:18.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 -- 172.21.15.16:0/52247696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8106a80 msgr2=0x7f22d8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/52247696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8106a80 0x7f22d8106e60 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009930 tx=0x7f22c802a420 comp rx=0 tx=0).stop 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 -- 172.21.15.16:0/52247696 shutdown_connections 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/52247696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8101420 0x7f22d810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/52247696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8100a80 0x7f22d8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/52247696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8106a80 0x7f22d8106e60 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 -- 172.21.15.16:0/52247696 >> 172.21.15.16:0/52247696 conn(0x7f22d80fc7a0 msgr2=0x7f22d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 -- 172.21.15.16:0/52247696 shutdown_connections 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.513+0000 7f22dd6e2640 1 -- 172.21.15.16:0/52247696 wait complete. 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 Processor -- start 2024-09-12T03:01:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 -- start start 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 0x7f22d8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8106a80 0x7f22d81a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8110f20 con 0x7f22d8100a80 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8110da0 con 0x7f22d8106a80 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22dd6e2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81110a0 con 0x7f22d8101420 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38374/0 (socket says 172.21.15.16:38374) 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 -- 172.21.15.16:0/2941151203 learned_addr learned my addr 172.21.15.16:0/2941151203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d67fc640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 0x7f22d8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d77fe640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8106a80 0x7f22d81a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 -- 172.21.15.16:0/2941151203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 msgr2=0x7f22d8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 0x7f22d8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 -- 172.21.15.16:0/2941151203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8106a80 msgr2=0x7f22d81a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8106a80 0x7f22d81a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d6ffd640 1 -- 172.21.15.16:0/2941151203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d81a4a10 con 0x7f22d8100a80 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.514+0000 7f22d67fc640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 0x7f22d8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22d6ffd640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009a50 tx=0x7f22c8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22bffff640 1 -- 172.21.15.16:0/2941151203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c803f030 con 0x7f22d8100a80 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d81a4be0 con 0x7f22d8100a80 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22bffff640 1 -- 172.21.15.16:0/2941151203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c8003ce0 con 0x7f22d8100a80 2024-09-12T03:01:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22bffff640 1 -- 172.21.15.16:0/2941151203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8035070 con 0x7f22d8100a80 2024-09-12T03:01:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.515+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d81a5030 con 0x7f22d8100a80 2024-09-12T03:01:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.517+0000 7f22bffff640 1 -- 172.21.15.16:0/2941151203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f22c8042030 con 0x7f22d8100a80 2024-09-12T03:01:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 Processor -- start 2024-09-12T03:01:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 -- start start 2024-09-12T03:01:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8606b40 0x556bc8605090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22bffff640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 0x7f22c00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc8616b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556bc856b2f0 con 0x556bc8606b40 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556bc8512cf0 con 0x556bc86055d0 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22dfabb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556bc85672b0 con 0x556bc8614780 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d6ffd640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 0x7f22c00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54738/0 (socket says 172.21.15.16:54738) 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d67fc640 1 -- 172.21.15.16:0/4289057631 learned_addr learned my addr 172.21.15.16:0/4289057631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d77fe640 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8606b40 0x556bc8605090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22d77fe640 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc8616b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.518+0000 7f22bffff640 1 -- 172.21.15.16:0/2941151203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22c80ef6c0 con 0x7f22d8100a80 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 -- 172.21.15.16:0/4289057631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 msgr2=0x556bc8616b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc8616b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 -- 172.21.15.16:0/4289057631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8606b40 msgr2=0x556bc8605090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8606b40 0x556bc8605090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 -- 172.21.15.16:0/4289057631 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bc84f73a0 con 0x556bc86055d0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d6ffd640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 0x7f22c00a5040 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f22c8000c00 tx=0x7f22c802a420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22d67fc640 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f22cc007800 tx=0x7f22cc007cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f476cdcb75b0108 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22bd7fa640 1 -- 172.21.15.16:0/4289057631 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00d950 con 0x556bc86055d0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22bd7fa640 1 -- 172.21.15.16:0/4289057631 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc018070 con 0x556bc86055d0 2024-09-12T03:01:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.519+0000 7f22bd7fa640 1 -- 172.21.15.16:0/4289057631 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc0123e0 con 0x556bc86055d0 2024-09-12T03:01:18.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- 172.21.15.16:0/4289057631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 msgr2=0x556bc8605a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f22cc007800 tx=0x7f22cc007cc0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- 172.21.15.16:0/4289057631 shutdown_connections 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc8616b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc8605a10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 --2- 172.21.15.16:0/4289057631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8606b40 0x556bc8605090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- 172.21.15.16:0/4289057631 >> 172.21.15.16:0/4289057631 conn(0x556bc85ce310 msgr2=0x556bc8601c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- 172.21.15.16:0/4289057631 shutdown_connections 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- 172.21.15.16:0/4289057631 wait complete. 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 -- start start 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.520+0000 7f22dfabb740 1 Processor -- start 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc86055d0 0x556bc86140e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8611390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556bc8617480 con 0x556bc8614780 2024-09-12T03:01:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556bc8526060 con 0x556bc8606b40 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556bc8617600 con 0x556bc86055d0 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 10 client.0 _ll_register_callbacks cb 0x556bc82798c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38384/0 (socket says 172.21.15.16:38384) 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 -- 172.21.15.16:0/3974436256 learned_addr learned my addr 172.21.15.16:0/3974436256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc86055d0 0x556bc86140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d67fc640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8611390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 -- 172.21.15.16:0/3974436256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc86055d0 msgr2=0x556bc86140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc86055d0 0x556bc86140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 -- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 msgr2=0x556bc8611390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8611390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 -- 172.21.15.16:0/3974436256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bc82dc3c0 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d67fc640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8611390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22d77fe640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cee0 tx=0x7f22d0009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a022d91c1f34ef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22baff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0015040 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22d001b770 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.521+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22cc018070 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f22d001b970 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 10 client.18615 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bc8622f10 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc018070 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f22d0027070 con 0x556bc8614780 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 10 client.18615 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 10 client.18615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 20 client.18615 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc86243d0 0x556bc8626850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 4 client.18615 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bc8578860 con 0x556bc86243d0 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.522+0000 7f22d67fc640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc86243d0 0x556bc8626850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.523+0000 7f22d67fc640 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc86243d0 0x556bc8626850 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56c0035ad539e8e3 server_cookie=b3ad0e437e556f81 in_seq=0 out_seq=0 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.523+0000 7f22baff5640 10 client.18615 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22baff5640 1 -- 172.21.15.16:0/3974436256 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556bc8578860 con 0x556bc86243d0 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22baff5640 10 client.18615 handle_command_reply: tid=0 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 client.18615 shutdown 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 msgr2=0x556bc86118f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cee0 tx=0x7f22d0009630 comp rx=0 tx=0).stop 2024-09-12T03:01:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 shutdown_connections 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc86243d0 0x556bc8626850 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc86118f0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8611390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 --2- 172.21.15.16:0/3974436256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc86055d0 0x556bc86140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 >> 172.21.15.16:0/3974436256 conn(0x556bc85ce310 msgr2=0x556bc8607930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22baff5640 0 client.18615 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 shutdown_connections 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 1 -- 172.21.15.16:0/3974436256 wait complete. 2024-09-12T03:01:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.524+0000 7f22dfabb740 20 client.18615 trim_cache size 0 max 16384 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 Processor -- start 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 -- start start 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8615b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc86160b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ab380 con 0x556bc86055d0 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81acf80 con 0x556bc8606b40 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22dfabb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81aba50 con 0x556bc8614780 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38396/0 (socket says 172.21.15.16:38396) 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 -- 172.21.15.16:0/252859910 learned_addr learned my addr 172.21.15.16:0/252859910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d67fc640 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8615b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 -- 172.21.15.16:0/252859910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 msgr2=0x556bc86160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc86160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 -- 172.21.15.16:0/252859910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 msgr2=0x556bc8615b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8615b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d77fe640 1 -- 172.21.15.16:0/252859910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bc8616fb0 con 0x556bc86055d0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.537+0000 7f22d67fc640 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8615b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22d77fe640 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000f80 tx=0x7f22d000c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d24154f4357f227 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22bcff9640 1 -- 172.21.15.16:0/252859910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0009b60 con 0x556bc86055d0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22bcff9640 1 -- 172.21.15.16:0/252859910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d0024070 con 0x556bc86055d0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22bcff9640 1 -- 172.21.15.16:0/252859910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d001e840 con 0x556bc86055d0 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 -- 172.21.15.16:0/252859910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 msgr2=0x556bc8620750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000f80 tx=0x7f22d000c3d0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 -- 172.21.15.16:0/252859910 shutdown_connections 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8614780 0x556bc86160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc8606b40 0x556bc8615b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 --2- 172.21.15.16:0/252859910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc86055d0 0x556bc8620750 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.538+0000 7f22dfabb740 1 -- 172.21.15.16:0/252859910 >> 172.21.15.16:0/252859910 conn(0x556bc85ce310 msgr2=0x556bc8611350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- 172.21.15.16:0/252859910 shutdown_connections 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- 172.21.15.16:0/252859910 wait complete. 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- start start 2024-09-12T03:01:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 Processor -- start 2024-09-12T03:01:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc862b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8606b40 0x556bc862b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ac8b0 con 0x556bc8614780 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ac1e0 con 0x556bc86055d0 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81a9360 con 0x556bc8606b40 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22dfabb740 10 client.0 _ll_register_callbacks cb 0x7f22d000c130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38408/0 (socket says 172.21.15.16:38408) 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22d77fe640 1 -- 172.21.15.16:0/2597819320 learned_addr learned my addr 172.21.15.16:0/2597819320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.539+0000 7f22d77fe640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc862b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d67fc640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8606b40 0x556bc862b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 -- 172.21.15.16:0/2597819320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8606b40 msgr2=0x556bc862b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8606b40 0x556bc862b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 -- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 msgr2=0x556bc862b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc862b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 -- 172.21.15.16:0/2597819320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bc861be00 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22d77fe640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cee0 tx=0x7f22d001e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232f5d0c684eb0db server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22baff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d003a020 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22d001e850 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22d0024070 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f22d001bb10 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22dfabb740 10 client.18621 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.540+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bc861d100 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d0024070 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f22d0027040 con 0x556bc8614780 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 10 client.18621 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 10 client.18621 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:01:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 20 client.18621 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc868f0f0 0x556bc86914e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 4 client.18621 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bc8572d70 con 0x556bc868f0f0 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.541+0000 7f22d67fc640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc868f0f0 0x556bc86914e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22d67fc640 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc868f0f0 0x556bc86914e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c0c78a87d5a7f39 server_cookie=a0294358e966728e in_seq=0 out_seq=0 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22baff5640 10 client.18621 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22baff5640 1 -- 172.21.15.16:0/2597819320 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bc8572d70 con 0x556bc868f0f0 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22baff5640 10 client.18621 handle_command_reply: tid=0 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 client.18621 shutdown 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 msgr2=0x556bc861b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cee0 tx=0x7f22d001e420 comp rx=0 tx=0).stop 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 shutdown_connections 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x556bc868f0f0 0x556bc86914e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556bc8614780 0x556bc861b8c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556bc8606b40 0x556bc862b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22baff5640 0 client.18621 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 --2- 172.21.15.16:0/2597819320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556bc86055d0 0x556bc862b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 >> 172.21.15.16:0/2597819320 conn(0x556bc85ce310 msgr2=0x556bc860fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 shutdown_connections 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 1 -- 172.21.15.16:0/2597819320 wait complete. 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.542+0000 7f22dfabb740 20 client.18621 trim_cache size 0 max 16384 2024-09-12T03:01:18.548 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 msgr2=0x7f22c00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 0x7f22c00a5040 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f22c8000c00 tx=0x7f22c802a420 comp rx=0 tx=0).stop 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 msgr2=0x7f22d8199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009a50 tx=0x7f22c8004270 comp rx=0 tx=0).stop 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 shutdown_connections 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d8106a80 0x7f22d81a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d8101420 0x7f22d8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22c00a2b80 0x7f22c00a5040 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 --2- 172.21.15.16:0/2941151203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8100a80 0x7f22d8199450 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 >> 172.21.15.16:0/2941151203 conn(0x7f22d80fc7a0 msgr2=0x7f22d80fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 shutdown_connections 2024-09-12T03:01:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:18.543+0000 7f22dd6e2640 1 -- 172.21.15.16:0/2941151203 wait complete. 2024-09-12T03:01:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:18 smithi184 ceph-mon[35321]: pgmap v738: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.5 MiB/s wr, 297 op/s 2024-09-12T03:01:19.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 20.93 MB/sec warmup 20 sec latency 12002.039 ms 2024-09-12T03:01:19.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:01:19.573 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:20.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 19.93 MB/sec warmup 21 sec latency 13002.457 ms 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 -- 172.21.15.16:0/889522659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0106a80 msgr2=0x7f7ff0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 --2- 172.21.15.16:0/889522659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0106a80 0x7f7ff0106e60 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009930 tx=0x7f7fe002a420 comp rx=0 tx=0).stop 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 -- 172.21.15.16:0/889522659 shutdown_connections 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 --2- 172.21.15.16:0/889522659 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0101420 0x7f7ff010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 --2- 172.21.15.16:0/889522659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 --2- 172.21.15.16:0/889522659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0106a80 0x7f7ff0106e60 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 -- 172.21.15.16:0/889522659 >> 172.21.15.16:0/889522659 conn(0x7f7ff00fc7a0 msgr2=0x7f7ff00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 -- 172.21.15.16:0/889522659 shutdown_connections 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.027+0000 7f7ff6350640 1 -- 172.21.15.16:0/889522659 wait complete. 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 Processor -- start 2024-09-12T03:01:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 -- start start 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0106a80 0x7f7ff019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0111fa0 con 0x7f7ff0101420 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0111e20 con 0x7f7ff0100a80 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.028+0000 7f7ff6350640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0112120 con 0x7f7ff0106a80 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38426/0 (socket says 172.21.15.16:38426) 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 -- 172.21.15.16:0/1808754676 learned_addr learned my addr 172.21.15.16:0/1808754676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0106a80 0x7f7ff019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7feffff640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 -- 172.21.15.16:0/1808754676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0106a80 msgr2=0x7f7ff019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0106a80 0x7f7ff019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 -- 172.21.15.16:0/1808754676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 msgr2=0x7f7ff019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 -- 172.21.15.16:0/1808754676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff019f6d0 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7feffff640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400b4a0 tx=0x7f7fe400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fed7fa640 1 -- 172.21.15.16:0/1808754676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4014070 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fed7fa640 1 -- 172.21.15.16:0/1808754676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe4010af0 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff019f9c0 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7fed7fa640 1 -- 172.21.15.16:0/1808754676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4017d10 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.029+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff01a72a0 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7fed7fa640 1 -- 172.21.15.16:0/1808754676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7fe4017070 con 0x7f7ff0101420 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 Processor -- start 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 -- start start 2024-09-12T03:01:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3e8370 0x5647cf3ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647cf3432f0 con 0x5647cf3deb40 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647cf2eacf0 con 0x5647cf3e8370 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff8729740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647cf33f2b0 con 0x5647cf3dd5d0 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38436/0 (socket says 172.21.15.16:38436) 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7ff48ed640 1 -- 172.21.15.16:0/1591444606 learned_addr learned my addr 172.21.15.16:0/1591444606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7fed7fa640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 0x7f7fd80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3e8370 0x5647cf3ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.032+0000 7f7feffff640 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 -- 172.21.15.16:0/1591444606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 msgr2=0x5647cf3dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3dda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 -- 172.21.15.16:0/1591444606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3e8370 msgr2=0x5647cf3ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3e8370 0x5647cf3ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 -- 172.21.15.16:0/1591444606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cf2cf3a0 con 0x5647cf3deb40 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7fed7fa640 1 -- 172.21.15.16:0/1808754676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fe40ce4b0 con 0x7f7ff0101420 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 0x7f7fd80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800f480 tx=0x7f7fe800f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44db412419df8ed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7fd67fc640 1 -- 172.21.15.16:0/1591444606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe801c400 con 0x5647cf3deb40 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7fd67fc640 1 -- 172.21.15.16:0/1591444606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe801ca80 con 0x5647cf3deb40 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.033+0000 7f7fd67fc640 1 -- 172.21.15.16:0/1591444606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe801b820 con 0x5647cf3deb40 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- 172.21.15.16:0/1591444606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 msgr2=0x5647cf3dd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800f480 tx=0x7f7fe800f940 comp rx=0 tx=0).stop 2024-09-12T03:01:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 0x7f7fd80a5040 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8020000 tx=0x7f7fe801d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- 172.21.15.16:0/1591444606 shutdown_connections 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3e8370 0x5647cf3ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1591444606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3deb40 0x5647cf3dd090 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- 172.21.15.16:0/1591444606 >> 172.21.15.16:0/1591444606 conn(0x5647cf3a6310 msgr2=0x5647cf3d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- 172.21.15.16:0/1591444606 shutdown_connections 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- 172.21.15.16:0/1591444606 wait complete. 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.034+0000 7f7ff8729740 1 -- start start 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 Processor -- start 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3e5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647cf3eb070 con 0x5647cf3e8370 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647cf2fe060 con 0x5647cf3deb40 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647cf3eb1f0 con 0x5647cf3dd5d0 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff8729740 10 client.0 _ll_register_callbacks cb 0x5647cf0518c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38450/0 (socket says 172.21.15.16:38450) 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 -- 172.21.15.16:0/2062699207 learned_addr learned my addr 172.21.15.16:0/2062699207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7feffff640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 -- 172.21.15.16:0/2062699207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 msgr2=0x5647cf3e7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 -- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 msgr2=0x5647cf3e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.035+0000 7f7ff48ed640 1 -- 172.21.15.16:0/2062699207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cf0b43c0 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801b3e0 tx=0x7f7fe801cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8d3909e504f69b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8009c50 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fe801ca80 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fe801bad0 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7fe801b7d0 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7ff8729740 10 client.18633 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647cf3f6b40 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe801ca80 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f7fe801c710 con 0x5647cf3e8370 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.036+0000 7f7ff8729740 10 client.18633 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7ff8729740 10 client.18633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7ff8729740 20 client.18633 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf3f7fa0 0x5647cf3fa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7ff8729740 4 client.18633 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647cf350860 con 0x5647cf3f7fa0 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7feffff640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf3f7fa0 0x5647cf3fa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7fcf7fe640 10 client.18633 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.037+0000 7f7feffff640 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf3f7fa0 0x5647cf3fa390 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d926be279bc440 server_cookie=eb8c342a553c9789 in_seq=0 out_seq=0 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/2062699207 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647cf350860 con 0x5647cf3f7fa0 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7fcf7fe640 10 client.18633 handle_command_reply: tid=0 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 client.18633 shutdown 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 msgr2=0x5647cf3e5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801b3e0 tx=0x7f7fe801cee0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 shutdown_connections 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf3f7fa0 0x5647cf3fa390 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf3e5640 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 --2- 172.21.15.16:0/2062699207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7fcf7fe640 0 client.18633 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.038+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 >> 172.21.15.16:0/2062699207 conn(0x5647cf3a6310 msgr2=0x5647cf3df990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.039+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 shutdown_connections 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.039+0000 7f7ff8729740 1 -- 172.21.15.16:0/2062699207 wait complete. 2024-09-12T03:01:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.039+0000 7f7ff8729740 20 client.18633 trim_cache size 0 max 16384 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 Processor -- start 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 -- start start 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3f4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 0x5647cf3e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01af860 con 0x5647cf3dd5d0 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01b1460 con 0x5647cf3deb40 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7ff8729740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01aff30 con 0x5647cf3e8370 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7feffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7feffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38452/0 (socket says 172.21.15.16:38452) 2024-09-12T03:01:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.052+0000 7f7feffff640 1 -- 172.21.15.16:0/1304461644 learned_addr learned my addr 172.21.15.16:0/1304461644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 -- 172.21.15.16:0/1304461644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 msgr2=0x5647cf3e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 0x5647cf3e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 0x5647cf3e6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 -- 172.21.15.16:0/1304461644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 msgr2=0x5647cf3f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 -- 172.21.15.16:0/1304461644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cf3e3580 con 0x5647cf3dd5d0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 0x5647cf3e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7feffff640 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009930 tx=0x7f7fe0031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac3855309c0cfd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7fd5ffb640 1 -- 172.21.15.16:0/1304461644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0035040 con 0x5647cf3dd5d0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7fd5ffb640 1 -- 172.21.15.16:0/1304461644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe0048070 con 0x5647cf3dd5d0 2024-09-12T03:01:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.053+0000 7f7fd5ffb640 1 -- 172.21.15.16:0/1304461644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0004b80 con 0x5647cf3dd5d0 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- 172.21.15.16:0/1304461644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 msgr2=0x5647cf3e4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009930 tx=0x7f7fe0031510 comp rx=0 tx=0).stop 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- 172.21.15.16:0/1304461644 shutdown_connections 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3e8370 0x5647cf3e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- 172.21.15.16:0/1304461644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3dd5d0 0x5647cf3e4910 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- 172.21.15.16:0/1304461644 >> 172.21.15.16:0/1304461644 conn(0x5647cf3a6310 msgr2=0x5647cf3e6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- 172.21.15.16:0/1304461644 shutdown_connections 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- 172.21.15.16:0/1304461644 wait complete. 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- start start 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 Processor -- start 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01b0d90 con 0x5647cf3e8370 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01b06c0 con 0x5647cf3deb40 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.054+0000 7f7ff8729740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01ad840 con 0x5647cf3dd5d0 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff8729740 10 client.0 _ll_register_callbacks cb 0x7f7fe801f020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38468/0 (socket says 172.21.15.16:38468) 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 -- 172.21.15.16:0/3160243169 learned_addr learned my addr 172.21.15.16:0/3160243169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7feffff640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7fef7fe640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 -- 172.21.15.16:0/3160243169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 msgr2=0x5647cf3eb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3eb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 -- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 msgr2=0x5647cf3ebf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3ebf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 -- 172.21.15.16:0/3160243169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cf4610c0 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff48ed640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801ceb0 tx=0x7f7fe80098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80017cddc355bf5f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7fcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe801ca80 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fe800fe40 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.055+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fe0048070 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7fe8009c50 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 10 client.18639 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5647cf462330 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe0048070 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f7fe801c660 con 0x5647cf3e8370 2024-09-12T03:01:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 10 client.18639 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 10 client.18639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 20 client.18639 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf4629f0 0x5647cf464e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 4 client.18639 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647cf34ad70 con 0x5647cf4629f0 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.056+0000 7f7feffff640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf4629f0 0x5647cf464e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7feffff640 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf4629f0 0x5647cf464e10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=246318ddba5f3126 server_cookie=1d33a45ade0872e9 in_seq=0 out_seq=0 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7fcf7fe640 10 client.18639 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7fcf7fe640 1 -- 172.21.15.16:0/3160243169 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647cf34ad70 con 0x5647cf4629f0 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7fcf7fe640 10 client.18639 handle_command_reply: tid=0 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 client.18639 shutdown 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 msgr2=0x5647cf460b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801ceb0 tx=0x7f7fe80098a0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 shutdown_connections 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647cf4629f0 0x5647cf464e10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cf3e8370 0x5647cf460b30 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647cf3deb40 0x5647cf3ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7fcf7fe640 0 client.18639 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 --2- 172.21.15.16:0/3160243169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647cf3dd5d0 0x5647cf3eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 >> 172.21.15.16:0/3160243169 conn(0x5647cf3a6310 msgr2=0x5647cf3df110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 shutdown_connections 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 1 -- 172.21.15.16:0/3160243169 wait complete. 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.057+0000 7f7ff8729740 20 client.18639 trim_cache size 0 max 16384 2024-09-12T03:01:20.065 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 msgr2=0x7f7fd80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 0x7f7fd80a5040 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8020000 tx=0x7f7fe801d000 comp rx=0 tx=0).stop 2024-09-12T03:01:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 msgr2=0x7f7ff019ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400b4a0 tx=0x7f7fe400b960 comp rx=0 tx=0).stop 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 shutdown_connections 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ff0106a80 0x7f7ff019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd80a2b80 0x7f7fd80a5040 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff0101420 0x7f7ff019ac60 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 --2- 172.21.15.16:0/1808754676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ff0100a80 0x7f7ff019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 >> 172.21.15.16:0/1808754676 conn(0x7f7ff00fc7a0 msgr2=0x7f7ff010c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 shutdown_connections 2024-09-12T03:01:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:20.058+0000 7f7ff6350640 1 -- 172.21.15.16:0/1808754676 wait complete. 2024-09-12T03:01:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:20 smithi016 ceph-mon[30925]: pgmap v739: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 8.6 MiB/s wr, 213 op/s 2024-09-12T03:01:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:20 smithi033 ceph-mon[35092]: pgmap v739: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 8.6 MiB/s wr, 213 op/s 2024-09-12T03:01:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:20 smithi184 ceph-mon[35321]: pgmap v739: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 8.6 MiB/s wr, 213 op/s 2024-09-12T03:01:21.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 19.03 MB/sec warmup 22 sec latency 14002.494 ms 2024-09-12T03:01:21.790 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:01:21.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:01:22.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 18.20 MB/sec warmup 23 sec latency 15002.539 ms 2024-09-12T03:01:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:22.347 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:22 smithi184 ceph-mon[35321]: pgmap v740: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 8.5 MiB/s wr, 218 op/s 2024-09-12T03:01:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:22 smithi016 ceph-mon[30925]: pgmap v740: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 8.5 MiB/s wr, 218 op/s 2024-09-12T03:01:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:22 smithi033 ceph-mon[35092]: pgmap v740: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 8.5 MiB/s wr, 218 op/s 2024-09-12T03:01:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 -- 172.21.15.16:0/3827337191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102570 msgr2=0x7f46a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 --2- 172.21.15.16:0/3827337191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102570 0x7f46a01029b0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4690009900 tx=0x7f469002a480 comp rx=0 tx=0).stop 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 -- 172.21.15.16:0/3827337191 shutdown_connections 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 --2- 172.21.15.16:0/3827337191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0102ef0 0x7f46a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 --2- 172.21.15.16:0/3827337191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102570 0x7f46a01029b0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 --2- 172.21.15.16:0/3827337191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0100e20 0x7f46a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 -- 172.21.15.16:0/3827337191 >> 172.21.15.16:0/3827337191 conn(0x7f46a00fc7d0 msgr2=0x7f46a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 -- 172.21.15.16:0/3827337191 shutdown_connections 2024-09-12T03:01:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.525+0000 7f46a4992640 1 -- 172.21.15.16:0/3827337191 wait complete. 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.526+0000 7f46a4992640 1 Processor -- start 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.526+0000 7f46a4992640 1 -- start start 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 0x7f46a0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0102570 0x7f46a0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0110f80 con 0x7f46a0102ef0 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0110e00 con 0x7f46a0102570 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46a4992640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0111100 con 0x7f46a0100e20 2024-09-12T03:01:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469e59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 0x7f46a0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38490/0 (socket says 172.21.15.16:38490) 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 -- 172.21.15.16:0/2812657802 learned_addr learned my addr 172.21.15.16:0/2812657802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469dd9b640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0102570 0x7f46a0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 -- 172.21.15.16:0/2812657802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 msgr2=0x7f46a0199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 0x7f46a0199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 -- 172.21.15.16:0/2812657802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0102570 msgr2=0x7f46a0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0102570 0x7f46a0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 -- 172.21.15.16:0/2812657802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a01a48f0 con 0x7f46a0102ef0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469e59c640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 0x7f46a0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f469ed9d640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f469800cad0 tx=0x7f469800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.527+0000 7f46977fe640 1 -- 172.21.15.16:0/2812657802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800ae20 con 0x7f46a0102ef0 2024-09-12T03:01:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.528+0000 7f46977fe640 1 -- 172.21.15.16:0/2812657802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4698005020 con 0x7f46a0102ef0 2024-09-12T03:01:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.528+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a01a4ac0 con 0x7f46a0102ef0 2024-09-12T03:01:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.528+0000 7f46977fe640 1 -- 172.21.15.16:0/2812657802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800f6f0 con 0x7f46a0102ef0 2024-09-12T03:01:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.528+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a01a4f80 con 0x7f46a0102ef0 2024-09-12T03:01:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.530+0000 7f46977fe640 1 -- 172.21.15.16:0/2812657802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f46980051d0 con 0x7f46a0102ef0 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.530+0000 7f46a6d6b740 1 Processor -- start 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.530+0000 7f46a6d6b740 1 -- start start 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642b7227280 con 0x5642b72c1b60 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642b71ceb10 con 0x5642b72c2480 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5642b7223240 con 0x5642b72ccd40 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46977fe640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 0x7f468c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38496/0 (socket says 172.21.15.16:38496) 2024-09-12T03:01:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 -- 172.21.15.16:0/140915389 learned_addr learned my addr 172.21.15.16:0/140915389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469ed9d640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 0x7f468c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469e59c640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f46977fe640 1 -- 172.21.15.16:0/2812657802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f46980cf2c0 con 0x7f46a0102ef0 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 -- 172.21.15.16:0/140915389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 msgr2=0x5642b72cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 -- 172.21.15.16:0/140915389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 msgr2=0x5642b72cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469dd9b640 1 -- 172.21.15.16:0/140915389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642b71db1f0 con 0x5642b72c1b60 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.531+0000 7f469e59c640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f469ed9d640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 0x7f468c0a5040 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f469809b880 tx=0x7f469806d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f469dd9b640 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f46900096c0 tx=0x7f4690009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ab97446a042844 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f4694ff9640 1 -- 172.21.15.16:0/140915389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690035030 con 0x5642b72c1b60 2024-09-12T03:01:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f4694ff9640 1 -- 172.21.15.16:0/140915389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690003cd0 con 0x5642b72c1b60 2024-09-12T03:01:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f4694ff9640 1 -- 172.21.15.16:0/140915389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690031d30 con 0x5642b72c1b60 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f46a6d6b740 1 -- 172.21.15.16:0/140915389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 msgr2=0x5642b72c1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f46900096c0 tx=0x7f4690009610 comp rx=0 tx=0).stop 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.532+0000 7f46a6d6b740 1 -- 172.21.15.16:0/140915389 shutdown_connections 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/140915389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72c1f40 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- 172.21.15.16:0/140915389 >> 172.21.15.16:0/140915389 conn(0x5642b72be0f0 msgr2=0x5642b72be4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- 172.21.15.16:0/140915389 shutdown_connections 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- 172.21.15.16:0/140915389 wait complete. 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- start start 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 Processor -- start 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642b72cf910 con 0x5642b72ccd40 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642b71e1f10 con 0x5642b72c1b60 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5642b72cfa90 con 0x5642b72c2480 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f46a6d6b740 10 client.0 _ll_register_callbacks cb 0x5642b71aaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.533+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38500/0 (socket says 172.21.15.16:38500) 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 -- 172.21.15.16:0/2425704823 learned_addr learned my addr 172.21.15.16:0/2425704823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469e59c640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 -- 172.21.15.16:0/2425704823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 msgr2=0x5642b72ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 -- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 msgr2=0x5642b72c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 -- 172.21.15.16:0/2425704823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642b6f986a0 con 0x5642b72ccd40 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f469dd9b640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f46900315c0 tx=0x7f4690005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3876d65d220c8cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f4685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469004c030 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4690003cd0 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f46900037f0 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4690035050 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f46a6d6b740 10 client.18651 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642b72db540 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.534+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690003cd0 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f4690048420 con 0x5642b72ccd40 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 10 client.18651 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 10 client.18651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 20 client.18651 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b72dc890 0x5642b72dec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 4 client.18651 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642b72347f0 con 0x5642b72dc890 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f469e59c640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b72dc890 0x5642b72dec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f469e59c640 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b72dc890 0x5642b72dec80 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8070ef88eb6eb69 server_cookie=56acee158bd33238 in_seq=0 out_seq=0 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.535+0000 7f4685ffb640 10 client.18651 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f4685ffb640 1 -- 172.21.15.16:0/2425704823 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5642b72347f0 con 0x5642b72dc890 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f4685ffb640 10 client.18651 handle_command_reply: tid=0 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 client.18651 shutdown 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 msgr2=0x5642b72ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f46900315c0 tx=0x7f4690005fd0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 shutdown_connections 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b72dc890 0x5642b72dec80 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72ca6c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f4685ffb640 0 client.18651 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:22.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/2425704823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.536+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 >> 172.21.15.16:0/2425704823 conn(0x5642b72be0f0 msgr2=0x5642b72c3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.537+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 shutdown_connections 2024-09-12T03:01:22.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.537+0000 7f46a6d6b740 1 -- 172.21.15.16:0/2425704823 wait complete. 2024-09-12T03:01:22.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.537+0000 7f46a6d6b740 20 client.18651 trim_cache size 0 max 16384 2024-09-12T03:01:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 Processor -- start 2024-09-12T03:01:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 -- start start 2024-09-12T03:01:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72e21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ad4c0 con 0x5642b72c1b60 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01af0c0 con 0x5642b72c2480 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.554+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01adb90 con 0x5642b72ccd40 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38510/0 (socket says 172.21.15.16:38510) 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 -- 172.21.15.16:0/1764695552 learned_addr learned my addr 172.21.15.16:0/1764695552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469e59c640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72e21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 -- 172.21.15.16:0/1764695552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 msgr2=0x5642b72e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72e21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 -- 172.21.15.16:0/1764695552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 msgr2=0x5642b72e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72e1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 -- 172.21.15.16:0/1764695552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642b72cc510 con 0x5642b72c1b60 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469e59c640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f469dd9b640 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4690004170 tx=0x7f469002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cd679f316a12eb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f4687fff640 1 -- 172.21.15.16:0/1764695552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469004c030 con 0x5642b72c1b60 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f4687fff640 1 -- 172.21.15.16:0/1764695552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690040070 con 0x5642b72c1b60 2024-09-12T03:01:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.555+0000 7f4687fff640 1 -- 172.21.15.16:0/1764695552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690048680 con 0x5642b72c1b60 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- 172.21.15.16:0/1764695552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 msgr2=0x5642b72cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4690004170 tx=0x7f469002a480 comp rx=0 tx=0).stop 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- 172.21.15.16:0/1764695552 shutdown_connections 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72ccd40 0x5642b72e21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c2480 0x5642b72e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/1764695552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72c1b60 0x5642b72cbaa0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- 172.21.15.16:0/1764695552 >> 172.21.15.16:0/1764695552 conn(0x5642b72be0f0 msgr2=0x5642b72cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- 172.21.15.16:0/1764695552 shutdown_connections 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- 172.21.15.16:0/1764695552 wait complete. 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 -- start start 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.556+0000 7f46a6d6b740 1 Processor -- start 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72dfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72e00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ae9f0 con 0x5642b72ccd40 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ae320 con 0x5642b72c1b60 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ab4a0 con 0x5642b72c2480 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f46a6d6b740 10 client.0 _ll_register_callbacks cb 0x7f46900041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38520/0 (socket says 172.21.15.16:38520) 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 -- 172.21.15.16:0/4033588572 learned_addr learned my addr 172.21.15.16:0/4033588572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72dfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469e59c640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72e00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 -- 172.21.15.16:0/4033588572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 msgr2=0x5642b72e00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72e00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 -- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 msgr2=0x5642b72dfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72dfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 -- 172.21.15.16:0/4033588572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642b73462d0 con 0x5642b72ccd40 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f469dd9b640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4690031790 tx=0x7f4690031690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c10776d89e79e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.557+0000 7f4685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469006d060 con 0x5642b72ccd40 2024-09-12T03:01:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4690040070 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f46900037f0 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4690060040 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 10 client.18657 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642b73475d0 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690040070 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f4690066040 con 0x5642b72ccd40 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 10 client.18657 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 10 client.18657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 20 client.18657 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b7347b90 0x5642b7349f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 4 client.18657 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5642b722ed00 con 0x5642b7347b90 2024-09-12T03:01:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.558+0000 7f469e59c640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b7347b90 0x5642b7349f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.559+0000 7f469e59c640 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b7347b90 0x5642b7349f80 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff847273a2de49c4 server_cookie=6cd82ec2166c81f8 in_seq=0 out_seq=0 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.559+0000 7f4685ffb640 10 client.18657 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f4685ffb640 1 -- 172.21.15.16:0/4033588572 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5642b722ed00 con 0x5642b7347b90 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f4685ffb640 10 client.18657 handle_command_reply: tid=0 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 client.18657 shutdown 2024-09-12T03:01:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 msgr2=0x5642b72e05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4690031790 tx=0x7f4690031690 comp rx=0 tx=0).stop 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 shutdown_connections 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5642b7347b90 0x5642b7349f80 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642b72ccd40 0x5642b72e05e0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5642b72c2480 0x5642b72e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 --2- 172.21.15.16:0/4033588572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642b72c1b60 0x5642b72dfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 >> 172.21.15.16:0/4033588572 conn(0x5642b72be0f0 msgr2=0x5642b72d8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f4685ffb640 0 client.18657 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 shutdown_connections 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.560+0000 7f46a6d6b740 1 -- 172.21.15.16:0/4033588572 wait complete. 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a6d6b740 20 client.18657 trim_cache size 0 max 16384 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:01:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "2c664bfb-7b25-4a8e-a71a-76052bdfa4af", 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 msgr2=0x7f468c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 0x7f468c0a5040 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f469809b880 tx=0x7f469806d040 comp rx=0 tx=0).stop 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 msgr2=0x7f46a01a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f469800cad0 tx=0x7f469800cf90 comp rx=0 tx=0).stop 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 shutdown_connections 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f468c0a2b80 0x7f468c0a5040 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a0102ef0 0x7f46a01a43b0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46a0102570 0x7f46a0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 --2- 172.21.15.16:0/2812657802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46a0100e20 0x7f46a0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 >> 172.21.15.16:0/2812657802 conn(0x7f46a00fc7d0 msgr2=0x7f46a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 shutdown_connections 2024-09-12T03:01:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:22.561+0000 7f46a4992640 1 -- 172.21.15.16:0/2812657802 wait complete. 2024-09-12T03:01:23.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 17.44 MB/sec warmup 24 sec latency 16002.611 ms 2024-09-12T03:01:23.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:01:23.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:01:23.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:23 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:23 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.528 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:23 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:23.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 -- 172.21.15.16:0/3736404050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278101de0 msgr2=0x7f82781021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:23.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 --2- 172.21.15.16:0/3736404050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278101de0 0x7f82781021c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8268009900 tx=0x7f826802a410 comp rx=0 tx=0).stop 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 -- 172.21.15.16:0/3736404050 shutdown_connections 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 --2- 172.21.15.16:0/3736404050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278106f10 0x7f827810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 --2- 172.21.15.16:0/3736404050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278102790 0x7f82781067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 --2- 172.21.15.16:0/3736404050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278101de0 0x7f82781021c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 -- 172.21.15.16:0/3736404050 >> 172.21.15.16:0/3736404050 conn(0x7f82780fd8c0 msgr2=0x7f82780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 -- 172.21.15.16:0/3736404050 shutdown_connections 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.975+0000 7f827cc4a640 1 -- 172.21.15.16:0/3736404050 wait complete. 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 Processor -- start 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 -- start start 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278101de0 0x7f82781984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 0x7f8278198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8278111f70 con 0x7f8278106f10 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8278111df0 con 0x7f8278102790 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827cc4a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82781120f0 con 0x7f8278101de0 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f8276d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f8276d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38540/0 (socket says 172.21.15.16:38540) 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f8276d9d640 1 -- 172.21.15.16:0/2209021410 learned_addr learned my addr 172.21.15.16:0/2209021410 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.976+0000 7f827659c640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278101de0 0x7f82781984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 0x7f8278198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 -- 172.21.15.16:0/2209021410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278101de0 msgr2=0x7f82781984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278101de0 0x7f82781984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 -- 172.21.15.16:0/2209021410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 msgr2=0x7f8278198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 0x7f8278198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 -- 172.21.15.16:0/2209021410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827819d480 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 0x7f8278198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f8276d9d640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f827000eb10 tx=0x7f827000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f82677fe640 1 -- 172.21.15.16:0/2209021410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270004b20 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f82677fe640 1 -- 172.21.15.16:0/2209021410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8270004cc0 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827819d770 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f82677fe640 1 -- 172.21.15.16:0/2209021410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270010760 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.977+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82781a5050 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.980+0000 7f82677fe640 1 -- 172.21.15.16:0/2209021410 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8270005020 con 0x7f8278106f10 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.980+0000 7f827f023740 1 Processor -- start 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.980+0000 7f827f023740 1 -- start start 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.980+0000 7f827f023740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346110e360 0x56346110e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.980+0000 7f827f023740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827f023740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827f023740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5634610742e0 con 0x56346111da90 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827f023740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56346101bce0 con 0x56346110ec80 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827f023740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5634610702a0 con 0x56346110e360 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f82677fe640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 0x7f82600a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38542/0 (socket says 172.21.15.16:38542) 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8275d9b640 1 -- 172.21.15.16:0/2184653472 learned_addr learned my addr 172.21.15.16:0/2184653472 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8276d9d640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 0x7f82600a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f82677fe640 1 -- 172.21.15.16:0/2209021410 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f82700cf570 con 0x7f8278106f10 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346110e360 0x56346110e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 -- 172.21.15.16:0/2184653472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346110e360 msgr2=0x56346110e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346110e360 0x56346110e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 -- 172.21.15.16:0/2184653472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 msgr2=0x56346111fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 -- 172.21.15.16:0/2184653472 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563461000440 con 0x56346110ec80 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f8276d9d640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 0x7f82600a50c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f827009bd70 tx=0x7f82700669b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.981+0000 7f827659c640 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111d550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8268005940 tx=0x7f8268005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12b6303b235876e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f8264ff9640 1 -- 172.21.15.16:0/2184653472 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268040070 con 0x56346110ec80 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f8264ff9640 1 -- 172.21.15.16:0/2184653472 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82680318f0 con 0x56346110ec80 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f8264ff9640 1 -- 172.21.15.16:0/2184653472 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268035040 con 0x56346110ec80 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 -- 172.21.15.16:0/2184653472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 msgr2=0x56346111d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111d550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8268005940 tx=0x7f8268005e00 comp rx=0 tx=0).stop 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 -- 172.21.15.16:0/2184653472 shutdown_connections 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346111da90 0x56346111fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111d550 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 --2- 172.21.15.16:0/2184653472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346110e360 0x56346110e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.982+0000 7f827f023740 1 -- 172.21.15.16:0/2184653472 >> 172.21.15.16:0/2184653472 conn(0x56346110aa30 msgr2=0x56346110ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- 172.21.15.16:0/2184653472 shutdown_connections 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- 172.21.15.16:0/2184653472 wait complete. 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- start start 2024-09-12T03:01:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 Processor -- start 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x56346111ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563461120600 con 0x56346110e360 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56346102f050 con 0x56346110ec80 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563461120780 con 0x56346111da90 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827f023740 10 client.0 _ll_register_callbacks cb 0x56346106f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38544/0 (socket says 172.21.15.16:38544) 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f827659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f8275d9b640 1 -- 172.21.15.16:0/83762127 learned_addr learned my addr 172.21.15.16:0/83762127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.983+0000 7f8275d9b640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x56346111ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 -- 172.21.15.16:0/83762127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 msgr2=0x56346111ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x56346111ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 -- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 msgr2=0x56346111a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 -- 172.21.15.16:0/83762127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563460de53c0 con 0x56346110e360 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f827659c640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f8275d9b640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f826c012d50 tx=0x7f826c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bd76671e5db34e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f825dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01c040 con 0x56346110e360 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82680318f0 con 0x56346110e360 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f826c021620 con 0x56346110e360 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f826c021820 con 0x56346110e360 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f827f023740 10 client.18669 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.984+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56346112c070 con 0x56346110e360 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82680318f0 con 0x56346110e360 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f826c00f980 con 0x56346110e360 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 10 client.18669 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 10 client.18669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 20 client.18669 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56346112d480 0x56346112f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 4 client.18669 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563461081850 con 0x56346112d480 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827659c640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56346112d480 0x56346112f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f825dffb640 10 client.18669 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.985+0000 7f827659c640 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56346112d480 0x56346112f870 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae6348bb37048b8 server_cookie=b3617dacbd80a033 in_seq=0 out_seq=0 2024-09-12T03:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f825dffb640 1 -- 172.21.15.16:0/83762127 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563461081850 con 0x56346112d480 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f825dffb640 10 client.18669 handle_command_reply: tid=0 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 client.18669 shutdown 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 msgr2=0x56346111d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f826c012d50 tx=0x7f826c004730 comp rx=0 tx=0).stop 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 shutdown_connections 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56346112d480 0x56346112f870 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x56346111ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x56346111a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f825dffb640 0 client.18669 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 --2- 172.21.15.16:0/83762127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x56346111d400 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 >> 172.21.15.16:0/83762127 conn(0x56346110aa30 msgr2=0x563461110790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 shutdown_connections 2024-09-12T03:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.986+0000 7f827f023740 1 -- 172.21.15.16:0/83762127 wait complete. 2024-09-12T03:01:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:23.987+0000 7f827f023740 20 client.18669 trim_cache size 0 max 16384 2024-09-12T03:01:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 16.74 MB/sec warmup 25 sec latency 17002.674 ms 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 Processor -- start 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 -- start start 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 0x5634611298a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461135610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82781ab2f0 con 0x56346110ec80 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82781acef0 con 0x56346110e360 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827f023740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82781ab9c0 con 0x56346111da90 2024-09-12T03:01:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38556/0 (socket says 172.21.15.16:38556) 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 -- 172.21.15.16:0/4157577307 learned_addr learned my addr 172.21.15.16:0/4157577307 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f8275d9b640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f8275d9b640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 0x5634611298a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 -- 172.21.15.16:0/4157577307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 msgr2=0x563461135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461135610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 -- 172.21.15.16:0/4157577307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 msgr2=0x5634611298a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 0x5634611298a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 -- 172.21.15.16:0/4157577307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563461119660 con 0x56346110ec80 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f8275d9b640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 0x5634611298a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.004+0000 7f827659c640 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f82680057c0 tx=0x7f82680058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67560fed6908d565 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f825ffff640 1 -- 172.21.15.16:0/4157577307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268040040 con 0x56346110ec80 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f825ffff640 1 -- 172.21.15.16:0/4157577307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82680035f0 con 0x56346110ec80 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f825ffff640 1 -- 172.21.15.16:0/4157577307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268035040 con 0x56346110ec80 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 -- 172.21.15.16:0/4157577307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 msgr2=0x5634611350d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f82680057c0 tx=0x7f82680058d0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 -- 172.21.15.16:0/4157577307 shutdown_connections 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110ec80 0x5634611350d0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 --2- 172.21.15.16:0/4157577307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110e360 0x5634611298a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 -- 172.21.15.16:0/4157577307 >> 172.21.15.16:0/4157577307 conn(0x56346110aa30 msgr2=0x563461115d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 -- 172.21.15.16:0/4157577307 shutdown_connections 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.005+0000 7f827f023740 1 -- 172.21.15.16:0/4157577307 wait complete. 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 -- start start 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 Processor -- start 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x563461117550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461117bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82781ac820 con 0x56346110e360 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82781ac150 con 0x56346110ec80 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82781a92f0 con 0x56346111da90 2024-09-12T03:01:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827f023740 10 client.0 _ll_register_callbacks cb 0x7f826c00cd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38562/0 (socket says 172.21.15.16:38562) 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 -- 172.21.15.16:0/2547790914 learned_addr learned my addr 172.21.15.16:0/2547790914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f827659c640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x563461117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461117bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 -- 172.21.15.16:0/2547790914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 msgr2=0x563461117bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461117bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 -- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 msgr2=0x563461117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x563461117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.006+0000 7f8275d9b640 1 -- 172.21.15.16:0/2547790914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56346111af80 con 0x56346110e360 2024-09-12T03:01:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f8275d9b640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f826c0059f0 tx=0x7f826c012780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97377033149fcee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00c750 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82680035f0 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f826c0219c0 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f826c021bc0 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 10 client.18675 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563461197fa0 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82680035f0 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f826c00f600 con 0x56346110e360 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 10 client.18675 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 10 client.18675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 20 client.18675 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.007+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5634611986f0 0x56346119ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f827f023740 4 client.18675 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56346107bd60 con 0x5634611986f0 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f827659c640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5634611986f0 0x56346119ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f825dffb640 10 client.18675 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f827659c640 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5634611986f0 0x56346119ab10 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b0f2fa3d7c57149 server_cookie=cf27fce4d7fbae2d in_seq=0 out_seq=0 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f825dffb640 1 -- 172.21.15.16:0/2547790914 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56346107bd60 con 0x5634611986f0 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f825dffb640 10 client.18675 handle_command_reply: tid=0 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.008+0000 7f827f023740 1 client.18675 shutdown 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 msgr2=0x563461126280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f826c0059f0 tx=0x7f826c012780 comp rx=0 tx=0).stop 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 shutdown_connections 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5634611986f0 0x56346119ab10 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56346111da90 0x563461117bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56346110ec80 0x563461117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 --2- 172.21.15.16:0/2547790914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56346110e360 0x563461126280 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f825dffb640 0 client.18675 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 >> 172.21.15.16:0/2547790914 conn(0x56346110aa30 msgr2=0x563461109350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 shutdown_connections 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 1 -- 172.21.15.16:0/2547790914 wait complete. 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827f023740 20 client.18675 trim_cache size 0 max 16384 2024-09-12T03:01:24.022 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "2c664bfb-7b25-4a8e-a71a-76052bdfa4af": { 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "2c664bfb-7b25-4a8e-a71a-76052bdfa4af", 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:24.023 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 msgr2=0x7f82600a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 0x7f82600a50c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f827009bd70 tx=0x7f82700669b0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 msgr2=0x7f827819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.009+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f827000eb10 tx=0x7f827000c2d0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 shutdown_connections 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f82600a2c00 0x7f82600a50c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8278106f10 0x7f827819cda0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8278102790 0x7f8278198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 --2- 172.21.15.16:0/2209021410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8278101de0 0x7f82781984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 >> 172.21.15.16:0/2209021410 conn(0x7f82780fd8c0 msgr2=0x7f82780fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 shutdown_connections 2024-09-12T03:01:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:24.010+0000 7f827cc4a640 1 -- 172.21.15.16:0/2209021410 wait complete. 2024-09-12T03:01:24.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:24 smithi016 ceph-mon[30925]: pgmap v741: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 5.2 MiB/s wr, 105 op/s 2024-09-12T03:01:24.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:24.320 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.320 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:01:24.320 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:24 smithi033 ceph-mon[35092]: pgmap v741: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 5.2 MiB/s wr, 105 op/s 2024-09-12T03:01:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:01:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:24 smithi184 ceph-mon[35321]: pgmap v741: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 5.2 MiB/s wr, 105 op/s 2024-09-12T03:01:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:01:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:01:24.772 INFO:tasks.cephfs.filesystem:scrub status for tag:2c664bfb-7b25-4a8e-a71a-76052bdfa4af - {'path': '/', 'tag': '2c664bfb-7b25-4a8e-a71a-76052bdfa4af', 'options': 'recursive,force'} 2024-09-12T03:01:25.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 16.10 MB/sec warmup 26 sec latency 18002.777 ms 2024-09-12T03:01:26.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 15.50 MB/sec warmup 27 sec latency 19002.838 ms 2024-09-12T03:01:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:26 smithi016 ceph-mon[30925]: pgmap v742: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 87 op/s 2024-09-12T03:01:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:26 smithi033 ceph-mon[35092]: pgmap v742: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 87 op/s 2024-09-12T03:01:26.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:26 smithi184 ceph-mon[35321]: pgmap v742: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 87 op/s 2024-09-12T03:01:27.002 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 14.95 MB/sec warmup 28 sec latency 20002.934 ms 2024-09-12T03:01:28.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 14.43 MB/sec warmup 29 sec latency 21003.044 ms 2024-09-12T03:01:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:28 smithi016 ceph-mon[30925]: pgmap v743: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-09-12T03:01:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:28 smithi033 ceph-mon[35092]: pgmap v743: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-09-12T03:01:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:28 smithi184 ceph-mon[35321]: pgmap v743: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-09-12T03:01:29.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 13.95 MB/sec warmup 30 sec latency 22003.185 ms 2024-09-12T03:01:30.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 13.50 MB/sec warmup 31 sec latency 23003.311 ms 2024-09-12T03:01:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:30 smithi016 ceph-mon[30925]: pgmap v744: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.0 MiB/s wr, 87 op/s 2024-09-12T03:01:30.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:30 smithi033 ceph-mon[35092]: pgmap v744: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.0 MiB/s wr, 87 op/s 2024-09-12T03:01:30.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:30 smithi184 ceph-mon[35321]: pgmap v744: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.0 MiB/s wr, 87 op/s 2024-09-12T03:01:31.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 13.08 MB/sec warmup 32 sec latency 24003.432 ms 2024-09-12T03:01:32.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 12.69 MB/sec warmup 33 sec latency 25003.473 ms 2024-09-12T03:01:32.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:32 smithi016 ceph-mon[30925]: pgmap v745: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-09-12T03:01:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:32 smithi033 ceph-mon[35092]: pgmap v745: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-09-12T03:01:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:32 smithi184 ceph-mon[35321]: pgmap v745: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-09-12T03:01:33.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 12.31 MB/sec warmup 34 sec latency 26003.963 ms 2024-09-12T03:01:33.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:01:34.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 11.96 MB/sec warmup 35 sec latency 27004.028 ms 2024-09-12T03:01:34.165 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:34.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:34 smithi016 ceph-mon[30925]: pgmap v746: 97 pgs: 97 active+clean; 352 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 3.5 MiB/s wr, 91 op/s 2024-09-12T03:01:34.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:34 smithi033 ceph-mon[35092]: pgmap v746: 97 pgs: 97 active+clean; 352 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 3.5 MiB/s wr, 91 op/s 2024-09-12T03:01:34.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:34 smithi184 ceph-mon[35321]: pgmap v746: 97 pgs: 97 active+clean; 352 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 3.5 MiB/s wr, 91 op/s 2024-09-12T03:01:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 -- 172.21.15.16:0/1889758576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420102570 msgr2=0x7ff4201029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 --2- 172.21.15.16:0/1889758576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420102570 0x7ff4201029b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff408009900 tx=0x7ff40802a480 comp rx=0 tx=0).stop 2024-09-12T03:01:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 -- 172.21.15.16:0/1889758576 shutdown_connections 2024-09-12T03:01:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 --2- 172.21.15.16:0/1889758576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102ef0 0x7ff42010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 --2- 172.21.15.16:0/1889758576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420102570 0x7ff4201029b0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 --2- 172.21.15.16:0/1889758576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420100e20 0x7ff420101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 -- 172.21.15.16:0/1889758576 >> 172.21.15.16:0/1889758576 conn(0x7ff4200fc7d0 msgr2=0x7ff4200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 -- 172.21.15.16:0/1889758576 shutdown_connections 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.775+0000 7ff42684d640 1 -- 172.21.15.16:0/1889758576 wait complete. 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 Processor -- start 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 -- start start 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102570 0x7ff420198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420102ef0 0x7ff42019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4201120a0 con 0x7ff420100e20 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff420111f20 con 0x7ff420102ef0 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff42684d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff420112220 con 0x7ff420102570 2024-09-12T03:01:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52220/0 (socket says 172.21.15.16:52220) 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 -- 172.21.15.16:0/2481793717 learned_addr learned my addr 172.21.15.16:0/2481793717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff417fff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102570 0x7ff420198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff424dea640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420102ef0 0x7ff42019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 -- 172.21.15.16:0/2481793717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102570 msgr2=0x7ff420198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102570 0x7ff420198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 -- 172.21.15.16:0/2481793717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420102ef0 msgr2=0x7ff42019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420102ef0 0x7ff42019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.776+0000 7ff41ffff640 1 -- 172.21.15.16:0/2481793717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42019d4d0 con 0x7ff420100e20 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff41ffff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff41000d950 tx=0x7ff41000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff410017070 con 0x7ff420100e20 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42019d7c0 con 0x7ff420100e20 2024-09-12T03:01:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff410009c30 con 0x7ff420100e20 2024-09-12T03:01:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41000f6b0 con 0x7ff420100e20 2024-09-12T03:01:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.777+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4201a50a0 con 0x7ff420100e20 2024-09-12T03:01:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.778+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff420101200 con 0x7ff420100e20 2024-09-12T03:01:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.780+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff4100109d0 con 0x7ff420100e20 2024-09-12T03:01:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.780+0000 7ff41dffb640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 0x7ff40c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.781+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff4100cecc0 con 0x7ff420100e20 2024-09-12T03:01:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.781+0000 7ff417fff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 0x7ff40c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.781+0000 7ff417fff640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 0x7ff40c0a5040 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff408009230 tx=0x7ff4080058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:34.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.784+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff410009de0 con 0x7ff420100e20 2024-09-12T03:01:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.996+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff420109090 con 0x7ff420100e20 2024-09-12T03:01:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.997+0000 7ff41dffb640 1 -- 172.21.15.16:0/2481793717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11976 (secure 0 0 0) 0x7ff41009bbd0 con 0x7ff420100e20 2024-09-12T03:01:34.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:01:34.998 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"btime":"2024-09-12T03:01:04:048707+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:03.760889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:01:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 msgr2=0x7ff40c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 0x7ff40c0a5040 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff408009230 tx=0x7ff4080058d0 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 msgr2=0x7ff420198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff41000d950 tx=0x7ff41000de10 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 shutdown_connections 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff420102ef0 0x7ff42019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff420102570 0x7ff420198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff40c0a2b80 0x7ff40c0a5040 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:34.999+0000 7ff42684d640 1 --2- 172.21.15.16:0/2481793717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff420100e20 0x7ff420198520 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:35.000+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 >> 172.21.15.16:0/2481793717 conn(0x7ff4200fc7d0 msgr2=0x7ff4200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:35.000+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 shutdown_connections 2024-09-12T03:01:35.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:35.000+0000 7ff42684d640 1 -- 172.21.15.16:0/2481793717 wait complete. 2024-09-12T03:01:35.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 11.63 MB/sec warmup 36 sec latency 28004.081 ms 2024-09-12T03:01:35.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:35 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2481793717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:35.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:35 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2481793717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:35 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2481793717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:35.829 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 8c2a116 --release --await' 2024-09-12T03:01:35.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 8c2a116 --release --await 2024-09-12T03:01:36.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4461 11.31 MB/sec warmup 37 sec latency 29004.110 ms 2024-09-12T03:01:36.091 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:36.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:36 smithi016 ceph-mon[30925]: pgmap v747: 97 pgs: 97 active+clean; 376 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.4 MiB/s wr, 96 op/s 2024-09-12T03:01:36.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:36 smithi033 ceph-mon[35092]: pgmap v747: 97 pgs: 97 active+clean; 376 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.4 MiB/s wr, 96 op/s 2024-09-12T03:01:36.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:36 smithi184 ceph-mon[35321]: pgmap v747: 97 pgs: 97 active+clean; 376 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.4 MiB/s wr, 96 op/s 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 -- 172.21.15.16:0/2130441605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074510 msgr2=0x7f46880748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 --2- 172.21.15.16:0/2130441605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074510 0x7f46880748f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4678009900 tx=0x7f467802a480 comp rx=0 tx=0).stop 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 -- 172.21.15.16:0/2130441605 shutdown_connections 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 --2- 172.21.15.16:0/2130441605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f468810e440 0x7f4688110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 --2- 172.21.15.16:0/2130441605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4688074e30 0x7f468810df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 --2- 172.21.15.16:0/2130441605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074510 0x7f46880748f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 -- 172.21.15.16:0/2130441605 >> 172.21.15.16:0/2130441605 conn(0x7f46880fd8c0 msgr2=0x7f46880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 -- 172.21.15.16:0/2130441605 shutdown_connections 2024-09-12T03:01:36.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.767+0000 7f468dd9f640 1 -- 172.21.15.16:0/2130441605 wait complete. 2024-09-12T03:01:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.768+0000 7f468dd9f640 1 Processor -- start 2024-09-12T03:01:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.768+0000 7f468dd9f640 1 -- start start 2024-09-12T03:01:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4688074510 0x7f46881075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f468810e440 0x7f46881026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46881141c0 con 0x7f4688074e30 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4688114040 con 0x7f468810e440 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f468dd9f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4688114340 con 0x7f4688074510 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47082/0 (socket says 172.21.15.16:47082) 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 -- 172.21.15.16:0/799485149 learned_addr learned my addr 172.21.15.16:0/799485149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 -- 172.21.15.16:0/799485149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4688074510 msgr2=0x7f46881075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4688074510 0x7f46881075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 -- 172.21.15.16:0/799485149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f468810e440 msgr2=0x7f46881026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f468810e440 0x7f46881026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.769+0000 7f467effd640 1 -- 172.21.15.16:0/799485149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4688102fa0 con 0x7f4688074e30 2024-09-12T03:01:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.770+0000 7f467effd640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f467000b4a0 tx=0x7f467000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.770+0000 7f46857fa640 1 -- 172.21.15.16:0/799485149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670014070 con 0x7f4688074e30 2024-09-12T03:01:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.770+0000 7f46857fa640 1 -- 172.21.15.16:0/799485149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4670010af0 con 0x7f4688074e30 2024-09-12T03:01:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.770+0000 7f46857fa640 1 -- 172.21.15.16:0/799485149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670017d10 con 0x7f4688074e30 2024-09-12T03:01:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.770+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4688103290 con 0x7f4688074e30 2024-09-12T03:01:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.771+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468810d4b0 con 0x7f4688074e30 2024-09-12T03:01:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.775+0000 7f46857fa640 1 -- 172.21.15.16:0/799485149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4670017070 con 0x7f4688074e30 2024-09-12T03:01:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.783+0000 7f46857fa640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 0x7f46740a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.784+0000 7f46857fa640 1 -- 172.21.15.16:0/799485149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f46700ce3d0 con 0x7f4688074e30 2024-09-12T03:01:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.784+0000 7f46877fe640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 0x7f46740a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 Processor -- start 2024-09-12T03:01:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 -- start start 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f46877fe640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 0x7f46740a50c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f467802a9a0 tx=0x7f4678000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95c9fe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca08d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca0b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e95c963620 con 0x55e95c9fe060 2024-09-12T03:01:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e95c90ae50 con 0x55e95c9fe980 2024-09-12T03:01:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.786+0000 7f4690178740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e95c95f5e0 con 0x55e95ca09240 2024-09-12T03:01:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.788+0000 7f46877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95c9fe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.788+0000 7f46877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95c9fe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47098/0 (socket says 172.21.15.16:47098) 2024-09-12T03:01:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.788+0000 7f46877fe640 1 -- 172.21.15.16:0/2934332557 learned_addr learned my addr 172.21.15.16:0/2934332557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca0b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 -- 172.21.15.16:0/2934332557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 msgr2=0x55e95ca08d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca08d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 -- 172.21.15.16:0/2934332557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 msgr2=0x55e95c9fe440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95c9fe440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.789+0000 7f4687fff640 1 -- 172.21.15.16:0/2934332557 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e95c8fbd40 con 0x55e95ca09240 2024-09-12T03:01:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.792+0000 7f4687fff640 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca0b630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f468810f8d0 tx=0x7f468000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e323569ebda15dc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.794+0000 7f467dffb640 1 -- 172.21.15.16:0/2934332557 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680004b20 con 0x55e95ca09240 2024-09-12T03:01:36.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.794+0000 7f467dffb640 1 -- 172.21.15.16:0/2934332557 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4680003e50 con 0x55e95ca09240 2024-09-12T03:01:36.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.794+0000 7f467dffb640 1 -- 172.21.15.16:0/2934332557 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680005020 con 0x55e95ca09240 2024-09-12T03:01:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.795+0000 7f4690178740 1 -- 172.21.15.16:0/2934332557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 msgr2=0x55e95ca0b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.795+0000 7f4690178740 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca0b630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f468810f8d0 tx=0x7f468000e720 comp rx=0 tx=0).stop 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.800+0000 7f4690178740 1 -- 172.21.15.16:0/2934332557 shutdown_connections 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.800+0000 7f4690178740 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca0b630 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.800+0000 7f4690178740 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.800+0000 7f4690178740 1 --2- 172.21.15.16:0/2934332557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95c9fe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.800+0000 7f4690178740 1 -- 172.21.15.16:0/2934332557 >> 172.21.15.16:0/2934332557 conn(0x55e95c9fa5f0 msgr2=0x55e95c9fa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.801+0000 7f4690178740 1 -- 172.21.15.16:0/2934332557 shutdown_connections 2024-09-12T03:01:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.804+0000 7f4690178740 1 -- 172.21.15.16:0/2934332557 wait complete. 2024-09-12T03:01:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 -- start start 2024-09-12T03:01:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 Processor -- start 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95ca061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca06c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.805+0000 7f4690178740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e95ca0be10 con 0x55e95c9fe060 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.806+0000 7f4690178740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e95c8ebf10 con 0x55e95c9fe980 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.806+0000 7f4690178740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e95ca0bf90 con 0x55e95ca09240 2024-09-12T03:01:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.806+0000 7f4690178740 10 client.0 _ll_register_callbacks cb 0x55e95c90fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.807+0000 7f467effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.807+0000 7f467effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37532/0 (socket says 172.21.15.16:37532) 2024-09-12T03:01:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.807+0000 7f467effd640 1 -- 172.21.15.16:0/1916725230 learned_addr learned my addr 172.21.15.16:0/1916725230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.807+0000 7f46877fe640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95ca061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.807+0000 7f4687fff640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca06c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.808+0000 7f467effd640 1 -- 172.21.15.16:0/1916725230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 msgr2=0x55e95ca06c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.808+0000 7f467effd640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca06c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.808+0000 7f467effd640 1 -- 172.21.15.16:0/1916725230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 msgr2=0x55e95ca061f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.808+0000 7f467effd640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95ca061f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.808+0000 7f467effd640 1 -- 172.21.15.16:0/1916725230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e95c672580 con 0x55e95c9fe980 2024-09-12T03:01:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.811+0000 7f467effd640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f467000be60 tx=0x7f467009b340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7b3fa6ad32e815 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.812+0000 7f4666ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:01:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467001e070 con 0x55e95c9fe980 2024-09-12T03:01:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4680003e50 con 0x55e95c9fe980 2024-09-12T03:01:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4670017360 con 0x55e95c9fe980 2024-09-12T03:01:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f467009eff0 con 0x55e95c9fe980 2024-09-12T03:01:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4690178740 10 client.28055 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.814+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e95c73c960 con 0x55e95c9fe980 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46700a38b0 con 0x55e95c9fe980 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f4670096990 con 0x55e95c9fe980 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 10 client.28055 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 10 client.28055 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 20 client.28055 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95ca3b180 0x55e95ca3d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 4 client.28055 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4680003e50 con 0x55e95ca3b180 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.815+0000 7f46877fe640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95ca3b180 0x55e95ca3d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.816+0000 7f46877fe640 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95ca3b180 0x55e95ca3d590 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b00c57bb35a5df server_cookie=4741697b11e3be3e in_seq=0 out_seq=0 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.816+0000 7f4666ffd640 10 client.28055 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.817+0000 7f4666ffd640 1 -- 172.21.15.16:0/1916725230 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f4680003e50 con 0x55e95ca3b180 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.817+0000 7f4666ffd640 10 client.28055 handle_command_reply: tid=0 2024-09-12T03:01:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.817+0000 7f4690178740 1 client.28055 shutdown 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 msgr2=0x55e95ca06730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f467000be60 tx=0x7f467009b340 comp rx=0 tx=0).stop 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 shutdown_connections 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95ca3b180 0x55e95ca3d590 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95ca09240 0x55e95ca06c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4666ffd640 0 client.28055 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe980 0x55e95ca06730 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 --2- 172.21.15.16:0/1916725230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95c9fe060 0x55e95ca061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 >> 172.21.15.16:0/1916725230 conn(0x55e95c9fa5f0 msgr2=0x55e95ca00320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 shutdown_connections 2024-09-12T03:01:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.818+0000 7f4690178740 1 -- 172.21.15.16:0/1916725230 wait complete. 2024-09-12T03:01:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.819+0000 7f4690178740 20 client.28055 trim_cache size 0 max 16384 2024-09-12T03:01:36.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.855+0000 7f4690178740 1 Processor -- start 2024-09-12T03:01:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.856+0000 7f4690178740 1 -- start start 2024-09-12T03:01:36.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.856+0000 7f4690178740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 0x55e95c9f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.857+0000 7f4690178740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca42ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.857+0000 7f4690178740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.857+0000 7f4690178740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46881a95e0 con 0x55e95ca09240 2024-09-12T03:01:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f4690178740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46881ab1e0 con 0x55e95c9fe060 2024-09-12T03:01:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f4690178740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46881a9cb0 con 0x55e95c9fe980 2024-09-12T03:01:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f467effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca42ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f467effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca42ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41356/0 (socket says 172.21.15.16:41356) 2024-09-12T03:01:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f467effd640 1 -- 172.21.15.16:0/1184777376 learned_addr learned my addr 172.21.15.16:0/1184777376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.859+0000 7f4687fff640 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.858+0000 7f46877fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 0x55e95c9f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.861+0000 7f46877fe640 1 -- 172.21.15.16:0/1184777376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 msgr2=0x55e95ca42ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.861+0000 7f46877fe640 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca42ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.861+0000 7f46877fe640 1 -- 172.21.15.16:0/1184777376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 msgr2=0x55e95ca433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.861+0000 7f46877fe640 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.861+0000 7f46877fe640 1 -- 172.21.15.16:0/1184777376 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e95ca051e0 con 0x55e95c9fe060 2024-09-12T03:01:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.862+0000 7f46877fe640 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 0x55e95c9f9f40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55e95ca3dde0 tx=0x7f467803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f691afb1d31438b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.863+0000 7f467d7fa640 1 -- 172.21.15.16:0/1184777376 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678035040 con 0x55e95c9fe060 2024-09-12T03:01:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.864+0000 7f467d7fa640 1 -- 172.21.15.16:0/1184777376 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4678056070 con 0x55e95c9fe060 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 -- 172.21.15.16:0/1184777376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 msgr2=0x55e95c9f9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 0x55e95c9f9f40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55e95ca3dde0 tx=0x7f467803d000 comp rx=0 tx=0).stop 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 -- 172.21.15.16:0/1184777376 shutdown_connections 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca42ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 --2- 172.21.15.16:0/1184777376 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95c9fe060 0x55e95c9f9f40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 -- 172.21.15.16:0/1184777376 >> 172.21.15.16:0/1184777376 conn(0x55e95c9fa5f0 msgr2=0x55e95ca07a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 -- 172.21.15.16:0/1184777376 shutdown_connections 2024-09-12T03:01:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.865+0000 7f4690178740 1 -- 172.21.15.16:0/1184777376 wait complete. 2024-09-12T03:01:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 -- start start 2024-09-12T03:01:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 Processor -- start 2024-09-12T03:01:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca0e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95ca3b180 0x55e95ca0e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46881aab10 con 0x55e95ca09240 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46881aa440 con 0x55e95ca3b180 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.866+0000 7f4690178740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46881a75c0 con 0x55e95c9fe980 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.867+0000 7f4690178740 10 client.0 _ll_register_callbacks cb 0x7f467009a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.867+0000 7f46877fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.867+0000 7f46877fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41358/0 (socket says 172.21.15.16:41358) 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.867+0000 7f46877fe640 1 -- 172.21.15.16:0/737427672 learned_addr learned my addr 172.21.15.16:0/737427672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 -- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95ca3b180 msgr2=0x55e95ca0e720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95ca3b180 0x55e95ca0e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 -- 172.21.15.16:0/737427672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 msgr2=0x55e95ca0e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca0e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 -- 172.21.15.16:0/737427672 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e95ca0ef70 con 0x55e95c9fe980 2024-09-12T03:01:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f46877fe640 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46780059d0 tx=0x7f467802ac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e263c2487d514bf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.868+0000 7f4666ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:01:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467805f060 con 0x55e95c9fe980 2024-09-12T03:01:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e95c970b90 con 0x55e95c9fe980 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4678049b60 con 0x55e95c9fe980 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4678035070 con 0x55e95c9fe980 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4690178740 10 client.27964 fetch_fsmap learned FSMap version 33 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.870+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e95cab7f90 con 0x55e95c9fe980 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.871+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e95c970b90 con 0x55e95c9fe980 2024-09-12T03:01:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.871+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f467805a410 con 0x55e95c9fe980 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 10 client.27964 fetch_fsmap finished waiting for FSMap version 33 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 10 client.27964 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 20 client.27964 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95cab85e0 0x55e95cabaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 4 client.27964 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8c2a116", "release": true, "await": true}] 2024-09-12T03:01:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.872+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8c2a116", "release": true, "await": true}) -- 0x55e95c96b0a0 con 0x55e95cab85e0 2024-09-12T03:01:36.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.874+0000 7f467effd640 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95cab85e0 0x55e95cabaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.876+0000 7f4666ffd640 10 client.27964 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.876+0000 7f467effd640 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95cab85e0 0x55e95cabaa20 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d0c8159abcf8600 server_cookie=d784b5c9b56657f3 in_seq=0 out_seq=0 2024-09-12T03:01:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.930+0000 7f4666ffd640 1 -- 172.21.15.16:0/737427672 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e95c96b0a0 con 0x55e95cab85e0 2024-09-12T03:01:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.930+0000 7f4666ffd640 10 client.27964 handle_command_reply: tid=0 2024-09-12T03:01:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.930+0000 7f4690178740 1 client.27964 shutdown 2024-09-12T03:01:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 msgr2=0x55e95ca119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46780059d0 tx=0x7f467802ac00 comp rx=0 tx=0).stop 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 shutdown_connections 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e95cab85e0 0x55e95cabaa20 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e95ca3b180 0x55e95ca0e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4666ffd640 0 client.27964 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e95ca09240 0x55e95ca0e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 --2- 172.21.15.16:0/737427672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e95c9fe980 0x55e95ca119f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 >> 172.21.15.16:0/737427672 conn(0x55e95c9fa5f0 msgr2=0x55e95ca06010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 shutdown_connections 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 1 -- 172.21.15.16:0/737427672 wait complete. 2024-09-12T03:01:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.931+0000 7f4690178740 20 client.27964 trim_cache size 0 max 16384 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 33, 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 52, 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "8c2a116": { 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 52, 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:36.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:36.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 msgr2=0x7f46740a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 0x7f46740a50c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f467802a9a0 tx=0x7f4678000bc0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 msgr2=0x7f4688107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f467000b4a0 tx=0x7f467000b960 comp rx=0 tx=0).stop 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 shutdown_connections 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f468810e440 0x7f46881026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f46740a2c00 0x7f46740a50c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4688074e30 0x7f4688107b00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 --2- 172.21.15.16:0/799485149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4688074510 0x7f46881075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 >> 172.21.15.16:0/799485149 conn(0x7f46880fd8c0 msgr2=0x7f46880fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 shutdown_connections 2024-09-12T03:01:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:36.932+0000 7f468dd9f640 1 -- 172.21.15.16:0/799485149 wait complete. 2024-09-12T03:01:37.003 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4463 11.03 MB/sec warmup 38 sec latency 29999.794 ms 2024-09-12T03:01:37.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:37 smithi016 ceph-mon[30925]: pgmap v748: 97 pgs: 97 active+clean; 431 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 10 MiB/s wr, 111 op/s 2024-09-12T03:01:37.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:37 smithi033 ceph-mon[35092]: pgmap v748: 97 pgs: 97 active+clean; 431 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 10 MiB/s wr, 111 op/s 2024-09-12T03:01:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:37 smithi184 ceph-mon[35321]: pgmap v748: 97 pgs: 97 active+clean; 431 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 10 MiB/s wr, 111 op/s 2024-09-12T03:01:37.753 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8c2a116', total seconds elapsed: 33.8 2024-09-12T03:01:37.753 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-09-12T03:01:37.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:01:38.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4919 11.28 MB/sec warmup 39 sec latency 30014.568 ms 2024-09-12T03:01:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:01:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:01:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:01:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:01:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:01:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:01:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:01:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:01:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:01:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:01:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:01:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.596+0000 7f2428ef6640 1 -- 172.21.15.16:0/3248136706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424102570 msgr2=0x7f24241029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.596+0000 7f2428ef6640 1 --2- 172.21.15.16:0/3248136706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424102570 0x7f24241029b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f2418009900 tx=0x7f241802a480 comp rx=0 tx=0).stop 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.596+0000 7f2428ef6640 1 -- 172.21.15.16:0/3248136706 shutdown_connections 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.596+0000 7f2428ef6640 1 --2- 172.21.15.16:0/3248136706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102ef0 0x7f242410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.596+0000 7f2428ef6640 1 --2- 172.21.15.16:0/3248136706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424102570 0x7f24241029b0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.597+0000 7f2428ef6640 1 --2- 172.21.15.16:0/3248136706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424100e20 0x7f2424101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.597+0000 7f2428ef6640 1 -- 172.21.15.16:0/3248136706 >> 172.21.15.16:0/3248136706 conn(0x7f24240fc7d0 msgr2=0x7f24240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.597+0000 7f2428ef6640 1 -- 172.21.15.16:0/3248136706 shutdown_connections 2024-09-12T03:01:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.597+0000 7f2428ef6640 1 -- 172.21.15.16:0/3248136706 wait complete. 2024-09-12T03:01:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.598+0000 7f2428ef6640 1 Processor -- start 2024-09-12T03:01:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.598+0000 7f2428ef6640 1 -- start start 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 0x7f24241984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424102ef0 0x7f242419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24241120d0 con 0x7f2424100e20 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2424111f50 con 0x7f2424102ef0 2024-09-12T03:01:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2428ef6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2424112250 con 0x7f2424102570 2024-09-12T03:01:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2421d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f242259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 0x7f24241984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2421d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:41384/0 (socket says 172.21.15.16:41384) 2024-09-12T03:01:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2421d9b640 1 -- 172.21.15.16:0/2155704117 learned_addr learned my addr 172.21.15.16:0/2155704117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f242259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 0x7f24241984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47136/0 (socket says 172.21.15.16:47136) 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.599+0000 7f2422d9d640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424102ef0 0x7f242419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 -- 172.21.15.16:0/2155704117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424102ef0 msgr2=0x7f242419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424102ef0 0x7f242419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 -- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 msgr2=0x7f24241984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 0x7f24241984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 -- 172.21.15.16:0/2155704117 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242419d480 con 0x7f2424102570 2024-09-12T03:01:38.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.600+0000 7f2421d9b640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f241802a9f0 tx=0x7f2418003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.601+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241803f070 con 0x7f2424102570 2024-09-12T03:01:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.601+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242419d710 con 0x7f2424102570 2024-09-12T03:01:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.601+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f242410efb0 con 0x7f2424102570 2024-09-12T03:01:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.601+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2418031c20 con 0x7f2424102570 2024-09-12T03:01:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.602+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418035030 con 0x7f2424102570 2024-09-12T03:01:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.603+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2404005290 con 0x7f2424102570 2024-09-12T03:01:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.605+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2418042070 con 0x7f2424102570 2024-09-12T03:01:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.606+0000 7f24137fe640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 0x7f240c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.606+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f24180ef550 con 0x7f2424102570 2024-09-12T03:01:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.606+0000 7f242259c640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 0x7f240c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.608+0000 7f242259c640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 0x7f240c0a5250 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2414005d00 tx=0x7f241400f820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.613+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2418049030 con 0x7f2424102570 2024-09-12T03:01:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.917+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2404005580 con 0x7f2424102570 2024-09-12T03:01:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.918+0000 7f24137fe640 1 -- 172.21.15.16:0/2155704117 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11976 (secure 0 0 0) 0x7f24180bc460 con 0x7f2424102570 2024-09-12T03:01:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:01:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"btime":"2024-09-12T03:01:04:048707+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:03.760889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:01:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-09-12T03:01:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 msgr2=0x7f240c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 0x7f240c0a5250 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2414005d00 tx=0x7f241400f820 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 msgr2=0x7f2424198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f241802a9f0 tx=0x7f2418003b90 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 shutdown_connections 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2424102ef0 0x7f242419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2424102570 0x7f2424198a10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f240c0a2d90 0x7f240c0a5250 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 --2- 172.21.15.16:0/2155704117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2424100e20 0x7f24241984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 >> 172.21.15.16:0/2155704117 conn(0x7f24240fc7d0 msgr2=0x7f24240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 shutdown_connections 2024-09-12T03:01:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:38.921+0000 7f2428ef6640 1 -- 172.21.15.16:0/2155704117 wait complete. 2024-09-12T03:01:39.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5514 11.69 MB/sec warmup 40 sec latency 65.570 ms 2024-09-12T03:01:39.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:39 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:01:39.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:39 smithi016 ceph-mon[30925]: pgmap v749: 97 pgs: 97 active+clean; 464 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 112 op/s 2024-09-12T03:01:39.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:39 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2155704117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:39.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:39 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:01:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:39 smithi033 ceph-mon[35092]: pgmap v749: 97 pgs: 97 active+clean; 464 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 112 op/s 2024-09-12T03:01:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:39 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2155704117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:39 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:01:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:39 smithi184 ceph-mon[35321]: pgmap v749: 97 pgs: 97 active+clean; 464 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 112 op/s 2024-09-12T03:01:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:39 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2155704117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:39.750 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-09-12T03:01:39.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-09-12T03:01:40.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6440 12.42 MB/sec warmup 41 sec latency 62.064 ms 2024-09-12T03:01:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:40.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:40 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:40.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:40 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:40.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:40 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:40.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 -- 172.21.15.16:0/1150622505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 msgr2=0x7fa47010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 --2- 172.21.15.16:0/1150622505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47010da40 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fa468009ca0 tx=0x7fa46802a950 comp rx=0 tx=0).stop 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 -- 172.21.15.16:0/1150622505 shutdown_connections 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 --2- 172.21.15.16:0/1150622505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47010da40 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 --2- 172.21.15.16:0/1150622505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470102570 0x7fa4701029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 --2- 172.21.15.16:0/1150622505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470100e20 0x7fa470101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 -- 172.21.15.16:0/1150622505 >> 172.21.15.16:0/1150622505 conn(0x7fa4700fc7d0 msgr2=0x7fa4700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 -- 172.21.15.16:0/1150622505 shutdown_connections 2024-09-12T03:01:40.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.638+0000 7fa476410640 1 -- 172.21.15.16:0/1150622505 wait complete. 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.639+0000 7fa476410640 1 Processor -- start 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.639+0000 7fa476410640 1 -- start start 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.639+0000 7fa476410640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.639+0000 7fa476410640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470102570 0x7fa470197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.639+0000 7fa476410640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa476410640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa470110f80 con 0x7fa470102ef0 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa476410640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa470110e00 con 0x7fa470100e20 2024-09-12T03:01:40.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa476410640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa470111100 con 0x7fa470102570 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa4749ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37606/0 (socket says 172.21.15.16:37606) 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 -- 172.21.15.16:0/2636219567 learned_addr learned my addr 172.21.15.16:0/2636219567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470102570 0x7fa470197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 -- 172.21.15.16:0/2636219567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470102570 msgr2=0x7fa470197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470102570 0x7fa470197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 -- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 msgr2=0x7fa47019bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47019bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 -- 172.21.15.16:0/2636219567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47019c3f0 con 0x7fa470100e20 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46ffff640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa46000e9f0 tx=0x7fa46000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.640+0000 7fa46d7fa640 1 -- 172.21.15.16:0/2636219567 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46000cd80 con 0x7fa470100e20 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.641+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47019c6e0 con 0x7fa470100e20 2024-09-12T03:01:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.641+0000 7fa46d7fa640 1 -- 172.21.15.16:0/2636219567 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa460010910 con 0x7fa470100e20 2024-09-12T03:01:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.641+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa470069f70 con 0x7fa470100e20 2024-09-12T03:01:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.641+0000 7fa46d7fa640 1 -- 172.21.15.16:0/2636219567 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4600188f0 con 0x7fa470100e20 2024-09-12T03:01:40.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa46d7fa640 1 -- 172.21.15.16:0/2636219567 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa460010ac0 con 0x7fa470100e20 2024-09-12T03:01:40.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 Processor -- start 2024-09-12T03:01:40.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 -- start start 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4556400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4558d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5628d44b12a0 con 0x5628d454b760 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628d4458b10 con 0x5628d454c080 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.644+0000 7fa4787e9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628d44ad260 con 0x5628d4556940 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47168/0 (socket says 172.21.15.16:47168) 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 -- 172.21.15.16:0/2749585697 learned_addr learned my addr 172.21.15.16:0/2749585697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa46ffff640 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4556400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4558d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa46d7fa640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 0x7fa4580a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 -- 172.21.15.16:0/2749585697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 msgr2=0x5628d4558d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4558d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 -- 172.21.15.16:0/2749585697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 msgr2=0x5628d4556400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4556400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 -- 172.21.15.16:0/2749585697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628d4222b90 con 0x5628d454b760 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa46d7fa640 1 -- 172.21.15.16:0/2636219567 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa4600cf4d0 con 0x7fa470100e20 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.645+0000 7fa4749ad640 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa468035000 tx=0x7fa468005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935658f3467cb390 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.646+0000 7fa4749ad640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 0x7fa4580a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.646+0000 7fa4567fc640 1 -- 172.21.15.16:0/2749585697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468049070 con 0x5628d454b760 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.646+0000 7fa4567fc640 1 -- 172.21.15.16:0/2749585697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa46802fb30 con 0x5628d454b760 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.646+0000 7fa4567fc640 1 -- 172.21.15.16:0/2749585697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468034040 con 0x5628d454b760 2024-09-12T03:01:40.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- 172.21.15.16:0/2749585697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 msgr2=0x5628d454bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4749ad640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 0x7fa4580a50c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa468046000 tx=0x7fa468004430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa468035000 tx=0x7fa468005c50 comp rx=0 tx=0).stop 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- 172.21.15.16:0/2749585697 shutdown_connections 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4558d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4556400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 --2- 172.21.15.16:0/2749585697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d454bb40 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- 172.21.15.16:0/2749585697 >> 172.21.15.16:0/2749585697 conn(0x5628d4547cf0 msgr2=0x5628d45480d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- 172.21.15.16:0/2749585697 shutdown_connections 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- 172.21.15.16:0/2749585697 wait complete. 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.647+0000 7fa4787e9740 1 -- start start 2024-09-12T03:01:40.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 Processor -- start 2024-09-12T03:01:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4553d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4554280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5628d456a570 con 0x5628d454b760 2024-09-12T03:01:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628d44396e0 con 0x5628d454c080 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628d456a6f0 con 0x5628d4556940 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4787e9740 10 client.0 _ll_register_callbacks cb 0x5628d445da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4553d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa46ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa46ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47176/0 (socket says 172.21.15.16:47176) 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa46ffff640 1 -- 172.21.15.16:0/3043577344 learned_addr learned my addr 172.21.15.16:0/3043577344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4553d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37636/0 (socket says 172.21.15.16:37636) 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.648+0000 7fa4749ad640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4554280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 -- 172.21.15.16:0/3043577344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 msgr2=0x5628d4554280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4554280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 -- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 msgr2=0x5628d4553d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4553d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 -- 172.21.15.16:0/3043577344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628d4573a60 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa46ffff640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fa460098140 tx=0x7fa460098040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc85268b23e3fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa44f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4600cf2d0 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa460018b60 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.649+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa46802fb30 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa46009c3e0 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 10 client.18714 fetch_fsmap learned FSMap version 34 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5628d45763f0 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa46009bce0 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fa460097a30 con 0x5628d454b760 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 10 client.18714 fetch_fsmap finished waiting for FSMap version 34 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 10 client.18714 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 20 client.18714 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:40.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d4579cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 4 client.18714 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.650+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa46802fb30 con 0x5628d4577850 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.651+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d4579cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.651+0000 7fa44f7fe640 10 client.18714 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.651+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d4579cd0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111b845086a9c3f8 server_cookie=cacc8276c5e3ba43 in_seq=0 out_seq=0 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa44f7fe640 1 -- 172.21.15.16:0/3043577344 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fa46802fb30 con 0x5628d4577850 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa44f7fe640 10 client.18714 handle_command_reply: tid=0 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 client.18714 shutdown 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 msgr2=0x5628d4553800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fa460098140 tx=0x7fa460098040 comp rx=0 tx=0).stop 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 shutdown_connections 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d4579cd0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d4554280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4553d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 --2- 172.21.15.16:0/3043577344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d4553800 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa44f7fe640 0 client.18714 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 >> 172.21.15.16:0/3043577344 conn(0x5628d4547cf0 msgr2=0x5628d4546e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 shutdown_connections 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.652+0000 7fa4787e9740 1 -- 172.21.15.16:0/3043577344 wait complete. 2024-09-12T03:01:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.653+0000 7fa4787e9740 20 client.18714 trim_cache size 0 max 16384 2024-09-12T03:01:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 Processor -- start 2024-09-12T03:01:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 -- start start 2024-09-12T03:01:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d45540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d457f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa470072fc0 con 0x5628d454b760 2024-09-12T03:01:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4701a4760 con 0x5628d454c080 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4787e9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa470073690 con 0x5628d4556940 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37642/0 (socket says 172.21.15.16:37642) 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa46f7fe640 1 -- 172.21.15.16:0/702151866 learned_addr learned my addr 172.21.15.16:0/702151866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa46ffff640 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d45540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.677+0000 7fa4749ad640 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d457f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 -- 172.21.15.16:0/702151866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 msgr2=0x5628d457f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d457f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 -- 172.21.15.16:0/702151866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 msgr2=0x5628d45540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d45540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 -- 172.21.15.16:0/702151866 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628d4543260 con 0x5628d454c080 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa464000c00 tx=0x7fa46401da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf063ace87e81009 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa455ffb640 1 -- 172.21.15.16:0/702151866 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464005020 con 0x5628d454c080 2024-09-12T03:01:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa455ffb640 1 -- 172.21.15.16:0/702151866 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4640047a0 con 0x5628d454c080 2024-09-12T03:01:40.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.678+0000 7fa455ffb640 1 -- 172.21.15.16:0/702151866 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640097c0 con 0x5628d454c080 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 -- 172.21.15.16:0/702151866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 msgr2=0x5628d4546d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa464000c00 tx=0x7fa46401da30 comp rx=0 tx=0).stop 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 -- 172.21.15.16:0/702151866 shutdown_connections 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d457f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d4546d60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 --2- 172.21.15.16:0/702151866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d45540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 -- 172.21.15.16:0/702151866 >> 172.21.15.16:0/702151866 conn(0x5628d4547cf0 msgr2=0x5628d4546650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 -- 172.21.15.16:0/702151866 shutdown_connections 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.679+0000 7fa4787e9740 1 -- 172.21.15.16:0/702151866 wait complete. 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 -- start start 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 Processor -- start 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d456b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d456c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4701a4090 con 0x5628d454b760 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa470073e20 con 0x5628d454c080 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.680+0000 7fa4787e9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa470070fa0 con 0x5628d4556940 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa4787e9740 10 client.0 _ll_register_callbacks cb 0x7fa4600d2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d456b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:37650/0 (socket says 172.21.15.16:37650) 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 -- 172.21.15.16:0/773736838 learned_addr learned my addr 172.21.15.16:0/773736838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa4749ad640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d456c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 -- 172.21.15.16:0/773736838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 msgr2=0x5628d456c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d456c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 -- 172.21.15.16:0/773736838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 msgr2=0x5628d456b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d456b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 -- 172.21.15.16:0/773736838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628d45e1860 con 0x5628d454c080 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa46f7fe640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa464004770 tx=0x7fa46400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18fbee2da3b4be94 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa44f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464032060 con 0x5628d454c080 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4640047a0 con 0x5628d454c080 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.681+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa464004d20 con 0x5628d454c080 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa464005000 con 0x5628d454c080 2024-09-12T03:01:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 10 client.28082 fetch_fsmap learned FSMap version 34 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5628d45e2ac0 con 0x5628d454c080 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4640047a0 con 0x5628d454c080 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fa46402d450 con 0x5628d454c080 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 10 client.28082 fetch_fsmap finished waiting for FSMap version 34 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 10 client.28082 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 20 client.28082 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d45e51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 4 client.28082 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x5628d44be9d0 con 0x5628d4577850 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.682+0000 7fa46ffff640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d45e51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.683+0000 7fa44f7fe640 10 client.28082 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:40.683+0000 7fa46ffff640 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d45e51c0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddea92462eabd000 server_cookie=be8a78c4efd68895 in_seq=0 out_seq=0 2024-09-12T03:01:41.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6928 12.73 MB/sec warmup 42 sec latency 320.178 ms 2024-09-12T03:01:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa44f7fe640 1 -- 172.21.15.16:0/773736838 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5628d44be9d0 con 0x5628d4577850 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa44f7fe640 10 client.28082 handle_command_reply: tid=0 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 client.28082 shutdown 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 msgr2=0x5628d456be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa464004770 tx=0x7fa46400ed00 comp rx=0 tx=0).stop 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 shutdown_connections 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5628d4577850 0x5628d45e51c0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5628d4556940 0x5628d456c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628d454c080 0x5628d456be90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa44f7fe640 0 client.28082 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 --2- 172.21.15.16:0/773736838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628d454b760 0x5628d456b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 >> 172.21.15.16:0/773736838 conn(0x5628d4547cf0 msgr2=0x5628d4546e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 shutdown_connections 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 1 -- 172.21.15.16:0/773736838 wait complete. 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.228+0000 7fa4787e9740 20 client.28082 trim_cache size 0 max 16384 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 34, 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 54, 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "f59a6c7": { 2024-09-12T03:01:41.231 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 54, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:01:41.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 msgr2=0x7fa4580a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 0x7fa4580a50c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa468046000 tx=0x7fa468004430 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 msgr2=0x7fa470197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa46000e9f0 tx=0x7fa46000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 shutdown_connections 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa4580a2c00 0x7fa4580a50c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa470102ef0 0x7fa47019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa470102570 0x7fa470197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 --2- 172.21.15.16:0/2636219567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa470100e20 0x7fa470197440 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 >> 172.21.15.16:0/2636219567 conn(0x7fa4700fc7d0 msgr2=0x7fa4701068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 shutdown_connections 2024-09-12T03:01:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:41.229+0000 7fa476410640 1 -- 172.21.15.16:0/2636219567 wait complete. 2024-09-12T03:01:41.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:41 smithi016 ceph-mon[30925]: pgmap v750: 97 pgs: 97 active+clean; 471 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 15 MiB/s wr, 250 op/s 2024-09-12T03:01:41.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:41 smithi033 ceph-mon[35092]: pgmap v750: 97 pgs: 97 active+clean; 471 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 15 MiB/s wr, 250 op/s 2024-09-12T03:01:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:41 smithi184 ceph-mon[35321]: pgmap v750: 97 pgs: 97 active+clean; 471 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 15 MiB/s wr, 250 op/s 2024-09-12T03:01:42.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 12.44 MB/sec warmup 43 sec latency 1320.277 ms 2024-09-12T03:01:42.125 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f59a6c7', quiesce took 4.4 seconds. Will release after: 25.6 2024-09-12T03:01:43.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 12.15 MB/sec warmup 44 sec latency 2320.387 ms 2024-09-12T03:01:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:43 smithi016 ceph-mon[30925]: pgmap v751: 97 pgs: 97 active+clean; 466 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 15 MiB/s wr, 321 op/s 2024-09-12T03:01:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:43 smithi033 ceph-mon[35092]: pgmap v751: 97 pgs: 97 active+clean; 466 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 15 MiB/s wr, 321 op/s 2024-09-12T03:01:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:43 smithi184 ceph-mon[35321]: pgmap v751: 97 pgs: 97 active+clean; 466 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 15 MiB/s wr, 321 op/s 2024-09-12T03:01:44.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 11.88 MB/sec warmup 45 sec latency 3320.487 ms 2024-09-12T03:01:45.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 11.62 MB/sec warmup 46 sec latency 4320.552 ms 2024-09-12T03:01:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:45 smithi016 ceph-mon[30925]: pgmap v752: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 13 MiB/s wr, 312 op/s 2024-09-12T03:01:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:45 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:45 smithi033 ceph-mon[35092]: pgmap v752: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 13 MiB/s wr, 312 op/s 2024-09-12T03:01:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:45 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:45 smithi184 ceph-mon[35321]: pgmap v752: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 13 MiB/s wr, 312 op/s 2024-09-12T03:01:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:45 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:01:46.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 11.38 MB/sec warmup 47 sec latency 5320.652 ms 2024-09-12T03:01:47.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 11.14 MB/sec warmup 48 sec latency 6320.767 ms 2024-09-12T03:01:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:47 smithi016 ceph-mon[30925]: pgmap v753: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-12T03:01:47.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:47 smithi033 ceph-mon[35092]: pgmap v753: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-12T03:01:47.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:47 smithi184 ceph-mon[35321]: pgmap v753: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-12T03:01:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 10.91 MB/sec warmup 49 sec latency 7320.882 ms 2024-09-12T03:01:49.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 10.69 MB/sec warmup 50 sec latency 8321.011 ms 2024-09-12T03:01:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:49 smithi016 ceph-mon[30925]: pgmap v754: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 328 op/s 2024-09-12T03:01:49.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:49 smithi033 ceph-mon[35092]: pgmap v754: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 328 op/s 2024-09-12T03:01:49.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:49 smithi184 ceph-mon[35321]: pgmap v754: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 328 op/s 2024-09-12T03:01:50.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 10.49 MB/sec warmup 51 sec latency 9321.162 ms 2024-09-12T03:01:51.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 10.28 MB/sec warmup 52 sec latency 10321.236 ms 2024-09-12T03:01:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:51 smithi016 ceph-mon[30925]: pgmap v755: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 312 op/s 2024-09-12T03:01:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:51 smithi033 ceph-mon[35092]: pgmap v755: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 312 op/s 2024-09-12T03:01:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:51 smithi184 ceph-mon[35321]: pgmap v755: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 312 op/s 2024-09-12T03:01:52.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 10.09 MB/sec warmup 53 sec latency 11321.360 ms 2024-09-12T03:01:53.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.90 MB/sec warmup 54 sec latency 12321.417 ms 2024-09-12T03:01:53.597 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:53 smithi184 ceph-mon[35321]: pgmap v756: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-09-12T03:01:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:53 smithi016 ceph-mon[30925]: pgmap v756: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-09-12T03:01:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:53 smithi033 ceph-mon[35092]: pgmap v756: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-09-12T03:01:54.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.72 MB/sec warmup 55 sec latency 13321.531 ms 2024-09-12T03:01:54.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:01:55.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.55 MB/sec warmup 56 sec latency 14321.598 ms 2024-09-12T03:01:55.043 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:55.473 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:55 smithi016 ceph-mon[30925]: pgmap v757: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 415 KiB/s wr, 80 op/s 2024-09-12T03:01:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 -- 172.21.15.16:0/4294257072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 msgr2=0x7f16b0100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 --2- 172.21.15.16:0/4294257072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b0100cb0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009900 tx=0x7f16ac02a410 comp rx=0 tx=0).stop 2024-09-12T03:01:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 -- 172.21.15.16:0/4294257072 shutdown_connections 2024-09-12T03:01:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 --2- 172.21.15.16:0/4294257072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b010e480 0x7f16b0110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 --2- 172.21.15.16:0/4294257072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b0101280 0x7f16b010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 --2- 172.21.15.16:0/4294257072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b0100cb0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.556+0000 7f16b7290640 1 -- 172.21.15.16:0/4294257072 >> 172.21.15.16:0/4294257072 conn(0x7f16b00fc7a0 msgr2=0x7f16b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 -- 172.21.15.16:0/4294257072 shutdown_connections 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 -- 172.21.15.16:0/4294257072 wait complete. 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 Processor -- start 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 -- start start 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.557+0000 7f16b7290640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 0x7f16b019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010e480 0x7f16b019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01141b0 con 0x7f16b01008d0 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b0114030 con 0x7f16b010e480 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16b0114330 con 0x7f16b0101280 2024-09-12T03:01:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44140/0 (socket says 172.21.15.16:44140) 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 -- 172.21.15.16:0/927750357 learned_addr learned my addr 172.21.15.16:0/927750357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b482b640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 0x7f16b019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b582d640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010e480 0x7f16b019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 -- 172.21.15.16:0/927750357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 msgr2=0x7f16b019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 0x7f16b019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 -- 172.21.15.16:0/927750357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010e480 msgr2=0x7f16b019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010e480 0x7f16b019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 -- 172.21.15.16:0/927750357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b019f640 con 0x7f16b01008d0 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b482b640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 0x7f16b019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b502c640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f16ac006010 tx=0x7f16ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16a67fc640 1 -- 172.21.15.16:0/927750357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac03f030 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16a67fc640 1 -- 172.21.15.16:0/927750357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ac003ce0 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b019f8d0 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.558+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b01a7210 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.559+0000 7f16a67fc640 1 -- 172.21.15.16:0/927750357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac049060 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16a67fc640 1 -- 172.21.15.16:0/927750357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f16ac042030 con 0x7f16b01008d0 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 Processor -- start 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 -- start start 2024-09-12T03:01:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55643ce1f2e0 con 0x55643ceb94a0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55643cdc6ce0 con 0x55643ceb9dc0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.561+0000 7f16b9669740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55643ce1b2a0 con 0x55643cec4680 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16a67fc640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 0x7f169c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44146/0 (socket says 172.21.15.16:44146) 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 -- 172.21.15.16:0/2284683488 learned_addr learned my addr 172.21.15.16:0/2284683488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b502c640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 0x7f169c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b482b640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16a67fc640 1 -- 172.21.15.16:0/927750357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16ac0ef280 con 0x7f16b01008d0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 -- 172.21.15.16:0/2284683488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 msgr2=0x55643cec6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 -- 172.21.15.16:0/2284683488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 msgr2=0x55643cec4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 -- 172.21.15.16:0/2284683488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643cdab440 con 0x55643ceb94a0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b482b640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b502c640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f16a80092f0 tx=0x7f16a8012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741d3d4e86d4a38a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.562+0000 7f16b582d640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 0x7f169c0a5040 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f16a8012c60 tx=0x7f16a801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f169b7fe640 1 -- 172.21.15.16:0/2284683488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8004d20 con 0x55643ceb94a0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f169b7fe640 1 -- 172.21.15.16:0/2284683488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a8012e70 con 0x55643ceb94a0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f169b7fe640 1 -- 172.21.15.16:0/2284683488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a80057e0 con 0x55643ceb94a0 2024-09-12T03:01:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 -- 172.21.15.16:0/2284683488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 msgr2=0x55643ceb9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f16a80092f0 tx=0x7f16a8012760 comp rx=0 tx=0).stop 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 -- 172.21.15.16:0/2284683488 shutdown_connections 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 --2- 172.21.15.16:0/2284683488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643ceb9880 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 -- 172.21.15.16:0/2284683488 >> 172.21.15.16:0/2284683488 conn(0x55643ceb5a30 msgr2=0x55643ceb5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.563+0000 7f16b9669740 1 -- 172.21.15.16:0/2284683488 shutdown_connections 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 -- 172.21.15.16:0/2284683488 wait complete. 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 -- start start 2024-09-12T03:01:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 Processor -- start 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55643cec71f0 con 0x55643ceb94a0 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55643cdda050 con 0x55643ceb9dc0 2024-09-12T03:01:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55643cec7370 con 0x55643cec4680 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b9669740 10 client.0 _ll_register_callbacks cb 0x55643ce1a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.564+0000 7f16b502c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44156/0 (socket says 172.21.15.16:44156) 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 -- 172.21.15.16:0/3293355597 learned_addr learned my addr 172.21.15.16:0/3293355597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b482b640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b582d640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 -- 172.21.15.16:0/3293355597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 msgr2=0x55643cec1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 -- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 msgr2=0x55643cec12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 -- 172.21.15.16:0/3293355597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643cb903c0 con 0x55643ceb94a0 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b482b640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b502c640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0b84c0 tx=0x7f16ac02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c8bee4bbb9c7f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f1698ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0bb450 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16ac03bc50 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a8012e70 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f16ac03b580 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b9669740 10 client.18732 fetch_fsmap learned FSMap version 35 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.565+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55643ced2bc0 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a8012e70 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f16ac04c060 con 0x55643ceb94a0 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 10 client.18732 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 10 client.18732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 20 client.18732 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643ced3fd0 0x55643ced6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 4 client.18732 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55643ce2c850 con 0x55643ced3fd0 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b482b640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643ced3fd0 0x55643ced6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f16b482b640 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643ced3fd0 0x55643ced6400 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42cea4148db3368d server_cookie=942fb439bf062a50 in_seq=0 out_seq=0 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.566+0000 7f1698ff9640 10 client.18732 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f1698ff9640 1 -- 172.21.15.16:0/3293355597 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55643ce2c850 con 0x55643ced3fd0 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f1698ff9640 10 client.18732 handle_command_reply: tid=0 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 client.18732 shutdown 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 msgr2=0x55643cec3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0b84c0 tx=0x7f16ac02a410 comp rx=0 tx=0).stop 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 shutdown_connections 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643ced3fd0 0x55643ced6400 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643cec4680 0x55643cec1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cec12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 --2- 172.21.15.16:0/3293355597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb94a0 0x55643cec3ff0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f1698ff9640 0 client.18732 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.567+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 >> 172.21.15.16:0/3293355597 conn(0x55643ceb5a30 msgr2=0x55643cebb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.568+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 shutdown_connections 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.568+0000 7f16b9669740 1 -- 172.21.15.16:0/3293355597 wait complete. 2024-09-12T03:01:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.568+0000 7f16b9669740 20 client.18732 trim_cache size 0 max 16384 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 Processor -- start 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 -- start start 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643ced0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cebfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01af7d0 con 0x55643cec4680 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b13d0 con 0x55643ceb9dc0 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b9669740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01afea0 con 0x55643ceb94a0 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b582d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b582d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44158/0 (socket says 172.21.15.16:44158) 2024-09-12T03:01:55.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.585+0000 7f16b582d640 1 -- 172.21.15.16:0/993834736 learned_addr learned my addr 172.21.15.16:0/993834736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b502c640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643ced0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b482b640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cebfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 -- 172.21.15.16:0/993834736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 msgr2=0x55643ced0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643ced0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 -- 172.21.15.16:0/993834736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 msgr2=0x55643cebfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cebfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 -- 172.21.15.16:0/993834736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643cec0250 con 0x55643cec4680 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b502c640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643ced0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b482b640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cebfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:55.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f16b582d640 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f16a8020000 tx=0x7f16a8014d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82617e8393ee2dcb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f169affd640 1 -- 172.21.15.16:0/993834736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a802f070 con 0x55643cec4680 2024-09-12T03:01:55.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f169affd640 1 -- 172.21.15.16:0/993834736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a80049e0 con 0x55643cec4680 2024-09-12T03:01:55.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.586+0000 7f169affd640 1 -- 172.21.15.16:0/993834736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8005410 con 0x55643cec4680 2024-09-12T03:01:55.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- 172.21.15.16:0/993834736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 msgr2=0x55643cedbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f16a8020000 tx=0x7f16a8014d40 comp rx=0 tx=0).stop 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- 172.21.15.16:0/993834736 shutdown_connections 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643cec4680 0x55643cedbd10 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643ceb9dc0 0x55643cebfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 --2- 172.21.15.16:0/993834736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643ced0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- 172.21.15.16:0/993834736 >> 172.21.15.16:0/993834736 conn(0x55643ceb5a30 msgr2=0x55643cebc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- 172.21.15.16:0/993834736 shutdown_connections 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- 172.21.15.16:0/993834736 wait complete. 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 -- start start 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.587+0000 7f16b9669740 1 Processor -- start 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643cebe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643cec4680 0x55643cebedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b0d00 con 0x55643ceb9dc0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01b0630 con 0x55643cec4680 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16b01ad7b0 con 0x55643ceb94a0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b9669740 10 client.0 _ll_register_callbacks cb 0x7f16ac0a6040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44172/0 (socket says 172.21.15.16:44172) 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 -- 172.21.15.16:0/795135579 learned_addr learned my addr 172.21.15.16:0/795135579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b502c640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643cebe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b582d640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643cec4680 0x55643cebedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 -- 172.21.15.16:0/795135579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 msgr2=0x55643cebe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643cebe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 -- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643cec4680 msgr2=0x55643cebedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:55.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643cec4680 0x55643cebedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 -- 172.21.15.16:0/795135579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643cf3dd90 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f16b482b640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f16a000b390 tx=0x7f16a0013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6532252de9b7f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.588+0000 7f1698ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0004230 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f16a00043d0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a80049e0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f16a00035f0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 10 client.18738 fetch_fsmap learned FSMap version 35 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55643cf3f0e0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a80049e0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f16a001d9e0 con 0x55643ceb9dc0 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 10 client.18738 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:01:55.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 10 client.18738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 20 client.18738 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643cf3f890 0x55643cf41cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 4 client.18738 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.589+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55643ce26d60 con 0x55643cf3f890 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b502c640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643cf3f890 0x55643cf41cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b502c640 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643cf3f890 0x55643cf41cb0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef7bb9a789c828b4 server_cookie=c17805995792402a in_seq=0 out_seq=0 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f1698ff9640 10 client.18738 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f1698ff9640 1 -- 172.21.15.16:0/795135579 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55643ce26d60 con 0x55643cf3f890 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f1698ff9640 10 client.18738 handle_command_reply: tid=0 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 client.18738 shutdown 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 msgr2=0x55643cebe8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f16a000b390 tx=0x7f16a0013980 comp rx=0 tx=0).stop 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 shutdown_connections 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55643cf3f890 0x55643cf41cb0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55643cec4680 0x55643cebedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55643ceb9dc0 0x55643cebe8b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 --2- 172.21.15.16:0/795135579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55643ceb94a0 0x55643cebe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 >> 172.21.15.16:0/795135579 conn(0x55643ceb5a30 msgr2=0x55643cebc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f1698ff9640 0 client.18738 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.590+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 shutdown_connections 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b9669740 1 -- 172.21.15.16:0/795135579 wait complete. 2024-09-12T03:01:55.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b9669740 20 client.18738 trim_cache size 0 max 16384 2024-09-12T03:01:55.598 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:01:55.598 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:01:55.598 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:01:55.598 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 msgr2=0x7f169c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 0x7f169c0a5040 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f16a8012c60 tx=0x7f16a801e000 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 msgr2=0x7f16b019a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f16ac006010 tx=0x7f16ac0042f0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 shutdown_connections 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16b010e480 0x7f16b019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b0101280 0x7f16b019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f169c0a2b80 0x7f169c0a5040 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 --2- 172.21.15.16:0/927750357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b01008d0 0x7f16b019a690 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 >> 172.21.15.16:0/927750357 conn(0x7f16b00fc7a0 msgr2=0x7f16b00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 shutdown_connections 2024-09-12T03:01:55.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:55.591+0000 7f16b7290640 1 -- 172.21.15.16:0/927750357 wait complete. 2024-09-12T03:01:55.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:55 smithi033 ceph-mon[35092]: pgmap v757: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 415 KiB/s wr, 80 op/s 2024-09-12T03:01:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:55 smithi184 ceph-mon[35321]: pgmap v757: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 415 KiB/s wr, 80 op/s 2024-09-12T03:01:56.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.38 MB/sec warmup 57 sec latency 15321.680 ms 2024-09-12T03:01:56.443 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:01:56.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:01:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:57.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.22 MB/sec warmup 58 sec latency 16321.724 ms 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.163+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2754336337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100480 msgr2=0x7f6bf4100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.163+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2754336337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100480 0x7f6bf4100860 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000aa10 tx=0x7f6bf002b720 comp rx=0 tx=0).stop 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2754336337 shutdown_connections 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2754336337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 0x7f6bf410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2754336337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100e30 0x7f6bf4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2754336337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100480 0x7f6bf4100860 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2754336337 >> 172.21.15.16:0/2754336337 conn(0x7f6bf4077210 msgr2=0x7f6bf40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2754336337 shutdown_connections 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2754336337 wait complete. 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 Processor -- start 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.164+0000 7f6bfb9e9640 1 -- start start 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 0x7f6bf4196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 0x7f6bf4196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf410fdd0 con 0x7f6bf4100e30 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf410fc50 con 0x7f6bf4100480 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf410ff50 con 0x7f6bf41055b0 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44200/0 (socket says 172.21.15.16:44200) 2024-09-12T03:01:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 -- 172.21.15.16:0/2234916458 learned_addr learned my addr 172.21.15.16:0/2234916458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf9f86640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 0x7f6bf4196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf9785640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 0x7f6bf4196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 -- 172.21.15.16:0/2234916458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 msgr2=0x7f6bf4196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 0x7f6bf4196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 -- 172.21.15.16:0/2234916458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 msgr2=0x7f6bf4196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 0x7f6bf4196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 -- 172.21.15.16:0/2234916458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf41a69a0 con 0x7f6bf4100e30 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf9785640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 0x7f6bf4196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bf8f84640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b4a0 tx=0x7f6be400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4014070 con 0x7f6bf4100e30 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6be4010af0 con 0x7f6bf4100e30 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf41a6c90 con 0x7f6bf4100e30 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4017d10 con 0x7f6bf4100e30 2024-09-12T03:01:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.165+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf41a7180 con 0x7f6bf4100e30 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.167+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf4100860 con 0x7f6bf4100e30 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.168+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6be4017070 con 0x7f6bf4100e30 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.168+0000 7f6bea7fc640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 0x7f6be00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.168+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6be40ce450 con 0x7f6bf4100e30 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.168+0000 7f6bf9785640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 0x7f6be00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.169+0000 7f6bf9785640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 0x7f6be00a5040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0000c00 tx=0x7f6bf0030d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.173+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6be409b3c0 con 0x7f6bf4100e30 2024-09-12T03:01:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.385+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf4108790 con 0x7f6bf4100e30 2024-09-12T03:01:57.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:57 smithi016 ceph-mon[30925]: pgmap v758: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 332 KiB/s wr, 74 op/s 2024-09-12T03:01:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.386+0000 7f6bea7fc640 1 -- 172.21.15.16:0/2234916458 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11983 (secure 0 0 0) 0x7f6be409acc0 con 0x7f6bf4100e30 2024-09-12T03:01:57.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:01:57.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"btime":"2024-09-12T03:01:44:300565+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:43.958785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:01:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.388+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 msgr2=0x7f6be00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.388+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 0x7f6be00a5040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0000c00 tx=0x7f6bf0030d70 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.388+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 msgr2=0x7f6bf4196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.388+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b4a0 tx=0x7f6be400b960 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 shutdown_connections 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bf41055b0 0x7f6bf4196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6be00a2b80 0x7f6be00a5040 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bf4100e30 0x7f6bf4196640 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 --2- 172.21.15.16:0/2234916458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6bf4100480 0x7f6bf4196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 >> 172.21.15.16:0/2234916458 conn(0x7f6bf4077210 msgr2=0x7f6bf4103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 shutdown_connections 2024-09-12T03:01:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:57.389+0000 7f6bfb9e9640 1 -- 172.21.15.16:0/2234916458 wait complete. 2024-09-12T03:01:57.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:57 smithi033 ceph-mon[35092]: pgmap v758: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 332 KiB/s wr, 74 op/s 2024-09-12T03:01:57.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:57 smithi184 ceph-mon[35321]: pgmap v758: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 332 KiB/s wr, 74 op/s 2024-09-12T03:01:58.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 9.06 MB/sec warmup 59 sec latency 17321.794 ms 2024-09-12T03:01:58.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:01:58.279 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:58.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:58 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2234916458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:58.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 -- 172.21.15.16:0/705102147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101440 msgr2=0x7f322c101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 --2- 172.21.15.16:0/705102147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101440 0x7f322c101820 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3228009900 tx=0x7f322802a410 comp rx=0 tx=0).stop 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 -- 172.21.15.16:0/705102147 shutdown_connections 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 --2- 172.21.15.16:0/705102147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c106050 0x7f322c10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 --2- 172.21.15.16:0/705102147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c101df0 0x7f322c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 --2- 172.21.15.16:0/705102147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101440 0x7f322c101820 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 -- 172.21.15.16:0/705102147 >> 172.21.15.16:0/705102147 conn(0x7f322c077210 msgr2=0x7f322c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 -- 172.21.15.16:0/705102147 shutdown_connections 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.689+0000 7f3233fe0640 1 -- 172.21.15.16:0/705102147 wait complete. 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 Processor -- start 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 -- start start 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c101440 0x7f322c1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c106050 0x7f322c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1110c0 con 0x7f322c101df0 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f322c110f40 con 0x7f322c106050 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3233fe0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f322c111240 con 0x7f322c101440 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f323157b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f323157b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44226/0 (socket says 172.21.15.16:44226) 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f323157b640 1 -- 172.21.15.16:0/3024282890 learned_addr learned my addr 172.21.15.16:0/3024282890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.690+0000 7f3231d7c640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c101440 0x7f322c1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323257d640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c106050 0x7f322c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 -- 172.21.15.16:0/3024282890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c101440 msgr2=0x7f322c1a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c101440 0x7f322c1a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 -- 172.21.15.16:0/3024282890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c106050 msgr2=0x7f322c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c106050 0x7f322c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 -- 172.21.15.16:0/3024282890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c19af10 con 0x7f322c101df0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f323157b640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f321c00d950 tx=0x7f321c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f3222ffd640 1 -- 172.21.15.16:0/3024282890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c017070 con 0x7f322c101df0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c19b200 con 0x7f322c101df0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f3222ffd640 1 -- 172.21.15.16:0/3024282890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c009c30 con 0x7f322c101df0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f3222ffd640 1 -- 172.21.15.16:0/3024282890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c00f6b0 con 0x7f322c101df0 2024-09-12T03:01:58.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.691+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c1a7550 con 0x7f322c101df0 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f3222ffd640 1 -- 172.21.15.16:0/3024282890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f321c0109d0 con 0x7f322c101df0 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 Processor -- start 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 -- start start 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c40090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c4d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562621ba62f0 con 0x562621c405d0 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562621b4dcf0 con 0x562621c4b370 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f32363b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562621ba22b0 con 0x562621c41b40 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44236/0 (socket says 172.21.15.16:44236) 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f3231d7c640 1 -- 172.21.15.16:0/3588027680 learned_addr learned my addr 172.21.15.16:0/3588027680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f323257d640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c40090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.694+0000 7f323157b640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c4d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 -- 172.21.15.16:0/3588027680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 msgr2=0x562621c40090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c40090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 -- 172.21.15.16:0/3588027680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 msgr2=0x562621c4d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c4d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 -- 172.21.15.16:0/3588027680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562621b323a0 con 0x562621c405d0 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f323257d640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c40090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3222ffd640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 0x7f32180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3231d7c640 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3228031550 tx=0x7f32280041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a80639fdc4424a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f323257d640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 0x7f32180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3222ffd640 1 -- 172.21.15.16:0/3024282890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f321c0cebf0 con 0x7f322c101df0 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3217fff640 1 -- 172.21.15.16:0/3588027680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228035040 con 0x562621c405d0 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3217fff640 1 -- 172.21.15.16:0/3588027680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3228005520 con 0x562621c405d0 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.695+0000 7f3217fff640 1 -- 172.21.15.16:0/3588027680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228003850 con 0x562621c405d0 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f323257d640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 0x7f32180a5040 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3224010540 tx=0x7f322400a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 -- 172.21.15.16:0/3588027680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 msgr2=0x562621c40a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3228031550 tx=0x7f32280041a0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 -- 172.21.15.16:0/3588027680 shutdown_connections 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c4d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c40a10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 --2- 172.21.15.16:0/3588027680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c40090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 -- 172.21.15.16:0/3588027680 >> 172.21.15.16:0/3588027680 conn(0x562621c09310 msgr2=0x562621c3cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 -- 172.21.15.16:0/3588027680 shutdown_connections 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.696+0000 7f32363b9740 1 -- 172.21.15.16:0/3588027680 wait complete. 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 -- start start 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 Processor -- start 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c41b40 0x562621c48040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 0x562621c48650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562621c4e010 con 0x562621c405d0 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562621b61060 con 0x562621c41b40 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562621c4e190 con 0x562621c4b370 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.697+0000 7f32363b9740 10 client.0 _ll_register_callbacks cb 0x5626218b48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44240/0 (socket says 172.21.15.16:44240) 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 -- 172.21.15.16:0/1936730308 learned_addr learned my addr 172.21.15.16:0/1936730308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f323257d640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 0x562621c48650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f323157b640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c41b40 0x562621c48040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 -- 172.21.15.16:0/1936730308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 msgr2=0x562621c48650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 0x562621c48650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 -- 172.21.15.16:0/1936730308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c41b40 msgr2=0x562621c48040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c41b40 0x562621c48040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 -- 172.21.15.16:0/1936730308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626219173c0 con 0x562621c405d0 2024-09-12T03:01:58.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f323257d640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 0x562621c48650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f322802a930 tx=0x7f32280051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d0282552ba3726 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f32157fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228035070 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f32280048d0 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3228005520 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.698+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3228049070 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 10 client.18756 fetch_fsmap learned FSMap version 35 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562621c59b10 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3228005520 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f32280035f0 con 0x562621c405d0 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 10 client.18756 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 10 client.18756 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 20 client.18756 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:58.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621c5ae60 0x562621c5d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 4 client.18756 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:01:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562621bb3860 con 0x562621c5ae60 2024-09-12T03:01:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f323157b640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621c5ae60 0x562621c5d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.699+0000 7f32157fa640 10 client.18756 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:58.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f323157b640 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621c5ae60 0x562621c5d250 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3caa72a841f21fbc server_cookie=48cdadc54b615b38 in_seq=0 out_seq=0 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32157fa640 1 -- 172.21.15.16:0/1936730308 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562621bb3860 con 0x562621c5ae60 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32157fa640 10 client.18756 handle_command_reply: tid=0 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 client.18756 shutdown 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 msgr2=0x562621c4ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f322802a930 tx=0x7f32280051d0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 shutdown_connections 2024-09-12T03:01:58.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c4b370 0x562621c48650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621c5ae60 0x562621c5d250 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c41b40 0x562621c48040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 --2- 172.21.15.16:0/1936730308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c4ad90 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32157fa640 0 client.18756 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.700+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 >> 172.21.15.16:0/1936730308 conn(0x562621c09310 msgr2=0x562621c42930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.701+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 shutdown_connections 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.701+0000 7f32363b9740 1 -- 172.21.15.16:0/1936730308 wait complete. 2024-09-12T03:01:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.701+0000 7f32363b9740 20 client.18756 trim_cache size 0 max 16384 2024-09-12T03:01:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 Processor -- start 2024-09-12T03:01:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 -- start start 2024-09-12T03:01:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c3c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1af790 con 0x562621c405d0 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b1390 con 0x562621c4b370 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f32363b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1afe60 con 0x562621c41b40 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44244/0 (socket says 172.21.15.16:44244) 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 -- 172.21.15.16:0/1723895577 learned_addr learned my addr 172.21.15.16:0/1723895577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f323157b640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c3c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f323257d640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 -- 172.21.15.16:0/1723895577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 msgr2=0x562621c3c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c3c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 -- 172.21.15.16:0/1723895577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 msgr2=0x562621c3cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c3cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f3231d7c640 1 -- 172.21.15.16:0/1723895577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562621c3d960 con 0x562621c405d0 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.714+0000 7f323257d640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c3cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f3231d7c640 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a38e33950bca28 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32177fe640 1 -- 172.21.15.16:0/1723895577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228035070 con 0x562621c405d0 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32177fe640 1 -- 172.21.15.16:0/1723895577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3228004db0 con 0x562621c405d0 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32177fe640 1 -- 172.21.15.16:0/1723895577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280315b0 con 0x562621c405d0 2024-09-12T03:01:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 -- 172.21.15.16:0/1723895577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 msgr2=0x562621c57460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228031400 comp rx=0 tx=0).stop 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 -- 172.21.15.16:0/1723895577 shutdown_connections 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c3cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c41b40 0x562621c3c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 --2- 172.21.15.16:0/1723895577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c405d0 0x562621c57460 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.715+0000 7f32363b9740 1 -- 172.21.15.16:0/1723895577 >> 172.21.15.16:0/1723895577 conn(0x562621c09310 msgr2=0x562621c37a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 -- 172.21.15.16:0/1723895577 shutdown_connections 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 -- 172.21.15.16:0/1723895577 wait complete. 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 -- start start 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 Processor -- start 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c405d0 0x562621c60a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c5e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b0cc0 con 0x562621c41b40 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.716+0000 7f32363b9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b05f0 con 0x562621c4b370 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32363b9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ad770 con 0x562621c405d0 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32363b9740 10 client.0 _ll_register_callbacks cb 0x7f321c0d25a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:01:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44246/0 (socket says 172.21.15.16:44246) 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 -- 172.21.15.16:0/2815787447 learned_addr learned my addr 172.21.15.16:0/2815787447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323257d640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c5e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f3231d7c640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c405d0 0x562621c60a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 -- 172.21.15.16:0/2815787447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c405d0 msgr2=0x562621c60a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c405d0 0x562621c60a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 -- 172.21.15.16:0/2815787447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 msgr2=0x562621c5e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c5e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 -- 172.21.15.16:0/2815787447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562621c48fe0 con 0x562621c41b40 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f323157b640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f321c097970 tx=0x7f321c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d955fd35d60450 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32157fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c0d2040 con 0x562621c41b40 2024-09-12T03:01:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f321c099810 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3228004db0 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f321c020040 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32363b9740 10 client.18762 fetch_fsmap learned FSMap version 35 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.717+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562621cc5570 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3228004db0 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f321c0d5070 con 0x562621c41b40 2024-09-12T03:01:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 10 client.18762 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 10 client.18762 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 20 client.18762 populate_metadata read hostname 'smithi016' 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621cc5ca0 0x562621cc80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 4 client.18762 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562621badd70 con 0x562621cc5ca0 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f3231d7c640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621cc5ca0 0x562621cc80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f32157fa640 10 client.18762 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.718+0000 7f3231d7c640 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621cc5ca0 0x562621cc80c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7397834571fa4167 server_cookie=b75dbc5d787f14ce in_seq=0 out_seq=0 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32157fa640 1 -- 172.21.15.16:0/2815787447 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562621badd70 con 0x562621cc5ca0 2024-09-12T03:01:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32157fa640 10 client.18762 handle_command_reply: tid=0 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 client.18762 shutdown 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 msgr2=0x562621c5dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f321c097970 tx=0x7f321c005880 comp rx=0 tx=0).stop 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 shutdown_connections 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562621cc5ca0 0x562621cc80c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562621c4b370 0x562621c5e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562621c41b40 0x562621c5dd60 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32157fa640 0 client.18762 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 --2- 172.21.15.16:0/2815787447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562621c405d0 0x562621c60a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 >> 172.21.15.16:0/2815787447 conn(0x562621c09310 msgr2=0x562621c3c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 shutdown_connections 2024-09-12T03:01:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 1 -- 172.21.15.16:0/2815787447 wait complete. 2024-09-12T03:01:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.719+0000 7f32363b9740 20 client.18762 trim_cache size 0 max 16384 2024-09-12T03:01:58.736 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:01:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 msgr2=0x7f32180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 0x7f32180a5040 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3224010540 tx=0x7f322400a4d0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 msgr2=0x7f322c1a0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f321c00d950 tx=0x7f321c00de10 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 shutdown_connections 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f322c106050 0x7f322c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f32180a2b80 0x7f32180a5040 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c101df0 0x7f322c1a0b10 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 --2- 172.21.15.16:0/3024282890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322c101440 0x7f322c1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 >> 172.21.15.16:0/3024282890 conn(0x7f322c077210 msgr2=0x7f322c103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 shutdown_connections 2024-09-12T03:01:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:01:58.720+0000 7f3233fe0640 1 -- 172.21.15.16:0/3024282890 wait complete. 2024-09-12T03:01:58.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:58 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2234916458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:58.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:58 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2234916458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:01:59.005 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.91 MB/sec warmup 60 sec latency 18321.877 ms 2024-09-12T03:01:59.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:01:59 smithi016 ceph-mon[30925]: pgmap v759: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-09-12T03:01:59.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:01:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:01:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:01:59 smithi033 ceph-mon[35092]: pgmap v759: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-09-12T03:01:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:01:59 smithi184 ceph-mon[35321]: pgmap v759: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-09-12T03:02:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.77 MB/sec warmup 61 sec latency 19321.974 ms 2024-09-12T03:02:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 -- 172.21.15.16:0/2686880746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc102790 msgr2=0x7f14dc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 --2- 172.21.15.16:0/2686880746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc102790 0x7f14dc1067e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc02a410 comp rx=0 tx=0).stop 2024-09-12T03:02:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 -- 172.21.15.16:0/2686880746 shutdown_connections 2024-09-12T03:02:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 --2- 172.21.15.16:0/2686880746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc106f10 0x7f14dc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 --2- 172.21.15.16:0/2686880746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc102790 0x7f14dc1067e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 --2- 172.21.15.16:0/2686880746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 -- 172.21.15.16:0/2686880746 >> 172.21.15.16:0/2686880746 conn(0x7f14dc0fd8c0 msgr2=0x7f14dc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 -- 172.21.15.16:0/2686880746 shutdown_connections 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.091+0000 7f14e3552640 1 -- 172.21.15.16:0/2686880746 wait complete. 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 Processor -- start 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 -- start start 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc102790 0x7f14dc19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc111f70 con 0x7f14dc106f10 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc111df0 con 0x7f14dc101de0 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e3552640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1120f0 con 0x7f14dc102790 2024-09-12T03:02:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e1aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e12ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e1aef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44256/0 (socket says 172.21.15.16:44256) 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e1aef640 1 -- 172.21.15.16:0/1055969220 learned_addr learned my addr 172.21.15.16:0/1055969220 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.092+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc102790 0x7f14dc19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 -- 172.21.15.16:0/1055969220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc102790 msgr2=0x7f14dc19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc102790 0x7f14dc19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 -- 172.21.15.16:0/1055969220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 msgr2=0x7f14dc19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 -- 172.21.15.16:0/1055969220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14dc19f5c0 con 0x7f14dc106f10 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e1aef640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f14d400eaa0 tx=0x7f14d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14d27fc640 1 -- 172.21.15.16:0/1055969220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d400cdb0 con 0x7f14dc106f10 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14d27fc640 1 -- 172.21.15.16:0/1055969220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d4005020 con 0x7f14dc106f10 2024-09-12T03:02:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14dc19f8b0 con 0x7f14dc106f10 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14d27fc640 1 -- 172.21.15.16:0/1055969220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d40116b0 con 0x7f14dc106f10 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.093+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14dc1a7190 con 0x7f14dc106f10 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14d27fc640 1 -- 172.21.15.16:0/1055969220 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f14d4005560 con 0x7f14dc106f10 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 Processor -- start 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 -- start start 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x561313086090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313086a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561312fec2f0 con 0x561313091370 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561312f93cf0 con 0x5613130865d0 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14e592b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561312fe82b0 con 0x561313087b40 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.096+0000 7f14d27fc640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 0x7f14c80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e1aef640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 0x7f14c80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44272/0 (socket says 172.21.15.16:44272) 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 -- 172.21.15.16:0/4220053540 learned_addr learned my addr 172.21.15.16:0/4220053540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14d27fc640 1 -- 172.21.15.16:0/1055969220 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14d40cf110 con 0x7f14dc106f10 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e12ee640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313086a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x561313086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 -- 172.21.15.16:0/4220053540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 msgr2=0x561313086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x561313086090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 -- 172.21.15.16:0/4220053540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 msgr2=0x561313086a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313086a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 -- 172.21.15.16:0/4220053540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561312f783a0 con 0x561313091370 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e12ee640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313086a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e1aef640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 0x7f14c80a50c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000f80 tx=0x7f14d4086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14e0aed640 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f14cc031550 tx=0x7f14cc0314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5692a9a9933826e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.097+0000 7f14c77fe640 1 -- 172.21.15.16:0/4220053540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035070 con 0x561313091370 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14c77fe640 1 -- 172.21.15.16:0/4220053540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005020 con 0x561313091370 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14c77fe640 1 -- 172.21.15.16:0/4220053540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004a70 con 0x561313091370 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 -- 172.21.15.16:0/4220053540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 msgr2=0x561313093760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f14cc031550 tx=0x7f14cc0314e0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 -- 172.21.15.16:0/4220053540 shutdown_connections 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x561313093760 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313086a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 --2- 172.21.15.16:0/4220053540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x561313086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 -- 172.21.15.16:0/4220053540 >> 172.21.15.16:0/4220053540 conn(0x56131304f310 msgr2=0x561313082c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 -- 172.21.15.16:0/4220053540 shutdown_connections 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.098+0000 7f14e592b740 1 -- 172.21.15.16:0/4220053540 wait complete. 2024-09-12T03:02:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 -- start start 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 Processor -- start 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x561313090d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 0x56131308e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561313094010 con 0x561313091370 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561312fa7060 con 0x561313087b40 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561313094190 con 0x5613130865d0 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e592b740 10 client.0 _ll_register_callbacks cb 0x561312cfa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44286/0 (socket says 172.21.15.16:44286) 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e0aed640 1 -- 172.21.15.16:0/1090652179 learned_addr learned my addr 172.21.15.16:0/1090652179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.099+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x561313090d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 0x56131308e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 -- 172.21.15.16:0/1090652179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 msgr2=0x561313090d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x561313090d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 -- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 msgr2=0x56131308e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 0x56131308e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 -- 172.21.15.16:0/1090652179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561312d5d3c0 con 0x561313091370 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 0x56131308e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02a930 tx=0x7f14cc041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2268bfb3905b3d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035030 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14cc005020 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14cc004a10 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f14cc005610 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e592b740 10 client.18774 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56131309fab0 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.100+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14cc005020 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f14cc059020 con 0x561313091370 2024-09-12T03:02:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 10 client.18774 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 10 client.18774 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 20 client.18774 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5613130a0e00 0x5613130a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 4 client.18774 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561312ff9860 con 0x5613130a0e00 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5613130a0e00 0x5613130a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14c4ff9640 10 client.18774 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.101+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5613130a0e00 0x5613130a31f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dfd1f66223c9160c server_cookie=7bffe5e6ca9f7136 in_seq=0 out_seq=0 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14c4ff9640 1 -- 172.21.15.16:0/1090652179 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561312ff9860 con 0x5613130a0e00 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14c4ff9640 10 client.18774 handle_command_reply: tid=0 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 client.18774 shutdown 2024-09-12T03:02:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 msgr2=0x56131308e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02a930 tx=0x7f14cc041040 comp rx=0 tx=0).stop 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 shutdown_connections 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5613130a0e00 0x5613130a31f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x56131308e610 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313087b40 0x56131308e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14c4ff9640 0 client.18774 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 --2- 172.21.15.16:0/1090652179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x561313090d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 >> 172.21.15.16:0/1090652179 conn(0x56131304f310 msgr2=0x561313088960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.102+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 shutdown_connections 2024-09-12T03:02:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.103+0000 7f14e592b740 1 -- 172.21.15.16:0/1090652179 wait complete. 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.103+0000 7f14e592b740 20 client.18774 trim_cache size 0 max 16384 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 Processor -- start 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 -- start start 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313082e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1af7a0 con 0x561313091370 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1b13a0 con 0x5613130865d0 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e592b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1afe70 con 0x561313087b40 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.116+0000 7f14e12ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44300/0 (socket says 172.21.15.16:44300) 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 -- 172.21.15.16:0/1823630426 learned_addr learned my addr 172.21.15.16:0/1823630426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e12ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43830/0 (socket says 172.21.15.16:43830) 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 -- 172.21.15.16:0/1823630426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 msgr2=0x5613130a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 -- 172.21.15.16:0/1823630426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 msgr2=0x561313082e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313082e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 -- 172.21.15.16:0/1823630426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613130862d0 con 0x561313091370 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e12ee640 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14e0aed640 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f14cc031720 tx=0x7f14cc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824e6c9f4c43142a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14c6ffd640 1 -- 172.21.15.16:0/1823630426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035070 con 0x561313091370 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14c6ffd640 1 -- 172.21.15.16:0/1823630426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14cc051040 con 0x561313091370 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.117+0000 7f14c6ffd640 1 -- 172.21.15.16:0/1823630426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004a40 con 0x561313091370 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 -- 172.21.15.16:0/1823630426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 msgr2=0x5613130a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f14cc031720 tx=0x7f14cc005d50 comp rx=0 tx=0).stop 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 -- 172.21.15.16:0/1823630426 shutdown_connections 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313091370 0x5613130a6750 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561313087b40 0x5613130a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 --2- 172.21.15.16:0/1823630426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5613130865d0 0x561313082e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 -- 172.21.15.16:0/1823630426 >> 172.21.15.16:0/1823630426 conn(0x56131304f310 msgr2=0x561313082700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 -- 172.21.15.16:0/1823630426 shutdown_connections 2024-09-12T03:02:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.118+0000 7f14e592b740 1 -- 172.21.15.16:0/1823630426 wait complete. 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 -- start start 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 Processor -- start 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x5613130a40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313091370 0x5613130a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1b0cd0 con 0x561313087b40 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1b0600 con 0x561313091370 2024-09-12T03:02:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc1ad780 con 0x5613130865d0 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e592b740 10 client.0 _ll_register_callbacks cb 0x7f14d800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e12ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e12ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44302/0 (socket says 172.21.15.16:44302) 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e12ee640 1 -- 172.21.15.16:0/51370362 learned_addr learned my addr 172.21.15.16:0/51370362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e0aed640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x5613130a40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.119+0000 7f14e0aed640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313091370 0x5613130a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 -- 172.21.15.16:0/51370362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 msgr2=0x5613130a40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x5613130a40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 -- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313091370 msgr2=0x5613130a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313091370 0x5613130a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 -- 172.21.15.16:0/51370362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56131310a810 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e12ee640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f14d8005d00 tx=0x7f14d8011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a132a4ac82ea91e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8010b90 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14cc051040 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f14d8010d30 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f14d8005020 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e592b740 10 client.18780 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56131310bb80 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.120+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14cc051040 con 0x561313087b40 2024-09-12T03:02:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f14d8030430 con 0x561313087b40 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 10 client.18780 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 10 client.18780 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 20 client.18780 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56131310c3f0 0x56131310e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 4 client.18780 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561312ff3d70 con 0x56131310c3f0 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e0aed640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56131310c3f0 0x56131310e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14e0aed640 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56131310c3f0 0x56131310e810 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e873db2bfaa89f4b server_cookie=51ff20a7417681a0 in_seq=0 out_seq=0 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.121+0000 7f14c4ff9640 10 client.18780 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14c4ff9640 1 -- 172.21.15.16:0/51370362 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561312ff3d70 con 0x56131310c3f0 2024-09-12T03:02:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14c4ff9640 10 client.18780 handle_command_reply: tid=0 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 client.18780 shutdown 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 msgr2=0x5613130a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f14d8005d00 tx=0x7f14d8011ea0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 shutdown_connections 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56131310c3f0 0x56131310e810 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561313091370 0x5613130a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561313087b40 0x5613130a45f0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 --2- 172.21.15.16:0/51370362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5613130865d0 0x5613130a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 >> 172.21.15.16:0/51370362 conn(0x56131304f310 msgr2=0x5613130880e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14c4ff9640 0 client.18780 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 shutdown_connections 2024-09-12T03:02:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 1 -- 172.21.15.16:0/51370362 wait complete. 2024-09-12T03:02:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.122+0000 7f14e592b740 20 client.18780 trim_cache size 0 max 16384 2024-09-12T03:02:00.140 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 msgr2=0x7f14c80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 0x7f14c80a50c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000f80 tx=0x7f14d4086040 comp rx=0 tx=0).stop 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 msgr2=0x7f14dc19eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f14d400eaa0 tx=0x7f14d400ef60 comp rx=0 tx=0).stop 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 shutdown_connections 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f14c80a2c00 0x7f14c80a50c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14dc106f10 0x7f14dc19eee0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14dc102790 0x7f14dc19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 --2- 172.21.15.16:0/1055969220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f14dc101de0 0x7f14dc19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 >> 172.21.15.16:0/1055969220 conn(0x7f14dc0fd8c0 msgr2=0x7f14dc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 shutdown_connections 2024-09-12T03:02:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:00.123+0000 7f14e3552640 1 -- 172.21.15.16:0/1055969220 wait complete. 2024-09-12T03:02:00.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:02:01.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.62 MB/sec warmup 62 sec latency 20322.290 ms 2024-09-12T03:02:01.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:01.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:01 smithi016 ceph-mon[30925]: pgmap v760: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-09-12T03:02:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 -- 172.21.15.16:0/3378279668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4100e20 msgr2=0x7fd1d4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 --2- 172.21.15.16:0/3378279668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4100e20 0x7fd1d4101200 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009900 tx=0x7fd1c802a480 comp rx=0 tx=0).stop 2024-09-12T03:02:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 -- 172.21.15.16:0/3378279668 shutdown_connections 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 --2- 172.21.15.16:0/3378279668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4102ef0 0x7fd1d410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 --2- 172.21.15.16:0/3378279668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 --2- 172.21.15.16:0/3378279668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4100e20 0x7fd1d4101200 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 -- 172.21.15.16:0/3378279668 >> 172.21.15.16:0/3378279668 conn(0x7fd1d40fc7d0 msgr2=0x7fd1d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 -- 172.21.15.16:0/3378279668 shutdown_connections 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 -- 172.21.15.16:0/3378279668 wait complete. 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.639+0000 7fd1da022640 1 Processor -- start 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 -- start start 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4100e20 0x7fd1d41993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4110fa0 con 0x7fd1d4102ef0 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4110e20 con 0x7fd1d4102570 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1da022640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4111120 con 0x7fd1d4100e20 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44326/0 (socket says 172.21.15.16:44326) 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 -- 172.21.15.16:0/4159259258 learned_addr learned my addr 172.21.15.16:0/4159259258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4100e20 0x7fd1d41993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 -- 172.21.15.16:0/4159259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4100e20 msgr2=0x7fd1d41993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4100e20 0x7fd1d41993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 -- 172.21.15.16:0/4159259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 msgr2=0x7fd1d41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d3fff640 1 -- 172.21.15.16:0/4159259258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d41a4970 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.640+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1d3fff640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00cce0 tx=0x7fd1cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1d0ff9640 1 -- 172.21.15.16:0/4159259258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00f450 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1d0ff9640 1 -- 172.21.15.16:0/4159259258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc00fad0 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d41a4b40 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1d0ff9640 1 -- 172.21.15.16:0/4159259258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc018c10 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.641+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d41a5030 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.643+0000 7fd1d0ff9640 1 -- 172.21.15.16:0/4159259258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd1cc00f5f0 con 0x7fd1d4102ef0 2024-09-12T03:02:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.643+0000 7fd1dc3fb740 1 Processor -- start 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 -- start start 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11129a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa11136760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa1108f2f0 con 0x55aa1112ab40 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa11036cf0 con 0x55aa111295d0 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa1108b2b0 con 0x55aa11134370 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44338/0 (socket says 172.21.15.16:44338) 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2236429373 learned_addr learned my addr 172.21.15.16:0/2236429373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:01.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11129a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d0ff9640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 0x7fd1bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa11136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d3fff640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 0x7fd1bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2236429373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 msgr2=0x55aa11136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa11136760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2236429373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 msgr2=0x55aa11129a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11129a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2236429373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa1101b3a0 con 0x55aa1112ab40 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.644+0000 7fd1d0ff9640 1 -- 172.21.15.16:0/4159259258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd1cc0cf800 con 0x7fd1d4102ef0 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.645+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4006fd0 tx=0x7fd1c4007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44b095e237f2902 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.645+0000 7fd1d3fff640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 0x7fd1bc0a5040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc06f0a0 tx=0x7fd1cc06f0d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.645+0000 7fd1b9ffb640 1 -- 172.21.15.16:0/2236429373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c401c070 con 0x55aa1112ab40 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.645+0000 7fd1b9ffb640 1 -- 172.21.15.16:0/2236429373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c4010040 con 0x55aa1112ab40 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.645+0000 7fd1b9ffb640 1 -- 172.21.15.16:0/2236429373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4014840 con 0x55aa1112ab40 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2236429373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 msgr2=0x55aa11129090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4006fd0 tx=0x7fd1c4007420 comp rx=0 tx=0).stop 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2236429373 shutdown_connections 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa11136760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11129a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2236429373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa11129090 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2236429373 >> 172.21.15.16:0/2236429373 conn(0x55aa110f2310 msgr2=0x55aa11125c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2236429373 shutdown_connections 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2236429373 wait complete. 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 -- start start 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.646+0000 7fd1dc3fb740 1 Processor -- start 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa111295d0 0x55aa11133d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 0x55aa11130fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa11137070 con 0x55aa11134370 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aa1104a060 con 0x55aa1112ab40 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa111371f0 con 0x55aa111295d0 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1dc3fb740 10 client.0 _ll_register_callbacks cb 0x55aa10d9d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44350/0 (socket says 172.21.15.16:44350) 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2949840740 learned_addr learned my addr 172.21.15.16:0/2949840740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa111295d0 0x55aa11133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 0x55aa11130fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa111295d0 msgr2=0x55aa11133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa111295d0 0x55aa11133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2949840740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 msgr2=0x55aa11130fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 0x55aa11130fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/2949840740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa10e003c0 con 0x55aa11134370 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.647+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 0x55aa11130fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4014520 tx=0x7fd1c40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ea906931b57e3f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c402a020 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd1c4017600 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c4010040 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd1c401c040 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 10 client.18792 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aa11142ab0 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c4010040 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fd1c4022070 con 0x55aa11134370 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 10 client.18792 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 10 client.18792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:02:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 20 client.18792 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa11143f10 0x55aa11146300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 4 client.18792 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.648+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa1109c860 con 0x55aa11143f10 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.649+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa11143f10 0x55aa11146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.649+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa11143f10 0x55aa11146300 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9af764376408c43 server_cookie=cc5b5412ce6f26f4 in_seq=0 out_seq=0 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.649+0000 7fd1b2ffd640 10 client.18792 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.650+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/2949840740 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa1109c860 con 0x55aa11143f10 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.650+0000 7fd1b2ffd640 10 client.18792 handle_command_reply: tid=0 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.650+0000 7fd1dc3fb740 1 client.18792 shutdown 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.650+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 msgr2=0x55aa111315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.650+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4014520 tx=0x7fd1c40047d0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 shutdown_connections 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa111315d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa1112ab40 0x55aa11130fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa11143f10 0x55aa11146300 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/2949840740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa111295d0 0x55aa11133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 >> 172.21.15.16:0/2949840740 conn(0x55aa110f2310 msgr2=0x55aa1112b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1b2ffd640 0 client.18792 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 shutdown_connections 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/2949840740 wait complete. 2024-09-12T03:02:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.651+0000 7fd1dc3fb740 20 client.18792 trim_cache size 0 max 16384 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 Processor -- start 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 -- start start 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11140310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa1112c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ad560 con 0x55aa1112ab40 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41af160 con 0x55aa111295d0 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41adc30 con 0x55aa11134370 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44360/0 (socket says 172.21.15.16:44360) 2024-09-12T03:02:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 -- 172.21.15.16:0/3708473767 learned_addr learned my addr 172.21.15.16:0/3708473767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 -- 172.21.15.16:0/3708473767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 msgr2=0x55aa1112c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa1112c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 -- 172.21.15.16:0/3708473767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 msgr2=0x55aa11140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 -- 172.21.15.16:0/3708473767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa1112f530 con 0x55aa1112ab40 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.664+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8005940 tx=0x7fd1c80058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343053e550e769ac server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1b97fa640 1 -- 172.21.15.16:0/3708473767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8035040 con 0x55aa1112ab40 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1b97fa640 1 -- 172.21.15.16:0/3708473767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c8005500 con 0x55aa1112ab40 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1b97fa640 1 -- 172.21.15.16:0/3708473767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80035f0 con 0x55aa1112ab40 2024-09-12T03:02:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/3708473767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 msgr2=0x55aa1112c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8005940 tx=0x7fd1c80058b0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/3708473767 shutdown_connections 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa11134370 0x55aa1112c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa1112ab40 0x55aa1112c120 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/3708473767 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa11140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/3708473767 >> 172.21.15.16:0/3708473767 conn(0x55aa110f2310 msgr2=0x55aa11133060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/3708473767 shutdown_connections 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.665+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/3708473767 wait complete. 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 -- start start 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 Processor -- start 2024-09-12T03:02:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa1113ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa1112ab40 0x55aa11137e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41aea90 con 0x55aa11134370 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ae3c0 con 0x55aa111295d0 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ab540 con 0x55aa1112ab40 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1dc3fb740 10 client.0 _ll_register_callbacks cb 0x7fd1c401a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44370/0 (socket says 172.21.15.16:44370) 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.666+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/1022585395 learned_addr learned my addr 172.21.15.16:0/1022585395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa1113ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa1112ab40 0x55aa11137e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa1112ab40 msgr2=0x55aa11137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa1112ab40 0x55aa11137e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/1022585395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 msgr2=0x55aa1113ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa1113ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 -- 172.21.15.16:0/1022585395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa11138a80 con 0x55aa11134370 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1d2ffd640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4014f80 tx=0x7fd1c4007550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e10ce13ea5831d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c402a040 con 0x55aa11134370 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd1c4024a00 con 0x55aa11134370 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c8005500 con 0x55aa11134370 2024-09-12T03:02:01.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd1c4007d00 con 0x55aa11134370 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1dc3fb740 10 client.18798 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aa111ae960 con 0x55aa11134370 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.667+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c8005500 con 0x55aa11134370 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fd1c4022040 con 0x55aa11134370 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 10 client.18798 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 10 client.18798 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 20 client.18798 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa111aefa0 0x55aa111b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 4 client.18798 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa11096d70 con 0x55aa111aefa0 2024-09-12T03:02:01.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa111aefa0 0x55aa111b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1d37fe640 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa111aefa0 0x55aa111b13c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=868a9edcd35f16c9 server_cookie=dd204f7ef4086b05 in_seq=0 out_seq=0 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.668+0000 7fd1b2ffd640 10 client.18798 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1b2ffd640 1 -- 172.21.15.16:0/1022585395 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa11096d70 con 0x55aa111aefa0 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1b2ffd640 10 client.18798 handle_command_reply: tid=0 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 client.18798 shutdown 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 msgr2=0x55aa11138390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4014f80 tx=0x7fd1c4007550 comp rx=0 tx=0).stop 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 shutdown_connections 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa11134370 0x55aa11138390 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55aa111aefa0 0x55aa111b13c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa1112ab40 0x55aa11137e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 --2- 172.21.15.16:0/1022585395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aa111295d0 0x55aa1113ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1b2ffd640 0 client.18798 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:01.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 >> 172.21.15.16:0/1022585395 conn(0x55aa110f2310 msgr2=0x55aa1112f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.697 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 shutdown_connections 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 1 -- 172.21.15.16:0/1022585395 wait complete. 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.669+0000 7fd1dc3fb740 20 client.18798 trim_cache size 0 max 16384 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 msgr2=0x7fd1bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 0x7fd1bc0a5040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc06f0a0 tx=0x7fd1cc06f0d0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 msgr2=0x7fd1d41a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00cce0 tx=0x7fd1cc00a620 comp rx=0 tx=0).stop 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 shutdown_connections 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd1bc0a2b80 0x7fd1bc0a5040 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d4102ef0 0x7fd1d41a4430 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1d4102570 0x7fd1d41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 --2- 172.21.15.16:0/4159259258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4100e20 0x7fd1d41993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:01.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 >> 172.21.15.16:0/4159259258 conn(0x7fd1d40fc7d0 msgr2=0x7fd1d4105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:01.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 shutdown_connections 2024-09-12T03:02:01.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:01.670+0000 7fd1da022640 1 -- 172.21.15.16:0/4159259258 wait complete. 2024-09-12T03:02:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:01 smithi033 ceph-mon[35092]: pgmap v760: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-09-12T03:02:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:01 smithi184 ceph-mon[35321]: pgmap v760: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-09-12T03:02:02.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.49 MB/sec warmup 63 sec latency 21322.368 ms 2024-09-12T03:02:02.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:02:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:03.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.36 MB/sec warmup 64 sec latency 22322.667 ms 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 -- 172.21.15.16:0/1364546615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd4102df0 msgr2=0x7f6fd41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/1364546615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd4102df0 0x7f6fd41031d0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009900 tx=0x7f6fc402a410 comp rx=0 tx=0).stop 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 -- 172.21.15.16:0/1364546615 shutdown_connections 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/1364546615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd410c2f0 0x7f6fd410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/1364546615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd41037a0 0x7f6fd410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/1364546615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd4102df0 0x7f6fd41031d0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 -- 172.21.15.16:0/1364546615 >> 172.21.15.16:0/1364546615 conn(0x7f6fd40fc7d0 msgr2=0x7f6fd40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 -- 172.21.15.16:0/1364546615 shutdown_connections 2024-09-12T03:02:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.208+0000 7f6fda9d5640 1 -- 172.21.15.16:0/1364546615 wait complete. 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 Processor -- start 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 -- start start 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 0x7f6fd4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd41037a0 0x7f6fd4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 0x7f6fd419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.209+0000 7f6fda9d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41120f0 con 0x7f6fd410c2f0 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fda9d5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd4111f70 con 0x7f6fd4102df0 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fda9d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd4112270 con 0x7f6fd41037a0 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 0x7f6fd4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 0x7f6fd419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 0x7f6fd4198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53410/0 (socket says 172.21.15.16:53410) 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd3fff640 1 -- 172.21.15.16:0/308346500 learned_addr learned my addr 172.21.15.16:0/308346500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd41037a0 0x7f6fd4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 -- 172.21.15.16:0/308346500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd41037a0 msgr2=0x7f6fd4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd41037a0 0x7f6fd4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 -- 172.21.15.16:0/308346500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 msgr2=0x7f6fd4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 0x7f6fd4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 -- 172.21.15.16:0/308346500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd419d5e0 con 0x7f6fd410c2f0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd8f72640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 0x7f6fd419cee0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00da10 tx=0x7f6fcc00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.210+0000 7f6fd1ffb640 1 -- 172.21.15.16:0/308346500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00bd70 con 0x7f6fd410c2f0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.211+0000 7f6fd1ffb640 1 -- 172.21.15.16:0/308346500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fcc003e50 con 0x7f6fd410c2f0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.211+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd419d8d0 con 0x7f6fd410c2f0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.211+0000 7f6fd1ffb640 1 -- 172.21.15.16:0/308346500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0106a0 con 0x7f6fd410c2f0 2024-09-12T03:02:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.211+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd4100980 con 0x7f6fd410c2f0 2024-09-12T03:02:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.213+0000 7f6fd1ffb640 1 -- 172.21.15.16:0/308346500 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6fcc003970 con 0x7f6fd410c2f0 2024-09-12T03:02:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.213+0000 7f6fdcdae740 1 Processor -- start 2024-09-12T03:02:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 -- start start 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5660090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5660a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc566d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fc55c62f0 con 0x563fc566b370 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fc556dcf0 con 0x563fc56605d0 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fc55c22b0 con 0x563fc5661b40 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5660090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5660090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:43920/0 (socket says 172.21.15.16:43920) 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2864191343 learned_addr learned my addr 172.21.15.16:0/2864191343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fd1ffb640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 0x7f6fc00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc566d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5660a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fd1ffb640 1 -- 172.21.15.16:0/308346500 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fcc0cf370 con 0x7f6fd410c2f0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.214+0000 7f6fd8f72640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 0x7f6fc00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2864191343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 msgr2=0x563fc5660090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5660090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2864191343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 msgr2=0x563fc5660a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5660a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2864191343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc55523a0 con 0x563fc566b370 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc566d760 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0126c0 tx=0x7f6fbc00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8c6a1984e269e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fd8f72640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 0x7f6fc00a5040 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0cce80 tx=0x7f6fcc0046e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fca7fc640 1 -- 172.21.15.16:0/2864191343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017070 con 0x563fc566b370 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fca7fc640 1 -- 172.21.15.16:0/2864191343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc009720 con 0x563fc566b370 2024-09-12T03:02:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.215+0000 7f6fca7fc640 1 -- 172.21.15.16:0/2864191343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc003830 con 0x563fc566b370 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2864191343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 msgr2=0x563fc566d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc566d760 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0126c0 tx=0x7f6fbc00b410 comp rx=0 tx=0).stop 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2864191343 shutdown_connections 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc566d760 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5660a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2864191343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5660090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2864191343 >> 172.21.15.16:0/2864191343 conn(0x563fc5629310 msgr2=0x563fc565cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2864191343 shutdown_connections 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2864191343 wait complete. 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.216+0000 7f6fdcdae740 1 -- start start 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 Processor -- start 2024-09-12T03:02:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5668030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc566b370 0x563fc5668640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fc566e070 con 0x563fc56605d0 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fc5581060 con 0x563fc566b370 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fc566e1f0 con 0x563fc5661b40 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fdcdae740 10 client.0 _ll_register_callbacks cb 0x563fc52d48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44394/0 (socket says 172.21.15.16:44394) 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2500586509 learned_addr learned my addr 172.21.15.16:0/2500586509 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5668030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2500586509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 msgr2=0x563fc5668030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5668030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc566b370 msgr2=0x563fc5668640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc566b370 0x563fc5668640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.217+0000 7f6fcbfff640 1 -- 172.21.15.16:0/2500586509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc53373c0 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc01c040 tx=0x7f6fbc0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a59d5237641bf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017050 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fbc009b90 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fbc009720 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6fbc0035f0 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fdcdae740 10 client.18810 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563fc5679b40 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc009720 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.218+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f6fbc030070 con 0x563fc56605d0 2024-09-12T03:02:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 10 client.18810 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 10 client.18810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 20 client.18810 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc567ae90 0x563fc567d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 4 client.18810 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fc55d3860 con 0x563fc567ae90 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc567ae90 0x563fc567d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc567ae90 0x563fc567d280 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dddf6a82f61946f1 server_cookie=dabd7968afe94437 in_seq=0 out_seq=0 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.219+0000 7f6fb37fe640 10 client.18810 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fb37fe640 1 -- 172.21.15.16:0/2500586509 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563fc55d3860 con 0x563fc567ae90 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fb37fe640 10 client.18810 handle_command_reply: tid=0 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 client.18810 shutdown 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 msgr2=0x563fc566ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc01c040 tx=0x7f6fbc0056d0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 shutdown_connections 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc567ae90 0x563fc567d280 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc566b370 0x563fc5668640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5668030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/2500586509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc566ad80 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 >> 172.21.15.16:0/2500586509 conn(0x563fc5629310 msgr2=0x563fc5662960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.220+0000 7f6fb37fe640 0 client.18810 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.221+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 shutdown_connections 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.221+0000 7f6fdcdae740 1 -- 172.21.15.16:0/2500586509 wait complete. 2024-09-12T03:02:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.221+0000 7f6fdcdae740 20 client.18810 trim_cache size 0 max 16384 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 Processor -- start 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 -- start start 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc5661b40 0x563fc5682930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc566b370 0x563fc5682e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41af790 con 0x563fc56605d0 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41b1390 con 0x563fc5661b40 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41afe60 con 0x563fc566b370 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44400/0 (socket says 172.21.15.16:44400) 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1642085792 learned_addr learned my addr 172.21.15.16:0/1642085792 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc5661b40 0x563fc5682930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1642085792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc566b370 msgr2=0x563fc5682e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc566b370 0x563fc5682e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1642085792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc5661b40 msgr2=0x563fc5682930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc5661b40 0x563fc5682930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1642085792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc566df50 con 0x563fc56605d0 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.234+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00bbf0 tx=0x7f6fbc004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fc0cfa7d835fff server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/1642085792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc037070 con 0x563fc56605d0 2024-09-12T03:02:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/1642085792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc02d070 con 0x563fc56605d0 2024-09-12T03:02:03.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fc9ffb640 1 -- 172.21.15.16:0/1642085792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017030 con 0x563fc56605d0 2024-09-12T03:02:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1642085792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 msgr2=0x563fc5677310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00bbf0 tx=0x7f6fbc004540 comp rx=0 tx=0).stop 2024-09-12T03:02:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1642085792 shutdown_connections 2024-09-12T03:02:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc566b370 0x563fc5682e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc5661b40 0x563fc5682930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1642085792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc56605d0 0x563fc5677310 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1642085792 >> 172.21.15.16:0/1642085792 conn(0x563fc5629310 msgr2=0x563fc5667140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.235+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1642085792 shutdown_connections 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1642085792 wait complete. 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 -- start start 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 Processor -- start 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5673b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5664d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41b0cc0 con 0x563fc566b370 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41b05f0 con 0x563fc56605d0 2024-09-12T03:02:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd41ad770 con 0x563fc5661b40 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fdcdae740 10 client.0 _ll_register_callbacks cb 0x7f6fc4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44406/0 (socket says 172.21.15.16:44406) 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1292713186 learned_addr learned my addr 172.21.15.16:0/1292713186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5673b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.236+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5664d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1292713186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 msgr2=0x563fc5664d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5664d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 msgr2=0x563fc5673b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5673b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 -- 172.21.15.16:0/1292713186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc5665a40 con 0x563fc566b370 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5664d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fcbfff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x563fc56645a0 tx=0x7f6fbc014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f44d11fe11e614 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc045060 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6fbc009b90 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fbc02d070 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6fbc017030 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fdcdae740 10 client.18816 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.237+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563fc56e58a0 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc02d070 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f6fbc03d070 con 0x563fc566b370 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 10 client.18816 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 10 client.18816 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 20 client.18816 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc56e5eb0 0x563fc56e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 4 client.18816 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563fc55cdd70 con 0x563fc56e5eb0 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.238+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc56e5eb0 0x563fc56e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fd3fff640 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc56e5eb0 0x563fc56e82a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a707cb3d4543af27 server_cookie=6b1bd393b351593f in_seq=0 out_seq=0 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fb37fe640 10 client.18816 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fb37fe640 1 -- 172.21.15.16:0/1292713186 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563fc55cdd70 con 0x563fc56e5eb0 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fb37fe640 10 client.18816 handle_command_reply: tid=0 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 client.18816 shutdown 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 msgr2=0x563fc5665270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x563fc56645a0 tx=0x7f6fbc014040 comp rx=0 tx=0).stop 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 shutdown_connections 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fc56e5eb0 0x563fc56e82a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fb37fe640 0 client.18816 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fc566b370 0x563fc5665270 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fc5661b40 0x563fc5664d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 --2- 172.21.15.16:0/1292713186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fc56605d0 0x563fc5673b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 >> 172.21.15.16:0/1292713186 conn(0x563fc5629310 msgr2=0x563fc565b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 shutdown_connections 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 1 -- 172.21.15.16:0/1292713186 wait complete. 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.239+0000 7f6fdcdae740 20 client.18816 trim_cache size 0 max 16384 2024-09-12T03:02:03.245 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 msgr2=0x7f6fc00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 0x7f6fc00a5040 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0cce80 tx=0x7f6fcc0046e0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 msgr2=0x7f6fd419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 0x7f6fd419cee0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00da10 tx=0x7f6fcc00ded0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 shutdown_connections 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6fc00a2b80 0x7f6fc00a5040 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd410c2f0 0x7f6fd419cee0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fd41037a0 0x7f6fd4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 --2- 172.21.15.16:0/308346500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6fd4102df0 0x7f6fd4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 >> 172.21.15.16:0/308346500 conn(0x7f6fd40fc7d0 msgr2=0x7f6fd40fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 shutdown_connections 2024-09-12T03:02:03.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:03.240+0000 7f6fda9d5640 1 -- 172.21.15.16:0/308346500 wait complete. 2024-09-12T03:02:03.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:03 smithi016 ceph-mon[30925]: pgmap v761: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-12T03:02:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:03 smithi033 ceph-mon[35092]: pgmap v761: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-12T03:02:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:03 smithi184 ceph-mon[35321]: pgmap v761: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-12T03:02:03.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:02:04.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.23 MB/sec warmup 65 sec latency 23322.717 ms 2024-09-12T03:02:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 -- 172.21.15.16:0/87900629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940101de0 msgr2=0x7f29401021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 --2- 172.21.15.16:0/87900629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940101de0 0x7f29401021c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f293002a410 comp rx=0 tx=0).stop 2024-09-12T03:02:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 -- 172.21.15.16:0/87900629 shutdown_connections 2024-09-12T03:02:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 --2- 172.21.15.16:0/87900629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940106f10 0x7f294010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 --2- 172.21.15.16:0/87900629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940102790 0x7f29401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 --2- 172.21.15.16:0/87900629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940101de0 0x7f29401021c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 -- 172.21.15.16:0/87900629 >> 172.21.15.16:0/87900629 conn(0x7f29400fd8c0 msgr2=0x7f29400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 -- 172.21.15.16:0/87900629 shutdown_connections 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.697+0000 7f2944f95640 1 -- 172.21.15.16:0/87900629 wait complete. 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.698+0000 7f2944f95640 1 Processor -- start 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.698+0000 7f2944f95640 1 -- start start 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f2940198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940106f10 0x7f294019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111f70 con 0x7f2940102790 2024-09-12T03:02:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111df0 con 0x7f2940101de0 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29401120f0 con 0x7f2940106f10 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44434/0 (socket says 172.21.15.16:44434) 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 -- 172.21.15.16:0/3909237755 learned_addr learned my addr 172.21.15.16:0/3909237755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293e59c640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f2940198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293ed9d640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940106f10 0x7f294019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 -- 172.21.15.16:0/3909237755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940106f10 msgr2=0x7f294019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940106f10 0x7f294019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 -- 172.21.15.16:0/3909237755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 msgr2=0x7f2940198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f2940198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 -- 172.21.15.16:0/3909237755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294019d3e0 con 0x7f2940102790 2024-09-12T03:02:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f293dd9b640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f293400d770 tx=0x7f293400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f292f7fe640 1 -- 172.21.15.16:0/3909237755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934017070 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f292f7fe640 1 -- 172.21.15.16:0/3909237755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2934005500 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.699+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294019d6d0 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.700+0000 7f292f7fe640 1 -- 172.21.15.16:0/3909237755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934004e00 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.700+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401a4fb0 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f292f7fe640 1 -- 172.21.15.16:0/3909237755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2934004fa0 con 0x7f2940102790 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 Processor -- start 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 -- start start 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdce9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5611bdc4f2f0 con 0x5611bdceab40 2024-09-12T03:02:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5611bdbf6cf0 con 0x5611bdcf4370 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.702+0000 7f294736e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bdc4b2b0 con 0x5611bdce95d0 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44450/0 (socket says 172.21.15.16:44450) 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 -- 172.21.15.16:0/2958912203 learned_addr learned my addr 172.21.15.16:0/2958912203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f292f7fe640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 0x7f29280a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293e59c640 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdce9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293dd9b640 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 0x7f29280a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f292f7fe640 1 -- 172.21.15.16:0/3909237755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f29340cea00 con 0x7f2940102790 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 -- 172.21.15.16:0/2958912203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 msgr2=0x5611bdce9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdce9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 -- 172.21.15.16:0/2958912203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 msgr2=0x5611bdcf6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 -- 172.21.15.16:0/2958912203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bdbdb3a0 con 0x5611bdceab40 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f293ed9d640 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2938013940 tx=0x7f2938013e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7eb920395d3bf8a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.703+0000 7f292cff9640 1 -- 172.21.15.16:0/2958912203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938027040 con 0x5611bdceab40 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f292cff9640 1 -- 172.21.15.16:0/2958912203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293801bbc0 con 0x5611bdceab40 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f292cff9640 1 -- 172.21.15.16:0/2958912203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938023b10 con 0x5611bdceab40 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f293ed9d640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 0x7f29280a5040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2938024000 tx=0x7f2938022000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 -- 172.21.15.16:0/2958912203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 msgr2=0x5611bdce9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2938013940 tx=0x7f2938013e00 comp rx=0 tx=0).stop 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 -- 172.21.15.16:0/2958912203 shutdown_connections 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdce9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 --2- 172.21.15.16:0/2958912203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce9090 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 -- 172.21.15.16:0/2958912203 >> 172.21.15.16:0/2958912203 conn(0x5611bdcb2310 msgr2=0x5611bdce5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 -- 172.21.15.16:0/2958912203 shutdown_connections 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.704+0000 7f294736e740 1 -- 172.21.15.16:0/2958912203 wait complete. 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 -- start start 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 Processor -- start 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdceab40 0x5611bdcf1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5611bdcf7070 con 0x5611bdce95d0 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5611bdc0a060 con 0x5611bdcf4370 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bdcf71f0 con 0x5611bdceab40 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f294736e740 10 client.0 _ll_register_callbacks cb 0x5611bd95d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f293e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f293e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44452/0 (socket says 172.21.15.16:44452) 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f293e59c640 1 -- 172.21.15.16:0/392929430 learned_addr learned my addr 172.21.15.16:0/392929430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f293dd9b640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdceab40 0x5611bdcf1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.705+0000 7f293ed9d640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 -- 172.21.15.16:0/392929430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdceab40 msgr2=0x5611bdcf1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdceab40 0x5611bdcf1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 -- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 msgr2=0x5611bdcf1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 -- 172.21.15.16:0/392929430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bd9c03c0 con 0x5611bdce95d0 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293ed9d640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f293e59c640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f29300319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c84e0c49d6edb63 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930035040 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2930004300 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293801bbc0 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2930005020 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f294736e740 10 client.18828 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5611bdd02b10 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293801bbc0 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f29300035f0 con 0x5611bdce95d0 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f294736e740 10 client.18828 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.706+0000 7f294736e740 10 client.18828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f294736e740 20 client.18828 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd03fd0 0x5611bdd063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f294736e740 4 client.18828 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611bdc5c860 con 0x5611bdd03fd0 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f293dd9b640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd03fd0 0x5611bdd063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f2925ffb640 10 client.18828 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.707+0000 7f293dd9b640 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd03fd0 0x5611bdd063c0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f8c977f7066e00 server_cookie=e9926823efa04821 in_seq=0 out_seq=0 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f2925ffb640 1 -- 172.21.15.16:0/392929430 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5611bdc5c860 con 0x5611bdd03fd0 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f2925ffb640 10 client.18828 handle_command_reply: tid=0 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 client.18828 shutdown 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 msgr2=0x5611bdcf3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f29300319b0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 shutdown_connections 2024-09-12T03:02:04.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd03fd0 0x5611bdd063c0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdceab40 0x5611bdcf1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f2925ffb640 0 client.18828 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 --2- 172.21.15.16:0/392929430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdce95d0 0x5611bdcf3d60 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 >> 172.21.15.16:0/392929430 conn(0x5611bdcb2310 msgr2=0x5611bdceb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 shutdown_connections 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 1 -- 172.21.15.16:0/392929430 wait complete. 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.708+0000 7f294736e740 20 client.18828 trim_cache size 0 max 16384 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 Processor -- start 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 -- start start 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcf0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcec120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29401ab290 con 0x5611bdceab40 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29401ace90 con 0x5611bdcf4370 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f294736e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29401ab960 con 0x5611bdce95d0 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f293dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f293dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44462/0 (socket says 172.21.15.16:44462) 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.721+0000 7f293dd9b640 1 -- 172.21.15.16:0/4244686643 learned_addr learned my addr 172.21.15.16:0/4244686643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293e59c640 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcf0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 -- 172.21.15.16:0/4244686643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 msgr2=0x5611bdcf0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcf0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 -- 172.21.15.16:0/4244686643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 msgr2=0x5611bdcec120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 -- 172.21.15.16:0/4244686643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bdce4310 con 0x5611bdceab40 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f293dd9b640 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f293409b020 tx=0x7f29340bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a69ca564540b54 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f2927fff640 1 -- 172.21.15.16:0/4244686643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934020040 con 0x5611bdceab40 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f2927fff640 1 -- 172.21.15.16:0/4244686643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293409b320 con 0x5611bdceab40 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.722+0000 7f2927fff640 1 -- 172.21.15.16:0/4244686643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934009cc0 con 0x5611bdceab40 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- 172.21.15.16:0/4244686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 msgr2=0x5611bdce5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f293409b020 tx=0x7f29340bb040 comp rx=0 tx=0).stop 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- 172.21.15.16:0/4244686643 shutdown_connections 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdcf4370 0x5611bdcec120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdceab40 0x5611bdce5d70 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- 172.21.15.16:0/4244686643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcf0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- 172.21.15.16:0/4244686643 >> 172.21.15.16:0/4244686643 conn(0x5611bdcb2310 msgr2=0x5611bdce4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- 172.21.15.16:0/4244686643 shutdown_connections 2024-09-12T03:02:04.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- 172.21.15.16:0/4244686643 wait complete. 2024-09-12T03:02:04.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 -- start start 2024-09-12T03:02:04.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 Processor -- start 2024-09-12T03:02:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcfcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.723+0000 7f294736e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 0x5611bdcede00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29401ac7c0 con 0x5611bdcf4370 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29401ac0f0 con 0x5611bdceab40 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29401a9270 con 0x5611bdce95d0 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 10 client.0 _ll_register_callbacks cb 0x7f2930031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44466/0 (socket says 172.21.15.16:44466) 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 -- 172.21.15.16:0/454690427 learned_addr learned my addr 172.21.15.16:0/454690427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293e59c640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcfcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293dd9b640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 0x5611bdcede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 -- 172.21.15.16:0/454690427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 msgr2=0x5611bdcfcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcfcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 -- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 msgr2=0x5611bdcede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 0x5611bdcede00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 -- 172.21.15.16:0/454690427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bdceeb70 con 0x5611bdcf4370 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293dd9b640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 0x5611bdcede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f293ed9d640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2938013940 tx=0x7f2938013560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4429c8e119c60d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f2925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938027040 con 0x5611bdcf4370 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2938004a00 con 0x5611bdcf4370 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.724+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293409b320 con 0x5611bdcf4370 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f293800a4d0 con 0x5611bdcf4370 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 10 client.18834 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5611bdd6ec90 con 0x5611bdcf4370 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293409b320 con 0x5611bdcf4370 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f293801b4e0 con 0x5611bdcf4370 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 10 client.18834 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 10 client.18834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 20 client.18834 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd6f240 0x5611bdd71660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 4 client.18834 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611bdc56d70 con 0x5611bdd6f240 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.725+0000 7f293e59c640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd6f240 0x5611bdd71660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f293e59c640 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd6f240 0x5611bdd71660 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40eada467196336b server_cookie=513da60507c7e660 in_seq=0 out_seq=0 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f2925ffb640 10 client.18834 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f2925ffb640 1 -- 172.21.15.16:0/454690427 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611bdc56d70 con 0x5611bdd6f240 2024-09-12T03:02:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f2925ffb640 10 client.18834 handle_command_reply: tid=0 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 client.18834 shutdown 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 msgr2=0x5611bdcee340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2938013940 tx=0x7f2938013560 comp rx=0 tx=0).stop 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 shutdown_connections 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5611bdd6f240 0x5611bdd71660 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5611bdcf4370 0x5611bdcee340 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5611bdceab40 0x5611bdcede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f2925ffb640 0 client.18834 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 --2- 172.21.15.16:0/454690427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bdce95d0 0x5611bdcfcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 >> 172.21.15.16:0/454690427 conn(0x5611bdcb2310 msgr2=0x5611bdcf2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 shutdown_connections 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 1 -- 172.21.15.16:0/454690427 wait complete. 2024-09-12T03:02:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.726+0000 7f294736e740 20 client.18834 trim_cache size 0 max 16384 2024-09-12T03:02:04.743 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:04.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 msgr2=0x7f29280a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 0x7f29280a5040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2938024000 tx=0x7f2938022000 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 msgr2=0x7f2940198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f293400d770 tx=0x7f293400dc30 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 shutdown_connections 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940106f10 0x7f294019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f29280a2b80 0x7f29280a5040 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2940102790 0x7f2940198970 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 --2- 172.21.15.16:0/3909237755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2940101de0 0x7f2940198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 >> 172.21.15.16:0/3909237755 conn(0x7f29400fd8c0 msgr2=0x7f29400fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 shutdown_connections 2024-09-12T03:02:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:04.727+0000 7f2944f95640 1 -- 172.21.15.16:0/3909237755 wait complete. 2024-09-12T03:02:05.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 8.10 MB/sec warmup 66 sec latency 24322.779 ms 2024-09-12T03:02:05.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:05 smithi016 ceph-mon[30925]: pgmap v762: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T03:02:05.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:05 smithi033 ceph-mon[35092]: pgmap v762: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T03:02:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:05 smithi184 ceph-mon[35321]: pgmap v762: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-12T03:02:06.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.98 MB/sec warmup 67 sec latency 25322.879 ms 2024-09-12T03:02:06.450 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:02:06.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:02:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:07.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.86 MB/sec warmup 68 sec latency 26322.924 ms 2024-09-12T03:02:07.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 -- 172.21.15.16:0/1721045489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 msgr2=0x7efdcc076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 --2- 172.21.15.16:0/1721045489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc076b00 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009900 tx=0x7efdbc02a410 comp rx=0 tx=0).stop 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 -- 172.21.15.16:0/1721045489 shutdown_connections 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 --2- 172.21.15.16:0/1721045489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc112a30 0x7efdcc114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 --2- 172.21.15.16:0/1721045489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc077040 0x7efdcc1124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 --2- 172.21.15.16:0/1721045489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc076b00 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 -- 172.21.15.16:0/1721045489 >> 172.21.15.16:0/1721045489 conn(0x7efdcc0ffa90 msgr2=0x7efdcc101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 -- 172.21.15.16:0/1721045489 shutdown_connections 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 -- 172.21.15.16:0/1721045489 wait complete. 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.184+0000 7efdd17c5640 1 Processor -- start 2024-09-12T03:02:07.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 -- start start 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc077040 0x7efdcc1a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc112a30 0x7efdcc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1185c0 con 0x7efdcc076720 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc118440 con 0x7efdcc077040 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdd17c5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc118740 con 0x7efdcc112a30 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33528/0 (socket says 172.21.15.16:33528) 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 -- 172.21.15.16:0/3638957391 learned_addr learned my addr 172.21.15.16:0/3638957391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdca7fc640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc077040 0x7efdcc1a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc112a30 0x7efdcc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 -- 172.21.15.16:0/3638957391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc112a30 msgr2=0x7efdcc19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc112a30 0x7efdcc19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 -- 172.21.15.16:0/3638957391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc077040 msgr2=0x7efdcc1a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc077040 0x7efdcc1a4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.185+0000 7efdcaffd640 1 -- 172.21.15.16:0/3638957391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc19f000 con 0x7efdcc076720 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdb3fff640 1 -- 172.21.15.16:0/3638957391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc03f030 con 0x7efdcc076720 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc19f290 con 0x7efdcc076720 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1ab5e0 con 0x7efdcc076720 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdb3fff640 1 -- 172.21.15.16:0/3638957391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdbc003ce0 con 0x7efdcc076720 2024-09-12T03:02:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.186+0000 7efdb3fff640 1 -- 172.21.15.16:0/3638957391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc049060 con 0x7efdcc076720 2024-09-12T03:02:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.188+0000 7efdb3fff640 1 -- 172.21.15.16:0/3638957391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efdbc042030 con 0x7efdcc076720 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 Processor -- start 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 -- start start 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed52d280 con 0x55d3ed5c8480 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed4d4b10 con 0x55d3ed5d2d40 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed529240 con 0x55d3ed5c7b60 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33538/0 (socket says 172.21.15.16:33538) 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdcaffd640 1 -- 172.21.15.16:0/2837931762 learned_addr learned my addr 172.21.15.16:0/2837931762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdca7fc640 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.189+0000 7efdb3fff640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 0x7efdb40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 0x7efdb40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2837931762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5d2d40 msgr2=0x55d3ed5d5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdb3fff640 1 -- 172.21.15.16:0/3638957391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efdbc0eedb0 con 0x7efdcc076720 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2837931762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 msgr2=0x55d3ed5d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2837931762 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3ed4e11f0 con 0x55d3ed5c7b60 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcaffd640 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5c7f40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efdcc07aa10 tx=0x7efdc00035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f04c986c28271bd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 0x7efdb40a5040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efdc001b000 tx=0x7efdc001d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdb17fa640 1 -- 172.21.15.16:0/2837931762 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0005030 con 0x55d3ed5c7b60 2024-09-12T03:02:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdb17fa640 1 -- 172.21.15.16:0/2837931762 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc00047a0 con 0x55d3ed5c7b60 2024-09-12T03:02:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.190+0000 7efdb17fa640 1 -- 172.21.15.16:0/2837931762 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00056b0 con 0x55d3ed5c7b60 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2837931762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 msgr2=0x55d3ed5c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5c7f40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efdcc07aa10 tx=0x7efdc00035f0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2837931762 shutdown_connections 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2837931762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5c7f40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2837931762 >> 172.21.15.16:0/2837931762 conn(0x55d3ed5c40f0 msgr2=0x55d3ed5c44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2837931762 shutdown_connections 2024-09-12T03:02:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.191+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2837931762 wait complete. 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 -- start start 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 Processor -- start 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed5d5910 con 0x55d3ed5c8480 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed4e7f10 con 0x55d3ed5c7b60 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d3ed5d5a90 con 0x55d3ed5d2d40 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdd3b9e740 10 client.0 _ll_register_callbacks cb 0x55d3ed4b0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33542/0 (socket says 172.21.15.16:33542) 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 -- 172.21.15.16:0/162390693 learned_addr learned my addr 172.21.15.16:0/162390693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdcaffd640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 -- 172.21.15.16:0/162390693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 msgr2=0x55d3ed5d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 -- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 msgr2=0x55d3ed5cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdca7fc640 1 -- 172.21.15.16:0/162390693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3ed29e6a0 con 0x55d3ed5c8480 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.192+0000 7efdcaffd640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdca7fc640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7efdc4010a50 tx=0x7efdc4010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523e76f7bcf0a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc400fc90 con 0x55d3ed5c8480 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efdc400fe30 con 0x55d3ed5c8480 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc00047a0 con 0x55d3ed5c8480 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdc4004b20 con 0x55d3ed5c8480 2024-09-12T03:02:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 10 client.18846 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d3ed5e15e0 con 0x55d3ed5c8480 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc00047a0 con 0x55d3ed5c8480 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7efdc4024450 con 0x55d3ed5c8480 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 10 client.18846 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 10 client.18846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 20 client.18846 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed5e2a60 0x55d3ed5e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 4 client.18846 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.193+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3ed53a7f0 con 0x55d3ed5e2a60 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.194+0000 7efdcaffd640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed5e2a60 0x55d3ed5e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.194+0000 7efdaeff5640 10 client.18846 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.194+0000 7efdcaffd640 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed5e2a60 0x55d3ed5e4f00 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92df372eadff990 server_cookie=619c9ed15c8c87c8 in_seq=0 out_seq=0 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdaeff5640 1 -- 172.21.15.16:0/162390693 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d3ed53a7f0 con 0x55d3ed5e2a60 2024-09-12T03:02:07.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdaeff5640 10 client.18846 handle_command_reply: tid=0 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 client.18846 shutdown 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 msgr2=0x55d3ed5d0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7efdc4010a50 tx=0x7efdc4010e50 comp rx=0 tx=0).stop 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 shutdown_connections 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed5e2a60 0x55d3ed5e4f00 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c8480 0x55d3ed5d0100 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdaeff5640 0 client.18846 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/162390693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 >> 172.21.15.16:0/162390693 conn(0x55d3ed5c40f0 msgr2=0x55d3ed5c3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 shutdown_connections 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 1 -- 172.21.15.16:0/162390693 wait complete. 2024-09-12T03:02:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.195+0000 7efdd3b9e740 20 client.18846 trim_cache size 0 max 16384 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 Processor -- start 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 -- start start 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5ea860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 0x55d3ed5eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800e400 con 0x55d3ed5d2d40 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efdb8010000 con 0x55d3ed5c7b60 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800ead0 con 0x55d3ed5c8480 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33550/0 (socket says 172.21.15.16:33550) 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2898429713 learned_addr learned my addr 172.21.15.16:0/2898429713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdca7fc640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 0x55d3ed5eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcaffd640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2898429713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 msgr2=0x55d3ed5eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 0x55d3ed5eada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2898429713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 msgr2=0x55d3ed5ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdcb7fe640 1 -- 172.21.15.16:0/2898429713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3ed5d2490 con 0x55d3ed5d2d40 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.213+0000 7efdca7fc640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 0x55d3ed5eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7efdc0004fd0 tx=0x7efdc0004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1766a561ca4d4dfc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdb0ff9640 1 -- 172.21.15.16:0/2898429713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc002b070 con 0x55d3ed5d2d40 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdb0ff9640 1 -- 172.21.15.16:0/2898429713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc0005510 con 0x55d3ed5d2d40 2024-09-12T03:02:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdb0ff9640 1 -- 172.21.15.16:0/2898429713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0010e50 con 0x55d3ed5d2d40 2024-09-12T03:02:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2898429713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 msgr2=0x55d3ed5ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7efdc0004fd0 tx=0x7efdc0004160 comp rx=0 tx=0).stop 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2898429713 shutdown_connections 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5ce360 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5c8480 0x55d3ed5eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/2898429713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.214+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2898429713 >> 172.21.15.16:0/2898429713 conn(0x55d3ed5c40f0 msgr2=0x55d3ed5d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2898429713 shutdown_connections 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- 172.21.15.16:0/2898429713 wait complete. 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- start start 2024-09-12T03:02:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 Processor -- start 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c8480 0x55d3ed5db2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800f930 con 0x55d3ed5c7b60 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800f260 con 0x55d3ed5c8480 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800c3e0 con 0x55d3ed5d2d40 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdd3b9e740 10 client.0 _ll_register_callbacks cb 0x7efdc400a9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33564/0 (socket says 172.21.15.16:33564) 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdcaffd640 1 -- 172.21.15.16:0/3901855078 learned_addr learned my addr 172.21.15.16:0/3901855078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdca7fc640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c8480 0x55d3ed5db2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.215+0000 7efdcb7fe640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 -- 172.21.15.16:0/3901855078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 msgr2=0x55d3ed5d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 -- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c8480 msgr2=0x55d3ed5db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c8480 0x55d3ed5db2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 -- 172.21.15.16:0/3901855078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3ed5db9f0 con 0x55d3ed5c7b60 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdcaffd640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0b7b30 tx=0x7efdbc0b7850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31c5d17a9d8a3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc035040 con 0x55d3ed5c7b60 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efdbc03b510 con 0x55d3ed5c7b60 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc0005510 con 0x55d3ed5c7b60 2024-09-12T03:02:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdbc0bb030 con 0x55d3ed5c7b60 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdd3b9e740 10 client.18852 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d3ed64de80 con 0x55d3ed5c7b60 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.216+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc0005510 con 0x55d3ed5c7b60 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7efdbc0b7d40 con 0x55d3ed5c7b60 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 10 client.18852 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 10 client.18852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 20 client.18852 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed64e5a0 0x55d3ed6509c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 4 client.18852 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d3ed534d00 con 0x55d3ed64e5a0 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdca7fc640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed64e5a0 0x55d3ed6509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdca7fc640 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed64e5a0 0x55d3ed6509c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ad269576ce6b28 server_cookie=db6ac41b0f39f25c in_seq=0 out_seq=0 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.217+0000 7efdaeff5640 10 client.18852 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdaeff5640 1 -- 172.21.15.16:0/3901855078 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d3ed534d00 con 0x55d3ed64e5a0 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdaeff5640 10 client.18852 handle_command_reply: tid=0 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 client.18852 shutdown 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 msgr2=0x55d3ed5dad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0b7b30 tx=0x7efdbc0b7850 comp rx=0 tx=0).stop 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 shutdown_connections 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d3ed64e5a0 0x55d3ed6509c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d3ed5d2d40 0x55d3ed5d7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3ed5c8480 0x55d3ed5db2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 --2- 172.21.15.16:0/3901855078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3ed5c7b60 0x55d3ed5dad70 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdaeff5640 0 client.18852 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 >> 172.21.15.16:0/3901855078 conn(0x55d3ed5c40f0 msgr2=0x55d3ed5d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 shutdown_connections 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 1 -- 172.21.15.16:0/3901855078 wait complete. 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.219+0000 7efdd3b9e740 20 client.18852 trim_cache size 0 max 16384 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "ea6d580d-e5db-43dc-b20b-352425d04d9b", 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:02:07.225 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 msgr2=0x7efdb40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 0x7efdb40a5040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efdc001b000 tx=0x7efdc001d000 comp rx=0 tx=0).stop 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 msgr2=0x7efdcc1a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc0042f0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 shutdown_connections 2024-09-12T03:02:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdcc112a30 0x7efdcc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efdcc077040 0x7efdcc1a4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efdb40a2b80 0x7efdb40a5040 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 --2- 172.21.15.16:0/3638957391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdcc076720 0x7efdcc1a4710 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 >> 172.21.15.16:0/3638957391 conn(0x7efdcc0ffa90 msgr2=0x7efdcc101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 shutdown_connections 2024-09-12T03:02:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:07.220+0000 7efdd17c5640 1 -- 172.21.15.16:0/3638957391 wait complete. 2024-09-12T03:02:07.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: pgmap v763: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-12T03:02:07.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:02:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:07.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:02:07.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: pgmap v763: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-12T03:02:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:02:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: pgmap v763: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.006 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.75 MB/sec warmup 69 sec latency 27323.006 ms 2024-09-12T03:02:08.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:02:08.126 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 -- 172.21.15.16:0/3042650868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 msgr2=0x7f27241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 --2- 172.21.15.16:0/3042650868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241067e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f2714009900 tx=0x7f271402a410 comp rx=0 tx=0).stop 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 -- 172.21.15.16:0/3042650868 shutdown_connections 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 --2- 172.21.15.16:0/3042650868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724106f10 0x7f272410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 --2- 172.21.15.16:0/3042650868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241067e0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 --2- 172.21.15.16:0/3042650868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724101de0 0x7f27241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 -- 172.21.15.16:0/3042650868 >> 172.21.15.16:0/3042650868 conn(0x7f27240fd8c0 msgr2=0x7f27240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 -- 172.21.15.16:0/3042650868 shutdown_connections 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.548+0000 7f272c16b640 1 -- 172.21.15.16:0/3042650868 wait complete. 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 Processor -- start 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 -- start start 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101de0 0x7f27241983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 0x7f272419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2724111f70 con 0x7f2724102790 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2724111df0 con 0x7f2724101de0 2024-09-12T03:02:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f272c16b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27241120f0 con 0x7f2724106f10 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f2729706640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f2729706640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33578/0 (socket says 172.21.15.16:33578) 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.549+0000 7f2729706640 1 -- 172.21.15.16:0/2379535765 learned_addr learned my addr 172.21.15.16:0/2379535765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729f07640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101de0 0x7f27241983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f272a708640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 0x7f272419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 -- 172.21.15.16:0/2379535765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 msgr2=0x7f272419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 0x7f272419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 -- 172.21.15.16:0/2379535765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101de0 msgr2=0x7f27241983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101de0 0x7f27241983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 -- 172.21.15.16:0/2379535765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272419d360 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f272a708640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 0x7f272419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f2729706640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2714006010 tx=0x7f27140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271403f030 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2714003ce0 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272419d5f0 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714035070 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.550+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27241a4f30 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.551+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27241021c0 con 0x7f2724102790 2024-09-12T03:02:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.553+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2714042030 con 0x7f2724102790 2024-09-12T03:02:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.553+0000 7f271affd640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.553+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27140ef850 con 0x7f2724102790 2024-09-12T03:02:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.553+0000 7f2729f07640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.554+0000 7f2729f07640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2720009b60 tx=0x7f2720005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.556+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27140bc880 con 0x7f2724102790 2024-09-12T03:02:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:02:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:02:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:02:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:02:08.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:02:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.782+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2724199710 con 0x7f2724102790 2024-09-12T03:02:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.782+0000 7f271affd640 1 -- 172.21.15.16:0/2379535765 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11983 (secure 0 0 0) 0x7f2724199710 con 0x7f2724102790 2024-09-12T03:02:08.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:02:08.783 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"btime":"2024-09-12T03:01:44:300565+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:43.958785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:02:08.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.784+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 msgr2=0x7f27100a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.784+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2720009b60 tx=0x7f2720005f90 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.784+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 msgr2=0x7f27241988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.784+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2714006010 tx=0x7f27140042f0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 shutdown_connections 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106f10 0x7f272419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724102790 0x7f27241988f0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 --2- 172.21.15.16:0/2379535765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101de0 0x7f27241983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 >> 172.21.15.16:0/2379535765 conn(0x7f27240fd8c0 msgr2=0x7f27240fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 shutdown_connections 2024-09-12T03:02:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:08.785+0000 7f272c16b640 1 -- 172.21.15.16:0/2379535765 wait complete. 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:02:08.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:02:09.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.64 MB/sec warmup 70 sec latency 28323.080 ms 2024-09-12T03:02:09.452 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id f59a6c7 --release --await' 2024-09-12T03:02:09.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id f59a6c7 --release --await 2024-09-12T03:02:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1024137048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 msgr2=0x7f9e081067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1024137048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e081067f0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009900 tx=0x7f9dfc02a410 comp rx=0 tx=0).stop 2024-09-12T03:02:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1024137048 shutdown_connections 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1024137048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 0x7f9e0810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1024137048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e081067f0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1024137048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 0x7f9e081021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1024137048 >> 172.21.15.16:0/1024137048 conn(0x7f9e080fd8f0 msgr2=0x7f9e080ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1024137048 shutdown_connections 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1024137048 wait complete. 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.702+0000 7f9e0dfeb640 1 Processor -- start 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 -- start start 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 0x7f9e08198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 0x7f9e0819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e08111f40 con 0x7f9e081027a0 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e08111dc0 con 0x7f9e08106f20 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e0dfeb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081120c0 con 0x7f9e08101df0 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33592/0 (socket says 172.21.15.16:33592) 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 -- 172.21.15.16:0/1565257182 learned_addr learned my addr 172.21.15.16:0/1565257182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e077fe640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 0x7f9e08198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e07fff640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 0x7f9e0819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 -- 172.21.15.16:0/1565257182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 msgr2=0x7f9e08198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 0x7f9e08198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 -- 172.21.15.16:0/1565257182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 msgr2=0x7f9e0819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 0x7f9e0819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.703+0000 7f9e06ffd640 1 -- 172.21.15.16:0/1565257182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0819d3d0 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005ab0 tx=0x7f9dfc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e04ff9640 1 -- 172.21.15.16:0/1565257182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc03f030 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e04ff9640 1 -- 172.21.15.16:0/1565257182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dfc003ce0 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0819d660 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e04ff9640 1 -- 172.21.15.16:0/1565257182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc035070 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.704+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e081a4fa0 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.706+0000 7f9e04ff9640 1 -- 172.21.15.16:0/1565257182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9dfc042030 con 0x7f9e081027a0 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.706+0000 7f9e103c4740 1 Processor -- start 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 -- start start 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 0x55650d30b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d316140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55650d2712e0 con 0x55650d316680 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55650d218ce0 con 0x55650d30bdc0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e103c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55650d26d2a0 con 0x55650d30b4a0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e04ff9640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 0x7f9df00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33602/0 (socket says 172.21.15.16:33602) 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 -- 172.21.15.16:0/2994457258 learned_addr learned my addr 172.21.15.16:0/2994457258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e07fff640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 0x55650d30b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e077fe640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d316140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e04ff9640 1 -- 172.21.15.16:0/1565257182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9dfc0ef900 con 0x7f9e081027a0 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 -- 172.21.15.16:0/2994457258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 msgr2=0x55650d30b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 0x55650d30b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 -- 172.21.15.16:0/2994457258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 msgr2=0x55650d316140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d316140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.707+0000 7f9e06ffd640 1 -- 172.21.15.16:0/2994457258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55650d1fd440 con 0x55650d316680 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9e07fff640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 0x55650d30b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9e07fff640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 0x7f9df00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005ec0 tx=0x7f9dfc03b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1faab8b183dcdb97 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9dedffb640 1 -- 172.21.15.16:0/2994457258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc049070 con 0x55650d316680 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9dedffb640 1 -- 172.21.15.16:0/2994457258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dfc0bc870 con 0x55650d316680 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.708+0000 7f9dedffb640 1 -- 172.21.15.16:0/2994457258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc03be10 con 0x55650d316680 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- 172.21.15.16:0/2994457258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 msgr2=0x55650d318a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005ec0 tx=0x7f9dfc03b790 comp rx=0 tx=0).stop 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e07fff640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 0x7f9df00a5040 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000a2c0 tx=0x7f9e00004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- 172.21.15.16:0/2994457258 shutdown_connections 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d316680 0x55650d318a70 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d316140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 --2- 172.21.15.16:0/2994457258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30b4a0 0x55650d30b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- 172.21.15.16:0/2994457258 >> 172.21.15.16:0/2994457258 conn(0x55650d307a30 msgr2=0x55650d307e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- 172.21.15.16:0/2994457258 shutdown_connections 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- 172.21.15.16:0/2994457258 wait complete. 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.709+0000 7f9e103c4740 1 -- start start 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 Processor -- start 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30b4a0 0x55650d316020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30bdc0 0x55650d3132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55650d319010 con 0x55650d30bdc0 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55650d22c050 con 0x55650d30b4a0 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55650d319190 con 0x55650d316680 2024-09-12T03:02:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e103c4740 10 client.0 _ll_register_callbacks cb 0x55650d26c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33480/0 (socket says 172.21.15.16:33480) 2024-09-12T03:02:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e07fff640 1 -- 172.21.15.16:0/3362544694 learned_addr learned my addr 172.21.15.16:0/3362544694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30b4a0 0x55650d316020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.710+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30bdc0 0x55650d3132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 -- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30b4a0 msgr2=0x55650d316020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30b4a0 0x55650d316020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 -- 172.21.15.16:0/3362544694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30bdc0 msgr2=0x55650d3132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30bdc0 0x55650d3132d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 -- 172.21.15.16:0/3362544694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55650cfe23c0 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9e0001ee50 tx=0x7f9e00004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ba7f6263184a1b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e00027640 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dfc0bc870 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9e0001b410 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9e0000abf0 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e103c4740 10 client.28117 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.711+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55650d324a90 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dfc0bc870 con 0x55650d316680 2024-09-12T03:02:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f9e00025070 con 0x55650d316680 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 10 client.28117 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 10 client.28117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 20 client.28117 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d325ea0 0x55650d328300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 4 client.28117 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55650d27e850 con 0x55650d325ea0 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d325ea0 0x55650d328300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9de6ffd640 10 client.28117 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.712+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d325ea0 0x55650d328300 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a6dc0230d0c3259 server_cookie=51f0ad26582e1b47 in_seq=0 out_seq=0 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3362544694 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55650d27e850 con 0x55650d325ea0 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9de6ffd640 10 client.28117 handle_command_reply: tid=0 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 client.28117 shutdown 2024-09-12T03:02:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 msgr2=0x55650d313870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9e0001ee50 tx=0x7f9e00004ab0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 shutdown_connections 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d325ea0 0x55650d328300 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d313870 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30bdc0 0x55650d3132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3362544694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30b4a0 0x55650d316020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 >> 172.21.15.16:0/3362544694 conn(0x55650d307a30 msgr2=0x55650d30d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.713+0000 7f9de6ffd640 0 client.28117 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.714+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 shutdown_connections 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.714+0000 7f9e103c4740 1 -- 172.21.15.16:0/3362544694 wait complete. 2024-09-12T03:02:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.714+0000 7f9e103c4740 20 client.28117 trim_cache size 0 max 16384 2024-09-12T03:02:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.732+0000 7f9e103c4740 1 Processor -- start 2024-09-12T03:02:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.732+0000 7f9e103c4740 1 -- start start 2024-09-12T03:02:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d315240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d32dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081ad4f0 con 0x55650d30b4a0 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081af0f0 con 0x55650d30bdc0 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e103c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081adbc0 con 0x55650d316680 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33624/0 (socket says 172.21.15.16:33624) 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 -- 172.21.15.16:0/3946657369 learned_addr learned my addr 172.21.15.16:0/3946657369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d315240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d32dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 -- 172.21.15.16:0/3946657369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 msgr2=0x55650d32dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d32dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 -- 172.21.15.16:0/3946657369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 msgr2=0x55650d315240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d315240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 -- 172.21.15.16:0/3946657369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55650d311820 con 0x55650d30b4a0 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d32dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.733+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x55650d328b40 tx=0x7f9df800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ffe7282d43ed19 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.734+0000 7f9ded7fa640 1 -- 172.21.15.16:0/3946657369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df801d6b0 con 0x55650d30b4a0 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.734+0000 7f9ded7fa640 1 -- 172.21.15.16:0/3946657369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9df801dd30 con 0x55650d30b4a0 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.734+0000 7f9ded7fa640 1 -- 172.21.15.16:0/3946657369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df800f420 con 0x55650d30b4a0 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.734+0000 7f9e103c4740 1 -- 172.21.15.16:0/3946657369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 msgr2=0x55650d322270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.734+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x55650d328b40 tx=0x7f9df800a2d0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 -- 172.21.15.16:0/3946657369 shutdown_connections 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d316680 0x55650d32dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d30bdc0 0x55650d315240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3946657369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d322270 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 -- 172.21.15.16:0/3946657369 >> 172.21.15.16:0/3946657369 conn(0x55650d307a30 msgr2=0x55650d314b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 -- 172.21.15.16:0/3946657369 shutdown_connections 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 -- 172.21.15.16:0/3946657369 wait complete. 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 -- start start 2024-09-12T03:02:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.735+0000 7f9e103c4740 1 Processor -- start 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30bdc0 0x55650d31aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d316680 0x55650d31af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081aea20 con 0x55650d30b4a0 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081ae350 con 0x55650d316680 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e081ab4d0 con 0x55650d30bdc0 2024-09-12T03:02:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e103c4740 10 client.0 _ll_register_callbacks cb 0x7f9e0000af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33640/0 (socket says 172.21.15.16:33640) 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 -- 172.21.15.16:0/3355587756 learned_addr learned my addr 172.21.15.16:0/3355587756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30bdc0 0x55650d31aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 -- 172.21.15.16:0/3355587756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30bdc0 msgr2=0x55650d31aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30bdc0 0x55650d31aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 -- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d316680 msgr2=0x55650d31af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e07fff640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d316680 0x55650d31af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d316680 0x55650d31af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.736+0000 7f9e077fe640 1 -- 172.21.15.16:0/3355587756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55650d31b4a0 con 0x55650d30b4a0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e077fe640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f9df8000c00 tx=0x7f9df800ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59778abf888f7c39 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8005030 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9df80051d0 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9df801dd30 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9df8005410 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e103c4740 10 client.18876 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55650d39a080 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9df801dd30 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f9df8004a00 con 0x55650d30b4a0 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e103c4740 10 client.18876 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.737+0000 7f9e103c4740 10 client.18876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e103c4740 20 client.18876 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d39a780 0x55650d39cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e103c4740 4 client.18876 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:02:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55650d278d60 con 0x55650d39a780 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d39a780 0x55650d39cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9e06ffd640 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d39a780 0x55650d39cba0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96fb28b9aa735a2e server_cookie=3bc3e820866a7b7f in_seq=0 out_seq=0 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.738+0000 7f9de6ffd640 10 client.18876 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9de6ffd640 1 -- 172.21.15.16:0/3355587756 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55650d278d60 con 0x55650d39a780 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9de6ffd640 10 client.18876 handle_command_reply: tid=0 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 client.18876 shutdown 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 msgr2=0x55650d31e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f9df8000c00 tx=0x7f9df800ac40 comp rx=0 tx=0).stop 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 shutdown_connections 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55650d39a780 0x55650d39cba0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55650d316680 0x55650d31af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55650d30bdc0 0x55650d31aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 --2- 172.21.15.16:0/3355587756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55650d30b4a0 0x55650d31e3c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 >> 172.21.15.16:0/3355587756 conn(0x55650d307a30 msgr2=0x55650d315350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9de6ffd640 0 client.18876 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 shutdown_connections 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 1 -- 172.21.15.16:0/3355587756 wait complete. 2024-09-12T03:02:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.739+0000 7f9e103c4740 20 client.18876 trim_cache size 0 max 16384 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "ea6d580d-e5db-43dc-b20b-352425d04d9b": { 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "ea6d580d-e5db-43dc-b20b-352425d04d9b", 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:09.746 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 msgr2=0x7f9df00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 0x7f9df00a5040 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000a2c0 tx=0x7f9e00004a40 comp rx=0 tx=0).stop 2024-09-12T03:02:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 msgr2=0x7f9e08198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005ab0 tx=0x7f9dfc0042f0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 shutdown_connections 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e08106f20 0x7f9e0819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9df00a2b80 0x7f9df00a5040 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e081027a0 0x7f9e08198960 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 --2- 172.21.15.16:0/1565257182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e08101df0 0x7f9e08198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 >> 172.21.15.16:0/1565257182 conn(0x7f9e080fd8f0 msgr2=0x7f9e080fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 shutdown_connections 2024-09-12T03:02:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:09.740+0000 7f9e0dfeb640 1 -- 172.21.15.16:0/1565257182 wait complete. 2024-09-12T03:02:10.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.53 MB/sec warmup 71 sec latency 29323.126 ms 2024-09-12T03:02:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:09 smithi184 ceph-mon[35321]: pgmap v764: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-09-12T03:02:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:09 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2379535765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:10.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:09 smithi016 ceph-mon[30925]: pgmap v764: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-09-12T03:02:10.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:09 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2379535765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:09 smithi033 ceph-mon[35092]: pgmap v764: 97 pgs: 97 active+clean; 467 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-09-12T03:02:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:09 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2379535765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:10.478 INFO:tasks.cephfs.filesystem:scrub status for tag:ea6d580d-e5db-43dc-b20b-352425d04d9b - {'path': '/', 'tag': 'ea6d580d-e5db-43dc-b20b-352425d04d9b', 'options': 'recursive,force'} 2024-09-12T03:02:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:11.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6936 7.43 MB/sec warmup 72 sec latency 30323.157 ms 2024-09-12T03:02:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 -- 172.21.15.16:0/1181492915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9581049c0 msgr2=0x7fb958108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 --2- 172.21.15.16:0/1181492915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9581049c0 0x7fb958108a10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009900 tx=0x7fb94c02a3f0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 -- 172.21.15.16:0/1181492915 shutdown_connections 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 --2- 172.21.15.16:0/1181492915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb958110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 --2- 172.21.15.16:0/1181492915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9581049c0 0x7fb958108a10 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 --2- 172.21.15.16:0/1181492915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb958104010 0x7fb9581043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 -- 172.21.15.16:0/1181492915 >> 172.21.15.16:0/1181492915 conn(0x7fb9580ffa50 msgr2=0x7fb958101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 -- 172.21.15.16:0/1181492915 shutdown_connections 2024-09-12T03:02:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.008+0000 7fb95dba3640 1 -- 172.21.15.16:0/1181492915 wait complete. 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.009+0000 7fb95dba3640 1 Processor -- start 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.009+0000 7fb95dba3640 1 -- start start 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.009+0000 7fb95dba3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.009+0000 7fb95dba3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9581049c0 0x7fb95819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.009+0000 7fb95dba3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb95819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb95dba3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb958114140 con 0x7fb958104010 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb95dba3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb958113fc0 con 0x7fb9581049c0 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb95dba3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581142c0 con 0x7fb958109140 2024-09-12T03:02:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33652/0 (socket says 172.21.15.16:33652) 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 -- 172.21.15.16:0/3191507008 learned_addr learned my addr 172.21.15.16:0/3191507008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb956ffd640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9581049c0 0x7fb95819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb957fff640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb95819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 -- 172.21.15.16:0/3191507008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 msgr2=0x7fb95819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb95819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 -- 172.21.15.16:0/3191507008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9581049c0 msgr2=0x7fb95819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9581049c0 0x7fb95819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 -- 172.21.15.16:0/3191507008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95819f5d0 con 0x7fb958104010 2024-09-12T03:02:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb957fff640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb95819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb94800cce0 tx=0x7fb94800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb954ff9640 1 -- 172.21.15.16:0/3191507008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb948004b20 con 0x7fb958104010 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb954ff9640 1 -- 172.21.15.16:0/3191507008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb948004cc0 con 0x7fb958104010 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95819f8c0 con 0x7fb958104010 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.010+0000 7fb954ff9640 1 -- 172.21.15.16:0/3191507008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94800f7d0 con 0x7fb958104010 2024-09-12T03:02:11.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.011+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9581a71a0 con 0x7fb958104010 2024-09-12T03:02:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb954ff9640 1 -- 172.21.15.16:0/3191507008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb94800f970 con 0x7fb958104010 2024-09-12T03:02:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 Processor -- start 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 -- start start 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c32440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32980 0x55ebf3c3cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3b97620 con 0x55ebf3c3d240 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3b3ee50 con 0x55ebf3c32060 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.013+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3b935e0 con 0x55ebf3c32980 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33658/0 (socket says 172.21.15.16:33658) 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 -- 172.21.15.16:0/2414487932 learned_addr learned my addr 172.21.15.16:0/2414487932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb954ff9640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 0x7fb9400a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb9577fe640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32980 0x55ebf3c3cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb957fff640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c32440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb957fff640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 0x7fb9400a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 -- 172.21.15.16:0/2414487932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32980 msgr2=0x55ebf3c3cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32980 0x55ebf3c3cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 -- 172.21.15.16:0/2414487932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 msgr2=0x55ebf3c32440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c32440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb954ff9640 1 -- 172.21.15.16:0/3191507008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb9480cf1b0 con 0x7fb958104010 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 -- 172.21.15.16:0/2414487932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf3b2fd40 con 0x55ebf3c3d240 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb957fff640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c32440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.014+0000 7fb956ffd640 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005ab0 tx=0x7fb94c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdf9c108380e6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb93dffb640 1 -- 172.21.15.16:0/2414487932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c03f070 con 0x55ebf3c3d240 2024-09-12T03:02:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb957fff640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 0x7fb9400a5040 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb950005fd0 tx=0x7fb9500058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb93dffb640 1 -- 172.21.15.16:0/2414487932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb94c031a20 con 0x55ebf3c3d240 2024-09-12T03:02:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb93dffb640 1 -- 172.21.15.16:0/2414487932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c035070 con 0x55ebf3c3d240 2024-09-12T03:02:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2414487932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 msgr2=0x55ebf3c3f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005ab0 tx=0x7fb94c031630 comp rx=0 tx=0).stop 2024-09-12T03:02:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2414487932 shutdown_connections 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3f630 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32980 0x55ebf3c3cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2414487932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c32440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2414487932 >> 172.21.15.16:0/2414487932 conn(0x55ebf3c2e5f0 msgr2=0x55ebf3c2e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2414487932 shutdown_connections 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.015+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2414487932 wait complete. 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 -- start start 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 Processor -- start 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32060 0x55ebf3c3a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c3a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3c3fe10 con 0x55ebf3c3d240 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3b1ff10 con 0x55ebf3c32980 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebf3c3ff90 con 0x55ebf3c32060 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb95ff7c740 10 client.0 _ll_register_callbacks cb 0x55ebf3b43db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb957fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb957fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33672/0 (socket says 172.21.15.16:33672) 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.016+0000 7fb957fff640 1 -- 172.21.15.16:0/2708508479 learned_addr learned my addr 172.21.15.16:0/2708508479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb9577fe640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32060 0x55ebf3c3a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb956ffd640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c3a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 -- 172.21.15.16:0/2708508479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32060 msgr2=0x55ebf3c3a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32060 0x55ebf3c3a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 -- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 msgr2=0x55ebf3c3a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c3a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 -- 172.21.15.16:0/2708508479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf38a6580 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb956ffd640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c3a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb957fff640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb95000fa50 tx=0x7fb9500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332237e04a7cc025 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb936ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95000eda0 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb95001d410 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb94c031a20 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb95001d610 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb95ff7c740 10 client.18888 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.017+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ebf3970960 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb94c031a20 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fb950019070 con 0x55ebf3c3d240 2024-09-12T03:02:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 10 client.18888 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 10 client.18888 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 20 client.18888 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3c4cd00 0x55ebf3c4f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 4 client.18888 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebf3ba4b90 con 0x55ebf3c4cd00 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.018+0000 7fb9577fe640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3c4cd00 0x55ebf3c4f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.019+0000 7fb936ffd640 10 client.18888 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.019+0000 7fb9577fe640 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3c4cd00 0x55ebf3c4f1a0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf1dfb95166b0af2 server_cookie=76217745c5b70da7 in_seq=0 out_seq=0 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.019+0000 7fb936ffd640 1 -- 172.21.15.16:0/2708508479 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ebf3ba4b90 con 0x55ebf3c4cd00 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.019+0000 7fb936ffd640 10 client.18888 handle_command_reply: tid=0 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.019+0000 7fb95ff7c740 1 client.18888 shutdown 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 msgr2=0x55ebf3c3ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb95000fa50 tx=0x7fb9500046c0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 shutdown_connections 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3c4cd00 0x55ebf3c4f1a0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c3d240 0x55ebf3c3ab20 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c3a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/2708508479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c32060 0x55ebf3c3a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 >> 172.21.15.16:0/2708508479 conn(0x55ebf3c2e5f0 msgr2=0x55ebf3c342d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb936ffd640 0 client.18888 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 shutdown_connections 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 1 -- 172.21.15.16:0/2708508479 wait complete. 2024-09-12T03:02:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.020+0000 7fb95ff7c740 20 client.18888 trim_cache size 0 max 16384 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 Processor -- start 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 -- start start 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c55000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c37300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581af7a0 con 0x55ebf3c32060 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581b13a0 con 0x55ebf3c32980 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581afe70 con 0x55ebf3c3d240 2024-09-12T03:02:11.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33686/0 (socket says 172.21.15.16:33686) 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb9577fe640 1 -- 172.21.15.16:0/3599093310 learned_addr learned my addr 172.21.15.16:0/3599093310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.037+0000 7fb956ffd640 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c55000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 -- 172.21.15.16:0/3599093310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 msgr2=0x55ebf3c37300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c37300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 -- 172.21.15.16:0/3599093310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 msgr2=0x55ebf3c55000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c55000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 -- 172.21.15.16:0/3599093310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf3c2e420 con 0x55ebf3c32060 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb956ffd640 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c55000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb9577fe640 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb9480078d0 tx=0x7fb9480c65b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a5a762148ceed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb93d7fa640 1 -- 172.21.15.16:0/3599093310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94801d070 con 0x55ebf3c32060 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb93d7fa640 1 -- 172.21.15.16:0/3599093310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9480ce040 con 0x55ebf3c32060 2024-09-12T03:02:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.038+0000 7fb93d7fa640 1 -- 172.21.15.16:0/3599093310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9480a4200 con 0x55ebf3c32060 2024-09-12T03:02:11.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- 172.21.15.16:0/3599093310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 msgr2=0x55ebf3c54ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb9480078d0 tx=0x7fb9480c65b0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- 172.21.15.16:0/3599093310 shutdown_connections 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c37300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32980 0x55ebf3c55000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/3599093310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32060 0x55ebf3c54ac0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- 172.21.15.16:0/3599093310 >> 172.21.15.16:0/3599093310 conn(0x55ebf3c2e5f0 msgr2=0x55ebf3c38970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- 172.21.15.16:0/3599093310 shutdown_connections 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- 172.21.15.16:0/3599093310 wait complete. 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 -- start start 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 Processor -- start 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.039+0000 7fb95ff7c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c401a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581b0cd0 con 0x55ebf3c32980 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581b0600 con 0x55ebf3c32060 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb95ff7c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581ad780 con 0x55ebf3c3d240 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb95ff7c740 10 client.0 _ll_register_callbacks cb 0x7fb95000f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33688/0 (socket says 172.21.15.16:33688) 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 -- 172.21.15.16:0/145376114 learned_addr learned my addr 172.21.15.16:0/145376114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb957fff640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb9577fe640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 -- 172.21.15.16:0/145376114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 msgr2=0x55ebf3c401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 -- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 msgr2=0x55ebf3c453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 -- 172.21.15.16:0/145376114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf3c40890 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb956ffd640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005ab0 tx=0x7fb94c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9185e09492a1493d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb936ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c035040 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb94c005230 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.040+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9480ce040 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb94c03f040 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 10 client.18894 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ebf3cb8420 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9480ce040 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fb94c003710 con 0x55ebf3c32980 2024-09-12T03:02:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 10 client.18894 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 10 client.18894 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 20 client.18894 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3cb8700 0x55ebf3cbaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 4 client.18894 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f59a6c7", "release": true, "await": true}] 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f59a6c7", "release": true, "await": true}) -- 0x55ebf3c29950 con 0x55ebf3cb8700 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.041+0000 7fb9577fe640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3cb8700 0x55ebf3cbaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.042+0000 7fb936ffd640 10 client.18894 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.042+0000 7fb9577fe640 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3cb8700 0x55ebf3cbaaf0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d12cbe6c5c63f7ea server_cookie=6c98ef4626a7b3df in_seq=0 out_seq=0 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.091+0000 7fb936ffd640 1 -- 172.21.15.16:0/145376114 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ebf3c29950 con 0x55ebf3cb8700 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.091+0000 7fb936ffd640 10 client.18894 handle_command_reply: tid=0 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.091+0000 7fb95ff7c740 1 client.18894 shutdown 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.091+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 msgr2=0x55ebf3c45910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.091+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005ab0 tx=0x7fb94c0043b0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 shutdown_connections 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ebf3cb8700 0x55ebf3cbaaf0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebf3c3d240 0x55ebf3c401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ebf3c32980 0x55ebf3c45910 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 --2- 172.21.15.16:0/145376114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ebf3c32060 0x55ebf3c453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 >> 172.21.15.16:0/145376114 conn(0x55ebf3c2e5f0 msgr2=0x55ebf3c2d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb936ffd640 0 client.18894 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 shutdown_connections 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 1 -- 172.21.15.16:0/145376114 wait complete. 2024-09-12T03:02:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95ff7c740 20 client.18894 trim_cache size 0 max 16384 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 35, 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 56, 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "f59a6c7": { 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 56, 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:02:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:11.096 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 msgr2=0x7fb9400a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 0x7fb9400a5040 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb950005fd0 tx=0x7fb9500058e0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 msgr2=0x7fb95819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb94800cce0 tx=0x7fb94800a620 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 shutdown_connections 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb958109140 0x7fb95819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9581049c0 0x7fb95819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9400a2b80 0x7fb9400a5040 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 --2- 172.21.15.16:0/3191507008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958104010 0x7fb95819a620 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.092+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 >> 172.21.15.16:0/3191507008 conn(0x7fb9580ffa50 msgr2=0x7fb958100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.093+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 shutdown_connections 2024-09-12T03:02:11.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:11.093+0000 7fb95dba3640 1 -- 172.21.15.16:0/3191507008 wait complete. 2024-09-12T03:02:11.896 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f59a6c7', total seconds elapsed: 34.1 2024-09-12T03:02:12.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7585 7.94 MB/sec warmup 73 sec latency 30411.355 ms 2024-09-12T03:02:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:11 smithi184 ceph-mon[35321]: pgmap v765: 97 pgs: 97 active+clean; 474 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.4 MiB/s wr, 49 op/s 2024-09-12T03:02:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:11 smithi016 ceph-mon[30925]: pgmap v765: 97 pgs: 97 active+clean; 474 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.4 MiB/s wr, 49 op/s 2024-09-12T03:02:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:11 smithi033 ceph-mon[35092]: pgmap v765: 97 pgs: 97 active+clean; 474 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.4 MiB/s wr, 49 op/s 2024-09-12T03:02:13.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 8253 8.43 MB/sec warmup 74 sec latency 86.698 ms 2024-09-12T03:02:14.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 8922 8.87 MB/sec warmup 75 sec latency 76.206 ms 2024-09-12T03:02:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:13 smithi184 ceph-mon[35321]: pgmap v766: 97 pgs: 97 active+clean; 478 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.1 MiB/s wr, 75 op/s 2024-09-12T03:02:14.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:13 smithi016 ceph-mon[30925]: pgmap v766: 97 pgs: 97 active+clean; 478 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.1 MiB/s wr, 75 op/s 2024-09-12T03:02:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:13 smithi033 ceph-mon[35092]: pgmap v766: 97 pgs: 97 active+clean; 478 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.1 MiB/s wr, 75 op/s 2024-09-12T03:02:15.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9338 9.02 MB/sec warmup 76 sec latency 95.943 ms 2024-09-12T03:02:16.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9913 9.18 MB/sec warmup 77 sec latency 106.248 ms 2024-09-12T03:02:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:15 smithi184 ceph-mon[35321]: pgmap v767: 97 pgs: 97 active+clean; 461 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-09-12T03:02:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:15 smithi016 ceph-mon[30925]: pgmap v767: 97 pgs: 97 active+clean; 461 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-09-12T03:02:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:15 smithi033 ceph-mon[35092]: pgmap v767: 97 pgs: 97 active+clean; 461 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-09-12T03:02:17.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10642 9.54 MB/sec warmup 78 sec latency 63.500 ms 2024-09-12T03:02:18.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 11490 9.93 MB/sec warmup 79 sec latency 44.308 ms 2024-09-12T03:02:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:17 smithi184 ceph-mon[35321]: pgmap v768: 97 pgs: 97 active+clean; 456 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-09-12T03:02:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:17 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:02:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:17 smithi016 ceph-mon[30925]: pgmap v768: 97 pgs: 97 active+clean; 456 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-09-12T03:02:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:17 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:02:18.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:17 smithi033 ceph-mon[35092]: pgmap v768: 97 pgs: 97 active+clean; 456 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-09-12T03:02:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:17 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:02:19.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 12235 10.36 MB/sec warmup 80 sec latency 58.723 ms 2024-09-12T03:02:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:02:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:02:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:02:19.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:02:19.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:02:19.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:02:19.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:02:19.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:02:19.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:02:19.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:02:19.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:02:19.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:02:20.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 13116 10.95 MB/sec warmup 81 sec latency 48.577 ms 2024-09-12T03:02:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:19 smithi184 ceph-mon[35321]: pgmap v769: 97 pgs: 97 active+clean; 455 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T03:02:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:19 smithi016 ceph-mon[30925]: pgmap v769: 97 pgs: 97 active+clean; 455 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T03:02:20.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:19 smithi033 ceph-mon[35092]: pgmap v769: 97 pgs: 97 active+clean; 455 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.3 MiB/s wr, 252 op/s 2024-09-12T03:02:21.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 13704 11.25 MB/sec warmup 82 sec latency 44.087 ms 2024-09-12T03:02:22.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 14144 11.35 MB/sec warmup 83 sec latency 69.929 ms 2024-09-12T03:02:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:21 smithi184 ceph-mon[35321]: pgmap v770: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 458 op/s 2024-09-12T03:02:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:21 smithi016 ceph-mon[30925]: pgmap v770: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 458 op/s 2024-09-12T03:02:22.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:21 smithi033 ceph-mon[35092]: pgmap v770: 97 pgs: 97 active+clean; 465 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.8 MiB/s wr, 458 op/s 2024-09-12T03:02:23.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 14778 11.52 MB/sec warmup 84 sec latency 157.575 ms 2024-09-12T03:02:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 15642 11.92 MB/sec warmup 85 sec latency 54.951 ms 2024-09-12T03:02:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:23 smithi184 ceph-mon[35321]: pgmap v771: 97 pgs: 97 active+clean; 467 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2024-09-12T03:02:24.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:23 smithi033 ceph-mon[35092]: pgmap v771: 97 pgs: 97 active+clean; 467 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2024-09-12T03:02:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:23 smithi016 ceph-mon[30925]: pgmap v771: 97 pgs: 97 active+clean; 467 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2024-09-12T03:02:25.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 16446 12.28 MB/sec warmup 86 sec latency 33.197 ms 2024-09-12T03:02:26.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 17353 12.77 MB/sec warmup 87 sec latency 44.528 ms 2024-09-12T03:02:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:25 smithi184 ceph-mon[35321]: pgmap v772: 97 pgs: 97 active+clean; 471 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 484 op/s 2024-09-12T03:02:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:25 smithi016 ceph-mon[30925]: pgmap v772: 97 pgs: 97 active+clean; 471 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 484 op/s 2024-09-12T03:02:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:25 smithi033 ceph-mon[35092]: pgmap v772: 97 pgs: 97 active+clean; 471 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 484 op/s 2024-09-12T03:02:27.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18109 13.15 MB/sec warmup 88 sec latency 56.492 ms 2024-09-12T03:02:28.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18659 13.31 MB/sec warmup 89 sec latency 64.167 ms 2024-09-12T03:02:28.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:27 smithi184 ceph-mon[35321]: pgmap v773: 97 pgs: 97 active+clean; 482 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.3 MiB/s wr, 579 op/s 2024-09-12T03:02:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:27 smithi016 ceph-mon[30925]: pgmap v773: 97 pgs: 97 active+clean; 482 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.3 MiB/s wr, 579 op/s 2024-09-12T03:02:28.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:27 smithi033 ceph-mon[35092]: pgmap v773: 97 pgs: 97 active+clean; 482 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.3 MiB/s wr, 579 op/s 2024-09-12T03:02:29.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 19250 13.46 MB/sec warmup 90 sec latency 55.216 ms 2024-09-12T03:02:30.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 20065 13.78 MB/sec warmup 91 sec latency 51.230 ms 2024-09-12T03:02:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:29 smithi184 ceph-mon[35321]: pgmap v774: 97 pgs: 97 active+clean; 484 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 476 op/s 2024-09-12T03:02:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:29 smithi016 ceph-mon[30925]: pgmap v774: 97 pgs: 97 active+clean; 484 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 476 op/s 2024-09-12T03:02:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:29 smithi033 ceph-mon[35092]: pgmap v774: 97 pgs: 97 active+clean; 484 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 476 op/s 2024-09-12T03:02:31.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 20876 14.03 MB/sec warmup 92 sec latency 185.499 ms 2024-09-12T03:02:32.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 21588 14.34 MB/sec warmup 93 sec latency 126.586 ms 2024-09-12T03:02:32.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:31 smithi184 ceph-mon[35321]: pgmap v775: 97 pgs: 97 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.7 MiB/s wr, 613 op/s 2024-09-12T03:02:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:31 smithi016 ceph-mon[30925]: pgmap v775: 97 pgs: 97 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.7 MiB/s wr, 613 op/s 2024-09-12T03:02:32.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:31 smithi033 ceph-mon[35092]: pgmap v775: 97 pgs: 97 active+clean; 495 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.7 MiB/s wr, 613 op/s 2024-09-12T03:02:33.007 INFO:tasks.workunit.client.0.smithi016.stdout: 10 22422 14.73 MB/sec warmup 94 sec latency 41.379 ms 2024-09-12T03:02:34.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 23127 15.02 MB/sec warmup 95 sec latency 54.192 ms 2024-09-12T03:02:34.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:33 smithi184 ceph-mon[35321]: pgmap v776: 97 pgs: 97 active+clean; 502 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 470 op/s 2024-09-12T03:02:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:33 smithi016 ceph-mon[30925]: pgmap v776: 97 pgs: 97 active+clean; 502 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 470 op/s 2024-09-12T03:02:34.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:33 smithi033 ceph-mon[35092]: pgmap v776: 97 pgs: 97 active+clean; 502 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 470 op/s 2024-09-12T03:02:35.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 23653 15.14 MB/sec warmup 96 sec latency 38.164 ms 2024-09-12T03:02:36.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24428 15.34 MB/sec warmup 97 sec latency 54.686 ms 2024-09-12T03:02:36.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:35 smithi184 ceph-mon[35321]: pgmap v777: 97 pgs: 97 active+clean; 507 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.2 MiB/s wr, 501 op/s 2024-09-12T03:02:36.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:35 smithi016 ceph-mon[30925]: pgmap v777: 97 pgs: 97 active+clean; 507 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.2 MiB/s wr, 501 op/s 2024-09-12T03:02:36.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:35 smithi033 ceph-mon[35092]: pgmap v777: 97 pgs: 97 active+clean; 507 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.2 MiB/s wr, 501 op/s 2024-09-12T03:02:37.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 25356 15.66 MB/sec warmup 98 sec latency 91.835 ms 2024-09-12T03:02:38.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 26124 15.98 MB/sec warmup 99 sec latency 48.527 ms 2024-09-12T03:02:38.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:37 smithi184 ceph-mon[35321]: pgmap v778: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 545 op/s 2024-09-12T03:02:38.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:37 smithi016 ceph-mon[30925]: pgmap v778: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 545 op/s 2024-09-12T03:02:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:37 smithi033 ceph-mon[35092]: pgmap v778: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 545 op/s 2024-09-12T03:02:39.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 26911 16.23 MB/sec warmup 100 sec latency 55.000 ms 2024-09-12T03:02:40.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27631 16.54 MB/sec warmup 101 sec latency 57.927 ms 2024-09-12T03:02:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:39 smithi184 ceph-mon[35321]: pgmap v779: 97 pgs: 97 active+clean; 517 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:02:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:39 smithi016 ceph-mon[30925]: pgmap v779: 97 pgs: 97 active+clean; 517 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:02:40.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:39 smithi033 ceph-mon[35092]: pgmap v779: 97 pgs: 97 active+clean; 517 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:02:40.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:02:40.802 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:41.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 28157 16.58 MB/sec warmup 102 sec latency 115.460 ms 2024-09-12T03:02:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 -- 172.21.15.16:0/554019379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 msgr2=0x7efe1c110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 --2- 172.21.15.16:0/554019379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c110ca0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7efe14009be0 tx=0x7efe1402a6c0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 -- 172.21.15.16:0/554019379 shutdown_connections 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 --2- 172.21.15.16:0/554019379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c110ca0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 --2- 172.21.15.16:0/554019379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 0x7efe1c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 --2- 172.21.15.16:0/554019379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 0x7efe1c1043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 -- 172.21.15.16:0/554019379 >> 172.21.15.16:0/554019379 conn(0x7efe1c0ffab0 msgr2=0x7efe1c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 -- 172.21.15.16:0/554019379 shutdown_connections 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.460+0000 7efe20ded640 1 -- 172.21.15.16:0/554019379 wait complete. 2024-09-12T03:02:41.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.461+0000 7efe20ded640 1 Processor -- start 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 -- start start 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 0x7efe1c110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 0x7efe1c10dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c114130 con 0x7efe1c109120 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c113fb0 con 0x7efe1c103ff0 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe20ded640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1142b0 con 0x7efe1c1049a0 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 0x7efe1c110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33934/0 (socket says 172.21.15.16:33934) 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe1ad9d640 1 -- 172.21.15.16:0/1703196586 learned_addr learned my addr 172.21.15.16:0/1703196586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.462+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 0x7efe1c10dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 -- 172.21.15.16:0/1703196586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 msgr2=0x7efe1c10dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 0x7efe1c10dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 -- 172.21.15.16:0/1703196586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 msgr2=0x7efe1c110b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 0x7efe1c110b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 -- 172.21.15.16:0/1703196586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1c10e990 con 0x7efe1c109120 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe1ad9d640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7efe14009b30 tx=0x7efe140305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe13fff640 1 -- 172.21.15.16:0/1703196586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14041020 con 0x7efe1c109120 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1c199660 con 0x7efe1c109120 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1c199bf0 con 0x7efe1c109120 2024-09-12T03:02:41.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe13fff640 1 -- 172.21.15.16:0/1703196586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe14003e50 con 0x7efe1c109120 2024-09-12T03:02:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.463+0000 7efe13fff640 1 -- 172.21.15.16:0/1703196586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14036040 con 0x7efe1c109120 2024-09-12T03:02:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.467+0000 7efe13fff640 1 -- 172.21.15.16:0/1703196586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efe14044020 con 0x7efe1c109120 2024-09-12T03:02:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.467+0000 7efe231c6740 1 Processor -- start 2024-09-12T03:02:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.467+0000 7efe231c6740 1 -- start start 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b242880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b24d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55669b1a82e0 con 0x55669b242dc0 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55669b14fce0 con 0x55669b24d680 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe231c6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55669b1a42a0 con 0x55669b2424a0 2024-09-12T03:02:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe13fff640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 0x7efe080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b24d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b24d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33948/0 (socket says 172.21.15.16:33948) 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe1a59c640 1 -- 172.21.15.16:0/1697280660 learned_addr learned my addr 172.21.15.16:0/1697280660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b242880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe13fff640 1 -- 172.21.15.16:0/1703196586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efe140efcd0 con 0x7efe1c109120 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe1ad9d640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 0x7efe080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 -- 172.21.15.16:0/1697280660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 msgr2=0x55669b24fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 -- 172.21.15.16:0/1697280660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 msgr2=0x55669b24d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b24d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.468+0000 7efe11d9b640 1 -- 172.21.15.16:0/1697280660 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55669b134440 con 0x55669b2424a0 2024-09-12T03:02:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.469+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b242880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe1c10f380 tx=0x7efe0c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15f4fa08b4c339c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.469+0000 7efe1ad9d640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 0x7efe080a5040 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efe140ed7e0 tx=0x7efe14031900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.469+0000 7efe10d99640 1 -- 172.21.15.16:0/1697280660 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004050 con 0x55669b2424a0 2024-09-12T03:02:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.469+0000 7efe10d99640 1 -- 172.21.15.16:0/1697280660 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe0c005500 con 0x55669b2424a0 2024-09-12T03:02:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.469+0000 7efe10d99640 1 -- 172.21.15.16:0/1697280660 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004d30 con 0x55669b2424a0 2024-09-12T03:02:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 -- 172.21.15.16:0/1697280660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 msgr2=0x55669b242880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b242880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe1c10f380 tx=0x7efe0c004730 comp rx=0 tx=0).stop 2024-09-12T03:02:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 -- 172.21.15.16:0/1697280660 shutdown_connections 2024-09-12T03:02:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b24d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 --2- 172.21.15.16:0/1697280660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b242880 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 -- 172.21.15.16:0/1697280660 >> 172.21.15.16:0/1697280660 conn(0x55669b23ea30 msgr2=0x55669b23ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 -- 172.21.15.16:0/1697280660 shutdown_connections 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.470+0000 7efe231c6740 1 -- 172.21.15.16:0/1697280660 wait complete. 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 -- start start 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 Processor -- start 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b242dc0 0x55669b24a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55669b250250 con 0x55669b2424a0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55669b163050 con 0x55669b24d680 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55669b2503d0 con 0x55669b242dc0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe231c6740 10 client.0 _ll_register_callbacks cb 0x55669b1a3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b242dc0 0x55669b24a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33954/0 (socket says 172.21.15.16:33954) 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe11d9b640 1 -- 172.21.15.16:0/331250962 learned_addr learned my addr 172.21.15.16:0/331250962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.471+0000 7efe11d9b640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 -- 172.21.15.16:0/331250962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b242dc0 msgr2=0x55669b24a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b242dc0 0x55669b24a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 -- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 msgr2=0x55669b24a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 -- 172.21.15.16:0/331250962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55669af193c0 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe11d9b640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004650 tx=0x7efe0c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff3fe267796833d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004ce0 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efe0c020430 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe0c005500 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efe0c035050 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe231c6740 10 client.18906 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.472+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55669b25bcb0 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe0c005500 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7efe0c00f040 con 0x55669b2424a0 2024-09-12T03:02:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 10 client.18906 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 10 client.18906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 20 client.18906 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b25f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 4 client.18906 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55669b1b5850 con 0x55669b25d000 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe1a59c640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b25f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe1a59c640 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b25f3f0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7337144ffd528d8d server_cookie=17243ac63a8d6da in_seq=0 out_seq=0 2024-09-12T03:02:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.473+0000 7efe01ffb640 10 client.18906 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.476+0000 7efe01ffb640 1 -- 172.21.15.16:0/331250962 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55669b1b5850 con 0x55669b25d000 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.476+0000 7efe01ffb640 10 client.18906 handle_command_reply: tid=0 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.476+0000 7efe231c6740 1 client.18906 shutdown 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 msgr2=0x55669b24d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004650 tx=0x7efe0c0049f0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 shutdown_connections 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b25f3f0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b24a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe01ffb640 0 client.18906 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b242dc0 0x55669b24a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 --2- 172.21.15.16:0/331250962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b2424a0 0x55669b24d070 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 >> 172.21.15.16:0/331250962 conn(0x55669b23ea30 msgr2=0x55669b244760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 shutdown_connections 2024-09-12T03:02:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 1 -- 172.21.15.16:0/331250962 wait complete. 2024-09-12T03:02:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.477+0000 7efe231c6740 20 client.18906 trim_cache size 0 max 16384 2024-09-12T03:02:41.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 Processor -- start 2024-09-12T03:02:41.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 -- start start 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b242dc0 0x55669b245900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b24d680 0x55669b264b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1bc3a0 con 0x55669b24d680 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1bdfa0 con 0x55669b242dc0 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe231c6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1bca70 con 0x55669b2424a0 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54938/0 (socket says 172.21.15.16:54938) 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe11d9b640 1 -- 172.21.15.16:0/2820443795 learned_addr learned my addr 172.21.15.16:0/2820443795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.502+0000 7efe11d9b640 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b24d680 0x55669b264b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe1a59c640 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b242dc0 0x55669b245900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 -- 172.21.15.16:0/2820443795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b242dc0 msgr2=0x55669b245900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b242dc0 0x55669b245900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 -- 172.21.15.16:0/2820443795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b24d680 msgr2=0x55669b264b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b24d680 0x55669b264b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 -- 172.21.15.16:0/2820443795 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55669b242bf0 con 0x55669b2424a0 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe11d9b640 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004620 tx=0x7efe0c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4649946ff99536 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe03fff640 1 -- 172.21.15.16:0/2820443795 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004680 con 0x55669b2424a0 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe03fff640 1 -- 172.21.15.16:0/2820443795 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe0c009e40 con 0x55669b2424a0 2024-09-12T03:02:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.503+0000 7efe03fff640 1 -- 172.21.15.16:0/2820443795 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c020430 con 0x55669b2424a0 2024-09-12T03:02:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 -- 172.21.15.16:0/2820443795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 msgr2=0x55669b2594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efe0c004620 tx=0x7efe0c009260 comp rx=0 tx=0).stop 2024-09-12T03:02:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 -- 172.21.15.16:0/2820443795 shutdown_connections 2024-09-12T03:02:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b24d680 0x55669b264b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b242dc0 0x55669b245900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 --2- 172.21.15.16:0/2820443795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b2594b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.504+0000 7efe231c6740 1 -- 172.21.15.16:0/2820443795 >> 172.21.15.16:0/2820443795 conn(0x55669b23ea30 msgr2=0x55669b23ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 -- 172.21.15.16:0/2820443795 shutdown_connections 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 -- 172.21.15.16:0/2820443795 wait complete. 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 -- start start 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 Processor -- start 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b250f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.505+0000 7efe231c6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b2514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe231c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1bd8d0 con 0x55669b242dc0 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe231c6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1bd200 con 0x55669b24d680 2024-09-12T03:02:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe231c6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1a0310 con 0x55669b2424a0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe231c6740 10 client.0 _ll_register_callbacks cb 0x7efe04009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54940/0 (socket says 172.21.15.16:54940) 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b250f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 -- 172.21.15.16:0/1976383573 learned_addr learned my addr 172.21.15.16:0/1976383573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b2514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 -- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 msgr2=0x55669b2514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b2514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 -- 172.21.15.16:0/1976383573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 msgr2=0x55669b250f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b250f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 -- 172.21.15.16:0/1976383573 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55669b251bc0 con 0x55669b2424a0 2024-09-12T03:02:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe11d9b640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009be0 tx=0x7efe0c010650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53581225a8bfcb6d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.506+0000 7efe01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c013070 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efe0c013210 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe0c009e40 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efe0c0108d0 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe231c6740 10 client.28147 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55669b2c79f0 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe0c009e40 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7efe0c0206e0 con 0x55669b2424a0 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.507+0000 7efe231c6740 10 client.28147 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:41.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe231c6740 10 client.28147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe231c6740 20 client.28147 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b2ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe231c6740 4 client.28147 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:02:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55669b1afd60 con 0x55669b25d000 2024-09-12T03:02:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.508+0000 7efe1a59c640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b2ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.509+0000 7efe1a59c640 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b2ca050 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c37f5962ad9096fe server_cookie=90ba8945ca2ce03c in_seq=0 out_seq=0 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.509+0000 7efe01ffb640 10 client.28147 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe01ffb640 1 -- 172.21.15.16:0/1976383573 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55669b1afd60 con 0x55669b25d000 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe01ffb640 10 client.28147 handle_command_reply: tid=0 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 client.28147 shutdown 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 msgr2=0x55669b255c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009be0 tx=0x7efe0c010650 comp rx=0 tx=0).stop 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 shutdown_connections 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55669b25d000 0x55669b2ca050 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55669b24d680 0x55669b2514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55669b242dc0 0x55669b250f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 --2- 172.21.15.16:0/1976383573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55669b2424a0 0x55669b255c60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 >> 172.21.15.16:0/1976383573 conn(0x55669b23ea30 msgr2=0x55669b23db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe01ffb640 0 client.28147 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 shutdown_connections 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.511+0000 7efe231c6740 1 -- 172.21.15.16:0/1976383573 wait complete. 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.512+0000 7efe231c6740 20 client.28147 trim_cache size 0 max 16384 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:02:41.516 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.512+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 msgr2=0x7efe080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.512+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 0x7efe080a5040 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efe140ed7e0 tx=0x7efe14031900 comp rx=0 tx=0).stop 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.512+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 msgr2=0x7efe1c10e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7efe14009b30 tx=0x7efe140305d0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 shutdown_connections 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efe080a2b80 0x7efe080a5040 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe1c109120 0x7efe1c10e100 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe1c1049a0 0x7efe1c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 --2- 172.21.15.16:0/1703196586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c103ff0 0x7efe1c110b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 >> 172.21.15.16:0/1703196586 conn(0x7efe1c0ffab0 msgr2=0x7efe1c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.513+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 shutdown_connections 2024-09-12T03:02:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:41.514+0000 7efe20ded640 1 -- 172.21.15.16:0/1703196586 wait complete. 2024-09-12T03:02:42.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 28840 16.78 MB/sec warmup 103 sec latency 71.275 ms 2024-09-12T03:02:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:41 smithi184 ceph-mon[35321]: pgmap v780: 97 pgs: 97 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.0 MiB/s wr, 626 op/s 2024-09-12T03:02:42.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:41 smithi016 ceph-mon[30925]: pgmap v780: 97 pgs: 97 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.0 MiB/s wr, 626 op/s 2024-09-12T03:02:42.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:41 smithi033 ceph-mon[35092]: pgmap v780: 97 pgs: 97 active+clean; 516 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.0 MiB/s wr, 626 op/s 2024-09-12T03:02:42.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:02:42.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:02:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:43.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 29712 17.01 MB/sec warmup 104 sec latency 57.536 ms 2024-09-12T03:02:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:02:43.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"btime":"2024-09-12T03:01:44:300565+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:01:43.958785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 -- 172.21.15.16:0/155916954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 msgr2=0x7f80e410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 --2- 172.21.15.16:0/155916954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e410ec40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009cd0 tx=0x7f80dc02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 -- 172.21.15.16:0/155916954 shutdown_connections 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 --2- 172.21.15.16:0/155916954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e410ec40 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 --2- 172.21.15.16:0/155916954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 0x7f80e4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 --2- 172.21.15.16:0/155916954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 -- 172.21.15.16:0/155916954 >> 172.21.15.16:0/155916954 conn(0x7f80e40fc7d0 msgr2=0x7f80e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 -- 172.21.15.16:0/155916954 shutdown_connections 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.342+0000 7f80eb490640 1 -- 172.21.15.16:0/155916954 wait complete. 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 Processor -- start 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 -- start start 2024-09-12T03:02:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 0x7f80e4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41120f0 con 0x7f80e41082a0 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4111f70 con 0x7f80e4102d30 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80eb490640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4112270 con 0x7f80e41036e0 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80e9a2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80e922c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80e9a2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33974/0 (socket says 172.21.15.16:33974) 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80e9a2d640 1 -- 172.21.15.16:0/1041124908 learned_addr learned my addr 172.21.15.16:0/1041124908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.343+0000 7f80e8a2b640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 0x7f80e4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 -- 172.21.15.16:0/1041124908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 msgr2=0x7f80e4100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 0x7f80e4100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 -- 172.21.15.16:0/1041124908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 msgr2=0x7f80e4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 -- 172.21.15.16:0/1041124908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e41a6af0 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e922c640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80e9a2d640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f80dc033040 tx=0x7f80dc031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc004040 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0046c0 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e41a6d80 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc030ed0 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.344+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e41a72e0 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.345+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e4103110 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.346+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f80dc0041e0 con 0x7f80e41082a0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.347+0000 7f80da7fc640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 0x7f80d00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.347+0000 7f80e922c640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 0x7f80d00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.348+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f80dc02e030 con 0x7f80e41082a0 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.348+0000 7f80e922c640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 0x7f80d00a50c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f80e0011490 tx=0x7f80e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.351+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80dc04b060 con 0x7f80e41082a0 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.609+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80e410a2b0 con 0x7f80e41082a0 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.610+0000 7f80da7fc640 1 -- 172.21.15.16:0/1041124908 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11983 (secure 0 0 0) 0x7f80dc0bd4f0 con 0x7f80e41082a0 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 msgr2=0x7f80d00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 0x7f80d00a50c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f80e0011490 tx=0x7f80e0005f90 comp rx=0 tx=0).stop 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 msgr2=0x7f80e4101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f80dc033040 tx=0x7f80dc031e60 comp rx=0 tx=0).stop 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 shutdown_connections 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f80d00a2c00 0x7f80d00a50c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e41082a0 0x7f80e4101410 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80e41036e0 0x7f80e4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 --2- 172.21.15.16:0/1041124908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f80e4102d30 0x7f80e4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.612+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 >> 172.21.15.16:0/1041124908 conn(0x7f80e40fc7d0 msgr2=0x7f80e40fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.613+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 shutdown_connections 2024-09-12T03:02:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:43.613+0000 7f80eb490640 1 -- 172.21.15.16:0/1041124908 wait complete. 2024-09-12T03:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:43 smithi016 ceph-mon[30925]: pgmap v781: 97 pgs: 97 active+clean; 513 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-09-12T03:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:43 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1041124908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:44.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 30395 17.28 MB/sec warmup 105 sec latency 95.173 ms 2024-09-12T03:02:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:43 smithi184 ceph-mon[35321]: pgmap v781: 97 pgs: 97 active+clean; 513 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-09-12T03:02:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:43 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1041124908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:43 smithi033 ceph-mon[35092]: pgmap v781: 97 pgs: 97 active+clean; 513 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-09-12T03:02:44.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1041124908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:02:44.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:02:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:45.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 31112 17.48 MB/sec warmup 106 sec latency 58.539 ms 2024-09-12T03:02:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 -- 172.21.15.16:0/375383013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca4102360 msgr2=0x7feca410e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 --2- 172.21.15.16:0/375383013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca4102360 0x7feca410e890 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fec94009cd0 tx=0x7fec9402a8e0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 -- 172.21.15.16:0/375383013 shutdown_connections 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 --2- 172.21.15.16:0/375383013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca4102360 0x7feca410e890 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 --2- 172.21.15.16:0/375383013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca41019c0 0x7feca4101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 --2- 172.21.15.16:0/375383013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 0x7feca4107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 -- 172.21.15.16:0/375383013 >> 172.21.15.16:0/375383013 conn(0x7feca40fd740 msgr2=0x7feca40ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 -- 172.21.15.16:0/375383013 shutdown_connections 2024-09-12T03:02:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.155+0000 7feca9283640 1 -- 172.21.15.16:0/375383013 wait complete. 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.156+0000 7feca9283640 1 Processor -- start 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.156+0000 7feca9283640 1 -- start start 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.156+0000 7feca9283640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca9283640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca4102360 0x7feca419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca9283640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 0x7feca419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca9283640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111de0 con 0x7feca41019c0 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca9283640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111c60 con 0x7feca41079c0 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca9283640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111f60 con 0x7feca4102360 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33990/0 (socket says 172.21.15.16:33990) 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca4102360 0x7feca419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 -- 172.21.15.16:0/4189258302 learned_addr learned my addr 172.21.15.16:0/4189258302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca8a82640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 0x7feca419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 -- 172.21.15.16:0/4189258302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca4102360 msgr2=0x7feca419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca4102360 0x7feca419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 -- 172.21.15.16:0/4189258302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 msgr2=0x7feca419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 0x7feca419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 -- 172.21.15.16:0/4189258302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca419f400 con 0x7feca41019c0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca3fff640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00c920 tx=0x7fec9c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca17fa640 1 -- 172.21.15.16:0/4189258302 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c007bb0 con 0x7feca41019c0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.157+0000 7feca17fa640 1 -- 172.21.15.16:0/4189258302 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9c007d50 con 0x7feca41019c0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.158+0000 7feca17fa640 1 -- 172.21.15.16:0/4189258302 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0187d0 con 0x7feca41019c0 2024-09-12T03:02:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.158+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca419f6f0 con 0x7feca41019c0 2024-09-12T03:02:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.158+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41a6fd0 con 0x7feca41019c0 2024-09-12T03:02:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.160+0000 7feca17fa640 1 -- 172.21.15.16:0/4189258302 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fec9c00f450 con 0x7feca41019c0 2024-09-12T03:02:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.160+0000 7fecac8be740 1 Processor -- start 2024-09-12T03:02:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.160+0000 7fecac8be740 1 -- start start 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4aca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4acabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556db4a302f0 con 0x556db4aca7b0 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556db49d7cf0 con 0x556db4acbd20 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7fecac8be740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556db4a2c2b0 con 0x556db4ad5550 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4acabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7feca37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7feca37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54996/0 (socket says 172.21.15.16:54996) 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7feca37fe640 1 -- 172.21.15.16:0/1429772955 learned_addr learned my addr 172.21.15.16:0/1429772955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.161+0000 7feca8a82640 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4aca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 -- 172.21.15.16:0/1429772955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 msgr2=0x556db4aca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4aca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 -- 172.21.15.16:0/1429772955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 msgr2=0x556db4acabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4acabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 -- 172.21.15.16:0/1429772955 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db49bc3a0 con 0x556db4ad5550 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca17fa640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 0x7fec8c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca37fe640 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec98009a40 tx=0x7fec98009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca0653c3a2d533 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7feca17fa640 1 -- 172.21.15.16:0/4189258302 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec9c0cf030 con 0x7feca41019c0 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7fec8a7fc640 1 -- 172.21.15.16:0/1429772955 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98017070 con 0x556db4ad5550 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7fec8a7fc640 1 -- 172.21.15.16:0/1429772955 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec98013470 con 0x556db4ad5550 2024-09-12T03:02:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.162+0000 7fec8a7fc640 1 -- 172.21.15.16:0/1429772955 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98012630 con 0x556db4ad5550 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7feca8a82640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 0x7fec8c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 -- 172.21.15.16:0/1429772955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 msgr2=0x556db4ad7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec98009a40 tx=0x7fec98009f00 comp rx=0 tx=0).stop 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 -- 172.21.15.16:0/1429772955 shutdown_connections 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4ad7940 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4acabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 --2- 172.21.15.16:0/1429772955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4aca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.163+0000 7fecac8be740 1 -- 172.21.15.16:0/1429772955 >> 172.21.15.16:0/1429772955 conn(0x556db4a93310 msgr2=0x556db4ac6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.164+0000 7feca8a82640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 0x7fec8c0a50c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec940098d0 tx=0x7fec94009960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.164+0000 7fecac8be740 1 -- 172.21.15.16:0/1429772955 shutdown_connections 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.164+0000 7fecac8be740 1 -- 172.21.15.16:0/1429772955 wait complete. 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.165+0000 7fecac8be740 1 -- start start 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.165+0000 7fecac8be740 1 Processor -- start 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4aca7b0 0x556db4ad4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 0x556db4ad2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556db4ad8250 con 0x556db4ad5550 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4aca7b0 0x556db4ad4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca8a82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 0x556db4ad2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca8a82640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 0x556db4ad2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34000/0 (socket says 172.21.15.16:34000) 2024-09-12T03:02:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca8a82640 1 -- 172.21.15.16:0/3758193535 learned_addr learned my addr 172.21.15.16:0/3758193535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7feca37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43780/0 (socket says 172.21.15.16:43780) 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556db49eb060 con 0x556db4acbd20 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.166+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556db4ad83d0 con 0x556db4aca7b0 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 -- 172.21.15.16:0/3758193535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4aca7b0 msgr2=0x556db4ad4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4aca7b0 0x556db4ad4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 -- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 msgr2=0x556db4ad2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 0x556db4ad2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db47a13c0 con 0x556db4acbd20 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7fecac8be740 10 client.0 _ll_register_callbacks cb 0x556db473e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec98013470 con 0x556db4acbd20 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7fec837fe640 10 client.28247 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7feca37fe640 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fec98009a10 tx=0x7fec98006eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e3ceca56ab78632 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7fec837fe640 1 -- 172.21.15.16:0/3758193535 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98017050 con 0x556db4acbd20 2024-09-12T03:02:45.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.167+0000 7fec837fe640 1 -- 172.21.15.16:0/3758193535 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec9801c450 con 0x556db4acbd20 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fec837fe640 1 -- 172.21.15.16:0/3758193535 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec9801c6b0 con 0x556db4acbd20 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 10 client.28247 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=35}) -- 0x556db4ae3b40 con 0x556db4acbd20 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fec837fe640 1 -- 172.21.15.16:0/3758193535 <== mon.2 v2:172.21.15.33:3300/0 4 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fec9801c850 con 0x556db4acbd20 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 10 client.28247 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 10 client.28247 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 20 client.28247 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4ae4e90 0x556db4ae7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 4 client.28247 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556db4a3d860 con 0x556db4ae4e90 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.168+0000 7feca3fff640 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4ae4e90 0x556db4ae7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fec837fe640 10 client.28247 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:02:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7feca3fff640 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4ae4e90 0x556db4ae7280 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=54e64ba36347e572 server_cookie=9199394b7e9e9337 in_seq=0 out_seq=0 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fec837fe640 1 -- 172.21.15.16:0/3758193535 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556db4a3d860 con 0x556db4ae4e90 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fec837fe640 10 client.28247 handle_command_reply: tid=0 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fecac8be740 1 client.28247 shutdown 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 msgr2=0x556db4ad21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.169+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fec98009a10 tx=0x7fec98006eb0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 shutdown_connections 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4ad5550 0x556db4ad2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4ae4e90 0x556db4ae7280 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad21a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 --2- 172.21.15.16:0/3758193535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4aca7b0 0x556db4ad4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fec837fe640 0 client.28247 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:02:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 >> 172.21.15.16:0/3758193535 conn(0x556db4a93310 msgr2=0x556db4accb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 shutdown_connections 2024-09-12T03:02:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 1 -- 172.21.15.16:0/3758193535 wait complete. 2024-09-12T03:02:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.170+0000 7fecac8be740 20 client.28247 trim_cache size 0 max 16384 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.186+0000 7fecac8be740 1 Processor -- start 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.186+0000 7fecac8be740 1 -- start start 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4ae14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feca41af4d0 con 0x556db4aca7b0 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca41b10d0 con 0x556db4acbd20 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7fecac8be740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feca41afba0 con 0x556db4ad5550 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4ae14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:55018/0 (socket says 172.21.15.16:55018) 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 -- 172.21.15.16:0/3855336014 learned_addr learned my addr 172.21.15.16:0/3855336014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca37fe640 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 -- 172.21.15.16:0/3855336014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 msgr2=0x556db4ad1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 -- 172.21.15.16:0/3855336014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 msgr2=0x556db4ae14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4ae14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.187+0000 7feca8a82640 1 -- 172.21.15.16:0/3855336014 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db4ad2ab0 con 0x556db4ad5550 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.188+0000 7feca8a82640 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec94031be0 tx=0x7fec94040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cafcc4f605f7536 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.188+0000 7fec89ffb640 1 -- 172.21.15.16:0/3855336014 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94056070 con 0x556db4ad5550 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.188+0000 7fec89ffb640 1 -- 172.21.15.16:0/3855336014 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec94036040 con 0x556db4ad5550 2024-09-12T03:02:45.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.188+0000 7fec89ffb640 1 -- 172.21.15.16:0/3855336014 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec940307a0 con 0x556db4ad5550 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- 172.21.15.16:0/3855336014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 msgr2=0x556db4aec8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec94031be0 tx=0x7fec94040000 comp rx=0 tx=0).stop 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- 172.21.15.16:0/3855336014 shutdown_connections 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4aec8e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4ad1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 --2- 172.21.15.16:0/3855336014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4ae14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- 172.21.15.16:0/3855336014 >> 172.21.15.16:0/3855336014 conn(0x556db4a93310 msgr2=0x556db4ad0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- 172.21.15.16:0/3855336014 shutdown_connections 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- 172.21.15.16:0/3855336014 wait complete. 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.189+0000 7fecac8be740 1 -- start start 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.190+0000 7fecac8be740 1 Processor -- start 2024-09-12T03:02:45.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.190+0000 7fecac8be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.190+0000 7fecac8be740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4acefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.190+0000 7fecac8be740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4acf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4acefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4acefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43810/0 (socket says 172.21.15.16:43810) 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34016/0 (socket says 172.21.15.16:34016) 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca8a82640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4acf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca37fe640 1 -- 172.21.15.16:0/2112571065 learned_addr learned my addr 172.21.15.16:0/2112571065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feca41b0a00 con 0x556db4aca7b0 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feca41b0330 con 0x556db4acbd20 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feca41ad4b0 con 0x556db4ad5550 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7fecac8be740 10 client.0 _ll_register_callbacks cb 0x7fec9c0d29e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:45.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 -- 172.21.15.16:0/2112571065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 msgr2=0x556db4acf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4acf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 -- 172.21.15.16:0/2112571065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 msgr2=0x556db4acefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4acefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.191+0000 7feca3fff640 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db4acfda0 con 0x556db4aca7b0 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7feca3fff640 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0ccb90 tx=0x7fec9c0cca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7eef3c2d1d1b72b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:45.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7fec837fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:45.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c09bfa0 con 0x556db4aca7b0 2024-09-12T03:02:45.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec94036040 con 0x556db4aca7b0 2024-09-12T03:02:45.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fec9c00f740 con 0x556db4aca7b0 2024-09-12T03:02:45.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.192+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec9c09b8a0 con 0x556db4aca7b0 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 10 client.18936 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556db4b4f9a0 con 0x556db4aca7b0 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec94036040 con 0x556db4aca7b0 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fec9c097f10 con 0x556db4aca7b0 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 10 client.18936 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 10 client.18936 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 20 client.18936 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4b50070 0x556db4b52460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 4 client.18936 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.193+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556db4a37d70 con 0x556db4b50070 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.194+0000 7feca37fe640 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4b50070 0x556db4b52460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.194+0000 7fec837fe640 10 client.18936 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.194+0000 7feca37fe640 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4b50070 0x556db4b52460 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=48d7d197fea4af41 server_cookie=5676adccb93f838c in_seq=0 out_seq=0 2024-09-12T03:02:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fec837fe640 1 -- 172.21.15.16:0/2112571065 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556db4a37d70 con 0x556db4b50070 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fec837fe640 10 client.18936 handle_command_reply: tid=0 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 client.18936 shutdown 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 msgr2=0x556db4added0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0ccb90 tx=0x7fec9c0cca10 comp rx=0 tx=0).stop 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 shutdown_connections 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556db4ad5550 0x556db4acf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x556db4b50070 0x556db4b52460 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556db4acbd20 0x556db4acefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 --2- 172.21.15.16:0/2112571065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556db4aca7b0 0x556db4added0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 >> 172.21.15.16:0/2112571065 conn(0x556db4a93310 msgr2=0x556db4ac5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fec837fe640 0 client.18936 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:02:45.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 shutdown_connections 2024-09-12T03:02:45.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 1 -- 172.21.15.16:0/2112571065 wait complete. 2024-09-12T03:02:45.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.195+0000 7fecac8be740 20 client.18936 trim_cache size 0 max 16384 2024-09-12T03:02:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 msgr2=0x7fec8c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 0x7fec8c0a50c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec940098d0 tx=0x7fec94009960 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 msgr2=0x7feca419a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00c920 tx=0x7fec9c00cd20 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 shutdown_connections 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feca41079c0 0x7feca419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feca4102360 0x7feca419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fec8c0a2c00 0x7fec8c0a50c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 --2- 172.21.15.16:0/4189258302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca41019c0 0x7feca419a450 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 >> 172.21.15.16:0/4189258302 conn(0x7feca40fd740 msgr2=0x7feca410d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 shutdown_connections 2024-09-12T03:02:45.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:45.196+0000 7feca9283640 1 -- 172.21.15.16:0/4189258302 wait complete. 2024-09-12T03:02:45.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:45 smithi016 ceph-mon[30925]: pgmap v782: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.2 MiB/s wr, 470 op/s 2024-09-12T03:02:46.008 INFO:tasks.workunit.client.0.smithi016.stdout: 10 31876 17.66 MB/sec warmup 107 sec latency 48.008 ms 2024-09-12T03:02:46.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:02:46.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:45 smithi184 ceph-mon[35321]: pgmap v782: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.2 MiB/s wr, 470 op/s 2024-09-12T03:02:46.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:45 smithi033 ceph-mon[35092]: pgmap v782: 97 pgs: 97 active+clean; 515 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.2 MiB/s wr, 470 op/s 2024-09-12T03:02:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:46.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 -- 172.21.15.16:0/4112603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 msgr2=0x7fdedc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:46.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 --2- 172.21.15.16:0/4112603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc101200 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009900 tx=0x7fdecc02a480 comp rx=0 tx=0).stop 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 -- 172.21.15.16:0/4112603 shutdown_connections 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 --2- 172.21.15.16:0/4112603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 --2- 172.21.15.16:0/4112603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 0x7fdedc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 --2- 172.21.15.16:0/4112603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc101200 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 -- 172.21.15.16:0/4112603 >> 172.21.15.16:0/4112603 conn(0x7fdedc0fc7d0 msgr2=0x7fdedc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 -- 172.21.15.16:0/4112603 shutdown_connections 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.904+0000 7fdee13f3640 1 -- 172.21.15.16:0/4112603 wait complete. 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.905+0000 7fdee13f3640 1 Processor -- start 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.905+0000 7fdee13f3640 1 -- start start 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.905+0000 7fdee13f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.905+0000 7fdee13f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 0x7fdedc19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.905+0000 7fdee13f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdee13f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc110f50 con 0x7fdedc100e20 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdee13f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc110dd0 con 0x7fdedc102570 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdee13f3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1110d0 con 0x7fdedc102ef0 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43330/0 (socket says 172.21.15.16:43330) 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedaffd640 1 -- 172.21.15.16:0/2536700742 learned_addr learned my addr 172.21.15.16:0/2536700742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 0x7fdedc19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 -- 172.21.15.16:0/2536700742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 msgr2=0x7fdedc19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 0x7fdedc19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 -- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 msgr2=0x7fdedc19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:46.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 -- 172.21.15.16:0/2536700742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc19f6f0 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.906+0000 7fdedb7fe640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc19f010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fded4009fd0 tx=0x7fded400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.907+0000 7fdec3fff640 1 -- 172.21.15.16:0/2536700742 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded400ce20 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.907+0000 7fdec3fff640 1 -- 172.21.15.16:0/2536700742 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fded4005020 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.907+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc19f9e0 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.907+0000 7fdec3fff640 1 -- 172.21.15.16:0/2536700742 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4010720 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.907+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc1a72c0 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.910+0000 7fdee37cc740 1 Processor -- start 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.912+0000 7fdec3fff640 1 -- 172.21.15.16:0/2536700742 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fded4010900 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.913+0000 7fdec3fff640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 0x7fdec40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.914+0000 7fdec3fff640 1 -- 172.21.15.16:0/2536700742 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fded40cf740 con 0x7fdedc102ef0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.914+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 0x7fdec40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.915+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 0x7fdec40a5040 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdedc10ddf0 tx=0x7fded0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 -- start start 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5ee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5eea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5fb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558aed5542f0 con 0x558aed5efb40 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558aed4fbcf0 con 0x558aed5f9370 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.916+0000 7fdee37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558aed5502b0 con 0x558aed5ee5d0 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.917+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.917+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43338/0 (socket says 172.21.15.16:43338) 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.917+0000 7fdedaffd640 1 -- 172.21.15.16:0/1363659172 learned_addr learned my addr 172.21.15.16:0/1363659172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.917+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdedb7fe640 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5fb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdeda7fc640 1 -- 172.21.15.16:0/1363659172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 msgr2=0x558aed5fb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5fb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdeda7fc640 1 -- 172.21.15.16:0/1363659172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 msgr2=0x558aed5ee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5ee090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.918+0000 7fdeda7fc640 1 -- 172.21.15.16:0/1363659172 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aed4e03a0 con 0x558aed5ee5d0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.919+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5eea10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fded00118d0 tx=0x7fded0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cccaaa46e7baf014 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.920+0000 7fdec17fa640 1 -- 172.21.15.16:0/1363659172 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0004030 con 0x558aed5ee5d0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.920+0000 7fdec17fa640 1 -- 172.21.15.16:0/1363659172 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fded00041d0 con 0x558aed5ee5d0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.921+0000 7fdec17fa640 1 -- 172.21.15.16:0/1363659172 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded002e700 con 0x558aed5ee5d0 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.921+0000 7fdee37cc740 1 -- 172.21.15.16:0/1363659172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 msgr2=0x558aed5eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:46.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.921+0000 7fdee37cc740 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5eea10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fded00118d0 tx=0x7fded0014000 comp rx=0 tx=0).stop 2024-09-12T03:02:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.937+0000 7fdee37cc740 1 -- 172.21.15.16:0/1363659172 shutdown_connections 2024-09-12T03:02:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.937+0000 7fdee37cc740 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5fb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.937+0000 7fdee37cc740 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5eea10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.937+0000 7fdee37cc740 1 --2- 172.21.15.16:0/1363659172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5ee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.937+0000 7fdee37cc740 1 -- 172.21.15.16:0/1363659172 >> 172.21.15.16:0/1363659172 conn(0x558aed5b7310 msgr2=0x558aed5eac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:46.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.944+0000 7fdee37cc740 1 -- 172.21.15.16:0/1363659172 shutdown_connections 2024-09-12T03:02:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.946+0000 7fdee37cc740 1 -- 172.21.15.16:0/1363659172 wait complete. 2024-09-12T03:02:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.947+0000 7fdee37cc740 1 -- start start 2024-09-12T03:02:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.947+0000 7fdee37cc740 1 Processor -- start 2024-09-12T03:02:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.948+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5f8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.948+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.948+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdee37cc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558aed5fc0c0 con 0x558aed5efb40 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdee37cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558aed50f060 con 0x558aed5f9370 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5f8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdedb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdedb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5f64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46888/0 (socket says 172.21.15.16:46888) 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdedb7fe640 1 -- 172.21.15.16:0/3627513071 learned_addr learned my addr 172.21.15.16:0/3627513071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558aed5fc240 con 0x558aed5ee5d0 2024-09-12T03:02:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.949+0000 7fdee37cc740 10 client.0 _ll_register_callbacks cb 0x558aed2628c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:46.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdedaffd640 1 -- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 msgr2=0x558aed5f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:46.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdedaffd640 1 -- 172.21.15.16:0/3627513071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 msgr2=0x558aed5f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.951+0000 7fdedaffd640 1 -- 172.21.15.16:0/3627513071 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aed2c53c0 con 0x558aed5ee5d0 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.952+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5f8ce0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009230 tx=0x7fdecc0042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5318148b57a8127 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.952+0000 7fdebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.952+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc035040 con 0x558aed5ee5d0 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.952+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdecc0037f0 con 0x558aed5ee5d0 2024-09-12T03:02:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.952+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded00041d0 con 0x558aed5ee5d0 2024-09-12T03:02:46.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.953+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdecc0044d0 con 0x558aed5ee5d0 2024-09-12T03:02:46.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.954+0000 7fdee37cc740 10 client.28180 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:46.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.954+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558aed607b20 con 0x558aed5ee5d0 2024-09-12T03:02:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.956+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdecc031540 con 0x558aed5ee5d0 2024-09-12T03:02:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fdecc0464b0 con 0x558aed5ee5d0 2024-09-12T03:02:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 10 client.28180 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 10 client.28180 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:02:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 20 client.28180 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed608fe0 0x558aed60b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 4 client.28180 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.957+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fded00041d0 con 0x558aed608fe0 2024-09-12T03:02:46.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.959+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed608fe0 0x558aed60b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.965+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed608fe0 0x558aed60b3d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d947486a48acdda6 server_cookie=d25413a0d25c77c6 in_seq=0 out_seq=0 2024-09-12T03:02:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.965+0000 7fdebeff5640 10 client.28180 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.966+0000 7fdebeff5640 1 -- 172.21.15.16:0/3627513071 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fded00041d0 con 0x558aed608fe0 2024-09-12T03:02:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.966+0000 7fdebeff5640 10 client.28180 handle_command_reply: tid=0 2024-09-12T03:02:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.966+0000 7fdee37cc740 1 client.28180 shutdown 2024-09-12T03:02:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.967+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 msgr2=0x558aed5f8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.967+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5f8ce0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009230 tx=0x7fdecc0042e0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 shutdown_connections 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed608fe0 0x558aed60b3d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5f9370 0x558aed5f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5efb40 0x558aed5f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3627513071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed5f8ce0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 >> 172.21.15.16:0/3627513071 conn(0x558aed5b7310 msgr2=0x558aed5f0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.969+0000 7fdebeff5640 0 client.28180 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:46.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.970+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 shutdown_connections 2024-09-12T03:02:46.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.971+0000 7fdee37cc740 1 -- 172.21.15.16:0/3627513071 wait complete. 2024-09-12T03:02:46.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:46.975+0000 7fdee37cc740 20 client.28180 trim_cache size 0 max 16384 2024-09-12T03:02:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.001+0000 7fdee37cc740 1 Processor -- start 2024-09-12T03:02:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.002+0000 7fdee37cc740 1 -- start start 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.002+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 0x558aed5f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed6053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 0x558aed610cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdedb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 0x558aed610cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdedb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 0x558aed610cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46898/0 (socket says 172.21.15.16:46898) 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdedb7fe640 1 -- 172.21.15.16:0/3104337100 learned_addr learned my addr 172.21.15.16:0/3104337100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.003+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1ab5b0 con 0x558aed5b7310 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.004+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1ad1b0 con 0x558aed5efb40 2024-09-12T03:02:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.004+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1abc80 con 0x558aed5ee5d0 2024-09-12T03:02:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.005+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed6053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:47.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 32428 17.83 MB/sec warmup 108 sec latency 103.790 ms 2024-09-12T03:02:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.005+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 0x558aed5f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.005+0000 7fdedaffd640 1 -- 172.21.15.16:0/3104337100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 msgr2=0x558aed6053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed6053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdedaffd640 1 -- 172.21.15.16:0/3104337100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 msgr2=0x558aed610cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 0x558aed610cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdedaffd640 1 -- 172.21.15.16:0/3104337100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aed5f9050 con 0x558aed5b7310 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdedaffd640 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 0x558aed5f5f60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fdecc005ab0 tx=0x7fdecc005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f994be17425abee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdec0ff9640 1 -- 172.21.15.16:0/3104337100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc046470 con 0x558aed5b7310 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdec0ff9640 1 -- 172.21.15.16:0/3104337100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdecc046610 con 0x558aed5b7310 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.006+0000 7fdec0ff9640 1 -- 172.21.15.16:0/3104337100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc004920 con 0x558aed5b7310 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.007+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 msgr2=0x558aed5f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.007+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 0x558aed5f5f60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fdecc005ab0 tx=0x7fdecc005e20 comp rx=0 tx=0).stop 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.009+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 shutdown_connections 2024-09-12T03:02:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.009+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5efb40 0x558aed610cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.009+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5ee5d0 0x558aed6053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.009+0000 7fdee37cc740 1 --2- 172.21.15.16:0/3104337100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5b7310 0x558aed5f5f60 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.009+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 >> 172.21.15.16:0/3104337100 conn(0x558aed5f19a0 msgr2=0x558aed5f5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.010+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 shutdown_connections 2024-09-12T03:02:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.010+0000 7fdee37cc740 1 -- 172.21.15.16:0/3104337100 wait complete. 2024-09-12T03:02:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.011+0000 7fdee37cc740 1 -- start start 2024-09-12T03:02:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.011+0000 7fdee37cc740 1 Processor -- start 2024-09-12T03:02:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.012+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5b7310 0x558aed66f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.012+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.013+0000 7fdee37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5efb40 0x558aed601a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.013+0000 7fdee37cc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1acae0 con 0x558aed5ee5d0 2024-09-12T03:02:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.013+0000 7fdee37cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1ac410 con 0x558aed5b7310 2024-09-12T03:02:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.013+0000 7fdee37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1a95b0 con 0x558aed5efb40 2024-09-12T03:02:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.014+0000 7fdee37cc740 10 client.0 _ll_register_callbacks cb 0x7fdecc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.014+0000 7fdeda7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.014+0000 7fdeda7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43382/0 (socket says 172.21.15.16:43382) 2024-09-12T03:02:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.014+0000 7fdeda7fc640 1 -- 172.21.15.16:0/2440152253 learned_addr learned my addr 172.21.15.16:0/2440152253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.015+0000 7fdeda7fc640 1 -- 172.21.15.16:0/2440152253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5efb40 msgr2=0x558aed601a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:02:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.015+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5efb40 0x558aed601a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.015+0000 7fdeda7fc640 1 -- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5b7310 msgr2=0x558aed66f610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:02:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.015+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5b7310 0x558aed66f610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.016+0000 7fdeda7fc640 1 -- 172.21.15.16:0/2440152253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aed5f2e90 con 0x558aed5ee5d0 2024-09-12T03:02:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.017+0000 7fdeda7fc640 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fded0000c00 tx=0x7fded00119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea83f943d2b75ab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.017+0000 7fdebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.018+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded000ec80 con 0x558aed5ee5d0 2024-09-12T03:02:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.018+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fded0009040 con 0x558aed5ee5d0 2024-09-12T03:02:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.018+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdecc046610 con 0x558aed5ee5d0 2024-09-12T03:02:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.018+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fded000ee20 con 0x558aed5ee5d0 2024-09-12T03:02:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.021+0000 7fdee37cc740 10 client.18954 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.021+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558aed684f10 con 0x558aed5ee5d0 2024-09-12T03:02:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fded00035f0 con 0x558aed5ee5d0 2024-09-12T03:02:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fded0044030 con 0x558aed5ee5d0 2024-09-12T03:02:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 10 client.18954 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 10 client.18954 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:02:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 20 client.18954 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:47.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed5f9370 0x558aed6874a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:47.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 4 client.18954 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:47.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.022+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fdecc046610 con 0x558aed5f9370 2024-09-12T03:02:47.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.023+0000 7fdedaffd640 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed5f9370 0x558aed6874a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.026+0000 7fdedaffd640 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed5f9370 0x558aed6874a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83b3df659d51d67a server_cookie=739200731e70b908 in_seq=0 out_seq=0 2024-09-12T03:02:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.026+0000 7fdebeff5640 10 client.18954 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.030+0000 7fdebeff5640 1 -- 172.21.15.16:0/2440152253 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fdecc046610 con 0x558aed5f9370 2024-09-12T03:02:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.030+0000 7fdebeff5640 10 client.18954 handle_command_reply: tid=0 2024-09-12T03:02:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.030+0000 7fdee37cc740 1 client.18954 shutdown 2024-09-12T03:02:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.031+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 msgr2=0x558aed601520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.031+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fded0000c00 tx=0x7fded00119f0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.031+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 shutdown_connections 2024-09-12T03:02:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.031+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x558aed5f9370 0x558aed6874a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558aed5efb40 0x558aed601a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558aed5ee5d0 0x558aed601520 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 --2- 172.21.15.16:0/2440152253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558aed5b7310 0x558aed66f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 >> 172.21.15.16:0/2440152253 conn(0x558aed5f19a0 msgr2=0x558aed5f0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdebeff5640 0 client.18954 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:02:47.034 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 shutdown_connections 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.032+0000 7fdee37cc740 1 -- 172.21.15.16:0/2440152253 wait complete. 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.033+0000 7fdee37cc740 20 client.18954 trim_cache size 0 max 16384 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 msgr2=0x7fdec40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 0x7fdec40a5040 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdedc10ddf0 tx=0x7fded0005f90 comp rx=0 tx=0).stop 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 msgr2=0x7fdedc19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc19f010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fded4009fd0 tx=0x7fded400ef10 comp rx=0 tx=0).stop 2024-09-12T03:02:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 shutdown_connections 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc102ef0 0x7fdedc19f010 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdedc102570 0x7fdedc19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdec40a2b80 0x7fdec40a5040 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 --2- 172.21.15.16:0/2536700742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdedc100e20 0x7fdedc19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.034+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 >> 172.21.15.16:0/2536700742 conn(0x7fdedc0fc7d0 msgr2=0x7fdedc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.035+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 shutdown_connections 2024-09-12T03:02:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:47.035+0000 7fdee13f3640 1 -- 172.21.15.16:0/2536700742 wait complete. 2024-09-12T03:02:47.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:02:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:47 smithi016 ceph-mon[30925]: pgmap v783: 97 pgs: 97 active+clean; 523 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-09-12T03:02:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 32884 17.87 MB/sec warmup 109 sec latency 245.769 ms 2024-09-12T03:02:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:48.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:47 smithi033 ceph-mon[35092]: pgmap v783: 97 pgs: 97 active+clean; 523 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-09-12T03:02:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:47 smithi184 ceph-mon[35321]: pgmap v783: 97 pgs: 97 active+clean; 523 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 -- 172.21.15.16:0/490482880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8100e20 msgr2=0x7fcfb8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/490482880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8100e20 0x7fcfb8101200 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009900 tx=0x7fcfac02a480 comp rx=0 tx=0).stop 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 -- 172.21.15.16:0/490482880 shutdown_connections 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/490482880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 0x7fcfb810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/490482880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfb8102570 0x7fcfb81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/490482880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8100e20 0x7fcfb8101200 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.688+0000 7fcfbf09e640 1 -- 172.21.15.16:0/490482880 >> 172.21.15.16:0/490482880 conn(0x7fcfb80fc7d0 msgr2=0x7fcfb80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.689+0000 7fcfbf09e640 1 -- 172.21.15.16:0/490482880 shutdown_connections 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.689+0000 7fcfbf09e640 1 -- 172.21.15.16:0/490482880 wait complete. 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.690+0000 7fcfbf09e640 1 Processor -- start 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.690+0000 7fcfbf09e640 1 -- start start 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfb8100e20 0x7fcfb8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb806a300 con 0x7fcfb8102570 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43404/0 (socket says 172.21.15.16:43404) 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3904001798 learned_addr learned my addr 172.21.15.16:0/3904001798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:48.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb806a180 con 0x7fcfb8100e20 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb806a480 con 0x7fcfb8102ef0 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3904001798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 msgr2=0x7fcfb819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3904001798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfb8100e20 msgr2=0x7fcfb8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfb8100e20 0x7fcfb8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.691+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3904001798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb819d440 con 0x7fcfb8102570 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9f0 tx=0x7fcfa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfb5ffb640 1 -- 172.21.15.16:0/3904001798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400cd80 con 0x7fcfb8102570 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfb5ffb640 1 -- 172.21.15.16:0/3904001798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4010910 con 0x7fcfb8102570 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb819d730 con 0x7fcfb8102570 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfb5ffb640 1 -- 172.21.15.16:0/3904001798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4018990 con 0x7fcfb8102570 2024-09-12T03:02:48.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.692+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb81a5010 con 0x7fcfb8102570 2024-09-12T03:02:48.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.696+0000 7fcfb5ffb640 1 -- 172.21.15.16:0/3904001798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcfa4010ac0 con 0x7fcfb8102570 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.696+0000 7fcfc1477740 1 Processor -- start 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.696+0000 7fcfb5ffb640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 0x7fcfa80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 -- start start 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 0x7fcfa80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfb5ffb640 1 -- 172.21.15.16:0/3904001798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcfa40cf350 con 0x7fcfb8102570 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff0e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff1b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561cbfe742f0 con 0x561cbff0fb40 2024-09-12T03:02:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561cbfe1bcf0 con 0x561cbff0e5d0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfc1477740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561cbfe702b0 con 0x561cbff19370 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfbce3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff0e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46938/0 (socket says 172.21.15.16:46938) 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfb7fff640 1 -- 172.21.15.16:0/1350697662 learned_addr learned my addr 172.21.15.16:0/1350697662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.697+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff1b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 -- 172.21.15.16:0/1350697662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 msgr2=0x561cbff1b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff1b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 -- 172.21.15.16:0/1350697662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 msgr2=0x561cbff0e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff0e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 0x7fcfa80a50c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0005fd0 tx=0x7fcfb0011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 -- 172.21.15.16:0/1350697662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cbfe003a0 con 0x561cbff0e5d0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcfa409bb50 tx=0x7fcfa409b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5214eab86e2c6fd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1350697662 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa40a4830 con 0x561cbff0e5d0 2024-09-12T03:02:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1350697662 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4099ec0 con 0x561cbff0e5d0 2024-09-12T03:02:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.698+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1350697662 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa409b270 con 0x561cbff0e5d0 2024-09-12T03:02:48.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 -- 172.21.15.16:0/1350697662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 msgr2=0x561cbff0ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcfa409bb50 tx=0x7fcfa409b640 comp rx=0 tx=0).stop 2024-09-12T03:02:48.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 -- 172.21.15.16:0/1350697662 shutdown_connections 2024-09-12T03:02:48.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff1b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff0ea10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 --2- 172.21.15.16:0/1350697662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff0e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 -- 172.21.15.16:0/1350697662 >> 172.21.15.16:0/1350697662 conn(0x561cbfed7310 msgr2=0x561cbff0ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.699+0000 7fcfc1477740 1 -- 172.21.15.16:0/1350697662 shutdown_connections 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.700+0000 7fcfc1477740 1 -- 172.21.15.16:0/1350697662 wait complete. 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.700+0000 7fcfc1477740 1 -- start start 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.700+0000 7fcfc1477740 1 Processor -- start 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0e5d0 0x561cbff18990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0fb40 0x561cbff11c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561cbff20420 con 0x561cbff0e5d0 2024-09-12T03:02:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561cbfe2f060 con 0x561cbff19370 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561cbff205a0 con 0x561cbff0fb40 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfc1477740 10 client.0 _ll_register_callbacks cb 0x561cbfb828c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0fb40 0x561cbff11c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbce3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0e5d0 0x561cbff18990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46950/0 (socket says 172.21.15.16:46950) 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 -- 172.21.15.16:0/3376654858 learned_addr learned my addr 172.21.15.16:0/3376654858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 -- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0fb40 msgr2=0x561cbff11c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0fb40 0x561cbff11c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 -- 172.21.15.16:0/3376654858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0e5d0 msgr2=0x561cbff18990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.701+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0e5d0 0x561cbff18990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcfbd63b640 1 -- 172.21.15.16:0/3376654858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbfbe53c0 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcfb001db90 tx=0x7fcfb0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151ff11a739c987b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcf97fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0017040 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa4099ec0 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.702+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcfb0009040 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.703+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcfb000fc70 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.703+0000 7fcfc1477740 10 client.28274 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.703+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561cbff303a0 con 0x561cbff19370 2024-09-12T03:02:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.703+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4099ec0 con 0x561cbff19370 2024-09-12T03:02:48.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fcfb00057a0 con 0x561cbff19370 2024-09-12T03:02:48.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 10 client.28274 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 10 client.28274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 20 client.28274 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff33cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 4 client.28274 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.704+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cbfe81860 con 0x561cbff318b0 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.705+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff33cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.705+0000 7fcf97fff640 10 client.28274 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.705+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff33cf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=376bf915c5acfe82 server_cookie=6441a1511086c37f in_seq=0 out_seq=0 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.706+0000 7fcf97fff640 1 -- 172.21.15.16:0/3376654858 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561cbfe81860 con 0x561cbff318b0 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.706+0000 7fcf97fff640 10 client.28274 handle_command_reply: tid=0 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.706+0000 7fcfc1477740 1 client.28274 shutdown 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 msgr2=0x561cbff12160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcfb001db90 tx=0x7fcfb0004ab0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 shutdown_connections 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff19370 0x561cbff12160 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff33cf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0fb40 0x561cbff11c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcf97fff640 0 client.28274 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3376654858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0e5d0 0x561cbff18990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 >> 172.21.15.16:0/3376654858 conn(0x561cbfed7310 msgr2=0x561cbff10960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 shutdown_connections 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.707+0000 7fcfc1477740 1 -- 172.21.15.16:0/3376654858 wait complete. 2024-09-12T03:02:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.708+0000 7fcfc1477740 20 client.28274 trim_cache size 0 max 16384 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 Processor -- start 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 -- start start 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff1c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff39aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ab310 con 0x561cbff0fb40 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81acf10 con 0x561cbff0e5d0 2024-09-12T03:02:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfc1477740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ab9e0 con 0x561cbff19370 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43432/0 (socket says 172.21.15.16:43432) 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3454991154 learned_addr learned my addr 172.21.15.16:0/3454991154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff1c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff39aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3454991154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 msgr2=0x561cbff39aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff39aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3454991154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 msgr2=0x561cbff1c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff1c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.725+0000 7fcfb7fff640 1 -- 172.21.15.16:0/3454991154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cbff342d0 con 0x561cbff0fb40 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff1c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfbd63b640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff39aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40cceb0 tx=0x7fcfa40a00f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede94af8a6d09b81 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfa27fc640 1 -- 172.21.15.16:0/3454991154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa40a9070 con 0x561cbff0fb40 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfa27fc640 1 -- 172.21.15.16:0/3454991154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4098230 con 0x561cbff0fb40 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.726+0000 7fcfa27fc640 1 -- 172.21.15.16:0/3454991154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa409b010 con 0x561cbff0fb40 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- 172.21.15.16:0/3454991154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 msgr2=0x561cbff39560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40cceb0 tx=0x7fcfa40a00f0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- 172.21.15.16:0/3454991154 shutdown_connections 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff19370 0x561cbff39aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff0fb40 0x561cbff39560 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3454991154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0e5d0 0x561cbff1c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- 172.21.15.16:0/3454991154 >> 172.21.15.16:0/3454991154 conn(0x561cbfed7310 msgr2=0x561cbff09d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- 172.21.15.16:0/3454991154 shutdown_connections 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- 172.21.15.16:0/3454991154 wait complete. 2024-09-12T03:02:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 -- start start 2024-09-12T03:02:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.727+0000 7fcfc1477740 1 Processor -- start 2024-09-12T03:02:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0fb40 0x561cbff29c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff19370 0x561cbff2a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ac840 con 0x561cbff19370 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ac170 con 0x561cbff0fb40 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81a92f0 con 0x561cbff0e5d0 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfc1477740 10 client.0 _ll_register_callbacks cb 0x7fcfb001d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbd63b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff19370 0x561cbff2a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48586/0 (socket says 172.21.15.16:48586) 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 -- 172.21.15.16:0/3587783437 learned_addr learned my addr 172.21.15.16:0/3587783437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0fb40 0x561cbff29c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 -- 172.21.15.16:0/3587783437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0fb40 msgr2=0x561cbff29c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0fb40 0x561cbff29c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 -- 172.21.15.16:0/3587783437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff19370 msgr2=0x561cbff2a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff19370 0x561cbff2a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.728+0000 7fcfbce3a640 1 -- 172.21.15.16:0/3587783437 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbff2a9c0 con 0x561cbff0e5d0 2024-09-12T03:02:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcfbce3a640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009900 tx=0x7fcfac004130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=683caba0ad5e710f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcf97fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac035040 con 0x561cbff0e5d0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcfac005240 con 0x561cbff0e5d0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa4098230 con 0x561cbff0e5d0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcfac005440 con 0x561cbff0e5d0 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcfc1477740 10 client.28198 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561cbff9c560 con 0x561cbff0e5d0 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.729+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4098230 con 0x561cbff0e5d0 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fcfac0036a0 con 0x561cbff0e5d0 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 10 client.28198 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 10 client.28198 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 20 client.28198 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff9ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 4 client.28198 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561cbfe7bd70 con 0x561cbff318b0 2024-09-12T03:02:48.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff9ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcfb7fff640 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff9ee30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c62aa8661c843b3e server_cookie=82776ff954d51a58 in_seq=0 out_seq=0 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.730+0000 7fcf97fff640 10 client.28198 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.731+0000 7fcf97fff640 1 -- 172.21.15.16:0/3587783437 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561cbfe7bd70 con 0x561cbff318b0 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.731+0000 7fcf97fff640 10 client.28198 handle_command_reply: tid=0 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.731+0000 7fcfc1477740 1 client.28198 shutdown 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.731+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 msgr2=0x561cbff2e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.731+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009900 tx=0x7fcfac004130 comp rx=0 tx=0).stop 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 shutdown_connections 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cbff19370 0x561cbff2a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561cbff0fb40 0x561cbff29c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561cbff318b0 0x561cbff9ee30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 --2- 172.21.15.16:0/3587783437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561cbff0e5d0 0x561cbff2e950 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 >> 172.21.15.16:0/3587783437 conn(0x561cbfed7310 msgr2=0x561cbff10110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcf97fff640 0 client.28198 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:02:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 shutdown_connections 2024-09-12T03:02:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 1 -- 172.21.15.16:0/3587783437 wait complete. 2024-09-12T03:02:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.732+0000 7fcfc1477740 20 client.28198 trim_cache size 0 max 16384 2024-09-12T03:02:48.740 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 msgr2=0x7fcfa80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 0x7fcfa80a50c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0005fd0 tx=0x7fcfb0011810 comp rx=0 tx=0).stop 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 msgr2=0x7fcfb81989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9f0 tx=0x7fcfa400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 shutdown_connections 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcfa80a2c00 0x7fcfa80a50c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfb8102570 0x7fcfb81989d0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 --2- 172.21.15.16:0/3904001798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcfb8100e20 0x7fcfb8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 >> 172.21.15.16:0/3904001798 conn(0x7fcfb80fc7d0 msgr2=0x7fcfb8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 shutdown_connections 2024-09-12T03:02:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:48.733+0000 7fcfbf09e640 1 -- 172.21.15.16:0/3904001798 wait complete. 2024-09-12T03:02:49.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 33803 18.06 MB/sec warmup 110 sec latency 41.577 ms 2024-09-12T03:02:49.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:49 smithi016 ceph-mon[30925]: pgmap v784: 97 pgs: 97 active+clean; 528 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-12T03:02:50.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34682 18.37 MB/sec warmup 111 sec latency 42.238 ms 2024-09-12T03:02:50.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:02:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:49 smithi033 ceph-mon[35092]: pgmap v784: 97 pgs: 97 active+clean; 528 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-12T03:02:50.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:49 smithi184 ceph-mon[35321]: pgmap v784: 97 pgs: 97 active+clean; 528 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-12T03:02:50.398 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:51.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 35400 18.63 MB/sec warmup 112 sec latency 52.361 ms 2024-09-12T03:02:51.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 -- 172.21.15.16:0/2289554397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea800ffe60 msgr2=0x7fea80100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 --2- 172.21.15.16:0/2289554397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea800ffe60 0x7fea80100240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fea78009860 tx=0x7fea7802a3d0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 -- 172.21.15.16:0/2289554397 shutdown_connections 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 --2- 172.21.15.16:0/2289554397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea8010a2f0 0x7fea80110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 --2- 172.21.15.16:0/2289554397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea80100780 0x7fea80109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 --2- 172.21.15.16:0/2289554397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea800ffe60 0x7fea80100240 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.043+0000 7fea879bd640 1 -- 172.21.15.16:0/2289554397 >> 172.21.15.16:0/2289554397 conn(0x7fea800fba90 msgr2=0x7fea800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.045+0000 7fea879bd640 1 -- 172.21.15.16:0/2289554397 shutdown_connections 2024-09-12T03:02:51.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.045+0000 7fea879bd640 1 -- 172.21.15.16:0/2289554397 wait complete. 2024-09-12T03:02:51.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.046+0000 7fea879bd640 1 Processor -- start 2024-09-12T03:02:51.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.046+0000 7fea879bd640 1 -- start start 2024-09-12T03:02:51.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea800ffe60 0x7fea80103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8010a2f0 0x7fea80104300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea85f5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8010a2f0 0x7fea80104300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48598/0 (socket says 172.21.15.16:48598) 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 -- 172.21.15.16:0/2732000352 learned_addr learned my addr 172.21.15.16:0/2732000352 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea80114110 con 0x7fea8010a2f0 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea80113f90 con 0x7fea800ffe60 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea80114290 con 0x7fea80100780 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea85759640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea800ffe60 0x7fea80103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 -- 172.21.15.16:0/2732000352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea800ffe60 msgr2=0x7fea80103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea800ffe60 0x7fea80103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 -- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8010a2f0 msgr2=0x7fea80104300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8010a2f0 0x7fea80104300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 -- 172.21.15.16:0/2732000352 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea80072c40 con 0x7fea80100780 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.047+0000 7fea84f58640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e9c0 tx=0x7fea7400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.048+0000 7fea7e7fc640 1 -- 172.21.15.16:0/2732000352 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7400cde0 con 0x7fea80100780 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.048+0000 7fea7e7fc640 1 -- 172.21.15.16:0/2732000352 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea740035f0 con 0x7fea80100780 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.048+0000 7fea7e7fc640 1 -- 172.21.15.16:0/2732000352 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74010690 con 0x7fea80100780 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.048+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea80072f30 con 0x7fea80100780 2024-09-12T03:02:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.048+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea80073420 con 0x7fea80100780 2024-09-12T03:02:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.051+0000 7fea7e7fc640 1 -- 172.21.15.16:0/2732000352 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fea740037a0 con 0x7fea80100780 2024-09-12T03:02:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.051+0000 7fea89d96740 1 Processor -- start 2024-09-12T03:02:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 -- start start 2024-09-12T03:02:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b83ff7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b83f5d2f0 con 0x563b83ff8b40 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563b83f04cf0 con 0x563b84002370 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.052+0000 7fea89d96740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563b83f592b0 con 0x563b83ff75d0 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea84f58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea84f58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47012/0 (socket says 172.21.15.16:47012) 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea84f58640 1 -- 172.21.15.16:0/3046986286 learned_addr learned my addr 172.21.15.16:0/3046986286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85759640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b83ff7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 -- 172.21.15.16:0/3046986286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 msgr2=0x563b83ff7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b83ff7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 -- 172.21.15.16:0/3046986286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 msgr2=0x563b84004760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.053+0000 7fea85f5a640 1 -- 172.21.15.16:0/3046986286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b83ee93a0 con 0x563b83ff8b40 2024-09-12T03:02:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.054+0000 7fea85759640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b83ff7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.054+0000 7fea84f58640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.054+0000 7fea85f5a640 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff7090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fea70009d90 tx=0x7fea7000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3403b8c2f33dbc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.054+0000 7fea677fe640 1 -- 172.21.15.16:0/3046986286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70003970 con 0x563b83ff8b40 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.054+0000 7fea677fe640 1 -- 172.21.15.16:0/3046986286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea70004030 con 0x563b83ff8b40 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.055+0000 7fea7e7fc640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 0x7fea680a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.055+0000 7fea85759640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 0x7fea680a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.055+0000 7fea677fe640 1 -- 172.21.15.16:0/3046986286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70011800 con 0x563b83ff8b40 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.055+0000 7fea7e7fc640 1 -- 172.21.15.16:0/2732000352 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea740cf3d0 con 0x7fea80100780 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- 172.21.15.16:0/3046986286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 msgr2=0x563b83ff7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff7090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fea70009d90 tx=0x7fea7000f730 comp rx=0 tx=0).stop 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- 172.21.15.16:0/3046986286 shutdown_connections 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b84004760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b83ff7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 --2- 172.21.15.16:0/3046986286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff7090 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- 172.21.15.16:0/3046986286 >> 172.21.15.16:0/3046986286 conn(0x563b83fc0310 msgr2=0x563b83ff3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea85759640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 0x7fea680a50c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x563b83ff4ac0 tx=0x7fea7802f4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- 172.21.15.16:0/3046986286 shutdown_connections 2024-09-12T03:02:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- 172.21.15.16:0/3046986286 wait complete. 2024-09-12T03:02:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.056+0000 7fea89d96740 1 -- start start 2024-09-12T03:02:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.057+0000 7fea89d96740 1 Processor -- start 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ffefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b83fff570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48628/0 (socket says 172.21.15.16:48628) 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 -- 172.21.15.16:0/2340889665 learned_addr learned my addr 172.21.15.16:0/2340889665 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563b840050c0 con 0x563b83ff8b40 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563b83f18060 con 0x563b84002370 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563b84005240 con 0x563b83ff75d0 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85f5a640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b83fff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea84f58640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ffefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea89d96740 10 client.0 _ll_register_callbacks cb 0x563b83c6b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 -- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 msgr2=0x563b83fff570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b83fff570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 -- 172.21.15.16:0/2340889665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 msgr2=0x563b83ffefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ffefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.058+0000 7fea85759640 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b83cce3c0 con 0x563b83ff75d0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.059+0000 7fea85759640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea78034000 tx=0x7fea7802a3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52444ba53a74e953 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.059+0000 7fea64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.059+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78005040 con 0x563b83ff75d0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.059+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea780051e0 con 0x563b83ff75d0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.059+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea70004030 con 0x563b83ff75d0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fea780053e0 con 0x563b83ff75d0 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea89d96740 10 client.28210 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563b84010b20 con 0x563b83ff75d0 2024-09-12T03:02:51.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea70004030 con 0x563b83ff75d0 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fea7802f7a0 con 0x563b83ff75d0 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea89d96740 10 client.28210 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.060+0000 7fea89d96740 10 client.28210 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea89d96740 20 client.28210 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b84011fe0 0x563b84014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea89d96740 4 client.28210 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b83f6a860 con 0x563b84011fe0 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea84f58640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b84011fe0 0x563b84014430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea84f58640 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b84011fe0 0x563b84014430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=149b5bbb74d9cd7d server_cookie=cde7b33c15b4e70a in_seq=0 out_seq=0 2024-09-12T03:02:51.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.061+0000 7fea64ff9640 10 client.28210 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.063+0000 7fea64ff9640 1 -- 172.21.15.16:0/2340889665 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b83f6a860 con 0x563b84011fe0 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.063+0000 7fea64ff9640 10 client.28210 handle_command_reply: tid=0 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.063+0000 7fea89d96740 1 client.28210 shutdown 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 msgr2=0x563b84001d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea78034000 tx=0x7fea7802a3d0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 shutdown_connections 2024-09-12T03:02:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b84011fe0 0x563b84014430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b84002370 0x563b83fff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ffefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 --2- 172.21.15.16:0/2340889665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84001d20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 >> 172.21.15.16:0/2340889665 conn(0x563b83fc0310 msgr2=0x563b83ff9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea64ff9640 0 client.28210 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 shutdown_connections 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 1 -- 172.21.15.16:0/2340889665 wait complete. 2024-09-12T03:02:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.064+0000 7fea89d96740 20 client.28210 trim_cache size 0 max 16384 2024-09-12T03:02:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.082+0000 7fea89d96740 1 Processor -- start 2024-09-12T03:02:51.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 -- start start 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 0x563b83ffdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b84002370 0x563b83ffa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea801bc420 con 0x563b83ff8b40 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea801be020 con 0x563b83ff75d0 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea89d96740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea801bcaf0 con 0x563b84002370 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea84f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea85759640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 0x563b83ffdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea84f58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43478/0 (socket says 172.21.15.16:43478) 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea84f58640 1 -- 172.21.15.16:0/598143839 learned_addr learned my addr 172.21.15.16:0/598143839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.083+0000 7fea85f5a640 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b84002370 0x563b83ffa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 -- 172.21.15.16:0/598143839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b84002370 msgr2=0x563b83ffa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b84002370 0x563b83ffa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 -- 172.21.15.16:0/598143839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 msgr2=0x563b83ff2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 -- 172.21.15.16:0/598143839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b84002050 con 0x563b83ff75d0 2024-09-12T03:02:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea85759640 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 0x563b83ffdf10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea7802a8f0 tx=0x7fea78030770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b06c47f6e822f48e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea66ffd640 1 -- 172.21.15.16:0/598143839 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7802fe30 con 0x563b83ff75d0 2024-09-12T03:02:51.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea66ffd640 1 -- 172.21.15.16:0/598143839 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea7804d070 con 0x563b83ff75d0 2024-09-12T03:02:51.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.084+0000 7fea66ffd640 1 -- 172.21.15.16:0/598143839 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78052930 con 0x563b83ff75d0 2024-09-12T03:02:51.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 -- 172.21.15.16:0/598143839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 msgr2=0x563b83ffdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 0x563b83ffdf10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea7802a8f0 tx=0x7fea78030770 comp rx=0 tx=0).stop 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 -- 172.21.15.16:0/598143839 shutdown_connections 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b84002370 0x563b83ffa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b83ff8b40 0x563b83ff2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 --2- 172.21.15.16:0/598143839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff75d0 0x563b83ffdf10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 -- 172.21.15.16:0/598143839 >> 172.21.15.16:0/598143839 conn(0x563b83fc0310 msgr2=0x563b83ff24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 -- 172.21.15.16:0/598143839 shutdown_connections 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.085+0000 7fea89d96740 1 -- 172.21.15.16:0/598143839 wait complete. 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.086+0000 7fea89d96740 1 -- start start 2024-09-12T03:02:51.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.086+0000 7fea89d96740 1 Processor -- start 2024-09-12T03:02:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 0x563b84006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 0x563b84074c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea85759640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea85759640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48648/0 (socket says 172.21.15.16:48648) 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea84f58640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 0x563b84006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea85759640 1 -- 172.21.15.16:0/3597804857 learned_addr learned my addr 172.21.15.16:0/3597804857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea801bd950 con 0x563b84002370 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea801bd280 con 0x563b83ff8b40 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea801ba400 con 0x563b83ff75d0 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.087+0000 7fea89d96740 10 client.0 _ll_register_callbacks cb 0x7fea700093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85f5a640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 0x563b84074c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 -- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 msgr2=0x563b84006180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 0x563b84006180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 -- 172.21.15.16:0/3597804857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 msgr2=0x563b84074c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 0x563b84074c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b840752d0 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85f5a640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 0x563b84074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea84f58640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 0x563b84006180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea85759640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fea7802fc40 tx=0x7fea780096b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54671c74295ef2b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.088+0000 7fea64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78005640 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fea78004240 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea7804d070 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fea7806c050 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea89d96740 10 client.28216 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.089+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563b8407c600 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea78004440 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fea78003ae0 con 0x563b83ff75d0 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 10 client.28216 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 10 client.28216 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:02:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 20 client.28216 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b8407cc60 0x563b8407f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 4 client.28216 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:51.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.090+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fea7804d070 con 0x563b8407cc60 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.091+0000 7fea84f58640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b8407cc60 0x563b8407f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.091+0000 7fea84f58640 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b8407cc60 0x563b8407f050 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3f40e5ae7e883ad server_cookie=54e2dbd72013eed1 in_seq=0 out_seq=0 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.091+0000 7fea64ff9640 10 client.28216 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea64ff9640 1 -- 172.21.15.16:0/3597804857 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fea7804d070 con 0x563b8407cc60 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea64ff9640 10 client.28216 handle_command_reply: tid=0 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 client.28216 shutdown 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 msgr2=0x563b84005c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fea7802fc40 tx=0x7fea780096b0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 shutdown_connections 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563b8407cc60 0x563b8407f050 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563b84002370 0x563b84074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563b83ff8b40 0x563b84006180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea64ff9640 0 client.28216 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 --2- 172.21.15.16:0/3597804857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563b83ff75d0 0x563b84005c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 >> 172.21.15.16:0/3597804857 conn(0x563b83fc0310 msgr2=0x563b83ff2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 shutdown_connections 2024-09-12T03:02:51.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.094+0000 7fea89d96740 1 -- 172.21.15.16:0/3597804857 wait complete. 2024-09-12T03:02:51.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea89d96740 20 client.28216 trim_cache size 0 max 16384 2024-09-12T03:02:51.097 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 msgr2=0x7fea680a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 0x7fea680a50c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x563b83ff4ac0 tx=0x7fea7802f4a0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 msgr2=0x7fea80103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e9c0 tx=0x7fea7400ee80 comp rx=0 tx=0).stop 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 shutdown_connections 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fea680a2c00 0x7fea680a50c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8010a2f0 0x7fea80104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea80100780 0x7fea80103dc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 --2- 172.21.15.16:0/2732000352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea800ffe60 0x7fea80103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.095+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 >> 172.21.15.16:0/2732000352 conn(0x7fea800fba90 msgr2=0x7fea80107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.096+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 shutdown_connections 2024-09-12T03:02:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:51.096+0000 7fea879bd640 1 -- 172.21.15.16:0/2732000352 wait complete. 2024-09-12T03:02:51.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:51 smithi016 ceph-mon[30925]: pgmap v785: 97 pgs: 97 active+clean; 537 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.3 MiB/s wr, 678 op/s 2024-09-12T03:02:52.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36139 18.78 MB/sec warmup 113 sec latency 78.797 ms 2024-09-12T03:02:52.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:51 smithi033 ceph-mon[35092]: pgmap v785: 97 pgs: 97 active+clean; 537 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.3 MiB/s wr, 678 op/s 2024-09-12T03:02:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:51 smithi184 ceph-mon[35321]: pgmap v785: 97 pgs: 97 active+clean; 537 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.3 MiB/s wr, 678 op/s 2024-09-12T03:02:52.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:02:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:53.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36575 18.82 MB/sec warmup 114 sec latency 176.872 ms 2024-09-12T03:02:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 -- 172.21.15.16:0/3956207055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c102570 msgr2=0x7f447c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 --2- 172.21.15.16:0/3956207055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c102570 0x7f447c1029b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f4464009900 tx=0x7f446402a480 comp rx=0 tx=0).stop 2024-09-12T03:02:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 -- 172.21.15.16:0/3956207055 shutdown_connections 2024-09-12T03:02:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 --2- 172.21.15.16:0/3956207055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102ef0 0x7f447c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 --2- 172.21.15.16:0/3956207055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c102570 0x7f447c1029b0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 --2- 172.21.15.16:0/3956207055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c100e20 0x7f447c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.340+0000 7f4480e13640 1 -- 172.21.15.16:0/3956207055 >> 172.21.15.16:0/3956207055 conn(0x7f447c0fc7d0 msgr2=0x7f447c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.341+0000 7f4480e13640 1 -- 172.21.15.16:0/3956207055 shutdown_connections 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.341+0000 7f4480e13640 1 -- 172.21.15.16:0/3956207055 wait complete. 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.341+0000 7f4480e13640 1 Processor -- start 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.341+0000 7f4480e13640 1 -- start start 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102570 0x7f447c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 0x7f447c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f447c110f50 con 0x7f447c100e20 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f447c110dd0 con 0x7f447c102ef0 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4480e13640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1110d0 con 0x7f447c102570 2024-09-12T03:02:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43512/0 (socket says 172.21.15.16:43512) 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447a59c640 1 -- 172.21.15.16:0/278010465 learned_addr learned my addr 172.21.15.16:0/278010465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 0x7f447c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f4473fff640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102570 0x7f447c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 -- 172.21.15.16:0/278010465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102570 msgr2=0x7f447c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102570 0x7f447c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 -- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 msgr2=0x7f447c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 -- 172.21.15.16:0/278010465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c19d550 con 0x7f447c102ef0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447a59c640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.342+0000 7f447ad9d640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 0x7f447c19ce70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f447400da10 tx=0x7f447400ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.343+0000 7f44737fe640 1 -- 172.21.15.16:0/278010465 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447400bd70 con 0x7f447c102ef0 2024-09-12T03:02:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.343+0000 7f44737fe640 1 -- 172.21.15.16:0/278010465 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4474003e50 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.343+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c19d840 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.343+0000 7f44737fe640 1 -- 172.21.15.16:0/278010465 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44740106a0 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.343+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c1a5120 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.345+0000 7f44737fe640 1 -- 172.21.15.16:0/278010465 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4474003970 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.346+0000 7f44737fe640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 0x7f44680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.346+0000 7f44831ec740 1 Processor -- start 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.346+0000 7f44737fe640 1 -- 172.21.15.16:0/278010465 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44740cf340 con 0x7f447c102ef0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.346+0000 7f447a59c640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 0x7f44680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 -- start start 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c35a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c46b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638b6b9b2f0 con 0x5638b6c36b40 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638b6b42cf0 con 0x5638b6c44780 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f44831ec740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5638b6b972b0 con 0x5638b6c355d0 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f447a59c640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 0x7f44680a5040 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f446c005fd0 tx=0x7f446c00b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f447a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f447a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43518/0 (socket says 172.21.15.16:43518) 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f447a59c640 1 -- 172.21.15.16:0/4006599325 learned_addr learned my addr 172.21.15.16:0/4006599325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f447ad9d640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c46b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.347+0000 7f4473fff640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c35a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 -- 172.21.15.16:0/4006599325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 msgr2=0x5638b6c35a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c35a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 -- 172.21.15.16:0/4006599325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 msgr2=0x5638b6c46b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c46b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 -- 172.21.15.16:0/4006599325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b6b273a0 con 0x5638b6c36b40 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447ad9d640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c46b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.349+0000 7f447a59c640 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f446c011870 tx=0x7f446c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be6b1819d8847b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.350+0000 7f4470ff9640 1 -- 172.21.15.16:0/4006599325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c00eb30 con 0x5638b6c36b40 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.350+0000 7f4470ff9640 1 -- 172.21.15.16:0/4006599325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c00ecd0 con 0x5638b6c36b40 2024-09-12T03:02:53.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.350+0000 7f4470ff9640 1 -- 172.21.15.16:0/4006599325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c01d700 con 0x5638b6c36b40 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.351+0000 7f44831ec740 1 -- 172.21.15.16:0/4006599325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 msgr2=0x5638b6c35090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.351+0000 7f44831ec740 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f446c011870 tx=0x7f446c013000 comp rx=0 tx=0).stop 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 -- 172.21.15.16:0/4006599325 shutdown_connections 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c46b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c35a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 --2- 172.21.15.16:0/4006599325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c36b40 0x5638b6c35090 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 -- 172.21.15.16:0/4006599325 >> 172.21.15.16:0/4006599325 conn(0x5638b6bfe310 msgr2=0x5638b6c31c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 -- 172.21.15.16:0/4006599325 shutdown_connections 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 -- 172.21.15.16:0/4006599325 wait complete. 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 -- start start 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.352+0000 7f44831ec740 1 Processor -- start 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c355d0 0x5638b6c43dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c3d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638b6c47480 con 0x5638b6c355d0 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5638b6b56060 con 0x5638b6c44780 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5638b6c47600 con 0x5638b6c36b40 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f44831ec740 10 client.0 _ll_register_callbacks cb 0x5638b68a98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c355d0 0x5638b6c43dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47080/0 (socket says 172.21.15.16:47080) 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 -- 172.21.15.16:0/1431604940 learned_addr learned my addr 172.21.15.16:0/1431604940 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f4473fff640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c3d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 -- 172.21.15.16:0/1431604940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 msgr2=0x5638b6c3d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c3d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 -- 172.21.15.16:0/1431604940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c355d0 msgr2=0x5638b6c43dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c355d0 0x5638b6c43dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f447ad9d640 1 -- 172.21.15.16:0/1431604940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b690c3c0 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.353+0000 7f4473fff640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f447ad9d640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4474098580 tx=0x7f447409bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a7d7163b94b0adc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f4461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44740b2030 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4474004000 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f446c00ecd0 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4474003c60 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f44831ec740 10 client.28310 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.354+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5638b6c57350 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c00ecd0 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f4474024060 con 0x5638b6c44780 2024-09-12T03:02:53.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 10 client.28310 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 10 client.28310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 20 client.28310 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6c5ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 4 client.28310 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638b6ba8860 con 0x5638b6c58810 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.355+0000 7f447a59c640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6c5ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.356+0000 7f447a59c640 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6c5ac60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55cb7e4cee7547dc server_cookie=e341d65ee1be0e0f in_seq=0 out_seq=0 2024-09-12T03:02:53.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.356+0000 7f4461ffb640 10 client.28310 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f4461ffb640 1 -- 172.21.15.16:0/1431604940 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638b6ba8860 con 0x5638b6c58810 2024-09-12T03:02:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f4461ffb640 10 client.28310 handle_command_reply: tid=0 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 client.28310 shutdown 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 msgr2=0x5638b6c3d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4474098580 tx=0x7f447409bc70 comp rx=0 tx=0).stop 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 shutdown_connections 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6c5ac60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c44780 0x5638b6c3d590 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f4461ffb640 0 client.28310 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 --2- 172.21.15.16:0/1431604940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c355d0 0x5638b6c43dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.358+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 >> 172.21.15.16:0/1431604940 conn(0x5638b6bfe310 msgr2=0x5638b6c37960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.359+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 shutdown_connections 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.359+0000 7f44831ec740 1 -- 172.21.15.16:0/1431604940 wait complete. 2024-09-12T03:02:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.359+0000 7f44831ec740 20 client.28310 trim_cache size 0 max 16384 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.378+0000 7f44831ec740 1 Processor -- start 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.378+0000 7f44831ec740 1 -- start start 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c31d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c36b40 0x5638b6c38120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c38660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1ab350 con 0x5638b6c44780 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1acf50 con 0x5638b6c36b40 2024-09-12T03:02:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f44831ec740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1aba20 con 0x5638b6c355d0 2024-09-12T03:02:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c38660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c38660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43536/0 (socket says 172.21.15.16:43536) 2024-09-12T03:02:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447ad9d640 1 -- 172.21.15.16:0/3555081780 learned_addr learned my addr 172.21.15.16:0/3555081780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c31d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f4473fff640 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c36b40 0x5638b6c38120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 -- 172.21.15.16:0/3555081780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c36b40 msgr2=0x5638b6c38120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c36b40 0x5638b6c38120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 -- 172.21.15.16:0/3555081780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 msgr2=0x5638b6c38660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c38660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.379+0000 7f447a59c640 1 -- 172.21.15.16:0/3555081780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b6c417f0 con 0x5638b6c355d0 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.380+0000 7f447a59c640 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c31d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f446c01ee50 tx=0x7f446c01eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd2d0f823df91276 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.380+0000 7f4463fff640 1 -- 172.21.15.16:0/3555081780 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c00ebb0 con 0x5638b6c355d0 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.380+0000 7f4463fff640 1 -- 172.21.15.16:0/3555081780 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c00ed50 con 0x5638b6c355d0 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.380+0000 7f4463fff640 1 -- 172.21.15.16:0/3555081780 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c004e50 con 0x5638b6c355d0 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- 172.21.15.16:0/3555081780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 msgr2=0x5638b6c31d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c31d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f446c01ee50 tx=0x7f446c01eeb0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- 172.21.15.16:0/3555081780 shutdown_connections 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c38660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c36b40 0x5638b6c38120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 --2- 172.21.15.16:0/3555081780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c355d0 0x5638b6c31d70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- 172.21.15.16:0/3555081780 >> 172.21.15.16:0/3555081780 conn(0x5638b6bfe310 msgr2=0x5638b6c410e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- 172.21.15.16:0/3555081780 shutdown_connections 2024-09-12T03:02:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- 172.21.15.16:0/3555081780 wait complete. 2024-09-12T03:02:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 -- start start 2024-09-12T03:02:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.381+0000 7f44831ec740 1 Processor -- start 2024-09-12T03:02:53.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c355d0 0x5638b6c55400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c55940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1ac880 con 0x5638b6c44780 2024-09-12T03:02:53.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1ac1b0 con 0x5638b6c355d0 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1a9330 con 0x5638b6c36b40 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f44831ec740 10 client.0 _ll_register_callbacks cb 0x7f4474097e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c55940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43542/0 (socket says 172.21.15.16:43542) 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f447ad9d640 1 -- 172.21.15.16:0/3239122006 learned_addr learned my addr 172.21.15.16:0/3239122006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f447a59c640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c355d0 0x5638b6c55400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 -- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c355d0 msgr2=0x5638b6c55400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c355d0 0x5638b6c55400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 -- 172.21.15.16:0/3239122006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 msgr2=0x5638b6c399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f4473fff640 1 -- 172.21.15.16:0/3239122006 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b6c3a1c0 con 0x5638b6c36b40 2024-09-12T03:02:53.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.382+0000 7f447ad9d640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f4473fff640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c55940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4464005940 tx=0x7f44640035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b11a028600d20f96 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f4461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464035040 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f446c00ed50 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f44640039b0 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4464003bb0 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f44831ec740 10 client.28231 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.383+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5638b6cc3030 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c00ed50 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f4464046580 con 0x5638b6c36b40 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 10 client.28231 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 10 client.28231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:53.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 20 client.28231 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6cc57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 4 client.28231 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638b6ba2d70 con 0x5638b6c58810 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.384+0000 7f447a59c640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6cc57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.385+0000 7f4461ffb640 10 client.28231 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:53.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.385+0000 7f447a59c640 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6cc57d0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ffa6e99852747c server_cookie=cc0d668bd75caa44 in_seq=0 out_seq=0 2024-09-12T03:02:53.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f4461ffb640 1 -- 172.21.15.16:0/3239122006 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638b6ba2d70 con 0x5638b6c58810 2024-09-12T03:02:53.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f4461ffb640 10 client.28231 handle_command_reply: tid=0 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 client.28231 shutdown 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 msgr2=0x5638b6c55940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c55940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4464005940 tx=0x7f44640035f0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 shutdown_connections 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5638b6c58810 0x5638b6cc57d0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638b6c44780 0x5638b6c399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5638b6c36b40 0x5638b6c55940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f4461ffb640 0 client.28231 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 --2- 172.21.15.16:0/3239122006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5638b6c355d0 0x5638b6c55400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 >> 172.21.15.16:0/3239122006 conn(0x5638b6bfe310 msgr2=0x5638b6c30310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.386+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 shutdown_connections 2024-09-12T03:02:53.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f44831ec740 1 -- 172.21.15.16:0/3239122006 wait complete. 2024-09-12T03:02:53.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f44831ec740 20 client.28231 trim_cache size 0 max 16384 2024-09-12T03:02:53.392 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 msgr2=0x7f44680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 0x7f44680a5040 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f446c005fd0 tx=0x7f446c00b4a0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 msgr2=0x7f447c19ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 0x7f447c19ce70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f447400da10 tx=0x7f447400ded0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 shutdown_connections 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f447c102ef0 0x7f447c19ce70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c102570 0x7f447c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f44680a2b80 0x7f44680a5040 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 --2- 172.21.15.16:0/278010465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c100e20 0x7f447c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:53.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.387+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 >> 172.21.15.16:0/278010465 conn(0x7f447c0fc7d0 msgr2=0x7f447c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:53.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.388+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 shutdown_connections 2024-09-12T03:02:53.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:53.388+0000 7f4480e13640 1 -- 172.21.15.16:0/278010465 wait complete. 2024-09-12T03:02:54.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 37112 18.91 MB/sec warmup 115 sec latency 62.581 ms 2024-09-12T03:02:54.096 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:53 smithi184 ceph-mon[35321]: pgmap v786: 97 pgs: 97 active+clean; 539 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 565 op/s 2024-09-12T03:02:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:53 smithi016 ceph-mon[30925]: pgmap v786: 97 pgs: 97 active+clean; 539 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 565 op/s 2024-09-12T03:02:54.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:53 smithi033 ceph-mon[35092]: pgmap v786: 97 pgs: 97 active+clean; 539 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 565 op/s 2024-09-12T03:02:55.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 37930 19.09 MB/sec warmup 116 sec latency 39.039 ms 2024-09-12T03:02:55.268 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:02:55.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:02:55.591 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:56.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38831 19.29 MB/sec warmup 117 sec latency 41.030 ms 2024-09-12T03:02:56.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:55 smithi016 ceph-mon[30925]: pgmap v787: 97 pgs: 97 active+clean; 544 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.8 MiB/s wr, 616 op/s 2024-09-12T03:02:56.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:55 smithi033 ceph-mon[35092]: pgmap v787: 97 pgs: 97 active+clean; 544 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.8 MiB/s wr, 616 op/s 2024-09-12T03:02:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:55 smithi184 ceph-mon[35321]: pgmap v787: 97 pgs: 97 active+clean; 544 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.8 MiB/s wr, 616 op/s 2024-09-12T03:02:56.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 -- 172.21.15.16:0/3828223070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 msgr2=0x7f2d2c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 --2- 172.21.15.16:0/3828223070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c10ec40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009cd0 tx=0x7f2d2402a8e0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 -- 172.21.15.16:0/3828223070 shutdown_connections 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 --2- 172.21.15.16:0/3828223070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c10ec40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 --2- 172.21.15.16:0/3828223070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 --2- 172.21.15.16:0/3828223070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c102d30 0x7f2d2c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 -- 172.21.15.16:0/3828223070 >> 172.21.15.16:0/3828223070 conn(0x7f2d2c0fc7d0 msgr2=0x7f2d2c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 -- 172.21.15.16:0/3828223070 shutdown_connections 2024-09-12T03:02:56.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.468+0000 7f2d33676640 1 -- 172.21.15.16:0/3828223070 wait complete. 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 Processor -- start 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 -- start start 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 0x7f2d2c100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1120a0 con 0x7f2d2c102d30 2024-09-12T03:02:56.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c111f20 con 0x7f2d2c1082a0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d33676640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c112220 con 0x7f2d2c1036e0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d31412640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 0x7f2d2c100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d30c11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d31412640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 0x7f2d2c100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47774/0 (socket says 172.21.15.16:47774) 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d31412640 1 -- 172.21.15.16:0/1997483422 learned_addr learned my addr 172.21.15.16:0/1997483422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.469+0000 7f2d30c11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34646/0 (socket says 172.21.15.16:34646) 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d31c13640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 -- 172.21.15.16:0/1997483422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 msgr2=0x7f2d2c101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 -- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 msgr2=0x7f2d2c100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 0x7f2d2c100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 -- 172.21.15.16:0/1997483422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c1a6a20 con 0x7f2d2c1036e0 2024-09-12T03:02:56.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d30c11640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0697d0 tx=0x7f2d1c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d227fc640 1 -- 172.21.15.16:0/1997483422 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00cc40 con 0x7f2d2c1036e0 2024-09-12T03:02:56.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c1a6d10 con 0x7f2d2c1036e0 2024-09-12T03:02:56.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c1a7240 con 0x7f2d2c1036e0 2024-09-12T03:02:56.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d227fc640 1 -- 172.21.15.16:0/1997483422 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c00cde0 con 0x7f2d2c1036e0 2024-09-12T03:02:56.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.470+0000 7f2d227fc640 1 -- 172.21.15.16:0/1997483422 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c018710 con 0x7f2d2c1036e0 2024-09-12T03:02:56.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 Processor -- start 2024-09-12T03:02:56.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 -- start start 2024-09-12T03:02:56.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a853010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a855940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c84a7a5280 con 0x55c84a853550 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c84a74cb10 con 0x55c84a83fa80 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.474+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c84a7a1240 con 0x55c84a8403a0 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39726/0 (socket says 172.21.15.16:39726) 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 -- 172.21.15.16:0/2584919925 learned_addr learned my addr 172.21.15.16:0/2584919925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d30c11640 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a855940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d227fc640 1 -- 172.21.15.16:0/1997483422 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2d1c010a30 con 0x7f2d2c1036e0 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 -- 172.21.15.16:0/2584919925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 msgr2=0x55c84a853010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a853010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 -- 172.21.15.16:0/2584919925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 msgr2=0x55c84a855940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a855940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.475+0000 7f2d31c13640 1 -- 172.21.15.16:0/2584919925 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84a7591f0 con 0x55c84a83fa80 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.476+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009cd0 tx=0x7f2d24009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd685193a1e92042 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.476+0000 7f2d177fe640 1 -- 172.21.15.16:0/2584919925 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24048070 con 0x55c84a83fa80 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.476+0000 7f2d177fe640 1 -- 172.21.15.16:0/2584919925 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24036070 con 0x55c84a83fa80 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.476+0000 7f2d177fe640 1 -- 172.21.15.16:0/2584919925 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24004440 con 0x55c84a83fa80 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.476+0000 7f2d227fc640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 0x7f2d180a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d31412640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 0x7f2d180a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d227fc640 1 -- 172.21.15.16:0/1997483422 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2d1c0cf400 con 0x7f2d2c1036e0 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2584919925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 msgr2=0x55c84a83fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009cd0 tx=0x7f2d24009a20 comp rx=0 tx=0).stop 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2584919925 shutdown_connections 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d31412640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 0x7f2d180a50c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009cb0 tx=0x7f2d28005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a855940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a853010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2584919925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a83fe60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2584919925 >> 172.21.15.16:0/2584919925 conn(0x55c84a83c0f0 msgr2=0x55c84a83c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2584919925 shutdown_connections 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.477+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2584919925 wait complete. 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 -- start start 2024-09-12T03:02:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 Processor -- start 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a850590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a850ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c84a856060 con 0x55c84a853550 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c84a75ff10 con 0x55c84a83fa80 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c84a8561e0 con 0x55c84a8403a0 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d35a4f740 10 client.0 _ll_register_callbacks cb 0x55c84a728a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d31412640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a850590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47800/0 (socket says 172.21.15.16:47800) 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d31c13640 1 -- 172.21.15.16:0/2624080937 learned_addr learned my addr 172.21.15.16:0/2624080937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.478+0000 7f2d30c11640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a850ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 -- 172.21.15.16:0/2624080937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 msgr2=0x55c84a850ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a850ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 -- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 msgr2=0x55c84a850590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a850590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 -- 172.21.15.16:0/2624080937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84a5166a0 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d31c13640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2d24005e30 tx=0x7f2d24031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ec4f0962c32afc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24048040 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2d24004140 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.479+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d24036070 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d24004340 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 10 client.19014 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c84a856060 con 0x55c84a853550 2024-09-12T03:02:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24036070 con 0x55c84a853550 2024-09-12T03:02:56.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f2d24042030 con 0x55c84a853550 2024-09-12T03:02:56.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 10 client.19014 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:56.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 10 client.19014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:56.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 20 client.19014 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:56.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a865360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 4 client.19014 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.480+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c84a7b27f0 con 0x55c84a862f70 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.481+0000 7f2d31412640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a865360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.481+0000 7f2d31412640 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a865360 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bcca3d240fc7084 server_cookie=41b388b1dcdbf0b in_seq=0 out_seq=0 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.481+0000 7f2d14ff9640 10 client.19014 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.483+0000 7f2d14ff9640 1 -- 172.21.15.16:0/2624080937 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c84a7b27f0 con 0x55c84a862f70 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.483+0000 7f2d14ff9640 10 client.19014 handle_command_reply: tid=0 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 client.19014 shutdown 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 msgr2=0x55c84a851010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2d24005e30 tx=0x7f2d24031cb0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 shutdown_connections 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a865360 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a851010 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d14ff9640 0 client.19014 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a8403a0 0x55c84a850ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/2624080937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a83fa80 0x55c84a850590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 >> 172.21.15.16:0/2624080937 conn(0x55c84a83c0f0 msgr2=0x55c84a83b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 shutdown_connections 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.484+0000 7f2d35a4f740 1 -- 172.21.15.16:0/2624080937 wait complete. 2024-09-12T03:02:56.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.485+0000 7f2d35a4f740 20 client.19014 trim_cache size 0 max 16384 2024-09-12T03:02:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 Processor -- start 2024-09-12T03:02:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 -- start start 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a86a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 0x55c84a86aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ab310 con 0x55c84a83fa80 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1acf10 con 0x55c84a8403a0 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ab9e0 con 0x55c84a853550 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31412640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d30c11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a86a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 0x55c84a86aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31412640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47808/0 (socket says 172.21.15.16:47808) 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31412640 1 -- 172.21.15.16:0/669320122 learned_addr learned my addr 172.21.15.16:0/669320122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 -- 172.21.15.16:0/669320122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 msgr2=0x55c84a86a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a86a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 -- 172.21.15.16:0/669320122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 msgr2=0x55c84a8527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31c13640 1 -- 172.21.15.16:0/669320122 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84a84b320 con 0x55c84a853550 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.507+0000 7f2d31412640 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.508+0000 7f2d31c13640 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 0x55c84a86aed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2d24002230 tx=0x7f2d24009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ba48b9a713add9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.508+0000 7f2d16ffd640 1 -- 172.21.15.16:0/669320122 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24048070 con 0x55c84a853550 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.508+0000 7f2d16ffd640 1 -- 172.21.15.16:0/669320122 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24036040 con 0x55c84a853550 2024-09-12T03:02:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.508+0000 7f2d16ffd640 1 -- 172.21.15.16:0/669320122 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240305a0 con 0x55c84a853550 2024-09-12T03:02:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- 172.21.15.16:0/669320122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 msgr2=0x55c84a86aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 0x55c84a86aed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2d24002230 tx=0x7f2d24009a20 comp rx=0 tx=0).stop 2024-09-12T03:02:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- 172.21.15.16:0/669320122 shutdown_connections 2024-09-12T03:02:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a853550 0x55c84a86aed0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a86a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/669320122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a83fa80 0x55c84a8527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- 172.21.15.16:0/669320122 >> 172.21.15.16:0/669320122 conn(0x55c84a83c0f0 msgr2=0x55c84a84f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- 172.21.15.16:0/669320122 shutdown_connections 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- 172.21.15.16:0/669320122 wait complete. 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 -- start start 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 Processor -- start 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 0x55c84a857100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.509+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a857640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a857b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ac840 con 0x55c84a853550 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ac170 con 0x55c84a8403a0 2024-09-12T03:02:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1a9310 con 0x55c84a83fa80 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 10 client.0 _ll_register_callbacks cb 0x7f2d24046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a857b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d31412640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 0x55c84a857100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a857640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d31c13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a857b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47812/0 (socket says 172.21.15.16:47812) 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d31412640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 0x55c84a857100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34672/0 (socket says 172.21.15.16:34672) 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d31c13640 1 -- 172.21.15.16:0/3511095071 learned_addr learned my addr 172.21.15.16:0/3511095071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 -- 172.21.15.16:0/3511095071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 msgr2=0x55c84a857100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 0x55c84a857100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 -- 172.21.15.16:0/3511095071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 msgr2=0x55c84a857b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a857b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 -- 172.21.15.16:0/3511095071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84a8401d0 con 0x55c84a8403a0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d30c11640 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a857640 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c098120 tx=0x7f2d1c09ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b1c13be946831 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c09b310 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2d1c097a10 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.510+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d24036040 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c09bcc0 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 10 client.28334 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c84a8cd9f0 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c0b2030 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7f2d1c09ab80 con 0x55c84a8403a0 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 10 client.28334 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 10 client.28334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 20 client.28334 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a8d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 4 client.28334 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:02:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c84a7acd00 con 0x55c84a862f70 2024-09-12T03:02:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.511+0000 7f2d31412640 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a8d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.512+0000 7f2d31412640 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a8d0230 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecdec0d27513398 server_cookie=3e3771e05b50e50 in_seq=0 out_seq=0 2024-09-12T03:02:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.512+0000 7f2d14ff9640 10 client.28334 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.514+0000 7f2d14ff9640 1 -- 172.21.15.16:0/3511095071 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c84a7acd00 con 0x55c84a862f70 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.514+0000 7f2d14ff9640 10 client.28334 handle_command_reply: tid=0 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.514+0000 7f2d35a4f740 1 client.28334 shutdown 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 msgr2=0x55c84a857640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a857640 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c098120 tx=0x7f2d1c09ba90 comp rx=0 tx=0).stop 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 shutdown_connections 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c84a862f70 0x55c84a8d0230 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c84a853550 0x55c84a857b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c84a8403a0 0x55c84a857640 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 --2- 172.21.15.16:0/3511095071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c84a83fa80 0x55c84a857100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 >> 172.21.15.16:0/3511095071 conn(0x55c84a83c0f0 msgr2=0x55c84a84f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d14ff9640 0 client.28334 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 shutdown_connections 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 1 -- 172.21.15.16:0/3511095071 wait complete. 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.515+0000 7f2d35a4f740 20 client.28334 trim_cache size 0 max 16384 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:02:56.519 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "0c09c20a-553b-492a-9204-672562f7ebb9", 2024-09-12T03:02:56.520 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:02:56.520 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 msgr2=0x7f2d180a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 0x7f2d180a50c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009cb0 tx=0x7f2d28005f90 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 msgr2=0x7f2d2c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0697d0 tx=0x7f2d1c00ee50 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 shutdown_connections 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.516+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d2c1082a0 0x7f2d2c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d2c1036e0 0x7f2d2c100ed0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2d180a2c00 0x7f2d180a50c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 --2- 172.21.15.16:0/1997483422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c102d30 0x7f2d2c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 >> 172.21.15.16:0/1997483422 conn(0x7f2d2c0fc7d0 msgr2=0x7f2d2c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 shutdown_connections 2024-09-12T03:02:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:56.517+0000 7f2d33676640 1 -- 172.21.15.16:0/1997483422 wait complete. 2024-09-12T03:02:57.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 39494 19.50 MB/sec warmup 118 sec latency 74.263 ms 2024-09-12T03:02:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:56 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:02:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:56 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:56 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:56 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:56 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:02:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:56 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:56 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:57.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:56 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:56 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:02:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:56 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:56 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:57.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:56 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:57.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:02:58.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 39957 19.59 MB/sec warmup 119 sec latency 283.996 ms 2024-09-12T03:02:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:02:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: pgmap v788: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 740 op/s 2024-09-12T03:02:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:57 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: pgmap v788: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 740 op/s 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:57 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: pgmap v788: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 740 op/s 2024-09-12T03:02:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:57 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 -- 172.21.15.16:0/3395300400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc40ffc00 msgr2=0x7fbbc4100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 --2- 172.21.15.16:0/3395300400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc40ffc00 0x7fbbc4100060 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009900 tx=0x7fbbac02a410 comp rx=0 tx=0).stop 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 -- 172.21.15.16:0/3395300400 shutdown_connections 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 --2- 172.21.15.16:0/3395300400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc41005a0 0x7fbbc410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 --2- 172.21.15.16:0/3395300400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc40ffc00 0x7fbbc4100060 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 --2- 172.21.15.16:0/3395300400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 -- 172.21.15.16:0/3395300400 >> 172.21.15.16:0/3395300400 conn(0x7fbbc40fba90 msgr2=0x7fbbc40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 -- 172.21.15.16:0/3395300400 shutdown_connections 2024-09-12T03:02:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 -- 172.21.15.16:0/3395300400 wait complete. 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.609+0000 7fbbca932640 1 Processor -- start 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 -- start start 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc40ffc00 0x7fbbc419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 0x7fbbc419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410fe00 con 0x7fbbc41005a0 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410fc80 con 0x7fbbc4106b80 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbca932640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410ff80 con 0x7fbbc40ffc00 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc40ffc00 0x7fbbc419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbc8ecf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbc8ecf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39780/0 (socket says 172.21.15.16:39780) 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbc8ecf640 1 -- 172.21.15.16:0/337784719 learned_addr learned my addr 172.21.15.16:0/337784719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.610+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 0x7fbbc419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 -- 172.21.15.16:0/337784719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc40ffc00 msgr2=0x7fbbc419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc40ffc00 0x7fbbc419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 -- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 msgr2=0x7fbbc419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 0x7fbbc419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 -- 172.21.15.16:0/337784719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc41969f0 con 0x7fbbc4106b80 2024-09-12T03:02:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 0x7fbbc419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00dad0 tx=0x7fbbbc00df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc17fa640 1 -- 172.21.15.16:0/337784719 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc015070 con 0x7fbbc4106b80 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc4196ce0 con 0x7fbbc4106b80 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc419ad10 con 0x7fbbc4106b80 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.611+0000 7fbbc17fa640 1 -- 172.21.15.16:0/337784719 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc00be30 con 0x7fbbc4106b80 2024-09-12T03:02:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.612+0000 7fbbc17fa640 1 -- 172.21.15.16:0/337784719 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc00f420 con 0x7fbbc4106b80 2024-09-12T03:02:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.614+0000 7fbbccd0b740 1 Processor -- start 2024-09-12T03:02:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc17fa640 1 -- 172.21.15.16:0/337784719 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbbbc004a40 con 0x7fbbc4106b80 2024-09-12T03:02:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 -- start start 2024-09-12T03:02:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a551a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55632a4aa2e0 con 0x55632a544dc0 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55632a451ce0 con 0x55632a54f680 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55632a4a62a0 con 0x55632a5444a0 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34706/0 (socket says 172.21.15.16:34706) 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47830/0 (socket says 172.21.15.16:47830) 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.615+0000 7fbbc3fff640 1 -- 172.21.15.16:0/3857986691 learned_addr learned my addr 172.21.15.16:0/3857986691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a551a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 -- 172.21.15.16:0/3857986691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 msgr2=0x55632a544880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 -- 172.21.15.16:0/3857986691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 msgr2=0x55632a551a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a551a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 -- 172.21.15.16:0/3857986691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632a436440 con 0x55632a544dc0 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.616+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009c60 tx=0x7fbbb4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51191543474b4ca server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.617+0000 7fbbba7fc640 1 -- 172.21.15.16:0/3857986691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4004230 con 0x55632a544dc0 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.617+0000 7fbbba7fc640 1 -- 172.21.15.16:0/3857986691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb40043d0 con 0x55632a544dc0 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.617+0000 7fbbba7fc640 1 -- 172.21.15.16:0/3857986691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40197c0 con 0x55632a544dc0 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.617+0000 7fbbc17fa640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 0x7fbbb00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 0x7fbbb00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbc17fa640 1 -- 172.21.15.16:0/337784719 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbbbc0cebc0 con 0x7fbbc4106b80 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 -- 172.21.15.16:0/3857986691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 msgr2=0x55632a54f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009c60 tx=0x7fbbb4011810 comp rx=0 tx=0).stop 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 -- 172.21.15.16:0/3857986691 shutdown_connections 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a551a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a544dc0 0x55632a54f140 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/3857986691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a5444a0 0x55632a544880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 -- 172.21.15.16:0/3857986691 >> 172.21.15.16:0/3857986691 conn(0x55632a540a30 msgr2=0x55632a540e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 0x7fbbb00a50c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbbac000c00 tx=0x7fbbac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 -- 172.21.15.16:0/3857986691 shutdown_connections 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.618+0000 7fbbccd0b740 1 -- 172.21.15.16:0/3857986691 wait complete. 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 -- start start 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 Processor -- start 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a54ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a54c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55632a552250 con 0x55632a5444a0 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55632a465050 con 0x55632a54f680 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.619+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55632a5523d0 con 0x55632a544dc0 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a54ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34712/0 (socket says 172.21.15.16:34712) 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc37fe640 1 -- 172.21.15.16:0/785005246 learned_addr learned my addr 172.21.15.16:0/785005246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a54c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbccd0b740 10 client.0 _ll_register_callbacks cb 0x55632a4a5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 -- 172.21.15.16:0/785005246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 msgr2=0x55632a54c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 -- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 msgr2=0x55632a54c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a54c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc3fff640 1 -- 172.21.15.16:0/785005246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55632a21b3c0 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.620+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a54ef60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4011d10 tx=0x7fbbb4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b37f0a25e5110d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4027020 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbbb4003a10 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbb40043d0 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbbb4009850 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbbccd0b740 10 client.19032 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55632a55dc00 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.621+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb40043d0 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fbbb4018070 con 0x55632a5444a0 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 10 client.19032 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 10 client.19032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:58.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 20 client.19032 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a55f470 0x55632a561930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 4 client.19032 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55632a4b7850 con 0x55632a55f470 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a55f470 0x55632a561930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbba37fe640 10 client.19032 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:58.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.622+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a55f470 0x55632a561930 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c3f2167b859a7c1 server_cookie=f996249e7a278377 in_seq=0 out_seq=0 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbba37fe640 1 -- 172.21.15.16:0/785005246 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55632a4b7850 con 0x55632a55f470 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbba37fe640 10 client.19032 handle_command_reply: tid=0 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 client.19032 shutdown 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 msgr2=0x55632a54ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a54ef60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4011d10 tx=0x7fbbb4003820 comp rx=0 tx=0).stop 2024-09-12T03:02:58.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 shutdown_connections 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a55f470 0x55632a561930 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a54f680 0x55632a54c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a54c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/785005246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a54ef60 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 >> 172.21.15.16:0/785005246 conn(0x55632a540a30 msgr2=0x55632a546710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.623+0000 7fbba37fe640 0 client.19032 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.624+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 shutdown_connections 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.624+0000 7fbbccd0b740 1 -- 172.21.15.16:0/785005246 wait complete. 2024-09-12T03:02:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.624+0000 7fbbccd0b740 20 client.19032 trim_cache size 0 max 16384 2024-09-12T03:02:58.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 Processor -- start 2024-09-12T03:02:58.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 -- start start 2024-09-12T03:02:58.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 0x55632a567240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a54f680 0x55632a567780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba400e400 con 0x55632a5444a0 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbba4010000 con 0x55632a544dc0 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbba400ead0 con 0x55632a54f680 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47856/0 (socket says 172.21.15.16:47856) 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc3fff640 1 -- 172.21.15.16:0/681634242 learned_addr learned my addr 172.21.15.16:0/681634242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 0x55632a567240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a54f680 0x55632a567780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 -- 172.21.15.16:0/681634242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a54f680 msgr2=0x55632a567780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a54f680 0x55632a567780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 -- 172.21.15.16:0/681634242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 msgr2=0x55632a55b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc37fe640 1 -- 172.21.15.16:0/681634242 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632a54b1c0 con 0x55632a544dc0 2024-09-12T03:02:58.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.645+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:58.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.646+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 0x55632a567240 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55632a54d850 tx=0x7fbbac040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd7a42cf958b33cc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.646+0000 7fbbb9ffb640 1 -- 172.21.15.16:0/681634242 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003d60 con 0x55632a544dc0 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.646+0000 7fbbb9ffb640 1 -- 172.21.15.16:0/681634242 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbac058030 con 0x55632a544dc0 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.646+0000 7fbbb9ffb640 1 -- 172.21.15.16:0/681634242 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac052880 con 0x55632a544dc0 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- 172.21.15.16:0/681634242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 msgr2=0x55632a567240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 0x55632a567240 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55632a54d850 tx=0x7fbbac040000 comp rx=0 tx=0).stop 2024-09-12T03:02:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- 172.21.15.16:0/681634242 shutdown_connections 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a54f680 0x55632a567780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a544dc0 0x55632a567240 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/681634242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a5444a0 0x55632a55b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- 172.21.15.16:0/681634242 >> 172.21.15.16:0/681634242 conn(0x55632a540a30 msgr2=0x55632a5470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- 172.21.15.16:0/681634242 shutdown_connections 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- 172.21.15.16:0/681634242 wait complete. 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.647+0000 7fbbccd0b740 1 -- start start 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 Processor -- start 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 0x55632a5527a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 0x55632a5c8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba400f930 con 0x55632a54f680 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbba400f260 con 0x55632a5444a0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbba400c3e0 con 0x55632a544dc0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbccd0b740 10 client.0 _ll_register_callbacks cb 0x7fbbb4009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbc37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34736/0 (socket says 172.21.15.16:34736) 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbc37fe640 1 -- 172.21.15.16:0/1051244852 learned_addr learned my addr 172.21.15.16:0/1051244852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 0x55632a5527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.648+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 0x55632a5c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 -- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 msgr2=0x55632a5527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 0x55632a5527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 -- 172.21.15.16:0/1051244852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 msgr2=0x55632a5c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 0x55632a5c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 -- 172.21.15.16:0/1051244852 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55632a544bf0 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 0x55632a5527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc8ecf640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 0x55632a5c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbc37fe640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009850 tx=0x7fbbac0524e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed781dfd9c15297f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003de0 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbbac052600 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.649+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbac058030 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbbac052820 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 10 client.28264 fetch_fsmap learned FSMap version 35 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55632a5c9760 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbac058030 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 35) ==== 8394+0+0 (secure 0 0 0) 0x7fbbac06f020 con 0x55632a544dc0 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 10 client.28264 fetch_fsmap finished waiting for FSMap version 35 2024-09-12T03:02:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 10 client.28264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:02:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 20 client.28264 populate_metadata read hostname 'smithi016' 2024-09-12T03:02:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a5c9f00 0x55632a5cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:02:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 4 client.28264 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:02:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55632a4b1d60 con 0x55632a5c9f00 2024-09-12T03:02:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.650+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a5c9f00 0x55632a5cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:02:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbba37fe640 10 client.28264 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbc3fff640 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a5c9f00 0x55632a5cc320 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49c17042ce5dccf9 server_cookie=b9f3c2777cfc28f9 in_seq=0 out_seq=0 2024-09-12T03:02:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbba37fe640 1 -- 172.21.15.16:0/1051244852 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55632a4b1d60 con 0x55632a5c9f00 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbba37fe640 10 client.28264 handle_command_reply: tid=0 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 client.28264 shutdown 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 msgr2=0x55632a552ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009850 tx=0x7fbbac0524e0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 shutdown_connections 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55632a5c9f00 0x55632a5cc320 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55632a54f680 0x55632a5c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55632a544dc0 0x55632a552ce0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 --2- 172.21.15.16:0/1051244852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55632a5444a0 0x55632a5527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 >> 172.21.15.16:0/1051244852 conn(0x55632a540a30 msgr2=0x55632a53f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbba37fe640 0 client.28264 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 shutdown_connections 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 1 -- 172.21.15.16:0/1051244852 wait complete. 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.651+0000 7fbbccd0b740 20 client.28264 trim_cache size 0 max 16384 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stdout: "0c09c20a-553b-492a-9204-672562f7ebb9": { 2024-09-12T03:02:58.658 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:02:58.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "0c09c20a-553b-492a-9204-672562f7ebb9", 2024-09-12T03:02:58.659 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:02:58.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:58.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:02:58.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 msgr2=0x7fbbb00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 0x7fbbb00a50c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbbac000c00 tx=0x7fbbac009260 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 msgr2=0x7fbbc4196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00dad0 tx=0x7fbbbc00df90 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 shutdown_connections 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc4106b80 0x7fbbc4196130 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbbb00a2c00 0x7fbbb00a50c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc41005a0 0x7fbbc419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 --2- 172.21.15.16:0/337784719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbc40ffc00 0x7fbbc419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:02:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.652+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 >> 172.21.15.16:0/337784719 conn(0x7fbbc40fba90 msgr2=0x7fbbc40fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:02:58.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.653+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 shutdown_connections 2024-09-12T03:02:58.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:02:58.653+0000 7fbbca932640 1 -- 172.21.15.16:0/337784719 wait complete. 2024-09-12T03:02:59.533 INFO:tasks.cephfs.filesystem:scrub status for tag:0c09c20a-553b-492a-9204-672562f7ebb9 - {'path': '/', 'tag': '0c09c20a-553b-492a-9204-672562f7ebb9', 'options': 'recursive,force'} 2024-09-12T03:03:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 41238 34.22 MB/sec execute 1 sec latency 69.638 ms 2024-09-12T03:03:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:02:59 smithi016 ceph-mon[30925]: pgmap v789: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:03:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:02:59 smithi033 ceph-mon[35092]: pgmap v789: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:03:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:02:59 smithi184 ceph-mon[35321]: pgmap v789: 97 pgs: 97 active+clean; 552 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:03:01.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 41853 33.08 MB/sec execute 2 sec latency 56.007 ms 2024-09-12T03:03:02.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 42708 35.53 MB/sec execute 3 sec latency 34.472 ms 2024-09-12T03:03:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:01 smithi016 ceph-mon[30925]: pgmap v790: 97 pgs: 97 active+clean; 536 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 747 op/s 2024-09-12T03:03:02.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:01 smithi033 ceph-mon[35092]: pgmap v790: 97 pgs: 97 active+clean; 536 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 747 op/s 2024-09-12T03:03:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:01 smithi184 ceph-mon[35321]: pgmap v790: 97 pgs: 97 active+clean; 536 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 747 op/s 2024-09-12T03:03:03.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43249 34.68 MB/sec execute 4 sec latency 184.344 ms 2024-09-12T03:03:04.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44029 36.52 MB/sec execute 5 sec latency 55.202 ms 2024-09-12T03:03:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:03 smithi016 ceph-mon[30925]: pgmap v791: 97 pgs: 97 active+clean; 535 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 708 op/s 2024-09-12T03:03:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:03 smithi033 ceph-mon[35092]: pgmap v791: 97 pgs: 97 active+clean; 535 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 708 op/s 2024-09-12T03:03:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:03 smithi184 ceph-mon[35321]: pgmap v791: 97 pgs: 97 active+clean; 535 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 708 op/s 2024-09-12T03:03:05.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44760 37.43 MB/sec execute 6 sec latency 77.959 ms 2024-09-12T03:03:06.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 45407 37.90 MB/sec execute 7 sec latency 37.760 ms 2024-09-12T03:03:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:05 smithi016 ceph-mon[30925]: pgmap v792: 97 pgs: 97 active+clean; 540 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 697 op/s 2024-09-12T03:03:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:05 smithi033 ceph-mon[35092]: pgmap v792: 97 pgs: 97 active+clean; 540 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 697 op/s 2024-09-12T03:03:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:05 smithi184 ceph-mon[35321]: pgmap v792: 97 pgs: 97 active+clean; 540 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 697 op/s 2024-09-12T03:03:07.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46004 36.31 MB/sec execute 8 sec latency 133.362 ms 2024-09-12T03:03:08.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46439 34.78 MB/sec execute 9 sec latency 238.573 ms 2024-09-12T03:03:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:07 smithi016 ceph-mon[30925]: pgmap v793: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:03:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:07 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:07 smithi033 ceph-mon[35092]: pgmap v793: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:03:08.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:07 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:07 smithi184 ceph-mon[35321]: pgmap v793: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:03:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:07 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:09.009 INFO:tasks.workunit.client.0.smithi016.stdout: 10 47016 34.17 MB/sec execute 10 sec latency 90.540 ms 2024-09-12T03:03:09.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:08 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:03:09.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:08 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:03:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:08 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:03:10.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 47756 34.71 MB/sec execute 11 sec latency 51.646 ms 2024-09-12T03:03:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:09 smithi016 ceph-mon[30925]: pgmap v794: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 627 op/s 2024-09-12T03:03:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:03:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:03:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:03:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:09 smithi033 ceph-mon[35092]: pgmap v794: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 627 op/s 2024-09-12T03:03:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:03:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:03:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:03:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:09 smithi184 ceph-mon[35321]: pgmap v794: 97 pgs: 97 active+clean; 543 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 627 op/s 2024-09-12T03:03:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:03:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:03:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:03:11.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 48568 36.01 MB/sec execute 12 sec latency 73.854 ms 2024-09-12T03:03:12.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 49239 35.79 MB/sec execute 13 sec latency 72.958 ms 2024-09-12T03:03:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:11 smithi016 ceph-mon[30925]: pgmap v795: 97 pgs: 97 active+clean; 553 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 771 op/s 2024-09-12T03:03:12.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:11 smithi033 ceph-mon[35092]: pgmap v795: 97 pgs: 97 active+clean; 553 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 771 op/s 2024-09-12T03:03:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:11 smithi184 ceph-mon[35321]: pgmap v795: 97 pgs: 97 active+clean; 553 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 771 op/s 2024-09-12T03:03:13.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 49700 34.98 MB/sec execute 14 sec latency 212.735 ms 2024-09-12T03:03:14.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 50224 34.43 MB/sec execute 15 sec latency 160.975 ms 2024-09-12T03:03:14.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:13 smithi016 ceph-mon[30925]: pgmap v796: 97 pgs: 97 active+clean; 556 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 696 op/s 2024-09-12T03:03:14.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:13 smithi033 ceph-mon[35092]: pgmap v796: 97 pgs: 97 active+clean; 556 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 696 op/s 2024-09-12T03:03:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:13 smithi184 ceph-mon[35321]: pgmap v796: 97 pgs: 97 active+clean; 556 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 696 op/s 2024-09-12T03:03:15.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 51045 35.09 MB/sec execute 16 sec latency 43.351 ms 2024-09-12T03:03:16.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 51787 35.38 MB/sec execute 17 sec latency 59.935 ms 2024-09-12T03:03:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:15 smithi016 ceph-mon[30925]: pgmap v797: 97 pgs: 97 active+clean; 559 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.7 MiB/s wr, 622 op/s 2024-09-12T03:03:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:15 smithi033 ceph-mon[35092]: pgmap v797: 97 pgs: 97 active+clean; 559 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.7 MiB/s wr, 622 op/s 2024-09-12T03:03:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:15 smithi184 ceph-mon[35321]: pgmap v797: 97 pgs: 97 active+clean; 559 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.7 MiB/s wr, 622 op/s 2024-09-12T03:03:17.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 52501 35.40 MB/sec execute 18 sec latency 69.930 ms 2024-09-12T03:03:18.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53109 35.30 MB/sec execute 19 sec latency 243.562 ms 2024-09-12T03:03:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:17 smithi016 ceph-mon[30925]: pgmap v798: 97 pgs: 97 active+clean; 567 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:03:18.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:17 smithi033 ceph-mon[35092]: pgmap v798: 97 pgs: 97 active+clean; 567 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:03:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:17 smithi184 ceph-mon[35321]: pgmap v798: 97 pgs: 97 active+clean; 567 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:03:19.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53668 35.43 MB/sec execute 20 sec latency 71.475 ms 2024-09-12T03:03:20.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54310 35.15 MB/sec execute 21 sec latency 49.715 ms 2024-09-12T03:03:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:19 smithi016 ceph-mon[30925]: pgmap v799: 97 pgs: 97 active+clean; 566 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.3 MiB/s wr, 636 op/s 2024-09-12T03:03:20.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:19 smithi033 ceph-mon[35092]: pgmap v799: 97 pgs: 97 active+clean; 566 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.3 MiB/s wr, 636 op/s 2024-09-12T03:03:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:19 smithi184 ceph-mon[35321]: pgmap v799: 97 pgs: 97 active+clean; 566 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.3 MiB/s wr, 636 op/s 2024-09-12T03:03:21.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54944 34.90 MB/sec execute 22 sec latency 59.769 ms 2024-09-12T03:03:22.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 55723 35.32 MB/sec execute 23 sec latency 53.601 ms 2024-09-12T03:03:22.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:21 smithi016 ceph-mon[30925]: pgmap v800: 97 pgs: 97 active+clean; 568 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-09-12T03:03:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:21 smithi033 ceph-mon[35092]: pgmap v800: 97 pgs: 97 active+clean; 568 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-09-12T03:03:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:21 smithi184 ceph-mon[35321]: pgmap v800: 97 pgs: 97 active+clean; 568 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 779 op/s 2024-09-12T03:03:23.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 56288 35.06 MB/sec execute 24 sec latency 149.382 ms 2024-09-12T03:03:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 56881 35.03 MB/sec execute 25 sec latency 143.135 ms 2024-09-12T03:03:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:23 smithi016 ceph-mon[30925]: pgmap v801: 97 pgs: 97 active+clean; 553 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 669 op/s 2024-09-12T03:03:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:23 smithi033 ceph-mon[35092]: pgmap v801: 97 pgs: 97 active+clean; 553 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 669 op/s 2024-09-12T03:03:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:23 smithi184 ceph-mon[35321]: pgmap v801: 97 pgs: 97 active+clean; 553 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 669 op/s 2024-09-12T03:03:25.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 57631 35.46 MB/sec execute 26 sec latency 62.242 ms 2024-09-12T03:03:26.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58252 35.43 MB/sec execute 27 sec latency 48.938 ms 2024-09-12T03:03:26.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:25 smithi016 ceph-mon[30925]: pgmap v802: 97 pgs: 97 active+clean; 557 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 620 op/s 2024-09-12T03:03:26.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:25 smithi033 ceph-mon[35092]: pgmap v802: 97 pgs: 97 active+clean; 557 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 620 op/s 2024-09-12T03:03:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:25 smithi184 ceph-mon[35321]: pgmap v802: 97 pgs: 97 active+clean; 557 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 620 op/s 2024-09-12T03:03:27.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58932 35.27 MB/sec execute 28 sec latency 56.135 ms 2024-09-12T03:03:28.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 59284 34.63 MB/sec execute 29 sec latency 304.846 ms 2024-09-12T03:03:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:27 smithi016 ceph-mon[30925]: pgmap v803: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 761 op/s 2024-09-12T03:03:28.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:27 smithi033 ceph-mon[35092]: pgmap v803: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 761 op/s 2024-09-12T03:03:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:27 smithi184 ceph-mon[35321]: pgmap v803: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 761 op/s 2024-09-12T03:03:29.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 59770 34.36 MB/sec execute 30 sec latency 197.185 ms 2024-09-12T03:03:29.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:03:29.842 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:30.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 60588 34.47 MB/sec execute 31 sec latency 41.960 ms 2024-09-12T03:03:30.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:29 smithi016 ceph-mon[30925]: pgmap v804: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 647 op/s 2024-09-12T03:03:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:29 smithi033 ceph-mon[35092]: pgmap v804: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 647 op/s 2024-09-12T03:03:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:29 smithi184 ceph-mon[35321]: pgmap v804: 97 pgs: 97 active+clean; 568 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 647 op/s 2024-09-12T03:03:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 -- 172.21.15.16:0/2200248052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c103ff0 msgr2=0x7f617c1043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 --2- 172.21.15.16:0/2200248052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c103ff0 0x7f617c1043d0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6170009900 tx=0x7f617002a3f0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 -- 172.21.15.16:0/2200248052 shutdown_connections 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 --2- 172.21.15.16:0/2200248052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 --2- 172.21.15.16:0/2200248052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c1049a0 0x7f617c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 --2- 172.21.15.16:0/2200248052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c103ff0 0x7f617c1043d0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.373+0000 7f6182da4640 1 -- 172.21.15.16:0/2200248052 >> 172.21.15.16:0/2200248052 conn(0x7f617c0ffab0 msgr2=0x7f617c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.374+0000 7f6182da4640 1 -- 172.21.15.16:0/2200248052 shutdown_connections 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.374+0000 7f6182da4640 1 -- 172.21.15.16:0/2200248052 wait complete. 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.374+0000 7f6182da4640 1 Processor -- start 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.374+0000 7f6182da4640 1 -- start start 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c103ff0 0x7f617c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 0x7f617c19ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f617c114120 con 0x7f617c1049a0 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f617c113fa0 con 0x7f617c103ff0 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6182da4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1142a0 con 0x7f617c109120 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f617bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 0x7f617c19ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f617bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 0x7f617c19ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54066/0 (socket says 172.21.15.16:54066) 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f617bfff640 1 -- 172.21.15.16:0/2042722896 learned_addr learned my addr 172.21.15.16:0/2042722896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6180b40640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c103ff0 0x7f617c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 -- 172.21.15.16:0/2042722896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c103ff0 msgr2=0x7f617c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c103ff0 0x7f617c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 -- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 msgr2=0x7f617c19ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 0x7f617c19ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.375+0000 7f6181341640 1 -- 172.21.15.16:0/2042722896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c19f690 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6181341640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c19efb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6174009fd0 tx=0x7f617400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6179ffb640 1 -- 172.21.15.16:0/2042722896 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617400ce20 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6179ffb640 1 -- 172.21.15.16:0/2042722896 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6174005020 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c19f980 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6179ffb640 1 -- 172.21.15.16:0/2042722896 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174010720 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.376+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c1a7260 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f6179ffb640 1 -- 172.21.15.16:0/2042722896 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6174005540 con 0x7f617c109120 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 Processor -- start 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 -- start start 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ed140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7ed680 0x5625fe7efa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625fe7482e0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625fe6efce0 con 0x5625fe7ed680 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f618517d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5625fe7442a0 con 0x5625fe7e24a0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f617bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f617bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44634/0 (socket says 172.21.15.16:44634) 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.379+0000 7f617bfff640 1 -- 172.21.15.16:0/116269318 learned_addr learned my addr 172.21.15.16:0/116269318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7ed680 0x5625fe7efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f6180b40640 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 -- 172.21.15.16:0/116269318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7ed680 msgr2=0x5625fe7efa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7ed680 0x5625fe7efa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 -- 172.21.15.16:0/116269318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e2dc0 msgr2=0x5625fe7ed140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ed140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 -- 172.21.15.16:0/116269318 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625fe6d4440 con 0x5625fe7e24a0 2024-09-12T03:03:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f6179ffb640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 0x7f616c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f617bfff640 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f616800bfd0 tx=0x7f616800b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e095f3b3a71104e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f6179ffb640 1 -- 172.21.15.16:0/2042722896 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f61740cf6b0 con 0x7f617c109120 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f6180b40640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 0x7f616c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.380+0000 7f6166ffd640 1 -- 172.21.15.16:0/116269318 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168012d80 con 0x5625fe7e24a0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f6166ffd640 1 -- 172.21.15.16:0/116269318 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6168010770 con 0x5625fe7e24a0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f6166ffd640 1 -- 172.21.15.16:0/116269318 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168003850 con 0x5625fe7e24a0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 -- 172.21.15.16:0/116269318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 msgr2=0x5625fe7e2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f616800bfd0 tx=0x7f616800b830 comp rx=0 tx=0).stop 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f6180b40640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 0x7f616c0a5040 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6170009230 tx=0x7f61700058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 -- 172.21.15.16:0/116269318 shutdown_connections 2024-09-12T03:03:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7ed680 0x5625fe7efa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.381+0000 7f618517d740 1 --2- 172.21.15.16:0/116269318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e2880 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.382+0000 7f618517d740 1 -- 172.21.15.16:0/116269318 >> 172.21.15.16:0/116269318 conn(0x5625fe7dea30 msgr2=0x5625fe7dee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.382+0000 7f618517d740 1 -- 172.21.15.16:0/116269318 shutdown_connections 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.382+0000 7f618517d740 1 -- 172.21.15.16:0/116269318 wait complete. 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.382+0000 7f618517d740 1 -- start start 2024-09-12T03:03:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.382+0000 7f618517d740 1 Processor -- start 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 0x5625fe7ed040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7ed680 0x5625fe7ea8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625fe7f0250 con 0x5625fe7e24a0 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625fe703050 con 0x5625fe7e2dc0 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5625fe7f03d0 con 0x5625fe7ed680 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f6180b40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 0x5625fe7ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f6180b40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 0x5625fe7ed040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54072/0 (socket says 172.21.15.16:54072) 2024-09-12T03:03:30.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f6180b40640 1 -- 172.21.15.16:0/1574495186 learned_addr learned my addr 172.21.15.16:0/1574495186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f6181341640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7ed680 0x5625fe7ea8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f618517d740 10 client.0 _ll_register_callbacks cb 0x5625fe743160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.383+0000 7f617bfff640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 -- 172.21.15.16:0/1574495186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7ed680 msgr2=0x5625fe7ea8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7ed680 0x5625fe7ea8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 -- 172.21.15.16:0/1574495186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 msgr2=0x5625fe7ed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 0x5625fe7ed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 -- 172.21.15.16:0/1574495186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625fe4b93c0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f617bfff640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ea2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6168010740 tx=0x7f616800b570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d4cf01ad71c9f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f615bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:03:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168010b20 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6168010cc0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.384+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6168010770 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6168005040 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f618517d740 10 client.28364 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5625fe7fbc70 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6168010770 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f61680035f0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f618517d740 10 client.28364 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.385+0000 7f618517d740 10 client.28364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f618517d740 20 client.28364 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe7fd0d0 0x5625fe7ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f618517d740 4 client.28364 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625fe755850 con 0x5625fe7fd0d0 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f6180b40640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe7fd0d0 0x5625fe7ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f615bfff640 10 client.28364 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.386+0000 7f6180b40640 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe7fd0d0 0x5625fe7ff4c0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5f65670278bf42 server_cookie=c914fc215d1964ef in_seq=0 out_seq=0 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.387+0000 7f615bfff640 1 -- 172.21.15.16:0/1574495186 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625fe755850 con 0x5625fe7fd0d0 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.387+0000 7f615bfff640 10 client.28364 handle_command_reply: tid=0 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.387+0000 7f618517d740 1 client.28364 shutdown 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 msgr2=0x5625fe7ea2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ea2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6168010740 tx=0x7f616800b570 comp rx=0 tx=0).stop 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 shutdown_connections 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe7fd0d0 0x5625fe7ff4c0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7ed680 0x5625fe7ea8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ea2f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f615bfff640 0 client.28364 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 --2- 172.21.15.16:0/1574495186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7e24a0 0x5625fe7ed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 >> 172.21.15.16:0/1574495186 conn(0x5625fe7dea30 msgr2=0x5625fe7e4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 shutdown_connections 2024-09-12T03:03:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.388+0000 7f618517d740 1 -- 172.21.15.16:0/1574495186 wait complete. 2024-09-12T03:03:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.389+0000 7f618517d740 20 client.28364 trim_cache size 0 max 16384 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 Processor -- start 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 -- start start 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1af6e0 con 0x5625fe7ed680 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1b12e0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f618517d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1afdb0 con 0x5625fe7e24a0 2024-09-12T03:03:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6181341640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6181341640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54074/0 (socket says 172.21.15.16:54074) 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6181341640 1 -- 172.21.15.16:0/2073777341 learned_addr learned my addr 172.21.15.16:0/2073777341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44650/0 (socket says 172.21.15.16:44650) 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f617bfff640 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 -- 172.21.15.16:0/2073777341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 msgr2=0x5625fe7ec260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ec260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 -- 172.21.15.16:0/2073777341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 msgr2=0x5625fe804c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6180b40640 1 -- 172.21.15.16:0/2073777341 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625fe7ed3b0 con 0x5625fe7e24a0 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.412+0000 7f6181341640 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.413+0000 7f6180b40640 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5625fe7ffb70 tx=0x7f617003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1bee66a311a2fdd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.413+0000 7f61667fc640 1 -- 172.21.15.16:0/2073777341 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170035040 con 0x5625fe7e24a0 2024-09-12T03:03:30.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.413+0000 7f61667fc640 1 -- 172.21.15.16:0/2073777341 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6170003b90 con 0x5625fe7e24a0 2024-09-12T03:03:30.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.413+0000 7f61667fc640 1 -- 172.21.15.16:0/2073777341 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170053c80 con 0x5625fe7e24a0 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- 172.21.15.16:0/2073777341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 msgr2=0x5625fe7e9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5625fe7ffb70 tx=0x7f617003e000 comp rx=0 tx=0).stop 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- 172.21.15.16:0/2073777341 shutdown_connections 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe804c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7ec260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 --2- 172.21.15.16:0/2073777341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7e9bb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- 172.21.15.16:0/2073777341 >> 172.21.15.16:0/2073777341 conn(0x5625fe7dea30 msgr2=0x5625fe7ebb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- 172.21.15.16:0/2073777341 shutdown_connections 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- 172.21.15.16:0/2073777341 wait complete. 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 -- start start 2024-09-12T03:03:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.414+0000 7f618517d740 1 Processor -- start 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe7f3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1b0c10 con 0x5625fe7ed680 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1b0540 con 0x5625fe7e2dc0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1ad6c0 con 0x5625fe7e24a0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f618517d740 10 client.0 _ll_register_callbacks cb 0x7f616800bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f6181341640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe7f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39202/0 (socket says 172.21.15.16:39202) 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 -- 172.21.15.16:0/1326773465 learned_addr learned my addr 172.21.15.16:0/1326773465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f6180b40640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 -- 172.21.15.16:0/1326773465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 msgr2=0x5625fe7f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 -- 172.21.15.16:0/1326773465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 msgr2=0x5625fe7f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe7f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.415+0000 7f617bfff640 1 -- 172.21.15.16:0/1326773465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625fe7f3e40 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f617bfff640 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5625fe7eb930 tx=0x7f6168012730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e785b58fd892a999 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f615bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168003dc0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6170003b90 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f61680051e0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6168044050 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f618517d740 10 client.28370 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5625fe867c20 con 0x5625fe7e2dc0 2024-09-12T03:03:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.416+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6170003b90 con 0x5625fe7e2dc0 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f61680044e0 con 0x5625fe7e2dc0 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 10 client.28370 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 10 client.28370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 20 client.28370 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe8682e0 0x5625fe86a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 4 client.28370 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f6180b40640 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe8682e0 0x5625fe86a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.417+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625fe74fd60 con 0x5625fe8682e0 2024-09-12T03:03:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.418+0000 7f6180b40640 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe8682e0 0x5625fe86a700 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6264af37f6c5fb5c server_cookie=53a963475cc5a595 in_seq=0 out_seq=0 2024-09-12T03:03:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.418+0000 7f615bfff640 10 client.28370 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f615bfff640 1 -- 172.21.15.16:0/1326773465 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5625fe74fd60 con 0x5625fe8682e0 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f615bfff640 10 client.28370 handle_command_reply: tid=0 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 client.28370 shutdown 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 msgr2=0x5625fe7f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5625fe7eb930 tx=0x7f6168012730 comp rx=0 tx=0).stop 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 shutdown_connections 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5625fe8682e0 0x5625fe86a700 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5625fe7ed680 0x5625fe7f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f615bfff640 0 client.28370 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625fe7e2dc0 0x5625fe7f3100 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 --2- 172.21.15.16:0/1326773465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5625fe7e24a0 0x5625fe7f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 >> 172.21.15.16:0/1326773465 conn(0x5625fe7dea30 msgr2=0x5625fe7ec370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 shutdown_connections 2024-09-12T03:03:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.419+0000 7f618517d740 1 -- 172.21.15.16:0/1326773465 wait complete. 2024-09-12T03:03:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f618517d740 20 client.28370 trim_cache size 0 max 16384 2024-09-12T03:03:30.424 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:03:30.424 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:03:30.424 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:03:30.424 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 msgr2=0x7f616c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 0x7f616c0a5040 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6170009230 tx=0x7f61700058d0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 msgr2=0x7f617c19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c19efb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6174009fd0 tx=0x7f617400ef10 comp rx=0 tx=0).stop 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 shutdown_connections 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c109120 0x7f617c19efb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f616c0a2b80 0x7f616c0a5040 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c1049a0 0x7f617c19ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 --2- 172.21.15.16:0/2042722896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f617c103ff0 0x7f617c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 >> 172.21.15.16:0/2042722896 conn(0x7f617c0ffab0 msgr2=0x7f617c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 shutdown_connections 2024-09-12T03:03:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:30.420+0000 7f6182da4640 1 -- 172.21.15.16:0/2042722896 wait complete. 2024-09-12T03:03:31.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 61504 35.13 MB/sec execute 32 sec latency 56.060 ms 2024-09-12T03:03:31.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:30 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:31.241 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:03:31.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:03:31.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:30 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:31.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:30 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:32.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62208 35.57 MB/sec execute 33 sec latency 66.413 ms 2024-09-12T03:03:32.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:31 smithi016 ceph-mon[30925]: pgmap v805: 97 pgs: 97 active+clean; 573 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 750 op/s 2024-09-12T03:03:32.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:31 smithi033 ceph-mon[35092]: pgmap v805: 97 pgs: 97 active+clean; 573 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 750 op/s 2024-09-12T03:03:32.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/1035500277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8102570 msgr2=0x7f6eb81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:32.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/1035500277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8102570 0x7f6eb81029b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009900 tx=0x7f6ea002a480 comp rx=0 tx=0).stop 2024-09-12T03:03:32.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/1035500277 shutdown_connections 2024-09-12T03:03:32.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/1035500277 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 0x7f6eb810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/1035500277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8102570 0x7f6eb81029b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/1035500277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8100e20 0x7f6eb8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/1035500277 >> 172.21.15.16:0/1035500277 conn(0x7f6eb80fc7d0 msgr2=0x7f6eb80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/1035500277 shutdown_connections 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.155+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/1035500277 wait complete. 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 Processor -- start 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 -- start start 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8102570 0x7f6eb81976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 0x7f6eb8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb8110f80 con 0x7f6eb8100e20 2024-09-12T03:03:32.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb8110e00 con 0x7f6eb8102ef0 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6ebdaa6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb8111100 con 0x7f6eb8102570 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6eaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6eaf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54100/0 (socket says 172.21.15.16:54100) 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6eaf7fe640 1 -- 172.21.15.16:0/266229156 learned_addr learned my addr 172.21.15.16:0/266229156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6eb77fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8102570 0x7f6eb81976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.156+0000 7f6eb7fff640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 0x7f6eb8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 -- 172.21.15.16:0/266229156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8102570 msgr2=0x7f6eb81976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8102570 0x7f6eb81976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 -- 172.21.15.16:0/266229156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 msgr2=0x7f6eb8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 0x7f6eb8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 -- 172.21.15.16:0/266229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb81a69f0 con 0x7f6eb8100e20 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eaf7fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e9f0 tx=0x7f6ea800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea800cd80 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb81a6ce0 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8005500 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8018630 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.157+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb81a71a0 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.158+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb8101200 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.159+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6ea80046c0 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.160+0000 7f6eb57fa640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 0x7f6ea40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.160+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6ea80cf510 con 0x7f6eb8100e20 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.160+0000 7f6eb77fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 0x7f6ea40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.161+0000 7f6eb77fe640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 0x7f6ea40a5040 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009230 tx=0x7f6ea00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.165+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6ea809c480 con 0x7f6eb8100e20 2024-09-12T03:03:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:31 smithi184 ceph-mon[35321]: pgmap v805: 97 pgs: 97 active+clean; 573 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 750 op/s 2024-09-12T03:03:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.413+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eb8108b00 con 0x7f6eb8100e20 2024-09-12T03:03:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.414+0000 7f6eb57fa640 1 -- 172.21.15.16:0/266229156 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11977 (secure 0 0 0) 0x7f6ea809bd80 con 0x7f6eb8100e20 2024-09-12T03:03:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:03:32.415 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"btime":"2024-09-12T03:03:29:858513+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:28.964408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14595,24443,24445,24467,24500]},"id":1}]} 2024-09-12T03:03:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 msgr2=0x7f6ea40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 0x7f6ea40a5040 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009230 tx=0x7f6ea00058d0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 msgr2=0x7f6eb8197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e9f0 tx=0x7f6ea800edf0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 shutdown_connections 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb8102ef0 0x7f6eb8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eb8102570 0x7f6eb81976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6ea40a2b80 0x7f6ea40a5040 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 --2- 172.21.15.16:0/266229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb8100e20 0x7f6eb8197180 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 >> 172.21.15.16:0/266229156 conn(0x7f6eb80fc7d0 msgr2=0x7f6eb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 shutdown_connections 2024-09-12T03:03:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:32.416+0000 7f6ebdaa6640 1 -- 172.21.15.16:0/266229156 wait complete. 2024-09-12T03:03:33.010 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62707 35.19 MB/sec execute 34 sec latency 65.153 ms 2024-09-12T03:03:33.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/266229156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:03:33.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/266229156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:03:33.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/266229156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:03:33.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:03:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:34.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63191 34.79 MB/sec execute 35 sec latency 203.310 ms 2024-09-12T03:03:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:33 smithi016 ceph-mon[30925]: pgmap v806: 97 pgs: 97 active+clean; 577 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 661 op/s 2024-09-12T03:03:34.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:33 smithi033 ceph-mon[35092]: pgmap v806: 97 pgs: 97 active+clean; 577 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 661 op/s 2024-09-12T03:03:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:33 smithi184 ceph-mon[35321]: pgmap v806: 97 pgs: 97 active+clean; 577 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 661 op/s 2024-09-12T03:03:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 -- 172.21.15.16:0/147725891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0101430 msgr2=0x7f40d010d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 --2- 172.21.15.16:0/147725891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0101430 0x7f40d010d960 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009cd0 tx=0x7f40c802a930 comp rx=0 tx=0).stop 2024-09-12T03:03:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 -- 172.21.15.16:0/147725891 shutdown_connections 2024-09-12T03:03:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 --2- 172.21.15.16:0/147725891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0101430 0x7f40d010d960 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 --2- 172.21.15.16:0/147725891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0100a90 0x7f40d0100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 --2- 172.21.15.16:0/147725891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0106a90 0x7f40d0106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 -- 172.21.15.16:0/147725891 >> 172.21.15.16:0/147725891 conn(0x7f40d00fc7d0 msgr2=0x7f40d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 -- 172.21.15.16:0/147725891 shutdown_connections 2024-09-12T03:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.298+0000 7f40d5a43640 1 -- 172.21.15.16:0/147725891 wait complete. 2024-09-12T03:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.299+0000 7f40d5a43640 1 Processor -- start 2024-09-12T03:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 -- start start 2024-09-12T03:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 0x7f40d00fce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0110ee0 con 0x7f40d0106a90 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0110d60 con 0x7f40d0100a90 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40d5a43640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0111060 con 0x7f40d0101430 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44696/0 (socket says 172.21.15.16:44696) 2024-09-12T03:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54124/0 (socket says 172.21.15.16:54124) 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.300+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1905094697 learned_addr learned my addr 172.21.15.16:0/1905094697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 -- 172.21.15.16:0/1905094697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 msgr2=0x7f40d00fd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 -- 172.21.15.16:0/1905094697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 msgr2=0x7f40d00fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 0x7f40d00fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 0x7f40d00fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 -- 172.21.15.16:0/1905094697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d01a68c0 con 0x7f40d0106a90 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 0x7f40d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40cf7fe640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f40c802ae50 tx=0x7f40c8031720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40d4a41640 1 -- 172.21.15.16:0/1905094697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c802fa50 con 0x7f40d0106a90 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40d4a41640 1 -- 172.21.15.16:0/1905094697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c802d070 con 0x7f40d0106a90 2024-09-12T03:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40d4a41640 1 -- 172.21.15.16:0/1905094697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c80466e0 con 0x7f40d0106a90 2024-09-12T03:03:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.301+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d01a6b50 con 0x7f40d0106a90 2024-09-12T03:03:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.302+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d01a7010 con 0x7f40d0106a90 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.304+0000 7f40d4a41640 1 -- 172.21.15.16:0/1905094697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f40c802fd80 con 0x7f40d0106a90 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 Processor -- start 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 -- start start 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b5da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b6a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561493ac32f0 con 0x561493b5d5d0 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561493a6acf0 con 0x561493b68370 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561493abf2b0 con 0x561493b5eb40 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b5da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44702/0 (socket says 172.21.15.16:44702) 2024-09-12T03:03:34.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40ce7fc640 1 -- 172.21.15.16:0/3013676127 learned_addr learned my addr 172.21.15.16:0/3013676127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d4a41640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 0x7f40b80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b6a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.305+0000 7f40d4a41640 1 -- 172.21.15.16:0/1905094697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f40c80f01e0 con 0x7f40d0106a90 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 -- 172.21.15.16:0/3013676127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 msgr2=0x561493b6a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b6a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 -- 172.21.15.16:0/3013676127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 msgr2=0x561493b5da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b5da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40cf7fe640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 0x7f40b80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 -- 172.21.15.16:0/3013676127 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561493a4f3a0 con 0x561493b5eb40 2024-09-12T03:03:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ceffd640 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009fd0 tx=0x7f40c4009d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f720df5d41e4333 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40cf7fe640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 0x7f40b80a50c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f40c8030c10 tx=0x7f40c802a930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40b5ffb640 1 -- 172.21.15.16:0/3013676127 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c400f940 con 0x561493b5eb40 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40b5ffb640 1 -- 172.21.15.16:0/3013676127 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c400e3b0 con 0x561493b5eb40 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.306+0000 7f40b5ffb640 1 -- 172.21.15.16:0/3013676127 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c400d5a0 con 0x561493b5eb40 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 -- 172.21.15.16:0/3013676127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 msgr2=0x561493b5d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009fd0 tx=0x7f40c4009d10 comp rx=0 tx=0).stop 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 -- 172.21.15.16:0/3013676127 shutdown_connections 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b6a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/3013676127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b5d090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 -- 172.21.15.16:0/3013676127 >> 172.21.15.16:0/3013676127 conn(0x561493b26310 msgr2=0x561493b59c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 -- 172.21.15.16:0/3013676127 shutdown_connections 2024-09-12T03:03:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.307+0000 7f40d7e1c740 1 -- 172.21.15.16:0/3013676127 wait complete. 2024-09-12T03:03:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.308+0000 7f40d7e1c740 1 -- start start 2024-09-12T03:03:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.308+0000 7f40d7e1c740 1 Processor -- start 2024-09-12T03:03:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.308+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.308+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b65020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.308+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b65610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561493b6b070 con 0x561493b5d5d0 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561493a7e060 con 0x561493b68370 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561493b6b1f0 con 0x561493b5eb40 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54136/0 (socket says 172.21.15.16:54136) 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1907979735 learned_addr learned my addr 172.21.15.16:0/1907979735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40d7e1c740 10 client.0 _ll_register_callbacks cb 0x5614937d18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b65020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b65610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 -- 172.21.15.16:0/1907979735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 msgr2=0x561493b65610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b65610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 -- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 msgr2=0x561493b67d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 -- 172.21.15.16:0/1907979735 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614938343c0 con 0x561493b5eb40 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b65020 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009d90 tx=0x7f40c0023830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14fc2f601d2428ed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.309+0000 7f40aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0005020 con 0x561493b5eb40 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f40c00051c0 con 0x561493b5eb40 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c400e3b0 con 0x561493b5eb40 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f40c00053c0 con 0x561493b5eb40 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40d7e1c740 10 client.28300 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561493b76b80 con 0x561493b5eb40 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.310+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c0005560 con 0x561493b5eb40 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f40c000da50 con 0x561493b5eb40 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 10 client.28300 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 10 client.28300 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 20 client.28300 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493b78090 0x561493b7a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 4 client.28300 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40c400e3b0 con 0x561493b78090 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.311+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493b78090 0x561493b7a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.312+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493b78090 0x561493b7a4b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=83c9c122d628ce32 server_cookie=67ae7d3e4288a084 in_seq=0 out_seq=0 2024-09-12T03:03:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.312+0000 7f40aeffd640 10 client.28300 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.312+0000 7f40aeffd640 1 -- 172.21.15.16:0/1907979735 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f40c400e3b0 con 0x561493b78090 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.312+0000 7f40aeffd640 10 client.28300 handle_command_reply: tid=0 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.312+0000 7f40d7e1c740 1 client.28300 shutdown 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 msgr2=0x561493b65020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b65020 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009d90 tx=0x7f40c0023830 comp rx=0 tx=0).stop 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 shutdown_connections 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493b78090 0x561493b7a4b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b65610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5eb40 0x561493b65020 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1907979735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b67d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40aeffd640 0 client.28300 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:03:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 >> 172.21.15.16:0/1907979735 conn(0x561493b26310 msgr2=0x561493b5f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 shutdown_connections 2024-09-12T03:03:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1907979735 wait complete. 2024-09-12T03:03:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.313+0000 7f40d7e1c740 20 client.28300 trim_cache size 0 max 16384 2024-09-12T03:03:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 Processor -- start 2024-09-12T03:03:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 -- start start 2024-09-12T03:03:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5eb40 0x561493b66f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b60120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40d006fcd0 con 0x561493b5eb40 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40d00718d0 con 0x561493b68370 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40d00703a0 con 0x561493b5d5d0 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44722/0 (socket says 172.21.15.16:44722) 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1873974080 learned_addr learned my addr 172.21.15.16:0/1873974080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b60120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ceffd640 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5eb40 0x561493b66f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1873974080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 msgr2=0x561493b60120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b60120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1873974080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5eb40 msgr2=0x561493b66f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5eb40 0x561493b66f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.330+0000 7f40ce7fc640 1 -- 172.21.15.16:0/1873974080 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561493b59510 con 0x561493b5d5d0 2024-09-12T03:03:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.331+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ea30 tx=0x7f40c4011580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc47be43631cfc74 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.331+0000 7f40b57fa640 1 -- 172.21.15.16:0/1873974080 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4031070 con 0x561493b5d5d0 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.331+0000 7f40b57fa640 1 -- 172.21.15.16:0/1873974080 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c400e470 con 0x561493b5d5d0 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.331+0000 7f40b57fa640 1 -- 172.21.15.16:0/1873974080 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40035f0 con 0x561493b5d5d0 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1873974080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 msgr2=0x561493b59d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ea30 tx=0x7f40c4011580 comp rx=0 tx=0).stop 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1873974080 shutdown_connections 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b68370 0x561493b60120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5eb40 0x561493b66f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/1873974080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b5d5d0 0x561493b59d70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1873974080 >> 172.21.15.16:0/1873974080 conn(0x561493b26310 msgr2=0x561493b66880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1873974080 shutdown_connections 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- 172.21.15.16:0/1873974080 wait complete. 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.332+0000 7f40d7e1c740 1 -- start start 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 Processor -- start 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 0x561493b70bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b68370 0x561493b618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0071200 con 0x561493b5d5d0 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0070b30 con 0x561493b5eb40 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40d019c280 con 0x561493b68370 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40ce7fc640 1 -- 172.21.15.16:0/4278496802 learned_addr learned my addr 172.21.15.16:0/4278496802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40ceffd640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 0x561493b70bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b68370 0x561493b618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.333+0000 7f40d7e1c740 10 client.0 _ll_register_callbacks cb 0x7f40c0009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 -- 172.21.15.16:0/4278496802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b68370 msgr2=0x561493b618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b68370 0x561493b618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 -- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 msgr2=0x561493b70690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 -- 172.21.15.16:0/4278496802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561493b620d0 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40ceffd640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 0x561493b70bd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f40c000def0 tx=0x7f40c0023e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c8dec1620d6836b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c000f890 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c400e470 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f40c000fa30 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f40c000fc70 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40d7e1c740 10 client.28391 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.334+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561493be27d0 con 0x561493b5eb40 2024-09-12T03:03:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c400e470 con 0x561493b5eb40 2024-09-12T03:03:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f40c0003650 con 0x561493b5eb40 2024-09-12T03:03:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 10 client.28391 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 10 client.28391 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 20 client.28391 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493be2f50 0x561493be5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 4 client.28391 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561493ad0860 con 0x561493be2f50 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.335+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493be2f50 0x561493be5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40aeffd640 10 client.28391 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40ce7fc640 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493be2f50 0x561493be5370 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32c960d23ac26bbe server_cookie=e682a70f7d44f790 in_seq=0 out_seq=0 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40aeffd640 1 -- 172.21.15.16:0/4278496802 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561493ad0860 con 0x561493be2f50 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40aeffd640 10 client.28391 handle_command_reply: tid=0 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 client.28391 shutdown 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 msgr2=0x561493b70bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 0x561493b70bd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f40c000def0 tx=0x7f40c0023e00 comp rx=0 tx=0).stop 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 shutdown_connections 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561493b68370 0x561493b618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561493be2f50 0x561493be5370 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561493b5eb40 0x561493b70bd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40aeffd640 0 client.28391 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 --2- 172.21.15.16:0/4278496802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561493b5d5d0 0x561493b70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.336+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 >> 172.21.15.16:0/4278496802 conn(0x561493b26310 msgr2=0x561493b670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.337+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 shutdown_connections 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.337+0000 7f40d7e1c740 1 -- 172.21.15.16:0/4278496802 wait complete. 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.337+0000 7f40d7e1c740 20 client.28391 trim_cache size 0 max 16384 2024-09-12T03:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:03:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 msgr2=0x7f40b80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 0x7f40b80a50c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f40c8030c10 tx=0x7f40c802a930 comp rx=0 tx=0).stop 2024-09-12T03:03:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 msgr2=0x7f40d00fd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f40c802ae50 tx=0x7f40c8031720 comp rx=0 tx=0).stop 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 shutdown_connections 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f40b80a2c00 0x7f40b80a50c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0106a90 0x7f40d00fd890 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40d0101430 0x7f40d00fd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 --2- 172.21.15.16:0/1905094697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40d0100a90 0x7f40d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 >> 172.21.15.16:0/1905094697 conn(0x7f40d00fc7d0 msgr2=0x7f40d006a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 shutdown_connections 2024-09-12T03:03:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:34.338+0000 7f40d5a43640 1 -- 172.21.15.16:0/1905094697 wait complete. 2024-09-12T03:03:35.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63924 34.91 MB/sec execute 36 sec latency 49.746 ms 2024-09-12T03:03:35.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:03:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:36.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 64668 34.94 MB/sec execute 37 sec latency 38.202 ms 2024-09-12T03:03:36.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:35 smithi016 ceph-mon[30925]: pgmap v807: 97 pgs: 97 active+clean; 581 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3339623754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1036e0 msgr2=0x7f4cbc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3339623754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc107b90 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009930 tx=0x7f4cb002a420 comp rx=0 tx=0).stop 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3339623754 shutdown_connections 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3339623754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc1082a0 0x7f4cbc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3339623754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc107b90 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3339623754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc102d30 0x7f4cbc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3339623754 >> 172.21.15.16:0/3339623754 conn(0x7f4cbc0fc7d0 msgr2=0x7f4cbc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3339623754 shutdown_connections 2024-09-12T03:03:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.221+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3339623754 wait complete. 2024-09-12T03:03:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 Processor -- start 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 -- start start 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1082a0 0x7f4cbc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1120a0 con 0x7f4cbc1082a0 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc111f20 con 0x7f4cbc102d30 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cc1a47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc112220 con 0x7f4cbc1036e0 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1082a0 0x7f4cbc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36262/0 (socket says 172.21.15.16:36262) 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3401812793 learned_addr learned my addr 172.21.15.16:0/3401812793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.222+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3401812793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc1036e0 msgr2=0x7f4cbc198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1082a0 msgr2=0x7f4cbc19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1082a0 0x7f4cbc19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3401812793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc19d590 con 0x7f4cbc102d30 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9f0 tx=0x7f4cac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cc0a45640 1 -- 172.21.15.16:0/3401812793 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00cd80 con 0x7f4cbc102d30 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc19d880 con 0x7f4cbc102d30 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cc0a45640 1 -- 172.21.15.16:0/3401812793 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac005500 con 0x7f4cbc102d30 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cc0a45640 1 -- 172.21.15.16:0/3401812793 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac010690 con 0x7f4cbc102d30 2024-09-12T03:03:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.223+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc1a5160 con 0x7f4cbc102d30 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.225+0000 7f4cc0a45640 1 -- 172.21.15.16:0/3401812793 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4cac0046c0 con 0x7f4cbc102d30 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.225+0000 7f4cc3e20740 1 Processor -- start 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 -- start start 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e443b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e443ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e4448760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5636e43a12f0 con 0x5636e443b5d0 2024-09-12T03:03:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5636e4348cf0 con 0x5636e443cb40 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5636e439d2b0 con 0x5636e4446370 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e443ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e443b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e4448760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e443b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36264/0 (socket says 172.21.15.16:36264) 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/2044254049 learned_addr learned my addr 172.21.15.16:0/2044254049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc0a45640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 0x7f4ca40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 0x7f4ca40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 -- 172.21.15.16:0/2044254049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 msgr2=0x5636e443b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e443b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 -- 172.21.15.16:0/2044254049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 msgr2=0x5636e443ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e443ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cba7fc640 1 -- 172.21.15.16:0/2044254049 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e432d3a0 con 0x5636e4446370 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.226+0000 7f4cc0a45640 1 -- 172.21.15.16:0/3401812793 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4cac0cf470 con 0x7f4cbc102d30 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.227+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e4448760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002a940 tx=0x7f4cb0005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b14cac067c33d06f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.227+0000 7f4ca1ffb640 1 -- 172.21.15.16:0/2044254049 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0035040 con 0x5636e4446370 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.227+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 0x7f4ca40a5040 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40049d0 tx=0x7f4cb4004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.227+0000 7f4ca1ffb640 1 -- 172.21.15.16:0/2044254049 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb0031870 con 0x5636e4446370 2024-09-12T03:03:36.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.227+0000 7f4ca1ffb640 1 -- 172.21.15.16:0/2044254049 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb00049d0 con 0x5636e4446370 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- 172.21.15.16:0/2044254049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 msgr2=0x5636e4448760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e4448760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002a940 tx=0x7f4cb0005c00 comp rx=0 tx=0).stop 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- 172.21.15.16:0/2044254049 shutdown_connections 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e4448760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e443ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/2044254049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e443b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- 172.21.15.16:0/2044254049 >> 172.21.15.16:0/2044254049 conn(0x5636e4404310 msgr2=0x5636e4437c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- 172.21.15.16:0/2044254049 shutdown_connections 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- 172.21.15.16:0/2044254049 wait complete. 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 -- start start 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.228+0000 7f4cc3e20740 1 Processor -- start 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e4442f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e44434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5636e4449010 con 0x5636e443b5d0 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5636e435c060 con 0x5636e443cb40 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5636e4449190 con 0x5636e4446370 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cc3e20740 10 client.0 _ll_register_callbacks cb 0x5636e40af8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37964/0 (socket says 172.21.15.16:37964) 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3570456871 learned_addr learned my addr 172.21.15.16:0/3570456871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e4442f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e44434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3570456871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 msgr2=0x5636e44434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e44434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 msgr2=0x5636e4442f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e4442f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 -- 172.21.15.16:0/3570456871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e41123c0 con 0x5636e443b5d0 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e4442f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9c0 tx=0x7f4cac004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc1b2a9186c7ccd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.229+0000 7f4c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac09bc80 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4cac0a4460 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb0031870 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4cac0a0030 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 10 client.19092 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5636e4454a80 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb0031870 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f4cac09b0d0 con 0x5636e443b5d0 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 10 client.19092 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 10 client.19092 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 20 client.19092 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e4455e90 0x5636e4458300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 4 client.19092 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.230+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e43ae860 con 0x5636e4455e90 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.231+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e4455e90 0x5636e4458300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.231+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e4455e90 0x5636e4458300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e3789d72f97317b5 server_cookie=ca632cb8c090b6eb in_seq=0 out_seq=0 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.231+0000 7f4c9affd640 10 client.19092 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4c9affd640 1 -- 172.21.15.16:0/3570456871 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5636e43ae860 con 0x5636e4455e90 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4c9affd640 10 client.19092 handle_command_reply: tid=0 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 client.19092 shutdown 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 msgr2=0x5636e4445cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9c0 tx=0x7f4cac004ae0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 shutdown_connections 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e4455e90 0x5636e4458300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4c9affd640 0 client.19092 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e44434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443cb40 0x5636e4442f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3570456871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443b5d0 0x5636e4445cc0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 >> 172.21.15.16:0/3570456871 conn(0x5636e4404310 msgr2=0x5636e443d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 shutdown_connections 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3570456871 wait complete. 2024-09-12T03:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.232+0000 7f4cc3e20740 20 client.19092 trim_cache size 0 max 16384 2024-09-12T03:03:36.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:35 smithi033 ceph-mon[35092]: pgmap v807: 97 pgs: 97 active+clean; 581 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.250+0000 7f4cc3e20740 1 Processor -- start 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 -- start start 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 0x5636e4437e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e445dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ad520 con 0x5636e443cb40 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1af120 con 0x5636e443b5d0 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1adbf0 con 0x5636e4446370 2024-09-12T03:03:36.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37980/0 (socket says 172.21.15.16:37980) 2024-09-12T03:03:36.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 -- 172.21.15.16:0/3901871874 learned_addr learned my addr 172.21.15.16:0/3901871874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:36.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e445dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 0x5636e4437e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 -- 172.21.15.16:0/3901871874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 msgr2=0x5636e445dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e445dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 -- 172.21.15.16:0/3901871874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 msgr2=0x5636e4437e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 0x5636e4437e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cba7fc640 1 -- 172.21.15.16:0/3901871874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e44458e0 con 0x5636e443cb40 2024-09-12T03:03:36.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e445dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.251+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 0x5636e4437e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002a940 tx=0x7f4cb0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf534051882a91b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4ca17fa640 1 -- 172.21.15.16:0/3901871874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0035040 con 0x5636e443cb40 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4ca17fa640 1 -- 172.21.15.16:0/3901871874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb00035f0 con 0x5636e443cb40 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4ca17fa640 1 -- 172.21.15.16:0/3901871874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb003c6a0 con 0x5636e443cb40 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3901871874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 msgr2=0x5636e44522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.252+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4cb002a940 tx=0x7f4cb0005fd0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3901871874 shutdown_connections 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e4446370 0x5636e445dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e44522b0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/3901871874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e443b5d0 0x5636e4437e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3901871874 >> 172.21.15.16:0/3901871874 conn(0x5636e4404310 msgr2=0x5636e4437700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3901871874 shutdown_connections 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 -- 172.21.15.16:0/3901871874 wait complete. 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 -- start start 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 Processor -- start 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.253+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e443b5d0 0x5636e4440030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e4440570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1aea50 con 0x5636e443cb40 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ae380 con 0x5636e4446370 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ab500 con 0x5636e443b5d0 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cc3e20740 10 client.0 _ll_register_callbacks cb 0x7f4cac0d2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e4440570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36298/0 (socket says 172.21.15.16:36298) 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/490207897 learned_addr learned my addr 172.21.15.16:0/490207897 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e443b5d0 0x5636e4440030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/490207897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e443b5d0 msgr2=0x5636e4440030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e443b5d0 0x5636e4440030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/490207897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 msgr2=0x5636e4440570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e4440570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/490207897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e44bfc20 con 0x5636e4446370 2024-09-12T03:03:36.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.254+0000 7f4cba7fc640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e4440570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400fc80 tx=0x7f4cb4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d43f68d3ccfc337c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:36.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4010e40 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb00035f0 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4cb4004b20 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4cb4004d20 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4cc3e20740 10 client.28409 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5636e44c0f60 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb00035f0 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.255+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f4cb401f040 con 0x5636e4446370 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 10 client.28409 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 10 client.28409 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 20 client.28409 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e44c1710 0x5636e44c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:36.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 4 client.28409 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636e43a8d70 con 0x5636e44c1710 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e44c1710 0x5636e44c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cbaffd640 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e44c1710 0x5636e44c3b30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=968864bdcefcbee0 server_cookie=54838b443e4bac97 in_seq=0 out_seq=0 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4c9affd640 10 client.28409 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4c9affd640 1 -- 172.21.15.16:0/490207897 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636e43a8d70 con 0x5636e44c1710 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4c9affd640 10 client.28409 handle_command_reply: tid=0 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.256+0000 7f4cc3e20740 1 client.28409 shutdown 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 msgr2=0x5636e4440ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400fc80 tx=0x7f4cb4004a40 comp rx=0 tx=0).stop 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 shutdown_connections 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5636e44c1710 0x5636e44c3b30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5636e4446370 0x5636e4440ab0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5636e443cb40 0x5636e4440570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 --2- 172.21.15.16:0/490207897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636e443b5d0 0x5636e4440030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 >> 172.21.15.16:0/490207897 conn(0x5636e4404310 msgr2=0x5636e44364e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4c9affd640 0 client.28409 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 shutdown_connections 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 1 -- 172.21.15.16:0/490207897 wait complete. 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.257+0000 7f4cc3e20740 20 client.28409 trim_cache size 0 max 16384 2024-09-12T03:03:36.262 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:03:36.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 msgr2=0x7f4ca40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 0x7f4ca40a5040 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40049d0 tx=0x7f4cb4004ab0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 msgr2=0x7f4cbc1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:36.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9f0 tx=0x7f4cac00edf0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 shutdown_connections 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4ca40a2b80 0x7f4ca40a5040 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc1082a0 0x7f4cbc19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cbc1036e0 0x7f4cbc198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 --2- 172.21.15.16:0/3401812793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cbc102d30 0x7f4cbc1985e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 >> 172.21.15.16:0/3401812793 conn(0x7f4cbc0fc7d0 msgr2=0x7f4cbc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 shutdown_connections 2024-09-12T03:03:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:36.258+0000 7f4cc1a47640 1 -- 172.21.15.16:0/3401812793 wait complete. 2024-09-12T03:03:36.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:35 smithi184 ceph-mon[35321]: pgmap v807: 97 pgs: 97 active+clean; 581 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-09-12T03:03:37.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65537 35.15 MB/sec execute 38 sec latency 48.995 ms 2024-09-12T03:03:37.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 -- 172.21.15.16:0/826550029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4301091f0 msgr2=0x7fe4301095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 --2- 172.21.15.16:0/826550029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4301091f0 0x7fe4301095d0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009900 tx=0x7fe42c02a400 comp rx=0 tx=0).stop 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 -- 172.21.15.16:0/826550029 shutdown_connections 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 --2- 172.21.15.16:0/826550029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430116ea0 0x7fe430119290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 --2- 172.21.15.16:0/826550029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe430109b10 0x7fe430116960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 --2- 172.21.15.16:0/826550029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4301091f0 0x7fe4301095d0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.896+0000 7fe437f1f640 1 -- 172.21.15.16:0/826550029 >> 172.21.15.16:0/826550029 conn(0x7fe43006e140 msgr2=0x7fe43006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.897+0000 7fe437f1f640 1 -- 172.21.15.16:0/826550029 shutdown_connections 2024-09-12T03:03:37.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.897+0000 7fe437f1f640 1 -- 172.21.15.16:0/826550029 wait complete. 2024-09-12T03:03:37.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.898+0000 7fe437f1f640 1 Processor -- start 2024-09-12T03:03:37.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.898+0000 7fe437f1f640 1 -- start start 2024-09-12T03:03:37.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430109b10 0x7fe4301a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 0x7fe4301a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe43011c9b0 con 0x7fe430116ea0 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe43011c830 con 0x7fe430109b10 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe437f1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe43011cb30 con 0x7fe4301091f0 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46510/0 (socket says 172.21.15.16:46510) 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe435cbb640 1 -- 172.21.15.16:0/3138866068 learned_addr learned my addr 172.21.15.16:0/3138866068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430109b10 0x7fe4301a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.899+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 0x7fe4301a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 -- 172.21.15.16:0/3138866068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430109b10 msgr2=0x7fe4301a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430109b10 0x7fe4301a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 -- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 msgr2=0x7fe4301a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 0x7fe4301a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 -- 172.21.15.16:0/3138866068 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4301a3230 con 0x7fe4301091f0 2024-09-12T03:03:37.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 0x7fe4301a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:03:37.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.900+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0305d0 tx=0x7fe42c030600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.901+0000 7fe426ffd640 1 -- 172.21.15.16:0/3138866068 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c03f070 con 0x7fe4301091f0 2024-09-12T03:03:37.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.901+0000 7fe426ffd640 1 -- 172.21.15.16:0/3138866068 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c004460 con 0x7fe4301091f0 2024-09-12T03:03:37.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.901+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4301a34c0 con 0x7fe4301091f0 2024-09-12T03:03:37.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.901+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4301af810 con 0x7fe4301091f0 2024-09-12T03:03:37.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.901+0000 7fe426ffd640 1 -- 172.21.15.16:0/3138866068 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c035030 con 0x7fe4301091f0 2024-09-12T03:03:37.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.905+0000 7fe43a2f8740 1 Processor -- start 2024-09-12T03:03:37.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.905+0000 7fe43a2f8740 1 -- start start 2024-09-12T03:03:37.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.905+0000 7fe426ffd640 1 -- 172.21.15.16:0/3138866068 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe42c042070 con 0x7fe4301091f0 2024-09-12T03:03:37.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac888220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac894f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560cac7e52f0 con 0x560cac887de0 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560cac78ccf0 con 0x560cac889350 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cac7e12b0 con 0x560cac892b80 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe4364bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe4364bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36322/0 (socket says 172.21.15.16:36322) 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe4364bc640 1 -- 172.21.15.16:0/3641970971 learned_addr learned my addr 172.21.15.16:0/3641970971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac888220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac894f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.906+0000 7fe426ffd640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 0x7fe41c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe435cbb640 1 -- 172.21.15.16:0/3641970971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 msgr2=0x560cac894f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac894f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe435cbb640 1 -- 172.21.15.16:0/3641970971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 msgr2=0x560cac8878a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe435cbb640 1 -- 172.21.15.16:0/3641970971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cac7713a0 con 0x560cac887de0 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.907+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac894f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe426ffd640 1 -- 172.21.15.16:0/3138866068 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe42c0ef660 con 0x7fe4301091f0 2024-09-12T03:03:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac888220 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b8840 tx=0x7fe42c0bb7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fe95e7044f4e91 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe41bfff640 1 -- 172.21.15.16:0/3641970971 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c0c0220 con 0x560cac887de0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe41bfff640 1 -- 172.21.15.16:0/3641970971 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c0c4ae0 con 0x560cac887de0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.908+0000 7fe41bfff640 1 -- 172.21.15.16:0/3641970971 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c0b84e0 con 0x560cac887de0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 0x7fe41c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 0x7fe41c0a5210 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe428010490 tx=0x7fe428004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3641970971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 msgr2=0x560cac888220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac888220 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b8840 tx=0x7fe42c0bb7d0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3641970971 shutdown_connections 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac894f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac888220 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3641970971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac8878a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.909+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3641970971 >> 172.21.15.16:0/3641970971 conn(0x560cac848310 msgr2=0x560cac884470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.910+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3641970971 shutdown_connections 2024-09-12T03:03:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.910+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3641970971 wait complete. 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.910+0000 7fe43a2f8740 1 -- start start 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 Processor -- start 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac88f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac88fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38036/0 (socket says 172.21.15.16:38036) 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe435cbb640 1 -- 172.21.15.16:0/2631987751 learned_addr learned my addr 172.21.15.16:0/2631987751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560cac7e52f0 con 0x560cac887de0 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560cac7a0060 con 0x560cac889350 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.911+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cac895640 con 0x560cac892b80 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe43a2f8740 10 client.0 _ll_register_callbacks cb 0x560cac4f38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe4364bc640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac88fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe435cbb640 1 -- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 msgr2=0x560cac88fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe435cbb640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac88fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe435cbb640 1 -- 172.21.15.16:0/2631987751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 msgr2=0x560cac88f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe4354ba640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac88f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe435cbb640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac88f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe435cbb640 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cac5563c0 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.912+0000 7fe4354ba640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac88f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe435cbb640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b82d0 tx=0x7fe42c02a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8142843fe6e908fc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c0c0220 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe42c0b84e0 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42c0c4ae0 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe42c0035f0 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe43a2f8740 10 client.19107 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560cac8a0f60 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c003790 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.913+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fe42c0bb460 con 0x560cac887de0 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 10 client.19107 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 10 client.19107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 20 client.19107 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac8a2150 0x560cac8a4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 4 client.19107 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe42c0c4ae0 con 0x560cac8a2150 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe4354ba640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac8a2150 0x560cac8a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe4197fa640 10 client.19107 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.914+0000 7fe4354ba640 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac8a2150 0x560cac8a4560 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aed6fb6b8ccf701b server_cookie=768fddf2e42b431a in_seq=0 out_seq=0 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe4197fa640 1 -- 172.21.15.16:0/2631987751 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fe42c0c4ae0 con 0x560cac8a2150 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe4197fa640 10 client.19107 handle_command_reply: tid=0 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 client.19107 shutdown 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 msgr2=0x560cac8925b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b82d0 tx=0x7fe42c02a400 comp rx=0 tx=0).stop 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 shutdown_connections 2024-09-12T03:03:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac8a2150 0x560cac8a4560 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac892b80 0x560cac88fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac889350 0x560cac88f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/2631987751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac8925b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 >> 172.21.15.16:0/2631987751 conn(0x560cac848310 msgr2=0x560cac88a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe4197fa640 0 client.19107 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 shutdown_connections 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 1 -- 172.21.15.16:0/2631987751 wait complete. 2024-09-12T03:03:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.916+0000 7fe43a2f8740 20 client.19107 trim_cache size 0 max 16384 2024-09-12T03:03:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 Processor -- start 2024-09-12T03:03:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 -- start start 2024-09-12T03:03:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac889350 0x560cac8a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400e420 con 0x560cac887de0 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe414010020 con 0x560cac892b80 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400eaf0 con 0x560cac889350 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe4354ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac889350 0x560cac8a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe435cbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38038/0 (socket says 172.21.15.16:38038) 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe435cbb640 1 -- 172.21.15.16:0/683970605 learned_addr learned my addr 172.21.15.16:0/683970605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.932+0000 7fe4364bc640 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 -- 172.21.15.16:0/683970605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac889350 msgr2=0x560cac8a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac889350 0x560cac8a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 -- 172.21.15.16:0/683970605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 msgr2=0x560cac8aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 -- 172.21.15.16:0/683970605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cac87a310 con 0x560cac887de0 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe435cbb640 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b6910 tx=0x7fe42c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a77db511be18dde server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe41b7fe640 1 -- 172.21.15.16:0/683970605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c003790 con 0x560cac887de0 2024-09-12T03:03:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe41b7fe640 1 -- 172.21.15.16:0/683970605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c0c0220 con 0x560cac887de0 2024-09-12T03:03:37.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.933+0000 7fe41b7fe640 1 -- 172.21.15.16:0/683970605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c0c4650 con 0x560cac887de0 2024-09-12T03:03:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 -- 172.21.15.16:0/683970605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 msgr2=0x560cac88e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0b6910 tx=0x7fe42c004610 comp rx=0 tx=0).stop 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 -- 172.21.15.16:0/683970605 shutdown_connections 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac889350 0x560cac8a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/683970605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac887de0 0x560cac88e840 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 -- 172.21.15.16:0/683970605 >> 172.21.15.16:0/683970605 conn(0x560cac848310 msgr2=0x560cac88e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 -- 172.21.15.16:0/683970605 shutdown_connections 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.934+0000 7fe43a2f8740 1 -- 172.21.15.16:0/683970605 wait complete. 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 -- start start 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 Processor -- start 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac887de0 0x560cac89ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8965f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe4354ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe43a2f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400f950 con 0x560cac889350 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe4354ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38054/0 (socket says 172.21.15.16:38054) 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.935+0000 7fe4354ba640 1 -- 172.21.15.16:0/3976594613 learned_addr learned my addr 172.21.15.16:0/3976594613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400f280 con 0x560cac892b80 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400c400 con 0x560cac887de0 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac887de0 0x560cac89ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 -- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac887de0 msgr2=0x560cac89ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac887de0 0x560cac89ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 -- 172.21.15.16:0/3976594613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 msgr2=0x560cac8965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cac896bb0 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4364bc640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 10 client.0 _ll_register_callbacks cb 0x7fe4280084d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42c0c0220 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4354ba640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe420004700 tx=0x7fe4200045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91086449b7c0fa0e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4197fa640 10 client.19113 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4197fa640 1 -- 172.21.15.16:0/3976594613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420010630 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4197fa640 1 -- 172.21.15.16:0/3976594613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe4200107d0 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe4197fa640 1 -- 172.21.15.16:0/3976594613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe420010a30 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 10 client.19113 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.936+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=36}) -- 0x560cac90ca60 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe4197fa640 1 -- 172.21.15.16:0/3976594613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fe420010bd0 con 0x560cac889350 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 10 client.19113 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 10 client.19113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 20 client.19113 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac90d1b0 0x560cac90f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 4 client.19113 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560cac7f2860 con 0x560cac90d1b0 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.937+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac90d1b0 0x560cac90f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.938+0000 7fe4197fa640 10 client.19113 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:03:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.938+0000 7fe435cbb640 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac90d1b0 0x560cac90f610 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30182f66473fdbc3 server_cookie=fd289f54774a21bd in_seq=0 out_seq=0 2024-09-12T03:03:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe4197fa640 1 -- 172.21.15.16:0/3976594613 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560cac7f2860 con 0x560cac90d1b0 2024-09-12T03:03:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe4197fa640 10 client.19113 handle_command_reply: tid=0 2024-09-12T03:03:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 client.19113 shutdown 2024-09-12T03:03:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 msgr2=0x560cac8960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe420004700 tx=0x7fe4200045b0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 shutdown_connections 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560cac892b80 0x560cac8965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560cac889350 0x560cac8960b0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560cac90d1b0 0x560cac90f610 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 --2- 172.21.15.16:0/3976594613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cac887de0 0x560cac89ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 >> 172.21.15.16:0/3976594613 conn(0x560cac848310 msgr2=0x560cac87a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe4197fa640 0 client.19113 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.941+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 shutdown_connections 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe43a2f8740 1 -- 172.21.15.16:0/3976594613 wait complete. 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe43a2f8740 20 client.19113 trim_cache size 0 max 16384 2024-09-12T03:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:03:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 msgr2=0x7fe41c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 0x7fe41c0a5210 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe428010490 tx=0x7fe428004a40 comp rx=0 tx=0).stop 2024-09-12T03:03:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 msgr2=0x7fe4301a8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.942+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0305d0 tx=0x7fe42c030600 comp rx=0 tx=0).stop 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 shutdown_connections 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe41c0a2d50 0x7fe41c0a5210 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430116ea0 0x7fe4301a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe430109b10 0x7fe4301a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 --2- 172.21.15.16:0/3138866068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4301091f0 0x7fe4301a8940 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 >> 172.21.15.16:0/3138866068 conn(0x7fe43006e140 msgr2=0x7fe430117280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:37.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 shutdown_connections 2024-09-12T03:03:37.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:37.943+0000 7fe437f1f640 1 -- 172.21.15.16:0/3138866068 wait complete. 2024-09-12T03:03:38.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 66405 35.82 MB/sec execute 39 sec latency 49.317 ms 2024-09-12T03:03:38.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:37 smithi016 ceph-mon[30925]: pgmap v808: 97 pgs: 97 active+clean; 586 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 768 op/s 2024-09-12T03:03:38.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:37 smithi033 ceph-mon[35092]: pgmap v808: 97 pgs: 97 active+clean; 586 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 768 op/s 2024-09-12T03:03:38.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:37 smithi184 ceph-mon[35321]: pgmap v808: 97 pgs: 97 active+clean; 586 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 768 op/s 2024-09-12T03:03:38.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:03:39.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 66832 35.51 MB/sec execute 40 sec latency 187.575 ms 2024-09-12T03:03:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.533+0000 7fad29917640 1 -- 172.21.15.16:0/2259374093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 msgr2=0x7fad24103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.533+0000 7fad29917640 1 --2- 172.21.15.16:0/2259374093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad24103130 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fad14009900 tx=0x7fad1402a480 comp rx=0 tx=0).stop 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 -- 172.21.15.16:0/2259374093 shutdown_connections 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 --2- 172.21.15.16:0/2259374093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad2410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 --2- 172.21.15.16:0/2259374093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 0x7fad2410bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 --2- 172.21.15.16:0/2259374093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad24103130 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 -- 172.21.15.16:0/2259374093 >> 172.21.15.16:0/2259374093 conn(0x7fad240fc7d0 msgr2=0x7fad240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 -- 172.21.15.16:0/2259374093 shutdown_connections 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.534+0000 7fad29917640 1 -- 172.21.15.16:0/2259374093 wait complete. 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 Processor -- start 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 -- start start 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 0x7fad24100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad24101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111f80 con 0x7fad24102d50 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111e00 con 0x7fad24103670 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad29917640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112100 con 0x7fad2410c180 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad237fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad24101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38078/0 (socket says 172.21.15.16:38078) 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad22ffd640 1 -- 172.21.15.16:0/2552751229 learned_addr learned my addr 172.21.15.16:0/2552751229 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.535+0000 7fad227fc640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 0x7fad24100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 -- 172.21.15.16:0/2552751229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 msgr2=0x7fad24101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad24101430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 -- 172.21.15.16:0/2552751229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 msgr2=0x7fad24100ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 0x7fad24100ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 -- 172.21.15.16:0/2552751229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241a68b0 con 0x7fad24102d50 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad237fe640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad24101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad22ffd640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fad14006010 tx=0x7fad140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad28915640 1 -- 172.21.15.16:0/2552751229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1403f030 con 0x7fad24102d50 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad241a6b40 con 0x7fad24102d50 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad28915640 1 -- 172.21.15.16:0/2552751229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad14003ce0 con 0x7fad24102d50 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad28915640 1 -- 172.21.15.16:0/2552751229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14035070 con 0x7fad24102d50 2024-09-12T03:03:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.536+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241a7070 con 0x7fad24102d50 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 Processor -- start 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 -- start start 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad28915640 1 -- 172.21.15.16:0/2552751229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fad14042030 con 0x7fad24102d50 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ecc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ecca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9e322f0 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9dd9cf0 con 0x55e0d9ed7370 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.539+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9e2e2b0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ecca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ecc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad227fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ecca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38084/0 (socket says 172.21.15.16:38084) 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad22ffd640 1 -- 172.21.15.16:0/2824796562 learned_addr learned my addr 172.21.15.16:0/2824796562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 -- 172.21.15.16:0/2824796562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ed7370 msgr2=0x55e0d9ed9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 -- 172.21.15.16:0/2824796562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 msgr2=0x55e0d9ecca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ecca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 -- 172.21.15.16:0/2824796562 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d9dbe3a0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad237fe640 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ecc090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fad1c010710 tx=0x7fad1c010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4acfb7d7eb1fffc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.540+0000 7fad28915640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 0x7fad0c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad09ffb640 1 -- 172.21.15.16:0/2824796562 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c004b20 con 0x55e0d9ecdb40 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad09ffb640 1 -- 172.21.15.16:0/2824796562 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c00edb0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad09ffb640 1 -- 172.21.15.16:0/2824796562 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c005670 con 0x55e0d9ecdb40 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad22ffd640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 0x7fad0c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad28915640 1 -- 172.21.15.16:0/2552751229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad140ef440 con 0x7fad24102d50 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2824796562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 msgr2=0x55e0d9ecc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad22ffd640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 0x7fad0c0a50c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fad14000c00 tx=0x7fad140f2f20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.541+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ecc090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fad1c010710 tx=0x7fad1c010bd0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2824796562 shutdown_connections 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ecca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2824796562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ecc090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2824796562 >> 172.21.15.16:0/2824796562 conn(0x55e0d9e95310 msgr2=0x55e0d9ec8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2824796562 shutdown_connections 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2824796562 wait complete. 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 -- start start 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.542+0000 7fad2bcf0740 1 Processor -- start 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ed6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9eda070 con 0x55e0d9ecdb40 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9ded060 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e0d9eda1f0 con 0x55e0d9ed7370 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad2bcf0740 10 client.0 _ll_register_callbacks cb 0x55e0d9b408c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ed6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38092/0 (socket says 172.21.15.16:38092) 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 -- 172.21.15.16:0/2359260311 learned_addr learned my addr 172.21.15.16:0/2359260311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad237fe640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 -- 172.21.15.16:0/2359260311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 msgr2=0x55e0d9ed45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 -- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 msgr2=0x55e0d9ed6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ed6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad227fc640 1 -- 172.21.15.16:0/2359260311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d9ba33c0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.543+0000 7fad237fe640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:03:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad227fc640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fad1800bef0 tx=0x7fad18011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4324cd3d3daad61a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18004020 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad180041c0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c00edb0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fad180043c0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad2bcf0740 10 client.19125 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.544+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0d9eee330 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c00edb0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fad18010700 con 0x55e0d9ecdb40 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 10 client.19125 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 10 client.19125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:03:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 20 client.19125 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9eef800 0x55e0d9ef1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 4 client.19125 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0d9e3f860 con 0x55e0d9eef800 2024-09-12T03:03:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.545+0000 7fad237fe640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9eef800 0x55e0d9ef1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.547+0000 7fad02ffd640 10 client.19125 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:03:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.547+0000 7fad237fe640 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9eef800 0x55e0d9ef1c20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c5a8145a251a5fa server_cookie=e8451ac400afc90b in_seq=0 out_seq=0 2024-09-12T03:03:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad02ffd640 1 -- 172.21.15.16:0/2359260311 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e0d9e3f860 con 0x55e0d9eef800 2024-09-12T03:03:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad02ffd640 10 client.19125 handle_command_reply: tid=0 2024-09-12T03:03:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 client.19125 shutdown 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 msgr2=0x55e0d9ed4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fad1800bef0 tx=0x7fad18011810 comp rx=0 tx=0).stop 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 shutdown_connections 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9eef800 0x55e0d9ef1c20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ed45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ed4000 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/2359260311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ed6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 >> 172.21.15.16:0/2359260311 conn(0x55e0d9e95310 msgr2=0x55e0d9ece930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad02ffd640 0 client.19125 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 shutdown_connections 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 1 -- 172.21.15.16:0/2359260311 wait complete. 2024-09-12T03:03:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.548+0000 7fad2bcf0740 20 client.19125 trim_cache size 0 max 16384 2024-09-12T03:03:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 Processor -- start 2024-09-12T03:03:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 -- start start 2024-09-12T03:03:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ef7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ab2e0 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad241acee0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ab9b0 con 0x55e0d9ed7370 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad227fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38094/0 (socket says 172.21.15.16:38094) 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad227fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36414/0 (socket says 172.21.15.16:36414) 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.563+0000 7fad237fe640 1 -- 172.21.15.16:0/216959117 learned_addr learned my addr 172.21.15.16:0/216959117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ef7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 -- 172.21.15.16:0/216959117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 msgr2=0x55e0d9ef7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ef7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 -- 172.21.15.16:0/216959117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 msgr2=0x55e0d9ef74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 -- 172.21.15.16:0/216959117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d9ecd3a0 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad227fc640 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad237fe640 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0104d0 tx=0x7fad1c0052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5eb88537cf1acc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad097fa640 1 -- 172.21.15.16:0/216959117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c026040 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad097fa640 1 -- 172.21.15.16:0/216959117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c027040 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.564+0000 7fad097fa640 1 -- 172.21.15.16:0/216959117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c005930 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 -- 172.21.15.16:0/216959117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 msgr2=0x55e0d9eebbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0104d0 tx=0x7fad1c0052b0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 -- 172.21.15.16:0/216959117 shutdown_connections 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ed7370 0x55e0d9ef7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ef74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/216959117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9eebbd0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 -- 172.21.15.16:0/216959117 >> 172.21.15.16:0/216959117 conn(0x55e0d9e95310 msgr2=0x55e0d9ed6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 -- 172.21.15.16:0/216959117 shutdown_connections 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.565+0000 7fad2bcf0740 1 -- 172.21.15.16:0/216959117 wait complete. 2024-09-12T03:03:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 -- start start 2024-09-12T03:03:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 Processor -- start 2024-09-12T03:03:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ee8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ed7370 0x55e0d9ee3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ac810 con 0x55e0d9ed7370 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ac140 con 0x55e0d9ecc5d0 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad241a92c0 con 0x55e0d9ecdb40 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.566+0000 7fad2bcf0740 10 client.0 _ll_register_callbacks cb 0x7fad1800bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46590/0 (socket says 172.21.15.16:46590) 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 -- 172.21.15.16:0/3754509209 learned_addr learned my addr 172.21.15.16:0/3754509209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad237fe640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ed7370 0x55e0d9ee3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad237fe640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ee8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 -- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 msgr2=0x55e0d9ee8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ee8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 -- 172.21.15.16:0/3754509209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ed7370 msgr2=0x55e0d9ee3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ed7370 0x55e0d9ee3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad227fc640 1 -- 172.21.15.16:0/3754509209 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d9ec8510 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.567+0000 7fad237fe640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ee8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad227fc640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55e0d9ed5640 tx=0x7fad18011da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beb9930617e3a435 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18033060 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c027040 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad1802b070 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fad18010700 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad2bcf0740 10 client.28360 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.568+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0d9f59f10 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c027040 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fad18004780 con 0x55e0d9ecdb40 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 10 client.28360 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 10 client.28360 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 20 client.28360 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9f5a6c0 0x55e0d9f5cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 4 client.28360 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0d9e39d70 con 0x55e0d9f5a6c0 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.569+0000 7fad237fe640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9f5a6c0 0x55e0d9f5cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad237fe640 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9f5a6c0 0x55e0d9f5cae0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f9af7782f6ef6e0 server_cookie=f61eb7515cb380a7 in_seq=0 out_seq=0 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad02ffd640 10 client.28360 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad02ffd640 1 -- 172.21.15.16:0/3754509209 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0d9e39d70 con 0x55e0d9f5a6c0 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad02ffd640 10 client.28360 handle_command_reply: tid=0 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 client.28360 shutdown 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 msgr2=0x55e0d9ee85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55e0d9ed5640 tx=0x7fad18011da0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 shutdown_connections 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55e0d9f5a6c0 0x55e0d9f5cae0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0d9ed7370 0x55e0d9ee3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e0d9ecdb40 0x55e0d9ee85d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad02ffd640 0 client.28360 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 --2- 172.21.15.16:0/3754509209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e0d9ecc5d0 0x55e0d9ee8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 >> 172.21.15.16:0/3754509209 conn(0x55e0d9e95310 msgr2=0x55e0d9ec74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 shutdown_connections 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 1 -- 172.21.15.16:0/3754509209 wait complete. 2024-09-12T03:03:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.570+0000 7fad2bcf0740 20 client.28360 trim_cache size 0 max 16384 2024-09-12T03:03:39.577 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:03:39.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 msgr2=0x7fad0c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 0x7fad0c0a50c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fad14000c00 tx=0x7fad140f2f20 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 msgr2=0x7fad241009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fad14006010 tx=0x7fad140042f0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 shutdown_connections 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410c180 0x7fad24101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24103670 0x7fad24100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad0c0a2c00 0x7fad0c0a50c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 --2- 172.21.15.16:0/2552751229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad24102d50 0x7fad241009b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 >> 172.21.15.16:0/2552751229 conn(0x7fad240fc7d0 msgr2=0x7fad240fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 shutdown_connections 2024-09-12T03:03:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:39.571+0000 7fad29917640 1 -- 172.21.15.16:0/2552751229 wait complete. 2024-09-12T03:03:40.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 67342 35.27 MB/sec execute 41 sec latency 62.630 ms 2024-09-12T03:03:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:39 smithi016 ceph-mon[30925]: pgmap v809: 97 pgs: 97 active+clean; 589 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.9 MiB/s wr, 630 op/s 2024-09-12T03:03:40.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:39 smithi033 ceph-mon[35092]: pgmap v809: 97 pgs: 97 active+clean; 589 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.9 MiB/s wr, 630 op/s 2024-09-12T03:03:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:39 smithi184 ceph-mon[35321]: pgmap v809: 97 pgs: 97 active+clean; 589 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.9 MiB/s wr, 630 op/s 2024-09-12T03:03:40.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:03:40.726 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:41.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 68097 35.14 MB/sec execute 42 sec latency 62.521 ms 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 -- 172.21.15.16:0/614470284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c102ef0 msgr2=0x7fd99c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 --2- 172.21.15.16:0/614470284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c102ef0 0x7fd99c10da40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd994009ca0 tx=0x7fd99402a950 comp rx=0 tx=0).stop 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 -- 172.21.15.16:0/614470284 shutdown_connections 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 --2- 172.21.15.16:0/614470284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c102ef0 0x7fd99c10da40 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 --2- 172.21.15.16:0/614470284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 0x7fd99c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 --2- 172.21.15.16:0/614470284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c100e20 0x7fd99c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 -- 172.21.15.16:0/614470284 >> 172.21.15.16:0/614470284 conn(0x7fd99c0fc7d0 msgr2=0x7fd99c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 -- 172.21.15.16:0/614470284 shutdown_connections 2024-09-12T03:03:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.277+0000 7fd9a318e640 1 -- 172.21.15.16:0/614470284 wait complete. 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 Processor -- start 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 -- start start 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 0x7fd99c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c102ef0 0x7fd99c1a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c110fd0 con 0x7fd99c100e20 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c110e50 con 0x7fd99c102570 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a318e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c111150 con 0x7fd99c102ef0 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a0f2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a0f2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38124/0 (socket says 172.21.15.16:38124) 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a0f2a640 1 -- 172.21.15.16:0/1405050466 learned_addr learned my addr 172.21.15.16:0/1405050466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.278+0000 7fd9a172b640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c102ef0 0x7fd99c1a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd99bfff640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 0x7fd99c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 -- 172.21.15.16:0/1405050466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c102ef0 msgr2=0x7fd99c1a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c102ef0 0x7fd99c1a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 -- 172.21.15.16:0/1405050466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 msgr2=0x7fd99c199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 0x7fd99c199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 -- 172.21.15.16:0/1405050466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c1a48c0 con 0x7fd99c100e20 2024-09-12T03:03:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd99000e9f0 tx=0x7fd99000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd999ffb640 1 -- 172.21.15.16:0/1405050466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99000cd80 con 0x7fd99c100e20 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c1a4a90 con 0x7fd99c100e20 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd999ffb640 1 -- 172.21.15.16:0/1405050466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd990010910 con 0x7fd99c100e20 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.279+0000 7fd999ffb640 1 -- 172.21.15.16:0/1405050466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9900188f0 con 0x7fd99c100e20 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.280+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a5020 con 0x7fd99c100e20 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.282+0000 7fd9a5567740 1 Processor -- start 2024-09-12T03:03:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.282+0000 7fd9a5567740 1 -- start start 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd999ffb640 1 -- 172.21.15.16:0/1405050466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd990010ac0 con 0x7fd99c100e20 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf30560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf45f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55deeae962f0 con 0x55deeaf31840 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55deeae3dcf0 con 0x55deeaf30b80 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a5567740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55deeae922b0 con 0x55deeaf30120 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38136/0 (socket says 172.21.15.16:38136) 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a172b640 1 -- 172.21.15.16:0/3507423605 learned_addr learned my addr 172.21.15.16:0/3507423605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.283+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf30560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 -- 172.21.15.16:0/3507423605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 msgr2=0x55deeaf30560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf30560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd99bfff640 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf45f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 -- 172.21.15.16:0/3507423605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 msgr2=0x55deeaf45f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf45f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 -- 172.21.15.16:0/3507423605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deeae223a0 con 0x55deeaf31840 2024-09-12T03:03:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.284+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd994009ca0 tx=0x7fd994005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33cbbf5c3bfaa18 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd986ffd640 1 -- 172.21.15.16:0/3507423605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994048070 con 0x55deeaf31840 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd986ffd640 1 -- 172.21.15.16:0/3507423605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994003e50 con 0x55deeaf31840 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd986ffd640 1 -- 172.21.15.16:0/3507423605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994034040 con 0x55deeaf31840 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd999ffb640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 0x7fd98c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 0x7fd98c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd999ffb640 1 -- 172.21.15.16:0/1405050466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd9900cf3f0 con 0x7fd99c100e20 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd9a5567740 1 -- 172.21.15.16:0/3507423605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 msgr2=0x55deeaf31c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.285+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd994009ca0 tx=0x7fd994005f80 comp rx=0 tx=0).stop 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 -- 172.21.15.16:0/3507423605 shutdown_connections 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf45f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf30560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3507423605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf31c20 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 -- 172.21.15.16:0/3507423605 >> 172.21.15.16:0/3507423605 conn(0x55deeaef9310 msgr2=0x55deeaf2cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 -- 172.21.15.16:0/3507423605 shutdown_connections 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a0f2a640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 0x7fd98c0a50c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd9900cd270 tx=0x7fd9900d4040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 -- 172.21.15.16:0/3507423605 wait complete. 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 -- start start 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.286+0000 7fd9a5567740 1 Processor -- start 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 0x55deeaf45df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf31840 0x55deeaf43600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55deeaf468d0 con 0x55deeaf30120 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55deeae51060 con 0x55deeaf30b80 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55deeaf46a50 con 0x55deeaf31840 2024-09-12T03:03:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a5567740 10 client.0 _ll_register_callbacks cb 0x55deeaba48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 0x55deeaf45df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36466/0 (socket says 172.21.15.16:36466) 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd99bfff640 1 -- 172.21.15.16:0/1872132719 learned_addr learned my addr 172.21.15.16:0/1872132719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.287+0000 7fd9a172b640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf31840 0x55deeaf43600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 -- 172.21.15.16:0/1872132719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf31840 msgr2=0x55deeaf43600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf31840 0x55deeaf43600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 -- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 msgr2=0x55deeaf430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 -- 172.21.15.16:0/1872132719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deeac073c0 con 0x55deeaf30120 2024-09-12T03:03:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd99bfff640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a172b640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 0x55deeaf45df0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd994046040 tx=0x7fd9940095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1e121919f559e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd97bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994048050 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd994003e50 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.288+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd994004000 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd994034040 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd9a5567740 10 client.19143 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55deeaf52310 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994003e50 con 0x55deeaf30120 2024-09-12T03:03:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fd994030890 con 0x55deeaf30120 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd9a5567740 10 client.19143 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd9a5567740 10 client.19143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.289+0000 7fd9a5567740 20 client.19143 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeaf53660 0x55deeaf55a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd9a5567740 4 client.19143 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55deeaea3860 con 0x55deeaf53660 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd99bfff640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeaf53660 0x55deeaf55a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd97bfff640 10 client.19143 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.290+0000 7fd99bfff640 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeaf53660 0x55deeaf55a50 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f49a38a59f9d957 server_cookie=20c22ea4d601964 in_seq=0 out_seq=0 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.291+0000 7fd97bfff640 1 -- 172.21.15.16:0/1872132719 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55deeaea3860 con 0x55deeaf53660 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.291+0000 7fd97bfff640 10 client.19143 handle_command_reply: tid=0 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 client.19143 shutdown 2024-09-12T03:03:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 msgr2=0x55deeaf45df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 0x55deeaf45df0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd994046040 tx=0x7fd9940095c0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 shutdown_connections 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeaf53660 0x55deeaf55a50 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf31840 0x55deeaf43600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 --2- 172.21.15.16:0/1872132719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30120 0x55deeaf45df0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 >> 172.21.15.16:0/1872132719 conn(0x55deeaef9310 msgr2=0x55deeaf329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd97bfff640 0 client.19143 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 shutdown_connections 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 1 -- 172.21.15.16:0/1872132719 wait complete. 2024-09-12T03:03:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.292+0000 7fd9a5567740 20 client.19143 trim_cache size 0 max 16384 2024-09-12T03:03:41.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 Processor -- start 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 -- start start 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf2ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ab2e0 con 0x55deeaf31840 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1acee0 con 0x55deeaf30b80 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a5567740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ab9b0 con 0x55deeaf30120 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf2ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46644/0 (socket says 172.21.15.16:46644) 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36470/0 (socket says 172.21.15.16:36470) 2024-09-12T03:03:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 -- 172.21.15.16:0/3081576645 learned_addr learned my addr 172.21.15.16:0/3081576645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 -- 172.21.15.16:0/3081576645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 msgr2=0x55deeaf2b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 -- 172.21.15.16:0/3081576645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 msgr2=0x55deeaf2ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf2ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.309+0000 7fd9a172b640 1 -- 172.21.15.16:0/3081576645 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deeaf2ced0 con 0x55deeaf30120 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.310+0000 7fd99bfff640 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:41.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.310+0000 7fd9a172b640 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd994004aa0 tx=0x7fd994000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d99d0b6e45b3d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.310+0000 7fd9867fc640 1 -- 172.21.15.16:0/3081576645 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994048070 con 0x55deeaf30120 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.310+0000 7fd9867fc640 1 -- 172.21.15.16:0/3081576645 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99405f040 con 0x55deeaf30120 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.310+0000 7fd9867fc640 1 -- 172.21.15.16:0/3081576645 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99405e070 con 0x55deeaf30120 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 -- 172.21.15.16:0/3081576645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 msgr2=0x55deeaf36f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd994004aa0 tx=0x7fd994000f40 comp rx=0 tx=0).stop 2024-09-12T03:03:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 -- 172.21.15.16:0/3081576645 shutdown_connections 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf31840 0x55deeaf2ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf30b80 0x55deeaf2b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 --2- 172.21.15.16:0/3081576645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf36f70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 -- 172.21.15.16:0/3081576645 >> 172.21.15.16:0/3081576645 conn(0x55deeaef9310 msgr2=0x55deeaf36860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 -- 172.21.15.16:0/3081576645 shutdown_connections 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.311+0000 7fd9a5567740 1 -- 172.21.15.16:0/3081576645 wait complete. 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 -- start start 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 Processor -- start 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30b80 0x55deeaf55d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf31840 0x55deeafbbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ac810 con 0x55deeaf30b80 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ac140 con 0x55deeaf31840 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1a92c0 con 0x55deeaf30120 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a5567740 10 client.0 _ll_register_callbacks cb 0x7fd994046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.312+0000 7fd9a172b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46656/0 (socket says 172.21.15.16:46656) 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 -- 172.21.15.16:0/387153937 learned_addr learned my addr 172.21.15.16:0/387153937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf31840 0x55deeafbbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd99bfff640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30b80 0x55deeaf55d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 -- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf31840 msgr2=0x55deeafbbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf31840 0x55deeafbbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 -- 172.21.15.16:0/387153937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30b80 msgr2=0x55deeaf55d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30b80 0x55deeaf55d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 -- 172.21.15.16:0/387153937 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deeafbc4d0 con 0x55deeaf30120 2024-09-12T03:03:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd9a172b640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd99402fc20 tx=0x7fd994005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32c0182c77e679b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.313+0000 7fd97bfff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994048070 con 0x55deeaf30120 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd99403ad80 con 0x55deeaf30120 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99405f040 con 0x55deeaf30120 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd99405e040 con 0x55deeaf30120 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd9a5567740 10 client.28378 fetch_fsmap learned FSMap version 36 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55deeafbd6b0 con 0x55deeaf30120 2024-09-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.314+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99405f040 con 0x55deeaf30120 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fd99406f050 con 0x55deeaf30120 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 10 client.28378 fetch_fsmap finished waiting for FSMap version 36 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 10 client.28378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 20 client.28378 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeafbdc10 0x55deeafc0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 4 client.28378 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55deeae9dd70 con 0x55deeafbdc10 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.315+0000 7fd99bfff640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeafbdc10 0x55deeafc0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.316+0000 7fd97bfff640 10 client.28378 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.316+0000 7fd99bfff640 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeafbdc10 0x55deeafc0000 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd63b25d431ed4b server_cookie=aecb113c70876fe in_seq=0 out_seq=0 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd97bfff640 1 -- 172.21.15.16:0/387153937 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55deeae9dd70 con 0x55deeafbdc10 2024-09-12T03:03:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd97bfff640 10 client.28378 handle_command_reply: tid=0 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 client.28378 shutdown 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 msgr2=0x55deeaf55850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd99402fc20 tx=0x7fd994005f80 comp rx=0 tx=0).stop 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 shutdown_connections 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55deeafbdc10 0x55deeafc0000 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55deeaf31840 0x55deeafbbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55deeaf30b80 0x55deeaf55d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd97bfff640 0 client.28378 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 --2- 172.21.15.16:0/387153937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55deeaf30120 0x55deeaf55850 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 >> 172.21.15.16:0/387153937 conn(0x55deeaef9310 msgr2=0x55deeaf36f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 shutdown_connections 2024-09-12T03:03:41.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 1 -- 172.21.15.16:0/387153937 wait complete. 2024-09-12T03:03:41.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.318+0000 7fd9a5567740 20 client.28378 trim_cache size 0 max 16384 2024-09-12T03:03:41.322 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:03:41.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 msgr2=0x7fd98c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 0x7fd98c0a50c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd9900cd270 tx=0x7fd9900d4040 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 msgr2=0x7fd99c199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd99000e9f0 tx=0x7fd99000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 shutdown_connections 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd99c102ef0 0x7fd99c1a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd99c102570 0x7fd99c199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd98c0a2c00 0x7fd98c0a50c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 --2- 172.21.15.16:0/1405050466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd99c100e20 0x7fd99c199300 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 >> 172.21.15.16:0/1405050466 conn(0x7fd99c0fc7d0 msgr2=0x7fd99c106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 shutdown_connections 2024-09-12T03:03:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:41.319+0000 7fd9a318e640 1 -- 172.21.15.16:0/1405050466 wait complete. 2024-09-12T03:03:41.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:41 smithi016 ceph-mon[30925]: pgmap v810: 97 pgs: 97 active+clean; 584 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 701 op/s 2024-09-12T03:03:42.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 68836 35.36 MB/sec execute 43 sec latency 53.432 ms 2024-09-12T03:03:42.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:41 smithi033 ceph-mon[35092]: pgmap v810: 97 pgs: 97 active+clean; 584 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 701 op/s 2024-09-12T03:03:42.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:41 smithi184 ceph-mon[35321]: pgmap v810: 97 pgs: 97 active+clean; 584 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 701 op/s 2024-09-12T03:03:43.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 69501 35.33 MB/sec execute 44 sec latency 46.448 ms 2024-09-12T03:03:43.094 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:03:43.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:03:43.452 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:44.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 70133 35.18 MB/sec execute 45 sec latency 214.236 ms 2024-09-12T03:03:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 -- 172.21.15.16:0/3332133601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1036e0 msgr2=0x7f8a5c107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 --2- 172.21.15.16:0/3332133601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c107b90 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009930 tx=0x7f8a5002a420 comp rx=0 tx=0).stop 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 -- 172.21.15.16:0/3332133601 shutdown_connections 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 --2- 172.21.15.16:0/3332133601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 --2- 172.21.15.16:0/3332133601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c107b90 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 --2- 172.21.15.16:0/3332133601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 -- 172.21.15.16:0/3332133601 >> 172.21.15.16:0/3332133601 conn(0x7f8a5c0fc7d0 msgr2=0x7f8a5c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 -- 172.21.15.16:0/3332133601 shutdown_connections 2024-09-12T03:03:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.034+0000 7f8a61f11640 1 -- 172.21.15.16:0/3332133601 wait complete. 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.035+0000 7f8a61f11640 1 Processor -- start 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.035+0000 7f8a61f11640 1 -- start start 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c06c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36500/0 (socket says 172.21.15.16:36500) 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c112090 con 0x7f8a5c1082a0 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c111f10 con 0x7f8a5c102d30 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c112210 con 0x7f8a5c1036e0 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.036+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2438808523 learned_addr learned my addr 172.21.15.16:0/2438808523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5affd640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2438808523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1036e0 msgr2=0x7f8a5c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c06bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 msgr2=0x7f8a5c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c06c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c06ca40 con 0x7f8a5c102d30 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c06c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9f0 tx=0x7f8a4c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.037+0000 7f8a58ff9640 1 -- 172.21.15.16:0/2438808523 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00ccc0 con 0x7f8a5c102d30 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.038+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c1ad460 con 0x7f8a5c102d30 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.038+0000 7f8a58ff9640 1 -- 172.21.15.16:0/2438808523 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c010910 con 0x7f8a5c102d30 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.038+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c1ad9c0 con 0x7f8a5c102d30 2024-09-12T03:03:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.038+0000 7f8a58ff9640 1 -- 172.21.15.16:0/2438808523 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0188a0 con 0x7f8a5c102d30 2024-09-12T03:03:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.040+0000 7f8a642ea740 1 Processor -- start 2024-09-12T03:03:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 -- start start 2024-09-12T03:03:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f76f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f81800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557df8edc280 con 0x557df8f77480 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557df8e83b10 con 0x557df8f81d40 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a642ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557df8ed8240 con 0x557df8f76b60 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:36516/0 (socket says 172.21.15.16:36516) 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.041+0000 7f8a5affd640 1 -- 172.21.15.16:0/3226672954 learned_addr learned my addr 172.21.15.16:0/3226672954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 -- 172.21.15.16:0/3226672954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 msgr2=0x557df8f76f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f76f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 -- 172.21.15.16:0/3226672954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 msgr2=0x557df8f81800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f81800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 -- 172.21.15.16:0/3226672954 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557df8e901f0 con 0x557df8f81d40 2024-09-12T03:03:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a58ff9640 1 -- 172.21.15.16:0/2438808523 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8a4c010ac0 con 0x7f8a5c102d30 2024-09-12T03:03:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.042+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a50005ab0 tx=0x7f8a50005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c153c839c4e812 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.043+0000 7f8a41ffb640 1 -- 172.21.15.16:0/3226672954 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50040070 con 0x557df8f81d40 2024-09-12T03:03:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.043+0000 7f8a41ffb640 1 -- 172.21.15.16:0/3226672954 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a50004400 con 0x557df8f81d40 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.043+0000 7f8a41ffb640 1 -- 172.21.15.16:0/3226672954 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50035040 con 0x557df8f81d40 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.044+0000 7f8a58ff9640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 0x7f8a440a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.044+0000 7f8a58ff9640 1 -- 172.21.15.16:0/2438808523 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8a4c0cf3e0 con 0x7f8a5c102d30 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.044+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 0x7f8a440a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.044+0000 7f8a642ea740 1 -- 172.21.15.16:0/3226672954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 msgr2=0x557df8f84130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.044+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a50005ab0 tx=0x7f8a50005c00 comp rx=0 tx=0).stop 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 -- 172.21.15.16:0/3226672954 shutdown_connections 2024-09-12T03:03:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8f84130 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f81800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3226672954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f76f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 -- 172.21.15.16:0/3226672954 >> 172.21.15.16:0/3226672954 conn(0x557df8f730f0 msgr2=0x557df8f734d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 -- 172.21.15.16:0/3226672954 shutdown_connections 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 -- 172.21.15.16:0/3226672954 wait complete. 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a642ea740 1 -- start start 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.045+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 0x7f8a440a50c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005fd0 tx=0x7f8a54009900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 Processor -- start 2024-09-12T03:03:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 0x557df8f7a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f81d40 0x557df8f7ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557df8f88d20 con 0x557df8f81d40 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557df8e96f10 con 0x557df8f77480 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557df8f88ea0 con 0x557df8f76b60 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a642ea740 10 client.0 _ll_register_callbacks cb 0x557df8e5fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f81d40 0x557df8f7ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46690/0 (socket says 172.21.15.16:46690) 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2153385673 learned_addr learned my addr 172.21.15.16:0/2153385673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.046+0000 7f8a5affd640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 0x557df8f7a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 msgr2=0x557df8f7a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 0x557df8f7a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2153385673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f81d40 msgr2=0x557df8f7ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f81d40 0x557df8f7ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/2153385673 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557df8c4d6a0 con 0x557df8f76b60 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5affd640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 0x557df8f7a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0ccef0 tx=0x7f8a4c09b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333c283e97fc972 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a3affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:03:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0b1030 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a50004400 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8a4c099f50 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a4c0a0000 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a642ea740 10 client.28387 fetch_fsmap learned FSMap version 37 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.047+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557df8f99010 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c0b4e50 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f8a4c07aac0 con 0x557df8f76b60 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 10 client.28387 fetch_fsmap finished waiting for FSMap version 37 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 10 client.28387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 20 client.28387 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df8f9a360 0x557df8f9c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 4 client.28387 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557df8ee97f0 con 0x557df8f9a360 2024-09-12T03:03:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.048+0000 7f8a5affd640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df8f9a360 0x557df8f9c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.049+0000 7f8a5affd640 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df8f9a360 0x557df8f9c750 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e13afeca4deef8 server_cookie=e728bbe1a9ee72ee in_seq=0 out_seq=0 2024-09-12T03:03:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.049+0000 7f8a3affd640 10 client.28387 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a3affd640 1 -- 172.21.15.16:0/2153385673 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557df8ee97f0 con 0x557df8f9a360 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a3affd640 10 client.28387 handle_command_reply: tid=0 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 client.28387 shutdown 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 msgr2=0x557df8f815c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0ccef0 tx=0x7f8a4c09b880 comp rx=0 tx=0).stop 2024-09-12T03:03:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 shutdown_connections 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df8f9a360 0x557df8f9c750 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a3affd640 0 client.28387 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f81d40 0x557df8f7ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f77480 0x557df8f7a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 --2- 172.21.15.16:0/2153385673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f76b60 0x557df8f815c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.052+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 >> 172.21.15.16:0/2153385673 conn(0x557df8f730f0 msgr2=0x557df8f78e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.053+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 shutdown_connections 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.053+0000 7f8a642ea740 1 -- 172.21.15.16:0/2153385673 wait complete. 2024-09-12T03:03:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.053+0000 7f8a642ea740 20 client.28387 trim_cache size 0 max 16384 2024-09-12T03:03:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.072+0000 7f8a642ea740 1 Processor -- start 2024-09-12T03:03:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.072+0000 7f8a642ea740 1 -- start start 2024-09-12T03:03:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 0x557df8f79610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8fa24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b3b70 con 0x557df8f76b60 2024-09-12T03:03:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b5740 con 0x557df8f81d40 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a642ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b4240 con 0x557df8f77480 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 0x557df8f79610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:46706/0 (socket says 172.21.15.16:46706) 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.073+0000 7f8a5affd640 1 -- 172.21.15.16:0/3605322152 learned_addr learned my addr 172.21.15.16:0/3605322152 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8fa24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/3605322152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 msgr2=0x557df8fa1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/3605322152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 msgr2=0x557df8fa24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8fa24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 -- 172.21.15.16:0/3605322152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557df8f881c0 con 0x557df8f76b60 2024-09-12T03:03:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 0x557df8f79610 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0980b0 tx=0x7f8a4c09ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3dfdd8c4d1859ef server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a417fa640 1 -- 172.21.15.16:0/3605322152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0b1070 con 0x557df8f76b60 2024-09-12T03:03:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a417fa640 1 -- 172.21.15.16:0/3605322152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c0b4e50 con 0x557df8f76b60 2024-09-12T03:03:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.074+0000 7f8a417fa640 1 -- 172.21.15.16:0/3605322152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c09b300 con 0x557df8f76b60 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- 172.21.15.16:0/3605322152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 msgr2=0x557df8f79610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 0x557df8f79610 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0980b0 tx=0x7f8a4c09ab10 comp rx=0 tx=0).stop 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- 172.21.15.16:0/3605322152 shutdown_connections 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f81d40 0x557df8fa24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f77480 0x557df8fa1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3605322152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f76b60 0x557df8f79610 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- 172.21.15.16:0/3605322152 >> 172.21.15.16:0/3605322152 conn(0x557df8f730f0 msgr2=0x557df8f808c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- 172.21.15.16:0/3605322152 shutdown_connections 2024-09-12T03:03:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- 172.21.15.16:0/3605322152 wait complete. 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.075+0000 7f8a642ea740 1 -- start start 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 Processor -- start 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 0x557df8f96bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f81d40 0x557df8f939a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b5070 con 0x557df8f77480 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b49a0 con 0x557df8f76b60 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b1b70 con 0x557df8f81d40 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a642ea740 10 client.0 _ll_register_callbacks cb 0x7f8a4c010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 0x557df8f96bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a5affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38222/0 (socket says 172.21.15.16:38222) 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a5affd640 1 -- 172.21.15.16:0/3747828462 learned_addr learned my addr 172.21.15.16:0/3747828462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.076+0000 7f8a5bfff640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f81d40 0x557df8f939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 -- 172.21.15.16:0/3747828462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f81d40 msgr2=0x557df8f939a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f81d40 0x557df8f939a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 -- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 msgr2=0x557df8f96bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 0x557df8f96bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 -- 172.21.15.16:0/3747828462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557df8f940b0 con 0x557df8f77480 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 0x557df8f96bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a5affd640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006010 tx=0x7f8a50005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6694064265ff8830 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a3affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50057070 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8a50031c90 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c0b4e50 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a50035040 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a642ea740 10 client.19164 fetch_fsmap learned FSMap version 37 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.077+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557df90059a0 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c0b4e50 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f8a5005a030 con 0x557df8f77480 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 10 client.19164 fetch_fsmap finished waiting for FSMap version 37 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 10 client.19164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 20 client.19164 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df9005c80 0x557df9008070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 4 client.19164 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557df8ee3d00 con 0x557df9005c80 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.078+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df9005c80 0x557df9008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.079+0000 7f8a5b7fe640 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df9005c80 0x557df9008070 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b36051c1a3b06b9e server_cookie=cf5ff91a0e39b4b8 in_seq=0 out_seq=0 2024-09-12T03:03:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.079+0000 7f8a3affd640 10 client.19164 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a3affd640 1 -- 172.21.15.16:0/3747828462 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557df8ee3d00 con 0x557df9005c80 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a3affd640 10 client.19164 handle_command_reply: tid=0 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 client.19164 shutdown 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 msgr2=0x557df8f97110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006010 tx=0x7f8a50005d90 comp rx=0 tx=0).stop 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 shutdown_connections 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x557df9005c80 0x557df9008070 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557df8f81d40 0x557df8f939a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557df8f77480 0x557df8f97110 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 --2- 172.21.15.16:0/3747828462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557df8f76b60 0x557df8f96bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 >> 172.21.15.16:0/3747828462 conn(0x557df8f730f0 msgr2=0x557df8f79e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.081+0000 7f8a3affd640 0 client.19164 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 shutdown_connections 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a642ea740 1 -- 172.21.15.16:0/3747828462 wait complete. 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a642ea740 20 client.19164 trim_cache size 0 max 16384 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "88f4c0eb-5270-42db-ad56-785890587e0d", 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:03:44.087 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 msgr2=0x7f8a440a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 0x7f8a440a50c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005fd0 tx=0x7f8a54009900 comp rx=0 tx=0).stop 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 msgr2=0x7f8a5c073d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.082+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9f0 tx=0x7f8a4c00edf0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 shutdown_connections 2024-09-12T03:03:44.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8a440a2c00 0x7f8a440a50c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a5c1082a0 0x7f8a5c06c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a5c1036e0 0x7f8a5c06bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 --2- 172.21.15.16:0/2438808523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a5c102d30 0x7f8a5c073d10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 >> 172.21.15.16:0/2438808523 conn(0x7f8a5c0fc7d0 msgr2=0x7f8a5c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 shutdown_connections 2024-09-12T03:03:44.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:44.083+0000 7f8a61f11640 1 -- 172.21.15.16:0/2438808523 wait complete. 2024-09-12T03:03:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:43 smithi016 ceph-mon[30925]: pgmap v811: 97 pgs: 97 active+clean; 575 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-09-12T03:03:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:43 smithi033 ceph-mon[35092]: pgmap v811: 97 pgs: 97 active+clean; 575 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-09-12T03:03:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:43 smithi184 ceph-mon[35321]: pgmap v811: 97 pgs: 97 active+clean; 575 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-09-12T03:03:45.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 70948 35.67 MB/sec execute 46 sec latency 71.039 ms 2024-09-12T03:03:45.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:03:45.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:45.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:45.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:44 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:44 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.761 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:45.761 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.761 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:45.761 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:45.761 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:44 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 71549 35.53 MB/sec execute 47 sec latency 60.042 ms 2024-09-12T03:03:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:03:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: pgmap v812: 97 pgs: 97 active+clean; 580 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.7 MiB/s wr, 646 op/s 2024-09-12T03:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:45 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: pgmap v812: 97 pgs: 97 active+clean; 580 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.7 MiB/s wr, 646 op/s 2024-09-12T03:03:46.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:45 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: pgmap v812: 97 pgs: 97 active+clean; 580 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.7 MiB/s wr, 646 op/s 2024-09-12T03:03:46.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:45 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 -- 172.21.15.16:0/3023270733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 msgr2=0x7f911c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 --2- 172.21.15.16:0/3023270733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c1021c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f9118009900 tx=0x7f911802a410 comp rx=0 tx=0).stop 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 -- 172.21.15.16:0/3023270733 shutdown_connections 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 --2- 172.21.15.16:0/3023270733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c106f10 0x7f911c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 --2- 172.21.15.16:0/3023270733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c102790 0x7f911c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 --2- 172.21.15.16:0/3023270733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c1021c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 -- 172.21.15.16:0/3023270733 >> 172.21.15.16:0/3023270733 conn(0x7f911c0fd8c0 msgr2=0x7f911c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 -- 172.21.15.16:0/3023270733 shutdown_connections 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.590+0000 7f9123530640 1 -- 172.21.15.16:0/3023270733 wait complete. 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.591+0000 7f9123530640 1 Processor -- start 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.591+0000 7f9123530640 1 -- start start 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.591+0000 7f9123530640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.591+0000 7f9123530640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 0x7f911c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9123530640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c106f10 0x7f911c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9123530640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f911c111f70 con 0x7f911c101de0 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9123530640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f911c111df0 con 0x7f911c106f10 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9123530640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1120f0 con 0x7f911c102790 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51502/0 (socket says 172.21.15.16:51502) 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 -- 172.21.15.16:0/432920086 learned_addr learned my addr 172.21.15.16:0/432920086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9121acd640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c106f10 0x7f911c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9120acb640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 0x7f911c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 -- 172.21.15.16:0/432920086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 msgr2=0x7f911c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 0x7f911c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 -- 172.21.15.16:0/432920086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c106f10 msgr2=0x7f911c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c106f10 0x7f911c19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 -- 172.21.15.16:0/432920086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911c19f5e0 con 0x7f911c101de0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f9120acb640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 0x7f911c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.592+0000 7f91212cc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9118006010 tx=0x7f9118004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.593+0000 7f91127fc640 1 -- 172.21.15.16:0/432920086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911803f030 con 0x7f911c101de0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.593+0000 7f91127fc640 1 -- 172.21.15.16:0/432920086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9118004440 con 0x7f911c101de0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.593+0000 7f91127fc640 1 -- 172.21.15.16:0/432920086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118035070 con 0x7f911c101de0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.593+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911c19f870 con 0x7f911c101de0 2024-09-12T03:03:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.593+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911c19fd00 con 0x7f911c101de0 2024-09-12T03:03:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 Processor -- start 2024-09-12T03:03:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 -- start start 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff2908c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff2908c680 0x55ff2908ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff28fe72e0 con 0x55ff29081dc0 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff28f8ece0 con 0x55ff2908c680 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.596+0000 7f9125909740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff28fe32a0 con 0x55ff290814a0 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:44172/0 (socket says 172.21.15.16:44172) 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 -- 172.21.15.16:0/3258269132 learned_addr learned my addr 172.21.15.16:0/3258269132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9120acb640 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff2908c680 0x55ff2908ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f91127fc640 1 -- 172.21.15.16:0/432920086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9118042030 con 0x7f911c101de0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 -- 172.21.15.16:0/3258269132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff2908c680 msgr2=0x55ff2908ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff2908c680 0x55ff2908ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 -- 172.21.15.16:0/3258269132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 msgr2=0x55ff2908c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff2908c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 -- 172.21.15.16:0/3258269132 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff28f73440 con 0x55ff290814a0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.597+0000 7f9121acd640 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f50 tx=0x7f91140118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14af4e9bb019ed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.598+0000 7f91077fe640 1 -- 172.21.15.16:0/3258269132 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114004d20 con 0x55ff290814a0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.598+0000 7f91077fe640 1 -- 172.21.15.16:0/3258269132 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9114004050 con 0x55ff290814a0 2024-09-12T03:03:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.598+0000 7f91077fe640 1 -- 172.21.15.16:0/3258269132 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114005280 con 0x55ff290814a0 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.598+0000 7f91127fc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 0x7f91080a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.598+0000 7f91212cc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 0x7f91080a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- 172.21.15.16:0/3258269132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 msgr2=0x55ff29081880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f91127fc640 1 -- 172.21.15.16:0/432920086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f91180ef2a0 con 0x7f911c101de0 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f50 tx=0x7f91140118d0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- 172.21.15.16:0/3258269132 shutdown_connections 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff2908c680 0x55ff2908ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff2908c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 --2- 172.21.15.16:0/3258269132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff290814a0 0x55ff29081880 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- 172.21.15.16:0/3258269132 >> 172.21.15.16:0/3258269132 conn(0x55ff2907da30 msgr2=0x55ff2907de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- 172.21.15.16:0/3258269132 shutdown_connections 2024-09-12T03:03:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f91212cc640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 0x7f91080a5190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f91180b85a0 tx=0x7f91180b7f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- 172.21.15.16:0/3258269132 wait complete. 2024-09-12T03:03:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 -- start start 2024-09-12T03:03:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.599+0000 7f9125909740 1 Processor -- start 2024-09-12T03:03:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff2908f2f0 con 0x55ff29081dc0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff28fa2050 con 0x55ff290814a0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff2908f470 con 0x55ff2908c680 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9125909740 10 client.0 _ll_register_callbacks cb 0x55ff28fe2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9120acb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9121acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9121acd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41422/0 (socket says 172.21.15.16:41422) 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9121acd640 1 -- 172.21.15.16:0/511420385 learned_addr learned my addr 172.21.15.16:0/511420385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9120acb640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51506/0 (socket says 172.21.15.16:51506) 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.600+0000 7f9121acd640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 -- 172.21.15.16:0/511420385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 msgr2=0x55ff290897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290897b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 -- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 msgr2=0x55ff2908bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 -- 172.21.15.16:0/511420385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff28d583c0 con 0x55ff29081dc0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9120acb640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f911c0ff750 tx=0x7f910c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbff072dd8676b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9104ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c012110 con 0x55ff29081dc0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9114004050 con 0x55ff29081dc0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f910c004050 con 0x55ff29081dc0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f910c0122b0 con 0x55ff29081dc0 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9125909740 10 client.19173 fetch_fsmap learned FSMap version 38 2024-09-12T03:03:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.601+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ff2909ad80 con 0x55ff29081dc0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9114004050 con 0x55ff29081dc0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 38) ==== 8374+0+0 (secure 0 0 0) 0x7f910c0104c0 con 0x55ff29081dc0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 10 client.19173 fetch_fsmap finished waiting for FSMap version 38 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 10 client.19173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 20 client.19173 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff2909e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 4 client.19173 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff28ff4850 con 0x55ff2909c1e0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.602+0000 7f9121acd640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff2909e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.603+0000 7f9104ff9640 10 client.19173 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.603+0000 7f9121acd640 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff2909e600 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8430eda6bfc338c4 server_cookie=15958d5d44920d52 in_seq=0 out_seq=0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9104ff9640 1 -- 172.21.15.16:0/511420385 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ff28ff4850 con 0x55ff2909c1e0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9104ff9640 10 client.19173 handle_command_reply: tid=0 2024-09-12T03:03:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 client.19173 shutdown 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 msgr2=0x55ff29089270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f911c0ff750 tx=0x7f910c00f790 comp rx=0 tx=0).stop 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 shutdown_connections 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff2909e600 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9104ff9640 0 client.19173 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290897b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff29089270 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 --2- 172.21.15.16:0/511420385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 >> 172.21.15.16:0/511420385 conn(0x55ff2907da30 msgr2=0x55ff29083760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 shutdown_connections 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 1 -- 172.21.15.16:0/511420385 wait complete. 2024-09-12T03:03:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.604+0000 7f9125909740 20 client.19173 trim_cache size 0 max 16384 2024-09-12T03:03:46.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.623+0000 7f9125909740 1 Processor -- start 2024-09-12T03:03:46.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.623+0000 7f9125909740 1 -- start start 2024-09-12T03:03:46.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.623+0000 7f9125909740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.623+0000 7f9125909740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9125909740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290a4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9125909740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1af750 con 0x55ff29081dc0 2024-09-12T03:03:46.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9125909740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1b1320 con 0x55ff290814a0 2024-09-12T03:03:46.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9125909740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1afe20 con 0x55ff2908c680 2024-09-12T03:03:46.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51510/0 (socket says 172.21.15.16:51510) 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 -- 172.21.15.16:0/491730205 learned_addr learned my addr 172.21.15.16:0/491730205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9121acd640 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9121acd640 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 -- 172.21.15.16:0/491730205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 msgr2=0x55ff290a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 -- 172.21.15.16:0/491730205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 msgr2=0x55ff2908b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 -- 172.21.15.16:0/491730205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2909e970 con 0x55ff29081dc0 2024-09-12T03:03:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9120acb640 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f910c010ad0 tx=0x7f910c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a3a35ae5e89b82 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9106ffd640 1 -- 172.21.15.16:0/491730205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c010b50 con 0x55ff29081dc0 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9106ffd640 1 -- 172.21.15.16:0/491730205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f910c010cf0 con 0x55ff29081dc0 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.624+0000 7f9106ffd640 1 -- 172.21.15.16:0/491730205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c003850 con 0x55ff29081dc0 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 -- 172.21.15.16:0/491730205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 msgr2=0x55ff290a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f910c010ad0 tx=0x7f910c009c00 comp rx=0 tx=0).stop 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 -- 172.21.15.16:0/491730205 shutdown_connections 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff29081dc0 0x55ff290a3ed0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 --2- 172.21.15.16:0/491730205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff290814a0 0x55ff2908b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 -- 172.21.15.16:0/491730205 >> 172.21.15.16:0/491730205 conn(0x55ff2907da30 msgr2=0x55ff2907cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 -- 172.21.15.16:0/491730205 shutdown_connections 2024-09-12T03:03:46.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.625+0000 7f9125909740 1 -- 172.21.15.16:0/491730205 wait complete. 2024-09-12T03:03:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 -- start start 2024-09-12T03:03:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 Processor -- start 2024-09-12T03:03:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 0x55ff29094c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 0x55ff29091f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290924c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1b0c50 con 0x55ff290814a0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1b0580 con 0x55ff29081dc0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f911c1ad730 con 0x55ff2908c680 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.626+0000 7f9125909740 10 client.0 _ll_register_callbacks cb 0x7f910c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9121acd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 0x55ff29094c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 0x55ff29091f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9121acd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 0x55ff29094c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9121acd640 1 -- 172.21.15.16:0/389997723 learned_addr learned my addr 172.21.15.16:0/389997723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9121acd640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 -- 172.21.15.16:0/389997723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 msgr2=0x55ff290924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290924c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 -- 172.21.15.16:0/389997723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 msgr2=0x55ff29094c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 0x55ff29094c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 -- 172.21.15.16:0/389997723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff29092ab0 con 0x55ff29081dc0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9120acb640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 0x55ff29091f80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ff29084fc0 tx=0x7f910c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e3fa766514dc37 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9104ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.627+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c005670 con 0x55ff29081dc0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f910c010cf0 con 0x55ff29081dc0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f910c0035f0 con 0x55ff29081dc0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f910c010b00 con 0x55ff29081dc0 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 10 client.28490 fetch_fsmap learned FSMap version 38 2024-09-12T03:03:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ff29106cc0 con 0x55ff29081dc0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f910c010cf0 con 0x55ff29081dc0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 38) ==== 8374+0+0 (secure 0 0 0) 0x7f910c01c870 con 0x55ff29081dc0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 10 client.28490 fetch_fsmap finished waiting for FSMap version 38 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 10 client.28490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 20 client.28490 populate_metadata read hostname 'smithi016' 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff29109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 4 client.28490 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.628+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ff28feed60 con 0x55ff2909c1e0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.629+0000 7f9121acd640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff29109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.629+0000 7f9121acd640 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff29109300 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321885bc8786d4ac server_cookie=25411fa918074b5d in_seq=0 out_seq=0 2024-09-12T03:03:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.629+0000 7f9104ff9640 10 client.28490 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:46.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9104ff9640 1 -- 172.21.15.16:0/389997723 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ff28feed60 con 0x55ff2909c1e0 2024-09-12T03:03:46.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9104ff9640 10 client.28490 handle_command_reply: tid=0 2024-09-12T03:03:46.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 client.28490 shutdown 2024-09-12T03:03:46.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 msgr2=0x55ff29091f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 0x55ff29091f80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ff29084fc0 tx=0x7f910c005270 comp rx=0 tx=0).stop 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 shutdown_connections 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff2909c1e0 0x55ff29109300 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff2908c680 0x55ff290924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff29081dc0 0x55ff29091f80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9104ff9640 0 client.28490 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 --2- 172.21.15.16:0/389997723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff290814a0 0x55ff29094c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 >> 172.21.15.16:0/389997723 conn(0x55ff2907da30 msgr2=0x55ff29088b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 shutdown_connections 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 1 -- 172.21.15.16:0/389997723 wait complete. 2024-09-12T03:03:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.631+0000 7f9125909740 20 client.28490 trim_cache size 0 max 16384 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "88f4c0eb-5270-42db-ad56-785890587e0d": { 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "88f4c0eb-5270-42db-ad56-785890587e0d", 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:03:46.637 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:03:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 msgr2=0x7f91080a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 0x7f91080a5190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f91180b85a0 tx=0x7f91180b7f40 comp rx=0 tx=0).stop 2024-09-12T03:03:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 msgr2=0x7f911c19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:03:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9118006010 tx=0x7f9118004040 comp rx=0 tx=0).stop 2024-09-12T03:03:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 shutdown_connections 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f911c106f10 0x7f911c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f911c102790 0x7f911c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f91080a2cd0 0x7f91080a5190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 --2- 172.21.15.16:0/432920086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911c101de0 0x7f911c19a630 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.632+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 >> 172.21.15.16:0/432920086 conn(0x7f911c0fd8c0 msgr2=0x7f911c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.633+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 shutdown_connections 2024-09-12T03:03:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:03:46.633+0000 7f9123530640 1 -- 172.21.15.16:0/432920086 wait complete. 2024-09-12T03:03:47.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72205 35.47 MB/sec execute 48 sec latency 77.908 ms 2024-09-12T03:03:47.403 INFO:tasks.cephfs.filesystem:scrub status for tag:88f4c0eb-5270-42db-ad56-785890587e0d - {'path': '/', 'tag': '88f4c0eb-5270-42db-ad56-785890587e0d', 'options': 'recursive,force'} 2024-09-12T03:03:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72974 35.64 MB/sec execute 49 sec latency 75.406 ms 2024-09-12T03:03:48.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:47 smithi033 ceph-mon[35092]: pgmap v813: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.6 MiB/s wr, 716 op/s 2024-09-12T03:03:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:47 smithi184 ceph-mon[35321]: pgmap v813: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.6 MiB/s wr, 716 op/s 2024-09-12T03:03:48.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:47 smithi016 ceph-mon[30925]: pgmap v813: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.6 MiB/s wr, 716 op/s 2024-09-12T03:03:49.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 73364 35.32 MB/sec execute 50 sec latency 248.494 ms 2024-09-12T03:03:50.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 74008 35.22 MB/sec execute 51 sec latency 55.622 ms 2024-09-12T03:03:50.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:49 smithi033 ceph-mon[35092]: pgmap v814: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.1 MiB/s wr, 603 op/s 2024-09-12T03:03:50.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:49 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:49 smithi184 ceph-mon[35321]: pgmap v814: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.1 MiB/s wr, 603 op/s 2024-09-12T03:03:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:49 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:49 smithi016 ceph-mon[30925]: pgmap v814: 97 pgs: 97 active+clean; 587 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.1 MiB/s wr, 603 op/s 2024-09-12T03:03:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:49 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:03:51.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 74910 35.53 MB/sec execute 52 sec latency 52.913 ms 2024-09-12T03:03:52.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 75694 35.77 MB/sec execute 53 sec latency 45.717 ms 2024-09-12T03:03:52.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:51 smithi033 ceph-mon[35092]: pgmap v815: 97 pgs: 97 active+clean; 598 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 747 op/s 2024-09-12T03:03:52.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:51 smithi184 ceph-mon[35321]: pgmap v815: 97 pgs: 97 active+clean; 598 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 747 op/s 2024-09-12T03:03:52.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:51 smithi016 ceph-mon[30925]: pgmap v815: 97 pgs: 97 active+clean; 598 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 747 op/s 2024-09-12T03:03:53.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 76288 35.66 MB/sec execute 54 sec latency 90.862 ms 2024-09-12T03:03:54.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 76772 35.46 MB/sec execute 55 sec latency 273.381 ms 2024-09-12T03:03:54.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:53 smithi033 ceph-mon[35092]: pgmap v816: 97 pgs: 97 active+clean; 599 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 711 op/s 2024-09-12T03:03:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:53 smithi184 ceph-mon[35321]: pgmap v816: 97 pgs: 97 active+clean; 599 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 711 op/s 2024-09-12T03:03:54.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:53 smithi016 ceph-mon[30925]: pgmap v816: 97 pgs: 97 active+clean; 599 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 711 op/s 2024-09-12T03:03:55.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77498 35.50 MB/sec execute 56 sec latency 82.275 ms 2024-09-12T03:03:56.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78183 35.50 MB/sec execute 57 sec latency 68.257 ms 2024-09-12T03:03:56.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:55 smithi033 ceph-mon[35092]: pgmap v817: 97 pgs: 97 active+clean; 607 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 686 op/s 2024-09-12T03:03:56.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:55 smithi184 ceph-mon[35321]: pgmap v817: 97 pgs: 97 active+clean; 607 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 686 op/s 2024-09-12T03:03:56.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:55 smithi016 ceph-mon[30925]: pgmap v817: 97 pgs: 97 active+clean; 607 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 686 op/s 2024-09-12T03:03:57.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78893 35.45 MB/sec execute 58 sec latency 49.087 ms 2024-09-12T03:03:58.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 79812 35.88 MB/sec execute 59 sec latency 36.201 ms 2024-09-12T03:03:58.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:57 smithi033 ceph-mon[35092]: pgmap v818: 97 pgs: 97 active+clean; 614 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-09-12T03:03:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:57 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:57 smithi184 ceph-mon[35321]: pgmap v818: 97 pgs: 97 active+clean; 614 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-09-12T03:03:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:57 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:57 smithi016 ceph-mon[30925]: pgmap v818: 97 pgs: 97 active+clean; 614 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-09-12T03:03:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:57 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80349 35.71 MB/sec execute 60 sec latency 203.450 ms 2024-09-12T03:03:59.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:59.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:58 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:03:59.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:59.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:58 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:03:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:03:59.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:58 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:00.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80928 35.59 MB/sec execute 61 sec latency 83.287 ms 2024-09-12T03:04:00.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:03:59 smithi033 ceph-mon[35092]: pgmap v819: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.2 MiB/s wr, 632 op/s 2024-09-12T03:04:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:03:59 smithi184 ceph-mon[35321]: pgmap v819: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.2 MiB/s wr, 632 op/s 2024-09-12T03:04:00.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:03:59 smithi016 ceph-mon[30925]: pgmap v819: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.2 MiB/s wr, 632 op/s 2024-09-12T03:04:01.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 81630 35.65 MB/sec execute 62 sec latency 54.520 ms 2024-09-12T03:04:02.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 82344 35.75 MB/sec execute 63 sec latency 50.173 ms 2024-09-12T03:04:02.209 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:01 smithi016 ceph-mon[30925]: pgmap v820: 97 pgs: 97 active+clean; 609 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-09-12T03:04:02.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:01 smithi033 ceph-mon[35092]: pgmap v820: 97 pgs: 97 active+clean; 609 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-09-12T03:04:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:01 smithi184 ceph-mon[35321]: pgmap v820: 97 pgs: 97 active+clean; 609 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-09-12T03:04:03.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83130 35.75 MB/sec execute 64 sec latency 42.853 ms 2024-09-12T03:04:03.326 INFO:tasks.quiescer.fs.[cephfs]:Total duration (48.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T03:04:03.326 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.25 2024-09-12T03:04:03.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:04:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:04.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83604 35.63 MB/sec execute 65 sec latency 201.570 ms 2024-09-12T03:04:04.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:03 smithi016 ceph-mon[30925]: pgmap v821: 97 pgs: 97 active+clean; 604 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 650 op/s 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 -- 172.21.15.16:0/1337828458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 msgr2=0x7f757c1043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 --2- 172.21.15.16:0/1337828458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c1043d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f7578009900 tx=0x7f757802a3f0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 -- 172.21.15.16:0/1337828458 shutdown_connections 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 --2- 172.21.15.16:0/1337828458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f757c109120 0x7f757c110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 --2- 172.21.15.16:0/1337828458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c1049a0 0x7f757c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.199+0000 7f7583e26640 1 --2- 172.21.15.16:0/1337828458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c1043d0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.200+0000 7f7583e26640 1 -- 172.21.15.16:0/1337828458 >> 172.21.15.16:0/1337828458 conn(0x7f757c0ffab0 msgr2=0x7f757c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.200+0000 7f7583e26640 1 -- 172.21.15.16:0/1337828458 shutdown_connections 2024-09-12T03:04:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.200+0000 7f7583e26640 1 -- 172.21.15.16:0/1337828458 wait complete. 2024-09-12T03:04:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 Processor -- start 2024-09-12T03:04:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 -- start start 2024-09-12T03:04:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f757c1049a0 0x7f757c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c109120 0x7f757c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f757c114170 con 0x7f757c103ff0 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f757c113ff0 con 0x7f757c1049a0 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7583e26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f757c1142f0 con 0x7f757c109120 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7581bc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7581bc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42744/0 (socket says 172.21.15.16:42744) 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f7581bc2640 1 -- 172.21.15.16:0/1154409543 learned_addr learned my addr 172.21.15.16:0/1154409543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.201+0000 7f75823c3640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c109120 0x7f757c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 -- 172.21.15.16:0/1154409543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c109120 msgr2=0x7f757c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c109120 0x7f757c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 -- 172.21.15.16:0/1154409543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f757c1049a0 msgr2=0x7f757c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f757c1049a0 0x7f757c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 -- 172.21.15.16:0/1154409543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757c19f6e0 con 0x7f757c103ff0 2024-09-12T03:04:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.202+0000 7f7581bc2640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7578004770 tx=0x7f7578003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.203+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757803f070 con 0x7f757c103ff0 2024-09-12T03:04:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.203+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7578031ca0 con 0x7f757c103ff0 2024-09-12T03:04:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.203+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757c19f970 con 0x7f757c103ff0 2024-09-12T03:04:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.203+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f757c1a72b0 con 0x7f757c103ff0 2024-09-12T03:04:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.204+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578035030 con 0x7f757c103ff0 2024-09-12T03:04:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.205+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f757c1054d0 con 0x7f757c103ff0 2024-09-12T03:04:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.207+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7578042070 con 0x7f757c103ff0 2024-09-12T03:04:04.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.207+0000 7f7572ffd640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 0x7f75680a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.208+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f75780ef960 con 0x7f757c103ff0 2024-09-12T03:04:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.208+0000 7f75813c1640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 0x7f75680a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.208+0000 7f75813c1640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 0x7f75680a4f90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f756c005d00 tx=0x7f756c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:04.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.212+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75780bc8d0 con 0x7f757c103ff0 2024-09-12T03:04:04.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:03 smithi033 ceph-mon[35092]: pgmap v821: 97 pgs: 97 active+clean; 604 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 650 op/s 2024-09-12T03:04:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:03 smithi184 ceph-mon[35321]: pgmap v821: 97 pgs: 97 active+clean; 604 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 650 op/s 2024-09-12T03:04:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.486+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f757c104400 con 0x7f757c103ff0 2024-09-12T03:04:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.487+0000 7f7572ffd640 1 -- 172.21.15.16:0/1154409543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f757c104400 con 0x7f757c103ff0 2024-09-12T03:04:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:04:04.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:04:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 msgr2=0x7f75680a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 0x7f75680a4f90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f756c005d00 tx=0x7f756c005c50 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 msgr2=0x7f757c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7578004770 tx=0x7f7578003c10 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 shutdown_connections 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f757c109120 0x7f757c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f757c1049a0 0x7f757c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f75680a2ad0 0x7f75680a4f90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 --2- 172.21.15.16:0/1154409543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757c103ff0 0x7f757c19a730 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 >> 172.21.15.16:0/1154409543 conn(0x7f757c0ffab0 msgr2=0x7f757c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 shutdown_connections 2024-09-12T03:04:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:04.491+0000 7f7583e26640 1 -- 172.21.15.16:0/1154409543 wait complete. 2024-09-12T03:04:05.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84271 35.70 MB/sec execute 66 sec latency 214.138 ms 2024-09-12T03:04:05.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:04 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1154409543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:05.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:04 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1154409543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:05.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:04 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1154409543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:05.522 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 144.25 --await-for 45.0' 2024-09-12T03:04:05.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 144.25 --await-for 45.0 2024-09-12T03:04:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:06.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84981 35.71 MB/sec execute 67 sec latency 46.579 ms 2024-09-12T03:04:06.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:05 smithi016 ceph-mon[30925]: pgmap v822: 97 pgs: 97 active+clean; 606 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.9 MiB/s wr, 656 op/s 2024-09-12T03:04:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:05 smithi033 ceph-mon[35092]: pgmap v822: 97 pgs: 97 active+clean; 606 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.9 MiB/s wr, 656 op/s 2024-09-12T03:04:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:05 smithi184 ceph-mon[35321]: pgmap v822: 97 pgs: 97 active+clean; 606 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.9 MiB/s wr, 656 op/s 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 -- 172.21.15.16:0/2999399047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202410e4d0 msgr2=0x7f20241108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 --2- 172.21.15.16:0/2999399047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202410e4d0 0x7f20241108e0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f201c009ca0 tx=0x7f201c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 -- 172.21.15.16:0/2999399047 shutdown_connections 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 --2- 172.21.15.16:0/2999399047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202410e4d0 0x7f20241108e0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 --2- 172.21.15.16:0/2999399047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2024101360 0x7f202410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 --2- 172.21.15.16:0/2999399047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20241009b0 0x7f2024100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 -- 172.21.15.16:0/2999399047 >> 172.21.15.16:0/2999399047 conn(0x7f20240fc7d0 msgr2=0x7f20240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 -- 172.21.15.16:0/2999399047 shutdown_connections 2024-09-12T03:04:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.353+0000 7f202a1a0640 1 -- 172.21.15.16:0/2999399047 wait complete. 2024-09-12T03:04:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.354+0000 7f202a1a0640 1 Processor -- start 2024-09-12T03:04:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.354+0000 7f202a1a0640 1 -- start start 2024-09-12T03:04:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 0x7f202419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f202410e4d0 0x7f202419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2024114200 con 0x7f2024101360 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2024114080 con 0x7f20241009b0 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f202a1a0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2024114380 con 0x7f202410e4d0 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 0x7f202419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43960/0 (socket says 172.21.15.16:43960) 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 -- 172.21.15.16:0/3573098194 learned_addr learned my addr 172.21.15.16:0/3573098194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f2023fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f202410e4d0 0x7f202419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 -- 172.21.15.16:0/3573098194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f202410e4d0 msgr2=0x7f202419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f202410e4d0 0x7f202419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 -- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 msgr2=0x7f202419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 0x7f202419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.355+0000 7f20237fe640 1 -- 172.21.15.16:0/3573098194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202419f690 con 0x7f20241009b0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f2022ffd640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 0x7f202419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f20237fe640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f201400e9f0 tx=0x7f201400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f2020ff9640 1 -- 172.21.15.16:0/3573098194 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201400cd80 con 0x7f20241009b0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f2020ff9640 1 -- 172.21.15.16:0/3573098194 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2014010910 con 0x7f20241009b0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202419f980 con 0x7f20241009b0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.356+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2024102b30 con 0x7f20241009b0 2024-09-12T03:04:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.357+0000 7f2020ff9640 1 -- 172.21.15.16:0/3573098194 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014018990 con 0x7f20241009b0 2024-09-12T03:04:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.360+0000 7f2020ff9640 1 -- 172.21.15.16:0/3573098194 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2014018b30 con 0x7f20241009b0 2024-09-12T03:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f2020ff9640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 0x7f200c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 Processor -- start 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 -- start start 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f2020ff9640 1 -- 172.21.15.16:0/3573098194 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f20140cf200 con 0x7f20241009b0 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606cc52a160 con 0x5606cc5c6170 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5606cc4d1720 con 0x5606cc5c5850 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.361+0000 7f202c579740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606cc526120 con 0x5606cc5d0a30 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43976/0 (socket says 172.21.15.16:43976) 2024-09-12T03:04:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 -- 172.21.15.16:0/910172359 learned_addr learned my addr 172.21.15.16:0/910172359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f2022ffd640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 0x7f200c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 -- 172.21.15.16:0/910172359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 msgr2=0x5606cc5d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 -- 172.21.15.16:0/910172359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 msgr2=0x5606cc5d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f20237fe640 1 -- 172.21.15.16:0/910172359 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606cc29bba0 con 0x5606cc5c5850 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.362+0000 7f2022ffd640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 0x7f200c0a5210 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f202410ed80 tx=0x7f2018009b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.363+0000 7f20237fe640 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2014097f20 tx=0x7f201409bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd24351a33a6b85d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.363+0000 7f2009ffb640 1 -- 172.21.15.16:0/910172359 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20140b4070 con 0x5606cc5c5850 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.363+0000 7f2009ffb640 1 -- 172.21.15.16:0/910172359 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20140ccdc0 con 0x5606cc5c5850 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.363+0000 7f2009ffb640 1 -- 172.21.15.16:0/910172359 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20140ce070 con 0x5606cc5c5850 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- 172.21.15.16:0/910172359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 msgr2=0x5606cc5c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2014097f20 tx=0x7f201409bb80 comp rx=0 tx=0).stop 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- 172.21.15.16:0/910172359 shutdown_connections 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 --2- 172.21.15.16:0/910172359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5c5c30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- 172.21.15.16:0/910172359 >> 172.21.15.16:0/910172359 conn(0x5606cc5c1de0 msgr2=0x5606cc5c21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- 172.21.15.16:0/910172359 shutdown_connections 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- 172.21.15.16:0/910172359 wait complete. 2024-09-12T03:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.364+0000 7f202c579740 1 -- start start 2024-09-12T03:04:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 Processor -- start 2024-09-12T03:04:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5606cc5d3600 con 0x5606cc5c6170 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5606cc4b2900 con 0x5606cc5c5850 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606cc5d3780 con 0x5606cc5d0a30 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f202c579740 10 client.0 _ll_register_callbacks cb 0x5606cc4b0780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f20237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:43984/0 (socket says 172.21.15.16:43984) 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f20237fe640 1 -- 172.21.15.16:0/2236768541 learned_addr learned my addr 172.21.15.16:0/2236768541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.365+0000 7f2023fff640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 -- 172.21.15.16:0/2236768541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 msgr2=0x5606cc5ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 -- 172.21.15.16:0/2236768541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 msgr2=0x5606cc5cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 -- 172.21.15.16:0/2236768541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606cc4f2c50 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f20237fe640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f201400c6a0 tx=0x7f2014018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57a6856c0c6a35b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f2002ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20140ce030 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2014010640 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20140ccdc0 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f20140b4040 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f202c579740 10 client.28505 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5606cc5df320 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.366+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20140ccdc0 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f201409a980 con 0x5606cc5c5850 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 10 client.28505 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 10 client.28505 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 20 client.28505 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc5e0670 0x5606cc5e2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 4 client.28505 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606cc537890 con 0x5606cc5e0670 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f2022ffd640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc5e0670 0x5606cc5e2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f2022ffd640 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc5e0670 0x5606cc5e2a60 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2d132af3d6b39d8 server_cookie=d4265c3cf44d4abc in_seq=0 out_seq=0 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.367+0000 7f2002ffd640 10 client.28505 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f2002ffd640 1 -- 172.21.15.16:0/2236768541 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5606cc537890 con 0x5606cc5e0670 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f2002ffd640 10 client.28505 handle_command_reply: tid=0 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 client.28505 shutdown 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 msgr2=0x5606cc5cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f201400c6a0 tx=0x7f2014018420 comp rx=0 tx=0).stop 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 shutdown_connections 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc5e0670 0x5606cc5e2a60 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 --2- 172.21.15.16:0/2236768541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c5850 0x5606cc5cd930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 >> 172.21.15.16:0/2236768541 conn(0x5606cc5c1de0 msgr2=0x5606cc5c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.368+0000 7f2002ffd640 0 client.28505 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.369+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 shutdown_connections 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.369+0000 7f202c579740 1 -- 172.21.15.16:0/2236768541 wait complete. 2024-09-12T03:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.369+0000 7f202c579740 20 client.28505 trim_cache size 0 max 16384 2024-09-12T03:04:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 Processor -- start 2024-09-12T03:04:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 -- start start 2024-09-12T03:04:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 0x5606cc5e8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5e8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 0x5606cc5cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f201000e400 con 0x5606cc5c6170 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f201000ff40 con 0x5606cc5d0a30 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.389+0000 7f202c579740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f201000ead0 con 0x5606cc5c5850 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5e8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33146/0 (socket says 172.21.15.16:33146) 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f2022ffd640 1 -- 172.21.15.16:0/3695210876 learned_addr learned my addr 172.21.15.16:0/3695210876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 0x5606cc5e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f2023fff640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 0x5606cc5cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 -- 172.21.15.16:0/3695210876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 msgr2=0x5606cc5cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 0x5606cc5cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 -- 172.21.15.16:0/3695210876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 msgr2=0x5606cc5e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 -- 172.21.15.16:0/3695210876 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606cc5bd3f0 con 0x5606cc5c5850 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f2023fff640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 0x5606cc5cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20237fe640 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 0x5606cc5e8150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f201409ad70 tx=0x7f201409bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ccff98bc0d125ad server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.390+0000 7f20097fa640 1 -- 172.21.15.16:0/3695210876 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201409af10 con 0x5606cc5c5850 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f20097fa640 1 -- 172.21.15.16:0/3695210876 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2014010640 con 0x5606cc5c5850 2024-09-12T03:04:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f20097fa640 1 -- 172.21.15.16:0/3695210876 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014010d20 con 0x5606cc5c5850 2024-09-12T03:04:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 -- 172.21.15.16:0/3695210876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 msgr2=0x5606cc5e8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 0x5606cc5e8150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f201409ad70 tx=0x7f201409bb80 comp rx=0 tx=0).stop 2024-09-12T03:04:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 -- 172.21.15.16:0/3695210876 shutdown_connections 2024-09-12T03:04:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5d0a30 0x5606cc5cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c6170 0x5606cc5e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 --2- 172.21.15.16:0/3695210876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5c5850 0x5606cc5e8150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 -- 172.21.15.16:0/3695210876 >> 172.21.15.16:0/3695210876 conn(0x5606cc5c1de0 msgr2=0x5606cc5c0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 -- 172.21.15.16:0/3695210876 shutdown_connections 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.391+0000 7f202c579740 1 -- 172.21.15.16:0/3695210876 wait complete. 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 -- start start 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 Processor -- start 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 0x5606cc5d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f201000f870 con 0x5606cc5c5850 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f201000f1a0 con 0x5606cc5c6170 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.392+0000 7f202c579740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f201000c3e0 con 0x5606cc5d0a30 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f202c579740 10 client.0 _ll_register_callbacks cb 0x7f20140b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2023fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:44002/0 (socket says 172.21.15.16:44002) 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 -- 172.21.15.16:0/535440098 learned_addr learned my addr 172.21.15.16:0/535440098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f20237fe640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 0x5606cc5d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 -- 172.21.15.16:0/535440098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 msgr2=0x5606cc5d5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 -- 172.21.15.16:0/535440098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 msgr2=0x5606cc5d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 0x5606cc5d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 -- 172.21.15.16:0/535440098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606cc5c2a70 con 0x5606cc5c6170 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f20237fe640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 0x5606cc5d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2023fff640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2022ffd640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2018011b60 tx=0x7f2018013070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbea02b1d1094f85 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.393+0000 7f2002ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018011120 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2014010640 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f20180112c0 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f20180220d0 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f202c579740 10 client.28511 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5606cc64b4f0 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2014010640 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.394+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f2018032030 con 0x5606cc5c6170 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 10 client.28511 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 10 client.28511 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 20 client.28511 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc64bcc0 0x5606cc64e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 4 client.28511 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.25, "await_for": 45.0}] 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.25, "await_for": 45.0}) -- 0x5606cc531be0 con 0x5606cc64bcc0 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f20237fe640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc64bcc0 0x5606cc64e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f2002ffd640 10 client.28511 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:06.395+0000 7f20237fe640 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc64bcc0 0x5606cc64e0e0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da96a452c5a569b3 server_cookie=2dfbbe769d3c6438 in_seq=0 out_seq=0 2024-09-12T03:04:07.012 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85254 35.41 MB/sec execute 68 sec latency 589.308 ms 2024-09-12T03:04:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.030+0000 7f2002ffd640 1 -- 172.21.15.16:0/535440098 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5606cc531be0 con 0x5606cc64bcc0 2024-09-12T03:04:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.030+0000 7f2002ffd640 10 client.28511 handle_command_reply: tid=0 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.030+0000 7f202c579740 1 client.28511 shutdown 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 msgr2=0x5606cc5d4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2018011b60 tx=0x7f2018013070 comp rx=0 tx=0).stop 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 shutdown_connections 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5606cc64bcc0 0x5606cc64e0e0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606cc5d0a30 0x5606cc5d5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f2002ffd640 0 client.28511 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5606cc5c6170 0x5606cc5d4f50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 --2- 172.21.15.16:0/535440098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5606cc5c5850 0x5606cc5d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 >> 172.21.15.16:0/535440098 conn(0x5606cc5c1de0 msgr2=0x5606cc5c06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 shutdown_connections 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 1 -- 172.21.15.16:0/535440098 wait complete. 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.031+0000 7f202c579740 20 client.28511 trim_cache size 0 max 16384 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 58, 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "f0607e58": { 2024-09-12T03:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 58, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.2, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 msgr2=0x7f200c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 0x7f200c0a5210 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f202410ed80 tx=0x7f2018009b50 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 msgr2=0x7f202419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f201400e9f0 tx=0x7f201400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 shutdown_connections 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f202410e4d0 0x7f202419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f200c0a2d50 0x7f200c0a5210 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2024101360 0x7f202419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 --2- 172.21.15.16:0/3573098194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20241009b0 0x7f202419a6e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 >> 172.21.15.16:0/3573098194 conn(0x7f20240fc7d0 msgr2=0x7f20240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 shutdown_connections 2024-09-12T03:04:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:07.032+0000 7f202a1a0640 1 -- 172.21.15.16:0/3573098194 wait complete. 2024-09-12T03:04:07.679 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f0607e58', quiesce took 4.4 seconds. Will release after: 19.85 2024-09-12T03:04:08.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 34.90 MB/sec execute 69 sec latency 1583.851 ms 2024-09-12T03:04:08.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:07 smithi033 ceph-mon[35092]: pgmap v823: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:04:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:07 smithi184 ceph-mon[35321]: pgmap v823: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:04:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:07 smithi016 ceph-mon[30925]: pgmap v823: 97 pgs: 97 active+clean; 616 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 709 op/s 2024-09-12T03:04:09.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 34.40 MB/sec execute 70 sec latency 2583.897 ms 2024-09-12T03:04:09.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:08 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:04:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:08 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:04:09.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:08 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:04:10.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 33.91 MB/sec execute 71 sec latency 3583.995 ms 2024-09-12T03:04:10.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:09 smithi033 ceph-mon[35092]: pgmap v824: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 656 op/s 2024-09-12T03:04:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:04:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:04:10.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:09 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:04:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:09 smithi184 ceph-mon[35321]: pgmap v824: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 656 op/s 2024-09-12T03:04:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:04:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:04:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:09 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:04:10.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:09 smithi016 ceph-mon[30925]: pgmap v824: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 656 op/s 2024-09-12T03:04:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:04:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:04:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:09 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:04:11.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 33.44 MB/sec execute 72 sec latency 4584.054 ms 2024-09-12T03:04:12.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 32.99 MB/sec execute 73 sec latency 5584.102 ms 2024-09-12T03:04:12.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:11 smithi033 ceph-mon[35092]: pgmap v825: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 689 op/s 2024-09-12T03:04:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:11 smithi184 ceph-mon[35321]: pgmap v825: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 689 op/s 2024-09-12T03:04:12.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:11 smithi016 ceph-mon[30925]: pgmap v825: 97 pgs: 97 active+clean; 618 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 689 op/s 2024-09-12T03:04:13.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 32.54 MB/sec execute 74 sec latency 6584.139 ms 2024-09-12T03:04:14.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 32.11 MB/sec execute 75 sec latency 7584.180 ms 2024-09-12T03:04:14.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:13 smithi033 ceph-mon[35092]: pgmap v826: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.1 MiB/s wr, 557 op/s 2024-09-12T03:04:14.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:13 smithi016 ceph-mon[30925]: pgmap v826: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.1 MiB/s wr, 557 op/s 2024-09-12T03:04:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:13 smithi184 ceph-mon[35321]: pgmap v826: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.1 MiB/s wr, 557 op/s 2024-09-12T03:04:15.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 31.68 MB/sec execute 76 sec latency 8584.293 ms 2024-09-12T03:04:16.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 31.27 MB/sec execute 77 sec latency 9584.400 ms 2024-09-12T03:04:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:15 smithi033 ceph-mon[35092]: pgmap v827: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.2 MiB/s wr, 476 op/s 2024-09-12T03:04:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:15 smithi184 ceph-mon[35321]: pgmap v827: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.2 MiB/s wr, 476 op/s 2024-09-12T03:04:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:15 smithi016 ceph-mon[30925]: pgmap v827: 97 pgs: 97 active+clean; 623 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.2 MiB/s wr, 476 op/s 2024-09-12T03:04:17.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 30.87 MB/sec execute 78 sec latency 10584.505 ms 2024-09-12T03:04:17.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:04:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:18.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 30.48 MB/sec execute 79 sec latency 11584.546 ms 2024-09-12T03:04:18.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:17 smithi016 ceph-mon[30925]: pgmap v828: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.1 MiB/s wr, 421 op/s 2024-09-12T03:04:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/496479665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc106a80 msgr2=0x7f1dbc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/496479665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc106a80 0x7f1dbc106e60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f1dac009cd0 tx=0x7f1dac02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/496479665 shutdown_connections 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/496479665 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc101420 0x7f1dbc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/496479665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc100a80 0x7f1dbc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/496479665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc106a80 0x7f1dbc106e60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/496479665 >> 172.21.15.16:0/496479665 conn(0x7f1dbc0fc7a0 msgr2=0x7f1dbc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/496479665 shutdown_connections 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.148+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/496479665 wait complete. 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 Processor -- start 2024-09-12T03:04:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 -- start start 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 0x7f1dbc198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc106a80 0x7f1dbc19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc110e80 con 0x7f1dbc100a80 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc110d00 con 0x7f1dbc106a80 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dc0fb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc111000 con 0x7f1dbc101420 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51920/0 (socket says 172.21.15.16:51920) 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.149+0000 7f1dba59c640 1 -- 172.21.15.16:0/934497411 learned_addr learned my addr 172.21.15.16:0/934497411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc106a80 0x7f1dbc19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 0x7f1dbc198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 -- 172.21.15.16:0/934497411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 msgr2=0x7f1dbc198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 0x7f1dbc198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 -- 172.21.15.16:0/934497411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc106a80 msgr2=0x7f1dbc19cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc106a80 0x7f1dbc19cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 -- 172.21.15.16:0/934497411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc19d4b0 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 0x7f1dbc198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dba59c640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1dac02ae00 tx=0x7f1dac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1db37fe640 1 -- 172.21.15.16:0/934497411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac004040 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1db37fe640 1 -- 172.21.15.16:0/934497411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dac0046c0 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dbc19d740 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1db37fe640 1 -- 172.21.15.16:0/934497411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac046480 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.150+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dbc1a5080 con 0x7f1dbc100a80 2024-09-12T03:04:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.153+0000 7f1db37fe640 1 -- 172.21.15.16:0/934497411 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1dac0305a0 con 0x7f1dbc100a80 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.153+0000 7f1dc338f740 1 Processor -- start 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 -- start start 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44be5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff44b4b2e0 con 0x55ff44bf4a90 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff44af2ce0 con 0x55ff44be5c80 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dc338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff44b472a0 con 0x55ff44be5360 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dbad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44be5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51932/0 (socket says 172.21.15.16:51932) 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1465843989 learned_addr learned my addr 172.21.15.16:0/1465843989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db37fe640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 0x7f1da80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1465843989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 msgr2=0x55ff44be5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44be5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1465843989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 msgr2=0x55ff44bf4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.154+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1465843989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff44ad7440 con 0x55ff44bf4a90 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 0x7f1da80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db37fe640 1 -- 172.21.15.16:0/934497411 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1dac02e030 con 0x7f1dbc100a80 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1da400cc60 tx=0x7f1da400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd19478f1924c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db0ff9640 1 -- 172.21.15.16:0/1465843989 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da4014070 con 0x55ff44bf4a90 2024-09-12T03:04:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 0x7f1da80a5040 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b6e0 tx=0x7f1db40046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db0ff9640 1 -- 172.21.15.16:0/1465843989 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1da40047a0 con 0x55ff44bf4a90 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.155+0000 7f1db0ff9640 1 -- 172.21.15.16:0/1465843989 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da4010720 con 0x55ff44bf4a90 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- 172.21.15.16:0/1465843989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 msgr2=0x55ff44bf6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1da400cc60 tx=0x7f1da400e720 comp rx=0 tx=0).stop 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- 172.21.15.16:0/1465843989 shutdown_connections 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44bf6e80 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1465843989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44be5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- 172.21.15.16:0/1465843989 >> 172.21.15.16:0/1465843989 conn(0x55ff44be1a30 msgr2=0x55ff44be1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- 172.21.15.16:0/1465843989 shutdown_connections 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- 172.21.15.16:0/1465843989 wait complete. 2024-09-12T03:04:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 -- start start 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 Processor -- start 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44bf4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.156+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44bf4a90 0x55ff44bf1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff44bf7660 con 0x55ff44be5c80 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff44b06050 con 0x55ff44bf4a90 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff44bf77e0 con 0x55ff44be5360 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 10 client.0 _ll_register_callbacks cb 0x55ff44b46160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51936/0 (socket says 172.21.15.16:51936) 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1804728980 learned_addr learned my addr 172.21.15.16:0/1804728980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44bf4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44bf4a90 0x55ff44bf1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1804728980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 msgr2=0x55ff44bf4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44bf4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44bf4a90 msgr2=0x55ff44bf1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44bf4a90 0x55ff44bf1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 -- 172.21.15.16:0/1804728980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff448bc3c0 con 0x55ff44be5c80 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1da400ec20 tx=0x7f1da4010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4825234b5bcce547 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1da1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da4027060 con 0x55ff44be5c80 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1da4022410 con 0x55ff44be5c80 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.157+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1da40047a0 con 0x55ff44be5c80 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1da4014070 con 0x55ff44be5c80 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 10 client.19215 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ff44c030b0 con 0x55ff44be5c80 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1da40047a0 con 0x55ff44be5c80 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1da40185d0 con 0x55ff44be5c80 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 10 client.19215 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 10 client.19215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 20 client.19215 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c04510 0x55ff44c06900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 4 client.19215 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff44b58850 con 0x55ff44c04510 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.158+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c04510 0x55ff44c06900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c04510 0x55ff44c06900 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b08377cacc5043ee server_cookie=38880826737ecbea in_seq=0 out_seq=0 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1da1ffb640 10 client.19215 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1804728980 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ff44b58850 con 0x55ff44c04510 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1da1ffb640 10 client.19215 handle_command_reply: tid=0 2024-09-12T03:04:18.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 client.19215 shutdown 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 msgr2=0x55ff44bf16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1da400ec20 tx=0x7f1da4010b70 comp rx=0 tx=0).stop 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 shutdown_connections 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c04510 0x55ff44c06900 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44bf4a90 0x55ff44bf1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5c80 0x55ff44bf16f0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1da1ffb640 0 client.19215 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1804728980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44bf4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 >> 172.21.15.16:0/1804728980 conn(0x55ff44be1a30 msgr2=0x55ff44be7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 shutdown_connections 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.159+0000 7f1dc338f740 1 -- 172.21.15.16:0/1804728980 wait complete. 2024-09-12T03:04:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.160+0000 7f1dc338f740 20 client.19215 trim_cache size 0 max 16384 2024-09-12T03:04:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 Processor -- start 2024-09-12T03:04:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 -- start start 2024-09-12T03:04:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44c00900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44c0bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1ab2a0 con 0x55ff44bf4a90 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1acde0 con 0x55ff44be5c80 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.177+0000 7f1dc338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1ab970 con 0x55ff44be5360 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51952/0 (socket says 172.21.15.16:51952) 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 -- 172.21.15.16:0/1925010860 learned_addr learned my addr 172.21.15.16:0/1925010860 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44c00900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44c0bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 -- 172.21.15.16:0/1925010860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 msgr2=0x55ff44c00900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44c00900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 -- 172.21.15.16:0/1925010860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 msgr2=0x55ff44c0bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44c0bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 -- 172.21.15.16:0/1925010860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff44be5ab0 con 0x55ff44bf4a90 2024-09-12T03:04:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44c0bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f1db4003cc0 tx=0x7f1db4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf31ab73af5741e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1da3fff640 1 -- 172.21.15.16:0/1925010860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db40119c0 con 0x55ff44bf4a90 2024-09-12T03:04:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1da3fff640 1 -- 172.21.15.16:0/1925010860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1db4011b60 con 0x55ff44bf4a90 2024-09-12T03:04:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.178+0000 7f1da3fff640 1 -- 172.21.15.16:0/1925010860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db401c680 con 0x55ff44bf4a90 2024-09-12T03:04:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- 172.21.15.16:0/1925010860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 msgr2=0x55ff44c0c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f1db4003cc0 tx=0x7f1db4016000 comp rx=0 tx=0).stop 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- 172.21.15.16:0/1925010860 shutdown_connections 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44bf4a90 0x55ff44c0c440 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44c0bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1925010860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44be5360 0x55ff44c00900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- 172.21.15.16:0/1925010860 >> 172.21.15.16:0/1925010860 conn(0x55ff44be1a30 msgr2=0x55ff44becd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- 172.21.15.16:0/1925010860 shutdown_connections 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- 172.21.15.16:0/1925010860 wait complete. 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.179+0000 7f1dc338f740 1 -- start start 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 Processor -- start 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44bf4a90 0x55ff44bf9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1ac710 con 0x55ff44be5360 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1ac040 con 0x55ff44be5c80 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1a9280 con 0x55ff44bf4a90 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dc338f740 10 client.0 _ll_register_callbacks cb 0x7f1db4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51964/0 (socket says 172.21.15.16:51964) 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 -- 172.21.15.16:0/1214629786 learned_addr learned my addr 172.21.15.16:0/1214629786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dbad9d640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44bf4a90 0x55ff44bf9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 -- 172.21.15.16:0/1214629786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44bf4a90 msgr2=0x55ff44bf9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44bf4a90 0x55ff44bf9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 -- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 msgr2=0x55ff44bf8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1dba59c640 1 -- 172.21.15.16:0/1214629786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff44bf2000 con 0x55ff44be5360 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.180+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dba59c640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f1dac009ca0 tx=0x7f1dac03d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cc0b6844d52c5f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0bd0a0 con 0x55ff44be5360 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1dac0dd0b0 con 0x55ff44be5360 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1db4011b60 con 0x55ff44be5360 2024-09-12T03:04:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1dac0c1450 con 0x55ff44be5360 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 10 client.19221 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ff44c6f1e0 con 0x55ff44be5360 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1db4011b60 con 0x55ff44be5360 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1dac0d8060 con 0x55ff44be5360 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 10 client.19221 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 10 client.19221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 20 client.19221 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c6f9a0 0x55ff44c71d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 4 client.19221 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.181+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ff44b52d60 con 0x55ff44c6f9a0 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c6f9a0 0x55ff44c71d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1db9d9b640 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c6f9a0 0x55ff44c71d90 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edc1093c818a6700 server_cookie=9fbd1866f028322b in_seq=0 out_seq=0 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1da1ffb640 10 client.19221 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1da1ffb640 1 -- 172.21.15.16:0/1214629786 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ff44b52d60 con 0x55ff44c6f9a0 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1da1ffb640 10 client.19221 handle_command_reply: tid=0 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 client.19221 shutdown 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 msgr2=0x55ff44bf8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f1dac009ca0 tx=0x7f1dac03d040 comp rx=0 tx=0).stop 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 shutdown_connections 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff44c6f9a0 0x55ff44c71d90 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff44bf4a90 0x55ff44bf9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff44be5c80 0x55ff44bf8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 --2- 172.21.15.16:0/1214629786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff44be5360 0x55ff44bf8a80 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 >> 172.21.15.16:0/1214629786 conn(0x55ff44be1a30 msgr2=0x55ff44bf2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1da1ffb640 0 client.19221 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.182+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 shutdown_connections 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc338f740 1 -- 172.21.15.16:0/1214629786 wait complete. 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc338f740 20 client.19221 trim_cache size 0 max 16384 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:04:18.189 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 msgr2=0x7f1da80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 0x7f1da80a5040 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b6e0 tx=0x7f1db40046c0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 msgr2=0x7f1dbc198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1dac02ae00 tx=0x7f1dac003970 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 shutdown_connections 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1dbc106a80 0x7f1dbc19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dbc101420 0x7f1dbc198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1da80a2b80 0x7f1da80a5040 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 --2- 172.21.15.16:0/934497411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1dbc100a80 0x7f1dbc198500 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 >> 172.21.15.16:0/934497411 conn(0x7f1dbc0fc7a0 msgr2=0x7f1dbc06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 shutdown_connections 2024-09-12T03:04:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:18.183+0000 7f1dc0fb6640 1 -- 172.21.15.16:0/934497411 wait complete. 2024-09-12T03:04:18.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:17 smithi033 ceph-mon[35092]: pgmap v828: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.1 MiB/s wr, 421 op/s 2024-09-12T03:04:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:17 smithi184 ceph-mon[35321]: pgmap v828: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.1 MiB/s wr, 421 op/s 2024-09-12T03:04:18.914 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:04:18.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:04:19.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 30.10 MB/sec execute 80 sec latency 12584.604 ms 2024-09-12T03:04:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 -- 172.21.15.16:0/2854720101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570102790 msgr2=0x7fc5701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 --2- 172.21.15.16:0/2854720101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570102790 0x7fc5701067e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc560009900 tx=0x7fc56002a420 comp rx=0 tx=0).stop 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 -- 172.21.15.16:0/2854720101 shutdown_connections 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 --2- 172.21.15.16:0/2854720101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc570106f10 0x7fc57010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 --2- 172.21.15.16:0/2854720101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570102790 0x7fc5701067e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 --2- 172.21.15.16:0/2854720101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570101de0 0x7fc5701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.966+0000 7fc57815f640 1 -- 172.21.15.16:0/2854720101 >> 172.21.15.16:0/2854720101 conn(0x7fc5700fd8e0 msgr2=0x7fc5700ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.969+0000 7fc57815f640 1 -- 172.21.15.16:0/2854720101 shutdown_connections 2024-09-12T03:04:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.969+0000 7fc57815f640 1 -- 172.21.15.16:0/2854720101 wait complete. 2024-09-12T03:04:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 Processor -- start 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 -- start start 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570101de0 0x7fc5700744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc570102790 0x7fc570074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc570112000 con 0x7fc570102790 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc570111e80 con 0x7fc570101de0 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.970+0000 7fc57815f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc570112180 con 0x7fc570106f10 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.971+0000 7fc5766fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.971+0000 7fc575efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570101de0 0x7fc5700744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.971+0000 7fc5766fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42690/0 (socket says 172.21.15.16:42690) 2024-09-12T03:04:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.971+0000 7fc5766fc640 1 -- 172.21.15.16:0/2537320989 learned_addr learned my addr 172.21.15.16:0/2537320989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 -- 172.21.15.16:0/2537320989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570101de0 msgr2=0x7fc5700744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570101de0 0x7fc5700744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 -- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc570102790 msgr2=0x7fc570074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc570102790 0x7fc570074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 -- 172.21.15.16:0/2537320989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5701aaeb0 con 0x7fc570106f10 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc5766fc640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc56800eaa0 tx=0x7fc56800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56800cdb0 con 0x7fc570106f10 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc568005020 con 0x7fc570106f10 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5701ab0e0 con 0x7fc570106f10 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.972+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5701ab620 con 0x7fc570106f10 2024-09-12T03:04:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.973+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5680116d0 con 0x7fc570106f10 2024-09-12T03:04:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.974+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5701032c0 con 0x7fc570106f10 2024-09-12T03:04:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.976+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc5680051d0 con 0x7fc570106f10 2024-09-12T03:04:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.977+0000 7fc566ffd640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 0x7fc55c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.977+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc5680cf2e0 con 0x7fc570106f10 2024-09-12T03:04:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.977+0000 7fc575efb640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 0x7fc55c0a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.978+0000 7fc575efb640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 0x7fc55c0a5140 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006fd0 tx=0x7fc56c0090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:19.982+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc568016030 con 0x7fc570106f10 2024-09-12T03:04:20.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 29.73 MB/sec execute 81 sec latency 13584.637 ms 2024-09-12T03:04:20.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:19 smithi016 ceph-mon[30925]: pgmap v829: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 299 op/s 2024-09-12T03:04:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.212+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc57010a150 con 0x7fc570106f10 2024-09-12T03:04:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.212+0000 7fc566ffd640 1 -- 172.21.15.16:0/2537320989 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fc56809c1f0 con 0x7fc570106f10 2024-09-12T03:04:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:04:20.213 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:04:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 msgr2=0x7fc55c0a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 0x7fc55c0a5140 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006fd0 tx=0x7fc56c0090e0 comp rx=0 tx=0).stop 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 msgr2=0x7fc570074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc56800eaa0 tx=0x7fc56800ef60 comp rx=0 tx=0).stop 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 shutdown_connections 2024-09-12T03:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc570106f10 0x7fc570074f70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc55c0a2c80 0x7fc55c0a5140 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc570102790 0x7fc570074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 --2- 172.21.15.16:0/2537320989 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc570101de0 0x7fc5700744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 >> 172.21.15.16:0/2537320989 conn(0x7fc5700fd8e0 msgr2=0x7fc5700febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 shutdown_connections 2024-09-12T03:04:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:20.215+0000 7fc57815f640 1 -- 172.21.15.16:0/2537320989 wait complete. 2024-09-12T03:04:20.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:19 smithi184 ceph-mon[35321]: pgmap v829: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 299 op/s 2024-09-12T03:04:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:19 smithi033 ceph-mon[35092]: pgmap v829: 97 pgs: 97 active+clean; 620 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 299 op/s 2024-09-12T03:04:20.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:04:21.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 29.36 MB/sec execute 82 sec latency 14584.695 ms 2024-09-12T03:04:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:21.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:21 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2537320989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:21 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2537320989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:21.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:21 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2537320989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 -- 172.21.15.16:0/1370128699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 msgr2=0x7fedd010bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 --2- 172.21.15.16:0/1370128699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd010bb60 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009900 tx=0x7fedc402a480 comp rx=0 tx=0).stop 2024-09-12T03:04:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 -- 172.21.15.16:0/1370128699 shutdown_connections 2024-09-12T03:04:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 --2- 172.21.15.16:0/1370128699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd010c180 0x7fedd010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 --2- 172.21.15.16:0/1370128699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd010bb60 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 --2- 172.21.15.16:0/1370128699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd0102d50 0x7fedd0103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.589+0000 7fedd5874640 1 -- 172.21.15.16:0/1370128699 >> 172.21.15.16:0/1370128699 conn(0x7fedd00fc7d0 msgr2=0x7fedd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.590+0000 7fedd5874640 1 -- 172.21.15.16:0/1370128699 shutdown_connections 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.590+0000 7fedd5874640 1 -- 172.21.15.16:0/1370128699 wait complete. 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.590+0000 7fedd5874640 1 Processor -- start 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 -- start start 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd0102d50 0x7fedd01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd010c180 0x7fedd019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd0111f80 con 0x7fedd0103670 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd0111e00 con 0x7fedd010c180 2024-09-12T03:04:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedd5874640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd0112100 con 0x7fedd0102d50 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52000/0 (socket says 172.21.15.16:52000) 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 -- 172.21.15.16:0/1074823043 learned_addr learned my addr 172.21.15.16:0/1074823043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedceffd640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd0102d50 0x7fedd01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd010c180 0x7fedd019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 -- 172.21.15.16:0/1074823043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd0102d50 msgr2=0x7fedd01983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd0102d50 0x7fedd01983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 -- 172.21.15.16:0/1074823043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd010c180 msgr2=0x7fedd019ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd010c180 0x7fedd019ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.591+0000 7fedce7fc640 1 -- 172.21.15.16:0/1074823043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd019d390 con 0x7fedd0103670 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedce7fc640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006010 tx=0x7fedc40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedd4872640 1 -- 172.21.15.16:0/1074823043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc403f030 con 0x7fedd0103670 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedd4872640 1 -- 172.21.15.16:0/1074823043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc4003ce0 con 0x7fedd0103670 2024-09-12T03:04:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd019d620 con 0x7fedd0103670 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedd4872640 1 -- 172.21.15.16:0/1074823043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4035070 con 0x7fedd0103670 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.592+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd01a4f60 con 0x7fedd0103670 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.594+0000 7fedd4872640 1 -- 172.21.15.16:0/1074823043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fedc4042030 con 0x7fedd0103670 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.594+0000 7fedd7c4d740 1 Processor -- start 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.594+0000 7fedd7c4d740 1 -- start start 2024-09-12T03:04:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.594+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.594+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 0x557617dc6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dd3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557617d2c2f0 con 0x557617dc7b40 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557617cd3cf0 con 0x557617dd1370 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557617d282b0 con 0x557617dc65d0 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52002/0 (socket says 172.21.15.16:52002) 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 -- 172.21.15.16:0/2475192998 learned_addr learned my addr 172.21.15.16:0/2475192998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedce7fc640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dd3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedceffd640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 0x557617dc6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 -- 172.21.15.16:0/2475192998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 msgr2=0x557617dc6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 0x557617dc6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 -- 172.21.15.16:0/2475192998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 msgr2=0x557617dd3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dd3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 -- 172.21.15.16:0/2475192998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557617cb83a0 con 0x557617dc7b40 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedceffd640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 0x557617dc6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009f70 tx=0x7fedc8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7a4e63dd7883c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd4872640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 0x7fedb80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedb5ffb640 1 -- 172.21.15.16:0/2475192998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8016070 con 0x557617dc7b40 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedceffd640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 0x7fedb80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedb5ffb640 1 -- 172.21.15.16:0/2475192998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8004d20 con 0x557617dc7b40 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedb5ffb640 1 -- 172.21.15.16:0/2475192998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8012910 con 0x557617dc7b40 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.595+0000 7fedd4872640 1 -- 172.21.15.16:0/1074823043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedc40ef2d0 con 0x7fedd0103670 2024-09-12T03:04:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2475192998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 msgr2=0x557617dc6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedceffd640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 0x7fedb80a50c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x557617dc3ac0 tx=0x7fedc000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009f70 tx=0x7fedc8010800 comp rx=0 tx=0).stop 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2475192998 shutdown_connections 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dd3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc65d0 0x557617dc6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2475192998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dc6090 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2475192998 >> 172.21.15.16:0/2475192998 conn(0x557617d8f310 msgr2=0x557617dc2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2475192998 shutdown_connections 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.596+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2475192998 wait complete. 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 -- start start 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 Processor -- start 2024-09-12T03:04:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557617dd4070 con 0x557617dc65d0 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557617ce7060 con 0x557617dd1370 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557617dd41f0 con 0x557617dc7b40 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedd7c4d740 10 client.0 _ll_register_callbacks cb 0x557617a3a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52010/0 (socket says 172.21.15.16:52010) 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedceffd640 1 -- 172.21.15.16:0/890347589 learned_addr learned my addr 172.21.15.16:0/890347589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.597+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 -- 172.21.15.16:0/890347589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 msgr2=0x557617dce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 -- 172.21.15.16:0/890347589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 msgr2=0x557617dce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 -- 172.21.15.16:0/890347589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557617a9d3c0 con 0x557617dc65d0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedceffd640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fedc0012010 tx=0x7fedc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eec5857c19e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc000fea0 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedc0005020 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc8004d20 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fedc0005220 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedd7c4d740 10 client.19236 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.598+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557617ddfae0 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8004d20 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fedc001f4b0 con 0x557617dc65d0 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 10 client.19236 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 10 client.19236 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 20 client.19236 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617de0fa0 0x557617de3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 4 client.19236 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557617d39860 con 0x557617de0fa0 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedce7fc640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617de0fa0 0x557617de3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedaeffd640 10 client.19236 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.599+0000 7fedce7fc640 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617de0fa0 0x557617de3390 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c6987f577141f1fe server_cookie=32bc1e54ede8019 in_seq=0 out_seq=0 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedaeffd640 1 -- 172.21.15.16:0/890347589 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557617d39860 con 0x557617de0fa0 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedaeffd640 10 client.19236 handle_command_reply: tid=0 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 client.19236 shutdown 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 msgr2=0x557617dd0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fedc0012010 tx=0x7fedc00046c0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 shutdown_connections 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617de0fa0 0x557617de3390 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617dce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/890347589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617dd0d60 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedaeffd640 0 client.19236 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:04:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 >> 172.21.15.16:0/890347589 conn(0x557617d8f310 msgr2=0x557617dc8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 shutdown_connections 2024-09-12T03:04:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.600+0000 7fedd7c4d740 1 -- 172.21.15.16:0/890347589 wait complete. 2024-09-12T03:04:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.601+0000 7fedd7c4d740 20 client.19236 trim_cache size 0 max 16384 2024-09-12T03:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.613+0000 7fedd7c4d740 1 Processor -- start 2024-09-12T03:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.613+0000 7fedd7c4d740 1 -- start start 2024-09-12T03:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dcd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617de8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01ad580 con 0x557617dc65d0 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01af0c0 con 0x557617dd1370 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01adc50 con 0x557617dc7b40 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52012/0 (socket says 172.21.15.16:52012) 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 -- 172.21.15.16:0/1208343462 learned_addr learned my addr 172.21.15.16:0/1208343462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedce7fc640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dcd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617de8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 -- 172.21.15.16:0/1208343462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 msgr2=0x557617dcd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dcd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 -- 172.21.15.16:0/1208343462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 msgr2=0x557617de8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617de8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 -- 172.21.15.16:0/1208343462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557617dcebe0 con 0x557617dc65d0 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617de8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedce7fc640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dcd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedceffd640 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fedc0012870 tx=0x7fedc0020f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b8732d7643e64f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.614+0000 7fedb57fa640 1 -- 172.21.15.16:0/1208343462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc00053c0 con 0x557617dc65d0 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedb57fa640 1 -- 172.21.15.16:0/1208343462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc0005560 con 0x557617dc65d0 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedb57fa640 1 -- 172.21.15.16:0/1208343462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc000fb70 con 0x557617dc65d0 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 -- 172.21.15.16:0/1208343462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 msgr2=0x557617ddd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fedc0012870 tx=0x7fedc0020f40 comp rx=0 tx=0).stop 2024-09-12T03:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 -- 172.21.15.16:0/1208343462 shutdown_connections 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dd1370 0x557617de8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dc7b40 0x557617dcd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/1208343462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc65d0 0x557617ddd310 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 -- 172.21.15.16:0/1208343462 >> 172.21.15.16:0/1208343462 conn(0x557617d8f310 msgr2=0x557617dcc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 -- 172.21.15.16:0/1208343462 shutdown_connections 2024-09-12T03:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.615+0000 7fedd7c4d740 1 -- 172.21.15.16:0/1208343462 wait complete. 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 -- start start 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 Processor -- start 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dc65d0 0x557617dd4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 0x557617dd92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01ae9f0 con 0x557617dc7b40 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01ae320 con 0x557617dc65d0 2024-09-12T03:04:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd01ab560 con 0x557617dd1370 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedd7c4d740 10 client.0 _ll_register_callbacks cb 0x7fedc0005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52016/0 (socket says 172.21.15.16:52016) 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedce7fc640 1 -- 172.21.15.16:0/2370613007 learned_addr learned my addr 172.21.15.16:0/2370613007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedceffd640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dc65d0 0x557617dd4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.616+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 0x557617dd92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 -- 172.21.15.16:0/2370613007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 msgr2=0x557617dd92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 0x557617dd92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 -- 172.21.15.16:0/2370613007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dc65d0 msgr2=0x557617dd4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dc65d0 0x557617dd4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 -- 172.21.15.16:0/2370613007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557617dd9990 con 0x557617dc7b40 2024-09-12T03:04:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedcf7fe640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 0x557617dd92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedce7fc640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fedc40ecde0 tx=0x7fedc403b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f58f02bb29d6988 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4049040 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedc40b9e40 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc0005560 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fedc40bc240 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedd7c4d740 10 client.19242 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557617e4b7a0 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc0005560 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.617+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fedc40da2b0 con 0x557617dc7b40 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 10 client.19242 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 10 client.19242 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:04:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 20 client.19242 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617e4bf50 0x557617e4e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 4 client.19242 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:04:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557617d33d70 con 0x557617e4bf50 2024-09-12T03:04:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedceffd640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617e4bf50 0x557617e4e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedceffd640 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617e4bf50 0x557617e4e340 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=874fa345dd9418ed server_cookie=43d16bcac4b31432 in_seq=0 out_seq=0 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedaeffd640 10 client.19242 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedaeffd640 1 -- 172.21.15.16:0/2370613007 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557617d33d70 con 0x557617e4bf50 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedaeffd640 10 client.19242 handle_command_reply: tid=0 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 1 client.19242 shutdown 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 msgr2=0x557617dd5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.618+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fedc40ecde0 tx=0x7fedc403b5d0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 shutdown_connections 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557617dd1370 0x557617dd92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557617dc7b40 0x557617dd5160 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x557617e4bf50 0x557617e4e340 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 --2- 172.21.15.16:0/2370613007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557617dc65d0 0x557617dd4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 >> 172.21.15.16:0/2370613007 conn(0x557617d8f310 msgr2=0x557617dc14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedaeffd640 0 client.19242 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:04:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 shutdown_connections 2024-09-12T03:04:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 1 -- 172.21.15.16:0/2370613007 wait complete. 2024-09-12T03:04:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd7c4d740 20 client.19242 trim_cache size 0 max 16384 2024-09-12T03:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 msgr2=0x7fedb80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 0x7fedb80a50c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x557617dc3ac0 tx=0x7fedc000a2d0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 msgr2=0x7fedd0198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006010 tx=0x7fedc40042f0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 shutdown_connections 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd010c180 0x7fedd019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedb80a2c00 0x7fedb80a50c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd0103670 0x7fedd0198920 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 --2- 172.21.15.16:0/1074823043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd0102d50 0x7fedd01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.619+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 >> 172.21.15.16:0/1074823043 conn(0x7fedd00fc7d0 msgr2=0x7fedd00fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.620+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 shutdown_connections 2024-09-12T03:04:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:21.620+0000 7fedd5874640 1 -- 172.21.15.16:0/1074823043 wait complete. 2024-09-12T03:04:22.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 29.01 MB/sec execute 83 sec latency 15584.727 ms 2024-09-12T03:04:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:22 smithi184 ceph-mon[35321]: pgmap v830: 97 pgs: 97 active+clean; 625 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-12T03:04:22.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:22 smithi016 ceph-mon[30925]: pgmap v830: 97 pgs: 97 active+clean; 625 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-12T03:04:22.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:04:22.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:22 smithi033 ceph-mon[35092]: pgmap v830: 97 pgs: 97 active+clean; 625 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 272 op/s 2024-09-12T03:04:22.637 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:23.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 28.67 MB/sec execute 84 sec latency 16584.840 ms 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 -- 172.21.15.16:0/2748018434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102ef0 msgr2=0x7faa2810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/2748018434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102ef0 0x7faa2810da40 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7faa20009900 tx=0x7faa2002a480 comp rx=0 tx=0).stop 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 -- 172.21.15.16:0/2748018434 shutdown_connections 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/2748018434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102ef0 0x7faa2810da40 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/2748018434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28102570 0x7faa281029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/2748018434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28100e20 0x7faa28101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 -- 172.21.15.16:0/2748018434 >> 172.21.15.16:0/2748018434 conn(0x7faa280fc7d0 msgr2=0x7faa280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 -- 172.21.15.16:0/2748018434 shutdown_connections 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.334+0000 7faa2cf6b640 1 -- 172.21.15.16:0/2748018434 wait complete. 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 Processor -- start 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 -- start start 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28100e20 0x7faa28198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28102ef0 0x7faa2819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa28110f80 con 0x7faa28102570 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faa28110e00 con 0x7faa28100e20 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2cf6b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa28111100 con 0x7faa28102ef0 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52030/0 (socket says 172.21.15.16:52030) 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 -- 172.21.15.16:0/1807308847 learned_addr learned my addr 172.21.15.16:0/1807308847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa2659c640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28100e20 0x7faa28198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa26d9d640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28102ef0 0x7faa2819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 -- 172.21.15.16:0/1807308847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28102ef0 msgr2=0x7faa2819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28102ef0 0x7faa2819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 -- 172.21.15.16:0/1807308847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28100e20 msgr2=0x7faa28198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28100e20 0x7faa28198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 -- 172.21.15.16:0/1807308847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2819d4b0 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.335+0000 7faa25d9b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00d950 tx=0x7faa1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.336+0000 7faa177fe640 1 -- 172.21.15.16:0/1807308847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c017070 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.336+0000 7faa177fe640 1 -- 172.21.15.16:0/1807308847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c009c30 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.336+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2819d7a0 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.336+0000 7faa177fe640 1 -- 172.21.15.16:0/1807308847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c00f6b0 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.336+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa281a5080 con 0x7faa28102570 2024-09-12T03:04:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.338+0000 7faa177fe640 1 -- 172.21.15.16:0/1807308847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7faa1c0109d0 con 0x7faa28102570 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.338+0000 7faa2f344740 1 Processor -- start 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 -- start start 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a7192e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a7192ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556a718942f0 con 0x556a71939370 2024-09-12T03:04:23.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a7183bcf0 con 0x556a7192fb40 2024-09-12T03:04:23.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2f344740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556a718902b0 con 0x556a7192e5d0 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa177fe640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 0x7faa100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a7192e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa2659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a7192ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52032/0 (socket says 172.21.15.16:52032) 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa25d9b640 1 -- 172.21.15.16:0/3966446446 learned_addr learned my addr 172.21.15.16:0/3966446446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa177fe640 1 -- 172.21.15.16:0/1807308847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faa1c0cebc0 con 0x7faa28102570 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.339+0000 7faa26d9d640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 0x7faa100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 -- 172.21.15.16:0/3966446446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 msgr2=0x556a7192ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a7192ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 -- 172.21.15.16:0/3966446446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 msgr2=0x556a7192e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a7192e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 -- 172.21.15.16:0/3966446446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a718203a0 con 0x556a71939370 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7faa1c010fd0 tx=0x7faa1c0c5fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6229d15ac6398098 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa26d9d640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 0x7faa100a5040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faa2002a9a0 tx=0x7faa20005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa14ff9640 1 -- 172.21.15.16:0/3966446446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c020050 con 0x556a71939370 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa14ff9640 1 -- 172.21.15.16:0/3966446446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c09bb30 con 0x556a71939370 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.340+0000 7faa14ff9640 1 -- 172.21.15.16:0/3966446446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0a40a0 con 0x556a71939370 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- 172.21.15.16:0/3966446446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 msgr2=0x556a7193b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7faa1c010fd0 tx=0x7faa1c0c5fc0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- 172.21.15.16:0/3966446446 shutdown_connections 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193b760 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a7192ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 --2- 172.21.15.16:0/3966446446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a7192e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- 172.21.15.16:0/3966446446 >> 172.21.15.16:0/3966446446 conn(0x556a718f7310 msgr2=0x556a7192ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- 172.21.15.16:0/3966446446 shutdown_connections 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- 172.21.15.16:0/3966446446 wait complete. 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 -- start start 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.341+0000 7faa2f344740 1 Processor -- start 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a71936060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a719367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556a7193c070 con 0x556a71939370 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556a7184f060 con 0x556a7192fb40 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556a7193c1f0 con 0x556a7192e5d0 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2f344740 10 client.0 _ll_register_callbacks cb 0x556a715a28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a719367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a719367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52034/0 (socket says 172.21.15.16:52034) 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42772/0 (socket says 172.21.15.16:42772) 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a71936060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa26d9d640 1 -- 172.21.15.16:0/3194416122 learned_addr learned my addr 172.21.15.16:0/3194416122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 -- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 msgr2=0x556a71936060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a71936060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 -- 172.21.15.16:0/3194416122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 msgr2=0x556a719367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a719367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa2659c640 1 -- 172.21.15.16:0/3194416122 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a716053c0 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.342+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a71936060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2659c640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa18005fd0 tx=0x7faa180117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60a633e57c9ada93 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18003b70 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faa18003d10 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c09bb30 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7faa18004050 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 10 client.28474 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556a71947bc0 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c09bb30 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7faa1801a020 con 0x556a7192e5d0 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 10 client.28474 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 10 client.28474 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 20 client.28474 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a71949080 0x556a7194b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.343+0000 7faa2f344740 4 client.28474 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.344+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a718a1860 con 0x556a71949080 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.344+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a71949080 0x556a7194b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.344+0000 7faa0dffb640 10 client.28474 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.344+0000 7faa25d9b640 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a71949080 0x556a7194b4d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66aebe6819f3f3a2 server_cookie=82e154cba388f71e in_seq=0 out_seq=0 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa0dffb640 1 -- 172.21.15.16:0/3194416122 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a718a1860 con 0x556a71949080 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa0dffb640 10 client.28474 handle_command_reply: tid=0 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 client.28474 shutdown 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 msgr2=0x556a71938db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa18005fd0 tx=0x7faa180117b0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 shutdown_connections 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a71949080 0x556a7194b4d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a719367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192fb40 0x556a71936060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa0dffb640 0 client.28474 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 --2- 172.21.15.16:0/3194416122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192e5d0 0x556a71938db0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 >> 172.21.15.16:0/3194416122 conn(0x556a718f7310 msgr2=0x556a71930990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 shutdown_connections 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 1 -- 172.21.15.16:0/3194416122 wait complete. 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.345+0000 7faa2f344740 20 client.28474 trim_cache size 0 max 16384 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 Processor -- start 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 -- start start 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a7192f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a71939370 0x556a7193b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ad500 con 0x556a7192fb40 2024-09-12T03:04:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faa281af040 con 0x556a7192e5d0 2024-09-12T03:04:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2f344740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa281adbd0 con 0x556a71939370 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52042/0 (socket says 172.21.15.16:52042) 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa25d9b640 1 -- 172.21.15.16:0/2193772025 learned_addr learned my addr 172.21.15.16:0/2193772025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa2659c640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a7192f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.359+0000 7faa26d9d640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a71939370 0x556a7193b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 -- 172.21.15.16:0/2193772025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a71939370 msgr2=0x556a7193b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a71939370 0x556a7193b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 -- 172.21.15.16:0/2193772025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 msgr2=0x556a7192f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a7192f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 -- 172.21.15.16:0/2193772025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7192b960 con 0x556a7192fb40 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa2659c640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a7192f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa25d9b640 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7faa1c097e00 tx=0x7faa1c097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a62d8cb3591031 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa0ffff640 1 -- 172.21.15.16:0/2193772025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c020040 con 0x556a7192fb40 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa0ffff640 1 -- 172.21.15.16:0/2193772025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c09bb30 con 0x556a7192fb40 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.360+0000 7faa0ffff640 1 -- 172.21.15.16:0/2193772025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c097aa0 con 0x556a7192fb40 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- 172.21.15.16:0/2193772025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 msgr2=0x556a71935150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7faa1c097e00 tx=0x7faa1c097040 comp rx=0 tx=0).stop 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- 172.21.15.16:0/2193772025 shutdown_connections 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a71939370 0x556a7193b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a7192fb40 0x556a71935150 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 --2- 172.21.15.16:0/2193772025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a7192f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- 172.21.15.16:0/2193772025 >> 172.21.15.16:0/2193772025 conn(0x556a718f7310 msgr2=0x556a71925a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- 172.21.15.16:0/2193772025 shutdown_connections 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- 172.21.15.16:0/2193772025 wait complete. 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 -- start start 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.361+0000 7faa2f344740 1 Processor -- start 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a71939fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192fb40 0x556a7193a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ae970 con 0x556a71939370 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ae2a0 con 0x556a7192e5d0 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ab4e0 con 0x556a7192fb40 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2f344740 10 client.0 _ll_register_callbacks cb 0x7faa18009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:23.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52052/0 (socket says 172.21.15.16:52052) 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 -- 172.21.15.16:0/513127382 learned_addr learned my addr 172.21.15.16:0/513127382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa25d9b640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192fb40 0x556a7193a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa2659c640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a71939fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 -- 172.21.15.16:0/513127382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192fb40 msgr2=0x556a7193a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192fb40 0x556a7193a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 -- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 msgr2=0x556a71939fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a71939fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 -- 172.21.15.16:0/513127382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a71940770 con 0x556a71939370 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa26d9d640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7faa20003d50 tx=0x7faa20009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b34d185e4a3394 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.362+0000 7faa0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20051070 con 0x556a71939370 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7faa20056460 con 0x556a71939370 2024-09-12T03:04:23.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c09bb30 con 0x556a71939370 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7faa20035040 con 0x556a71939370 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 10 client.19260 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556a71941a70 con 0x556a71939370 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c09bb30 con 0x556a71939370 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7faa2003a030 con 0x556a71939370 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 10 client.19260 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 10 client.19260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 20 client.19260 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a719b3d70 0x556a719b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:23.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 4 client.19260 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a7189bd70 con 0x556a719b3d70 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.363+0000 7faa2659c640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a719b3d70 0x556a719b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa0dffb640 10 client.19260 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2659c640 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a719b3d70 0x556a719b6160 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=28a2383241a6816e server_cookie=f8d9c5ef5f409720 in_seq=0 out_seq=0 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa0dffb640 1 -- 172.21.15.16:0/513127382 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a7189bd70 con 0x556a719b3d70 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa0dffb640 10 client.19260 handle_command_reply: tid=0 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 client.19260 shutdown 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 msgr2=0x556a7193aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7faa20003d50 tx=0x7faa20009260 comp rx=0 tx=0).stop 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 shutdown_connections 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x556a719b3d70 0x556a719b6160 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a71939370 0x556a7193aa60 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa0dffb640 0 client.19260 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:04:23.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556a7192fb40 0x556a7193a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 --2- 172.21.15.16:0/513127382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556a7192e5d0 0x556a71939fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.364+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 >> 172.21.15.16:0/513127382 conn(0x556a718f7310 msgr2=0x556a71935150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 shutdown_connections 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2f344740 1 -- 172.21.15.16:0/513127382 wait complete. 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2f344740 20 client.19260 trim_cache size 0 max 16384 2024-09-12T03:04:23.381 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 msgr2=0x7faa100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 0x7faa100a5040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faa2002a9a0 tx=0x7faa20005fd0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 msgr2=0x7faa28198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00d950 tx=0x7faa1c00de10 comp rx=0 tx=0).stop 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 shutdown_connections 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa28102ef0 0x7faa2819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7faa100a2b80 0x7faa100a5040 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa28102570 0x7faa28198a40 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 --2- 172.21.15.16:0/1807308847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa28100e20 0x7faa28198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:23.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.365+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 >> 172.21.15.16:0/1807308847 conn(0x7faa280fc7d0 msgr2=0x7faa2806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:23.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.366+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 shutdown_connections 2024-09-12T03:04:23.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:23.366+0000 7faa2cf6b640 1 -- 172.21.15.16:0/1807308847 wait complete. 2024-09-12T03:04:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 28.33 MB/sec execute 85 sec latency 17584.894 ms 2024-09-12T03:04:24.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:04:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:24 smithi184 ceph-mon[35321]: pgmap v831: 97 pgs: 97 active+clean; 619 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 5.6 MiB/s wr, 210 op/s 2024-09-12T03:04:24.298 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:24 smithi033 ceph-mon[35092]: pgmap v831: 97 pgs: 97 active+clean; 619 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 5.6 MiB/s wr, 210 op/s 2024-09-12T03:04:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:24.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:24 smithi016 ceph-mon[30925]: pgmap v831: 97 pgs: 97 active+clean; 619 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 5.6 MiB/s wr, 210 op/s 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 -- 172.21.15.16:0/761297414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd914103660 msgr2=0x7fd91410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 --2- 172.21.15.16:0/761297414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd914103660 0x7fd91410bb50 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd900009900 tx=0x7fd90002a480 comp rx=0 tx=0).stop 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 -- 172.21.15.16:0/761297414 shutdown_connections 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 --2- 172.21.15.16:0/761297414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91410c170 0x7fd91410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 --2- 172.21.15.16:0/761297414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd914103660 0x7fd91410bb50 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 --2- 172.21.15.16:0/761297414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 0x7fd914103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 -- 172.21.15.16:0/761297414 >> 172.21.15.16:0/761297414 conn(0x7fd9140fc7a0 msgr2=0x7fd9140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 -- 172.21.15.16:0/761297414 shutdown_connections 2024-09-12T03:04:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.875+0000 7fd91adc0640 1 -- 172.21.15.16:0/761297414 wait complete. 2024-09-12T03:04:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 Processor -- start 2024-09-12T03:04:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 -- start start 2024-09-12T03:04:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 0x7fd914198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd914103660 0x7fd9141989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd914111fc0 con 0x7fd91410c170 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd914111e40 con 0x7fd914102d40 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91adc0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd914112140 con 0x7fd914103660 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91935d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91935d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52066/0 (socket says 172.21.15.16:52066) 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd91935d640 1 -- 172.21.15.16:0/3274697039 learned_addr learned my addr 172.21.15.16:0/3274697039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.876+0000 7fd918b5c640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 0x7fd914198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd913fff640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd914103660 0x7fd9141989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 -- 172.21.15.16:0/3274697039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd914103660 msgr2=0x7fd9141989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd914103660 0x7fd9141989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 -- 172.21.15.16:0/3274697039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 msgr2=0x7fd914198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 0x7fd914198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 -- 172.21.15.16:0/3274697039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91419d430 con 0x7fd91410c170 2024-09-12T03:04:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91935d640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00eaa0 tx=0x7fd90c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd911ffb640 1 -- 172.21.15.16:0/3274697039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00cdb0 con 0x7fd91410c170 2024-09-12T03:04:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd911ffb640 1 -- 172.21.15.16:0/3274697039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c005020 con 0x7fd91410c170 2024-09-12T03:04:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91419d720 con 0x7fd91410c170 2024-09-12T03:04:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd911ffb640 1 -- 172.21.15.16:0/3274697039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0116b0 con 0x7fd91410c170 2024-09-12T03:04:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.877+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9141a5000 con 0x7fd91410c170 2024-09-12T03:04:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.879+0000 7fd911ffb640 1 -- 172.21.15.16:0/3274697039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd90c005560 con 0x7fd91410c170 2024-09-12T03:04:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 Processor -- start 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 -- start start 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b854090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b861760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557b5b7ba2f0 con 0x557b5b8545d0 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557b5b761cf0 con 0x557b5b85f370 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91d199740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557b5b7b62b0 con 0x557b5b855b40 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd911ffb640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 0x7fd9040a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd918b5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd91935d640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 0x7fd9040a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd918b5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52068/0 (socket says 172.21.15.16:52068) 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd918b5c640 1 -- 172.21.15.16:0/1577231793 learned_addr learned my addr 172.21.15.16:0/1577231793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.880+0000 7fd913fff640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b854090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd913fff640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b861760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd911ffb640 1 -- 172.21.15.16:0/3274697039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd90c0cf210 con 0x7fd91410c170 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 -- 172.21.15.16:0/1577231793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 msgr2=0x557b5b854090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b854090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 -- 172.21.15.16:0/1577231793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 msgr2=0x557b5b861760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b861760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 -- 172.21.15.16:0/1577231793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b5b7463a0 con 0x557b5b8545d0 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd913fff640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b854090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd91935d640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 0x7fd9040a50c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0a4160 tx=0x7fd90c066b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd90800ac20 tx=0x7fd90800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37ce17a497ea8ce server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd8feffd640 1 -- 172.21.15.16:0/1577231793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90801d070 con 0x557b5b8545d0 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd8feffd640 1 -- 172.21.15.16:0/1577231793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd908018b20 con 0x557b5b8545d0 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.881+0000 7fd8feffd640 1 -- 172.21.15.16:0/1577231793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908010ad0 con 0x557b5b8545d0 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- 172.21.15.16:0/1577231793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 msgr2=0x557b5b854a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd90800ac20 tx=0x7fd90800f790 comp rx=0 tx=0).stop 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- 172.21.15.16:0/1577231793 shutdown_connections 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b861760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b854a10 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 --2- 172.21.15.16:0/1577231793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b854090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- 172.21.15.16:0/1577231793 >> 172.21.15.16:0/1577231793 conn(0x557b5b81d310 msgr2=0x557b5b850c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- 172.21.15.16:0/1577231793 shutdown_connections 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- 172.21.15.16:0/1577231793 wait complete. 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.882+0000 7fd91d199740 1 -- start start 2024-09-12T03:04:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 Processor -- start 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b855b40 0x557b5b85c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b85f370 0x557b5b85c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557b5b862070 con 0x557b5b8545d0 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557b5b775060 con 0x557b5b855b40 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557b5b8621f0 con 0x557b5b85f370 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd91d199740 10 client.0 _ll_register_callbacks cb 0x557b5b4c88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52076/0 (socket says 172.21.15.16:52076) 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 -- 172.21.15.16:0/3419616510 learned_addr learned my addr 172.21.15.16:0/3419616510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd918b5c640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b855b40 0x557b5b85c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b85f370 0x557b5b85c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 -- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b85f370 msgr2=0x557b5b85c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b85f370 0x557b5b85c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 -- 172.21.15.16:0/3419616510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b855b40 msgr2=0x557b5b85c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b855b40 0x557b5b85c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.883+0000 7fd913fff640 1 -- 172.21.15.16:0/3419616510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b5b52b3c0 con 0x557b5b8545d0 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd913fff640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd900009230 tx=0x7fd900005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be132efac2a9c29 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900035040 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd9000048f0 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd908018b20 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd900004af0 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd91d199740 10 client.19272 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557b5b86daf0 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd908018b20 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.884+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd90003a040 con 0x557b5b8545d0 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 10 client.19272 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 10 client.19272 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 20 client.19272 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b86efb0 0x557b5b8713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 4 client.19272 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b5b7c7860 con 0x557b5b86efb0 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd918b5c640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b86efb0 0x557b5b8713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd918b5c640 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b86efb0 0x557b5b8713a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4fe78759933144ee server_cookie=c7a42ad4b7627c32 in_seq=0 out_seq=0 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.885+0000 7fd8f3fff640 10 client.19272 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd8f3fff640 1 -- 172.21.15.16:0/3419616510 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557b5b7c7860 con 0x557b5b86efb0 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd8f3fff640 10 client.19272 handle_command_reply: tid=0 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 client.19272 shutdown 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 msgr2=0x557b5b85ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd900009230 tx=0x7fd900005dd0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 shutdown_connections 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b86efb0 0x557b5b8713a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b85f370 0x557b5b85c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b855b40 0x557b5b85c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 --2- 172.21.15.16:0/3419616510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b85ed50 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 >> 172.21.15.16:0/3419616510 conn(0x557b5b81d310 msgr2=0x557b5b856960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd8f3fff640 0 client.19272 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 shutdown_connections 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 1 -- 172.21.15.16:0/3419616510 wait complete. 2024-09-12T03:04:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.886+0000 7fd91d199740 20 client.19272 trim_cache size 0 max 16384 2024-09-12T03:04:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.899+0000 7fd91d199740 1 Processor -- start 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.899+0000 7fd91d199740 1 -- start start 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b8545d0 0x557b5b84fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b867860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141ad5b0 con 0x557b5b85f370 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141af0f0 con 0x557b5b8545d0 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd91d199740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141adc80 con 0x557b5b855b40 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52082/0 (socket says 172.21.15.16:52082) 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 -- 172.21.15.16:0/1204645131 learned_addr learned my addr 172.21.15.16:0/1204645131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b867860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b8545d0 0x557b5b84fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 -- 172.21.15.16:0/1204645131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 msgr2=0x557b5b867860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b867860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 -- 172.21.15.16:0/1204645131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b8545d0 msgr2=0x557b5b84fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b8545d0 0x557b5b84fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 -- 172.21.15.16:0/1204645131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b5b85a570 con 0x557b5b85f370 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b867860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.900+0000 7fd913fff640 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fd900031460 tx=0x7fd90002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7997c70dd0bd7904 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd8fe7fc640 1 -- 172.21.15.16:0/1204645131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900035040 con 0x557b5b85f370 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd8fe7fc640 1 -- 172.21.15.16:0/1204645131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd900005490 con 0x557b5b85f370 2024-09-12T03:04:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd8fe7fc640 1 -- 172.21.15.16:0/1204645131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900031a90 con 0x557b5b85f370 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 -- 172.21.15.16:0/1204645131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 msgr2=0x557b5b867da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fd900031460 tx=0x7fd90002a480 comp rx=0 tx=0).stop 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 -- 172.21.15.16:0/1204645131 shutdown_connections 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b85f370 0x557b5b867da0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b867860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 --2- 172.21.15.16:0/1204645131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b8545d0 0x557b5b84fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 -- 172.21.15.16:0/1204645131 >> 172.21.15.16:0/1204645131 conn(0x557b5b81d310 msgr2=0x557b5b84f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 -- 172.21.15.16:0/1204645131 shutdown_connections 2024-09-12T03:04:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.901+0000 7fd91d199740 1 -- 172.21.15.16:0/1204645131 wait complete. 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 -- start start 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 Processor -- start 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b8766e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b876c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141aea20 con 0x557b5b8545d0 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141ae350 con 0x557b5b85f370 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141ab590 con 0x557b5b855b40 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd91d199740 10 client.0 _ll_register_callbacks cb 0x7fd900009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.902+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52098/0 (socket says 172.21.15.16:52098) 2024-09-12T03:04:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 -- 172.21.15.16:0/1683369028 learned_addr learned my addr 172.21.15.16:0/1683369028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b876c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b8766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 -- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 msgr2=0x557b5b8766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b8766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 -- 172.21.15.16:0/1683369028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 msgr2=0x557b5b876c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b876c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 -- 172.21.15.16:0/1683369028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b5b877400 con 0x557b5b8545d0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd913fff640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd900004ec0 tx=0x7fd900005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42dc8f0930f43ec8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd8f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900035040 con 0x557b5b8545d0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd900003a70 con 0x557b5b8545d0 2024-09-12T03:04:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd900005490 con 0x557b5b8545d0 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.903+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd900003f00 con 0x557b5b8545d0 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 10 client.19278 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557b5b8ddc60 con 0x557b5b8545d0 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd900005490 con 0x557b5b8545d0 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd90004b5d0 con 0x557b5b8545d0 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 10 client.19278 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 10 client.19278 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 20 client.19278 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b8de260 0x557b5b8e0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 4 client.19278 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b5b7c1d70 con 0x557b5b8de260 2024-09-12T03:04:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.904+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b8de260 0x557b5b8e0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd918b5c640 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b8de260 0x557b5b8e0650 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3928daa66ddf82c server_cookie=cf468ec78578c145 in_seq=0 out_seq=0 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd8f3fff640 10 client.19278 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd8f3fff640 1 -- 172.21.15.16:0/1683369028 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b5b7c1d70 con 0x557b5b8de260 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd8f3fff640 10 client.19278 handle_command_reply: tid=0 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 client.19278 shutdown 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 msgr2=0x557b5b859b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd900004ec0 tx=0x7fd900005350 comp rx=0 tx=0).stop 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 shutdown_connections 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b5b85f370 0x557b5b876c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x557b5b8de260 0x557b5b8e0650 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b5b855b40 0x557b5b8766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 --2- 172.21.15.16:0/1683369028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b5b8545d0 0x557b5b859b80 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 >> 172.21.15.16:0/1683369028 conn(0x557b5b81d310 msgr2=0x557b5b84fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd8f3fff640 0 client.19278 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:04:24.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 shutdown_connections 2024-09-12T03:04:24.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 1 -- 172.21.15.16:0/1683369028 wait complete. 2024-09-12T03:04:24.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.905+0000 7fd91d199740 20 client.19278 trim_cache size 0 max 16384 2024-09-12T03:04:24.911 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 msgr2=0x7fd9040a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 0x7fd9040a50c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0a4160 tx=0x7fd90c066b40 comp rx=0 tx=0).stop 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 msgr2=0x7fd91419cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00eaa0 tx=0x7fd90c00ef60 comp rx=0 tx=0).stop 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 shutdown_connections 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd9040a2c00 0x7fd9040a50c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd91410c170 0x7fd91419cd50 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd914103660 0x7fd9141989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 --2- 172.21.15.16:0/3274697039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd914102d40 0x7fd914198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:24.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 >> 172.21.15.16:0/3274697039 conn(0x7fd9140fc7a0 msgr2=0x7fd9140fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:24.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 shutdown_connections 2024-09-12T03:04:24.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:24.906+0000 7fd91adc0640 1 -- 172.21.15.16:0/3274697039 wait complete. 2024-09-12T03:04:25.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 28.00 MB/sec execute 86 sec latency 18584.938 ms 2024-09-12T03:04:25.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:04:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:26.013 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 27.68 MB/sec execute 87 sec latency 19584.982 ms 2024-09-12T03:04:26.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:26 smithi016 ceph-mon[30925]: pgmap v832: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 5.1 MiB/s wr, 196 op/s 2024-09-12T03:04:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:26 smithi184 ceph-mon[35321]: pgmap v832: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 5.1 MiB/s wr, 196 op/s 2024-09-12T03:04:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:26 smithi033 ceph-mon[35092]: pgmap v832: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 5.1 MiB/s wr, 196 op/s 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 -- 172.21.15.16:0/3933002646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd4103660 msgr2=0x7fedd410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 --2- 172.21.15.16:0/3933002646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd4103660 0x7fedd410bb50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009900 tx=0x7fedc802a480 comp rx=0 tx=0).stop 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 -- 172.21.15.16:0/3933002646 shutdown_connections 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 --2- 172.21.15.16:0/3933002646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd410c170 0x7fedd410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 --2- 172.21.15.16:0/3933002646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd4103660 0x7fedd410bb50 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 --2- 172.21.15.16:0/3933002646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 0x7fedd4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 -- 172.21.15.16:0/3933002646 >> 172.21.15.16:0/3933002646 conn(0x7fedd40fc7a0 msgr2=0x7fedd40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 -- 172.21.15.16:0/3933002646 shutdown_connections 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.476+0000 7fedd91df640 1 -- 172.21.15.16:0/3933002646 wait complete. 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.477+0000 7fedd91df640 1 Processor -- start 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.477+0000 7fedd91df640 1 -- start start 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 0x7fedd41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 0x7fedd4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd4111f70 con 0x7fedd410c170 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd4111df0 con 0x7fedd4103660 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd91df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41120f0 con 0x7fedd4102d40 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52160/0 (socket says 172.21.15.16:52160) 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 -- 172.21.15.16:0/250180487 learned_addr learned my addr 172.21.15.16:0/250180487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 0x7fedd41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd259c640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 0x7fedd4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 -- 172.21.15.16:0/250180487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 msgr2=0x7fedd41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 0x7fedd41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 -- 172.21.15.16:0/250180487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 msgr2=0x7fedd4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 0x7fedd4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 -- 172.21.15.16:0/250180487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd419d480 con 0x7fedd410c170 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd259c640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 0x7fedd4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.478+0000 7fedd359e640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00eaa0 tx=0x7fedcc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.479+0000 7fedbbfff640 1 -- 172.21.15.16:0/250180487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc00cdb0 con 0x7fedd410c170 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.479+0000 7fedbbfff640 1 -- 172.21.15.16:0/250180487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedcc005020 con 0x7fedd410c170 2024-09-12T03:04:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.479+0000 7fedbbfff640 1 -- 172.21.15.16:0/250180487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0116b0 con 0x7fedd410c170 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.479+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd419d770 con 0x7fedd410c170 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.479+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd41a5050 con 0x7fedd410c170 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.481+0000 7fedbbfff640 1 -- 172.21.15.16:0/250180487 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fedcc005560 con 0x7fedd410c170 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.481+0000 7feddb5b8740 1 Processor -- start 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.481+0000 7feddb5b8740 1 -- start start 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba92d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba92da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5576ba8932f0 con 0x5576ba93c780 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576ba83acf0 con 0x5576ba92d5d0 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7feddb5b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5576ba88f2b0 con 0x5576ba92eb40 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52170/0 (socket says 172.21.15.16:52170) 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 -- 172.21.15.16:0/1501699359 learned_addr learned my addr 172.21.15.16:0/1501699359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba92da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba92d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 -- 172.21.15.16:0/1501699359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 msgr2=0x5576ba92d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba92d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 -- 172.21.15.16:0/1501699359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 msgr2=0x5576ba92da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba92da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 -- 172.21.15.16:0/1501699359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576ba81f3a0 con 0x5576ba93c780 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedbbfff640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 0x7fedbc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd259c640 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fedc80096c0 tx=0x7fedc8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3971559d1992d49a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.482+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 0x7fedbc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7fedbbfff640 1 -- 172.21.15.16:0/250180487 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedcc0cf1f0 con 0x7fedd410c170 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7fedb97fa640 1 -- 172.21.15.16:0/1501699359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8035040 con 0x5576ba93c780 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7fedb97fa640 1 -- 172.21.15.16:0/1501699359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8004030 con 0x5576ba93c780 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7fedb97fa640 1 -- 172.21.15.16:0/1501699359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8003bf0 con 0x5576ba93c780 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 0x7fedbc0a50c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x5576ba92aac0 tx=0x7fedc400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7feddb5b8740 1 -- 172.21.15.16:0/1501699359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 msgr2=0x5576ba93eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.483+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fedc80096c0 tx=0x7fedc8009560 comp rx=0 tx=0).stop 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- 172.21.15.16:0/1501699359 shutdown_connections 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba93c780 0x5576ba93eb70 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba92da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1501699359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba92d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- 172.21.15.16:0/1501699359 >> 172.21.15.16:0/1501699359 conn(0x5576ba8f6310 msgr2=0x5576ba929c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- 172.21.15.16:0/1501699359 shutdown_connections 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- 172.21.15.16:0/1501699359 wait complete. 2024-09-12T03:04:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- start start 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 Processor -- start 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92eb40 0x5576ba939480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba939c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5576ba93f420 con 0x5576ba92d5d0 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576ba84e060 con 0x5576ba92eb40 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5576ba93f5a0 con 0x5576ba93c780 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7feddb5b8740 10 client.0 _ll_register_callbacks cb 0x5576ba5a18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7fedd2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7fedd2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52174/0 (socket says 172.21.15.16:52174) 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7fedd2d9d640 1 -- 172.21.15.16:0/1568637819 learned_addr learned my addr 172.21.15.16:0/1568637819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.484+0000 7fedd259c640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92eb40 0x5576ba939480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd359e640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba939c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 -- 172.21.15.16:0/1568637819 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 msgr2=0x5576ba939c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba939c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 -- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92eb40 msgr2=0x5576ba939480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92eb40 0x5576ba939480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 -- 172.21.15.16:0/1568637819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576ba6043c0 con 0x5576ba92d5d0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd359e640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba939c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fedc4003e60 tx=0x7fedc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364f674b54d03955 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedb6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc402c060 con 0x5576ba92d5d0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedc4017030 con 0x5576ba92d5d0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc8004030 con 0x5576ba92d5d0 2024-09-12T03:04:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fedc4027410 con 0x5576ba92d5d0 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7feddb5b8740 10 client.19290 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.485+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5576ba94af90 con 0x5576ba92d5d0 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8004030 con 0x5576ba92d5d0 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fedc40117e0 con 0x5576ba92d5d0 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 10 client.19290 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 10 client.19290 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 20 client.19290 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba94c450 0x5576ba94e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 4 client.19290 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576ba8a0860 con 0x5576ba94c450 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.486+0000 7fedd259c640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba94c450 0x5576ba94e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7fedb6ff5640 10 client.19290 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:04:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7fedd259c640 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba94c450 0x5576ba94e840 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d563140761f9412 server_cookie=676f352e4e5d9ba3 in_seq=0 out_seq=0 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7fedb6ff5640 1 -- 172.21.15.16:0/1568637819 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5576ba8a0860 con 0x5576ba94c450 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7fedb6ff5640 10 client.19290 handle_command_reply: tid=0 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7feddb5b8740 1 client.19290 shutdown 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 msgr2=0x5576ba93c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.487+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fedc4003e60 tx=0x7fedc4004730 comp rx=0 tx=0).stop 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 shutdown_connections 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba94c450 0x5576ba94e840 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba939c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92eb40 0x5576ba939480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 --2- 172.21.15.16:0/1568637819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba93c1d0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7fedb6ff5640 0 client.19290 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 >> 172.21.15.16:0/1568637819 conn(0x5576ba8f6310 msgr2=0x5576ba92f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 shutdown_connections 2024-09-12T03:04:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 1 -- 172.21.15.16:0/1568637819 wait complete. 2024-09-12T03:04:26.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.488+0000 7feddb5b8740 20 client.19290 trim_cache size 0 max 16384 2024-09-12T03:04:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 Processor -- start 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 -- start start 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba9540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba93c780 0x5576ba954600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41ab370 con 0x5576ba92d5d0 2024-09-12T03:04:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41aceb0 con 0x5576ba93c780 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7feddb5b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41aba40 con 0x5576ba92eb40 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52176/0 (socket says 172.21.15.16:52176) 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 -- 172.21.15.16:0/2360465243 learned_addr learned my addr 172.21.15.16:0/2360465243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd359e640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba93c780 0x5576ba954600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd259c640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba9540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 -- 172.21.15.16:0/2360465243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 msgr2=0x5576ba9540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba9540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 -- 172.21.15.16:0/2360465243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba93c780 msgr2=0x5576ba954600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba93c780 0x5576ba954600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 -- 172.21.15.16:0/2360465243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576ba94ee70 con 0x5576ba92d5d0 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd259c640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba9540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.506+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fedc4018040 tx=0x7fedc4011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21af1fe8d9f3d88e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7fedb8ff9640 1 -- 172.21.15.16:0/2360465243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4037070 con 0x5576ba92d5d0 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7fedb8ff9640 1 -- 172.21.15.16:0/2360465243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc40275b0 con 0x5576ba92d5d0 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7fedb8ff9640 1 -- 172.21.15.16:0/2360465243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc40336b0 con 0x5576ba92d5d0 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 -- 172.21.15.16:0/2360465243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 msgr2=0x5576ba929e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fedc4018040 tx=0x7fedc4011fd0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 -- 172.21.15.16:0/2360465243 shutdown_connections 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba93c780 0x5576ba954600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba92eb40 0x5576ba9540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2360465243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92d5d0 0x5576ba929e10 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 -- 172.21.15.16:0/2360465243 >> 172.21.15.16:0/2360465243 conn(0x5576ba8f6310 msgr2=0x5576ba929700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 -- 172.21.15.16:0/2360465243 shutdown_connections 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.507+0000 7feddb5b8740 1 -- 172.21.15.16:0/2360465243 wait complete. 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 -- start start 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 Processor -- start 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba936650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba9370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41ac7e0 con 0x5576ba92eb40 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41ac110 con 0x5576ba92d5d0 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41a9350 con 0x5576ba93c780 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7feddb5b8740 10 client.0 _ll_register_callbacks cb 0x7fedc400bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7fedd259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7fedd259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52178/0 (socket says 172.21.15.16:52178) 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7fedd259c640 1 -- 172.21.15.16:0/2529639422 learned_addr learned my addr 172.21.15.16:0/2529639422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.508+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba936650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd359e640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba9370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 -- 172.21.15.16:0/2529639422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 msgr2=0x5576ba9370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba9370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 -- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 msgr2=0x5576ba936650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba936650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 -- 172.21.15.16:0/2529639422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576ba9b6290 con 0x5576ba92eb40 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd359e640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba9370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedd259c640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009900 tx=0x7fedc802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3d01cebf90db20 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedb6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8055070 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc40275b0 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fedc80317c0 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fedc8061050 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7feddb5b8740 10 client.19296 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5576ba9b74a0 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.509+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc40275b0 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fedc803a040 con 0x5576ba92eb40 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 10 client.19296 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 10 client.19296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 20 client.19296 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba9b77e0 0x5576ba9b9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 4 client.19296 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:04:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576ba89ad70 con 0x5576ba9b77e0 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba9b77e0 0x5576ba9b9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7fedb6ff5640 10 client.19296 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.510+0000 7fedd2d9d640 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba9b77e0 0x5576ba9b9bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e79e4e86f5f587be server_cookie=a8d400a749a05fa3 in_seq=0 out_seq=0 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedb6ff5640 1 -- 172.21.15.16:0/2529639422 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576ba89ad70 con 0x5576ba9b77e0 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedb6ff5640 10 client.19296 handle_command_reply: tid=0 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 client.19296 shutdown 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 msgr2=0x5576ba936b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009900 tx=0x7fedc802a480 comp rx=0 tx=0).stop 2024-09-12T03:04:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 shutdown_connections 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5576ba9b77e0 0x5576ba9b9bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5576ba93c780 0x5576ba9370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5576ba92eb40 0x5576ba936b90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 --2- 172.21.15.16:0/2529639422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576ba92d5d0 0x5576ba936650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 >> 172.21.15.16:0/2529639422 conn(0x5576ba8f6310 msgr2=0x5576ba92f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedb6ff5640 0 client.19296 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 shutdown_connections 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 1 -- 172.21.15.16:0/2529639422 wait complete. 2024-09-12T03:04:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7feddb5b8740 20 client.19296 trim_cache size 0 max 16384 2024-09-12T03:04:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 msgr2=0x7fedbc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 0x7fedbc0a50c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x5576ba92aac0 tx=0x7fedc400b390 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 msgr2=0x7fedd419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.511+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00eaa0 tx=0x7fedcc00ef60 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 shutdown_connections 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fedbc0a2c00 0x7fedbc0a50c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd410c170 0x7fedd419cda0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedd4103660 0x7fedd4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 --2- 172.21.15.16:0/250180487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd4102d40 0x7fedd41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 >> 172.21.15.16:0/250180487 conn(0x7fedd40fc7a0 msgr2=0x7fedd40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 shutdown_connections 2024-09-12T03:04:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:26.512+0000 7fedd91df640 1 -- 172.21.15.16:0/250180487 wait complete. 2024-09-12T03:04:27.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 27.36 MB/sec execute 88 sec latency 20585.016 ms 2024-09-12T03:04:27.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:04:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:27.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 -- 172.21.15.16:0/3060312585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102570 msgr2=0x7f36d41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 --2- 172.21.15.16:0/3060312585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102570 0x7f36d41029b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009900 tx=0x7f36c002a480 comp rx=0 tx=0).stop 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 -- 172.21.15.16:0/3060312585 shutdown_connections 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 --2- 172.21.15.16:0/3060312585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102ef0 0x7f36d410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 --2- 172.21.15.16:0/3060312585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102570 0x7f36d41029b0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 --2- 172.21.15.16:0/3060312585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 0x7f36d4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 -- 172.21.15.16:0/3060312585 >> 172.21.15.16:0/3060312585 conn(0x7f36d40fc7d0 msgr2=0x7f36d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 -- 172.21.15.16:0/3060312585 shutdown_connections 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.929+0000 7f36dae9b640 1 -- 172.21.15.16:0/3060312585 wait complete. 2024-09-12T03:04:27.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 Processor -- start 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 -- start start 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 0x7f36d419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102570 0x7f36d419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4110f80 con 0x7f36d4102ef0 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4110e00 con 0x7f36d4102570 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36dae9b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4111100 con 0x7f36d4100e20 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52194/0 (socket says 172.21.15.16:52194) 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36d9438640 1 -- 172.21.15.16:0/1110684467 learned_addr learned my addr 172.21.15.16:0/1110684467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.930+0000 7f36d3fff640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102570 0x7f36d419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d8c37640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 0x7f36d419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 -- 172.21.15.16:0/1110684467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 msgr2=0x7f36d419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 0x7f36d419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 -- 172.21.15.16:0/1110684467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102570 msgr2=0x7f36d419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102570 0x7f36d419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 -- 172.21.15.16:0/1110684467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d419f6a0 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d9438640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00eaa0 tx=0x7f36cc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d1ffb640 1 -- 172.21.15.16:0/1110684467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00cdb0 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d1ffb640 1 -- 172.21.15.16:0/1110684467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36cc005020 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d419f990 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36d1ffb640 1 -- 172.21.15.16:0/1110684467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0116b0 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.931+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d41a7270 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.933+0000 7f36d1ffb640 1 -- 172.21.15.16:0/1110684467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f36cc005560 con 0x7f36d4102ef0 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 Processor -- start 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 -- start start 2024-09-12T03:04:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 0x5585c2bada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d1ffb640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 0x7f36c40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585c2b132f0 con 0x5585c2bb8370 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585c2abacf0 con 0x5585c2baeb40 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36dd274740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5585c2b0f2b0 con 0x5585c2bad5d0 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d8c37640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 0x7f36c40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52202/0 (socket says 172.21.15.16:52202) 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d9438640 1 -- 172.21.15.16:0/2542507313 learned_addr learned my addr 172.21.15.16:0/2542507313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d1ffb640 1 -- 172.21.15.16:0/1110684467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f36cc0cf310 con 0x7f36d4102ef0 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.934+0000 7f36d3fff640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d3fff640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 0x5585c2bada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 -- 172.21.15.16:0/2542507313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 msgr2=0x5585c2bada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 0x5585c2bada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 -- 172.21.15.16:0/2542507313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 msgr2=0x5585c2bad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 -- 172.21.15.16:0/2542507313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c2a9f3a0 con 0x5585c2bb8370 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d3fff640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 0x5585c2bada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d8c37640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 0x7f36c40a5040 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009b60 tx=0x7f36c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36d9438640 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00efd0 tx=0x7f36cc01eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c610a76fc7e686 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36beffd640 1 -- 172.21.15.16:0/2542507313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc09c220 con 0x5585c2bb8370 2024-09-12T03:04:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36beffd640 1 -- 172.21.15.16:0/2542507313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36cc09bb20 con 0x5585c2bb8370 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.935+0000 7f36beffd640 1 -- 172.21.15.16:0/2542507313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc098190 con 0x5585c2bb8370 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- 172.21.15.16:0/2542507313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 msgr2=0x5585c2bba760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00efd0 tx=0x7f36cc01eb40 comp rx=0 tx=0).stop 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- 172.21.15.16:0/2542507313 shutdown_connections 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bba760 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bad5d0 0x5585c2bada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 --2- 172.21.15.16:0/2542507313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- 172.21.15.16:0/2542507313 >> 172.21.15.16:0/2542507313 conn(0x5585c2b76310 msgr2=0x5585c2ba9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- 172.21.15.16:0/2542507313 shutdown_connections 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- 172.21.15.16:0/2542507313 wait complete. 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 -- start start 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.936+0000 7f36dd274740 1 Processor -- start 2024-09-12T03:04:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bb7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bb5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585c2bbb070 con 0x5585c2bb8370 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5585c2ace060 con 0x5585c2bad5d0 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5585c2bbb1f0 con 0x5585c2baeb40 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 10 client.0 _ll_register_callbacks cb 0x5585c28218c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52218/0 (socket says 172.21.15.16:52218) 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 -- 172.21.15.16:0/334529055 learned_addr learned my addr 172.21.15.16:0/334529055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d8c37640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bb7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d3fff640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bb5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 -- 172.21.15.16:0/334529055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 msgr2=0x5585c2bb5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bb5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 -- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 msgr2=0x5585c2bb7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bb7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 -- 172.21.15.16:0/334529055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c28843c0 con 0x5585c2bb8370 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d8c37640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bb7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d3fff640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bb5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36d9438640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0bd010 tx=0x7f36cc0cd0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e654a971e7f015 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc09c220 con 0x5585c2bb8370 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f36cc0b9070 con 0x5585c2bb8370 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.937+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36cc09bb20 con 0x5585c2bb8370 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f36cc095050 con 0x5585c2bb8370 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 10 client.19308 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5585c2bc6b60 con 0x5585c2bb8370 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36cc09bb20 con 0x5585c2bb8370 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f36cc00f040 con 0x5585c2bb8370 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 10 client.19308 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 10 client.19308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 20 client.19308 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2bc7eb0 0x5585c2bca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 4 client.19308 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585c2b20860 con 0x5585c2bc7eb0 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.938+0000 7f36d8c37640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2bc7eb0 0x5585c2bca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.939+0000 7f36d8c37640 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2bc7eb0 0x5585c2bca2a0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71988d296656c743 server_cookie=9085caa0bd5f13ae in_seq=0 out_seq=0 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.939+0000 7f36b3fff640 10 client.19308 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.939+0000 7f36b3fff640 1 -- 172.21.15.16:0/334529055 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5585c2b20860 con 0x5585c2bc7eb0 2024-09-12T03:04:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.939+0000 7f36b3fff640 10 client.19308 handle_command_reply: tid=0 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.939+0000 7f36dd274740 1 client.19308 shutdown 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 msgr2=0x5585c2bb57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0bd010 tx=0x7f36cc0cd0a0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 shutdown_connections 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2bc7eb0 0x5585c2bca2a0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bb57f0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bb5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 --2- 172.21.15.16:0/334529055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bb7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 >> 172.21.15.16:0/334529055 conn(0x5585c2b76310 msgr2=0x5585c2baf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36b3fff640 0 client.19308 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 shutdown_connections 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 1 -- 172.21.15.16:0/334529055 wait complete. 2024-09-12T03:04:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.940+0000 7f36dd274740 20 client.19308 trim_cache size 0 max 16384 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 Processor -- start 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 -- start start 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bb4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bb8370 0x5585c2bcd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41af6e0 con 0x5585c2bad5d0 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b1220 con 0x5585c2baeb40 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36dd274740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41afdb0 con 0x5585c2bb8370 2024-09-12T03:04:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36d8c37640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36d8c37640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52220/0 (socket says 172.21.15.16:52220) 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.953+0000 7f36d8c37640 1 -- 172.21.15.16:0/3931659826 learned_addr learned my addr 172.21.15.16:0/3931659826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d3fff640 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bb4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d9438640 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bb8370 0x5585c2bcd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 -- 172.21.15.16:0/3931659826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bb8370 msgr2=0x5585c2bcd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bb8370 0x5585c2bcd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 -- 172.21.15.16:0/3931659826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 msgr2=0x5585c2bb4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bb4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 -- 172.21.15.16:0/3931659826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c2bb0120 con 0x5585c2bad5d0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x5585c2bbb1b0 tx=0x7f36c8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4d368a8f3e5ab9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36be7fc640 1 -- 172.21.15.16:0/3931659826 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8004310 con 0x5585c2bad5d0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36be7fc640 1 -- 172.21.15.16:0/3931659826 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c802f070 con 0x5585c2bad5d0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.954+0000 7f36be7fc640 1 -- 172.21.15.16:0/3931659826 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8029720 con 0x5585c2bad5d0 2024-09-12T03:04:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- 172.21.15.16:0/3931659826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 msgr2=0x5585c2ba9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x5585c2bbb1b0 tx=0x7f36c8014000 comp rx=0 tx=0).stop 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- 172.21.15.16:0/3931659826 shutdown_connections 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2bb8370 0x5585c2bcd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2baeb40 0x5585c2bb4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 --2- 172.21.15.16:0/3931659826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bad5d0 0x5585c2ba9d70 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- 172.21.15.16:0/3931659826 >> 172.21.15.16:0/3931659826 conn(0x5585c2b76310 msgr2=0x5585c2bb3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- 172.21.15.16:0/3931659826 shutdown_connections 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- 172.21.15.16:0/3931659826 wait complete. 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 -- start start 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.955+0000 7f36dd274740 1 Processor -- start 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bcda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bcadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b0b50 con 0x5585c2bb8370 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b0480 con 0x5585c2bad5d0 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41ad6c0 con 0x5585c2baeb40 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36dd274740 10 client.0 _ll_register_callbacks cb 0x7f36c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d9438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52234/0 (socket says 172.21.15.16:52234) 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d9438640 1 -- 172.21.15.16:0/3213307961 learned_addr learned my addr 172.21.15.16:0/3213307961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bcda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d3fff640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bcadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 -- 172.21.15.16:0/3213307961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 msgr2=0x5585c2bcadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bcadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 -- 172.21.15.16:0/3213307961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 msgr2=0x5585c2bcb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d8c37640 1 -- 172.21.15.16:0/3213307961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c2bcbb30 con 0x5585c2bad5d0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.956+0000 7f36d9438640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36d8c37640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bcda50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011870 tx=0x7f36c8012560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=940b947538536dde server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c80044e0 con 0x5585c2bad5d0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f36c80037f0 con 0x5585c2bad5d0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36c802f070 con 0x5585c2bad5d0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f36c8003a30 con 0x5585c2bad5d0 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36dd274740 10 client.28616 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5585c2c32820 con 0x5585c2bad5d0 2024-09-12T03:04:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.957+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c802f070 con 0x5585c2bad5d0 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f36c80204f0 con 0x5585c2bad5d0 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 10 client.28616 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 10 client.28616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 20 client.28616 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2c32ee0 0x5585c2c352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 4 client.28616 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585c2b1ad70 con 0x5585c2c32ee0 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36d3fff640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2c32ee0 0x5585c2c352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36d3fff640 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2c32ee0 0x5585c2c352d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ac287c507b5cd84 server_cookie=6dfeaf6d1f653594 in_seq=0 out_seq=0 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36b3fff640 10 client.28616 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36b3fff640 1 -- 172.21.15.16:0/3213307961 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585c2b1ad70 con 0x5585c2c32ee0 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36b3fff640 10 client.28616 handle_command_reply: tid=0 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.958+0000 7f36dd274740 1 client.28616 shutdown 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 msgr2=0x5585c2bcda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bcda50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011870 tx=0x7f36c8012560 comp rx=0 tx=0).stop 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 shutdown_connections 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5585c2c32ee0 0x5585c2c352d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585c2bb8370 0x5585c2bcb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585c2baeb40 0x5585c2bcadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 --2- 172.21.15.16:0/3213307961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5585c2bad5d0 0x5585c2bcda50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36b3fff640 0 client.28616 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 >> 172.21.15.16:0/3213307961 conn(0x5585c2b76310 msgr2=0x5585c2bb68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 shutdown_connections 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 1 -- 172.21.15.16:0/3213307961 wait complete. 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.959+0000 7f36dd274740 20 client.28616 trim_cache size 0 max 16384 2024-09-12T03:04:27.966 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 msgr2=0x7f36c40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 0x7f36c40a5040 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009b60 tx=0x7f36c8005f90 comp rx=0 tx=0).stop 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 msgr2=0x7f36d419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00eaa0 tx=0x7f36cc00ef60 comp rx=0 tx=0).stop 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 shutdown_connections 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36c40a2b80 0x7f36c40a5040 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4102ef0 0x7f36d419efc0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36d4102570 0x7f36d419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 --2- 172.21.15.16:0/1110684467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d4100e20 0x7f36d419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 >> 172.21.15.16:0/1110684467 conn(0x7f36d40fc7d0 msgr2=0x7f36d406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 shutdown_connections 2024-09-12T03:04:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:27.960+0000 7f36dae9b640 1 -- 172.21.15.16:0/1110684467 wait complete. 2024-09-12T03:04:28.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 27.06 MB/sec execute 89 sec latency 21585.050 ms 2024-09-12T03:04:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:28 smithi016 ceph-mon[30925]: pgmap v833: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 5.6 MiB/s wr, 214 op/s 2024-09-12T03:04:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:28 smithi184 ceph-mon[35321]: pgmap v833: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 5.6 MiB/s wr, 214 op/s 2024-09-12T03:04:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:28 smithi033 ceph-mon[35092]: pgmap v833: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 5.6 MiB/s wr, 214 op/s 2024-09-12T03:04:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:29.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 26.75 MB/sec execute 90 sec latency 22585.095 ms 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 -- 172.21.15.16:0/1539400556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00102570 msgr2=0x7f8c001029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/1539400556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00102570 0x7f8c001029b0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009900 tx=0x7f8bf402a480 comp rx=0 tx=0).stop 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 -- 172.21.15.16:0/1539400556 shutdown_connections 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/1539400556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102ef0 0x7f8c0010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/1539400556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00102570 0x7f8c001029b0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/1539400556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00100e20 0x7f8c00101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 -- 172.21.15.16:0/1539400556 >> 172.21.15.16:0/1539400556 conn(0x7f8c000fc7d0 msgr2=0x7f8c000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 -- 172.21.15.16:0/1539400556 shutdown_connections 2024-09-12T03:04:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.134+0000 7f8c04f9e640 1 -- 172.21.15.16:0/1539400556 wait complete. 2024-09-12T03:04:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 Processor -- start 2024-09-12T03:04:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 -- start start 2024-09-12T03:04:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102570 0x7f8c00198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 0x7f8c0019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c00110f80 con 0x7f8c00100e20 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8c00110e00 con 0x7f8c00102570 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8c04f9e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8c00111100 con 0x7f8c00102ef0 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8bfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8bfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52250/0 (socket says 172.21.15.16:52250) 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.135+0000 7f8bfe59c640 1 -- 172.21.15.16:0/3419615704 learned_addr learned my addr 172.21.15.16:0/3419615704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfed9d640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 0x7f8c0019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfdd9b640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102570 0x7f8c00198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 -- 172.21.15.16:0/3419615704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 msgr2=0x7f8c0019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 0x7f8c0019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 -- 172.21.15.16:0/3419615704 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102570 msgr2=0x7f8c00198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102570 0x7f8c00198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 -- 172.21.15.16:0/3419615704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0019d480 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfed9d640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 0x7f8c0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bfe59c640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000e9f0 tx=0x7f8bf000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000cd80 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0019d770 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0005500 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0010690 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.136+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c001a5050 con 0x7f8c00100e20 2024-09-12T03:04:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.137+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c00101200 con 0x7f8c00100e20 2024-09-12T03:04:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.138+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8bf00046c0 con 0x7f8c00100e20 2024-09-12T03:04:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.139+0000 7f8bef7fe640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 0x7f8be80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.139+0000 7f8bfdd9b640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 0x7f8be80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.139+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8bf00cf5f0 con 0x7f8c00100e20 2024-09-12T03:04:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.140+0000 7f8bfdd9b640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 0x7f8be80a5040 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8c000fffe0 tx=0x7f8bf4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:29.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.144+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8bf009c500 con 0x7f8c00100e20 2024-09-12T03:04:29.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.416+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c0010a0a0 con 0x7f8c00100e20 2024-09-12T03:04:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.420+0000 7f8bef7fe640 1 -- 172.21.15.16:0/3419615704 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f8bf009be00 con 0x7f8c00100e20 2024-09-12T03:04:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:04:29.423 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:04:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:04:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.427+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 msgr2=0x7f8be80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.427+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 0x7f8be80a5040 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8c000fffe0 tx=0x7f8bf4005a20 comp rx=0 tx=0).stop 2024-09-12T03:04:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.427+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 msgr2=0x7f8c001984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.427+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000e9f0 tx=0x7f8bf000edf0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 shutdown_connections 2024-09-12T03:04:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c00102ef0 0x7f8c0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c00102570 0x7f8c00198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8be80a2b80 0x7f8be80a5040 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 --2- 172.21.15.16:0/3419615704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c00100e20 0x7f8c001984d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.429+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 >> 172.21.15.16:0/3419615704 conn(0x7f8c000fc7d0 msgr2=0x7f8c0006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.431+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 shutdown_connections 2024-09-12T03:04:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:29.433+0000 7f8c04f9e640 1 -- 172.21.15.16:0/3419615704 wait complete. 2024-09-12T03:04:29.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:29 smithi016 ceph-mon[30925]: pgmap v834: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-12T03:04:29.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:29 smithi033 ceph-mon[35092]: pgmap v834: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-12T03:04:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:29 smithi184 ceph-mon[35321]: pgmap v834: 97 pgs: 97 active+clean; 615 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-12T03:04:29.669 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:04:29.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:04:30.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 26.46 MB/sec execute 91 sec latency 23585.181 ms 2024-09-12T03:04:30.127 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id f0607e58 --release --await' 2024-09-12T03:04:30.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id f0607e58 --release --await 2024-09-12T03:04:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:30.382 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3419615704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:30 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3419615704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:30.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:30 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3419615704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 -- 172.21.15.16:0/2712271460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4106f10 msgr2=0x7f26b410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/2712271460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4106f10 0x7f26b410ea90 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009ca0 tx=0x7f26ac02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 -- 172.21.15.16:0/2712271460 shutdown_connections 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/2712271460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4106f10 0x7f26b410ea90 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/2712271460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4102790 0x7f26b41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/2712271460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4101de0 0x7f26b41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 -- 172.21.15.16:0/2712271460 >> 172.21.15.16:0/2712271460 conn(0x7f26b40fd8c0 msgr2=0x7f26b40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 -- 172.21.15.16:0/2712271460 shutdown_connections 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.759+0000 7f26b9a9a640 1 -- 172.21.15.16:0/2712271460 wait complete. 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 Processor -- start 2024-09-12T03:04:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 -- start start 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 0x7f26b4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4106f10 0x7f26b419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b4111f70 con 0x7f26b4101de0 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b4111df0 con 0x7f26b4106f10 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b9a9a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41120f0 con 0x7f26b4102790 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52270/0 (socket says 172.21.15.16:52270) 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b37fe640 1 -- 172.21.15.16:0/3941102082 learned_addr learned my addr 172.21.15.16:0/3941102082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.760+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4106f10 0x7f26b419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 0x7f26b4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 -- 172.21.15.16:0/3941102082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 msgr2=0x7f26b4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 0x7f26b4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 -- 172.21.15.16:0/3941102082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4106f10 msgr2=0x7f26b419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4106f10 0x7f26b419cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 -- 172.21.15.16:0/3941102082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b419d380 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 0x7f26b4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f26a400d950 tx=0x7f26a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b0ff9640 1 -- 172.21.15.16:0/3941102082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4017030 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b419d670 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b0ff9640 1 -- 172.21.15.16:0/3941102082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a4004020 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b0ff9640 1 -- 172.21.15.16:0/3941102082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a40047a0 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.761+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b41a4f50 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26b0ff9640 1 -- 172.21.15.16:0/3941102082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f26a40041d0 con 0x7f26b4101de0 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 Processor -- start 2024-09-12T03:04:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 -- start start 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f494e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55803f3fa280 con 0x55803f4a4150 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55803f3a1b10 con 0x55803f494a40 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.764+0000 7f26bbe73740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55803f3f6240 con 0x55803f495360 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52274/0 (socket says 172.21.15.16:52274) 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 -- 172.21.15.16:0/3858885009 learned_addr learned my addr 172.21.15.16:0/3858885009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f494e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b0ff9640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 0x7f269c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 -- 172.21.15.16:0/3858885009 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 msgr2=0x55803f4a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 -- 172.21.15.16:0/3858885009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 msgr2=0x55803f494e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f494e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 -- 172.21.15.16:0/3858885009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55803f3ae1f0 con 0x55803f4a4150 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 0x7f269c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b0ff9640 1 -- 172.21.15.16:0/3941102082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26a40cea70 con 0x7f26b4101de0 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f26a800bac0 tx=0x7f26a800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a115d4debc43ef0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.765+0000 7f2699ffb640 1 -- 172.21.15.16:0/3858885009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8017420 con 0x55803f4a4150 2024-09-12T03:04:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f2699ffb640 1 -- 172.21.15.16:0/3858885009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a8017aa0 con 0x55803f4a4150 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f2699ffb640 1 -- 172.21.15.16:0/3858885009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a800fbd0 con 0x55803f4a4150 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 -- 172.21.15.16:0/3858885009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 msgr2=0x55803f4a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 0x7f269c0a5040 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f26ac005e30 tx=0x7f26ac0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f26a800bac0 tx=0x7f26a800bf80 comp rx=0 tx=0).stop 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 -- 172.21.15.16:0/3858885009 shutdown_connections 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f4a4150 0x55803f4a6540 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3858885009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f494e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 -- 172.21.15.16:0/3858885009 >> 172.21.15.16:0/3858885009 conn(0x55803f4910f0 msgr2=0x55803f4914d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 -- 172.21.15.16:0/3858885009 shutdown_connections 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.766+0000 7f26bbe73740 1 -- 172.21.15.16:0/3858885009 wait complete. 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 -- start start 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 Processor -- start 2024-09-12T03:04:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f494a40 0x55803f4a1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f4a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55803f4a6d20 con 0x55803f495360 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55803f3b4f10 con 0x55803f4a4150 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55803f4a6ea0 con 0x55803f494a40 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26bbe73740 10 client.0 _ll_register_callbacks cb 0x55803f37da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52288/0 (socket says 172.21.15.16:52288) 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.767+0000 7f26b2ffd640 1 -- 172.21.15.16:0/2635757333 learned_addr learned my addr 172.21.15.16:0/2635757333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b3fff640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f4a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b37fe640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f494a40 0x55803f4a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 -- 172.21.15.16:0/2635757333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f494a40 msgr2=0x55803f4a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f494a40 0x55803f4a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 -- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 msgr2=0x55803f4a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f4a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 -- 172.21.15.16:0/2635757333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55803f16b6a0 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f26a800f4d0 tx=0x7f26a8005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f291f0144c0dae61 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f2692ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8027020 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26a8017620 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a8017aa0 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26a8017820 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26bbe73740 10 client.19332 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55803f4b29a0 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.768+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a8017aa0 con 0x55803f495360 2024-09-12T03:04:30.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f26a80239d0 con 0x55803f495360 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 10 client.19332 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 10 client.19332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 20 client.19332 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f4b3d50 0x55803f4b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 4 client.19332 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55803f4077f0 con 0x55803f4b3d50 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26b37fe640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f4b3d50 0x55803f4b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f26b37fe640 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f4b3d50 0x55803f4b6140 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a831094bb61741b server_cookie=e692f9b85c4915d5 in_seq=0 out_seq=0 2024-09-12T03:04:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.769+0000 7f2692ffd640 10 client.19332 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f2692ffd640 1 -- 172.21.15.16:0/2635757333 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55803f4077f0 con 0x55803f4b3d50 2024-09-12T03:04:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f2692ffd640 10 client.19332 handle_command_reply: tid=0 2024-09-12T03:04:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 client.19332 shutdown 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 msgr2=0x55803f4a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f26a800f4d0 tx=0x7f26a8005eb0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 shutdown_connections 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f4b3d50 0x55803f4b6140 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f4a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4a15d0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 --2- 172.21.15.16:0/2635757333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f494a40 0x55803f4a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 >> 172.21.15.16:0/2635757333 conn(0x55803f4910f0 msgr2=0x55803f496e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.770+0000 7f2692ffd640 0 client.19332 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.771+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 shutdown_connections 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.771+0000 7f26bbe73740 1 -- 172.21.15.16:0/2635757333 wait complete. 2024-09-12T03:04:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.771+0000 7f26bbe73740 20 client.19332 trim_cache size 0 max 16384 2024-09-12T03:04:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.788+0000 7f26bbe73740 1 Processor -- start 2024-09-12T03:04:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.788+0000 7f26bbe73740 1 -- start start 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.788+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f4bb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f4a4150 0x55803f49e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26bbe73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41ab3b0 con 0x55803f495360 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26bbe73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41acef0 con 0x55803f494a40 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26bbe73740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41aba80 con 0x55803f4a4150 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52298/0 (socket says 172.21.15.16:52298) 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 -- 172.21.15.16:0/635128066 learned_addr learned my addr 172.21.15.16:0/635128066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b3fff640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f4a4150 0x55803f49e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b37fe640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f4bb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 -- 172.21.15.16:0/635128066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f4a4150 msgr2=0x55803f49e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f4a4150 0x55803f49e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 -- 172.21.15.16:0/635128066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 msgr2=0x55803f4bb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f4bb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 -- 172.21.15.16:0/635128066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55803f4a6e60 con 0x55803f495360 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b37fe640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f4bb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.789+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f26a800b430 tx=0x7f26a8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b632a24393ae3a17 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26997fa640 1 -- 172.21.15.16:0/635128066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8027050 con 0x55803f495360 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26997fa640 1 -- 172.21.15.16:0/635128066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a800fcf0 con 0x55803f495360 2024-09-12T03:04:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26997fa640 1 -- 172.21.15.16:0/635128066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8021a20 con 0x55803f495360 2024-09-12T03:04:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26bbe73740 1 -- 172.21.15.16:0/635128066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 msgr2=0x55803f4bbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26bbe73740 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f26a800b430 tx=0x7f26a8001e50 comp rx=0 tx=0).stop 2024-09-12T03:04:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26bbe73740 1 -- 172.21.15.16:0/635128066 shutdown_connections 2024-09-12T03:04:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.790+0000 7f26bbe73740 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f4a4150 0x55803f49e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f495360 0x55803f4bbca0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 --2- 172.21.15.16:0/635128066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f494a40 0x55803f4bb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 -- 172.21.15.16:0/635128066 >> 172.21.15.16:0/635128066 conn(0x55803f4910f0 msgr2=0x55803f4a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 -- 172.21.15.16:0/635128066 shutdown_connections 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 -- 172.21.15.16:0/635128066 wait complete. 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 -- start start 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 Processor -- start 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 0x55803f4a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.791+0000 7f26bbe73740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f51d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26bbe73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41ac820 con 0x55803f494a40 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26bbe73740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41ac150 con 0x55803f4a4150 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26bbe73740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b41a9390 con 0x55803f495360 2024-09-12T03:04:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 0x55803f4a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26bbe73740 10 client.0 _ll_register_callbacks cb 0x7f26ac005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 0x55803f4a8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52302/0 (socket says 172.21.15.16:52302) 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b37fe640 1 -- 172.21.15.16:0/3603955084 learned_addr learned my addr 172.21.15.16:0/3603955084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f51d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 -- 172.21.15.16:0/3603955084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 msgr2=0x55803f4a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 -- 172.21.15.16:0/3603955084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 msgr2=0x55803f4a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 0x55803f4a8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b3fff640 1 -- 172.21.15.16:0/3603955084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55803f4a9020 con 0x55803f4a4150 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.792+0000 7f26b2ffd640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f26b3fff640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f51d730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006010 tx=0x7f26ac042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a82bd87d8c778678 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f2692ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac02fce0 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26ac003970 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a800fcf0 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26ac02fe80 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f26bbe73740 10 client.28637 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55803f51edc0 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.793+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a800fcf0 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f26ac0305c0 con 0x55803f4a4150 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 10 client.28637 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 10 client.28637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 20 client.28637 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f51f510 0x55803f521950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 4 client.28637 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:04:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55803f401d00 con 0x55803f51f510 2024-09-12T03:04:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f51f510 0x55803f521950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f2692ffd640 10 client.28637 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.794+0000 7f26b37fe640 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f51f510 0x55803f521950 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5eec5fce658cc01 server_cookie=77109c6aead9ba4e in_seq=0 out_seq=0 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f2692ffd640 1 -- 172.21.15.16:0/3603955084 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55803f401d00 con 0x55803f51f510 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f2692ffd640 10 client.28637 handle_command_reply: tid=0 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 client.28637 shutdown 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 msgr2=0x55803f51d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f51d730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006010 tx=0x7f26ac042000 comp rx=0 tx=0).stop 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 shutdown_connections 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55803f51f510 0x55803f521950 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55803f4a4150 0x55803f51d730 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55803f495360 0x55803f4a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f2692ffd640 0 client.28637 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 --2- 172.21.15.16:0/3603955084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55803f494a40 0x55803f4a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 >> 172.21.15.16:0/3603955084 conn(0x55803f4910f0 msgr2=0x55803f48fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 shutdown_connections 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 1 -- 172.21.15.16:0/3603955084 wait complete. 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.796+0000 7f26bbe73740 20 client.28637 trim_cache size 0 max 16384 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "54aace7f-e74e-427b-9156-e57958d5a5dd", 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:04:30.801 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 msgr2=0x7f269c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 0x7f269c0a5040 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f26ac005e30 tx=0x7f26ac0095c0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 msgr2=0x7f26b41983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f26a400d950 tx=0x7f26a400de10 comp rx=0 tx=0).stop 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 shutdown_connections 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b4106f10 0x7f26b419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b4102790 0x7f26b4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f269c0a2b80 0x7f269c0a5040 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 --2- 172.21.15.16:0/3941102082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b4101de0 0x7f26b41983d0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 >> 172.21.15.16:0/3941102082 conn(0x7f26b40fd8c0 msgr2=0x7f26b40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 shutdown_connections 2024-09-12T03:04:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:30.797+0000 7f26b9a9a640 1 -- 172.21.15.16:0/3941102082 wait complete. 2024-09-12T03:04:30.812 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:31.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 26.17 MB/sec execute 92 sec latency 24585.227 ms 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: pgmap v835: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 6.4 MiB/s wr, 232 op/s 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:31.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:31 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: pgmap v835: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 6.4 MiB/s wr, 232 op/s 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:31.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:31.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:31 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: pgmap v835: 97 pgs: 97 active+clean; 611 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 6.4 MiB/s wr, 232 op/s 2024-09-12T03:04:31.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:04:31.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:31 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:31.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:04:32.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85265 25.89 MB/sec execute 93 sec latency 25585.274 ms 2024-09-12T03:04:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 -- 172.21.15.16:0/3992084177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c106f20 msgr2=0x7f3c5c10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 --2- 172.21.15.16:0/3992084177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c106f20 0x7f3c5c10eaa0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009ca0 tx=0x7f3c5402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 -- 172.21.15.16:0/3992084177 shutdown_connections 2024-09-12T03:04:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 --2- 172.21.15.16:0/3992084177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c106f20 0x7f3c5c10eaa0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 --2- 172.21.15.16:0/3992084177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 --2- 172.21.15.16:0/3992084177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 0x7f3c5c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 -- 172.21.15.16:0/3992084177 >> 172.21.15.16:0/3992084177 conn(0x7f3c5c0fd8b0 msgr2=0x7f3c5c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 -- 172.21.15.16:0/3992084177 shutdown_connections 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.194+0000 7f3c630f5640 1 -- 172.21.15.16:0/3992084177 wait complete. 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 Processor -- start 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 -- start start 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 0x7f3c5c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c106f20 0x7f3c5c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c111f80 con 0x7f3c5c1027a0 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c111e00 con 0x7f3c5c106f20 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c630f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c112100 con 0x7f3c5c101df0 2024-09-12T03:04:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52314/0 (socket says 172.21.15.16:52314) 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c5bfff640 1 -- 172.21.15.16:0/1031050958 learned_addr learned my addr 172.21.15.16:0/1031050958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.195+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 0x7f3c5c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c61692640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c106f20 0x7f3c5c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 -- 172.21.15.16:0/1031050958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 msgr2=0x7f3c5c198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 0x7f3c5c198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 -- 172.21.15.16:0/1031050958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c106f20 msgr2=0x7f3c5c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c106f20 0x7f3c5c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 -- 172.21.15.16:0/1031050958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c19d330 con 0x7f3c5c1027a0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d950 tx=0x7f3c4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c59ffb640 1 -- 172.21.15.16:0/1031050958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48017070 con 0x7f3c5c1027a0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c59ffb640 1 -- 172.21.15.16:0/1031050958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c48009c30 con 0x7f3c5c1027a0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c19d620 con 0x7f3c5c1027a0 2024-09-12T03:04:32.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c59ffb640 1 -- 172.21.15.16:0/1031050958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4800f6b0 con 0x7f3c5c1027a0 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.196+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c1a4f00 con 0x7f3c5c1027a0 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.198+0000 7f3c59ffb640 1 -- 172.21.15.16:0/1031050958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3c480109d0 con 0x7f3c5c1027a0 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 Processor -- start 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 -- start start 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd13440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd20630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bc78620 con 0x55cd4bd13980 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bc1fe50 con 0x55cd4bd13060 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c654ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bc745e0 con 0x55cd4bd1e240 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c59ffb640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 0x7f3c4c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52330/0 (socket says 172.21.15.16:52330) 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.199+0000 7f3c60e91640 1 -- 172.21.15.16:0/1386195731 learned_addr learned my addr 172.21.15.16:0/1386195731 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c61692640 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd13440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd20630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c59ffb640 1 -- 172.21.15.16:0/1031050958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3c480cebc0 con 0x7f3c5c1027a0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c61692640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 0x7f3c4c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 -- 172.21.15.16:0/1386195731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 msgr2=0x55cd4bd20630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd20630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 -- 172.21.15.16:0/1386195731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 msgr2=0x55cd4bd13440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd13440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 -- 172.21.15.16:0/1386195731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4bc10d40 con 0x55cd4bd13980 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009d20 tx=0x7f3c50011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4816667eccedac9b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.200+0000 7f3c61692640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 0x7f3c4c0a5040 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3c54004af0 tx=0x7f3c5402a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.204+0000 7f3c46ffd640 1 -- 172.21.15.16:0/1386195731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5001e070 con 0x55cd4bd13980 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.204+0000 7f3c46ffd640 1 -- 172.21.15.16:0/1386195731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c50019b10 con 0x55cd4bd13980 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.204+0000 7f3c46ffd640 1 -- 172.21.15.16:0/1386195731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000eb30 con 0x55cd4bd13980 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- 172.21.15.16:0/1386195731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 msgr2=0x55cd4bd1dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009d20 tx=0x7f3c50011810 comp rx=0 tx=0).stop 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- 172.21.15.16:0/1386195731 shutdown_connections 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd20630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1dd00 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1386195731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd13440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- 172.21.15.16:0/1386195731 >> 172.21.15.16:0/1386195731 conn(0x55cd4bd0f5f0 msgr2=0x55cd4bd0f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- 172.21.15.16:0/1386195731 shutdown_connections 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- 172.21.15.16:0/1386195731 wait complete. 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 -- start start 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.205+0000 7f3c654ce740 1 Processor -- start 2024-09-12T03:04:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13060 0x55cd4bd1b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 0x55cd4bd1bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bd20e10 con 0x55cd4bd13980 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bc00f10 con 0x55cd4bd1e240 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cd4bd20f90 con 0x55cd4bd13060 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c654ce740 10 client.0 _ll_register_callbacks cb 0x55cd4bc24db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52336/0 (socket says 172.21.15.16:52336) 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 -- 172.21.15.16:0/2415415358 learned_addr learned my addr 172.21.15.16:0/2415415358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c60e91640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13060 0x55cd4bd1b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c61692640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 0x55cd4bd1bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 -- 172.21.15.16:0/2415415358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13060 msgr2=0x55cd4bd1b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13060 0x55cd4bd1b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 -- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 msgr2=0x55cd4bd1bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 0x55cd4bd1bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c5bfff640 1 -- 172.21.15.16:0/2415415358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4b987580 con 0x55cd4bd13980 2024-09-12T03:04:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.206+0000 7f3c61692640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 0x55cd4bd1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.207+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009610 tx=0x7f3c48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378b17643b2f352 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.207+0000 7f3c3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48014070 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3c480ab2a0 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c50019b10 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3c48020040 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c654ce740 10 client.19350 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.209+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cd4ba51960 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c50019b10 con 0x55cd4bd13980 2024-09-12T03:04:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3c480ccb90 con 0x55cd4bd13980 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 10 client.19350 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 10 client.19350 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 20 client.19350 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd2dcd0 0x55cd4bd30110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 4 client.19350 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.210+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd4bc85b90 con 0x55cd4bd2dcd0 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.211+0000 7f3c60e91640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd2dcd0 0x55cd4bd30110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.211+0000 7f3c60e91640 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd2dcd0 0x55cd4bd30110 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2995441266fbd2cd server_cookie=126bfaab947c8c9f in_seq=0 out_seq=0 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.211+0000 7f3c3bfff640 10 client.19350 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c3bfff640 1 -- 172.21.15.16:0/2415415358 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cd4bc85b90 con 0x55cd4bd2dcd0 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c3bfff640 10 client.19350 handle_command_reply: tid=0 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 client.19350 shutdown 2024-09-12T03:04:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 msgr2=0x55cd4bd1b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009610 tx=0x7f3c48005f90 comp rx=0 tx=0).stop 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 shutdown_connections 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd2dcd0 0x55cd4bd30110 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd1e240 0x55cd4bd1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13980 0x55cd4bd1b5f0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 --2- 172.21.15.16:0/2415415358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13060 0x55cd4bd1b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 >> 172.21.15.16:0/2415415358 conn(0x55cd4bd0f5f0 msgr2=0x55cd4bd15320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c3bfff640 0 client.19350 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 shutdown_connections 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 1 -- 172.21.15.16:0/2415415358 wait complete. 2024-09-12T03:04:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.212+0000 7f3c654ce740 20 client.19350 trim_cache size 0 max 16384 2024-09-12T03:04:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 Processor -- start 2024-09-12T03:04:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 -- start start 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:04:32.231 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.231 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:32 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 0x55cd4bd35ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ad4c0 con 0x55cd4bd1e240 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1af000 con 0x55cd4bd13060 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c654ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1adb90 con 0x55cd4bd13980 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c61692640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 0x55cd4bd35ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38198/0 (socket says 172.21.15.16:38198) 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:53988/0 (socket says 172.21.15.16:53988) 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.230+0000 7f3c60e91640 1 -- 172.21.15.16:0/1083970264 learned_addr learned my addr 172.21.15.16:0/1083970264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 -- 172.21.15.16:0/1083970264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 msgr2=0x55cd4bd35990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 -- 172.21.15.16:0/1083970264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 msgr2=0x55cd4bd35ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 0x55cd4bd35ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 -- 172.21.15.16:0/1083970264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4bd0f420 con 0x55cd4bd13060 2024-09-12T03:04:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c61692640 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 0x55cd4bd35ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c60e91640 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009230 tx=0x7f3c5000e770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c25ab2de97e7102 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c467fc640 1 -- 172.21.15.16:0/1083970264 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50029070 con 0x55cd4bd13060 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c467fc640 1 -- 172.21.15.16:0/1083970264 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5001f040 con 0x55cd4bd13060 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.231+0000 7f3c467fc640 1 -- 172.21.15.16:0/1083970264 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5001e040 con 0x55cd4bd13060 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- 172.21.15.16:0/1083970264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 msgr2=0x55cd4bd0e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009230 tx=0x7f3c5000e770 comp rx=0 tx=0).stop 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- 172.21.15.16:0/1083970264 shutdown_connections 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd1e240 0x55cd4bd35ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd13980 0x55cd4bd35990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 --2- 172.21.15.16:0/1083970264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13060 0x55cd4bd0e640 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- 172.21.15.16:0/1083970264 >> 172.21.15.16:0/1083970264 conn(0x55cd4bd0f5f0 msgr2=0x55cd4bd0df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- 172.21.15.16:0/1083970264 shutdown_connections 2024-09-12T03:04:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- 172.21.15.16:0/1083970264 wait complete. 2024-09-12T03:04:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.232+0000 7f3c654ce740 1 -- start start 2024-09-12T03:04:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 Processor -- start 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 0x55cd4bd267f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd226f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ae930 con 0x55cd4bd13060 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ae260 con 0x55cd4bd13980 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ab4a0 con 0x55cd4bd1e240 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c654ce740 10 client.0 _ll_register_callbacks cb 0x7f3c4809afb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52346/0 (socket says 172.21.15.16:52346) 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 -- 172.21.15.16:0/50470185 learned_addr learned my addr 172.21.15.16:0/50470185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 0x55cd4bd267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c61692640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd226f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 -- 172.21.15.16:0/50470185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 msgr2=0x55cd4bd226f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd226f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 -- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 msgr2=0x55cd4bd267f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 0x55cd4bd267f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.233+0000 7f3c60e91640 1 -- 172.21.15.16:0/50470185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4bd22f20 con 0x55cd4bd13060 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 0x55cd4bd267f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c60e91640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f3c5001c040 tx=0x7f3c50005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeaccf306bfd7c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5003c060 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3c5000e9e0 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c5001f040 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3c50029070 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c654ce740 10 client.19356 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cd4bd99410 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.234+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5001f040 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3c50003b60 con 0x55cd4bd13060 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 10 client.19356 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 10 client.19356 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 20 client.19356 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd998d0 0x55cd4bd9bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 4 client.19356 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f0607e58", "release": true, "await": true}] 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f0607e58", "release": true, "await": true}) -- 0x55cd4bc800a0 con 0x55cd4bd998d0 2024-09-12T03:04:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.235+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd998d0 0x55cd4bd9bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.236+0000 7f3c3bfff640 10 client.19356 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.236+0000 7f3c5bfff640 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd998d0 0x55cd4bd9bcc0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=320f3aaa436ceb6f server_cookie=a8d277087fc27c51 in_seq=0 out_seq=0 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c3bfff640 1 -- 172.21.15.16:0/50470185 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cd4bc800a0 con 0x55cd4bd998d0 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c3bfff640 10 client.19356 handle_command_reply: tid=0 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 client.19356 shutdown 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 msgr2=0x55cd4bd262b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f3c5001c040 tx=0x7f3c50005d30 comp rx=0 tx=0).stop 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 shutdown_connections 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd4bd998d0 0x55cd4bd9bcc0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd4bd1e240 0x55cd4bd226f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c3bfff640 0 client.19356 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:32.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd4bd13980 0x55cd4bd267f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 --2- 172.21.15.16:0/50470185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd4bd13060 0x55cd4bd262b0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.287+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 >> 172.21.15.16:0/50470185 conn(0x55cd4bd0f5f0 msgr2=0x55cd4bd14b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 shutdown_connections 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c654ce740 1 -- 172.21.15.16:0/50470185 wait complete. 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c654ce740 20 client.19356 trim_cache size 0 max 16384 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 60, 2024-09-12T03:04:32.289 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "f0607e58": { 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 60, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.2, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:04:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:32.291 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 msgr2=0x7f3c4c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 0x7f3c4c0a5040 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3c54004af0 tx=0x7f3c5402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 msgr2=0x7f3c5c1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.288+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d950 tx=0x7f3c4800de10 comp rx=0 tx=0).stop 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 shutdown_connections 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3c5c106f20 0x7f3c5c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3c4c0a2b80 0x7f3c4c0a5040 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c1027a0 0x7f3c5c1988c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 --2- 172.21.15.16:0/1031050958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c5c101df0 0x7f3c5c198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 >> 172.21.15.16:0/1031050958 conn(0x7f3c5c0fd8b0 msgr2=0x7f3c5c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 shutdown_connections 2024-09-12T03:04:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:32.289+0000 7f3c630f5640 1 -- 172.21.15.16:0/1031050958 wait complete. 2024-09-12T03:04:32.294 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:04:32.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:32 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:32 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:04:33.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85647 25.83 MB/sec execute 94 sec latency 25861.638 ms 2024-09-12T03:04:33.201 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f0607e58', total seconds elapsed: 29.9 2024-09-12T03:04:33.201 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.25 2024-09-12T03:04:33.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:04:33.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:33 smithi016 ceph-mon[30925]: pgmap v836: 97 pgs: 97 active+clean; 607 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 6.4 MiB/s wr, 239 op/s 2024-09-12T03:04:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 -- 172.21.15.16:0/2159147290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa41082a0 msgr2=0x7f5fa410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2159147290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa41082a0 0x7f5fa410ec40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009cd0 tx=0x7f5f9c02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 -- 172.21.15.16:0/2159147290 shutdown_connections 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2159147290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa41082a0 0x7f5fa410ec40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2159147290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2159147290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa4102d30 0x7f5fa4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 -- 172.21.15.16:0/2159147290 >> 172.21.15.16:0/2159147290 conn(0x7f5fa40fc7d0 msgr2=0x7f5fa40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 -- 172.21.15.16:0/2159147290 shutdown_connections 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.427+0000 7f5fabf76640 1 -- 172.21.15.16:0/2159147290 wait complete. 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 Processor -- start 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 -- start start 2024-09-12T03:04:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa41082a0 0x7f5fa419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41120a0 con 0x7f5fa4102d30 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4111f20 con 0x7f5fa41036e0 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fabf76640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4112220 con 0x7f5fa41082a0 2024-09-12T03:04:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52370/0 (socket says 172.21.15.16:52370) 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fa9d12640 1 -- 172.21.15.16:0/2426444201 learned_addr learned my addr 172.21.15.16:0/2426444201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5fa9511640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.428+0000 7f5faa513640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa41082a0 0x7f5fa419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 -- 172.21.15.16:0/2426444201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa41082a0 msgr2=0x7f5fa419cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa41082a0 0x7f5fa419cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 -- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 msgr2=0x7f5fa4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 -- 172.21.15.16:0/2426444201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa419d4c0 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fa9511640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4198a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40697d0 tx=0x7f5f9400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5f9affd640 1 -- 172.21.15.16:0/2426444201 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9401e070 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa419d7b0 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5f9affd640 1 -- 172.21.15.16:0/2426444201 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f940198f0 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5f9affd640 1 -- 172.21.15.16:0/2426444201 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94018730 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.429+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa41008c0 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.432+0000 7f5f9affd640 1 -- 172.21.15.16:0/2426444201 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5f94021070 con 0x7f5fa41036e0 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.432+0000 7f5fae34f740 1 Processor -- start 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 -- start start 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b1e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b2ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aae0a842e0 con 0x55aae0b1edc0 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aae0a2bce0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fae34f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aae0a802a0 con 0x55aae0b29680 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52376/0 (socket says 172.21.15.16:52376) 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 -- 172.21.15.16:0/35968320 learned_addr learned my addr 172.21.15.16:0/35968320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9511640 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b2ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5faa513640 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b1e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 -- 172.21.15.16:0/35968320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 msgr2=0x55aae0b2ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b2ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 -- 172.21.15.16:0/35968320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 msgr2=0x55aae0b1e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.433+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b1e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5fa9d12640 1 -- 172.21.15.16:0/35968320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aae0a10440 con 0x55aae0b1edc0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5f9affd640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 0x7f5f900a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0011650 tx=0x7f5fa0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572e75d479bb2cf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5faa513640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 0x7f5f900a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5f9affd640 1 -- 172.21.15.16:0/2426444201 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5f940d2770 con 0x7f5fa41036e0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5f8ffff640 1 -- 172.21.15.16:0/35968320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa001e070 con 0x55aae0b1edc0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5f8ffff640 1 -- 172.21.15.16:0/35968320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fa000fba0 con 0x55aae0b1edc0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.434+0000 7f5f8ffff640 1 -- 172.21.15.16:0/35968320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0019d40 con 0x55aae0b1edc0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- 172.21.15.16:0/35968320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 msgr2=0x55aae0b29140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0011650 tx=0x7f5fa0009f90 comp rx=0 tx=0).stop 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5faa513640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 0x7f5f900a5040 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c004ab0 tx=0x7f5f9c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- 172.21.15.16:0/35968320 shutdown_connections 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b2ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b29140 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 --2- 172.21.15.16:0/35968320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b1e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- 172.21.15.16:0/35968320 >> 172.21.15.16:0/35968320 conn(0x55aae0b1aa30 msgr2=0x55aae0b1ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- 172.21.15.16:0/35968320 shutdown_connections 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- 172.21.15.16:0/35968320 wait complete. 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.435+0000 7f5fae34f740 1 -- start start 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 Processor -- start 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b26830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aae0b2c250 con 0x55aae0b1edc0 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55aae0a3f050 con 0x55aae0b1e4a0 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aae0b2c3d0 con 0x55aae0b29680 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fae34f740 10 client.0 _ll_register_callbacks cb 0x55aae0a7f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fa9511640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fa9d12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38250/0 (socket says 172.21.15.16:38250) 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5faa513640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b26830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fa9d12640 1 -- 172.21.15.16:0/3338870626 learned_addr learned my addr 172.21.15.16:0/3338870626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.436+0000 7f5fa9511640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52386/0 (socket says 172.21.15.16:52386) 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 -- 172.21.15.16:0/3338870626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 msgr2=0x55aae0b26830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b26830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 -- 172.21.15.16:0/3338870626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 msgr2=0x55aae0b262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 -- 172.21.15.16:0/3338870626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aae07f53c0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9511640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000c00 tx=0x7f5fa000faf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c28d8476ab871a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5f8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa001e040 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa000fba0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5fa0009cc0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5fa0019a80 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fae34f740 10 client.28667 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.437+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55aae0b37c40 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fa000fba0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5fa002a030 con 0x55aae0b1e4a0 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 10 client.28667 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 10 client.28667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 20 client.28667 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0b39050 0x55aae0b3b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 4 client.28667 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aae0a91850 con 0x55aae0b39050 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5fa9511640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0b39050 0x55aae0b3b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.438+0000 7f5f8d7fa640 10 client.28667 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5fa9511640 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0b39050 0x55aae0b3b440 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9c2f72536d71133 server_cookie=a35045ad48de5fb7 in_seq=0 out_seq=0 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/3338870626 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aae0a91850 con 0x55aae0b39050 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5f8d7fa640 10 client.28667 handle_command_reply: tid=0 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5fae34f740 1 client.28667 shutdown 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 msgr2=0x55aae0b29000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.439+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000c00 tx=0x7f5fa000faf0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 shutdown_connections 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0b39050 0x55aae0b3b440 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b26830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b262b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5f8d7fa640 0 client.28667 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3338870626 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b29000 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 >> 172.21.15.16:0/3338870626 conn(0x55aae0b1aa30 msgr2=0x55aae0b20760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 shutdown_connections 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 1 -- 172.21.15.16:0/3338870626 wait complete. 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.440+0000 7f5fae34f740 20 client.28667 trim_cache size 0 max 16384 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.467+0000 7f5fae34f740 1 Processor -- start 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.467+0000 7f5fae34f740 1 -- start start 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b40c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41ab400 con 0x55aae0b1edc0 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41acf40 con 0x55aae0b1e4a0 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fae34f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41abad0 con 0x55aae0b29680 2024-09-12T03:04:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5faa513640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5faa513640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54028/0 (socket says 172.21.15.16:54028) 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5faa513640 1 -- 172.21.15.16:0/3323848945 learned_addr learned my addr 172.21.15.16:0/3323848945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.468+0000 7f5fa9511640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b40c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b19e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 -- 172.21.15.16:0/3323848945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 msgr2=0x55aae0b19e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b19e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 -- 172.21.15.16:0/3323848945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 msgr2=0x55aae0b40c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b40c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 -- 172.21.15.16:0/3323848945 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aae0b26ba0 con 0x55aae0b29680 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5fa9511640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b40c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5faa513640 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009cd0 tx=0x7f5f9c044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa85c96d40c1bc9c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.469+0000 7f5f8f7fe640 1 -- 172.21.15.16:0/3323848945 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c003ab0 con 0x55aae0b29680 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.470+0000 7f5f8f7fe640 1 -- 172.21.15.16:0/3323848945 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c056070 con 0x55aae0b29680 2024-09-12T03:04:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.470+0000 7f5f8f7fe640 1 -- 172.21.15.16:0/3323848945 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c030920 con 0x55aae0b29680 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- 172.21.15.16:0/3323848945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 msgr2=0x55aae0b41140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009cd0 tx=0x7f5f9c044000 comp rx=0 tx=0).stop 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- 172.21.15.16:0/3323848945 shutdown_connections 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b29680 0x55aae0b41140 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b1edc0 0x55aae0b40c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 --2- 172.21.15.16:0/3323848945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b19e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- 172.21.15.16:0/3323848945 >> 172.21.15.16:0/3323848945 conn(0x55aae0b1aa30 msgr2=0x55aae0b25270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- 172.21.15.16:0/3323848945 shutdown_connections 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- 172.21.15.16:0/3323848945 wait complete. 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.471+0000 7f5fae34f740 1 -- start start 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 Processor -- start 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b2e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b29680 0x55aae0baa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41ac870 con 0x55aae0b29680 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41ac1a0 con 0x55aae0b1e4a0 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41a93e0 con 0x55aae0b1edc0 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fae34f740 10 client.0 _ll_register_callbacks cb 0x7f5fa001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fa9511640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fa9511640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54036/0 (socket says 172.21.15.16:54036) 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5fa9511640 1 -- 172.21.15.16:0/1078044183 learned_addr learned my addr 172.21.15.16:0/1078044183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.472+0000 7f5faa513640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b29680 0x55aae0baa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 -- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 msgr2=0x55aae0b2e740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b2e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 -- 172.21.15.16:0/1078044183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b29680 msgr2=0x55aae0baa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b29680 0x55aae0baa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 -- 172.21.15.16:0/1078044183 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aae0baa920 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fa9511640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400bb90 tx=0x7f5f94012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=778999d5f7807b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5f8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f940bf0b0 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5f94019b20 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.473+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f9c056070 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.474+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5f9409f090 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.474+0000 7f5fae34f740 10 client.28579 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.474+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55aae0babae0 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.474+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c056070 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5f9409ade0 con 0x55aae0b1edc0 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 10 client.28579 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 10 client.28579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 20 client.28579 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0bac170 0x55aae0bae590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 4 client.28579 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.475+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aae0a8bd60 con 0x55aae0bac170 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.476+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0bac170 0x55aae0bae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.477+0000 7f5fa9d12640 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0bac170 0x55aae0bae590 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d9f9230f7f11e44 server_cookie=4acb8c98c7ca00e6 in_seq=0 out_seq=0 2024-09-12T03:04:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.477+0000 7f5f8d7fa640 10 client.28579 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.478+0000 7f5f8d7fa640 1 -- 172.21.15.16:0/1078044183 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55aae0a8bd60 con 0x55aae0bac170 2024-09-12T03:04:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.478+0000 7f5f8d7fa640 10 client.28579 handle_command_reply: tid=0 2024-09-12T03:04:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.478+0000 7f5fae34f740 1 client.28579 shutdown 2024-09-12T03:04:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.479+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 msgr2=0x55aae0b2ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.479+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400bb90 tx=0x7f5f94012040 comp rx=0 tx=0).stop 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 shutdown_connections 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55aae0bac170 0x55aae0bae590 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aae0b29680 0x55aae0baa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5f8d7fa640 0 client.28579 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aae0b1edc0 0x55aae0b2ec80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 --2- 172.21.15.16:0/1078044183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55aae0b1e4a0 0x55aae0b2e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.481+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 >> 172.21.15.16:0/1078044183 conn(0x55aae0b1aa30 msgr2=0x55aae0b27b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.482+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 shutdown_connections 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.482+0000 7f5fae34f740 1 -- 172.21.15.16:0/1078044183 wait complete. 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.482+0000 7f5fae34f740 20 client.28579 trim_cache size 0 max 16384 2024-09-12T03:04:33.495 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "54aace7f-e74e-427b-9156-e57958d5a5dd": { 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "54aace7f-e74e-427b-9156-e57958d5a5dd", 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:33.496 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 msgr2=0x7f5f900a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 0x7f5f900a5040 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c004ab0 tx=0x7f5f9c004a40 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 msgr2=0x7f5fa4198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4198a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40697d0 tx=0x7f5f9400b690 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 shutdown_connections 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5fa41082a0 0x7f5fa419cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa41036e0 0x7f5fa4198a50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5f900a2b80 0x7f5f900a5040 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 --2- 172.21.15.16:0/2426444201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fa4102d30 0x7f5fa4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.484+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 >> 172.21.15.16:0/2426444201 conn(0x7f5fa40fc7d0 msgr2=0x7f5fa4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.485+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 shutdown_connections 2024-09-12T03:04:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:33.485+0000 7f5fabf76640 1 -- 172.21.15.16:0/2426444201 wait complete. 2024-09-12T03:04:33.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:33 smithi033 ceph-mon[35092]: pgmap v836: 97 pgs: 97 active+clean; 607 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 6.4 MiB/s wr, 239 op/s 2024-09-12T03:04:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:33 smithi184 ceph-mon[35321]: pgmap v836: 97 pgs: 97 active+clean; 607 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 6.4 MiB/s wr, 239 op/s 2024-09-12T03:04:34.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86160 25.85 MB/sec execute 95 sec latency 109.743 ms 2024-09-12T03:04:34.443 INFO:tasks.cephfs.filesystem:scrub status for tag:54aace7f-e74e-427b-9156-e57958d5a5dd - {'path': '/', 'tag': '54aace7f-e74e-427b-9156-e57958d5a5dd', 'options': 'recursive,force'} 2024-09-12T03:04:34.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:35.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86704 25.87 MB/sec execute 96 sec latency 155.826 ms 2024-09-12T03:04:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.063+0000 7fb9d33a8640 1 -- 172.21.15.16:0/1779091482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 msgr2=0x7fb9cc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.063+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/1779091482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc101200 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009ca0 tx=0x7fb9c002a940 comp rx=0 tx=0).stop 2024-09-12T03:04:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 -- 172.21.15.16:0/1779091482 shutdown_connections 2024-09-12T03:04:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/1779091482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102ef0 0x7fb9cc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/1779091482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9cc102570 0x7fb9cc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/1779091482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc101200 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 -- 172.21.15.16:0/1779091482 >> 172.21.15.16:0/1779091482 conn(0x7fb9cc0fc7d0 msgr2=0x7fb9cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 -- 172.21.15.16:0/1779091482 shutdown_connections 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.064+0000 7fb9d33a8640 1 -- 172.21.15.16:0/1779091482 wait complete. 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.065+0000 7fb9d33a8640 1 Processor -- start 2024-09-12T03:04:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.065+0000 7fb9d33a8640 1 -- start start 2024-09-12T03:04:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9cc102ef0 0x7fb9cc19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10fec0 con 0x7fb9cc100e20 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10fd40 con 0x7fb9cc102ef0 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d33a8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc110040 con 0x7fb9cc102570 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d0943640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d0943640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54062/0 (socket says 172.21.15.16:54062) 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.066+0000 7fb9d0943640 1 -- 172.21.15.16:0/4112422458 learned_addr learned my addr 172.21.15.16:0/4112422458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d1144640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 -- 172.21.15.16:0/4112422458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9cc102ef0 msgr2=0x7fb9cc19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9cc102ef0 0x7fb9cc19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 -- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 msgr2=0x7fb9cc1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc1962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 -- 172.21.15.16:0/4112422458 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc19b290 con 0x7fb9cc102570 2024-09-12T03:04:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.067+0000 7fb9d0943640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800d950 tx=0x7fb9b800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.068+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8017070 con 0x7fb9cc102570 2024-09-12T03:04:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.068+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b8009c30 con 0x7fb9cc102570 2024-09-12T03:04:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.069+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b800f6b0 con 0x7fb9cc102570 2024-09-12T03:04:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.069+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc19b580 con 0x7fb9cc102570 2024-09-12T03:04:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.069+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc1a2e60 con 0x7fb9cc102570 2024-09-12T03:04:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.071+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc101200 con 0x7fb9cc102570 2024-09-12T03:04:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.072+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb9b80109d0 con 0x7fb9cc102570 2024-09-12T03:04:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.073+0000 7fb9ca7fc640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 0x7fb9bc0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.073+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb9b80ce540 con 0x7fb9cc102570 2024-09-12T03:04:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.073+0000 7fb9d1144640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 0x7fb9bc0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:35.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.074+0000 7fb9d1144640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 0x7fb9bc0a73e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0000f80 tx=0x7fb9c0009800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.080+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb9b8009de0 con 0x7fb9cc102570 2024-09-12T03:04:35.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.387+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9cc109090 con 0x7fb9cc102570 2024-09-12T03:04:35.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.388+0000 7fb9ca7fc640 1 -- 172.21.15.16:0/4112422458 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fb9b809b530 con 0x7fb9cc102570 2024-09-12T03:04:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:04:35.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:04:35.388 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:04:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 msgr2=0x7fb9bc0a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 0x7fb9bc0a73e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0000f80 tx=0x7fb9c0009800 comp rx=0 tx=0).stop 2024-09-12T03:04:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 msgr2=0x7fb9cc196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800d950 tx=0x7fb9b800de10 comp rx=0 tx=0).stop 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 shutdown_connections 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9cc102ef0 0x7fb9cc19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9cc102570 0x7fb9cc196820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb9bc0a4f20 0x7fb9bc0a73e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 --2- 172.21.15.16:0/4112422458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9cc100e20 0x7fb9cc1962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 >> 172.21.15.16:0/4112422458 conn(0x7fb9cc0fc7d0 msgr2=0x7fb9cc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 shutdown_connections 2024-09-12T03:04:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:35.390+0000 7fb9d33a8640 1 -- 172.21.15.16:0/4112422458 wait complete. 2024-09-12T03:04:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:35 smithi016 ceph-mon[30925]: pgmap v837: 97 pgs: 97 active+clean; 605 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 6.1 MiB/s wr, 231 op/s 2024-09-12T03:04:35.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:35 smithi033 ceph-mon[35092]: pgmap v837: 97 pgs: 97 active+clean; 605 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 6.1 MiB/s wr, 231 op/s 2024-09-12T03:04:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:35 smithi184 ceph-mon[35321]: pgmap v837: 97 pgs: 97 active+clean; 605 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 6.1 MiB/s wr, 231 op/s 2024-09-12T03:04:36.014 INFO:tasks.workunit.client.0.smithi016.stdout: 10 87589 26.05 MB/sec execute 97 sec latency 51.673 ms 2024-09-12T03:04:36.207 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 144.25 --await-for 45.0' 2024-09-12T03:04:36.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 144.25 --await-for 45.0 2024-09-12T03:04:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:36.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:36 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4112422458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:36.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:36 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4112422458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:36.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:36 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4112422458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:37.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88258 26.25 MB/sec execute 98 sec latency 52.820 ms 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 -- 172.21.15.16:0/769111713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 msgr2=0x7f50141029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 --2- 172.21.15.16:0/769111713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f50141029b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f5004009900 tx=0x7f500402a480 comp rx=0 tx=0).stop 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 -- 172.21.15.16:0/769111713 shutdown_connections 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 --2- 172.21.15.16:0/769111713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 0x7f501410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 --2- 172.21.15.16:0/769111713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f50141029b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 --2- 172.21.15.16:0/769111713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 0x7f5014101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.134+0000 7f501bc04640 1 -- 172.21.15.16:0/769111713 >> 172.21.15.16:0/769111713 conn(0x7f50140fc7d0 msgr2=0x7f50140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.135+0000 7f501bc04640 1 -- 172.21.15.16:0/769111713 shutdown_connections 2024-09-12T03:04:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.135+0000 7f501bc04640 1 -- 172.21.15.16:0/769111713 wait complete. 2024-09-12T03:04:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.136+0000 7f501bc04640 1 Processor -- start 2024-09-12T03:04:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 -- start start 2024-09-12T03:04:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 0x7f501419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 0x7f501419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5014110fd0 con 0x7f5014102570 2024-09-12T03:04:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43680/0 (socket says 172.21.15.16:43680) 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501919f640 1 -- 172.21.15.16:0/2992905843 learned_addr learned my addr 172.21.15.16:0/2992905843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501a1a1640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 0x7f501419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5014110e50 con 0x7f5014102ef0 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.137+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5014111150 con 0x7f5014100e20 2024-09-12T03:04:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f50199a0640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 0x7f501419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 -- 172.21.15.16:0/2992905843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 msgr2=0x7f501419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 0x7f501419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 -- 172.21.15.16:0/2992905843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 msgr2=0x7f501419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 0x7f501419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 -- 172.21.15.16:0/2992905843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501419f6d0 con 0x7f5014102570 2024-09-12T03:04:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.138+0000 7f501919f640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f500402aa70 tx=0x7f5004003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.139+0000 7f500affd640 1 -- 172.21.15.16:0/2992905843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500403f070 con 0x7f5014102570 2024-09-12T03:04:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.139+0000 7f500affd640 1 -- 172.21.15.16:0/2992905843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5004031c30 con 0x7f5014102570 2024-09-12T03:04:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.139+0000 7f500affd640 1 -- 172.21.15.16:0/2992905843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004035030 con 0x7f5014102570 2024-09-12T03:04:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.140+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501419f960 con 0x7f5014102570 2024-09-12T03:04:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.140+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f501419fd00 con 0x7f5014102570 2024-09-12T03:04:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.144+0000 7f500affd640 1 -- 172.21.15.16:0/2992905843 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5004042070 con 0x7f5014102570 2024-09-12T03:04:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.144+0000 7f501dfdd740 1 Processor -- start 2024-09-12T03:04:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.144+0000 7f501dfdd740 1 -- start start 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 0x55d7f5c71440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71980 0x55d7f5c7bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5bcd160 con 0x55d7f5c71980 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5b74720 con 0x55d7f5c71060 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501dfdd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5bc9120 con 0x55d7f5c7c240 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501919f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501919f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60978/0 (socket says 172.21.15.16:60978) 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501919f640 1 -- 172.21.15.16:0/1866999561 learned_addr learned my addr 172.21.15.16:0/1866999561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.145+0000 7f501a1a1640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 0x55d7f5c71440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f500affd640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 0x7f50000a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f50199a0640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71980 0x55d7f5c7bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 -- 172.21.15.16:0/1866999561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 msgr2=0x55d7f5c7e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 -- 172.21.15.16:0/1866999561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71980 msgr2=0x55d7f5c7bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71980 0x55d7f5c7bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f500affd640 1 -- 172.21.15.16:0/2992905843 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50040ef990 con 0x7f5014102570 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 -- 172.21.15.16:0/1866999561 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f593eba0 con 0x55d7f5c71060 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 0x7f50000a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501919f640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.146+0000 7f501a1a1640 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 0x55d7f5c71440 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f500c011490 tx=0x7f500c011950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1602384f7fa1694 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.147+0000 7f501a1a1640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 0x7f50000a5210 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f500c023000 tx=0x7f500c024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.147+0000 7f4ffffff640 1 -- 172.21.15.16:0/1866999561 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c017070 con 0x55d7f5c71060 2024-09-12T03:04:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.147+0000 7f4ffffff640 1 -- 172.21.15.16:0/1866999561 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f500c00fc20 con 0x55d7f5c71060 2024-09-12T03:04:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.147+0000 7f4ffffff640 1 -- 172.21.15.16:0/1866999561 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c01ccb0 con 0x55d7f5c71060 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- 172.21.15.16:0/1866999561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 msgr2=0x55d7f5c71440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 0x55d7f5c71440 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f500c011490 tx=0x7f500c011950 comp rx=0 tx=0).stop 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- 172.21.15.16:0/1866999561 shutdown_connections 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71980 0x55d7f5c7bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1866999561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71060 0x55d7f5c71440 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- 172.21.15.16:0/1866999561 >> 172.21.15.16:0/1866999561 conn(0x55d7f5c6d5f0 msgr2=0x55d7f5c6d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- 172.21.15.16:0/1866999561 shutdown_connections 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- 172.21.15.16:0/1866999561 wait complete. 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 -- start start 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.148+0000 7f501dfdd740 1 Processor -- start 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71980 0x55d7f5c79700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c7c240 0x55d7f5c79c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5c7ee10 con 0x55d7f5c71060 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5b55900 con 0x55d7f5c7c240 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501dfdd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d7f5c7ef90 con 0x55d7f5c71980 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f50199a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f50199a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43704/0 (socket says 172.21.15.16:43704) 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f50199a0640 1 -- 172.21.15.16:0/1053441164 learned_addr learned my addr 172.21.15.16:0/1053441164 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501a1a1640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c7c240 0x55d7f5c79c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.149+0000 7f501919f640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71980 0x55d7f5c79700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f501dfdd740 10 client.0 _ll_register_callbacks cb 0x55d7f5b53780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 -- 172.21.15.16:0/1053441164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71980 msgr2=0x55d7f5c79700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71980 0x55d7f5c79700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 -- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c7c240 msgr2=0x55d7f5c79c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c7c240 0x55d7f5c79c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 -- 172.21.15.16:0/1053441164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f5b95c50 con 0x55d7f5c71060 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.150+0000 7f50199a0640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f5010013710 tx=0x7f5010013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4fb68ecf76606a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f4ffd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010020060 con 0x55d7f5c71060 2024-09-12T03:04:37.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f500c00fc20 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5010011dd0 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f501000f450 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f501dfdd740 10 client.19392 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.151+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d7f5c8ab70 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f500c00fc20 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f501001a030 con 0x55d7f5c71060 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 10 client.19392 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 10 client.19392 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 20 client.19392 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5c8e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 4 client.19392 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7f5bda890 con 0x55d7f5c8c2c0 2024-09-12T03:04:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.152+0000 7f501919f640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5c8e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.153+0000 7f501919f640 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5c8e730 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be1c6a4b7eb4ff4 server_cookie=13f77a6d692fc249 in_seq=0 out_seq=0 2024-09-12T03:04:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.153+0000 7f4ffd7fa640 10 client.19392 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.154+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/1053441164 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d7f5bda890 con 0x55d7f5c8c2c0 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.154+0000 7f4ffd7fa640 10 client.19392 handle_command_reply: tid=0 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.154+0000 7f501dfdd740 1 client.19392 shutdown 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 msgr2=0x55d7f5c791c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f5010013710 tx=0x7f5010013b10 comp rx=0 tx=0).stop 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 shutdown_connections 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5c8e730 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c7c240 0x55d7f5c79c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71980 0x55d7f5c79700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f4ffd7fa640 0 client.19392 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 --2- 172.21.15.16:0/1053441164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c791c0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.155+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 >> 172.21.15.16:0/1053441164 conn(0x55d7f5c6d5f0 msgr2=0x55d7f5c73320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.156+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 shutdown_connections 2024-09-12T03:04:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.156+0000 7f501dfdd740 1 -- 172.21.15.16:0/1053441164 wait complete. 2024-09-12T03:04:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.156+0000 7f501dfdd740 20 client.19392 trim_cache size 0 max 16384 2024-09-12T03:04:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 Processor -- start 2024-09-12T03:04:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 -- start start 2024-09-12T03:04:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c94580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c76420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50141ab380 con 0x55d7f5c71060 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50141acec0 con 0x55d7f5c71980 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501dfdd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50141aba50 con 0x55d7f5c7c240 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f50199a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f50199a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43718/0 (socket says 172.21.15.16:43718) 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f50199a0640 1 -- 172.21.15.16:0/3969609747 learned_addr learned my addr 172.21.15.16:0/3969609747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501a1a1640 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c76420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.182+0000 7f501919f640 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c94580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 -- 172.21.15.16:0/3969609747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 msgr2=0x55d7f5c76420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c76420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 -- 172.21.15.16:0/3969609747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 msgr2=0x55d7f5c94580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c94580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 -- 172.21.15.16:0/3969609747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f5c74010 con 0x55d7f5c71060 2024-09-12T03:04:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f50199a0640 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5010005d80 tx=0x7f5010009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9594c9fa632cf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f4fff7fe640 1 -- 172.21.15.16:0/3969609747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010011dd0 con 0x55d7f5c71060 2024-09-12T03:04:37.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f4fff7fe640 1 -- 172.21.15.16:0/3969609747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010027450 con 0x55d7f5c71060 2024-09-12T03:04:37.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.183+0000 7f4fff7fe640 1 -- 172.21.15.16:0/3969609747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010003850 con 0x55d7f5c71060 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- 172.21.15.16:0/3969609747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 msgr2=0x55d7f5c94040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5010005d80 tx=0x7f5010009780 comp rx=0 tx=0).stop 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- 172.21.15.16:0/3969609747 shutdown_connections 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c7c240 0x55d7f5c76420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c94580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3969609747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c71060 0x55d7f5c94040 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- 172.21.15.16:0/3969609747 >> 172.21.15.16:0/3969609747 conn(0x55d7f5c6d5f0 msgr2=0x55d7f5c77a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- 172.21.15.16:0/3969609747 shutdown_connections 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- 172.21.15.16:0/3969609747 wait complete. 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.184+0000 7f501dfdd740 1 -- start start 2024-09-12T03:04:37.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 Processor -- start 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c807b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c7c240 0x55d7f5cf6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50141ac7f0 con 0x55d7f5c7c240 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50141ac120 con 0x55d7f5c71980 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50141a9360 con 0x55d7f5c71060 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501dfdd740 10 client.0 _ll_register_callbacks cb 0x7f50100096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:04:37.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:32782/0 (socket says 172.21.15.16:32782) 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 -- 172.21.15.16:0/3395691508 learned_addr learned my addr 172.21.15.16:0/3395691508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501919f640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c807b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501a1a1640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c7c240 0x55d7f5cf6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 -- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 msgr2=0x55d7f5c807b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c807b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 -- 172.21.15.16:0/3395691508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c7c240 msgr2=0x55d7f5cf6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c7c240 0x55d7f5cf6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f50199a0640 1 -- 172.21.15.16:0/3395691508 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f5c81040 con 0x55d7f5c71060 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.185+0000 7f501919f640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c807b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f50199a0640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5010000c00 tx=0x7f5010027ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1342636a0a10500c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f4ffd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010037020 con 0x55d7f5c71060 2024-09-12T03:04:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5010011dd0 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5010027450 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5010020070 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f501dfdd740 10 client.28603 fetch_fsmap learned FSMap version 39 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.186+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d7f5cf7600 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010027450 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f501001a040 con 0x55d7f5c71060 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 10 client.28603 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 10 client.28603 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 20 client.28603 populate_metadata read hostname 'smithi016' 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5cf9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 4 client.28603 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.25, "await_for": 45.0}] 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.25, "await_for": 45.0}) -- 0x55d7f5bd4be0 con 0x55d7f5c8c2c0 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501919f640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5cf9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:37.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f501919f640 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5cf9c80 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1692e669b2bb42 server_cookie=46c0f1d5af877763 in_seq=0 out_seq=0 2024-09-12T03:04:37.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.187+0000 7f4ffd7fa640 10 client.28603 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:37.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:37 smithi016 ceph-mon[30925]: pgmap v838: 97 pgs: 97 active+clean; 601 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 294 op/s 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f4ffd7fa640 1 -- 172.21.15.16:0/3395691508 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d7f5bd4be0 con 0x55d7f5c8c2c0 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f4ffd7fa640 10 client.28603 handle_command_reply: tid=0 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 client.28603 shutdown 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 msgr2=0x55d7f5c80270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5010000c00 tx=0x7f5010027ef0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 shutdown_connections 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55d7f5c8c2c0 0x55d7f5cf9c80 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d7f5c7c240 0x55d7f5cf6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d7f5c71980 0x55d7f5c807b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 --2- 172.21.15.16:0/3395691508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d7f5c71060 0x55d7f5c80270 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 >> 172.21.15.16:0/3395691508 conn(0x55d7f5c6d5f0 msgr2=0x55d7f5c77e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f4ffd7fa640 0 client.28603 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 shutdown_connections 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.723+0000 7f501dfdd740 1 -- 172.21.15.16:0/3395691508 wait complete. 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501dfdd740 20 client.28603 trim_cache size 0 max 16384 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 62, 2024-09-12T03:04:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "28c3f551": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 62, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.2, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:04:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:04:37.727 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 msgr2=0x7f50000a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 0x7f50000a5210 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f500c023000 tx=0x7f500c024000 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 msgr2=0x7f501419ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f500402aa70 tx=0x7f5004003c10 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 shutdown_connections 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5014102ef0 0x7f501419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50000a2d50 0x7f50000a5210 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5014102570 0x7f501419ac60 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 --2- 172.21.15.16:0/2992905843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014100e20 0x7f501419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 >> 172.21.15.16:0/2992905843 conn(0x7f50140fc7d0 msgr2=0x7f501406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 shutdown_connections 2024-09-12T03:04:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:37.724+0000 7f501bc04640 1 -- 172.21.15.16:0/2992905843 wait complete. 2024-09-12T03:04:37.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:37 smithi033 ceph-mon[35092]: pgmap v838: 97 pgs: 97 active+clean; 601 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 294 op/s 2024-09-12T03:04:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:37 smithi184 ceph-mon[35321]: pgmap v838: 97 pgs: 97 active+clean; 601 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 294 op/s 2024-09-12T03:04:38.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 26.08 MB/sec execute 99 sec latency 827.310 ms 2024-09-12T03:04:38.501 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '28c3f551', quiesce took 5.3 seconds. Will release after: 18.95 2024-09-12T03:04:39.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 25.82 MB/sec execute 100 sec latency 1827.343 ms 2024-09-12T03:04:39.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:39 smithi033 ceph-mon[35092]: pgmap v839: 97 pgs: 97 active+clean; 605 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.2 MiB/s wr, 403 op/s 2024-09-12T03:04:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:39 smithi184 ceph-mon[35321]: pgmap v839: 97 pgs: 97 active+clean; 605 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.2 MiB/s wr, 403 op/s 2024-09-12T03:04:39.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:39 smithi016 ceph-mon[30925]: pgmap v839: 97 pgs: 97 active+clean; 605 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.2 MiB/s wr, 403 op/s 2024-09-12T03:04:40.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 25.56 MB/sec execute 101 sec latency 2827.447 ms 2024-09-12T03:04:41.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 25.31 MB/sec execute 102 sec latency 3827.545 ms 2024-09-12T03:04:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:41 smithi016 ceph-mon[30925]: pgmap v840: 97 pgs: 97 active+clean; 600 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-09-12T03:04:41.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:41 smithi033 ceph-mon[35092]: pgmap v840: 97 pgs: 97 active+clean; 600 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-09-12T03:04:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:41 smithi184 ceph-mon[35321]: pgmap v840: 97 pgs: 97 active+clean; 600 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-09-12T03:04:42.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 25.06 MB/sec execute 103 sec latency 4827.671 ms 2024-09-12T03:04:43.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 24.82 MB/sec execute 104 sec latency 5827.716 ms 2024-09-12T03:04:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: pgmap v841: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.8 MiB/s wr, 465 op/s 2024-09-12T03:04:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:43 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: pgmap v841: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.8 MiB/s wr, 465 op/s 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:04:43.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:43 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: pgmap v841: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.8 MiB/s wr, 465 op/s 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:04:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:43 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:04:44.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 24.59 MB/sec execute 105 sec latency 6827.771 ms 2024-09-12T03:04:45.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 24.36 MB/sec execute 106 sec latency 7827.883 ms 2024-09-12T03:04:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:45 smithi016 ceph-mon[30925]: pgmap v842: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.8 MiB/s wr, 427 op/s 2024-09-12T03:04:45.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:45 smithi033 ceph-mon[35092]: pgmap v842: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.8 MiB/s wr, 427 op/s 2024-09-12T03:04:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:45 smithi184 ceph-mon[35321]: pgmap v842: 97 pgs: 97 active+clean; 605 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.8 MiB/s wr, 427 op/s 2024-09-12T03:04:46.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 24.13 MB/sec execute 107 sec latency 8827.984 ms 2024-09-12T03:04:47.015 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 23.90 MB/sec execute 108 sec latency 9828.093 ms 2024-09-12T03:04:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:47 smithi016 ceph-mon[30925]: pgmap v843: 97 pgs: 97 active+clean; 601 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 407 op/s 2024-09-12T03:04:47.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:47 smithi033 ceph-mon[35092]: pgmap v843: 97 pgs: 97 active+clean; 601 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 407 op/s 2024-09-12T03:04:47.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:47 smithi184 ceph-mon[35321]: pgmap v843: 97 pgs: 97 active+clean; 601 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 407 op/s 2024-09-12T03:04:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 23.68 MB/sec execute 109 sec latency 10828.219 ms 2024-09-12T03:04:49.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 23.47 MB/sec execute 110 sec latency 11828.300 ms 2024-09-12T03:04:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:49 smithi016 ceph-mon[30925]: pgmap v844: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-09-12T03:04:49.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:49 smithi033 ceph-mon[35092]: pgmap v844: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-09-12T03:04:49.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:49 smithi184 ceph-mon[35321]: pgmap v844: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-09-12T03:04:50.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 23.26 MB/sec execute 111 sec latency 12828.385 ms 2024-09-12T03:04:51.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 23.05 MB/sec execute 112 sec latency 13828.483 ms 2024-09-12T03:04:51.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:51 smithi016 ceph-mon[30925]: pgmap v845: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 169 op/s 2024-09-12T03:04:51.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:51 smithi033 ceph-mon[35092]: pgmap v845: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 169 op/s 2024-09-12T03:04:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:51 smithi184 ceph-mon[35321]: pgmap v845: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 169 op/s 2024-09-12T03:04:52.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 22.85 MB/sec execute 113 sec latency 14828.626 ms 2024-09-12T03:04:53.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 22.65 MB/sec execute 114 sec latency 15828.736 ms 2024-09-12T03:04:53.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:53 smithi016 ceph-mon[30925]: pgmap v846: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 929 KiB/s wr, 109 op/s 2024-09-12T03:04:53.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:53 smithi033 ceph-mon[35092]: pgmap v846: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 929 KiB/s wr, 109 op/s 2024-09-12T03:04:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:53 smithi184 ceph-mon[35321]: pgmap v846: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 929 KiB/s wr, 109 op/s 2024-09-12T03:04:54.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 22.45 MB/sec execute 115 sec latency 16828.814 ms 2024-09-12T03:04:55.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 22.26 MB/sec execute 116 sec latency 17828.949 ms 2024-09-12T03:04:55.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:55 smithi016 ceph-mon[30925]: pgmap v847: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 221 KiB/s wr, 34 op/s 2024-09-12T03:04:55.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:55 smithi033 ceph-mon[35092]: pgmap v847: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 221 KiB/s wr, 34 op/s 2024-09-12T03:04:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:55 smithi184 ceph-mon[35321]: pgmap v847: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 221 KiB/s wr, 34 op/s 2024-09-12T03:04:56.016 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 22.07 MB/sec execute 117 sec latency 18829.043 ms 2024-09-12T03:04:57.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 21.88 MB/sec execute 118 sec latency 19829.146 ms 2024-09-12T03:04:57.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:04:57.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:57 smithi016 ceph-mon[30925]: pgmap v848: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-09-12T03:04:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:57.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:57 smithi033 ceph-mon[35092]: pgmap v848: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-09-12T03:04:57.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:57 smithi184 ceph-mon[35321]: pgmap v848: 97 pgs: 97 active+clean; 598 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-09-12T03:04:58.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 21.69 MB/sec execute 119 sec latency 20829.214 ms 2024-09-12T03:04:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- 172.21.15.16:0/3468721336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 msgr2=0x7fb868101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 --2- 172.21.15.16:0/3468721336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868101200 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fb864009900 tx=0x7fb86402a480 comp rx=0 tx=0).stop 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- 172.21.15.16:0/3468721336 shutdown_connections 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 --2- 172.21.15.16:0/3468721336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 0x7fb86810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 --2- 172.21.15.16:0/3468721336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 0x7fb8681029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 --2- 172.21.15.16:0/3468721336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868101200 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- 172.21.15.16:0/3468721336 >> 172.21.15.16:0/3468721336 conn(0x7fb8680fc7d0 msgr2=0x7fb8680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- 172.21.15.16:0/3468721336 shutdown_connections 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- 172.21.15.16:0/3468721336 wait complete. 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 Processor -- start 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.166+0000 7fb8707de640 1 -- start start 2024-09-12T03:04:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 0x7fb8681998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 0x7fb8681a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb868110f80 con 0x7fb868100e20 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb868110e00 con 0x7fb868102ef0 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb868111100 con 0x7fb868102570 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49158/0 (socket says 172.21.15.16:49158) 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 -- 172.21.15.16:0/723198150 learned_addr learned my addr 172.21.15.16:0/723198150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86ed7b640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 0x7fb8681a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86dd79640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 0x7fb8681998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 -- 172.21.15.16:0/723198150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 msgr2=0x7fb8681998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 0x7fb8681998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 -- 172.21.15.16:0/723198150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 msgr2=0x7fb8681a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 0x7fb8681a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 -- 172.21.15.16:0/723198150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8681a4940 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb86e57a640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb864006010 tx=0x7fb864003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86403f070 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb864031c30 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8681a4b10 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.167+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864035030 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.168+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681a4f60 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.168+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb868101200 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.170+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb864042070 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.170+0000 7fb85f7fe640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 0x7fb8540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.170+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb8640ef890 con 0x7fb868100e20 2024-09-12T03:04:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.170+0000 7fb86dd79640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 0x7fb8540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:04:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.171+0000 7fb86dd79640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 0x7fb8540a5040 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb858005fd0 tx=0x7fb858005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:04:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.175+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb864031de0 con 0x7fb868100e20 2024-09-12T03:04:58.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.385+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb868109090 con 0x7fb868100e20 2024-09-12T03:04:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.386+0000 7fb85f7fe640 1 -- 172.21.15.16:0/723198150 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fb8640bc800 con 0x7fb868100e20 2024-09-12T03:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:04:58.388 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 msgr2=0x7fb8540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 0x7fb8540a5040 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb858005fd0 tx=0x7fb858005e20 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 msgr2=0x7fb868199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb864006010 tx=0x7fb864003c10 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 shutdown_connections 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb868102ef0 0x7fb8681a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb868102570 0x7fb8681998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb8540a2b80 0x7fb8540a5040 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 --2- 172.21.15.16:0/723198150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb868100e20 0x7fb868199380 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 >> 172.21.15.16:0/723198150 conn(0x7fb8680fc7d0 msgr2=0x7fb8680fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.388+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 shutdown_connections 2024-09-12T03:04:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:04:58.389+0000 7fb8707de640 1 -- 172.21.15.16:0/723198150 wait complete. 2024-09-12T03:04:59.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 21.51 MB/sec execute 120 sec latency 21829.498 ms 2024-09-12T03:04:59.255 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 28c3f551 --release --await' 2024-09-12T03:04:59.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 28c3f551 --release --await 2024-09-12T03:04:59.514 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:04:59.635 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:59 smithi016 ceph-mon[30925]: pgmap v849: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-09-12T03:04:59.635 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:04:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/723198150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:59.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:59 smithi033 ceph-mon[35092]: pgmap v849: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-09-12T03:04:59.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:04:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/723198150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:04:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:59 smithi184 ceph-mon[35321]: pgmap v849: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-09-12T03:04:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:04:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/723198150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 -- 172.21.15.16:0/435001163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 msgr2=0x7fa10c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 --2- 172.21.15.16:0/435001163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c10ea90 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fa104009ca0 tx=0x7fa10402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 -- 172.21.15.16:0/435001163 shutdown_connections 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 --2- 172.21.15.16:0/435001163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c10ea90 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 --2- 172.21.15.16:0/435001163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c102790 0x7fa10c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 --2- 172.21.15.16:0/435001163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c101de0 0x7fa10c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 -- 172.21.15.16:0/435001163 >> 172.21.15.16:0/435001163 conn(0x7fa10c0fd8c0 msgr2=0x7fa10c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 -- 172.21.15.16:0/435001163 shutdown_connections 2024-09-12T03:05:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.008+0000 7fa1129b0640 1 -- 172.21.15.16:0/435001163 wait complete. 2024-09-12T03:05:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.009+0000 7fa1129b0640 1 Processor -- start 2024-09-12T03:05:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.009+0000 7fa1129b0640 1 -- start start 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c101de0 0x7fa10c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 0x7fa10c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c111f70 con 0x7fa10c106f10 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c111df0 con 0x7fa10c102790 2024-09-12T03:05:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1120f0 con 0x7fa10c101de0 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49176/0 (socket says 172.21.15.16:49176) 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 -- 172.21.15.16:0/1754614934 learned_addr learned my addr 172.21.15.16:0/1754614934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa10bfff640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c101de0 0x7fa10c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 0x7fa10c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 -- 172.21.15.16:0/1754614934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c101de0 msgr2=0x7fa10c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c101de0 0x7fa10c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 -- 172.21.15.16:0/1754614934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 msgr2=0x7fa10c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 0x7fa10c1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 -- 172.21.15.16:0/1754614934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c19d420 con 0x7fa10c106f10 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 0x7fa10c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa110f4d640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fa104031a20 tx=0x7fa104031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1097fa640 1 -- 172.21.15.16:0/1754614934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104003c30 con 0x7fa10c106f10 2024-09-12T03:05:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1097fa640 1 -- 172.21.15.16:0/1754614934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa104003dd0 con 0x7fa10c106f10 2024-09-12T03:05:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c19d6b0 con 0x7fa10c106f10 2024-09-12T03:05:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.010+0000 7fa1097fa640 1 -- 172.21.15.16:0/1754614934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10402fc40 con 0x7fa10c106f10 2024-09-12T03:05:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.011+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10c1a4ff0 con 0x7fa10c106f10 2024-09-12T03:05:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa1097fa640 1 -- 172.21.15.16:0/1754614934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa104004050 con 0x7fa10c106f10 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 Processor -- start 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 -- start start 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c494340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c494880 0x55e12c4a3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e12c3f9620 con 0x55e12c4a3650 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e12c3a0e50 con 0x55e12c493f60 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa114d89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e12c3f55e0 con 0x55e12c494880 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.013+0000 7fa1097fa640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 0x7fa0f40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49184/0 (socket says 172.21.15.16:49184) 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 -- 172.21.15.16:0/801631236 learned_addr learned my addr 172.21.15.16:0/801631236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa1097fa640 1 -- 172.21.15.16:0/1754614934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa1040f03b0 con 0x7fa10c106f10 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa110f4d640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 0x7fa0f40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c494340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10bfff640 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c494880 0x55e12c4a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 -- 172.21.15.16:0/801631236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c494880 msgr2=0x55e12c4a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c494880 0x55e12c4a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 -- 172.21.15.16:0/801631236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 msgr2=0x55e12c494340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c494340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 -- 172.21.15.16:0/801631236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12c391d40 con 0x55e12c4a3650 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa110f4d640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 0x7fa0f40a50c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa104009620 tx=0x7fa104033040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fa100013610 tx=0x7fa10000c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328bb2af98d22c60 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.014+0000 7fa0f27fc640 1 -- 172.21.15.16:0/801631236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100019030 con 0x55e12c4a3650 2024-09-12T03:05:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa0f27fc640 1 -- 172.21.15.16:0/801631236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa100011a80 con 0x55e12c4a3650 2024-09-12T03:05:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa0f27fc640 1 -- 172.21.15.16:0/801631236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10000fa70 con 0x55e12c4a3650 2024-09-12T03:05:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 -- 172.21.15.16:0/801631236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 msgr2=0x55e12c4a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fa100013610 tx=0x7fa10000c6d0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 -- 172.21.15.16:0/801631236 shutdown_connections 2024-09-12T03:05:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c4a3650 0x55e12c4a5a40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c494880 0x55e12c4a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 --2- 172.21.15.16:0/801631236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c494340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 -- 172.21.15.16:0/801631236 >> 172.21.15.16:0/801631236 conn(0x55e12c4905f0 msgr2=0x55e12c4909d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 -- 172.21.15.16:0/801631236 shutdown_connections 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.015+0000 7fa114d89740 1 -- 172.21.15.16:0/801631236 wait complete. 2024-09-12T03:05:00.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 -- start start 2024-09-12T03:05:00.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88375 21.34 MB/sec execute 121 sec latency 22829.527 ms 2024-09-12T03:05:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 Processor -- start 2024-09-12T03:05:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c494880 0x55e12c4a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4a1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e12c4a6280 con 0x55e12c493f60 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e12c381f10 con 0x55e12c494880 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e12c4a6400 con 0x55e12c4a3650 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa114d89740 10 client.0 _ll_register_callbacks cb 0x55e12c3a5db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49188/0 (socket says 172.21.15.16:49188) 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa10b7fe640 1 -- 172.21.15.16:0/1296602091 learned_addr learned my addr 172.21.15.16:0/1296602091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.016+0000 7fa10bfff640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c494880 0x55e12c4a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 -- 172.21.15.16:0/1296602091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 msgr2=0x55e12c4a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 -- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c494880 msgr2=0x55e12c4a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c494880 0x55e12c4a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 -- 172.21.15.16:0/1296602091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12c108580 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fa1000058c0 tx=0x7fa100005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15164d8d0221e90a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa0eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100019070 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa100005450 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa100011a80 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa1000118d0 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa114d89740 10 client.19416 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.017+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e12c1d2960 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa100011a80 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa1000048e0 con 0x55e12c493f60 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 10 client.19416 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 10 client.19416 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 20 client.19416 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c4b31a0 0x55e12c4b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 4 client.19416 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e12c406b90 con 0x55e12c4b31a0 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa10bfff640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c4b31a0 0x55e12c4b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa0eb7fe640 10 client.19416 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.018+0000 7fa10bfff640 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c4b31a0 0x55e12c4b55b0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b97e0c851ef4a11 server_cookie=48cd57b39f3d85a9 in_seq=0 out_seq=0 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/1296602091 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e12c406b90 con 0x55e12c4b31a0 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa0eb7fe640 10 client.19416 handle_command_reply: tid=0 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 client.19416 shutdown 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 msgr2=0x55e12c4a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fa1000058c0 tx=0x7fa100005850 comp rx=0 tx=0).stop 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 shutdown_connections 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c4b31a0 0x55e12c4b55b0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c494880 0x55e12c4a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 --2- 172.21.15.16:0/1296602091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c493f60 0x55e12c4a0610 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa0eb7fe640 0 client.19416 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.019+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 >> 172.21.15.16:0/1296602091 conn(0x55e12c4905f0 msgr2=0x55e12c496350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.020+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 shutdown_connections 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.020+0000 7fa114d89740 1 -- 172.21.15.16:0/1296602091 wait complete. 2024-09-12T03:05:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.020+0000 7fa114d89740 20 client.19416 trim_cache size 0 max 16384 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.047+0000 7fa114d89740 1 Processor -- start 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 -- start start 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c493f60 0x55e12c4961a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c4a3650 0x55e12c4bb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ab2b0 con 0x55e12c494880 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1acdf0 con 0x55e12c4a3650 2024-09-12T03:05:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.048+0000 7fa114d89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ab980 con 0x55e12c493f60 2024-09-12T03:05:00.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.060+0000 7fa10bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.060+0000 7fa10bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49198/0 (socket says 172.21.15.16:49198) 2024-09-12T03:05:00.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.060+0000 7fa10bfff640 1 -- 172.21.15.16:0/2041789188 learned_addr learned my addr 172.21.15.16:0/2041789188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.065+0000 7fa10bfff640 1 -- 172.21.15.16:0/2041789188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c493f60 msgr2=0x55e12c4961a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:05:00.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.065+0000 7fa10bfff640 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c493f60 0x55e12c4961a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.065+0000 7fa10bfff640 1 -- 172.21.15.16:0/2041789188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c4a3650 msgr2=0x55e12c4bb3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:05:00.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.065+0000 7fa10bfff640 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c4a3650 0x55e12c4bb3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.065+0000 7fa10bfff640 1 -- 172.21.15.16:0/2041789188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12c4946b0 con 0x55e12c494880 2024-09-12T03:05:00.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.073+0000 7fa10bfff640 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x55e12c4b5e20 tx=0x7fa0fc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46578c492243c88 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.075+0000 7fa0f1ffb640 1 -- 172.21.15.16:0/2041789188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc022070 con 0x55e12c494880 2024-09-12T03:05:00.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.075+0000 7fa0f1ffb640 1 -- 172.21.15.16:0/2041789188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc0047a0 con 0x55e12c494880 2024-09-12T03:05:00.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.075+0000 7fa0f1ffb640 1 -- 172.21.15.16:0/2041789188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc01e8e0 con 0x55e12c494880 2024-09-12T03:05:00.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.076+0000 7fa114d89740 1 -- 172.21.15.16:0/2041789188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 msgr2=0x55e12c4bae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.077+0000 7fa114d89740 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x55e12c4b5e20 tx=0x7fa0fc005f90 comp rx=0 tx=0).stop 2024-09-12T03:05:00.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 -- 172.21.15.16:0/2041789188 shutdown_connections 2024-09-12T03:05:00.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c4a3650 0x55e12c4bb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4bae80 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 --2- 172.21.15.16:0/2041789188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c493f60 0x55e12c4961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 -- 172.21.15.16:0/2041789188 >> 172.21.15.16:0/2041789188 conn(0x55e12c4905f0 msgr2=0x55e12c495a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.080+0000 7fa114d89740 1 -- 172.21.15.16:0/2041789188 shutdown_connections 2024-09-12T03:05:00.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.081+0000 7fa114d89740 1 -- 172.21.15.16:0/2041789188 wait complete. 2024-09-12T03:05:00.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.081+0000 7fa114d89740 1 -- start start 2024-09-12T03:05:00.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.081+0000 7fa114d89740 1 Processor -- start 2024-09-12T03:05:00.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c49aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4ab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4abc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ac720 con 0x55e12c494880 2024-09-12T03:05:00.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ac050 con 0x55e12c493f60 2024-09-12T03:05:00.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1a9290 con 0x55e12c4a3650 2024-09-12T03:05:00.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.082+0000 7fa114d89740 10 client.0 _ll_register_callbacks cb 0x7fa10000cea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:00.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.084+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.084+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c49aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.084+0000 7fa10b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c49aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:50956/0 (socket says 172.21.15.16:50956) 2024-09-12T03:05:00.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.084+0000 7fa10b7fe640 1 -- 172.21.15.16:0/3740625816 learned_addr learned my addr 172.21.15.16:0/3740625816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.087+0000 7fa10b7fe640 1 -- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 msgr2=0x55e12c49aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.087+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c49aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.087+0000 7fa10b7fe640 1 -- 172.21.15.16:0/3740625816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 msgr2=0x55e12c4ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.087+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.087+0000 7fa10b7fe640 1 -- 172.21.15.16:0/3740625816 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12c497e20 con 0x55e12c4a3650 2024-09-12T03:05:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.090+0000 7fa10b7fe640 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4abc80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa100013610 tx=0x7fa100005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b6d2bdacb34ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.090+0000 7fa0eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:05:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100005250 con 0x55e12c4a3650 2024-09-12T03:05:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa1000053f0 con 0x55e12c4a3650 2024-09-12T03:05:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0fc0047a0 con 0x55e12c4a3650 2024-09-12T03:05:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa100047050 con 0x55e12c4a3650 2024-09-12T03:05:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa114d89740 10 client.28621 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.091+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e12c533940 con 0x55e12c4a3650 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc0047a0 con 0x55e12c4a3650 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa10000fa00 con 0x55e12c4a3650 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 10 client.28621 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 10 client.28621 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 20 client.28621 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c534010 0x55e12c536450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:00.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 4 client.28621 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "28c3f551", "release": true, "await": true}] 2024-09-12T03:05:00.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.093+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "28c3f551", "release": true, "await": true}) -- 0x55e12c4010a0 con 0x55e12c534010 2024-09-12T03:05:00.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.094+0000 7fa10bfff640 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c534010 0x55e12c536450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.097+0000 7fa0eb7fe640 10 client.28621 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.097+0000 7fa10bfff640 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c534010 0x55e12c536450 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6cc8f31e46b77af server_cookie=26be4483900fc38d in_seq=0 out_seq=0 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa0eb7fe640 1 -- 172.21.15.16:0/3740625816 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e12c4010a0 con 0x55e12c534010 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa0eb7fe640 10 client.28621 handle_command_reply: tid=0 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 client.28621 shutdown 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 msgr2=0x55e12c4abc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4abc80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa100013610 tx=0x7fa100005970 comp rx=0 tx=0).stop 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 shutdown_connections 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e12c534010 0x55e12c536450 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e12c4a3650 0x55e12c4abc80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e12c494880 0x55e12c4ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 --2- 172.21.15.16:0/3740625816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e12c493f60 0x55e12c49aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 >> 172.21.15.16:0/3740625816 conn(0x55e12c4905f0 msgr2=0x55e12c4962b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.204+0000 7fa0eb7fe640 0 client.28621 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 shutdown_connections 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa114d89740 1 -- 172.21.15.16:0/3740625816 wait complete. 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa114d89740 20 client.28621 trim_cache size 0 max 16384 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:05:00.207 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 64, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "28c3f551": { 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 64, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.2, 2024-09-12T03:05:00.208 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:00.209 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 msgr2=0x7fa0f40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 0x7fa0f40a50c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa104009620 tx=0x7fa104033040 comp rx=0 tx=0).stop 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 msgr2=0x7fa10c19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.205+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fa104031a20 tx=0x7fa104031a50 comp rx=0 tx=0).stop 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 shutdown_connections 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0f40a2c00 0x7fa0f40a50c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa10c106f10 0x7fa10c19cd40 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa10c102790 0x7fa10c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 --2- 172.21.15.16:0/1754614934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa10c101de0 0x7fa10c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 >> 172.21.15.16:0/1754614934 conn(0x7fa10c0fd8c0 msgr2=0x7fa10c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 shutdown_connections 2024-09-12T03:05:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:00.206+0000 7fa1129b0640 1 -- 172.21.15.16:0/1754614934 wait complete. 2024-09-12T03:05:00.932 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '28c3f551', total seconds elapsed: 27.7 2024-09-12T03:05:01.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88782 21.32 MB/sec execute 122 sec latency 23088.544 ms 2024-09-12T03:05:01.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:01 smithi016 ceph-mon[30925]: pgmap v850: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-09-12T03:05:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:01 smithi033 ceph-mon[35092]: pgmap v850: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-09-12T03:05:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:01 smithi184 ceph-mon[35321]: pgmap v850: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-09-12T03:05:02.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 89568 21.43 MB/sec execute 123 sec latency 47.218 ms 2024-09-12T03:05:03.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 90136 21.57 MB/sec execute 124 sec latency 52.720 ms 2024-09-12T03:05:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:03 smithi016 ceph-mon[30925]: pgmap v851: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-12T03:05:03.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:03 smithi033 ceph-mon[35092]: pgmap v851: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-12T03:05:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:03 smithi184 ceph-mon[35321]: pgmap v851: 97 pgs: 97 active+clean; 599 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-12T03:05:04.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 90809 21.60 MB/sec execute 125 sec latency 59.859 ms 2024-09-12T03:05:04.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:05:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:05.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 91704 21.85 MB/sec execute 126 sec latency 52.555 ms 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 -- 172.21.15.16:0/3339329219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c101de0 msgr2=0x7fcf7c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 --2- 172.21.15.16:0/3339329219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1021c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009900 tx=0x7fcf6c02a3f0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 -- 172.21.15.16:0/3339329219 shutdown_connections 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 --2- 172.21.15.16:0/3339329219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c106f10 0x7fcf7c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 --2- 172.21.15.16:0/3339329219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c102790 0x7fcf7c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 --2- 172.21.15.16:0/3339329219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1021c0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 -- 172.21.15.16:0/3339329219 >> 172.21.15.16:0/3339329219 conn(0x7fcf7c0fd8c0 msgr2=0x7fcf7c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.556+0000 7fcf82343640 1 -- 172.21.15.16:0/3339329219 shutdown_connections 2024-09-12T03:05:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.557+0000 7fcf82343640 1 -- 172.21.15.16:0/3339329219 wait complete. 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.557+0000 7fcf82343640 1 Processor -- start 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.557+0000 7fcf82343640 1 -- start start 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c102790 0x7fcf7c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 0x7fcf7c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c111f30 con 0x7fcf7c106f10 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c111db0 con 0x7fcf7c101de0 2024-09-12T03:05:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf82343640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1120b0 con 0x7fcf7c102790 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 0x7fcf7c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c102790 0x7fcf7c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53994/0 (socket says 172.21.15.16:53994) 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf7bfff640 1 -- 172.21.15.16:0/2370185956 learned_addr learned my addr 172.21.15.16:0/2370185956 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 -- 172.21.15.16:0/2370185956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c102790 msgr2=0x7fcf7c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c102790 0x7fcf7c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 -- 172.21.15.16:0/2370185956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 msgr2=0x7fcf7c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.558+0000 7fcf808e0640 1 -- 172.21.15.16:0/2370185956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7c19d390 con 0x7fcf7c106f10 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf808e0640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 0x7fcf7c19ccb0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009fd0 tx=0x7fcf7400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf79ffb640 1 -- 172.21.15.16:0/2370185956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7400ce20 con 0x7fcf7c106f10 2024-09-12T03:05:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf79ffb640 1 -- 172.21.15.16:0/2370185956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf74005020 con 0x7fcf7c106f10 2024-09-12T03:05:05.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf79ffb640 1 -- 172.21.15.16:0/2370185956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf74010720 con 0x7fcf7c106f10 2024-09-12T03:05:05.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf7c19d680 con 0x7fcf7c106f10 2024-09-12T03:05:05.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.559+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf7c074420 con 0x7fcf7c106f10 2024-09-12T03:05:05.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.561+0000 7fcf79ffb640 1 -- 172.21.15.16:0/2370185956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcf74005540 con 0x7fcf7c106f10 2024-09-12T03:05:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.562+0000 7fcf8471c740 1 Processor -- start 2024-09-12T03:05:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.562+0000 7fcf79ffb640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 0x7fcf680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.562+0000 7fcf8471c740 1 -- start start 2024-09-12T03:05:05.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf73fff640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 0x7fcf680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf79ffb640 1 -- 172.21.15.16:0/2370185956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcf740cf710 con 0x7fcf7c106f10 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffb1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffbea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a11ff172e0 con 0x55a11ffb1dc0 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a11febece0 con 0x55a11ffb14a0 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf8471c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a11ff132a0 con 0x55a11ffbc680 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43674/0 (socket says 172.21.15.16:43674) 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffb1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf73fff640 1 -- 172.21.15.16:0/3639295014 learned_addr learned my addr 172.21.15.16:0/3639295014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf808e0640 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffbea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 -- 172.21.15.16:0/3639295014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 msgr2=0x55a11ffbea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffbea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 -- 172.21.15.16:0/3639295014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 msgr2=0x55a11ffbc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf7bfff640 1 -- 172.21.15.16:0/3639295014 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11fea3440 con 0x55a11ffb14a0 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.563+0000 7fcf73fff640 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.564+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffb1880 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009230 tx=0x7fcf6c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23e02b5214f823cd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.564+0000 7fcf73fff640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 0x7fcf680a5040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x55a11ffae890 tx=0x7fcf6400b7c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.564+0000 7fcf727fc640 1 -- 172.21.15.16:0/3639295014 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c040070 con 0x55a11ffb14a0 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.564+0000 7fcf727fc640 1 -- 172.21.15.16:0/3639295014 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6c0317b0 con 0x55a11ffb14a0 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.564+0000 7fcf727fc640 1 -- 172.21.15.16:0/3639295014 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c035040 con 0x55a11ffb14a0 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- 172.21.15.16:0/3639295014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 msgr2=0x55a11ffb1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffb1880 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009230 tx=0x7fcf6c005e80 comp rx=0 tx=0).stop 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- 172.21.15.16:0/3639295014 shutdown_connections 2024-09-12T03:05:05.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffbea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffbc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 --2- 172.21.15.16:0/3639295014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffb1880 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- 172.21.15.16:0/3639295014 >> 172.21.15.16:0/3639295014 conn(0x55a11ffada30 msgr2=0x55a11ffade10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- 172.21.15.16:0/3639295014 shutdown_connections 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- 172.21.15.16:0/3639295014 wait complete. 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.565+0000 7fcf8471c740 1 -- start start 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 Processor -- start 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffb9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffb9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a11ffbf250 con 0x55a11ffb1dc0 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a11fed2050 con 0x55a11ffb14a0 2024-09-12T03:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a11ffbf3d0 con 0x55a11ffbc680 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf8471c740 10 client.0 _ll_register_callbacks cb 0x55a11ff12160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffb9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf808e0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffb9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf808e0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffb9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34436/0 (socket says 172.21.15.16:34436) 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf808e0640 1 -- 172.21.15.16:0/81117865 learned_addr learned my addr 172.21.15.16:0/81117865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54006/0 (socket says 172.21.15.16:54006) 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.566+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffb9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43676/0 (socket says 172.21.15.16:43676) 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 -- 172.21.15.16:0/81117865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 msgr2=0x55a11ffb9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffb9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 -- 172.21.15.16:0/81117865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 msgr2=0x55a11ffb9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffb9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 -- 172.21.15.16:0/81117865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11fc883c0 con 0x55a11ffb14a0 2024-09-12T03:05:05.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c000c00 tx=0x7fcf6c032ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f58ab648f563da1a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:05.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c035070 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf6c0317b0 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.567+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcf6c005550 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcf6c040040 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 10 client.28724 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a11ffcabd0 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6c0317b0 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcf6c003c00 con 0x55a11ffb14a0 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 10 client.28724 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 10 client.28724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 20 client.28724 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:05.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffcbf20 0x55a11ffce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 4 client.28724 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a11ff24850 con 0x55a11ffcbf20 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.568+0000 7fcf73fff640 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffcbf20 0x55a11ffce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf73fff640 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffcbf20 0x55a11ffce310 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67ae6c74b9eca9c5 server_cookie=f54ee603c88656eb in_seq=0 out_seq=0 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf5b7fe640 10 client.28724 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/81117865 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a11ff24850 con 0x55a11ffcbf20 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf5b7fe640 10 client.28724 handle_command_reply: tid=0 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf8471c740 1 client.28724 shutdown 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 msgr2=0x55a11ffbbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.569+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c000c00 tx=0x7fcf6c032ee0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 shutdown_connections 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffcbf20 0x55a11ffce310 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffbc680 0x55a11ffb9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb1dc0 0x55a11ffb9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 --2- 172.21.15.16:0/81117865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb14a0 0x55a11ffbbfa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf5b7fe640 0 client.28724 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 >> 172.21.15.16:0/81117865 conn(0x55a11ffada30 msgr2=0x55a11ffacb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 shutdown_connections 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 1 -- 172.21.15.16:0/81117865 wait complete. 2024-09-12T03:05:05.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.570+0000 7fcf8471c740 20 client.28724 trim_cache size 0 max 16384 2024-09-12T03:05:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 Processor -- start 2024-09-12T03:05:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 -- start start 2024-09-12T03:05:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffb14a0 0x55a11ffd1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a11ffd1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1af690 con 0x55a11ffada30 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1b11d0 con 0x55a11ffb1dc0 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf8471c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1afd60 con 0x55a11ffb14a0 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43684/0 (socket says 172.21.15.16:43684) 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 -- 172.21.15.16:0/2164422066 learned_addr learned my addr 172.21.15.16:0/2164422066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf73fff640 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffb14a0 0x55a11ffd1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf808e0640 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a11ffd1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 -- 172.21.15.16:0/2164422066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffb14a0 msgr2=0x55a11ffd1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffb14a0 0x55a11ffd1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 -- 172.21.15.16:0/2164422066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 msgr2=0x55a11ffd1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a11ffd1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.590+0000 7fcf7bfff640 1 -- 172.21.15.16:0/2164422066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11ffae6c0 con 0x55a11ffada30 2024-09-12T03:05:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.591+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c02a430 tx=0x7fcf6c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8537c4618b9efce0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.591+0000 7fcf71ffb640 1 -- 172.21.15.16:0/2164422066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c000e60 con 0x55a11ffada30 2024-09-12T03:05:05.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.591+0000 7fcf71ffb640 1 -- 172.21.15.16:0/2164422066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6c05c070 con 0x55a11ffada30 2024-09-12T03:05:05.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.591+0000 7fcf71ffb640 1 -- 172.21.15.16:0/2164422066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c035070 con 0x55a11ffada30 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- 172.21.15.16:0/2164422066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 msgr2=0x55a11ffb7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c02a430 tx=0x7fcf6c005e80 comp rx=0 tx=0).stop 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- 172.21.15.16:0/2164422066 shutdown_connections 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a11ffd1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffb14a0 0x55a11ffd1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 --2- 172.21.15.16:0/2164422066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffada30 0x55a11ffb7b20 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- 172.21.15.16:0/2164422066 >> 172.21.15.16:0/2164422066 conn(0x55a11ffb98f0 msgr2=0x55a11ffacb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- 172.21.15.16:0/2164422066 shutdown_connections 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- 172.21.15.16:0/2164422066 wait complete. 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.592+0000 7fcf8471c740 1 -- start start 2024-09-12T03:05:05.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 Processor -- start 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffada30 0x55a12002a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a12002b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1b0b00 con 0x55a11ffb14a0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1b0430 con 0x55a11ffb1dc0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1ad670 con 0x55a11ffada30 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf8471c740 10 client.0 _ll_register_callbacks cb 0x7fcf6400bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf808e0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a12002b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43686/0 (socket says 172.21.15.16:43686) 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 -- 172.21.15.16:0/211437134 learned_addr learned my addr 172.21.15.16:0/211437134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffada30 0x55a12002a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 -- 172.21.15.16:0/211437134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffada30 msgr2=0x55a12002a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffada30 0x55a12002a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 -- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 msgr2=0x55a12002b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a12002b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.593+0000 7fcf73fff640 1 -- 172.21.15.16:0/211437134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a120035ac0 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf808e0640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a12002b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf73fff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcf64005fd0 tx=0x7fcf6400b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df3686eb6cd5d76 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf64035060 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcf64027820 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf6c05c070 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcf64027a20 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf8471c740 10 client.19437 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.594+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a120036e30 con 0x55a11ffb14a0 2024-09-12T03:05:05.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6c05c070 con 0x55a11ffb14a0 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcf64030c50 con 0x55a11ffb14a0 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 10 client.19437 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 10 client.19437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 20 client.19437 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffbc680 0x55a120039540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:05.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 4 client.19437 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a11ff1ed60 con 0x55a11ffbc680 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.595+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffbc680 0x55a120039540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.596+0000 7fcf7bfff640 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffbc680 0x55a120039540 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a05415335f3ff93 server_cookie=6c950ea1a99496ce in_seq=0 out_seq=0 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.596+0000 7fcf5b7fe640 10 client.19437 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.598+0000 7fcf5b7fe640 1 -- 172.21.15.16:0/211437134 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a11ff1ed60 con 0x55a11ffbc680 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.598+0000 7fcf5b7fe640 10 client.19437 handle_command_reply: tid=0 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.598+0000 7fcf8471c740 1 client.19437 shutdown 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 msgr2=0x55a12002ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcf64005fd0 tx=0x7fcf6400b410 comp rx=0 tx=0).stop 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 shutdown_connections 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a11ffbc680 0x55a120039540 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a11ffb1dc0 0x55a12002b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a11ffb14a0 0x55a12002ad30 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 --2- 172.21.15.16:0/211437134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a11ffada30 0x55a12002a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf5b7fe640 0 client.19437 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 >> 172.21.15.16:0/211437134 conn(0x55a11ffb98f0 msgr2=0x55a11ffbaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 shutdown_connections 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 1 -- 172.21.15.16:0/211437134 wait complete. 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.599+0000 7fcf8471c740 20 client.19437 trim_cache size 0 max 16384 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:05:05.602 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 msgr2=0x7fcf680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 0x7fcf680a5040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x55a11ffae890 tx=0x7fcf6400b7c0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 msgr2=0x7fcf7c19ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 0x7fcf7c19ccb0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009fd0 tx=0x7fcf7400ef10 comp rx=0 tx=0).stop 2024-09-12T03:05:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 shutdown_connections 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcf680a2b80 0x7fcf680a5040 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c106f10 0x7fcf7c19ccb0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf7c102790 0x7fcf7c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 --2- 172.21.15.16:0/2370185956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcf7c101de0 0x7fcf7c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 >> 172.21.15.16:0/2370185956 conn(0x7fcf7c0fd8c0 msgr2=0x7fcf7c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 shutdown_connections 2024-09-12T03:05:05.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:05.600+0000 7fcf82343640 1 -- 172.21.15.16:0/2370185956 wait complete. 2024-09-12T03:05:05.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:05 smithi016 ceph-mon[30925]: pgmap v852: 97 pgs: 97 active+clean; 596 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 122 op/s 2024-09-12T03:05:05.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:05 smithi033 ceph-mon[35092]: pgmap v852: 97 pgs: 97 active+clean; 596 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 122 op/s 2024-09-12T03:05:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:05 smithi184 ceph-mon[35321]: pgmap v852: 97 pgs: 97 active+clean; 596 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 122 op/s 2024-09-12T03:05:06.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 92452 21.99 MB/sec execute 127 sec latency 51.368 ms 2024-09-12T03:05:06.562 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:05:06.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:05:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:07.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 93107 22.13 MB/sec execute 128 sec latency 73.728 ms 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3722604198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8108290 msgr2=0x7f2fd810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3722604198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8108290 0x7f2fd810ec30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009900 tx=0x7f2fd002a400 comp rx=0 tx=0).stop 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3722604198 shutdown_connections 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3722604198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8108290 0x7f2fd810ec30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3722604198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8103740 0x7f2fd8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3722604198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd8102d90 0x7f2fd8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3722604198 >> 172.21.15.16:0/3722604198 conn(0x7f2fd80fc7d0 msgr2=0x7f2fd80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3722604198 shutdown_connections 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3722604198 wait complete. 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.396+0000 7f2fdd82e640 1 Processor -- start 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 -- start start 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd8102d90 0x7f2fd81009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8103740 0x7f2fd8100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81120a0 con 0x7f2fd8102d90 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8111f20 con 0x7f2fd8108290 2024-09-12T03:05:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fdd82e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8112220 con 0x7f2fd8103740 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54042/0 (socket says 172.21.15.16:54042) 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.397+0000 7f2fd77fe640 1 -- 172.21.15.16:0/3032143794 learned_addr learned my addr 172.21.15.16:0/3032143794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd67fc640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8103740 0x7f2fd8100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 -- 172.21.15.16:0/3032143794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8103740 msgr2=0x7f2fd8100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8103740 0x7f2fd8100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 -- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd8102d90 msgr2=0x7f2fd81009f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd8102d90 0x7f2fd81009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 -- 172.21.15.16:0/3032143794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd81aae90 con 0x7f2fd8108290 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fd77fe640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fd002a920 tx=0x7f2fd0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.398+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd003f070 con 0x7f2fd8108290 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.399+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fd0031c20 con 0x7f2fd8108290 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.399+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0035030 con 0x7f2fd8108290 2024-09-12T03:05:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.399+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd81ab060 con 0x7f2fd8108290 2024-09-12T03:05:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.399+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd81ab490 con 0x7f2fd8108290 2024-09-12T03:05:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.401+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd806a7f0 con 0x7f2fd8108290 2024-09-12T03:05:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.401+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2fd0042070 con 0x7f2fd8108290 2024-09-12T03:05:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.402+0000 7f2fdc82c640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 0x7f2fc00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.402+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2fd00ef9f0 con 0x7f2fd8108290 2024-09-12T03:05:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.402+0000 7f2fd6ffd640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 0x7f2fc00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.403+0000 7f2fd6ffd640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 0x7f2fc00a50c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80058c0 tx=0x7f2fc8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.408+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2fd00bc900 con 0x7f2fd8108290 2024-09-12T03:05:07.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:07 smithi016 ceph-mon[30925]: pgmap v853: 97 pgs: 97 active+clean; 585 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-09-12T03:05:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.690+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fd81ab750 con 0x7f2fd8108290 2024-09-12T03:05:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.691+0000 7f2fdc82c640 1 -- 172.21.15.16:0/3032143794 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f2fd00bc200 con 0x7f2fd8108290 2024-09-12T03:05:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:05:07.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:05:07.693 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:05:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.693+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 msgr2=0x7f2fc00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.693+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 0x7f2fc00a50c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80058c0 tx=0x7f2fc8005850 comp rx=0 tx=0).stop 2024-09-12T03:05:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.693+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 msgr2=0x7f2fd8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.693+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fd002a920 tx=0x7f2fd0003b90 comp rx=0 tx=0).stop 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 shutdown_connections 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2fd8108290 0x7f2fd8101470 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd8103740 0x7f2fd8100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2fc00a2c00 0x7f2fc00a50c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 --2- 172.21.15.16:0/3032143794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd8102d90 0x7f2fd81009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 >> 172.21.15.16:0/3032143794 conn(0x7f2fd80fc7d0 msgr2=0x7f2fd8105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 shutdown_connections 2024-09-12T03:05:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:07.694+0000 7f2fdd82e640 1 -- 172.21.15.16:0/3032143794 wait complete. 2024-09-12T03:05:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:07 smithi033 ceph-mon[35092]: pgmap v853: 97 pgs: 97 active+clean; 585 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-09-12T03:05:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:07 smithi184 ceph-mon[35321]: pgmap v853: 97 pgs: 97 active+clean; 585 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-09-12T03:05:08.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 93835 22.29 MB/sec execute 129 sec latency 79.006 ms 2024-09-12T03:05:08.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:08 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3032143794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:08.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:05:08.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:08 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3032143794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:08 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3032143794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:09.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 94473 22.33 MB/sec execute 130 sec latency 59.179 ms 2024-09-12T03:05:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:09.465 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:09 smithi016 ceph-mon[30925]: pgmap v854: 97 pgs: 97 active+clean; 583 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2351731509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc100e20 msgr2=0x7fd5cc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2351731509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc100e20 0x7fd5cc101200 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009cd0 tx=0x7fd5c402a9c0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2351731509 shutdown_connections 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2351731509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 0x7fd5cc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2351731509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc102570 0x7fd5cc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2351731509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc100e20 0x7fd5cc101200 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009cd0 tx=0x7fd5c402a9c0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.577+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2351731509 >> 172.21.15.16:0/2351731509 conn(0x7fd5cc0fc7d0 msgr2=0x7fd5cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.578+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2351731509 shutdown_connections 2024-09-12T03:05:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.578+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2351731509 wait complete. 2024-09-12T03:05:09.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.579+0000 7fd5d25cd640 1 Processor -- start 2024-09-12T03:05:09.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.579+0000 7fd5d25cd640 1 -- start start 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc102570 0x7fd5cc109f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 0x7fd5cc10a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc10fe70 con 0x7fd5cc102ef0 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc10fcf0 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d25cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc10fff0 con 0x7fd5cc102570 2024-09-12T03:05:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d0b6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d0b6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54070/0 (socket says 172.21.15.16:54070) 2024-09-12T03:05:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 0x7fd5cc10a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/2122741527 learned_addr learned my addr 172.21.15.16:0/2122741527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.580+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc102570 0x7fd5cc109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/2122741527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc102570 msgr2=0x7fd5cc109f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc102570 0x7fd5cc109f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 msgr2=0x7fd5cc10a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 0x7fd5cc10a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/2122741527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5cc1ab270 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c970 tx=0x7fd5c000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5c97fa640 1 -- 172.21.15.16:0/2122741527 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0007bb0 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5c97fa640 1 -- 172.21.15.16:0/2122741527 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c0007d50 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5cc1ab560 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5c97fa640 1 -- 172.21.15.16:0/2122741527 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0018800 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.581+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5cc1aba50 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.584+0000 7fd5c97fa640 1 -- 172.21.15.16:0/2122741527 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd5c00189e0 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.584+0000 7fd5d49a6740 1 Processor -- start 2024-09-12T03:05:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.584+0000 7fd5d49a6740 1 -- start start 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f3370 0x55c7e60f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c7e604e2f0 con 0x55c7e60e9b40 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5ff5cf0 con 0x55c7e60e85d0 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c7e604a2b0 con 0x55c7e60f3370 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54078/0 (socket says 172.21.15.16:54078) 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5cbfff640 1 -- 172.21.15.16:0/1063802795 learned_addr learned my addr 172.21.15.16:0/1063802795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.585+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f3370 0x55c7e60f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 -- 172.21.15.16:0/1063802795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f3370 msgr2=0x55c7e60f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f3370 0x55c7e60f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 -- 172.21.15.16:0/1063802795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 msgr2=0x55c7e60e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 -- 172.21.15.16:0/1063802795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7e5fda3a0 con 0x55c7e60e85d0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009430 tx=0x7fd5c4030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b15e473c03b70e78 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5b27fc640 1 -- 172.21.15.16:0/1063802795 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4034020 con 0x55c7e60e85d0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.586+0000 7fd5c97fa640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 0x7fd5b40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5b27fc640 1 -- 172.21.15.16:0/1063802795 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c40098e0 con 0x55c7e60e85d0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5b27fc640 1 -- 172.21.15.16:0/1063802795 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c402f630 con 0x55c7e60e85d0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 0x7fd5b40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5c97fa640 1 -- 172.21.15.16:0/2122741527 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd5c00cf050 con 0x7fd5cc10a9c0 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5d49a6740 1 -- 172.21.15.16:0/1063802795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 msgr2=0x55c7e60e8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.587+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009430 tx=0x7fd5c4030000 comp rx=0 tx=0).stop 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.588+0000 7fd5d49a6740 1 -- 172.21.15.16:0/1063802795 shutdown_connections 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.588+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f3370 0x55c7e60f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.588+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60e85d0 0x55c7e60e8a10 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009430 tx=0x7fd5c4030000 comp rx=0 tx=0).stop 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.588+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/1063802795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.588+0000 7fd5d49a6740 1 -- 172.21.15.16:0/1063802795 >> 172.21.15.16:0/1063802795 conn(0x55c7e60b1310 msgr2=0x55c7e60e4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.589+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 0x7fd5b40a5040 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55c7e60e9070 tx=0x7fd5bc005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.589+0000 7fd5d49a6740 1 -- 172.21.15.16:0/1063802795 shutdown_connections 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.589+0000 7fd5d49a6740 1 -- 172.21.15.16:0/1063802795 wait complete. 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.589+0000 7fd5d49a6740 1 -- start start 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.590+0000 7fd5d49a6740 1 Processor -- start 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.590+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.590+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60f0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60f01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34500/0 (socket says 172.21.15.16:34500) 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5cbfff640 1 -- 172.21.15.16:0/813283474 learned_addr learned my addr 172.21.15.16:0/813283474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c7e60f6010 con 0x55c7e60f3370 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c7e6009060 con 0x55c7e60f0c60 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c7e60f6190 con 0x55c7e60e9b40 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d49a6740 10 client.0 _ll_register_callbacks cb 0x55c7e5d5c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.591+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 -- 172.21.15.16:0/813283474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 msgr2=0x55c7e60f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60f01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 -- 172.21.15.16:0/813283474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 msgr2=0x55c7e60ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7e5dbf3c0 con 0x55c7e60f3370 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.592+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60f0720 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005d00 tx=0x7fd5bc00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14ad2b049749fb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc005220 con 0x55c7e60f3370 2024-09-12T03:05:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd5bc0053c0 con 0x55c7e60f3370 2024-09-12T03:05:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c40098e0 con 0x55c7e60f3370 2024-09-12T03:05:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc010e10 con 0x55c7e60f3370 2024-09-12T03:05:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5d49a6740 10 client.19449 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.593+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c7e61020e0 con 0x55c7e60f3370 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c40098e0 con 0x55c7e60f3370 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd5bc01c040 con 0x55c7e60f3370 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 10 client.19449 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 10 client.19449 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 20 client.19449 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e61035a0 0x55c7e6105a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 4 client.19449 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7e605b860 con 0x55c7e61035a0 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.594+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e61035a0 0x55c7e6105a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.595+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e61035a0 0x55c7e6105a00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d415b3ce695fce3 server_cookie=bcae05fd97618427 in_seq=0 out_seq=0 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.595+0000 7fd5ab7fe640 10 client.19449 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.595+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/813283474 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c7e605b860 con 0x55c7e61035a0 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.595+0000 7fd5ab7fe640 10 client.19449 handle_command_reply: tid=0 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 client.19449 shutdown 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 msgr2=0x55c7e60f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60f0720 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005d00 tx=0x7fd5bc00b390 comp rx=0 tx=0).stop 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 shutdown_connections 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e61035a0 0x55c7e6105a00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60f0720 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/813283474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 >> 172.21.15.16:0/813283474 conn(0x55c7e60b1310 msgr2=0x55c7e60ea930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5ab7fe640 0 client.19449 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 shutdown_connections 2024-09-12T03:05:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 1 -- 172.21.15.16:0/813283474 wait complete. 2024-09-12T03:05:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.596+0000 7fd5d49a6740 20 client.19449 trim_cache size 0 max 16384 2024-09-12T03:05:09.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5d49a6740 1 Processor -- start 2024-09-12T03:05:09.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5d49a6740 1 -- start start 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60ff820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.618+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34508/0 (socket says 172.21.15.16:34508) 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54102/0 (socket says 172.21.15.16:54102) 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 -- 172.21.15.16:0/4024457366 learned_addr learned my addr 172.21.15.16:0/4024457366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b3f70 con 0x55c7e60f3370 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b5ab0 con 0x55c7e60f0c60 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b4640 con 0x55c7e60e9b40 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60ff820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 -- 172.21.15.16:0/4024457366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 msgr2=0x55c7e60ff2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 -- 172.21.15.16:0/4024457366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 msgr2=0x55c7e60ff820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60ff820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.619+0000 7fd5cbfff640 1 -- 172.21.15.16:0/4024457366 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7e60e5dd0 con 0x55c7e60e9b40 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.620+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402aee0 tx=0x7fd5c402b080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e69bafa053e362ee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.620+0000 7fd5b1ffb640 1 -- 172.21.15.16:0/4024457366 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c405b070 con 0x55c7e60e9b40 2024-09-12T03:05:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.620+0000 7fd5b1ffb640 1 -- 172.21.15.16:0/4024457366 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4009780 con 0x55c7e60e9b40 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.621+0000 7fd5b1ffb640 1 -- 172.21.15.16:0/4024457366 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4034040 con 0x55c7e60e9b40 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.621+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 msgr2=0x55c7e60efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.621+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402aee0 tx=0x7fd5c402b080 comp rx=0 tx=0).stop 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 shutdown_connections 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60f3370 0x55c7e60ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f0c60 0x55c7e60ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/4024457366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60e9b40 0x55c7e60efac0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 >> 172.21.15.16:0/4024457366 conn(0x55c7e60b1310 msgr2=0x55c7e60e3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 shutdown_connections 2024-09-12T03:05:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 -- 172.21.15.16:0/4024457366 wait complete. 2024-09-12T03:05:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.622+0000 7fd5d49a6740 1 -- start start 2024-09-12T03:05:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 Processor -- start 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f0c60 0x55c7e60f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b53e0 con 0x55c7e60e9b40 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b4d10 con 0x55c7e60f3370 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1b1f50 con 0x55c7e60f0c60 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d49a6740 10 client.0 _ll_register_callbacks cb 0x7fd5bc00ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d0b6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d0b6a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54106/0 (socket says 172.21.15.16:54106) 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/3022298709 learned_addr learned my addr 172.21.15.16:0/3022298709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.623+0000 7fd5cb7fe640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f0c60 0x55c7e60f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.624+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/3022298709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f0c60 msgr2=0x55c7e60f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.624+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f0c60 0x55c7e60f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.624+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 msgr2=0x55c7e60fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.624+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.624+0000 7fd5d0b6a640 1 -- 172.21.15.16:0/3022298709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7e60f7980 con 0x55c7e60f3370 2024-09-12T03:05:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.626+0000 7fd5d0b6a640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5c009b5c0 tx=0x7fd5c009c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f648cbc77155ccee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c00b9060 con 0x55c7e60f3370 2024-09-12T03:05:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c4009780 con 0x55c7e60f3370 2024-09-12T03:05:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd5c00a40a0 con 0x55c7e60f3370 2024-09-12T03:05:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5c00b4070 con 0x55c7e60f3370 2024-09-12T03:05:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5d49a6740 10 client.28754 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.627+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c7e616e170 con 0x55c7e60f3370 2024-09-12T03:05:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4009780 con 0x55c7e60f3370 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd5c0020030 con 0x55c7e60f3370 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 10 client.28754 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 10 client.28754 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 20 client.28754 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e616e7f0 0x55c7e6170be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 4 client.28754 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.628+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7e6055d70 con 0x55c7e616e7f0 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.629+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e616e7f0 0x55c7e6170be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.629+0000 7fd5cbfff640 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e616e7f0 0x55c7e6170be0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c3dd6414e83d20e server_cookie=358bb3c97762cef8 in_seq=0 out_seq=0 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.629+0000 7fd5ab7fe640 10 client.28754 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5ab7fe640 1 -- 172.21.15.16:0/3022298709 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7e6055d70 con 0x55c7e616e7f0 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5ab7fe640 10 client.28754 handle_command_reply: tid=0 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 client.28754 shutdown 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 msgr2=0x55c7e60f7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5c009b5c0 tx=0x7fd5c009c100 comp rx=0 tx=0).stop 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 shutdown_connections 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55c7e616e7f0 0x55c7e6170be0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c7e60f3370 0x55c7e60f7310 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7e60f0c60 0x55c7e60f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5ab7fe640 0 client.28754 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 --2- 172.21.15.16:0/3022298709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7e60e9b40 0x55c7e60fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.630+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 >> 172.21.15.16:0/3022298709 conn(0x55c7e60b1310 msgr2=0x55c7e60f1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.631+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 shutdown_connections 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.631+0000 7fd5d49a6740 1 -- 172.21.15.16:0/3022298709 wait complete. 2024-09-12T03:05:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.631+0000 7fd5d49a6740 20 client.28754 trim_cache size 0 max 16384 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 msgr2=0x7fd5b40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 0x7fd5b40a5040 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55c7e60e9070 tx=0x7fd5bc005c90 comp rx=0 tx=0).stop 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 msgr2=0x7fd5cc1aaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c970 tx=0x7fd5c000ce30 comp rx=0 tx=0).stop 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 shutdown_connections 2024-09-12T03:05:09.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5cc10a9c0 0x7fd5cc1aaaf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5b40a2b80 0x7fd5b40a5040 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5cc102ef0 0x7fd5cc10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 --2- 172.21.15.16:0/2122741527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5cc102570 0x7fd5cc109f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 >> 172.21.15.16:0/2122741527 conn(0x7fd5cc0fc7d0 msgr2=0x7fd5cc0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 shutdown_connections 2024-09-12T03:05:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:09.632+0000 7fd5d25cd640 1 -- 172.21.15.16:0/2122741527 wait complete. 2024-09-12T03:05:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:09 smithi033 ceph-mon[35092]: pgmap v854: 97 pgs: 97 active+clean; 583 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-09-12T03:05:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:09 smithi184 ceph-mon[35321]: pgmap v854: 97 pgs: 97 active+clean; 583 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-09-12T03:05:10.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 95092 22.40 MB/sec execute 131 sec latency 56.711 ms 2024-09-12T03:05:10.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:05:10.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:05:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:05:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T03:05:10.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T03:05:10.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T03:05:10.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T03:05:10.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:05:10.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:05:10.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:05:10.786 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi184 to 176.0M 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi033 to 177.0M 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi033 to 185700659: error parsing value: Value '185700659' is below minimum 939524096 2024-09-12T03:05:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi184 to 184596070: error parsing value: Value '184596070' is below minimum 939524096 2024-09-12T03:05:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-12T03:05:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:05:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:05:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:05:11.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 95921 22.55 MB/sec execute 132 sec latency 63.551 ms 2024-09-12T03:05:11.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/1332361861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 msgr2=0x7fcb281021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/1332361861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb281021b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009900 tx=0x7fcb1802a410 comp rx=0 tx=0).stop 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/1332361861 shutdown_connections 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/1332361861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28106f00 0x7fcb2810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/1332361861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28102780 0x7fcb281067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/1332361861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb281021b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/1332361861 >> 172.21.15.16:0/1332361861 conn(0x7fcb280fd8b0 msgr2=0x7fcb280ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/1332361861 shutdown_connections 2024-09-12T03:05:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.313+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/1332361861 wait complete. 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 Processor -- start 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 -- start start 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 0x7fcb28074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 0x7fcb28074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb28111f60 con 0x7fcb28101dd0 2024-09-12T03:05:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb28111de0 con 0x7fcb28102780 2024-09-12T03:05:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2c9c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281120e0 con 0x7fcb28106f00 2024-09-12T03:05:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.314+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 0x7fcb28074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43770/0 (socket says 172.21.15.16:43770) 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 -- 172.21.15.16:0/4283076926 learned_addr learned my addr 172.21.15.16:0/4283076926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 0x7fcb28074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 -- 172.21.15.16:0/4283076926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 msgr2=0x7fcb28074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 0x7fcb28074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 -- 172.21.15.16:0/4283076926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 msgr2=0x7fcb28074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 0x7fcb28074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 -- 172.21.15.16:0/4283076926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb281a6990 con 0x7fcb28101dd0 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 0x7fcb28074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 0x7fcb28074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2659c640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fcb18006010 tx=0x7fcb18004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb177fe640 1 -- 172.21.15.16:0/4283076926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1803f030 con 0x7fcb28101dd0 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb177fe640 1 -- 172.21.15.16:0/4283076926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb18004440 con 0x7fcb28101dd0 2024-09-12T03:05:11.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb177fe640 1 -- 172.21.15.16:0/4283076926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18035070 con 0x7fcb28101dd0 2024-09-12T03:05:11.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.315+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb281a6c20 con 0x7fcb28101dd0 2024-09-12T03:05:11.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.316+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb281a7100 con 0x7fcb28101dd0 2024-09-12T03:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 Processor -- start 2024-09-12T03:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 -- start start 2024-09-12T03:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be74090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be74a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e99bdda2f0 con 0x55e99be75b40 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e99bd81cf0 con 0x55e99be745d0 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.319+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e99bdd62b0 con 0x55e99be7f370 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be74090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb177fe640 1 -- 172.21.15.16:0/4283076926 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcb18042030 con 0x7fcb28101dd0 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34542/0 (socket says 172.21.15.16:34542) 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 -- 172.21.15.16:0/4027094841 learned_addr learned my addr 172.21.15.16:0/4027094841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 -- 172.21.15.16:0/4027094841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 msgr2=0x55e99be74a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb2659c640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be74a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be74a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 -- 172.21.15.16:0/4027094841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 msgr2=0x55e99be74090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be74090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb25d9b640 1 -- 172.21.15.16:0/4027094841 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e99bd663a0 con 0x55e99be7f370 2024-09-12T03:05:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.320+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be74090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.321+0000 7fcb2659c640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be74a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.321+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0114f0 tx=0x7fcb1c0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9dc70ab87642681 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.321+0000 7fcb14ff9640 1 -- 172.21.15.16:0/4027094841 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c004020 con 0x55e99be7f370 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.321+0000 7fcb14ff9640 1 -- 172.21.15.16:0/4027094841 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb1c0041c0 con 0x55e99be7f370 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.321+0000 7fcb14ff9640 1 -- 172.21.15.16:0/4027094841 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c00f950 con 0x55e99be7f370 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.322+0000 7fcb177fe640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 0x7fcb100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.322+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 0x7fcb100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.322+0000 7fcb177fe640 1 -- 172.21.15.16:0/4283076926 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb180ef020 con 0x7fcb28101dd0 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.322+0000 7fcb2eda0740 1 -- 172.21.15.16:0/4027094841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 msgr2=0x55e99be81760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.322+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0114f0 tx=0x7fcb1c0119b0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 -- 172.21.15.16:0/4027094841 shutdown_connections 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be81760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be74a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/4027094841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be74090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 -- 172.21.15.16:0/4027094841 >> 172.21.15.16:0/4027094841 conn(0x55e99be3d310 msgr2=0x55e99be70c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 -- 172.21.15.16:0/4027094841 shutdown_connections 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb2eda0740 1 -- 172.21.15.16:0/4027094841 wait complete. 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.323+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 0x7fcb100a5040 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009fd0 tx=0x7fcb20009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.324+0000 7fcb2eda0740 1 -- start start 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.324+0000 7fcb2eda0740 1 Processor -- start 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.324+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 0x55e99be7ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be7bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e99be82070 con 0x55e99be745d0 2024-09-12T03:05:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e99bd95060 con 0x55e99be7f370 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e99be821f0 con 0x55e99be75b40 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2eda0740 10 client.0 _ll_register_callbacks cb 0x55e99bae88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be7bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb26d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb26d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54146/0 (socket says 172.21.15.16:54146) 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb26d9d640 1 -- 172.21.15.16:0/3242015846 learned_addr learned my addr 172.21.15.16:0/3242015846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.325+0000 7fcb2659c640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 0x55e99be7ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 -- 172.21.15.16:0/3242015846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 msgr2=0x55e99be7bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be7bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 -- 172.21.15.16:0/3242015846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 msgr2=0x55e99be7ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 0x55e99be7ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 -- 172.21.15.16:0/3242015846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e99bb4b3c0 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be7bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb2659c640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 0x55e99be7ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000f910 tx=0x7fcb20004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5899965c347fed4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.326+0000 7fcb0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20011d10 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb2001f420 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb1c0041c0 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcb20011eb0 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb2eda0740 10 client.28763 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.327+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e99be962b0 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb1c0041c0 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcb20027070 con 0x55e99be7f370 2024-09-12T03:05:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 10 client.28763 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 10 client.28763 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 20 client.28763 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99be976c0 0x55e99be99b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 4 client.28763 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e99bde7860 con 0x55e99be976c0 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.328+0000 7fcb2659c640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99be976c0 0x55e99be99b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.329+0000 7fcb2659c640 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99be976c0 0x55e99be99b60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2056434f7d61864d server_cookie=1fc4d488e7a7ecd7 in_seq=0 out_seq=0 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.329+0000 7fcb0dffb640 10 client.28763 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb0dffb640 1 -- 172.21.15.16:0/3242015846 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e99bde7860 con 0x55e99be976c0 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb0dffb640 10 client.28763 handle_command_reply: tid=0 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 client.28763 shutdown 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 msgr2=0x55e99be7c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000f910 tx=0x7fcb20004ab0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 shutdown_connections 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99be976c0 0x55e99be99b60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be7f370 0x55e99be7c520 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be7bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/3242015846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be745d0 0x55e99be7ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb0dffb640 0 client.28763 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 >> 172.21.15.16:0/3242015846 conn(0x55e99be3d310 msgr2=0x55e99be76930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 shutdown_connections 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.331+0000 7fcb2eda0740 1 -- 172.21.15.16:0/3242015846 wait complete. 2024-09-12T03:05:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.332+0000 7fcb2eda0740 20 client.28763 trim_cache size 0 max 16384 2024-09-12T03:05:11.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 Processor -- start 2024-09-12T03:05:11.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 -- start start 2024-09-12T03:05:11.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be6fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be77120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281ab350 con 0x55e99be75b40 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281ace90 con 0x55e99be745d0 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281aba20 con 0x55e99be7f370 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be77120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb26d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be6fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb26d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34560/0 (socket says 172.21.15.16:34560) 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be6fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-09-12T03:05:11.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.347+0000 7fcb26d9d640 1 -- 172.21.15.16:0/2935658647 learned_addr learned my addr 172.21.15.16:0/2935658647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 -- 172.21.15.16:0/2935658647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 msgr2=0x55e99be6fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be6fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 -- 172.21.15.16:0/2935658647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 msgr2=0x55e99be77120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be77120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 -- 172.21.15.16:0/2935658647 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e99be7f060 con 0x55e99be7f370 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be77120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f80 tx=0x7fcb20005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6dba85962a79131 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.348+0000 7fcb0ffff640 1 -- 172.21.15.16:0/2935658647 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20038070 con 0x55e99be7f370 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb0ffff640 1 -- 172.21.15.16:0/2935658647 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb20012430 con 0x55e99be7f370 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb0ffff640 1 -- 172.21.15.16:0/2935658647 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb200298a0 con 0x55e99be7f370 2024-09-12T03:05:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2935658647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 msgr2=0x55e99be77660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f80 tx=0x7fcb20005c50 comp rx=0 tx=0).stop 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2935658647 shutdown_connections 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be7f370 0x55e99be77660 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be75b40 0x55e99be77120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2935658647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be6fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2935658647 >> 172.21.15.16:0/2935658647 conn(0x55e99be3d310 msgr2=0x55e99be6f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2935658647 shutdown_connections 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.349+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2935658647 wait complete. 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.350+0000 7fcb2eda0740 1 -- start start 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.350+0000 7fcb2eda0740 1 Processor -- start 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.350+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.350+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be79710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2eda0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 0x55e99be79c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281ac7c0 con 0x55e99be7f370 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281ac0f0 con 0x55e99be745d0 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be79710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2eda0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb281a9330 con 0x55e99be75b40 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54156/0 (socket says 172.21.15.16:54156) 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb25d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be79710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34574/0 (socket says 172.21.15.16:34574) 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 -- 172.21.15.16:0/2189711993 learned_addr learned my addr 172.21.15.16:0/2189711993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2eda0740 10 client.0 _ll_register_callbacks cb 0x7fcb20012840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 0x55e99be79c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 -- 172.21.15.16:0/2189711993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 msgr2=0x55e99be79710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be79710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 -- 172.21.15.16:0/2189711993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 msgr2=0x55e99be79c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 0x55e99be79c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 -- 172.21.15.16:0/2189711993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e99bf01650 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb26d9d640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 0x55e99be79c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb2659c640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcb180b7da0 tx=0x7fcb180bb940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d6b2dce197770c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18049040 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.351+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb180ecbb0 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb20012430 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcb18003cd0 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb2eda0740 10 client.28769 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e99bf02990 con 0x55e99be745d0 2024-09-12T03:05:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb20012430 con 0x55e99be745d0 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.352+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcb1804c060 con 0x55e99be745d0 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 10 client.28769 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 10 client.28769 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 20 client.28769 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99bf03040 0x55e99bf05460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 4 client.28769 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e99bde1d70 con 0x55e99bf03040 2024-09-12T03:05:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99bf03040 0x55e99bf05460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb0dffb640 10 client.28769 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.353+0000 7fcb25d9b640 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99bf03040 0x55e99bf05460 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=751fcdc7a180528d server_cookie=2f04042ca54df63 in_seq=0 out_seq=0 2024-09-12T03:05:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb0dffb640 1 -- 172.21.15.16:0/2189711993 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e99bde1d70 con 0x55e99bf03040 2024-09-12T03:05:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb0dffb640 10 client.28769 handle_command_reply: tid=0 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 client.28769 shutdown 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 msgr2=0x55e99be791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcb180b7da0 tx=0x7fcb180bb940 comp rx=0 tx=0).stop 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 shutdown_connections 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55e99bf03040 0x55e99bf05460 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e99be7f370 0x55e99be79c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb0dffb640 0 client.28769 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e99be75b40 0x55e99be79710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 --2- 172.21.15.16:0/2189711993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e99be745d0 0x55e99be791d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 >> 172.21.15.16:0/2189711993 conn(0x55e99be3d310 msgr2=0x55e99be6fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 shutdown_connections 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.354+0000 7fcb2eda0740 1 -- 172.21.15.16:0/2189711993 wait complete. 2024-09-12T03:05:11.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2eda0740 20 client.28769 trim_cache size 0 max 16384 2024-09-12T03:05:11.361 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:11.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 msgr2=0x7fcb100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 0x7fcb100a5040 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009fd0 tx=0x7fcb20009500 comp rx=0 tx=0).stop 2024-09-12T03:05:11.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 msgr2=0x7fcb280744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:11.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fcb18006010 tx=0x7fcb18004040 comp rx=0 tx=0).stop 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 shutdown_connections 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb28106f00 0x7fcb28074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb28102780 0x7fcb28074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcb100a2b80 0x7fcb100a5040 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.355+0000 7fcb2c9c7640 1 --2- 172.21.15.16:0/4283076926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb28101dd0 0x7fcb280744f0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.356+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 >> 172.21.15.16:0/4283076926 conn(0x7fcb280fd8b0 msgr2=0x7fcb280feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.356+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 shutdown_connections 2024-09-12T03:05:11.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:11.356+0000 7fcb2c9c7640 1 -- 172.21.15.16:0/4283076926 wait complete. 2024-09-12T03:05:12.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 96814 22.72 MB/sec execute 133 sec latency 39.766 ms 2024-09-12T03:05:12.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:05:12.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:12 smithi016 ceph-mon[30925]: pgmap v855: 97 pgs: 97 active+clean; 595 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 455 op/s 2024-09-12T03:05:12.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:12 smithi033 ceph-mon[35092]: pgmap v855: 97 pgs: 97 active+clean; 595 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 455 op/s 2024-09-12T03:05:12.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:12 smithi184 ceph-mon[35321]: pgmap v855: 97 pgs: 97 active+clean; 595 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 455 op/s 2024-09-12T03:05:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:13.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 97528 22.89 MB/sec execute 134 sec latency 82.446 ms 2024-09-12T03:05:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 -- 172.21.15.16:0/1519285844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8100100 msgr2=0x7f3ba8100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 --2- 172.21.15.16:0/1519285844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8100100 0x7f3ba8100520 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009900 tx=0x7f3b9402a3f0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 -- 172.21.15.16:0/1519285844 shutdown_connections 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 --2- 172.21.15.16:0/1519285844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba8110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 --2- 172.21.15.16:0/1519285844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8100100 0x7f3ba8100520 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 --2- 172.21.15.16:0/1519285844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8102a50 0x7f3ba8102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 -- 172.21.15.16:0/1519285844 >> 172.21.15.16:0/1519285844 conn(0x7f3ba8077230 msgr2=0x7f3ba80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 -- 172.21.15.16:0/1519285844 shutdown_connections 2024-09-12T03:05:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.094+0000 7f3baed16640 1 -- 172.21.15.16:0/1519285844 wait complete. 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 Processor -- start 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 -- start start 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 0x7f3ba810f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba810f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 0x7f3ba81a4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8104620 con 0x7f3ba8102a50 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81044a0 con 0x7f3ba8100100 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.095+0000 7f3baed16640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81047a0 con 0x7f3ba8100c80 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3bad2b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 0x7f3ba81a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 0x7f3ba810f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3bad2b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 0x7f3ba81a4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43820/0 (socket says 172.21.15.16:43820) 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 0x7f3ba810f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54176/0 (socket says 172.21.15.16:54176) 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3bad2b3640 1 -- 172.21.15.16:0/3949445344 learned_addr learned my addr 172.21.15.16:0/3949445344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba810f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 -- 172.21.15.16:0/3949445344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 msgr2=0x7f3ba810f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 0x7f3ba810f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 -- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 msgr2=0x7f3ba81a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 0x7f3ba81a4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 -- 172.21.15.16:0/3949445344 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba81a4c70 con 0x7f3ba8100c80 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.096+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba810f930 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3b94005940 tx=0x7f3b94004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.097+0000 7f3ba5ffb640 1 -- 172.21.15.16:0/3949445344 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9403f030 con 0x7f3ba8100c80 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.097+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba81a4f00 con 0x7f3ba8100c80 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.097+0000 7f3ba5ffb640 1 -- 172.21.15.16:0/3949445344 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b94003ad0 con 0x7f3ba8100c80 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.097+0000 7f3ba5ffb640 1 -- 172.21.15.16:0/3949445344 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94035070 con 0x7f3ba8100c80 2024-09-12T03:05:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.097+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba81ab5f0 con 0x7f3ba8100c80 2024-09-12T03:05:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3ba5ffb640 1 -- 172.21.15.16:0/3949445344 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3b94042030 con 0x7f3ba8100c80 2024-09-12T03:05:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3bb10ef740 1 Processor -- start 2024-09-12T03:05:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3bb10ef740 1 -- start start 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba12fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.101+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba13c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562aba0952f0 con 0x562aba13a370 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562aba03ccf0 con 0x562aba130b40 2024-09-12T03:05:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562aba0912b0 con 0x562aba12f5d0 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54186/0 (socket says 172.21.15.16:54186) 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 -- 172.21.15.16:0/1467397268 learned_addr learned my addr 172.21.15.16:0/1467397268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba13c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 -- 172.21.15.16:0/1467397268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 msgr2=0x562aba12fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba12fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 -- 172.21.15.16:0/1467397268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 msgr2=0x562aba13c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba13c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3bad2b3640 1 -- 172.21.15.16:0/1467397268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aba0213a0 con 0x562aba130b40 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.102+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba13c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3ba5ffb640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 0x7f3b980a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0011460 tx=0x7f3ba0011860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6e7237b45fb8f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3bacab2640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 0x7f3b980a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3b92ffd640 1 -- 172.21.15.16:0/1467397268 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0012710 con 0x562aba130b40 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3ba5ffb640 1 -- 172.21.15.16:0/3949445344 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3b940ef590 con 0x7f3ba8100c80 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3b92ffd640 1 -- 172.21.15.16:0/1467397268 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba0012d90 con 0x562aba130b40 2024-09-12T03:05:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.103+0000 7f3b92ffd640 1 -- 172.21.15.16:0/1467397268 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00193a0 con 0x562aba130b40 2024-09-12T03:05:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1467397268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 msgr2=0x562aba12f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bacab2640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 0x7f3b980a5040 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x562aba12cac0 tx=0x7f3b9c0097c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0011460 tx=0x7f3ba0011860 comp rx=0 tx=0).stop 2024-09-12T03:05:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1467397268 shutdown_connections 2024-09-12T03:05:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba13c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba12fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1467397268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba12f090 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1467397268 >> 172.21.15.16:0/1467397268 conn(0x562aba0f8310 msgr2=0x562aba12bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1467397268 shutdown_connections 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.104+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1467397268 wait complete. 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.105+0000 7f3bb10ef740 1 -- start start 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.105+0000 7f3bb10ef740 1 Processor -- start 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.105+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba139d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.105+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 0x562aba137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.105+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562aba13d070 con 0x562aba130b40 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba139d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34630/0 (socket says 172.21.15.16:34630) 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bacab2640 1 -- 172.21.15.16:0/2070262028 learned_addr learned my addr 172.21.15.16:0/2070262028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 0x562aba137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562aba050060 con 0x562aba13a370 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562aba13d1f0 con 0x562aba12f5d0 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 -- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 msgr2=0x562aba139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba139d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 -- 172.21.15.16:0/2070262028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 msgr2=0x562aba137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 0x562aba137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3bad2b3640 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ab9e063c0 con 0x562aba13a370 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.106+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 0x562aba137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3bb10ef740 10 client.0 _ll_register_callbacks cb 0x562ab9da38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba0012d90 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba1375d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0011460 tx=0x7f3ba001af70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95bf28aa1b50f19 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3b87fff640 10 client.28781 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3b87fff640 1 -- 172.21.15.16:0/2070262028 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0032050 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3b87fff640 1 -- 172.21.15.16:0/2070262028 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ba0004a40 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3b87fff640 1 -- 172.21.15.16:0/2070262028 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3ba0028070 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3bb10ef740 10 client.28781 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.107+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=39}) -- 0x562aba1488c0 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3b87fff640 1 -- 172.21.15.16:0/2070262028 <== mon.2 v2:172.21.15.33:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3ba000f040 con 0x562aba13a370 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 10 client.28781 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 10 client.28781 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:05:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 20 client.28781 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba14a220 0x562aba14c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 4 client.28781 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562aba0a2860 con 0x562aba14a220 2024-09-12T03:05:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.108+0000 7f3bacab2640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba14a220 0x562aba14c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.109+0000 7f3bacab2640 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba14a220 0x562aba14c6b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8625bb5215c570d9 server_cookie=bb8e4b3407a0269b in_seq=0 out_seq=0 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.109+0000 7f3b87fff640 10 client.28781 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.111+0000 7f3b87fff640 1 -- 172.21.15.16:0/2070262028 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562aba0a2860 con 0x562aba14a220 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.111+0000 7f3b87fff640 10 client.28781 handle_command_reply: tid=0 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.111+0000 7f3bb10ef740 1 client.28781 shutdown 2024-09-12T03:05:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.111+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 msgr2=0x562aba1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba1375d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0011460 tx=0x7f3ba001af70 comp rx=0 tx=0).stop 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 shutdown_connections 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba1375d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba130b40 0x562aba137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba14a220 0x562aba14c6b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/2070262028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 >> 172.21.15.16:0/2070262028 conn(0x562aba0f8310 msgr2=0x562aba131960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3b87fff640 0 client.28781 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 shutdown_connections 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 1 -- 172.21.15.16:0/2070262028 wait complete. 2024-09-12T03:05:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.112+0000 7f3bb10ef740 20 client.28781 trim_cache size 0 max 16384 2024-09-12T03:05:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 Processor -- start 2024-09-12T03:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 -- start start 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba152500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b3b00 con 0x562aba12f5d0 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b5640 con 0x562aba13a370 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b41d0 con 0x562aba130b40 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34642/0 (socket says 172.21.15.16:34642) 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43844/0 (socket says 172.21.15.16:43844) 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3ba7fff640 1 -- 172.21.15.16:0/823397538 learned_addr learned my addr 172.21.15.16:0/823397538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bad2b3640 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bacab2640 1 -- 172.21.15.16:0/823397538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 msgr2=0x562aba151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bacab2640 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.132+0000 7f3bacab2640 1 -- 172.21.15.16:0/823397538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 msgr2=0x562aba152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3bacab2640 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3bacab2640 1 -- 172.21.15.16:0/823397538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aba14cd30 con 0x562aba12f5d0 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3bacab2640 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00092f0 tx=0x7f3b9c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6199cef1eec2eb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3b927fc640 1 -- 172.21.15.16:0/823397538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c003ac0 con 0x562aba12f5d0 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3b927fc640 1 -- 172.21.15.16:0/823397538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b9c005500 con 0x562aba12f5d0 2024-09-12T03:05:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.133+0000 7f3b927fc640 1 -- 172.21.15.16:0/823397538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004d00 con 0x562aba12f5d0 2024-09-12T03:05:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- 172.21.15.16:0/823397538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 msgr2=0x562aba146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00092f0 tx=0x7f3b9c014000 comp rx=0 tx=0).stop 2024-09-12T03:05:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- 172.21.15.16:0/823397538 shutdown_connections 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba13a370 0x562aba152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba130b40 0x562aba151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/823397538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba12f5d0 0x562aba146340 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- 172.21.15.16:0/823397538 >> 172.21.15.16:0/823397538 conn(0x562aba0f8310 msgr2=0x562aba139080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- 172.21.15.16:0/823397538 shutdown_connections 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- 172.21.15.16:0/823397538 wait complete. 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 -- start start 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.134+0000 7f3bb10ef740 1 Processor -- start 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba13df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba1b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b4f70 con 0x562aba13a370 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b48a0 con 0x562aba130b40 2024-09-12T03:05:13.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b1ae0 con 0x562aba12f5d0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bb10ef740 10 client.0 _ll_register_callbacks cb 0x7f3ba0009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bad2b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba1b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34644/0 (socket says 172.21.15.16:34644) 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba13df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 -- 172.21.15.16:0/1563950388 learned_addr learned my addr 172.21.15.16:0/1563950388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 -- 172.21.15.16:0/1563950388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 msgr2=0x562aba13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba13df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 -- 172.21.15.16:0/1563950388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 msgr2=0x562aba1b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba1b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 -- 172.21.15.16:0/1563950388 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aba1b34f0 con 0x562aba12f5d0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3bacab2640 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d746ca1673e35b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:13.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.135+0000 7f3b87fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c049070 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b9c005500 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3b9c003cd0 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3b9c055050 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 10 client.28696 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562aba1b47f0 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b9c005500 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3b9c0285f0 con 0x562aba12f5d0 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 10 client.28696 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 10 client.28696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 20 client.28696 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba1b4e60 0x562aba1b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:13.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 4 client.28696 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.136+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562aba09cd70 con 0x562aba1b4e60 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.137+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba1b4e60 0x562aba1b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.137+0000 7f3ba7fff640 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba1b4e60 0x562aba1b7250 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45e4ce044990e772 server_cookie=28e73d96a75c7e42 in_seq=0 out_seq=0 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.137+0000 7f3b87fff640 10 client.28696 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3b87fff640 1 -- 172.21.15.16:0/1563950388 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562aba09cd70 con 0x562aba1b4e60 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3b87fff640 10 client.28696 handle_command_reply: tid=0 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 client.28696 shutdown 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 msgr2=0x562aba13da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c0046c0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 shutdown_connections 2024-09-12T03:05:13.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562aba13a370 0x562aba1b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562aba130b40 0x562aba13df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562aba1b4e60 0x562aba1b7250 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 --2- 172.21.15.16:0/1563950388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562aba12f5d0 0x562aba13da10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 >> 172.21.15.16:0/1563950388 conn(0x562aba0f8310 msgr2=0x562aba145330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.138+0000 7f3b87fff640 0 client.28696 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.139+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 shutdown_connections 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.139+0000 7f3bb10ef740 1 -- 172.21.15.16:0/1563950388 wait complete. 2024-09-12T03:05:13.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.139+0000 7f3bb10ef740 20 client.28696 trim_cache size 0 max 16384 2024-09-12T03:05:13.145 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:13.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 msgr2=0x7f3b980a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 0x7f3b980a5040 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x562aba12cac0 tx=0x7f3b9c0097c0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 msgr2=0x7f3ba810f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba810f930 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3b94005940 tx=0x7f3b94004270 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 shutdown_connections 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3b980a2b80 0x7f3b980a5040 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ba8102a50 0x7f3ba81a4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ba8100c80 0x7f3ba810f930 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 --2- 172.21.15.16:0/3949445344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ba8100100 0x7f3ba810f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 >> 172.21.15.16:0/3949445344 conn(0x7f3ba8077230 msgr2=0x7f3ba80fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 shutdown_connections 2024-09-12T03:05:13.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:13.140+0000 7f3baed16640 1 -- 172.21.15.16:0/3949445344 wait complete. 2024-09-12T03:05:13.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:05:14.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 98211 23.04 MB/sec execute 135 sec latency 46.408 ms 2024-09-12T03:05:14.307 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:14 smithi016 ceph-mon[30925]: pgmap v856: 97 pgs: 97 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 541 op/s 2024-09-12T03:05:14.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:14 smithi033 ceph-mon[35092]: pgmap v856: 97 pgs: 97 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 541 op/s 2024-09-12T03:05:14.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:14 smithi184 ceph-mon[35321]: pgmap v856: 97 pgs: 97 active+clean; 600 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 541 op/s 2024-09-12T03:05:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 -- 172.21.15.16:0/3238340171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 msgr2=0x7f9fbc10c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 --2- 172.21.15.16:0/3238340171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc10c9f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0009900 tx=0x7f9fb002a3f0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 -- 172.21.15.16:0/3238340171 shutdown_connections 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 --2- 172.21.15.16:0/3238340171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 0x7f9fbc119410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 --2- 172.21.15.16:0/3238340171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc10c9f0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 --2- 172.21.15.16:0/3238340171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 0x7f9fbc112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 -- 172.21.15.16:0/3238340171 >> 172.21.15.16:0/3238340171 conn(0x7f9fbc06cdc0 msgr2=0x7f9fbc06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 -- 172.21.15.16:0/3238340171 shutdown_connections 2024-09-12T03:05:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.850+0000 7f9fc2062640 1 -- 172.21.15.16:0/3238340171 wait complete. 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.851+0000 7f9fc2062640 1 Processor -- start 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.851+0000 7f9fc2062640 1 -- start start 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.851+0000 7f9fc2062640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.851+0000 7f9fc2062640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 0x7f9fbc1a3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.851+0000 7f9fc2062640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 0x7f9fbc1a7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fc2062640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc11c9a0 con 0x7f9fbc10c590 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fc2062640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc11c820 con 0x7f9fbc112640 2024-09-12T03:05:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fc2062640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc11cb20 con 0x7f9fbc10cf30 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43880/0 (socket says 172.21.15.16:43880) 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2534558718 learned_addr learned my addr 172.21.15.16:0/2534558718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 0x7f9fbc1a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 0x7f9fbc1a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2534558718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 msgr2=0x7f9fbc1a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 0x7f9fbc1a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2534558718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 msgr2=0x7f9fbc1a7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 0x7f9fbc1a7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2534558718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc1a7e00 con 0x7f9fbc10c590 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.852+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00d950 tx=0x7f9fac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.853+0000 7f9fb8ff9640 1 -- 172.21.15.16:0/2534558718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac017030 con 0x7f9fbc10c590 2024-09-12T03:05:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.853+0000 7f9fb8ff9640 1 -- 172.21.15.16:0/2534558718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fac004020 con 0x7f9fbc10c590 2024-09-12T03:05:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.853+0000 7f9fb8ff9640 1 -- 172.21.15.16:0/2534558718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0047a0 con 0x7f9fbc10c590 2024-09-12T03:05:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.854+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fbc1a80f0 con 0x7f9fbc10c590 2024-09-12T03:05:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.854+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fbc1af9d0 con 0x7f9fbc10c590 2024-09-12T03:05:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.857+0000 7f9fb8ff9640 1 -- 172.21.15.16:0/2534558718 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9fac0041d0 con 0x7f9fbc10c590 2024-09-12T03:05:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.857+0000 7f9fb8ff9640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 0x7f9fa40a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.858+0000 7f9fb8ff9640 1 -- 172.21.15.16:0/2534558718 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9fac0cdef0 con 0x7f9fbc10c590 2024-09-12T03:05:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.858+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 0x7f9fa40a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.858+0000 7f9fc443b740 1 Processor -- start 2024-09-12T03:05:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.858+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 0x7f9fa40a5190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0009230 tx=0x7f9fb00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 -- start start 2024-09-12T03:05:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b85a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b92760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f01aeb2f0 con 0x558f01b86b40 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43896/0 (socket says 172.21.15.16:43896) 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/379931362 learned_addr learned my addr 172.21.15.16:0/379931362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558f01a92cf0 con 0x558f01b90370 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558f01ae72b0 con 0x558f01b855d0 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b85a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.859+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b92760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/379931362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 msgr2=0x558f01b85a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b85a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/379931362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 msgr2=0x558f01b92760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b92760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/379931362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f01a773a0 con 0x558f01b86b40 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0a2f20 tx=0x7f9fac005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c465a376cbd7f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fa1ffb640 1 -- 172.21.15.16:0/379931362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac09eb90 con 0x558f01b86b40 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fa1ffb640 1 -- 172.21.15.16:0/379931362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fac0b3070 con 0x558f01b86b40 2024-09-12T03:05:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.860+0000 7f9fa1ffb640 1 -- 172.21.15.16:0/379931362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac099f40 con 0x558f01b86b40 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 msgr2=0x558f01b85090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0a2f20 tx=0x7f9fac005dc0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 shutdown_connections 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b92760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b85a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 --2- 172.21.15.16:0/379931362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b85090 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 >> 172.21.15.16:0/379931362 conn(0x558f01b4e310 msgr2=0x558f01b81c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 shutdown_connections 2024-09-12T03:05:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- 172.21.15.16:0/379931362 wait complete. 2024-09-12T03:05:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.861+0000 7f9fc443b740 1 -- start start 2024-09-12T03:05:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 Processor -- start 2024-09-12T03:05:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b8d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b8d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558f01b93010 con 0x558f01b86b40 2024-09-12T03:05:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558f01aa6060 con 0x558f01b90370 2024-09-12T03:05:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558f01b93190 con 0x558f01b855d0 2024-09-12T03:05:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b8d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34674/0 (socket says 172.21.15.16:34674) 2024-09-12T03:05:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2996942076 learned_addr learned my addr 172.21.15.16:0/2996942076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.862+0000 7f9fc443b740 10 client.0 _ll_register_callbacks cb 0x558f017f98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 msgr2=0x558f01b8d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b8d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2996942076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 msgr2=0x558f01b8d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b8d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 -- 172.21.15.16:0/2996942076 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f0185c3c0 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0ce210 tx=0x7f9fac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3575a212a5f359e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.863+0000 7f9f9affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0b9070 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fac0b3070 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9fac099ca0 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f9fac099f20 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9fc443b740 10 client.28708 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558f01b93010 con 0x558f01b855d0 2024-09-12T03:05:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.864+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fac014040 con 0x558f01b855d0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f9fac099860 con 0x558f01b855d0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 10 client.28708 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 10 client.28708 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 20 client.28708 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01b9fca0 0x558f01ba2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 4 client.28708 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fac0b3070 con 0x558f01b9fca0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01b9fca0 0x558f01ba2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.865+0000 7f9fbaffd640 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01b9fca0 0x558f01ba2090 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac14b582d6e13e92 server_cookie=65f5b88536751646 in_seq=0 out_seq=0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.866+0000 7f9f9affd640 10 client.28708 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9f9affd640 1 -- 172.21.15.16:0/2996942076 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f9fac0b3070 con 0x558f01b9fca0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9f9affd640 10 client.28708 handle_command_reply: tid=0 2024-09-12T03:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 client.28708 shutdown 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 msgr2=0x558f01b8fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0ce210 tx=0x7f9fac004730 comp rx=0 tx=0).stop 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 shutdown_connections 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01b9fca0 0x558f01ba2090 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b8d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b8d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 --2- 172.21.15.16:0/2996942076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b855d0 0x558f01b8fda0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9f9affd640 0 client.28708 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 >> 172.21.15.16:0/2996942076 conn(0x558f01b4e310 msgr2=0x558f01b87990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 shutdown_connections 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 1 -- 172.21.15.16:0/2996942076 wait complete. 2024-09-12T03:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.867+0000 7f9fc443b740 20 client.28708 trim_cache size 0 max 16384 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.883+0000 7f9fc443b740 1 Processor -- start 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.883+0000 7f9fc443b740 1 -- start start 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b855d0 0x558f01b8c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b88120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b8060 con 0x558f01b86b40 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b9ba0 con 0x558f01b855d0 2024-09-12T03:05:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fc443b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b8730 con 0x558f01b90370 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b88120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34688/0 (socket says 172.21.15.16:34688) 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 -- 172.21.15.16:0/4109017706 learned_addr learned my addr 172.21.15.16:0/4109017706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b855d0 0x558f01b8c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 -- 172.21.15.16:0/4109017706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b855d0 msgr2=0x558f01b8c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b855d0 0x558f01b8c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 -- 172.21.15.16:0/4109017706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 msgr2=0x558f01b88120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b88120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.884+0000 7f9fbbfff640 1 -- 172.21.15.16:0/4109017706 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f01b81510 con 0x558f01b90370 2024-09-12T03:05:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.885+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009fc0 tx=0x7f9fb40128e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567e86dd3f97b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.885+0000 7f9fa17fa640 1 -- 172.21.15.16:0/4109017706 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4012190 con 0x558f01b90370 2024-09-12T03:05:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.885+0000 7f9fa17fa640 1 -- 172.21.15.16:0/4109017706 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fb401c940 con 0x558f01b90370 2024-09-12T03:05:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.885+0000 7f9fa17fa640 1 -- 172.21.15.16:0/4109017706 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4010990 con 0x558f01b90370 2024-09-12T03:05:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- 172.21.15.16:0/4109017706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 msgr2=0x558f01b88660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009fc0 tx=0x7f9fb40128e0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- 172.21.15.16:0/4109017706 shutdown_connections 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b90370 0x558f01b88660 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b86b40 0x558f01b88120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 --2- 172.21.15.16:0/4109017706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b855d0 0x558f01b8c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- 172.21.15.16:0/4109017706 >> 172.21.15.16:0/4109017706 conn(0x558f01b4e310 msgr2=0x558f01b8c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- 172.21.15.16:0/4109017706 shutdown_connections 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- 172.21.15.16:0/4109017706 wait complete. 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 -- start start 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.886+0000 7f9fc443b740 1 Processor -- start 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b86b40 0x558f01b94660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b94d0 con 0x558f01b855d0 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b8e00 con 0x558f01b90370 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b6040 con 0x558f01b86b40 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fc443b740 10 client.0 _ll_register_callbacks cb 0x7f9fb0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b86b40 0x558f01b94660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54244/0 (socket says 172.21.15.16:54244) 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 -- 172.21.15.16:0/1197310540 learned_addr learned my addr 172.21.15.16:0/1197310540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43918/0 (socket says 172.21.15.16:43918) 2024-09-12T03:05:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 -- 172.21.15.16:0/1197310540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b86b40 msgr2=0x558f01b94660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b86b40 0x558f01b94660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 -- 172.21.15.16:0/1197310540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 msgr2=0x558f01b94120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 -- 172.21.15.16:0/1197310540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f01c09710 con 0x558f01b90370 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9fbbfff640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40123c0 tx=0x7f9fb401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ecf4f9ffdc8393 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.887+0000 7f9f9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4029060 con 0x558f01b90370 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fb401c940 con 0x558f01b90370 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9fb401cd00 con 0x558f01b90370 2024-09-12T03:05:14.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f9fb4010430 con 0x558f01b90370 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 10 client.28802 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558f01c0a990 con 0x558f01b90370 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fb401c940 con 0x558f01b90370 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f9fb4023030 con 0x558f01b90370 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 10 client.28802 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 10 client.28802 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 20 client.28802 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.888+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01c0b000 0x558f01c0d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9fc443b740 4 client.28802 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f01af8860 con 0x558f01c0b000 2024-09-12T03:05:14.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01c0b000 0x558f01c0d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9fbb7fe640 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01c0b000 0x558f01c0d3f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a7a46e2799f2441 server_cookie=cb2b8e3804612514 in_seq=0 out_seq=0 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9f9affd640 10 client.28802 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9f9affd640 1 -- 172.21.15.16:0/1197310540 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f01af8860 con 0x558f01c0b000 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9f9affd640 10 client.28802 handle_command_reply: tid=0 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.889+0000 7f9fc443b740 1 client.28802 shutdown 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 msgr2=0x558f01b94ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40123c0 tx=0x7f9fb401c8b0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 shutdown_connections 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558f01c0b000 0x558f01c0d3f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558f01b90370 0x558f01b94ba0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558f01b86b40 0x558f01b94660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9f9affd640 0 client.28802 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 --2- 172.21.15.16:0/1197310540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558f01b855d0 0x558f01b94120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 >> 172.21.15.16:0/1197310540 conn(0x558f01b4e310 msgr2=0x558f01b804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 shutdown_connections 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 1 -- 172.21.15.16:0/1197310540 wait complete. 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.890+0000 7f9fc443b740 20 client.28802 trim_cache size 0 max 16384 2024-09-12T03:05:14.897 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:14.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 msgr2=0x7f9fa40a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 0x7f9fa40a5190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0009230 tx=0x7f9fb00058d0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 msgr2=0x7f9fbc1a2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00d950 tx=0x7f9fac00de10 comp rx=0 tx=0).stop 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 shutdown_connections 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fbc112640 0x7f9fbc1a7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fbc10cf30 0x7f9fbc1a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9fa40a2cd0 0x7f9fa40a5190 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 --2- 172.21.15.16:0/2534558718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc10c590 0x7f9fbc1a2e50 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 >> 172.21.15.16:0/2534558718 conn(0x7f9fbc06cdc0 msgr2=0x7f9fbc10afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 shutdown_connections 2024-09-12T03:05:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:14.891+0000 7f9fc2062640 1 -- 172.21.15.16:0/2534558718 wait complete. 2024-09-12T03:05:15.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 98865 23.09 MB/sec execute 136 sec latency 60.079 ms 2024-09-12T03:05:15.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:05:16.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 99418 23.12 MB/sec execute 137 sec latency 85.721 ms 2024-09-12T03:05:16.018 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:16 smithi016 ceph-mon[30925]: pgmap v857: 97 pgs: 97 active+clean; 602 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 484 op/s 2024-09-12T03:05:16.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:16 smithi033 ceph-mon[35092]: pgmap v857: 97 pgs: 97 active+clean; 602 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 484 op/s 2024-09-12T03:05:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:16 smithi184 ceph-mon[35321]: pgmap v857: 97 pgs: 97 active+clean; 602 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 MiB/s wr, 484 op/s 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 -- 172.21.15.16:0/949056270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570100e20 msgr2=0x7ff570101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 --2- 172.21.15.16:0/949056270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570100e20 0x7ff570101200 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff56002a480 comp rx=0 tx=0).stop 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 -- 172.21.15.16:0/949056270 shutdown_connections 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 --2- 172.21.15.16:0/949056270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570102ef0 0x7ff57010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 --2- 172.21.15.16:0/949056270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102570 0x7ff5701029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 --2- 172.21.15.16:0/949056270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570100e20 0x7ff570101200 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 -- 172.21.15.16:0/949056270 >> 172.21.15.16:0/949056270 conn(0x7ff5700fc7d0 msgr2=0x7ff5700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 -- 172.21.15.16:0/949056270 shutdown_connections 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.572+0000 7ff574889640 1 -- 172.21.15.16:0/949056270 wait complete. 2024-09-12T03:05:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 Processor -- start 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 -- start start 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570100e20 0x7ff570103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 0x7ff570103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 0x7ff5701041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff570110fd0 con 0x7ff570102570 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff570110e50 con 0x7ff570100e20 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff574889640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff570111150 con 0x7ff570102ef0 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 0x7ff570103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 0x7ff570103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46992/0 (socket says 172.21.15.16:46992) 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff56dd9b640 1 -- 172.21.15.16:0/2401988866 learned_addr learned my addr 172.21.15.16:0/2401988866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 0x7ff5701041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.573+0000 7ff56e59c640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570100e20 0x7ff570103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 -- 172.21.15.16:0/2401988866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570100e20 msgr2=0x7ff570103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570100e20 0x7ff570103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 -- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 msgr2=0x7ff570103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 0x7ff570103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 -- 172.21.15.16:0/2401988866 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5701a68e0 con 0x7ff570102ef0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 0x7ff5701041d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff56800cab0 tx=0x7ff56800cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff55f7fe640 1 -- 172.21.15.16:0/2401988866 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56800ae20 con 0x7ff570102ef0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5701a6bd0 con 0x7ff570102ef0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff55f7fe640 1 -- 172.21.15.16:0/2401988866 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff56800f930 con 0x7ff570102ef0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff55f7fe640 1 -- 172.21.15.16:0/2401988866 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568018b30 con 0x7ff570102ef0 2024-09-12T03:05:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.574+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5701a7110 con 0x7ff570102ef0 2024-09-12T03:05:16.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.577+0000 7ff55f7fe640 1 -- 172.21.15.16:0/2401988866 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff56800fae0 con 0x7ff570102ef0 2024-09-12T03:05:16.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.577+0000 7ff576c62740 1 Processor -- start 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 -- start start 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a7c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a89760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632d49e22f0 con 0x5632d4a7db40 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632d4989cf0 con 0x5632d4a7c5d0 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff576c62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5632d49de2b0 con 0x5632d4a87370 2024-09-12T03:05:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a7c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40348/0 (socket says 172.21.15.16:40348) 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a7c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47008/0 (socket says 172.21.15.16:47008) 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56e59c640 1 -- 172.21.15.16:0/1656725019 learned_addr learned my addr 172.21.15.16:0/1656725019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff55f7fe640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 0x7ff5580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.578+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a89760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 -- 172.21.15.16:0/1656725019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 msgr2=0x5632d4a89760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a89760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff55f7fe640 1 -- 172.21.15.16:0/2401988866 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff5680cf6e0 con 0x7ff570102ef0 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 -- 172.21.15.16:0/1656725019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 msgr2=0x5632d4a7c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a7c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 -- 172.21.15.16:0/1656725019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632d496e3a0 con 0x5632d4a7c5d0 2024-09-12T03:05:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 0x7ff5580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff56e59c640 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff5600040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4685082c79d53 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff55cff9640 1 -- 172.21.15.16:0/1656725019 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560035040 con 0x5632d4a7c5d0 2024-09-12T03:05:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff55cff9640 1 -- 172.21.15.16:0/1656725019 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff560005520 con 0x5632d4a7c5d0 2024-09-12T03:05:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.579+0000 7ff55cff9640 1 -- 172.21.15.16:0/1656725019 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560003850 con 0x5632d4a7c5d0 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 -- 172.21.15.16:0/1656725019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 msgr2=0x5632d4a7ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff5600040f0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff56ed9d640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 0x7ff5580a5040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff568098350 tx=0x7ff568020f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 -- 172.21.15.16:0/1656725019 shutdown_connections 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a89760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a7ca10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 --2- 172.21.15.16:0/1656725019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a7c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 -- 172.21.15.16:0/1656725019 >> 172.21.15.16:0/1656725019 conn(0x5632d4a45310 msgr2=0x5632d4a78c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 -- 172.21.15.16:0/1656725019 shutdown_connections 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.580+0000 7ff576c62740 1 -- 172.21.15.16:0/1656725019 wait complete. 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 -- start start 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 Processor -- start 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a86d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a84570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632d4a8a010 con 0x5632d4a7db40 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632d499d060 con 0x5632d4a7c5d0 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5632d4a8a190 con 0x5632d4a87370 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff576c62740 10 client.0 _ll_register_callbacks cb 0x5632d46f08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.581+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a86d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47018/0 (socket says 172.21.15.16:47018) 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a86d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40362/0 (socket says 172.21.15.16:40362) 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 -- 172.21.15.16:0/3228208190 learned_addr learned my addr 172.21.15.16:0/3228208190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a84570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 -- 172.21.15.16:0/3228208190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 msgr2=0x5632d4a84570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a84570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 -- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 msgr2=0x5632d4a86d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a86d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 -- 172.21.15.16:0/3228208190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632d47533c0 con 0x5632d4a7db40 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff56e59c640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff5600045d0 tx=0x7ff56002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254b16e3b46bb1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff555ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560052060 con 0x5632d4a7db40 2024-09-12T03:05:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff560004f40 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.582+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff560005520 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff560035050 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 10 client.19518 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5632d4a95a90 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff560005520 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff560003ad0 con 0x5632d4a7db40 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 10 client.19518 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 10 client.19518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 20 client.19518 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4a96de0 0x5632d4a991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 4 client.19518 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.583+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632d49ef860 con 0x5632d4a96de0 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.584+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4a96de0 0x5632d4a991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.584+0000 7ff555ffb640 10 client.19518 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.584+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4a96de0 0x5632d4a991d0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2021b9fe8c14df9 server_cookie=397733b91bed9e5e in_seq=0 out_seq=0 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.585+0000 7ff555ffb640 1 -- 172.21.15.16:0/3228208190 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632d49ef860 con 0x5632d4a96de0 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.585+0000 7ff555ffb640 10 client.19518 handle_command_reply: tid=0 2024-09-12T03:05:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.585+0000 7ff576c62740 1 client.19518 shutdown 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 msgr2=0x5632d4a83fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff5600045d0 tx=0x7ff56002a480 comp rx=0 tx=0).stop 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 shutdown_connections 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4a96de0 0x5632d4a991d0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a87370 0x5632d4a84570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a83fd0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 --2- 172.21.15.16:0/3228208190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a7c5d0 0x5632d4a86d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 >> 172.21.15.16:0/3228208190 conn(0x5632d4a45310 msgr2=0x5632d4a7e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.587+0000 7ff555ffb640 0 client.19518 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.588+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 shutdown_connections 2024-09-12T03:05:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.588+0000 7ff576c62740 1 -- 172.21.15.16:0/3228208190 wait complete. 2024-09-12T03:05:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.588+0000 7ff576c62740 20 client.19518 trim_cache size 0 max 16384 2024-09-12T03:05:16.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.606+0000 7ff576c62740 1 Processor -- start 2024-09-12T03:05:16.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.606+0000 7ff576c62740 1 -- start start 2024-09-12T03:05:16.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7db40 0x5632d4a88760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4a88ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701af550 con 0x5632d4a7c5d0 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b1090 con 0x5632d4a87370 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff576c62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701afc20 con 0x5632d4a7db40 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47020/0 (socket says 172.21.15.16:47020) 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 -- 172.21.15.16:0/1015228191 learned_addr learned my addr 172.21.15.16:0/1015228191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56e59c640 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7db40 0x5632d4a88760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4a88ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 -- 172.21.15.16:0/1015228191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7db40 msgr2=0x5632d4a88760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7db40 0x5632d4a88760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 -- 172.21.15.16:0/1015228191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 msgr2=0x5632d4a88ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4a88ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.607+0000 7ff56dd9b640 1 -- 172.21.15.16:0/1015228191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632d4a79960 con 0x5632d4a7c5d0 2024-09-12T03:05:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.608+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff564005d80 tx=0x7ff5640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dff6aa1bf8fd59 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.608+0000 7ff557fff640 1 -- 172.21.15.16:0/1015228191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564017020 con 0x5632d4a7c5d0 2024-09-12T03:05:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.608+0000 7ff557fff640 1 -- 172.21.15.16:0/1015228191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5640044b0 con 0x5632d4a7c5d0 2024-09-12T03:05:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.608+0000 7ff557fff640 1 -- 172.21.15.16:0/1015228191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5640258e0 con 0x5632d4a7c5d0 2024-09-12T03:05:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- 172.21.15.16:0/1015228191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 msgr2=0x5632d4a893d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff564005d80 tx=0x7ff5640046c0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- 172.21.15.16:0/1015228191 shutdown_connections 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4a88ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7db40 0x5632d4a88760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- 172.21.15.16:0/1015228191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7c5d0 0x5632d4a893d0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- 172.21.15.16:0/1015228191 >> 172.21.15.16:0/1015228191 conn(0x5632d4a45310 msgr2=0x5632d4a73a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- 172.21.15.16:0/1015228191 shutdown_connections 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- 172.21.15.16:0/1015228191 wait complete. 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- start start 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 Processor -- start 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 0x5632d4a97960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4af96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b09c0 con 0x5632d4a7db40 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b02f0 con 0x5632d4a87370 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701ad530 con 0x5632d4a7c5d0 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.609+0000 7ff576c62740 10 client.0 _ll_register_callbacks cb 0x7ff564009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 0x5632d4a97960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47036/0 (socket says 172.21.15.16:47036) 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 0x5632d4a97960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:49624/0 (socket says 172.21.15.16:49624) 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 -- 172.21.15.16:0/999483256 learned_addr learned my addr 172.21.15.16:0/999483256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4af96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 -- 172.21.15.16:0/999483256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 msgr2=0x5632d4a97960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 0x5632d4a97960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 -- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 msgr2=0x5632d4af96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4af96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 -- 172.21.15.16:0/999483256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632d4a78510 con 0x5632d4a7db40 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4af96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff56e59c640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff560004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9d97c52a3bc4d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff555ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56005d030 con 0x5632d4a7db40 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5640044b0 con 0x5632d4a7db40 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.610+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff56004a070 con 0x5632d4a7db40 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff560052040 con 0x5632d4a7db40 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 10 client.19524 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5632d4b00f00 con 0x5632d4a7db40 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5640044b0 con 0x5632d4a7db40 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff560064480 con 0x5632d4a7db40 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 10 client.19524 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 10 client.19524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 20 client.19524 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4b01620 0x5632d4b03a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 4 client.19524 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632d49e9d70 con 0x5632d4b01620 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.611+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4b01620 0x5632d4b03a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff56dd9b640 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4b01620 0x5632d4b03a40 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a567f5be02e47db server_cookie=228983e75a3ffc7f in_seq=0 out_seq=0 2024-09-12T03:05:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff555ffb640 10 client.19524 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff555ffb640 1 -- 172.21.15.16:0/999483256 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632d49e9d70 con 0x5632d4b01620 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff555ffb640 10 client.19524 handle_command_reply: tid=0 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 client.19524 shutdown 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 msgr2=0x5632d4a97ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff560004220 comp rx=0 tx=0).stop 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 shutdown_connections 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5632d4b01620 0x5632d4b03a40 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff555ffb640 0 client.19524 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632d4a87370 0x5632d4af96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632d4a7db40 0x5632d4a97ea0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 --2- 172.21.15.16:0/999483256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5632d4a7c5d0 0x5632d4a97960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 >> 172.21.15.16:0/999483256 conn(0x5632d4a45310 msgr2=0x5632d4a82730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 shutdown_connections 2024-09-12T03:05:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 1 -- 172.21.15.16:0/999483256 wait complete. 2024-09-12T03:05:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.612+0000 7ff576c62740 20 client.19524 trim_cache size 0 max 16384 2024-09-12T03:05:16.619 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 msgr2=0x7ff5580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 0x7ff5580a5040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff568098350 tx=0x7ff568020f70 comp rx=0 tx=0).stop 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 msgr2=0x7ff5701041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 0x7ff5701041d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff56800cab0 tx=0x7ff56800cf70 comp rx=0 tx=0).stop 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 shutdown_connections 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff570102ef0 0x7ff5701041d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff5580a2b80 0x7ff5580a5040 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff570102570 0x7ff570103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 --2- 172.21.15.16:0/2401988866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff570100e20 0x7ff570103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 >> 172.21.15.16:0/2401988866 conn(0x7ff5700fc7d0 msgr2=0x7ff57006a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 shutdown_connections 2024-09-12T03:05:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:16.613+0000 7ff574889640 1 -- 172.21.15.16:0/2401988866 wait complete. 2024-09-12T03:05:17.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 100103 23.23 MB/sec execute 138 sec latency 52.704 ms 2024-09-12T03:05:18.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 100886 23.35 MB/sec execute 139 sec latency 56.283 ms 2024-09-12T03:05:18.317 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:05:18.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:05:18.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:18 smithi016 ceph-mon[30925]: pgmap v858: 97 pgs: 97 active+clean; 612 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-09-12T03:05:18.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:18 smithi033 ceph-mon[35092]: pgmap v858: 97 pgs: 97 active+clean; 612 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-09-12T03:05:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:18 smithi184 ceph-mon[35321]: pgmap v858: 97 pgs: 97 active+clean; 612 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-09-12T03:05:18.607 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:19.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 101639 23.47 MB/sec execute 140 sec latency 79.469 ms 2024-09-12T03:05:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- 172.21.15.16:0/2459002839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 msgr2=0x7fbe3c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/2459002839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c10ec50 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009cd0 tx=0x7fbe3402a8e0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- 172.21.15.16:0/2459002839 shutdown_connections 2024-09-12T03:05:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/2459002839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c10ec50 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/2459002839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 0x7fbe3c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/2459002839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- 172.21.15.16:0/2459002839 >> 172.21.15.16:0/2459002839 conn(0x7fbe3c0fc7d0 msgr2=0x7fbe3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- 172.21.15.16:0/2459002839 shutdown_connections 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- 172.21.15.16:0/2459002839 wait complete. 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 Processor -- start 2024-09-12T03:05:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.117+0000 7fbe42ab0640 1 -- start start 2024-09-12T03:05:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 0x7fbe3c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1120b0 con 0x7fbe3c1082e0 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c111f30 con 0x7fbe3c102db0 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe42ab0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c112230 con 0x7fbe3c103760 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47060/0 (socket says 172.21.15.16:47060) 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 -- 172.21.15.16:0/1948765151 learned_addr learned my addr 172.21.15.16:0/1948765151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 0x7fbe3c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4084c640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 -- 172.21.15.16:0/1948765151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 msgr2=0x7fbe3c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 0x7fbe3c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 -- 172.21.15.16:0/1948765151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 msgr2=0x7fbe3c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 -- 172.21.15.16:0/1948765151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c19d520 con 0x7fbe3c1082e0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4084c640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe4104d640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005cc0 tx=0x7fbe340093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe39ffb640 1 -- 172.21.15.16:0/1948765151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34004040 con 0x7fbe3c1082e0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.118+0000 7fbe39ffb640 1 -- 172.21.15.16:0/1948765151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe340046c0 con 0x7fbe3c1082e0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.119+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3c19d7b0 con 0x7fbe3c1082e0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.119+0000 7fbe39ffb640 1 -- 172.21.15.16:0/1948765151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34030e00 con 0x7fbe3c1082e0 2024-09-12T03:05:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.119+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3c1a50f0 con 0x7fbe3c1082e0 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.121+0000 7fbe39ffb640 1 -- 172.21.15.16:0/1948765151 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbe340305a0 con 0x7fbe3c1082e0 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.121+0000 7fbe44e89740 1 Processor -- start 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.121+0000 7fbe44e89740 1 -- start start 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5dbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5578af532280 con 0x5578af5cd360 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5578af4d9b10 con 0x5578af5dc150 2024-09-12T03:05:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe44e89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5578af52e240 con 0x5578af5cca40 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe4084c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5dbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe4084c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5dbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47062/0 (socket says 172.21.15.16:47062) 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe4084c640 1 -- 172.21.15.16:0/1107433072 learned_addr learned my addr 172.21.15.16:0/1107433072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe39ffb640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 0x7fbe2c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1107433072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 msgr2=0x5578af5de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1107433072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 msgr2=0x5578af5dbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe4104d640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 0x7fbe2c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5dbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1107433072 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578af4e61f0 con 0x5578af5cca40 2024-09-12T03:05:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.122+0000 7fbe39ffb640 1 -- 172.21.15.16:0/1948765151 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbe3402e030 con 0x7fbe3c1082e0 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009710 tx=0x7fbe280096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b1d7121e76b501 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1107433072 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004030 con 0x5578af5cca40 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1107433072 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe280041d0 con 0x5578af5cca40 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe4104d640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 0x7fbe2c0a50c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009ca0 tx=0x7fbe340dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1107433072 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800f6e0 con 0x5578af5cca40 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 -- 172.21.15.16:0/1107433072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 msgr2=0x5578af5cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009710 tx=0x7fbe280096a0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 -- 172.21.15.16:0/1107433072 shutdown_connections 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5dbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1107433072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cce20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.123+0000 7fbe44e89740 1 -- 172.21.15.16:0/1107433072 >> 172.21.15.16:0/1107433072 conn(0x5578af5c90f0 msgr2=0x5578af5c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- 172.21.15.16:0/1107433072 shutdown_connections 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- 172.21.15.16:0/1107433072 wait complete. 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- start start 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 Processor -- start 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5cca40 0x5578af5d90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5d95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5578af5ded20 con 0x5578af5dc150 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5578af4ecf10 con 0x5578af5cca40 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5578af5deea0 con 0x5578af5cd360 2024-09-12T03:05:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.124+0000 7fbe44e89740 10 client.0 _ll_register_callbacks cb 0x5578af4b5a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47074/0 (socket says 172.21.15.16:47074) 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1287461384 learned_addr learned my addr 172.21.15.16:0/1287461384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5cca40 0x5578af5d90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe4084c640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5d95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1287461384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 msgr2=0x5578af5d95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5d95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5cca40 msgr2=0x5578af5d90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5cca40 0x5578af5d90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 -- 172.21.15.16:0/1287461384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578af2a36a0 con 0x5578af5dc150 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fbe280097a0 tx=0x7fbe28003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd8eff71985e03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28003aa0 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe280041d0 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbe2802a040 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbe28036050 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe44e89740 10 client.19536 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.125+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5578af5ea9e0 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe280041d0 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fbe2800fd00 con 0x5578af5dc150 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 10 client.19536 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 10 client.19536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 20 client.19536 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af5ebd30 0x5578af5ee120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 4 client.19536 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.126+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578af53f7f0 con 0x5578af5ebd30 2024-09-12T03:05:19.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.149+0000 7fbe4084c640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af5ebd30 0x5578af5ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.150+0000 7fbe4084c640 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af5ebd30 0x5578af5ee120 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c07dcc5b03f2f93e server_cookie=7059edd57eb4bbb4 in_seq=0 out_seq=0 2024-09-12T03:05:19.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.150+0000 7fbe1bfff640 10 client.19536 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe1bfff640 1 -- 172.21.15.16:0/1287461384 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5578af53f7f0 con 0x5578af5ebd30 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe1bfff640 10 client.19536 handle_command_reply: tid=0 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 client.19536 shutdown 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 msgr2=0x5578af5d9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fbe280097a0 tx=0x7fbe28003e60 comp rx=0 tx=0).stop 2024-09-12T03:05:19.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 shutdown_connections 2024-09-12T03:05:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af5ebd30 0x5578af5ee120 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5dc150 0x5578af5d9b30 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5d95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 --2- 172.21.15.16:0/1287461384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5cca40 0x5578af5d90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe1bfff640 0 client.19536 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.151+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 >> 172.21.15.16:0/1287461384 conn(0x5578af5c90f0 msgr2=0x5578af5cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.152+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 shutdown_connections 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.152+0000 7fbe44e89740 1 -- 172.21.15.16:0/1287461384 wait complete. 2024-09-12T03:05:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.152+0000 7fbe44e89740 20 client.19536 trim_cache size 0 max 16384 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 Processor -- start 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 -- start start 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5f3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:49670/0 (socket says 172.21.15.16:49670) 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe3bfff640 1 -- 172.21.15.16:0/628158050 learned_addr learned my addr 172.21.15.16:0/628158050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1ab390 con 0x5578af5cd360 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1aced0 con 0x5578af5dc150 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.184+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1aba60 con 0x5578af5cca40 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 -- 172.21.15.16:0/628158050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 msgr2=0x5578af5f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 -- 172.21.15.16:0/628158050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 msgr2=0x5578af5f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe4084c640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 -- 172.21.15.16:0/628158050 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578af5cde20 con 0x5578af5cca40 2024-09-12T03:05:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe4084c640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:05:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.185+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000c00 tx=0x7fbe2801f760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a846f46c7398e6aa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.186+0000 7fbe267fc640 1 -- 172.21.15.16:0/628158050 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004060 con 0x5578af5cca40 2024-09-12T03:05:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.186+0000 7fbe267fc640 1 -- 172.21.15.16:0/628158050 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe28004200 con 0x5578af5cca40 2024-09-12T03:05:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.186+0000 7fbe267fc640 1 -- 172.21.15.16:0/628158050 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28005360 con 0x5578af5cca40 2024-09-12T03:05:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.187+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 msgr2=0x5578af5cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.187+0000 7fbe44e89740 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000c00 tx=0x7fbe2801f760 comp rx=0 tx=0).stop 2024-09-12T03:05:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.187+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 shutdown_connections 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.187+0000 7fbe44e89740 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.188+0000 7fbe44e89740 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5cd360 0x5578af5f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.188+0000 7fbe44e89740 1 --2- 172.21.15.16:0/628158050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cca40 0x5578af5cecb0 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000c00 tx=0x7fbe2801f760 comp rx=0 tx=0).stop 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.188+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 >> 172.21.15.16:0/628158050 conn(0x5578af5c90f0 msgr2=0x5578af5ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.188+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 shutdown_connections 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.188+0000 7fbe44e89740 1 -- 172.21.15.16:0/628158050 wait complete. 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.189+0000 7fbe44e89740 1 -- start start 2024-09-12T03:05:19.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.189+0000 7fbe44e89740 1 Processor -- start 2024-09-12T03:05:19.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5e0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5e05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5e0b10 0x5578af6558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5e0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:49684/0 (socket says 172.21.15.16:49684) 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe3bfff640 1 -- 172.21.15.16:0/3057905116 learned_addr learned my addr 172.21.15.16:0/3057905116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1ac800 con 0x5578af5e0b10 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1ac130 con 0x5578af5dc150 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c1a9370 con 0x5578af5cd360 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.190+0000 7fbe4084c640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5e05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe44e89740 10 client.0 _ll_register_callbacks cb 0x7fbe3000b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 -- 172.21.15.16:0/3057905116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 msgr2=0x5578af5e0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5e0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 -- 172.21.15.16:0/3057905116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5e0b10 msgr2=0x5578af6558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5e0b10 0x5578af6558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578af655ee0 con 0x5578af5dc150 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe4084c640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5e05d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5578af5d8940 tx=0x7fbe3000be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b73e969602ea35 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.191+0000 7fbe1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30005020 con 0x5578af5dc150 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbe300051c0 con 0x5578af5dc150 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe28004200 con 0x5578af5dc150 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbe300053c0 con 0x5578af5dc150 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 10 client.28835 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:19.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5578af657170 con 0x5578af5dc150 2024-09-12T03:05:19.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe28004200 con 0x5578af5dc150 2024-09-12T03:05:19.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fbe3001d710 con 0x5578af5dc150 2024-09-12T03:05:19.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 10 client.28835 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:19.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 10 client.28835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.192+0000 7fbe44e89740 20 client.28835 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.193+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af6578a0 0x5578af659cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.193+0000 7fbe44e89740 4 client.28835 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.193+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5578af539d00 con 0x5578af6578a0 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.193+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af6578a0 0x5578af659cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.194+0000 7fbe3bfff640 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af6578a0 0x5578af659cc0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1124d494cda5ab1 server_cookie=25466c3dcfc030d in_seq=0 out_seq=0 2024-09-12T03:05:19.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.194+0000 7fbe1bfff640 10 client.28835 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.196+0000 7fbe1bfff640 1 -- 172.21.15.16:0/3057905116 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5578af539d00 con 0x5578af6578a0 2024-09-12T03:05:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.196+0000 7fbe1bfff640 10 client.28835 handle_command_reply: tid=0 2024-09-12T03:05:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 client.28835 shutdown 2024-09-12T03:05:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 msgr2=0x5578af5e05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5e05d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5578af5d8940 tx=0x7fbe3000be90 comp rx=0 tx=0).stop 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 shutdown_connections 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578af6578a0 0x5578af659cc0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578af5e0b10 0x5578af6558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe1bfff640 0 client.28835 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578af5dc150 0x5578af5e05d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 --2- 172.21.15.16:0/3057905116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578af5cd360 0x5578af5e0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 >> 172.21.15.16:0/3057905116 conn(0x5578af5c90f0 msgr2=0x5578af5d9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 shutdown_connections 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 1 -- 172.21.15.16:0/3057905116 wait complete. 2024-09-12T03:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.197+0000 7fbe44e89740 20 client.28835 trim_cache size 0 max 16384 2024-09-12T03:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "aafed6b2-b7c3-4469-90a7-8e0c423e8cf2", 2024-09-12T03:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 msgr2=0x7fbe2c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 0x7fbe2c0a50c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009ca0 tx=0x7fbe340dd040 comp rx=0 tx=0).stop 2024-09-12T03:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 msgr2=0x7fbe3c19ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005cc0 tx=0x7fbe340093d0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 shutdown_connections 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbe2c0a2c00 0x7fbe2c0a50c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe3c1082e0 0x7fbe3c19ce40 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe3c103760 0x7fbe3c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 --2- 172.21.15.16:0/1948765151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe3c102db0 0x7fbe3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 >> 172.21.15.16:0/1948765151 conn(0x7fbe3c0fc7d0 msgr2=0x7fbe3c106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 shutdown_connections 2024-09-12T03:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:19.198+0000 7fbe42ab0640 1 -- 172.21.15.16:0/1948765151 wait complete. 2024-09-12T03:05:20.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 102196 23.54 MB/sec execute 141 sec latency 138.633 ms 2024-09-12T03:05:20.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:05:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: pgmap v859: 97 pgs: 97 active+clean; 615 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-09-12T03:05:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:05:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:20 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: pgmap v859: 97 pgs: 97 active+clean; 615 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:20 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: pgmap v859: 97 pgs: 97 active+clean; 615 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:05:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:20 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:05:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/3282538529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4102ef0 msgr2=0x7fe5e410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/3282538529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4102ef0 0x7fe5e410da40 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009ca0 tx=0x7fe5dc02a950 comp rx=0 tx=0).stop 2024-09-12T03:05:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/3282538529 shutdown_connections 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/3282538529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4102ef0 0x7fe5e410da40 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/3282538529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/3282538529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4100e20 0x7fe5e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/3282538529 >> 172.21.15.16:0/3282538529 conn(0x7fe5e40fc7d0 msgr2=0x7fe5e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/3282538529 shutdown_connections 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.814+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/3282538529 wait complete. 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 Processor -- start 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 -- start start 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4102ef0 0x7fe5e419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e410fec0 con 0x7fe5e4100e20 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e410fd40 con 0x7fe5e4102ef0 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e9dcc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4110040 con 0x7fe5e4102570 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47118/0 (socket says 172.21.15.16:47118) 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e37fe640 1 -- 172.21.15.16:0/1622191017 learned_addr learned my addr 172.21.15.16:0/1622191017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4102ef0 0x7fe5e419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 -- 172.21.15.16:0/1622191017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4102ef0 msgr2=0x7fe5e419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4102ef0 0x7fe5e419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 -- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 msgr2=0x7fe5e419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5dbfff640 1 -- 172.21.15.16:0/1622191017 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e419f610 con 0x7fe5e4102570 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.815+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e419aba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00e9f0 tx=0x7fe5cc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5e17fa640 1 -- 172.21.15.16:0/1622191017 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc00ccc0 con 0x7fe5e4102570 2024-09-12T03:05:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e419f900 con 0x7fe5e4102570 2024-09-12T03:05:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5e17fa640 1 -- 172.21.15.16:0/1622191017 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5cc010910 con 0x7fe5e4102570 2024-09-12T03:05:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5e17fa640 1 -- 172.21.15.16:0/1622191017 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc018960 con 0x7fe5e4102570 2024-09-12T03:05:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.816+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e41a71e0 con 0x7fe5e4102570 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e17fa640 1 -- 172.21.15.16:0/1622191017 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe5cc010ac0 con 0x7fe5e4102570 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 Processor -- start 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 -- start start 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 0x55e7fc78d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc79aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc6f32e0 con 0x55e7fc798680 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc69ace0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc6ef2a0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc79aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 0x55e7fc78d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40428/0 (socket says 172.21.15.16:40428) 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 0x55e7fc78d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:49712/0 (socket says 172.21.15.16:49712) 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 -- 172.21.15.16:0/3165714776 learned_addr learned my addr 172.21.15.16:0/3165714776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 -- 172.21.15.16:0/3165714776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 msgr2=0x55e7fc78d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 0x55e7fc78d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 -- 172.21.15.16:0/3165714776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 msgr2=0x55e7fc79aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc79aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.819+0000 7fe5e37fe640 1 -- 172.21.15.16:0/3165714776 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7fc67f440 con 0x55e7fc78ddc0 2024-09-12T03:05:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5e17fa640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 0x7fe5d00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4011650 tx=0x7fe5d4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21150eaf569a70e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 0x7fe5d00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5e17fa640 1 -- 172.21.15.16:0/1622191017 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe5cc0cf380 con 0x7fe5e4102570 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5d9ffb640 1 -- 172.21.15.16:0/3165714776 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d401e070 con 0x55e7fc78ddc0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5d9ffb640 1 -- 172.21.15.16:0/3165714776 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5d400fba0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.820+0000 7fe5d9ffb640 1 -- 172.21.15.16:0/3165714776 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4019ce0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/3165714776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 msgr2=0x55e7fc798140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4011650 tx=0x7fe5d4009f90 comp rx=0 tx=0).stop 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 0x7fe5d00a5040 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006010 tx=0x7fe5dc005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/3165714776 shutdown_connections 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc79aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc798140 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/3165714776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78d4a0 0x55e7fc78d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/3165714776 >> 172.21.15.16:0/3165714776 conn(0x55e7fc789a30 msgr2=0x55e7fc789e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/3165714776 shutdown_connections 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/3165714776 wait complete. 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 -- start start 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.821+0000 7fe5ec1a5740 1 Processor -- start 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7952a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc795800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc79b250 con 0x55e7fc798680 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc6ae050 con 0x55e7fc78d4a0 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e7fc79b3d0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5ec1a5740 10 client.0 _ll_register_callbacks cb 0x55e7fc6ee160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40442/0 (socket says 172.21.15.16:40442) 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 -- 172.21.15.16:0/2592812250 learned_addr learned my addr 172.21.15.16:0/2592812250 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc795800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 -- 172.21.15.16:0/2592812250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 msgr2=0x55e7fc7952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 -- 172.21.15.16:0/2592812250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 msgr2=0x55e7fc795800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc795800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 -- 172.21.15.16:0/2592812250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7fc4643c0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4019880 tx=0x7fe5d400faf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fea4e1a47b243e63 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.822+0000 7fe5c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d401e040 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe5d4009cc0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d400fba0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe5d4019a80 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 10 client.28847 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e7fc7a6c20 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5d400fba0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe5d4021070 con 0x55e7fc78d4a0 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 10 client.28847 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 10 client.28847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 20 client.28847 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc7a8440 0x55e7fc7aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 4 client.28847 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7fc700850 con 0x55e7fc7a8440 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.823+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc7a8440 0x55e7fc7aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.824+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc7a8440 0x55e7fc7aa830 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca6b6c897b9b1c5e server_cookie=c11409c010cb680a in_seq=0 out_seq=0 2024-09-12T03:05:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.824+0000 7fe5c2ffd640 10 client.28847 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.824+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2592812250 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e7fc700850 con 0x55e7fc7a8440 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.824+0000 7fe5c2ffd640 10 client.28847 handle_command_reply: tid=0 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 client.28847 shutdown 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 msgr2=0x55e7fc797ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4019880 tx=0x7fe5d400faf0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 shutdown_connections 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc7a8440 0x55e7fc7aa830 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc795800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2592812250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc797ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 >> 172.21.15.16:0/2592812250 conn(0x55e7fc789a30 msgr2=0x55e7fc78f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5c2ffd640 0 client.28847 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 shutdown_connections 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2592812250 wait complete. 2024-09-12T03:05:20.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.825+0000 7fe5ec1a5740 20 client.28847 trim_cache size 0 max 16384 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 Processor -- start 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 -- start start 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7ad380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc798680 0x55e7fc7ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41af6c0 con 0x55e7fc78d4a0 2024-09-12T03:05:20.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b1200 con 0x55e7fc78ddc0 2024-09-12T03:05:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41afd90 con 0x55e7fc798680 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.844+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47154/0 (socket says 172.21.15.16:47154) 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5e37fe640 1 -- 172.21.15.16:0/2648401991 learned_addr learned my addr 172.21.15.16:0/2648401991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc798680 0x55e7fc7ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 -- 172.21.15.16:0/2648401991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc798680 msgr2=0x55e7fc7ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc798680 0x55e7fc7ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 -- 172.21.15.16:0/2648401991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 msgr2=0x55e7fc790740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 -- 172.21.15.16:0/2648401991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7fc795b70 con 0x55e7fc78ddc0 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7ad380 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0ccee0 tx=0x7fe5cc0ccd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac077f129033f07f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5d97fa640 1 -- 172.21.15.16:0/2648401991 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc09b7e0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5d97fa640 1 -- 172.21.15.16:0/2648401991 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5cc09bca0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.845+0000 7fe5d97fa640 1 -- 172.21.15.16:0/2648401991 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc09b600 con 0x55e7fc78ddc0 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2648401991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 msgr2=0x55e7fc7ad380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7ad380 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0ccee0 tx=0x7fe5cc0ccd60 comp rx=0 tx=0).stop 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2648401991 shutdown_connections 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc798680 0x55e7fc7ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7ad380 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2648401991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc78d4a0 0x55e7fc790740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2648401991 >> 172.21.15.16:0/2648401991 conn(0x55e7fc789a30 msgr2=0x55e7fc790030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2648401991 shutdown_connections 2024-09-12T03:05:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.846+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2648401991 wait complete. 2024-09-12T03:05:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.847+0000 7fe5ec1a5740 1 -- start start 2024-09-12T03:05:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.847+0000 7fe5ec1a5740 1 Processor -- start 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.847+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc7aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.847+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7aa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.847+0000 7fe5ec1a5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b0b30 con 0x55e7fc798680 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5ec1a5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41b0460 con 0x55e7fc78d4a0 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47162/0 (socket says 172.21.15.16:47162) 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 -- 172.21.15.16:0/2577650072 learned_addr learned my addr 172.21.15.16:0/2577650072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ad6a0 con 0x55e7fc78ddc0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5ec1a5740 10 client.0 _ll_register_callbacks cb 0x7fe5d401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc7aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 -- 172.21.15.16:0/2577650072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 msgr2=0x55e7fc7aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 -- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 msgr2=0x55e7fc7aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc7aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 -- 172.21.15.16:0/2577650072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7fc811040 con 0x55e7fc798680 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5dbfff640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc7aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.848+0000 7fe5e3fff640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009ca0 tx=0x7fe5dc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acf9f3a1e806a92 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc034040 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe5dc004510 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5cc09bca0 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe5dc004710 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5ec1a5740 10 client.19554 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.849+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e7fc812330 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5cc09bca0 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe5dc05d570 con 0x55e7fc798680 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 10 client.19554 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 10 client.19554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 20 client.19554 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc812b00 0x55e7fc814f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 4 client.19554 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.850+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e7fc6fad60 con 0x55e7fc812b00 2024-09-12T03:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.851+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc812b00 0x55e7fc814f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.851+0000 7fe5e37fe640 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc812b00 0x55e7fc814f20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19774f2769fe73b1 server_cookie=8e43baad481d1c20 in_seq=0 out_seq=0 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.851+0000 7fe5c2ffd640 10 client.19554 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/2577650072 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e7fc6fad60 con 0x55e7fc812b00 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5c2ffd640 10 client.19554 handle_command_reply: tid=0 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 client.19554 shutdown 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 msgr2=0x55e7fc7aaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009ca0 tx=0x7fe5dc0095c0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 shutdown_connections 2024-09-12T03:05:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e7fc812b00 0x55e7fc814f20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5c2ffd640 0 client.19554 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7fc798680 0x55e7fc7aaed0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e7fc78ddc0 0x55e7fc7aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 --2- 172.21.15.16:0/2577650072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7fc78d4a0 0x55e7fc7aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 >> 172.21.15.16:0/2577650072 conn(0x55e7fc789a30 msgr2=0x55e7fc788350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 shutdown_connections 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 1 -- 172.21.15.16:0/2577650072 wait complete. 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.852+0000 7fe5ec1a5740 20 client.19554 trim_cache size 0 max 16384 2024-09-12T03:05:20.856 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "aafed6b2-b7c3-4469-90a7-8e0c423e8cf2": { 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "aafed6b2-b7c3-4469-90a7-8e0c423e8cf2", 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 msgr2=0x7fe5d00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 0x7fe5d00a5040 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006010 tx=0x7fe5dc005d80 comp rx=0 tx=0).stop 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 msgr2=0x7fe5e419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e419aba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00e9f0 tx=0x7fe5cc00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 shutdown_connections 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e4102ef0 0x7fe5e419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5e4102570 0x7fe5e419aba0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe5d00a2b80 0x7fe5d00a5040 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 --2- 172.21.15.16:0/1622191017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e4100e20 0x7fe5e419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 >> 172.21.15.16:0/1622191017 conn(0x7fe5e40fc7d0 msgr2=0x7fe5e41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 shutdown_connections 2024-09-12T03:05:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:20.853+0000 7fe5e9dcc640 1 -- 172.21.15.16:0/1622191017 wait complete. 2024-09-12T03:05:21.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 102831 23.67 MB/sec execute 142 sec latency 101.607 ms 2024-09-12T03:05:21.653 INFO:tasks.cephfs.filesystem:scrub status for tag:aafed6b2-b7c3-4469-90a7-8e0c423e8cf2 - {'path': '/', 'tag': 'aafed6b2-b7c3-4469-90a7-8e0c423e8cf2', 'options': 'recursive,force'} 2024-09-12T03:05:22.017 INFO:tasks.workunit.client.0.smithi016.stdout: 10 103392 23.68 MB/sec execute 143 sec latency 58.393 ms 2024-09-12T03:05:22.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:22 smithi033 ceph-mon[35092]: pgmap v860: 97 pgs: 97 active+clean; 627 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2024-09-12T03:05:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:22 smithi184 ceph-mon[35321]: pgmap v860: 97 pgs: 97 active+clean; 627 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2024-09-12T03:05:22.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:22 smithi016 ceph-mon[30925]: pgmap v860: 97 pgs: 97 active+clean; 627 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2024-09-12T03:05:23.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 104036 23.75 MB/sec execute 144 sec latency 51.049 ms 2024-09-12T03:05:24.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 104823 23.84 MB/sec execute 145 sec latency 54.950 ms 2024-09-12T03:05:24.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:24 smithi033 ceph-mon[35092]: pgmap v861: 97 pgs: 97 active+clean; 632 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:05:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:24 smithi184 ceph-mon[35321]: pgmap v861: 97 pgs: 97 active+clean; 632 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:05:24.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:24 smithi016 ceph-mon[30925]: pgmap v861: 97 pgs: 97 active+clean; 632 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:05:25.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 105544 23.98 MB/sec execute 146 sec latency 91.063 ms 2024-09-12T03:05:26.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106199 24.03 MB/sec execute 147 sec latency 69.669 ms 2024-09-12T03:05:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:26 smithi033 ceph-mon[35092]: pgmap v862: 97 pgs: 97 active+clean; 635 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-09-12T03:05:26.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:26 smithi184 ceph-mon[35321]: pgmap v862: 97 pgs: 97 active+clean; 635 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-09-12T03:05:26.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:26 smithi016 ceph-mon[30925]: pgmap v862: 97 pgs: 97 active+clean; 635 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-09-12T03:05:27.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 106890 24.12 MB/sec execute 148 sec latency 148.378 ms 2024-09-12T03:05:28.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 107567 24.23 MB/sec execute 149 sec latency 139.986 ms 2024-09-12T03:05:28.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:28 smithi033 ceph-mon[35092]: pgmap v863: 97 pgs: 97 active+clean; 634 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 576 op/s 2024-09-12T03:05:28.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:28 smithi184 ceph-mon[35321]: pgmap v863: 97 pgs: 97 active+clean; 634 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 576 op/s 2024-09-12T03:05:28.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:28 smithi016 ceph-mon[30925]: pgmap v863: 97 pgs: 97 active+clean; 634 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.6 MiB/s wr, 576 op/s 2024-09-12T03:05:29.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 108075 24.24 MB/sec execute 150 sec latency 90.033 ms 2024-09-12T03:05:30.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 108713 24.28 MB/sec execute 151 sec latency 73.232 ms 2024-09-12T03:05:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:30 smithi033 ceph-mon[35092]: pgmap v864: 97 pgs: 97 active+clean; 636 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-09-12T03:05:30.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:30 smithi184 ceph-mon[35321]: pgmap v864: 97 pgs: 97 active+clean; 636 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-09-12T03:05:30.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:30 smithi016 ceph-mon[30925]: pgmap v864: 97 pgs: 97 active+clean; 636 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-09-12T03:05:31.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 109540 24.43 MB/sec execute 152 sec latency 110.742 ms 2024-09-12T03:05:32.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 110227 24.54 MB/sec execute 153 sec latency 69.243 ms 2024-09-12T03:05:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:32 smithi033 ceph-mon[35092]: pgmap v865: 97 pgs: 97 active+clean; 638 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-09-12T03:05:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:32 smithi184 ceph-mon[35321]: pgmap v865: 97 pgs: 97 active+clean; 638 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-09-12T03:05:32.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:32 smithi016 ceph-mon[30925]: pgmap v865: 97 pgs: 97 active+clean; 638 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-09-12T03:05:33.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 110802 24.59 MB/sec execute 154 sec latency 49.166 ms 2024-09-12T03:05:33.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:05:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:05:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:05:33.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:33 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:05:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:05:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:05:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:05:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:33 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:05:33.558 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:05:33.558 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:05:33.558 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:05:33.558 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:33 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:05:34.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 111492 24.64 MB/sec execute 155 sec latency 56.591 ms 2024-09-12T03:05:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:34 smithi016 ceph-mon[30925]: pgmap v866: 97 pgs: 97 active+clean; 642 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 457 op/s 2024-09-12T03:05:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:34 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:05:34.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:34 smithi033 ceph-mon[35092]: pgmap v866: 97 pgs: 97 active+clean; 642 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 457 op/s 2024-09-12T03:05:34.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:34 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:05:34.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:34 smithi184 ceph-mon[35321]: pgmap v866: 97 pgs: 97 active+clean; 642 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 457 op/s 2024-09-12T03:05:34.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:34 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:05:35.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 112236 24.75 MB/sec execute 156 sec latency 96.332 ms 2024-09-12T03:05:35.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:35 smithi016 ceph-mon[30925]: pgmap v867: 97 pgs: 97 active+clean; 646 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-09-12T03:05:35.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:35 smithi033 ceph-mon[35092]: pgmap v867: 97 pgs: 97 active+clean; 646 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-09-12T03:05:35.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:35 smithi184 ceph-mon[35321]: pgmap v867: 97 pgs: 97 active+clean; 646 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-09-12T03:05:36.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 112800 24.80 MB/sec execute 157 sec latency 47.166 ms 2024-09-12T03:05:37.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 113277 24.75 MB/sec execute 158 sec latency 214.836 ms 2024-09-12T03:05:37.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:37 smithi016 ceph-mon[30925]: pgmap v868: 97 pgs: 97 active+clean; 656 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 561 op/s 2024-09-12T03:05:37.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:37 smithi033 ceph-mon[35092]: pgmap v868: 97 pgs: 97 active+clean; 656 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 561 op/s 2024-09-12T03:05:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:37 smithi184 ceph-mon[35321]: pgmap v868: 97 pgs: 97 active+clean; 656 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 561 op/s 2024-09-12T03:05:38.018 INFO:tasks.workunit.client.0.smithi016.stdout: 10 113979 24.86 MB/sec execute 159 sec latency 202.158 ms 2024-09-12T03:05:39.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 114754 25.02 MB/sec execute 160 sec latency 50.010 ms 2024-09-12T03:05:39.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:39 smithi016 ceph-mon[30925]: pgmap v869: 97 pgs: 97 active+clean; 666 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 449 op/s 2024-09-12T03:05:39.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:39 smithi033 ceph-mon[35092]: pgmap v869: 97 pgs: 97 active+clean; 666 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 449 op/s 2024-09-12T03:05:39.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:39 smithi184 ceph-mon[35321]: pgmap v869: 97 pgs: 97 active+clean; 666 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 449 op/s 2024-09-12T03:05:40.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 115316 25.03 MB/sec execute 161 sec latency 55.740 ms 2024-09-12T03:05:41.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 115960 25.10 MB/sec execute 162 sec latency 65.990 ms 2024-09-12T03:05:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:41 smithi016 ceph-mon[30925]: pgmap v870: 97 pgs: 97 active+clean; 673 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-09-12T03:05:41.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:41 smithi033 ceph-mon[35092]: pgmap v870: 97 pgs: 97 active+clean; 673 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-09-12T03:05:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:41 smithi184 ceph-mon[35321]: pgmap v870: 97 pgs: 97 active+clean; 673 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-09-12T03:05:42.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 116481 25.07 MB/sec execute 163 sec latency 209.464 ms 2024-09-12T03:05:43.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 117056 25.11 MB/sec execute 164 sec latency 119.628 ms 2024-09-12T03:05:43.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:43 smithi016 ceph-mon[30925]: pgmap v871: 97 pgs: 97 active+clean; 670 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 588 op/s 2024-09-12T03:05:43.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:43 smithi033 ceph-mon[35092]: pgmap v871: 97 pgs: 97 active+clean; 670 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 588 op/s 2024-09-12T03:05:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:43 smithi184 ceph-mon[35321]: pgmap v871: 97 pgs: 97 active+clean; 670 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 588 op/s 2024-09-12T03:05:44.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 117742 25.17 MB/sec execute 165 sec latency 43.834 ms 2024-09-12T03:05:45.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 118503 25.26 MB/sec execute 166 sec latency 90.532 ms 2024-09-12T03:05:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:45 smithi016 ceph-mon[30925]: pgmap v872: 97 pgs: 97 active+clean; 672 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-09-12T03:05:45.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:45 smithi033 ceph-mon[35092]: pgmap v872: 97 pgs: 97 active+clean; 672 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-09-12T03:05:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:45 smithi184 ceph-mon[35321]: pgmap v872: 97 pgs: 97 active+clean; 672 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-09-12T03:05:46.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 119293 25.41 MB/sec execute 167 sec latency 44.317 ms 2024-09-12T03:05:47.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 119832 25.45 MB/sec execute 168 sec latency 186.299 ms 2024-09-12T03:05:47.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:47 smithi016 ceph-mon[30925]: pgmap v873: 97 pgs: 97 active+clean; 680 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.3 MiB/s wr, 713 op/s 2024-09-12T03:05:47.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:47 smithi033 ceph-mon[35092]: pgmap v873: 97 pgs: 97 active+clean; 680 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.3 MiB/s wr, 713 op/s 2024-09-12T03:05:47.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:47 smithi184 ceph-mon[35321]: pgmap v873: 97 pgs: 97 active+clean; 680 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.3 MiB/s wr, 713 op/s 2024-09-12T03:05:48.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120312 25.44 MB/sec execute 169 sec latency 77.015 ms 2024-09-12T03:05:49.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 120919 25.44 MB/sec execute 170 sec latency 47.494 ms 2024-09-12T03:05:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:49 smithi016 ceph-mon[30925]: pgmap v874: 97 pgs: 97 active+clean; 684 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 618 op/s 2024-09-12T03:05:49.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:49 smithi033 ceph-mon[35092]: pgmap v874: 97 pgs: 97 active+clean; 684 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 618 op/s 2024-09-12T03:05:49.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:49 smithi184 ceph-mon[35321]: pgmap v874: 97 pgs: 97 active+clean; 684 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 618 op/s 2024-09-12T03:05:50.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 121763 25.58 MB/sec execute 171 sec latency 81.312 ms 2024-09-12T03:05:51.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 122474 25.63 MB/sec execute 172 sec latency 99.018 ms 2024-09-12T03:05:51.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:05:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:51 smithi016 ceph-mon[30925]: pgmap v875: 97 pgs: 97 active+clean; 667 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-09-12T03:05:51.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:51 smithi033 ceph-mon[35092]: pgmap v875: 97 pgs: 97 active+clean; 667 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-09-12T03:05:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:51 smithi184 ceph-mon[35321]: pgmap v875: 97 pgs: 97 active+clean; 667 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-09-12T03:05:52.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 123055 25.65 MB/sec execute 173 sec latency 171.146 ms 2024-09-12T03:05:52.116 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 -- 172.21.15.16:0/531844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64101410 msgr2=0x7f8b6410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 --2- 172.21.15.16:0/531844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64101410 0x7f8b6410d940 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009cd0 tx=0x7f8b5c02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 -- 172.21.15.16:0/531844156 shutdown_connections 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 --2- 172.21.15.16:0/531844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64101410 0x7f8b6410d940 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 --2- 172.21.15.16:0/531844156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64100a70 0x7f8b64100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 --2- 172.21.15.16:0/531844156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 0x7f8b64106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 -- 172.21.15.16:0/531844156 >> 172.21.15.16:0/531844156 conn(0x7f8b640fc7d0 msgr2=0x7f8b640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 -- 172.21.15.16:0/531844156 shutdown_connections 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.901+0000 7f8b68c92640 1 -- 172.21.15.16:0/531844156 wait complete. 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.902+0000 7f8b68c92640 1 Processor -- start 2024-09-12T03:05:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 -- start start 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64101410 0x7f8b64198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 0x7f8b6419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64110e70 con 0x7f8b64100a70 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64110cf0 con 0x7f8b64101410 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b68c92640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64110ff0 con 0x7f8b64106a70 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52510/0 (socket says 172.21.15.16:52510) 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b6259c640 1 -- 172.21.15.16:0/62397998 learned_addr learned my addr 172.21.15.16:0/62397998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 0x7f8b6419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.903+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64101410 0x7f8b64198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 -- 172.21.15.16:0/62397998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 msgr2=0x7f8b6419ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 0x7f8b6419ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 -- 172.21.15.16:0/62397998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64101410 msgr2=0x7f8b64198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64101410 0x7f8b64198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 -- 172.21.15.16:0/62397998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6419d3b0 con 0x7f8b64100a70 2024-09-12T03:05:52.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b6259c640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e9f0 tx=0x7f8b5400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b5bfff640 1 -- 172.21.15.16:0/62397998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400cd80 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6419d6a0 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b5bfff640 1 -- 172.21.15.16:0/62397998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b54005500 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b5bfff640 1 -- 172.21.15.16:0/62397998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54010690 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.904+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641a4f80 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.906+0000 7f8b5bfff640 1 -- 172.21.15.16:0/62397998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8b540046c0 con 0x7f8b64100a70 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.907+0000 7f8b6b06b740 1 Processor -- start 2024-09-12T03:05:52.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.907+0000 7f8b6b06b740 1 -- start start 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739368880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x560739373140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607392ce2e0 con 0x5607393684a0 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560739275ce0 con 0x560739373680 2024-09-12T03:05:52.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5607392ca2a0 con 0x560739368dc0 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b5bfff640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 0x7f8b500a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59914/0 (socket says 172.21.15.16:59914) 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 -- 172.21.15.16:0/2546945887 learned_addr learned my addr 172.21.15.16:0/2546945887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739368880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b6259c640 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x560739373140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b5bfff640 1 -- 172.21.15.16:0/62397998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8b540cf3e0 con 0x7f8b64100a70 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 0x7f8b500a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 -- 172.21.15.16:0/2546945887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 msgr2=0x560739373140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x560739373140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 -- 172.21.15.16:0/2546945887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 msgr2=0x560739368880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739368880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.908+0000 7f8b59d9b640 1 -- 172.21.15.16:0/2546945887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56073925a440 con 0x560739373680 2024-09-12T03:05:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.909+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009c90 tx=0x7f8b4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe0c519bbd7b70e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.909+0000 7f8b58d99640 1 -- 172.21.15.16:0/2546945887 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c01e070 con 0x560739373680 2024-09-12T03:05:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.909+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 0x7f8b500a50c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c004ab0 tx=0x7f8b5c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.909+0000 7f8b58d99640 1 -- 172.21.15.16:0/2546945887 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4c019b10 con 0x560739373680 2024-09-12T03:05:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.909+0000 7f8b58d99640 1 -- 172.21.15.16:0/2546945887 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00eac0 con 0x560739373680 2024-09-12T03:05:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2546945887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 msgr2=0x560739375a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009c90 tx=0x7f8b4c011810 comp rx=0 tx=0).stop 2024-09-12T03:05:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2546945887 shutdown_connections 2024-09-12T03:05:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739375a70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x560739373140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2546945887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739368880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2546945887 >> 172.21.15.16:0/2546945887 conn(0x560739364a30 msgr2=0x560739364e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2546945887 shutdown_connections 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2546945887 wait complete. 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 -- start start 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.910+0000 7f8b6b06b740 1 Processor -- start 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 0x5607393702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 0x560739370800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560739376250 con 0x5607393684a0 2024-09-12T03:05:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560739289050 con 0x560739368dc0 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5607393763d0 con 0x560739373680 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6b06b740 10 client.0 _ll_register_callbacks cb 0x5607392c9160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52538/0 (socket says 172.21.15.16:52538) 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 -- 172.21.15.16:0/2776983526 learned_addr learned my addr 172.21.15.16:0/2776983526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 0x5607393702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 0x560739370800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 -- 172.21.15.16:0/2776983526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 msgr2=0x560739370800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 0x560739370800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 -- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 msgr2=0x5607393702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 0x5607393702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.911+0000 7f8b6259c640 1 -- 172.21.15.16:0/2776983526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56073903f3c0 con 0x5607393684a0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 0x560739370800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 0x5607393702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b6259c640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8b54098050 tx=0x7f8b5409b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d52eaa5a80c4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540b2070 con 0x5607393684a0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b4c019b10 con 0x5607393684a0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8b540a45d0 con 0x5607393684a0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8b5409ffa0 con 0x5607393684a0 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b6b06b740 10 client.19566 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.912+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560739381c20 con 0x5607393684a0 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4c019b10 con 0x5607393684a0 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f8b5409ab40 con 0x5607393684a0 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 10 client.19566 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 10 client.19566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 20 client.19566 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560739383080 0x5607393854d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 4 client.19566 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607392db850 con 0x560739383080 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.913+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560739383080 0x5607393854d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.914+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560739383080 0x5607393854d0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b27d8f81d8becc8 server_cookie=b48591ee084bbf28 in_seq=0 out_seq=0 2024-09-12T03:05:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.914+0000 7f8b49ffb640 10 client.19566 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.915+0000 7f8b49ffb640 1 -- 172.21.15.16:0/2776983526 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5607392db850 con 0x560739383080 2024-09-12T03:05:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.915+0000 7f8b49ffb640 10 client.19566 handle_command_reply: tid=0 2024-09-12T03:05:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.915+0000 7f8b6b06b740 1 client.19566 shutdown 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 msgr2=0x560739372ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8b54098050 tx=0x7f8b5409b610 comp rx=0 tx=0).stop 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 shutdown_connections 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560739383080 0x5607393854d0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739373680 0x560739370800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739368dc0 0x5607393702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b49ffb640 0 client.19566 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/2776983526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x560739372ff0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 >> 172.21.15.16:0/2776983526 conn(0x560739364a30 msgr2=0x56073936a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 shutdown_connections 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 1 -- 172.21.15.16:0/2776983526 wait complete. 2024-09-12T03:05:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.916+0000 7f8b6b06b740 20 client.19566 trim_cache size 0 max 16384 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 Processor -- start 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 -- start start 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x56073936eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x56073938ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641ad4a0 con 0x5607393684a0 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641aefe0 con 0x560739373680 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641adb70 con 0x560739368dc0 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.942+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52546/0 (socket says 172.21.15.16:52546) 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b62d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x56073938ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 -- 172.21.15.16:0/821023347 learned_addr learned my addr 172.21.15.16:0/821023347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x56073936eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 -- 172.21.15.16:0/821023347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 msgr2=0x56073936eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x56073936eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 -- 172.21.15.16:0/821023347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 msgr2=0x56073938ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x56073938ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 -- 172.21.15.16:0/821023347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56073936f250 con 0x5607393684a0 2024-09-12T03:05:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b6259c640 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8b54098570 tx=0x7f8b5409bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ffb3c568eac11b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b4bfff640 1 -- 172.21.15.16:0/821023347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540b2070 con 0x5607393684a0 2024-09-12T03:05:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b4bfff640 1 -- 172.21.15.16:0/821023347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b540ba020 con 0x5607393684a0 2024-09-12T03:05:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.943+0000 7f8b4bfff640 1 -- 172.21.15.16:0/821023347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540a43d0 con 0x5607393684a0 2024-09-12T03:05:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 -- 172.21.15.16:0/821023347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 msgr2=0x56073937f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8b54098570 tx=0x7f8b5409bd70 comp rx=0 tx=0).stop 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 -- 172.21.15.16:0/821023347 shutdown_connections 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x56073938ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x56073936eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/821023347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937f4b0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 -- 172.21.15.16:0/821023347 >> 172.21.15.16:0/821023347 conn(0x560739364a30 msgr2=0x56073936b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 -- 172.21.15.16:0/821023347 shutdown_connections 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.944+0000 7f8b6b06b740 1 -- 172.21.15.16:0/821023347 wait complete. 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 -- start start 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 Processor -- start 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x5607393772b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739377a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641ae910 con 0x5607393684a0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641ae240 con 0x560739373680 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641ab480 con 0x560739368dc0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6b06b740 10 client.0 _ll_register_callbacks cb 0x7f8b54097790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52556/0 (socket says 172.21.15.16:52556) 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.945+0000 7f8b6259c640 1 -- 172.21.15.16:0/1850170895 learned_addr learned my addr 172.21.15.16:0/1850170895 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x5607393772b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739377a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 -- 172.21.15.16:0/1850170895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 msgr2=0x5607393772b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x5607393772b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 -- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 msgr2=0x560739377a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739377a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 -- 172.21.15.16:0/1850170895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560739368bf0 con 0x5607393684a0 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b62d9d640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739377a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6259c640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f8b540a0160 tx=0x7f8b540ccf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45ffadda1209253 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5408a070 con 0x5607393684a0 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8b5409b040 con 0x5607393684a0 2024-09-12T03:05:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.946+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b540ba020 con 0x5607393684a0 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8b540b2070 con 0x5607393684a0 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 10 client.19572 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560739368bf0 con 0x5607393684a0 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b540ba020 con 0x5607393684a0 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f8b5408d020 con 0x5607393684a0 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 10 client.19572 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 10 client.19572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 20 client.19572 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5607393ee070 0x5607393f0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 4 client.19572 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:05:52.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.947+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607392d5d60 con 0x5607393ee070 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.948+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5607393ee070 0x5607393f0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.949+0000 7f8b59d9b640 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5607393ee070 0x5607393f0460 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51073efca154c433 server_cookie=dfac864df09c3278 in_seq=0 out_seq=0 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.949+0000 7f8b49ffb640 10 client.19572 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b49ffb640 1 -- 172.21.15.16:0/1850170895 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5607392d5d60 con 0x5607393ee070 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b49ffb640 10 client.19572 handle_command_reply: tid=0 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 client.19572 shutdown 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 msgr2=0x56073937bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f8b540a0160 tx=0x7f8b540ccf10 comp rx=0 tx=0).stop 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 shutdown_connections 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5607393ee070 0x5607393f0460 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b49ffb640 0 client.19572 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560739373680 0x560739377a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560739368dc0 0x5607393772b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 --2- 172.21.15.16:0/1850170895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607393684a0 0x56073937bf30 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 >> 172.21.15.16:0/1850170895 conn(0x560739364a30 msgr2=0x560739363b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 shutdown_connections 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.950+0000 7f8b6b06b740 1 -- 172.21.15.16:0/1850170895 wait complete. 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.951+0000 7f8b6b06b740 20 client.19572 trim_cache size 0 max 16384 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:05:52.954 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.951+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 msgr2=0x7f8b500a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.951+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 0x7f8b500a50c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c004ab0 tx=0x7f8b5c004a40 comp rx=0 tx=0).stop 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.951+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 msgr2=0x7f8b64198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.951+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e9f0 tx=0x7f8b5400edf0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 shutdown_connections 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64106a70 0x7f8b6419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64101410 0x7f8b64198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f8b500a2c00 0x7f8b500a50c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 --2- 172.21.15.16:0/62397998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b64100a70 0x7f8b64198400 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 >> 172.21.15.16:0/62397998 conn(0x7f8b640fc7d0 msgr2=0x7f8b6406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 shutdown_connections 2024-09-12T03:05:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:52.952+0000 7f8b68c92640 1 -- 172.21.15.16:0/62397998 wait complete. 2024-09-12T03:05:53.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 123753 25.75 MB/sec execute 174 sec latency 294.182 ms 2024-09-12T03:05:53.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:53 smithi016 ceph-mon[30925]: pgmap v876: 97 pgs: 97 active+clean; 666 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-09-12T03:05:53.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:53 smithi033 ceph-mon[35092]: pgmap v876: 97 pgs: 97 active+clean; 666 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-09-12T03:05:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:53 smithi184 ceph-mon[35321]: pgmap v876: 97 pgs: 97 active+clean; 666 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-09-12T03:05:53.978 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:05:53.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:05:54.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 124465 25.88 MB/sec execute 175 sec latency 63.693 ms 2024-09-12T03:05:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:55.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 -- 172.21.15.16:0/3492031020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 msgr2=0x7f72a01043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/3492031020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a01043d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f729c009860 tx=0x7f729c02a340 comp rx=0 tx=0).stop 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 -- 172.21.15.16:0/3492031020 shutdown_connections 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/3492031020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a0110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/3492031020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 0x7f72a01089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/3492031020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a01043d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.995+0000 7f72a7aa3640 1 -- 172.21.15.16:0/3492031020 >> 172.21.15.16:0/3492031020 conn(0x7f72a00ffab0 msgr2=0x7f72a0101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.996+0000 7f72a7aa3640 1 -- 172.21.15.16:0/3492031020 shutdown_connections 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.996+0000 7f72a7aa3640 1 -- 172.21.15.16:0/3492031020 wait complete. 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.996+0000 7f72a7aa3640 1 Processor -- start 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.996+0000 7f72a7aa3640 1 -- start start 2024-09-12T03:05:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 0x7f72a019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72a0114120 con 0x7f72a0109120 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f72a0113fa0 con 0x7f72a01049a0 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a7aa3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f72a01142a0 con 0x7f72a0103ff0 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a6040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a6040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52574/0 (socket says 172.21.15.16:52574) 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a6040640 1 -- 172.21.15.16:0/2054716780 learned_addr learned my addr 172.21.15.16:0/2054716780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a503e640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 0x7f72a019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 -- 172.21.15.16:0/2054716780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 msgr2=0x7f72a019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 0x7f72a019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 -- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 msgr2=0x7f72a019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 -- 172.21.15.16:0/2054716780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a019f5b0 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a6040640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.997+0000 7f72a583f640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a019a600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f729c02a860 tx=0x7f729c033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.998+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c037040 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.998+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a019f840 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.998+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c033d50 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.998+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c003960 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.998+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a01a7180 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:54.999+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a01054d0 con 0x7f72a0103ff0 2024-09-12T03:05:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.000+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f729c06bd10 con 0x7f72a0103ff0 2024-09-12T03:05:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.001+0000 7f7296ffd640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 0x7f728c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.001+0000 7f72a503e640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 0x7f728c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.001+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f729c0ee870 con 0x7f72a0103ff0 2024-09-12T03:05:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.002+0000 7f72a503e640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 0x7f728c0a5040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f72a00fffd0 tx=0x7f7290005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.005+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f729c049020 con 0x7f72a0103ff0 2024-09-12T03:05:55.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 124901 25.85 MB/sec execute 176 sec latency 66.017 ms 2024-09-12T03:05:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.274+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72a01043d0 con 0x7f72a0103ff0 2024-09-12T03:05:55.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.275+0000 7f7296ffd640 1 -- 172.21.15.16:0/2054716780 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f72a01043d0 con 0x7f72a0103ff0 2024-09-12T03:05:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:05:55.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:05:55.277 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 msgr2=0x7f728c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 0x7f728c0a5040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f72a00fffd0 tx=0x7f7290005f90 comp rx=0 tx=0).stop 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 msgr2=0x7f72a019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a019a600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f729c02a860 tx=0x7f729c033b60 comp rx=0 tx=0).stop 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 shutdown_connections 2024-09-12T03:05:55.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f728c0a2b80 0x7f728c0a5040 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a0109120 0x7f72a019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72a01049a0 0x7f72a019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 --2- 172.21.15.16:0/2054716780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72a0103ff0 0x7f72a019a600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 >> 172.21.15.16:0/2054716780 conn(0x7f72a00ffab0 msgr2=0x7f72a0100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 shutdown_connections 2024-09-12T03:05:55.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:55.278+0000 7f72a7aa3640 1 -- 172.21.15.16:0/2054716780 wait complete. 2024-09-12T03:05:55.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:55 smithi016 ceph-mon[30925]: pgmap v877: 97 pgs: 97 active+clean; 669 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 674 op/s 2024-09-12T03:05:55.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:55 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2054716780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:55.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:55 smithi033 ceph-mon[35092]: pgmap v877: 97 pgs: 97 active+clean; 669 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 674 op/s 2024-09-12T03:05:55.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:55 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2054716780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:55 smithi184 ceph-mon[35321]: pgmap v877: 97 pgs: 97 active+clean; 669 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.2 MiB/s wr, 674 op/s 2024-09-12T03:05:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:55 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2054716780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:05:55.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:05:56.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 125479 25.83 MB/sec execute 177 sec latency 48.244 ms 2024-09-12T03:05:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 -- 172.21.15.16:0/940471772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901082a0 msgr2=0x7f699010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 --2- 172.21.15.16:0/940471772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901082a0 0x7f699010ec40 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f6988009930 tx=0x7f698802a420 comp rx=0 tx=0).stop 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 -- 172.21.15.16:0/940471772 shutdown_connections 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 --2- 172.21.15.16:0/940471772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901082a0 0x7f699010ec40 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 --2- 172.21.15.16:0/940471772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69901036e0 0x7f6990107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.849+0000 7f6994a1b640 1 --2- 172.21.15.16:0/940471772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6990102d30 0x7f6990103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.850+0000 7f6994a1b640 1 -- 172.21.15.16:0/940471772 >> 172.21.15.16:0/940471772 conn(0x7f69900fc7d0 msgr2=0x7f69900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.850+0000 7f6994a1b640 1 -- 172.21.15.16:0/940471772 shutdown_connections 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.850+0000 7f6994a1b640 1 -- 172.21.15.16:0/940471772 wait complete. 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.850+0000 7f6994a1b640 1 Processor -- start 2024-09-12T03:05:56.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 -- start start 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 0x7f69901a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901082a0 0x7f699019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6990101bc0 con 0x7f69901036e0 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6990101a40 con 0x7f6990102d30 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f6994a1b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6990101d40 con 0x7f69901082a0 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35388/0 (socket says 172.21.15.16:35388) 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f698e59c640 1 -- 172.21.15.16:0/2116850419 learned_addr learned my addr 172.21.15.16:0/2116850419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f698dd9b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 0x7f69901a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.851+0000 7f698ed9d640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901082a0 0x7f699019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 -- 172.21.15.16:0/2116850419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901082a0 msgr2=0x7f699019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901082a0 0x7f699019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 -- 172.21.15.16:0/2116850419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 msgr2=0x7f69901a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 -- 172.21.15.16:0/2116850419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699019acc0 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698e59c640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f698dd9b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 0x7f69901a0910 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f69900697d0 tx=0x7f698400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f697f7fe640 1 -- 172.21.15.16:0/2116850419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984010430 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699019afb0 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f697f7fe640 1 -- 172.21.15.16:0/2116850419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6984010ab0 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f697f7fe640 1 -- 172.21.15.16:0/2116850419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984017b30 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.852+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69901a72a0 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.855+0000 7f697f7fe640 1 -- 172.21.15.16:0/2116850419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6984017cd0 con 0x7f69901036e0 2024-09-12T03:05:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.855+0000 7f6996df4740 1 Processor -- start 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 -- start start 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559696de62f0 con 0x559696e81b40 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559696d8dcf0 con 0x559696e805d0 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38826/0 (socket says 172.21.15.16:38826) 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698dd9b640 1 -- 172.21.15.16:0/3626165676 learned_addr learned my addr 172.21.15.16:0/3626165676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:56.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559696de22b0 con 0x559696e8b370 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35394/0 (socket says 172.21.15.16:35394) 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 -- 172.21.15.16:0/3626165676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 msgr2=0x559696e8d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 -- 172.21.15.16:0/3626165676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 msgr2=0x559696e80090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f698e59c640 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e80090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.856+0000 7f697f7fe640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 0x7f69780a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f698e59c640 1 -- 172.21.15.16:0/3626165676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559696d723a0 con 0x559696e805d0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f697f7fe640 1 -- 172.21.15.16:0/2116850419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f69840cf810 con 0x7f69901036e0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f698e59c640 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6980011510 tx=0x7f69800119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372dd150e55bb050 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f698ed9d640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 0x7f69780a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f697cff9640 1 -- 172.21.15.16:0/3626165676 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698001e070 con 0x559696e805d0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f697cff9640 1 -- 172.21.15.16:0/3626165676 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f698000faa0 con 0x559696e805d0 2024-09-12T03:05:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.857+0000 7f697cff9640 1 -- 172.21.15.16:0/3626165676 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980019c30 con 0x559696e805d0 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 msgr2=0x559696e80a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6980011510 tx=0x7f69800119d0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f698ed9d640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 0x7f69780a50c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6988006010 tx=0x7f6988005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 shutdown_connections 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e80a10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 --2- 172.21.15.16:0/3626165676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e80090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 >> 172.21.15.16:0/3626165676 conn(0x559696e49310 msgr2=0x559696e7cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 shutdown_connections 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.858+0000 7f6996df4740 1 -- 172.21.15.16:0/3626165676 wait complete. 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 -- start start 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 Processor -- start 2024-09-12T03:05:56.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e805d0 0x559696e8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e81b40 0x559696e88040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559696e8e070 con 0x559696e805d0 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559696da1060 con 0x559696e8b370 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559696e8e1f0 con 0x559696e81b40 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f6996df4740 10 client.0 _ll_register_callbacks cb 0x559696af48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f698ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f698ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35402/0 (socket says 172.21.15.16:35402) 2024-09-12T03:05:56.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.859+0000 7f698ed9d640 1 -- 172.21.15.16:0/1315270676 learned_addr learned my addr 172.21.15.16:0/1315270676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:56.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698e59c640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e805d0 0x559696e8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698ed9d640 1 -- 172.21.15.16:0/1315270676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e81b40 msgr2=0x559696e88040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:05:56.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698ed9d640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e81b40 0x559696e88040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698ed9d640 1 -- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e805d0 msgr2=0x559696e8ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698ed9d640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e805d0 0x559696e8ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.860+0000 7f698ed9d640 1 -- 172.21.15.16:0/1315270676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559696b573c0 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.862+0000 7f698ed9d640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6988033000 tx=0x7f6988009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca670534b974bec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.862+0000 7f6975ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.863+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988035040 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.863+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6988003b80 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.863+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f698000faa0 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.863+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6988003d80 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.864+0000 7f6996df4740 10 client.28883 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.864+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559696ea2350 con 0x559696e8b370 2024-09-12T03:05:56.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.864+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6988057020 con 0x559696e8b370 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6988044030 con 0x559696e8b370 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 10 client.28883 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 10 client.28883 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 20 client.28883 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696ea3810 0x559696ea5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 4 client.28883 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.865+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f698000faa0 con 0x559696ea3810 2024-09-12T03:05:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.866+0000 7f698e59c640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696ea3810 0x559696ea5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.873+0000 7f698e59c640 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696ea3810 0x559696ea5c60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bbfdaf77a821eeb6 server_cookie=1e6e9187606ec889 in_seq=0 out_seq=0 2024-09-12T03:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.873+0000 7f6975ffb640 10 client.28883 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.878+0000 7f6975ffb640 1 -- 172.21.15.16:0/1315270676 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f698000faa0 con 0x559696ea3810 2024-09-12T03:05:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.878+0000 7f6975ffb640 10 client.28883 handle_command_reply: tid=0 2024-09-12T03:05:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.879+0000 7f6996df4740 1 client.28883 shutdown 2024-09-12T03:05:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.884+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 msgr2=0x559696e88650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.884+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6988033000 tx=0x7f6988009260 comp rx=0 tx=0).stop 2024-09-12T03:05:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 shutdown_connections 2024-09-12T03:05:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696ea3810 0x559696ea5c60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e88650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e81b40 0x559696e88040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 --2- 172.21.15.16:0/1315270676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e805d0 0x559696e8ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.887+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 >> 172.21.15.16:0/1315270676 conn(0x559696e49310 msgr2=0x559696e82930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.889+0000 7f6975ffb640 0 client.28883 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.892+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 shutdown_connections 2024-09-12T03:05:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.893+0000 7f6996df4740 1 -- 172.21.15.16:0/1315270676 wait complete. 2024-09-12T03:05:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.893+0000 7f6996df4740 20 client.28883 trim_cache size 0 max 16384 2024-09-12T03:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.933+0000 7f6996df4740 1 Processor -- start 2024-09-12T03:05:56.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.937+0000 7f6996df4740 1 -- start start 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.937+0000 7f6996df4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.938+0000 7f6996df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.938+0000 7f6996df4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.938+0000 7f6996df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69901ab4a0 con 0x559696e81b40 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.938+0000 7f6996df4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69901acfe0 con 0x559696e805d0 2024-09-12T03:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.938+0000 7f6996df4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f69901abb70 con 0x559696e8b370 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.939+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.939+0000 7f698e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35412/0 (socket says 172.21.15.16:35412) 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.940+0000 7f698e59c640 1 -- 172.21.15.16:0/3486271988 learned_addr learned my addr 172.21.15.16:0/3486271988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.940+0000 7f698dd9b640 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.941+0000 7f698e59c640 1 -- 172.21.15.16:0/3486271988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 msgr2=0x559696e8c760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.941+0000 7f698e59c640 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.941+0000 7f698e59c640 1 -- 172.21.15.16:0/3486271988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 msgr2=0x559696e9fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.941+0000 7f698e59c640 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.941+0000 7f698e59c640 1 -- 172.21.15.16:0/3486271988 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559696e7c4f0 con 0x559696e805d0 2024-09-12T03:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.944+0000 7f698e59c640 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6980009810 tx=0x7f69800197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a119902ca648953 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.944+0000 7f6977fff640 1 -- 172.21.15.16:0/3486271988 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698001e040 con 0x559696e805d0 2024-09-12T03:05:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.945+0000 7f6977fff640 1 -- 172.21.15.16:0/3486271988 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f698000faa0 con 0x559696e805d0 2024-09-12T03:05:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.945+0000 7f6977fff640 1 -- 172.21.15.16:0/3486271988 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800053b0 con 0x559696e805d0 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 -- 172.21.15.16:0/3486271988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 msgr2=0x559696e88010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6980009810 tx=0x7f69800197d0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 -- 172.21.15.16:0/3486271988 shutdown_connections 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e8b370 0x559696e8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 --2- 172.21.15.16:0/3486271988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e805d0 0x559696e88010 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6980009810 tx=0x7f69800197d0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.947+0000 7f6996df4740 1 -- 172.21.15.16:0/3486271988 >> 172.21.15.16:0/3486271988 conn(0x559696e49310 msgr2=0x559696e87900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.955+0000 7f6996df4740 1 -- 172.21.15.16:0/3486271988 shutdown_connections 2024-09-12T03:05:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.956+0000 7f6996df4740 1 -- 172.21.15.16:0/3486271988 wait complete. 2024-09-12T03:05:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 -- start start 2024-09-12T03:05:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 Processor -- start 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e9c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 0x559696e97150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69901ac910 con 0x559696e81b40 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69901ac240 con 0x559696e8b370 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.957+0000 7f6996df4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f69901a9500 con 0x559696e96cd0 2024-09-12T03:05:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.958+0000 7f6996df4740 10 client.0 _ll_register_callbacks cb 0x7f6988009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.959+0000 7f698dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e9c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.960+0000 7f698ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 0x559696e97150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.960+0000 7f698ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 0x559696e97150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38840/0 (socket says 172.21.15.16:38840) 2024-09-12T03:05:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.960+0000 7f698e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.960+0000 7f698ed9d640 1 -- 172.21.15.16:0/3189964664 learned_addr learned my addr 172.21.15.16:0/3189964664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:56.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.962+0000 7f698dd9b640 1 -- 172.21.15.16:0/3189964664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 msgr2=0x559696e97150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.962+0000 7f698dd9b640 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 0x559696e97150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.962+0000 7f698dd9b640 1 -- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 msgr2=0x559696e9bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.962+0000 7f698dd9b640 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.962+0000 7f698dd9b640 1 -- 172.21.15.16:0/3189964664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559696e97690 con 0x559696e8b370 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.963+0000 7f698dd9b640 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e9c260 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f698409bfb0 tx=0x7f69840cfb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd13fd90228bb088 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.964+0000 7f6975ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.964+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698409b980 con 0x559696e8b370 2024-09-12T03:05:56.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.964+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6984010c60 con 0x559696e8b370 2024-09-12T03:05:56.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.967+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f698000faa0 con 0x559696e8b370 2024-09-12T03:05:56.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.973+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f698409c720 con 0x559696e8b370 2024-09-12T03:05:56.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.975+0000 7f6996df4740 10 client.28889 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:56.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.975+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559696f0f140 con 0x559696e8b370 2024-09-12T03:05:56.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.977+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f698000faa0 con 0x559696e8b370 2024-09-12T03:05:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.978+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6984096b90 con 0x559696e8b370 2024-09-12T03:05:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 10 client.28889 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 10 client.28889 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:05:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 20 client.28889 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696f0f840 0x559696f11c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 4 client.28889 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.979+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559696df3860 con 0x559696f0f840 2024-09-12T03:05:56.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.981+0000 7f698e59c640 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696f0f840 0x559696f11c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:56.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.982+0000 7f698e59c640 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696f0f840 0x559696f11c60 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1268e653802fa32c server_cookie=b4b754950d4978a1 in_seq=0 out_seq=0 2024-09-12T03:05:56.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.982+0000 7f6975ffb640 10 client.28889 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:56.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.983+0000 7f6975ffb640 1 -- 172.21.15.16:0/3189964664 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559696df3860 con 0x559696f0f840 2024-09-12T03:05:56.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.983+0000 7f6975ffb640 10 client.28889 handle_command_reply: tid=0 2024-09-12T03:05:56.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.983+0000 7f6996df4740 1 client.28889 shutdown 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 msgr2=0x559696e9c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e9c260 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f698409bfb0 tx=0x7f69840cfb30 comp rx=0 tx=0).stop 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 shutdown_connections 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559696e96cd0 0x559696e97150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x559696f0f840 0x559696f11c60 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559696e8b370 0x559696e9c260 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 --2- 172.21.15.16:0/3189964664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559696e81b40 0x559696e9bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 >> 172.21.15.16:0/3189964664 conn(0x559696e49310 msgr2=0x559696e7bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.984+0000 7f6975ffb640 0 client.28889 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.985+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 shutdown_connections 2024-09-12T03:05:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.985+0000 7f6996df4740 1 -- 172.21.15.16:0/3189964664 wait complete. 2024-09-12T03:05:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.986+0000 7f6996df4740 20 client.28889 trim_cache size 0 max 16384 2024-09-12T03:05:56.988 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 msgr2=0x7f69780a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 0x7f69780a50c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6988006010 tx=0x7f6988005b50 comp rx=0 tx=0).stop 2024-09-12T03:05:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 msgr2=0x7f69901a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 0x7f69901a0910 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f69900697d0 tx=0x7f698400ef90 comp rx=0 tx=0).stop 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 shutdown_connections 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901082a0 0x7f699019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f69780a2c00 0x7f69780a50c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69901036e0 0x7f69901a0910 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 --2- 172.21.15.16:0/2116850419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6990102d30 0x7f69901a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.988+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 >> 172.21.15.16:0/2116850419 conn(0x7f69900fc7d0 msgr2=0x7f69900fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.989+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 shutdown_connections 2024-09-12T03:05:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:56.989+0000 7f6994a1b640 1 -- 172.21.15.16:0/2116850419 wait complete. 2024-09-12T03:05:57.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 126179 25.91 MB/sec execute 178 sec latency 157.090 ms 2024-09-12T03:05:57.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:57 smithi016 ceph-mon[30925]: pgmap v878: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 799 op/s 2024-09-12T03:05:57.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:57 smithi033 ceph-mon[35092]: pgmap v878: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 799 op/s 2024-09-12T03:05:57.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:57 smithi184 ceph-mon[35321]: pgmap v878: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 799 op/s 2024-09-12T03:05:57.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:05:58.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 126812 25.95 MB/sec execute 179 sec latency 256.910 ms 2024-09-12T03:05:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4126139161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e98106f20 msgr2=0x7f6e9810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4126139161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e98106f20 0x7f6e9810eaa0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009ca0 tx=0x7f6e9002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4126139161 shutdown_connections 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4126139161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e98106f20 0x7f6e9810eaa0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4126139161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e981027a0 0x7f6e981067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4126139161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98101df0 0x7f6e981021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4126139161 >> 172.21.15.16:0/4126139161 conn(0x7f6e980fd8b0 msgr2=0x7f6e980ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4126139161 shutdown_connections 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.736+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4126139161 wait complete. 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.737+0000 7f6e9e8e1640 1 Processor -- start 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.737+0000 7f6e9e8e1640 1 -- start start 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.737+0000 7f6e9e8e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.737+0000 7f6e9e8e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 0x7f6e981988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e9e8e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98106f20 0x7f6e9819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e9e8e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e98111f80 con 0x7f6e981027a0 2024-09-12T03:05:58.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e9e8e1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e98111e00 con 0x7f6e98106f20 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e9e8e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6e98112100 con 0x7f6e98101df0 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38844/0 (socket says 172.21.15.16:38844) 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 -- 172.21.15.16:0/4036650413 learned_addr learned my addr 172.21.15.16:0/4036650413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 0x7f6e981988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98106f20 0x7f6e9819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 -- 172.21.15.16:0/4036650413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98106f20 msgr2=0x7f6e9819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98106f20 0x7f6e9819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 -- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 msgr2=0x7f6e981988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 0x7f6e981988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 -- 172.21.15.16:0/4036650413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9819d360 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e977fe640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 0x7f6e981988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.738+0000 7f6e97fff640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9f0 tx=0x7f6e8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.739+0000 7f6e957fa640 1 -- 172.21.15.16:0/4036650413 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800cd80 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.739+0000 7f6e957fa640 1 -- 172.21.15.16:0/4036650413 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e88010910 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.739+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9819d650 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.739+0000 7f6e957fa640 1 -- 172.21.15.16:0/4036650413 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88018990 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.739+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e981a4f30 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.742+0000 7f6e957fa640 1 -- 172.21.15.16:0/4036650413 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6e88010ac0 con 0x7f6e98101df0 2024-09-12T03:05:58.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.742+0000 7f6ea0cba740 1 Processor -- start 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.742+0000 7f6ea0cba740 1 -- start start 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.742+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d81090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d8e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e957fa640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 0x7f6e800a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ef8ce72f0 con 0x559ef8d8c370 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ef8c8ecf0 con 0x559ef8d815d0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ef8ce32b0 con 0x559ef8d82b40 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e957fa640 1 -- 172.21.15.16:0/4036650413 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6e880cf340 con 0x7f6e98101df0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e9ce7e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d81090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35452/0 (socket says 172.21.15.16:35452) 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 -- 172.21.15.16:0/1877494403 learned_addr learned my addr 172.21.15.16:0/1877494403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 0x7f6e800a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e977fe640 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d8e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 -- 172.21.15.16:0/1877494403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 msgr2=0x559ef8d81090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d81090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 -- 172.21.15.16:0/1877494403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 msgr2=0x559ef8d8e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d8e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.743+0000 7f6e97fff640 1 -- 172.21.15.16:0/1877494403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef8c733a0 con 0x559ef8d815d0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.744+0000 7f6e97fff640 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6e88098630 tx=0x7f6e88025360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215ff6ee4119d054 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.744+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 0x7f6e800a5040 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6e90004af0 tx=0x7f6e90005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.744+0000 7f6e7e7fc640 1 -- 172.21.15.16:0/1877494403 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8809bc60 con 0x559ef8d815d0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.744+0000 7f6e7e7fc640 1 -- 172.21.15.16:0/1877494403 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e8809c540 con 0x559ef8d815d0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.744+0000 7f6e7e7fc640 1 -- 172.21.15.16:0/1877494403 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88025170 con 0x559ef8d815d0 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 -- 172.21.15.16:0/1877494403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 msgr2=0x559ef8d81a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6e88098630 tx=0x7f6e88025360 comp rx=0 tx=0).stop 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 -- 172.21.15.16:0/1877494403 shutdown_connections 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d8e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d815d0 0x559ef8d81a10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/1877494403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d81090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 -- 172.21.15.16:0/1877494403 >> 172.21.15.16:0/1877494403 conn(0x559ef8d4a310 msgr2=0x559ef8d7dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 -- 172.21.15.16:0/1877494403 shutdown_connections 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.745+0000 7f6ea0cba740 1 -- 172.21.15.16:0/1877494403 wait complete. 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 -- start start 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 Processor -- start 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d8bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 0x559ef8d89800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559ef8d8efb0 con 0x559ef8d815d0 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559ef8ca2060 con 0x559ef8d8c370 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559ef8d8f130 con 0x559ef8d82b40 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.746+0000 7f6ea0cba740 10 client.0 _ll_register_callbacks cb 0x559ef89f58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e9ce7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 0x559ef8d89800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38850/0 (socket says 172.21.15.16:38850) 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e9ce7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 0x559ef8d89800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35464/0 (socket says 172.21.15.16:35464) 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 -- 172.21.15.16:0/3216369292 learned_addr learned my addr 172.21.15.16:0/3216369292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d8bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 -- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 msgr2=0x559ef8d89800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 0x559ef8d89800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 -- 172.21.15.16:0/3216369292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 msgr2=0x559ef8d8bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d8bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 -- 172.21.15.16:0/3216369292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef8a583c0 con 0x559ef8d82b40 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e777fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.747+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c012d00 tx=0x7f6e8c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98ae84cd708210 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c009d90 con 0x559ef8d82b40 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6e8c012110 con 0x559ef8d82b40 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e8809c540 con 0x559ef8d82b40 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6e8c004050 con 0x559ef8d82b40 2024-09-12T03:05:58.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6ea0cba740 10 client.28816 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559ef8d9ab20 con 0x559ef8d82b40 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e8809c540 con 0x559ef8d82b40 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6e8c010430 con 0x559ef8d82b40 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6ea0cba740 10 client.28816 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.748+0000 7f6ea0cba740 10 client.28816 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6ea0cba740 20 client.28816 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8d9bfe0 0x559ef8d9e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6ea0cba740 4 client.28816 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:05:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ef8cf4860 con 0x559ef8d9bfe0 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8d9bfe0 0x559ef8d9e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6e777fe640 10 client.28816 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.749+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8d9bfe0 0x559ef8d9e430 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8029384323b6db5f server_cookie=baf0e46b4e87d685 in_seq=0 out_seq=0 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.751+0000 7f6e777fe640 1 -- 172.21.15.16:0/3216369292 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ef8cf4860 con 0x559ef8d9bfe0 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.751+0000 7f6e777fe640 10 client.28816 handle_command_reply: tid=0 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 client.28816 shutdown 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 msgr2=0x559ef8d89070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c012d00 tx=0x7f6e8c00f790 comp rx=0 tx=0).stop 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 shutdown_connections 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8d9bfe0 0x559ef8d9e430 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d8c370 0x559ef8d89800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d82b40 0x559ef8d89070 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3216369292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d8bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 >> 172.21.15.16:0/3216369292 conn(0x559ef8d4a310 msgr2=0x559ef8d839c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6e777fe640 0 client.28816 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 shutdown_connections 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3216369292 wait complete. 2024-09-12T03:05:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.752+0000 7f6ea0cba740 20 client.28816 trim_cache size 0 max 16384 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 Processor -- start 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 -- start start 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 0x559ef8d82410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d88930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981ab330 con 0x559ef8d8c370 2024-09-12T03:05:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981ace70 con 0x559ef8d82b40 2024-09-12T03:05:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981aba00 con 0x559ef8d815d0 2024-09-12T03:05:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45976/0 (socket says 172.21.15.16:45976) 2024-09-12T03:05:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 0x559ef8d82410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 -- 172.21.15.16:0/3963806159 learned_addr learned my addr 172.21.15.16:0/3963806159 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d88930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 -- 172.21.15.16:0/3963806159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 msgr2=0x559ef8d82410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 0x559ef8d82410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 -- 172.21.15.16:0/3963806159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 msgr2=0x559ef8d88930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d88930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e9ce7e640 1 -- 172.21.15.16:0/3963806159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef8d7e960 con 0x559ef8d8c370 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d88930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.769+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 0x559ef8d82410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.770+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009ca0 tx=0x7f6e9003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c15dcb88e7989c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.770+0000 7f6e7dffb640 1 -- 172.21.15.16:0/3963806159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90053070 con 0x559ef8d8c370 2024-09-12T03:05:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.770+0000 7f6e7dffb640 1 -- 172.21.15.16:0/3963806159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e90036070 con 0x559ef8d8c370 2024-09-12T03:05:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.770+0000 7f6e7dffb640 1 -- 172.21.15.16:0/3963806159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90030dd0 con 0x559ef8d8c370 2024-09-12T03:05:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3963806159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 msgr2=0x559ef8d84120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009ca0 tx=0x7f6e9003e000 comp rx=0 tx=0).stop 2024-09-12T03:05:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3963806159 shutdown_connections 2024-09-12T03:05:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d8c370 0x559ef8d84120 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d88930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3963806159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d815d0 0x559ef8d82410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3963806159 >> 172.21.15.16:0/3963806159 conn(0x559ef8d4a310 msgr2=0x559ef8d7d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3963806159 shutdown_connections 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3963806159 wait complete. 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 -- start start 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.771+0000 7f6ea0cba740 1 Processor -- start 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d8fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d8c370 0x559ef8d90470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981ac7a0 con 0x559ef8d815d0 2024-09-12T03:05:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981ac0d0 con 0x559ef8d82b40 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6e981a9310 con 0x559ef8d8c370 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45984/0 (socket says 172.21.15.16:45984) 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6ea0cba740 10 client.0 _ll_register_callbacks cb 0x7f6e8c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 -- 172.21.15.16:0/3381352018 learned_addr learned my addr 172.21.15.16:0/3381352018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e9ce7e640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d8c370 0x559ef8d90470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d8fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 -- 172.21.15.16:0/3381352018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d8c370 msgr2=0x559ef8d90470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d8c370 0x559ef8d90470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 -- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 msgr2=0x559ef8d8fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d8fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 -- 172.21.15.16:0/3381352018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef8d90ca0 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e97fff640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6e880980c0 tx=0x7f6e8809cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99b8ea760fa5235 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.772+0000 7f6e777fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88025170 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6e88099d00 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e90036070 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6e88099f60 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6ea0cba740 10 client.19614 fetch_fsmap learned FSMap version 39 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.773+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559ef8e06e10 con 0x559ef8d815d0 2024-09-12T03:05:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e90036070 con 0x559ef8d815d0 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6e8809cee0 con 0x559ef8d815d0 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 10 client.19614 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 10 client.19614 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 20 client.19614 populate_metadata read hostname 'smithi016' 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8e071c0 0x559ef8e095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 4 client.19614 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ef8ceed70 con 0x559ef8e071c0 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.774+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8e071c0 0x559ef8e095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.775+0000 7f6e777fe640 10 client.19614 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.775+0000 7f6e977fe640 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8e071c0 0x559ef8e095b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a6cf23c837533c server_cookie=78a1f8abc113a9b4 in_seq=0 out_seq=0 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6e777fe640 1 -- 172.21.15.16:0/3381352018 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ef8ceed70 con 0x559ef8e071c0 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6e777fe640 10 client.19614 handle_command_reply: tid=0 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 client.19614 shutdown 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 msgr2=0x559ef8d94bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6e880980c0 tx=0x7f6e8809cde0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 shutdown_connections 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559ef8e071c0 0x559ef8e095b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559ef8d8c370 0x559ef8d90470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559ef8d82b40 0x559ef8d8fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 --2- 172.21.15.16:0/3381352018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559ef8d815d0 0x559ef8d94bc0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 >> 172.21.15.16:0/3381352018 conn(0x559ef8d4a310 msgr2=0x559ef8d8a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.776+0000 7f6e777fe640 0 client.19614 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:05:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.777+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 shutdown_connections 2024-09-12T03:05:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.777+0000 7f6ea0cba740 1 -- 172.21.15.16:0/3381352018 wait complete. 2024-09-12T03:05:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.777+0000 7f6ea0cba740 20 client.19614 trim_cache size 0 max 16384 2024-09-12T03:05:58.784 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:05:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 msgr2=0x7f6e800a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 0x7f6e800a5040 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6e90004af0 tx=0x7f6e90005ef0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 msgr2=0x7f6e981983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9f0 tx=0x7f6e8800eeb0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 shutdown_connections 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e98106f20 0x7f6e9819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6e800a2b80 0x7f6e800a5040 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e981027a0 0x7f6e981988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 --2- 172.21.15.16:0/4036650413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e98101df0 0x7f6e981983b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 >> 172.21.15.16:0/4036650413 conn(0x7f6e980fd8b0 msgr2=0x7f6e980febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 shutdown_connections 2024-09-12T03:05:58.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:05:58.778+0000 7f6e9e8e1640 1 -- 172.21.15.16:0/4036650413 wait complete. 2024-09-12T03:05:59.019 INFO:tasks.workunit.client.0.smithi016.stdout: 10 127510 26.00 MB/sec execute 180 sec latency 53.433 ms 2024-09-12T03:05:59.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:05:59.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:05:59 smithi016 ceph-mon[30925]: pgmap v879: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.1 MiB/s wr, 697 op/s 2024-09-12T03:05:59.742 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:05:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:05:59 smithi033 ceph-mon[35092]: pgmap v879: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.1 MiB/s wr, 697 op/s 2024-09-12T03:05:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:05:59 smithi184 ceph-mon[35321]: pgmap v879: 97 pgs: 97 active+clean; 676 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.1 MiB/s wr, 697 op/s 2024-09-12T03:06:00.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 128308 26.11 MB/sec execute 181 sec latency 56.986 ms 2024-09-12T03:06:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/2545239546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54102570 msgr2=0x7f7b541029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/2545239546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54102570 0x7f7b541029b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009900 tx=0x7f7b4802a480 comp rx=0 tx=0).stop 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/2545239546 shutdown_connections 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/2545239546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102ef0 0x7f7b5410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/2545239546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54102570 0x7f7b541029b0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/2545239546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54100e20 0x7f7b54101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/2545239546 >> 172.21.15.16:0/2545239546 conn(0x7f7b540fc7d0 msgr2=0x7f7b540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/2545239546 shutdown_connections 2024-09-12T03:06:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.254+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/2545239546 wait complete. 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 Processor -- start 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 -- start start 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102570 0x7f7b54198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 0x7f7b5419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b54110f80 con 0x7f7b54100e20 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b54110e00 con 0x7f7b54102570 2024-09-12T03:06:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.255+0000 7f7b5a5ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b54111100 con 0x7f7b54102ef0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 0x7f7b5419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46000/0 (socket says 172.21.15.16:46000) 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b53fff640 1 -- 172.21.15.16:0/325571898 learned_addr learned my addr 172.21.15.16:0/325571898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b537fe640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102570 0x7f7b54198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 -- 172.21.15.16:0/325571898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102570 msgr2=0x7f7b54198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102570 0x7f7b54198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 -- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 msgr2=0x7f7b54198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 -- 172.21.15.16:0/325571898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5419d510 con 0x7f7b54102ef0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b53fff640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b58b67640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 0x7f7b5419ce30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009fd0 tx=0x7f7b4c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.256+0000 7f7b517fa640 1 -- 172.21.15.16:0/325571898 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c00ce20 con 0x7f7b54102ef0 2024-09-12T03:06:00.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.257+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5419d800 con 0x7f7b54102ef0 2024-09-12T03:06:00.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.257+0000 7f7b517fa640 1 -- 172.21.15.16:0/325571898 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4c005020 con 0x7f7b54102ef0 2024-09-12T03:06:00.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.257+0000 7f7b517fa640 1 -- 172.21.15.16:0/325571898 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c010720 con 0x7f7b54102ef0 2024-09-12T03:06:00.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.257+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b541a50e0 con 0x7f7b54102ef0 2024-09-12T03:06:00.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.259+0000 7f7b517fa640 1 -- 172.21.15.16:0/325571898 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7b4c005540 con 0x7f7b54102ef0 2024-09-12T03:06:00.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 Processor -- start 2024-09-12T03:06:00.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 -- start start 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfbb40 0x55d786bfa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d786b602f0 con 0x55d786bfa5d0 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d786b07cf0 con 0x55d786bfbb40 2024-09-12T03:06:00.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d786b5c2b0 con 0x55d786c09780 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b517fa640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 0x7f7b3c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46004/0 (socket says 172.21.15.16:46004) 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b53fff640 1 -- 172.21.15.16:0/2370928610 learned_addr learned my addr 172.21.15.16:0/2370928610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b537fe640 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfbb40 0x55d786bfa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b537fe640 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.260+0000 7f7b58b67640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 0x7f7b3c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b517fa640 1 -- 172.21.15.16:0/325571898 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b4c0cf6b0 con 0x7f7b54102ef0 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 -- 172.21.15.16:0/2370928610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 msgr2=0x55d786c0bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 -- 172.21.15.16:0/2370928610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfbb40 msgr2=0x55d786bfa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfbb40 0x55d786bfa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 -- 172.21.15.16:0/2370928610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d786aec3a0 con 0x55d786bfa5d0 2024-09-12T03:06:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b53fff640 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7b44011630 tx=0x7f7b44011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69decc439d195a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b58b67640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 0x7f7b3c0a5040 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00c760 tx=0x7f7b4c01c3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b3a7fc640 1 -- 172.21.15.16:0/2370928610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4401e070 con 0x55d786bfa5d0 2024-09-12T03:06:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b3a7fc640 1 -- 172.21.15.16:0/2370928610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4400fbd0 con 0x55d786bfa5d0 2024-09-12T03:06:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.261+0000 7f7b3a7fc640 1 -- 172.21.15.16:0/2370928610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44019d00 con 0x55d786bfa5d0 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/2370928610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 msgr2=0x55d786bfaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7b44011630 tx=0x7f7b44011af0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/2370928610 shutdown_connections 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bfaa10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/2370928610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfbb40 0x55d786bfa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.262+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/2370928610 >> 172.21.15.16:0/2370928610 conn(0x55d786bc3310 msgr2=0x55d786bf6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.263+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/2370928610 shutdown_connections 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.263+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/2370928610 wait complete. 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.263+0000 7f7b5c9a3740 1 -- start start 2024-09-12T03:06:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.263+0000 7f7b5c9a3740 1 Processor -- start 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786c06470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d786c0c480 con 0x55d786c09780 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d786b1b060 con 0x55d786bfa5d0 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d786c0c600 con 0x55d786bfbb40 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b5c9a3740 10 client.0 _ll_register_callbacks cb 0x55d78686e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46020/0 (socket says 172.21.15.16:46020) 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 -- 172.21.15.16:0/814135843 learned_addr learned my addr 172.21.15.16:0/814135843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b53fff640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786c06470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 -- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 msgr2=0x55d786c06470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786c06470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 -- 172.21.15.16:0/814135843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 msgr2=0x55d786c091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 -- 172.21.15.16:0/814135843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7868d13c0 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.264+0000 7f7b537fe640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7b48031470 tx=0x7f7b48031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1898bd7ac51d1170 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48046040 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b48032c00 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b4400fbd0 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7b48035040 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b5c9a3740 10 client.19626 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d786c17f70 con 0x55d786c09780 2024-09-12T03:06:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4400fbd0 con 0x55d786c09780 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.265+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7b4803d450 con 0x55d786c09780 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 10 client.19626 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 10 client.19626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 20 client.19626 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c19430 0x55d786c1b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 4 client.19626 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d786b6d860 con 0x55d786c19430 2024-09-12T03:06:00.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b53fff640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c19430 0x55d786c1b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b337fe640 10 client.19626 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.266+0000 7f7b53fff640 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c19430 0x55d786c1b820 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=523c17768561c6b4 server_cookie=bf999f6dff003185 in_seq=0 out_seq=0 2024-09-12T03:06:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.267+0000 7f7b337fe640 1 -- 172.21.15.16:0/814135843 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d786b6d860 con 0x55d786c19430 2024-09-12T03:06:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.267+0000 7f7b337fe640 10 client.19626 handle_command_reply: tid=0 2024-09-12T03:06:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.267+0000 7f7b5c9a3740 1 client.19626 shutdown 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 msgr2=0x55d786c06c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7b48031470 tx=0x7f7b48031400 comp rx=0 tx=0).stop 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 shutdown_connections 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786c09780 0x55d786c06c00 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c19430 0x55d786c1b820 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786c06470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/814135843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b337fe640 0 client.19626 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 >> 172.21.15.16:0/814135843 conn(0x55d786bc3310 msgr2=0x55d786bfc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 shutdown_connections 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/814135843 wait complete. 2024-09-12T03:06:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.268+0000 7f7b5c9a3740 20 client.19626 trim_cache size 0 max 16384 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 Processor -- start 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 -- start start 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786bf6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786c09780 0x55d786c21080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541ab340 con 0x55d786bfa5d0 2024-09-12T03:06:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541ace80 con 0x55d786c09780 2024-09-12T03:06:00.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541aba10 con 0x55d786bfbb40 2024-09-12T03:06:00.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46028/0 (socket says 172.21.15.16:46028) 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b537fe640 1 -- 172.21.15.16:0/868628719 learned_addr learned my addr 172.21.15.16:0/868628719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b53fff640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786bf6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.283+0000 7f7b537fe640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786c09780 0x55d786c21080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 -- 172.21.15.16:0/868628719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786c09780 msgr2=0x55d786c21080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786c09780 0x55d786c21080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 -- 172.21.15.16:0/868628719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 msgr2=0x55d786bf6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 -- 172.21.15.16:0/868628719 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d786c0bee0 con 0x55d786bfbb40 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b537fe640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b53fff640 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786bf6c40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400f3c0 tx=0x7f7b440097c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26108bc0784c917d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b39ffb640 1 -- 172.21.15.16:0/868628719 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4401e040 con 0x55d786bfbb40 2024-09-12T03:06:00.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b39ffb640 1 -- 172.21.15.16:0/868628719 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44019410 con 0x55d786bfbb40 2024-09-12T03:06:00.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.284+0000 7f7b39ffb640 1 -- 172.21.15.16:0/868628719 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400fa80 con 0x55d786bfbb40 2024-09-12T03:06:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/868628719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 msgr2=0x55d786bf6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786bf6c40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400f3c0 tx=0x7f7b440097c0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/868628719 shutdown_connections 2024-09-12T03:06:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786c09780 0x55d786c21080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786bfbb40 0x55d786bf6c40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/868628719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfa5d0 0x55d786bf6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/868628719 >> 172.21.15.16:0/868628719 conn(0x55d786bc3310 msgr2=0x55d786c01720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/868628719 shutdown_connections 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/868628719 wait complete. 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.285+0000 7f7b5c9a3740 1 -- start start 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 Processor -- start 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c11f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541ac7b0 con 0x55d786bfbb40 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541ac0e0 con 0x55d786bfa5d0 2024-09-12T03:06:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b541a9320 con 0x55d786c09780 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b5c9a3740 10 client.0 _ll_register_callbacks cb 0x7f7b48009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46034/0 (socket says 172.21.15.16:46034) 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 -- 172.21.15.16:0/3337666437 learned_addr learned my addr 172.21.15.16:0/3337666437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c11f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b537fe640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 -- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 msgr2=0x55d786c0d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 -- 172.21.15.16:0/3337666437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 msgr2=0x55d786c11f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c11f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.286+0000 7f7b53fff640 1 -- 172.21.15.16:0/3337666437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d786c0dfb0 con 0x55d786bfbb40 2024-09-12T03:06:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b53fff640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7b4401c040 tx=0x7f7b440195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cce9e8feedfe19a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b337fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44035070 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b44019410 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7b44009c30 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7b4401e040 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b5c9a3740 10 client.19632 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d786c83fc0 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.287+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44019410 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7b4400f720 con 0x55d786bfbb40 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 10 client.19632 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 10 client.19632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 20 client.19632 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c844f0 0x55d786c86910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 4 client.19632 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d786b67d70 con 0x55d786c844f0 2024-09-12T03:06:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b537fe640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c844f0 0x55d786c86910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b337fe640 10 client.19632 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.288+0000 7f7b537fe640 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c844f0 0x55d786c86910 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f01e5897f01f2c60 server_cookie=eb9ce78e65dc9e82 in_seq=0 out_seq=0 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.291+0000 7f7b337fe640 1 -- 172.21.15.16:0/3337666437 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d786b67d70 con 0x55d786c844f0 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.291+0000 7f7b337fe640 10 client.19632 handle_command_reply: tid=0 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.291+0000 7f7b5c9a3740 1 client.19632 shutdown 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 msgr2=0x55d786c0d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7b4401c040 tx=0x7f7b440195c0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 shutdown_connections 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d786c844f0 0x55d786c86910 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d786c09780 0x55d786c0d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d786bfbb40 0x55d786c0d270 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 --2- 172.21.15.16:0/3337666437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d786bfa5d0 0x55d786c11f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b337fe640 0 client.19632 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 >> 172.21.15.16:0/3337666437 conn(0x55d786bc3310 msgr2=0x55d786bf66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 shutdown_connections 2024-09-12T03:06:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 1 -- 172.21.15.16:0/3337666437 wait complete. 2024-09-12T03:06:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5c9a3740 20 client.19632 trim_cache size 0 max 16384 2024-09-12T03:06:00.296 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 msgr2=0x7f7b3c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 0x7f7b3c0a5040 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00c760 tx=0x7f7b4c01c3b0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 msgr2=0x7f7b5419ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.292+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 0x7f7b5419ce30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009fd0 tx=0x7f7b4c00ef10 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 shutdown_connections 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b54102ef0 0x7f7b5419ce30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b54102570 0x7f7b54198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7b3c0a2b80 0x7f7b3c0a5040 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 --2- 172.21.15.16:0/325571898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b54100e20 0x7f7b54198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 >> 172.21.15.16:0/325571898 conn(0x7f7b540fc7d0 msgr2=0x7f7b5406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 shutdown_connections 2024-09-12T03:06:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:00.293+0000 7f7b5a5ca640 1 -- 172.21.15.16:0/325571898 wait complete. 2024-09-12T03:06:01.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 129012 26.22 MB/sec execute 182 sec latency 44.033 ms 2024-09-12T03:06:01.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:06:01.542 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:01.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:01 smithi016 ceph-mon[30925]: pgmap v880: 97 pgs: 97 active+clean; 681 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 827 op/s 2024-09-12T03:06:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:01 smithi033 ceph-mon[35092]: pgmap v880: 97 pgs: 97 active+clean; 681 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 827 op/s 2024-09-12T03:06:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:01 smithi184 ceph-mon[35321]: pgmap v880: 97 pgs: 97 active+clean; 681 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 827 op/s 2024-09-12T03:06:02.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 129518 26.22 MB/sec execute 183 sec latency 77.035 ms 2024-09-12T03:06:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.075+0000 7fbdbe068640 1 -- 172.21.15.16:0/1263275960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb806ea30 msgr2=0x7fbdb806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.075+0000 7fbdbe068640 1 --2- 172.21.15.16:0/1263275960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb806ea30 0x7fbdb806ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009be0 tx=0x7fbdb002a6f0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.076+0000 7fbdbe068640 1 -- 172.21.15.16:0/1263275960 shutdown_connections 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.076+0000 7fbdbe068640 1 --2- 172.21.15.16:0/1263275960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb811b140 0x7fbdb811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.076+0000 7fbdbe068640 1 --2- 172.21.15.16:0/1263275960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb806ea30 0x7fbdb806ee70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.076+0000 7fbdbe068640 1 --2- 172.21.15.16:0/1263275960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 0x7fbdb806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.076+0000 7fbdbe068640 1 -- 172.21.15.16:0/1263275960 >> 172.21.15.16:0/1263275960 conn(0x7fbdb8073180 msgr2=0x7fbdb8073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.078+0000 7fbdbe068640 1 -- 172.21.15.16:0/1263275960 shutdown_connections 2024-09-12T03:06:02.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.078+0000 7fbdbe068640 1 -- 172.21.15.16:0/1263275960 wait complete. 2024-09-12T03:06:02.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.079+0000 7fbdbe068640 1 Processor -- start 2024-09-12T03:06:02.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.079+0000 7fbdbe068640 1 -- start start 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 0x7fbdb81a7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 0x7fbdb81a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 0x7fbdb81aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8120be0 con 0x7fbdb8110190 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8120a60 con 0x7fbdb811b140 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbe068640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8120d60 con 0x7fbdb806ea30 2024-09-12T03:06:02.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbd066640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 0x7fbdb81a7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbd066640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 0x7fbdb81a7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38936/0 (socket says 172.21.15.16:38936) 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbd066640 1 -- 172.21.15.16:0/659057206 learned_addr learned my addr 172.21.15.16:0/659057206 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbd867640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 0x7fbdb81aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.080+0000 7fbdbc865640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 0x7fbdb81a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 -- 172.21.15.16:0/659057206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 msgr2=0x7fbdb81a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 0x7fbdb81a7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 -- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 msgr2=0x7fbdb81a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 0x7fbdb81a76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 -- 172.21.15.16:0/659057206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb81ac120 con 0x7fbdb811b140 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbd867640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 0x7fbdb81aba40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbdb806b4a0 tx=0x7fbdb400da20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdae7fc640 1 -- 172.21.15.16:0/659057206 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4017070 con 0x7fbdb811b140 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb81ac410 con 0x7fbdb811b140 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.081+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb81b3cf0 con 0x7fbdb811b140 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.082+0000 7fbdae7fc640 1 -- 172.21.15.16:0/659057206 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb4005500 con 0x7fbdb811b140 2024-09-12T03:06:02.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.082+0000 7fbdae7fc640 1 -- 172.21.15.16:0/659057206 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb40096d0 con 0x7fbdb811b140 2024-09-12T03:06:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.085+0000 7fbdc16a3740 1 Processor -- start 2024-09-12T03:06:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.087+0000 7fbdae7fc640 1 -- 172.21.15.16:0/659057206 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbdb4005020 con 0x7fbdb811b140 2024-09-12T03:06:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdae7fc640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 0x7fbda40a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdae7fc640 1 -- 172.21.15.16:0/659057206 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbdb40ceaf0 con 0x7fbdb811b140 2024-09-12T03:06:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdc16a3740 1 -- start start 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812826cf0 0x55d812825240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812825780 0x55d812825bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.089+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d81278b2f0 con 0x55d812826cf0 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d812732cf0 con 0x55d812825780 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d8127872b0 con 0x55d812830520 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdbd066640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812826cf0 0x55d812825240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdbc865640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 0x7fbda40a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38938/0 (socket says 172.21.15.16:38938) 2024-09-12T03:06:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.090+0000 7fbdbd867640 1 -- 172.21.15.16:0/704828308 learned_addr learned my addr 172.21.15.16:0/704828308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbc865640 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812825780 0x55d812825bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbd867640 1 -- 172.21.15.16:0/704828308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812825780 msgr2=0x55d812825bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbd867640 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812825780 0x55d812825bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbd867640 1 -- 172.21.15.16:0/704828308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812826cf0 msgr2=0x55d812825240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbd867640 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812826cf0 0x55d812825240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.092+0000 7fbdbd867640 1 -- 172.21.15.16:0/704828308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8127173a0 con 0x55d812830520 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.093+0000 7fbdbd867640 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40a3a40 tx=0x7fbdb409af10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544c436b648e24f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.093+0000 7fbda37fe640 1 -- 172.21.15.16:0/704828308 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb409b300 con 0x55d812830520 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.093+0000 7fbda37fe640 1 -- 172.21.15.16:0/704828308 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb409f6b0 con 0x55d812830520 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.093+0000 7fbda37fe640 1 -- 172.21.15.16:0/704828308 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4097970 con 0x55d812830520 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 -- 172.21.15.16:0/704828308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 msgr2=0x55d812832910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40a3a40 tx=0x7fbdb409af10 comp rx=0 tx=0).stop 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 -- 172.21.15.16:0/704828308 shutdown_connections 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830520 0x55d812832910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812825780 0x55d812825bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/704828308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812826cf0 0x55d812825240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.094+0000 7fbdc16a3740 1 -- 172.21.15.16:0/704828308 >> 172.21.15.16:0/704828308 conn(0x55d8127ee310 msgr2=0x55d812821e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.097+0000 7fbdc16a3740 1 -- 172.21.15.16:0/704828308 shutdown_connections 2024-09-12T03:06:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.098+0000 7fbdc16a3740 1 -- 172.21.15.16:0/704828308 wait complete. 2024-09-12T03:06:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.098+0000 7fbdc16a3740 1 -- start start 2024-09-12T03:06:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.098+0000 7fbdc16a3740 1 Processor -- start 2024-09-12T03:06:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.099+0000 7fbdbc865640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 0x7fbda40a5180 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbdb811acf0 tx=0x7fbdb002f300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 0x55d81282d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d81282d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d812844170 con 0x55d812830520 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d812746060 con 0x55d81282dff0 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d8128442f0 con 0x55d812830110 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdc16a3740 10 client.0 _ll_register_callbacks cb 0x55d8124998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35554/0 (socket says 172.21.15.16:35554) 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.102+0000 7fbdbd867640 1 -- 172.21.15.16:0/3815046491 learned_addr learned my addr 172.21.15.16:0/3815046491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.103+0000 7fbdbc865640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d81282d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.104+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 0x55d81282d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.104+0000 7fbdbd867640 1 -- 172.21.15.16:0/3815046491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 msgr2=0x55d81282d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.104+0000 7fbdbd867640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d81282d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.104+0000 7fbdbd867640 1 -- 172.21.15.16:0/3815046491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 msgr2=0x55d81282d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.104+0000 7fbdbd867640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 0x55d81282d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.105+0000 7fbdbd867640 1 -- 172.21.15.16:0/3815046491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8124fc3c0 con 0x55d81282dff0 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.105+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 0x55d81282d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.105+0000 7fbdbd867640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4097cd0 tx=0x7fbdb4009340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41741e3f5737de server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.105+0000 7fbda0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:06:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.106+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4005310 con 0x55d81282dff0 2024-09-12T03:06:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.106+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb409f6b0 con 0x55d81282dff0 2024-09-12T03:06:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.106+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbdb40b41b0 con 0x55d81282dff0 2024-09-12T03:06:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbdb409b300 con 0x55d81282dff0 2024-09-12T03:06:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 10 client.28937 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d812850650 con 0x55d81282dff0 2024-09-12T03:06:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb409f6b0 con 0x55d81282dff0 2024-09-12T03:06:02.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fbdb40bb080 con 0x55d81282dff0 2024-09-12T03:06:02.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 10 client.28937 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:02.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 10 client.28937 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:06:02.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 20 client.28937 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:02.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d812851b50 0x55d812853fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 4 client.28937 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:02.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.107+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d812798860 con 0x55d812851b50 2024-09-12T03:06:02.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.108+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d812851b50 0x55d812853fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.109+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d812851b50 0x55d812853fa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9292ac1c6c7ab7a1 server_cookie=a8b6e3795b476639 in_seq=0 out_seq=0 2024-09-12T03:06:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.109+0000 7fbda0ff9640 10 client.28937 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.110+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3815046491 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d812798860 con 0x55d812851b50 2024-09-12T03:06:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.110+0000 7fbda0ff9640 10 client.28937 handle_command_reply: tid=0 2024-09-12T03:06:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.110+0000 7fbdc16a3740 1 client.28937 shutdown 2024-09-12T03:06:02.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.112+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 msgr2=0x55d81284d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.112+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4097cd0 tx=0x7fbdb4009340 comp rx=0 tx=0).stop 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 shutdown_connections 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d812851b50 0x55d812853fa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282dff0 0x55d81284d690 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d81282d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3815046491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d812830520 0x55d81282d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 >> 172.21.15.16:0/3815046491 conn(0x55d8127ee310 msgr2=0x55d812827ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.114+0000 7fbda0ff9640 0 client.28937 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:02.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.117+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 shutdown_connections 2024-09-12T03:06:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.118+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3815046491 wait complete. 2024-09-12T03:06:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.118+0000 7fbdc16a3740 20 client.28937 trim_cache size 0 max 16384 2024-09-12T03:06:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.144+0000 7fbdc16a3740 1 Processor -- start 2024-09-12T03:06:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 -- start start 2024-09-12T03:06:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d81284d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812831910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8118e10 con 0x55d81282dff0 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb81b5750 con 0x55d812830520 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb81b42e0 con 0x55d812830110 2024-09-12T03:06:02.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdbd867640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35560/0 (socket says 172.21.15.16:35560) 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdbd867640 1 -- 172.21.15.16:0/4292099647 learned_addr learned my addr 172.21.15.16:0/4292099647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.145+0000 7fbdbd066640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d81284d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbc865640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812831910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 -- 172.21.15.16:0/4292099647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 msgr2=0x55d812831910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812831910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 -- 172.21.15.16:0/4292099647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 msgr2=0x55d81284d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d81284d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 -- 172.21.15.16:0/4292099647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d81284d9c0 con 0x55d812830520 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd066640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d81284d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.146+0000 7fbdbd867640 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbdb409b000 tx=0x7fbdb40975c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99329c842ebb747d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.147+0000 7fbda2ffd640 1 -- 172.21.15.16:0/4292099647 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb407b020 con 0x55d812830520 2024-09-12T03:06:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.147+0000 7fbda2ffd640 1 -- 172.21.15.16:0/4292099647 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb4005310 con 0x55d812830520 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 -- 172.21.15.16:0/4292099647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 msgr2=0x55d812831e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbda2ffd640 1 -- 172.21.15.16:0/4292099647 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4097970 con 0x55d812830520 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbdb409b000 tx=0x7fbdb40975c0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 -- 172.21.15.16:0/4292099647 shutdown_connections 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d812830520 0x55d812831e50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812831910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/4292099647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d81284d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.148+0000 7fbdc16a3740 1 -- 172.21.15.16:0/4292099647 >> 172.21.15.16:0/4292099647 conn(0x55d8127ee310 msgr2=0x55d81282f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.149+0000 7fbdc16a3740 1 -- 172.21.15.16:0/4292099647 shutdown_connections 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.149+0000 7fbdc16a3740 1 -- 172.21.15.16:0/4292099647 wait complete. 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.149+0000 7fbdc16a3740 1 -- start start 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.149+0000 7fbdc16a3740 1 Processor -- start 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d812829a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282a570 0x55d8128485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb81b5080 con 0x55d81282dff0 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb81b49b0 con 0x55d81282a570 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8116df0 con 0x55d812830110 2024-09-12T03:06:02.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.150+0000 7fbdc16a3740 10 client.0 _ll_register_callbacks cb 0x7fbdb0034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:38968/0 (socket says 172.21.15.16:38968) 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 -- 172.21.15.16:0/3430303449 learned_addr learned my addr 172.21.15.16:0/3430303449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 -- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282a570 msgr2=0x55d8128485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282a570 0x55d8128485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 -- 172.21.15.16:0/3430303449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 msgr2=0x55d812829a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d812829a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 -- 172.21.15.16:0/3430303449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d812848df0 con 0x55d812830110 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbdbc865640 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009be0 tx=0x7fbdb002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1ecc5b1f79dad17 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.151+0000 7fbda0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.152+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0050020 con 0x55d812830110 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.152+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fbdb002fa10 con 0x55d812830110 2024-09-12T03:06:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.152+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb4005310 con 0x55d812830110 2024-09-12T03:06:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fbdb0036040 con 0x55d812830110 2024-09-12T03:06:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 10 client.28858 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d81284a090 con 0x55d812830110 2024-09-12T03:06:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdb4005310 con 0x55d812830110 2024-09-12T03:06:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fbdb0031bd0 con 0x55d812830110 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 10 client.28858 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 10 client.28858 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 20 client.28858 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d8128dec80 0x55d8128e10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 4 client.28858 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.154+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d812792d70 con 0x55d8128dec80 2024-09-12T03:06:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.158+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d8128dec80 0x55d8128e10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.159+0000 7fbda0ff9640 10 client.28858 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.159+0000 7fbdbd066640 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d8128dec80 0x55d8128e10e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c399bbd2359019d4 server_cookie=61b514dde8070cfe in_seq=0 out_seq=0 2024-09-12T03:06:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.159+0000 7fbda0ff9640 1 -- 172.21.15.16:0/3430303449 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d812792d70 con 0x55d8128dec80 2024-09-12T03:06:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.159+0000 7fbda0ff9640 10 client.28858 handle_command_reply: tid=0 2024-09-12T03:06:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 client.28858 shutdown 2024-09-12T03:06:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 msgr2=0x55d812829f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009be0 tx=0x7fbdb002a6f0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 shutdown_connections 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d8128dec80 0x55d8128e10e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d81282a570 0x55d8128485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d812830110 0x55d812829f90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 --2- 172.21.15.16:0/3430303449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d81282dff0 0x55d812829a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbda0ff9640 0 client.28858 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 >> 172.21.15.16:0/3430303449 conn(0x55d8127ee310 msgr2=0x55d8128272c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 shutdown_connections 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 1 -- 172.21.15.16:0/3430303449 wait complete. 2024-09-12T03:06:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.160+0000 7fbdc16a3740 20 client.28858 trim_cache size 0 max 16384 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 msgr2=0x7fbda40a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 0x7fbda40a5180 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbdb811acf0 tx=0x7fbdb002f300 comp rx=0 tx=0).stop 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 msgr2=0x7fbdb81aba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 0x7fbdb81aba40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbdb806b4a0 tx=0x7fbdb400da20 comp rx=0 tx=0).stop 2024-09-12T03:06:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 shutdown_connections 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdb811b140 0x7fbdb81aba40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fbda40a2cc0 0x7fbda40a5180 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb8110190 0x7fbdb81a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 --2- 172.21.15.16:0/659057206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb806ea30 0x7fbdb81a7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 >> 172.21.15.16:0/659057206 conn(0x7fbdb8073180 msgr2=0x7fbdb81193b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 shutdown_connections 2024-09-12T03:06:02.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:02.161+0000 7fbdbe068640 1 -- 172.21.15.16:0/659057206 wait complete. 2024-09-12T03:06:02.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:06:03.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 129970 26.19 MB/sec execute 184 sec latency 225.070 ms 2024-09-12T03:06:03.257 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:03.516 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:03 smithi016 ceph-mon[30925]: pgmap v881: 97 pgs: 97 active+clean; 684 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 757 op/s 2024-09-12T03:06:03.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:03 smithi033 ceph-mon[35092]: pgmap v881: 97 pgs: 97 active+clean; 684 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 757 op/s 2024-09-12T03:06:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:03 smithi184 ceph-mon[35321]: pgmap v881: 97 pgs: 97 active+clean; 684 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 757 op/s 2024-09-12T03:06:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 -- 172.21.15.16:0/1629047130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6581068d0 msgr2=0x7fa658106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 --2- 172.21.15.16:0/1629047130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6581068d0 0x7fa658106cb0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fa654009930 tx=0x7fa65402a420 comp rx=0 tx=0).stop 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 -- 172.21.15.16:0/1629047130 shutdown_connections 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 --2- 172.21.15.16:0/1629047130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa658101270 0x7fa65810d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 --2- 172.21.15.16:0/1629047130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581008d0 0x7fa658100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 --2- 172.21.15.16:0/1629047130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6581068d0 0x7fa658106cb0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 -- 172.21.15.16:0/1629047130 >> 172.21.15.16:0/1629047130 conn(0x7fa6580fc610 msgr2=0x7fa6580fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 -- 172.21.15.16:0/1629047130 shutdown_connections 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.963+0000 7fa65ec43640 1 -- 172.21.15.16:0/1629047130 wait complete. 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 Processor -- start 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 -- start start 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6581008d0 0x7fa658198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581068d0 0x7fa65819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa658110cf0 con 0x7fa658101270 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa658110b70 con 0x7fa6581068d0 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65ec43640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa658110e70 con 0x7fa6581008d0 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65d440640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65d440640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46110/0 (socket says 172.21.15.16:46110) 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65d440640 1 -- 172.21.15.16:0/3447960174 learned_addr learned my addr 172.21.15.16:0/3447960174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65dc41640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6581008d0 0x7fa658198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.964+0000 7fa65e442640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581068d0 0x7fa65819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 -- 172.21.15.16:0/3447960174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6581008d0 msgr2=0x7fa658198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6581008d0 0x7fa658198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 -- 172.21.15.16:0/3447960174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581068d0 msgr2=0x7fa65819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581068d0 0x7fa65819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 -- 172.21.15.16:0/3447960174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819d2b0 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65d440640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e9f0 tx=0x7fa64800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa64effd640 1 -- 172.21.15.16:0/3447960174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64800cd80 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa64effd640 1 -- 172.21.15.16:0/3447960174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648010910 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65819d5a0 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa64effd640 1 -- 172.21.15.16:0/3447960174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648018990 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.965+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a4e80 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.968+0000 7fa64effd640 1 -- 172.21.15.16:0/3447960174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa648010ac0 con 0x7fa658101270 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.968+0000 7fa66227e740 1 Processor -- start 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.968+0000 7fa66227e740 1 -- start start 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 0x564e25bf9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25c06940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e25b5f2f0 con 0x564e25bf97b0 2024-09-12T03:06:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e25b06cf0 con 0x564e25bfad20 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa66227e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e25b5b2b0 con 0x564e25c04550 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65e442640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 0x564e25bf9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46112/0 (socket says 172.21.15.16:46112) 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 -- 172.21.15.16:0/2206666300 learned_addr learned my addr 172.21.15.16:0/2206666300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65d440640 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25c06940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 -- 172.21.15.16:0/2206666300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 msgr2=0x564e25c06940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25c06940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 -- 172.21.15.16:0/2206666300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 msgr2=0x564e25bf9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 0x564e25bf9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65dc41640 1 -- 172.21.15.16:0/2206666300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e25aeb3a0 con 0x564e25bf97b0 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa65e442640 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 0x564e25bf9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.969+0000 7fa64effd640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 0x7fa6440a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa64effd640 1 -- 172.21.15.16:0/3447960174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa6480cf2a0 con 0x7fa658101270 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa65dc41640 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fa654009930 tx=0x7fa654005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9adb8b4f1df0e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa65e442640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 0x7fa6440a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa643fff640 1 -- 172.21.15.16:0/2206666300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654035040 con 0x564e25bf97b0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa643fff640 1 -- 172.21.15.16:0/2206666300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa654005520 con 0x564e25bf97b0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.970+0000 7fa643fff640 1 -- 172.21.15.16:0/2206666300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6540035f0 con 0x564e25bf97b0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- 172.21.15.16:0/2206666300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 msgr2=0x564e25bf9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa65e442640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 0x7fa6440a50c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa650005fd0 tx=0x7fa650015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fa654009930 tx=0x7fa654005de0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- 172.21.15.16:0/2206666300 shutdown_connections 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25c06940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bf97b0 0x564e25bf9bf0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 --2- 172.21.15.16:0/2206666300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bfad20 0x564e25bf9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- 172.21.15.16:0/2206666300 >> 172.21.15.16:0/2206666300 conn(0x564e25bc2310 msgr2=0x564e25bf5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- 172.21.15.16:0/2206666300 shutdown_connections 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- 172.21.15.16:0/2206666300 wait complete. 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 -- start start 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.971+0000 7fa66227e740 1 Processor -- start 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c03ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c01150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e25c071f0 con 0x564e25c04550 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e25b1a060 con 0x564e25bf97b0 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e25c07370 con 0x564e25bfad20 2024-09-12T03:06:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa66227e740 10 client.0 _ll_register_callbacks cb 0x564e2586d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65e442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65dc41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c03ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65e442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46124/0 (socket says 172.21.15.16:46124) 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65dc41640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c03ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35616/0 (socket says 172.21.15.16:35616) 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65e442640 1 -- 172.21.15.16:0/2216994756 learned_addr learned my addr 172.21.15.16:0/2216994756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.972+0000 7fa65d440640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c01150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 -- 172.21.15.16:0/2216994756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 msgr2=0x564e25c01150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c01150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 -- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 msgr2=0x564e25c03ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c03ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 -- 172.21.15.16:0/2216994756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e258d03c0 con 0x564e25c04550 2024-09-12T03:06:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa65e442640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fa65001f670 tx=0x7fa650004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37db5166600db7b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65002f060 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa65001d690 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa654005520 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa65001d890 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa66227e740 10 client.19659 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564e25c12ce0 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.973+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa654005520 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa65002abe0 con 0x564e25c04550 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 10 client.19659 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 10 client.19659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 20 client.19659 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c141a0 0x564e25c165b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 4 client.19659 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e25b6c860 con 0x564e25c141a0 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa65dc41640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c141a0 0x564e25c165b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa65dc41640 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c141a0 0x564e25c165b0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8259d709270414da server_cookie=48382cab9f19fcc0 in_seq=0 out_seq=0 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.974+0000 7fa6417fa640 10 client.19659 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa6417fa640 1 -- 172.21.15.16:0/2216994756 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e25b6c860 con 0x564e25c141a0 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa6417fa640 10 client.19659 handle_command_reply: tid=0 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 client.19659 shutdown 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 msgr2=0x564e25c016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fa65001f670 tx=0x7fa650004ab0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 shutdown_connections 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c141a0 0x564e25c165b0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25c016b0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c01150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 --2- 172.21.15.16:0/2216994756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c03ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 >> 172.21.15.16:0/2216994756 conn(0x564e25bc2310 msgr2=0x564e25bfbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.975+0000 7fa6417fa640 0 client.19659 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.976+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 shutdown_connections 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.976+0000 7fa66227e740 1 -- 172.21.15.16:0/2216994756 wait complete. 2024-09-12T03:06:03.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.976+0000 7fa66227e740 20 client.19659 trim_cache size 0 max 16384 2024-09-12T03:06:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 Processor -- start 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 -- start start 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25bf5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25bfc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ab1a0 con 0x564e25c04550 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581acce0 con 0x564e25bf97b0 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.991+0000 7fa66227e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ab870 con 0x564e25bfad20 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65d440640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25bfc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65d440640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:39024/0 (socket says 172.21.15.16:39024) 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65d440640 1 -- 172.21.15.16:0/764559702 learned_addr learned my addr 172.21.15.16:0/764559702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65dc41640 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25bf5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 -- 172.21.15.16:0/764559702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 msgr2=0x564e25c104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 -- 172.21.15.16:0/764559702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 msgr2=0x564e25bf5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25bf5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 -- 172.21.15.16:0/764559702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e25bff6c0 con 0x564e25c04550 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65d440640 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.992+0000 7fa65e442640 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25bfc300 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fa6500353d0 tx=0x7fa65000e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d51b367e86dcc71 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.993+0000 7fa6437fe640 1 -- 172.21.15.16:0/764559702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650042020 con 0x564e25c04550 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.993+0000 7fa6437fe640 1 -- 172.21.15.16:0/764559702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa650026040 con 0x564e25c04550 2024-09-12T03:06:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.993+0000 7fa6437fe640 1 -- 172.21.15.16:0/764559702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65003a040 con 0x564e25c04550 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- 172.21.15.16:0/764559702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 msgr2=0x564e25bfc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25bfc300 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fa6500353d0 tx=0x7fa65000e800 comp rx=0 tx=0).stop 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- 172.21.15.16:0/764559702 shutdown_connections 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25c04550 0x564e25bfc300 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25bfad20 0x564e25c104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 --2- 172.21.15.16:0/764559702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25bf5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:03.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- 172.21.15.16:0/764559702 >> 172.21.15.16:0/764559702 conn(0x564e25bc2310 msgr2=0x564e25bf58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- 172.21.15.16:0/764559702 shutdown_connections 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- 172.21.15.16:0/764559702 wait complete. 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 -- start start 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.994+0000 7fa66227e740 1 Processor -- start 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c0cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 0x564e25bfdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ac610 con 0x564e25bfad20 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581abf40 con 0x564e25bf97b0 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581a9180 con 0x564e25c04550 2024-09-12T03:06:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa66227e740 10 client.0 _ll_register_callbacks cb 0x7fa650009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa65e442640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa65e442640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:39040/0 (socket says 172.21.15.16:39040) 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa65e442640 1 -- 172.21.15.16:0/1455588772 learned_addr learned my addr 172.21.15.16:0/1455588772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa65d440640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 0x564e25bfdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.995+0000 7fa65dc41640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c0cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 -- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 msgr2=0x564e25c0cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c0cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 -- 172.21.15.16:0/1455588772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 msgr2=0x564e25bfdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 0x564e25bfdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 -- 172.21.15.16:0/1455588772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e25bfe7b0 con 0x564e25c04550 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65d440640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 0x564e25bfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa65e442640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa650009690 tx=0x7fa650005cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0b9620732cb866 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650042070 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa65000ed90 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.996+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa650026040 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa65003a040 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 10 client.28876 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564e25c87430 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa650026040 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa65003d070 con 0x564e25c04550 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 10 client.28876 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 10 client.28876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 20 client.28876 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:04.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c87a30 0x564e25c89e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 4 client.28876 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e25b66d70 con 0x564e25c87a30 2024-09-12T03:06:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.997+0000 7fa65dc41640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c87a30 0x564e25c89e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.998+0000 7fa6417fa640 10 client.28876 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.998+0000 7fa65dc41640 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c87a30 0x564e25c89e50 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2648caba545dfd server_cookie=ec3ab462534a506c in_seq=0 out_seq=0 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa6417fa640 1 -- 172.21.15.16:0/1455588772 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e25b66d70 con 0x564e25c87a30 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa6417fa640 10 client.28876 handle_command_reply: tid=0 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 client.28876 shutdown 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 msgr2=0x564e25bfe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa650009690 tx=0x7fa650005cd0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 shutdown_connections 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564e25c87a30 0x564e25c89e50 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e25c04550 0x564e25bfe1f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e25bfad20 0x564e25bfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa6417fa640 0 client.28876 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 --2- 172.21.15.16:0/1455588772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e25bf97b0 0x564e25c0cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 >> 172.21.15.16:0/1455588772 conn(0x564e25bc2310 msgr2=0x564e25bf4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 shutdown_connections 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 1 -- 172.21.15.16:0/1455588772 wait complete. 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:03.999+0000 7fa66227e740 20 client.28876 trim_cache size 0 max 16384 2024-09-12T03:06:04.005 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 msgr2=0x7fa6440a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 0x7fa6440a50c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa650005fd0 tx=0x7fa650015040 comp rx=0 tx=0).stop 2024-09-12T03:06:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 msgr2=0x7fa658198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e9f0 tx=0x7fa64800eeb0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 shutdown_connections 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa6581068d0 0x7fa65819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6440a2c00 0x7fa6440a50c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa658101270 0x7fa658198840 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 --2- 172.21.15.16:0/3447960174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6581008d0 0x7fa658198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 >> 172.21.15.16:0/3447960174 conn(0x7fa6580fc610 msgr2=0x7fa65806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 shutdown_connections 2024-09-12T03:06:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:04.000+0000 7fa65ec43640 1 -- 172.21.15.16:0/3447960174 wait complete. 2024-09-12T03:06:04.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 130593 26.21 MB/sec execute 185 sec latency 77.850 ms 2024-09-12T03:06:05.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 131301 26.27 MB/sec execute 186 sec latency 82.247 ms 2024-09-12T03:06:05.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:05 smithi016 ceph-mon[30925]: pgmap v882: 97 pgs: 97 active+clean; 691 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.0 MiB/s wr, 744 op/s 2024-09-12T03:06:05.695 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:06:05.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:06:05.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:05 smithi033 ceph-mon[35092]: pgmap v882: 97 pgs: 97 active+clean; 691 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.0 MiB/s wr, 744 op/s 2024-09-12T03:06:05.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:05 smithi184 ceph-mon[35321]: pgmap v882: 97 pgs: 97 active+clean; 691 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 9.0 MiB/s wr, 744 op/s 2024-09-12T03:06:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:06.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 132082 26.35 MB/sec execute 187 sec latency 38.428 ms 2024-09-12T03:06:06.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 -- 172.21.15.16:0/3082200508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404101de0 msgr2=0x7fa4041021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 --2- 172.21.15.16:0/3082200508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404101de0 0x7fa4041021c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f402a410 comp rx=0 tx=0).stop 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 -- 172.21.15.16:0/3082200508 shutdown_connections 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 --2- 172.21.15.16:0/3082200508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404106f10 0x7fa40410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 --2- 172.21.15.16:0/3082200508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404102790 0x7fa4041067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 --2- 172.21.15.16:0/3082200508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404101de0 0x7fa4041021c0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 -- 172.21.15.16:0/3082200508 >> 172.21.15.16:0/3082200508 conn(0x7fa4040fd8c0 msgr2=0x7fa4040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.781+0000 7fa408e3e640 1 -- 172.21.15.16:0/3082200508 shutdown_connections 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.782+0000 7fa408e3e640 1 -- 172.21.15.16:0/3082200508 wait complete. 2024-09-12T03:06:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.782+0000 7fa408e3e640 1 Processor -- start 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.782+0000 7fa408e3e640 1 -- start start 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404101de0 0x7fa4040744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404106f10 0x7fa404074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa404111f30 con 0x7fa404102790 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa404111db0 con 0x7fa404101de0 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa408e3e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041120b0 con 0x7fa404106f10 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47970/0 (socket says 172.21.15.16:47970) 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 -- 172.21.15.16:0/2874536956 learned_addr learned my addr 172.21.15.16:0/2874536956 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa40259c640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404101de0 0x7fa4040744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404106f10 0x7fa404074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 -- 172.21.15.16:0/2874536956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404106f10 msgr2=0x7fa404074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404106f10 0x7fa404074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 -- 172.21.15.16:0/2874536956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404101de0 msgr2=0x7fa4040744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404101de0 0x7fa4040744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.783+0000 7fa3fbfff640 1 -- 172.21.15.16:0/2874536956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4041a6910 con 0x7fa404102790 2024-09-12T03:06:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00e9f0 tx=0x7fa3ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa3fb7fe640 1 -- 172.21.15.16:0/2874536956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00cd80 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa3fb7fe640 1 -- 172.21.15.16:0/2874536956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec010910 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa3fb7fe640 1 -- 172.21.15.16:0/2874536956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec018980 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4041a6c00 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.784+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4041a70f0 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.787+0000 7fa3fb7fe640 1 -- 172.21.15.16:0/2874536956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa3ec010ac0 con 0x7fa404102790 2024-09-12T03:06:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.787+0000 7fa40b217740 1 Processor -- start 2024-09-12T03:06:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.787+0000 7fa40b217740 1 -- start start 2024-09-12T03:06:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cef130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560092c47280 con 0x560092cecd40 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560092beeb10 con 0x560092ce1b60 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40b217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560092c43240 con 0x560092ce2480 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa40259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47986/0 (socket says 172.21.15.16:47986) 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa3fbfff640 1 -- 172.21.15.16:0/534764126 learned_addr learned my addr 172.21.15.16:0/534764126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa3fb7fe640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 0x7fa3f00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 -- 172.21.15.16:0/534764126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 msgr2=0x560092cec800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cec800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 -- 172.21.15.16:0/534764126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 msgr2=0x560092cef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 -- 172.21.15.16:0/534764126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560092bfb1f0 con 0x560092ce1b60 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 0x7fa3f00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.788+0000 7fa3fb7fe640 1 -- 172.21.15.16:0/2874536956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3ec0cf2c0 con 0x7fa404102790 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.789+0000 7fa402d9d640 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce1f40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009fa0 tx=0x7fa3fc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f0af1dbd2ef4827 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.789+0000 7fa3f8ff9640 1 -- 172.21.15.16:0/534764126 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc017050 con 0x560092ce1b60 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.789+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 0x7fa3f00a50c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc022000 tx=0x7fa3fc023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.789+0000 7fa3f8ff9640 1 -- 172.21.15.16:0/534764126 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3fc003c30 con 0x560092ce1b60 2024-09-12T03:06:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.789+0000 7fa3f8ff9640 1 -- 172.21.15.16:0/534764126 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc01ba30 con 0x560092ce1b60 2024-09-12T03:06:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 -- 172.21.15.16:0/534764126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 msgr2=0x560092ce1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce1f40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009fa0 tx=0x7fa3fc0118d0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 -- 172.21.15.16:0/534764126 shutdown_connections 2024-09-12T03:06:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cec800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 --2- 172.21.15.16:0/534764126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce1f40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 -- 172.21.15.16:0/534764126 >> 172.21.15.16:0/534764126 conn(0x560092cde0f0 msgr2=0x560092cde4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 -- 172.21.15.16:0/534764126 shutdown_connections 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.790+0000 7fa40b217740 1 -- 172.21.15.16:0/534764126 wait complete. 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 -- start start 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 Processor -- start 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560092cef910 con 0x560092cecd40 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560092c01f10 con 0x560092ce1b60 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560092cefa90 con 0x560092ce2480 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40b217740 10 client.0 _ll_register_callbacks cb 0x560092bcaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa402d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33198/0 (socket says 172.21.15.16:33198) 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa40259c640 1 -- 172.21.15.16:0/4083882888 learned_addr learned my addr 172.21.15.16:0/4083882888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.791+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 -- 172.21.15.16:0/4083882888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 msgr2=0x560092cea180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cea180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 -- 172.21.15.16:0/4083882888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 msgr2=0x560092cea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 -- 172.21.15.16:0/4083882888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600929b86a0 con 0x560092ce1b60 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa402d9d640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cea180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40259c640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f40317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44bda951ec78f32a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035040 con 0x560092ce1b60 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560092c547f0 con 0x560092ce1b60 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3f4004260 con 0x560092ce1b60 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa3f4004460 con 0x560092ce1b60 2024-09-12T03:06:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40b217740 10 client.28970 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.792+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560092cfb540 con 0x560092ce1b60 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560092c547f0 con 0x560092ce1b60 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f4046790 con 0x560092ce1b60 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 10 client.28970 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 10 client.28970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 20 client.28970 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092cfc940 0x560092cfeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 4 client.28970 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560092c4ed00 con 0x560092cfc940 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.793+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092cfc940 0x560092cfeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.794+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092cfc940 0x560092cfeda0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a541414e2954c33 server_cookie=97cfce5fb8af2f04 in_seq=0 out_seq=0 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.794+0000 7fa3e9ffb640 10 client.28970 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.794+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/4083882888 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560092c4ed00 con 0x560092cfc940 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.794+0000 7fa3e9ffb640 10 client.28970 handle_command_reply: tid=0 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.794+0000 7fa40b217740 1 client.28970 shutdown 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 msgr2=0x560092ce9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f40317e0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 shutdown_connections 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092cfc940 0x560092cfeda0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092cea180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 --2- 172.21.15.16:0/4083882888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce9c40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa3e9ffb640 0 client.28970 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 >> 172.21.15.16:0/4083882888 conn(0x560092cde0f0 msgr2=0x560092cdd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 shutdown_connections 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 1 -- 172.21.15.16:0/4083882888 wait complete. 2024-09-12T03:06:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.795+0000 7fa40b217740 20 client.28970 trim_cache size 0 max 16384 2024-09-12T03:06:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 Processor -- start 2024-09-12T03:06:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 -- start start 2024-09-12T03:06:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092d046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041af770 con 0x560092cecd40 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041b12b0 con 0x560092ce1b60 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40b217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041afe40 con 0x560092ce2480 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa402d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa402d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48014/0 (socket says 172.21.15.16:48014) 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa402d9d640 1 -- 172.21.15.16:0/2276210185 learned_addr learned my addr 172.21.15.16:0/2276210185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092d046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.818+0000 7fa40259c640 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 -- 172.21.15.16:0/2276210185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 msgr2=0x560092d046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092d046c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 -- 172.21.15.16:0/2276210185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 msgr2=0x560092ce45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 -- 172.21.15.16:0/2276210185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560092cd9830 con 0x560092cecd40 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009e40 tx=0x7fa3fc01b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2763db55fc1b1490 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa3ebfff640 1 -- 172.21.15.16:0/2276210185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc01b760 con 0x560092cecd40 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa3ebfff640 1 -- 172.21.15.16:0/2276210185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3fc01b900 con 0x560092cecd40 2024-09-12T03:06:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.819+0000 7fa3ebfff640 1 -- 172.21.15.16:0/2276210185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc00fa00 con 0x560092cecd40 2024-09-12T03:06:06.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- 172.21.15.16:0/2276210185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 msgr2=0x560092d04c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009e40 tx=0x7fa3fc01b6f0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- 172.21.15.16:0/2276210185 shutdown_connections 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092d04c00 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce2480 0x560092d046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 --2- 172.21.15.16:0/2276210185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce1b60 0x560092ce45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- 172.21.15.16:0/2276210185 >> 172.21.15.16:0/2276210185 conn(0x560092cde0f0 msgr2=0x560092cebbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- 172.21.15.16:0/2276210185 shutdown_connections 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- 172.21.15.16:0/2276210185 wait complete. 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.820+0000 7fa40b217740 1 -- start start 2024-09-12T03:06:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 Processor -- start 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 0x560092cf0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cf0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041b0be0 con 0x560092cecd40 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041b0510 con 0x560092ce2480 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041ad750 con 0x560092ce1b60 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40b217740 10 client.0 _ll_register_callbacks cb 0x7fa3f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58988/0 (socket says 172.21.15.16:58988) 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 0x560092cf0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa40259c640 1 -- 172.21.15.16:0/2679022695 learned_addr learned my addr 172.21.15.16:0/2679022695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.821+0000 7fa402d9d640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cf0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 -- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 msgr2=0x560092cf0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 0x560092cf0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 -- 172.21.15.16:0/2679022695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 msgr2=0x560092cf0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cf0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 -- 172.21.15.16:0/2679022695 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560092cf15c0 con 0x560092ce1b60 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 0x560092cf0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40259c640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4046c30 tx=0x7fa3f4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea3d81892f46508 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035070 con 0x560092ce1b60 2024-09-12T03:06:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3f4005040 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3fc01b900 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa3f4061050 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40b217740 10 client.28894 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.822+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560092d67770 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3fc01b900 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f4004a30 con 0x560092ce1b60 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 10 client.28894 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 10 client.28894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 20 client.28894 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092d67ec0 0x560092d6a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:06.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 4 client.28894 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:06:06.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.823+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560092bcb110 con 0x560092d67ec0 2024-09-12T03:06:06.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.824+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092d67ec0 0x560092d6a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:06.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.824+0000 7fa3e9ffb640 10 client.28894 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:06.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.824+0000 7fa3fbfff640 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092d67ec0 0x560092d6a2e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50329a6613a42c58 server_cookie=59a6ff39dee1d753 in_seq=0 out_seq=0 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa3e9ffb640 1 -- 172.21.15.16:0/2679022695 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560092bcb110 con 0x560092d67ec0 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa3e9ffb640 10 client.28894 handle_command_reply: tid=0 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 client.28894 shutdown 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 msgr2=0x560092cf54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4046c30 tx=0x7fa3f4005c00 comp rx=0 tx=0).stop 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 shutdown_connections 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x560092d67ec0 0x560092d6a2e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560092cecd40 0x560092cf0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa3e9ffb640 0 client.28894 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560092ce2480 0x560092cf0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 --2- 172.21.15.16:0/2679022695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560092ce1b60 0x560092cf54e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 >> 172.21.15.16:0/2679022695 conn(0x560092cde0f0 msgr2=0x560092cdca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 shutdown_connections 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.826+0000 7fa40b217740 1 -- 172.21.15.16:0/2679022695 wait complete. 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa40b217740 20 client.28894 trim_cache size 0 max 16384 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:06:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "316e25ff-0a84-4c90-af26-e3957f9c254e", 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 msgr2=0x7fa3f00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 0x7fa3f00a50c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc022000 tx=0x7fa3fc023000 comp rx=0 tx=0).stop 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 msgr2=0x7fa404074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00e9f0 tx=0x7fa3ec00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 shutdown_connections 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa404106f10 0x7fa404074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3f00a2c00 0x7fa3f00a50c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa404102790 0x7fa404074a30 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 --2- 172.21.15.16:0/2874536956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa404101de0 0x7fa4040744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 >> 172.21.15.16:0/2874536956 conn(0x7fa4040fd8c0 msgr2=0x7fa4040fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 shutdown_connections 2024-09-12T03:06:06.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:06.827+0000 7fa408e3e640 1 -- 172.21.15.16:0/2874536956 wait complete. 2024-09-12T03:06:07.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 132782 26.39 MB/sec execute 188 sec latency 78.569 ms 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: pgmap v883: 97 pgs: 97 active+clean; 692 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 737 op/s 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:07.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.646 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:07.646 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:07 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: pgmap v883: 97 pgs: 97 active+clean; 692 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 737 op/s 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:07.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:07 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: pgmap v883: 97 pgs: 97 active+clean; 692 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 737 op/s 2024-09-12T03:06:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:06:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:07.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:07 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:08.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 133345 26.47 MB/sec execute 189 sec latency 216.447 ms 2024-09-12T03:06:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 -- 172.21.15.16:0/2293562216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc10e500 msgr2=0x7fdbbc110910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/2293562216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc10e500 0x7fdbbc110910 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009ca0 tx=0x7fdbb402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 -- 172.21.15.16:0/2293562216 shutdown_connections 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/2293562216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc10e500 0x7fdbbc110910 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/2293562216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc10df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/2293562216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc100b20 0x7fdbbc100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.629+0000 7fdbc1e50640 1 -- 172.21.15.16:0/2293562216 >> 172.21.15.16:0/2293562216 conn(0x7fdbbc0fc7d0 msgr2=0x7fdbbc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.630+0000 7fdbc1e50640 1 -- 172.21.15.16:0/2293562216 shutdown_connections 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.630+0000 7fdbc1e50640 1 -- 172.21.15.16:0/2293562216 wait complete. 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.630+0000 7fdbc1e50640 1 Processor -- start 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 -- start start 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc10e500 0x7fdbbc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc114250 con 0x7fdbbc100b20 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc1140d0 con 0x7fdbbc101440 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbc1e50640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc1143d0 con 0x7fdbbc10e500 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48022/0 (socket says 172.21.15.16:48022) 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbbaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/818848127 learned_addr learned my addr 172.21.15.16:0/818848127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.631+0000 7fdbbbfff640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc10e500 0x7fdbbc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/818848127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc10e500 msgr2=0x7fdbbc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc10e500 0x7fdbbc19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/818848127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 msgr2=0x7fdbbc19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/818848127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc19f6a0 con 0x7fdbbc100b20 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00e9f0 tx=0x7fdbac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbb8ff9640 1 -- 172.21.15.16:0/818848127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00cd80 con 0x7fdbbc100b20 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbbc19f990 con 0x7fdbbc100b20 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbb8ff9640 1 -- 172.21.15.16:0/818848127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac005500 con 0x7fdbbc100b20 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbb8ff9640 1 -- 172.21.15.16:0/818848127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac010690 con 0x7fdbbc100b20 2024-09-12T03:06:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.632+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbbc1a7270 con 0x7fdbbc100b20 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.635+0000 7fdbb8ff9640 1 -- 172.21.15.16:0/818848127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdbac0046c0 con 0x7fdbbc100b20 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.635+0000 7fdbc4229740 1 Processor -- start 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 -- start start 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 0x55cd3bae2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baed140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baefa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ba482e0 con 0x55cd3bae24a0 2024-09-12T03:06:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd3b9efce0 con 0x55cd3baed680 2024-09-12T03:06:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbc4229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ba442a0 con 0x55cd3bae2dc0 2024-09-12T03:06:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbb8ff9640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 0x7fdba40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:08.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:08 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 0x55cd3bae2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 0x55cd3bae2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48024/0 (socket says 172.21.15.16:48024) 2024-09-12T03:06:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbbfff640 1 -- 172.21.15.16:0/2143156881 learned_addr learned my addr 172.21.15.16:0/2143156881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baefa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbbbfff640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 0x7fdba40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.636+0000 7fdbb8ff9640 1 -- 172.21.15.16:0/818848127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdbac0cf410 con 0x7fdbbc100b20 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.637+0000 7fdbbaffd640 1 -- 172.21.15.16:0/2143156881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 msgr2=0x55cd3baed140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.637+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baed140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.637+0000 7fdbbaffd640 1 -- 172.21.15.16:0/2143156881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 msgr2=0x55cd3bae2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.637+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 0x55cd3bae2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.637+0000 7fdbbaffd640 1 -- 172.21.15.16:0/2143156881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd3b9d4440 con 0x55cd3baed680 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.638+0000 7fdbbbfff640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 0x7fdba40a5040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009a60 tx=0x7fdbb4004a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.638+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baefa70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06a140 tx=0x7fdbb00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7caa4f392ffb6b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.638+0000 7fdba1ffb640 1 -- 172.21.15.16:0/2143156881 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0004220 con 0x55cd3baed680 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.638+0000 7fdba1ffb640 1 -- 172.21.15.16:0/2143156881 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb00043c0 con 0x55cd3baed680 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.638+0000 7fdba1ffb640 1 -- 172.21.15.16:0/2143156881 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb00035f0 con 0x55cd3baed680 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 -- 172.21.15.16:0/2143156881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 msgr2=0x55cd3baefa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baefa70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06a140 tx=0x7fdbb00107a0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 -- 172.21.15.16:0/2143156881 shutdown_connections 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baefa70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2143156881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae24a0 0x55cd3bae2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 -- 172.21.15.16:0/2143156881 >> 172.21.15.16:0/2143156881 conn(0x55cd3badea30 msgr2=0x55cd3badee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 -- 172.21.15.16:0/2143156881 shutdown_connections 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.639+0000 7fdbc4229740 1 -- 172.21.15.16:0/2143156881 wait complete. 2024-09-12T03:06:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 -- start start 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 Processor -- start 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baea280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3baea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cd3baf0250 con 0x55cd3baed680 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ba03050 con 0x55cd3bae24a0 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cd3baf03d0 con 0x55cd3bae2dc0 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbc4229740 10 client.0 _ll_register_callbacks cb 0x55cd3ba43160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33272/0 (socket says 172.21.15.16:33272) 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/1081427081 learned_addr learned my addr 172.21.15.16:0/1081427081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.640+0000 7fdbbbfff640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3baea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baea280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/1081427081 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 msgr2=0x55cd3baea280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baea280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/1081427081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 msgr2=0x55cd3baea7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3baea7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/1081427081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd3b7b93c0 con 0x55cd3bae24a0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baea280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdbac098650 tx=0x7fdbac06acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef3e3e7401fb8d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdb9affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0a4660 con 0x55cd3bae24a0 2024-09-12T03:06:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdbac09b0d0 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb00043c0 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.641+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fdbac09c380 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 10 client.28988 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cd3bafbc80 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb00043c0 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fdbac09ab40 con 0x55cd3bae24a0 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 10 client.28988 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 10 client.28988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 20 client.28988 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bafd090 0x55cd3baff480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 4 client.28988 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.642+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd3ba55850 con 0x55cd3bafd090 2024-09-12T03:06:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.643+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bafd090 0x55cd3baff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.643+0000 7fdb9affd640 10 client.28988 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.643+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bafd090 0x55cd3baff480 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=924def79ac14c73c server_cookie=a9ac8f15d91c17f2 in_seq=0 out_seq=0 2024-09-12T03:06:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.644+0000 7fdb9affd640 1 -- 172.21.15.16:0/1081427081 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cd3ba55850 con 0x55cd3bafd090 2024-09-12T03:06:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.644+0000 7fdb9affd640 10 client.28988 handle_command_reply: tid=0 2024-09-12T03:06:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 client.28988 shutdown 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 msgr2=0x55cd3baecfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdbac098650 tx=0x7fdbac06acc0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 shutdown_connections 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bafd090 0x55cd3baff480 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3baea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3baea280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 --2- 172.21.15.16:0/1081427081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baecfd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 >> 172.21.15.16:0/1081427081 conn(0x55cd3badea30 msgr2=0x55cd3bae4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdb9affd640 0 client.28988 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 shutdown_connections 2024-09-12T03:06:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.645+0000 7fdbc4229740 1 -- 172.21.15.16:0/1081427081 wait complete. 2024-09-12T03:06:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.646+0000 7fdbc4229740 20 client.28988 trim_cache size 0 max 16384 2024-09-12T03:06:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.663+0000 7fdbc4229740 1 Processor -- start 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.663+0000 7fdbc4229740 1 -- start start 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3bae8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc06faf0 con 0x55cd3baed680 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc071630 con 0x55cd3bae24a0 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbc4229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc0701c0 con 0x55cd3bae2dc0 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3bae8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33276/0 (socket says 172.21.15.16:33276) 2024-09-12T03:06:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48042/0 (socket says 172.21.15.16:48042) 2024-09-12T03:06:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/935889189 learned_addr learned my addr 172.21.15.16:0/935889189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 -- 172.21.15.16:0/935889189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 msgr2=0x55cd3baf94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 -- 172.21.15.16:0/935889189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 msgr2=0x55cd3bae57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbaffd640 1 -- 172.21.15.16:0/935889189 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd3bae9230 con 0x55cd3bae2dc0 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbbfff640 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.664+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.665+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3bae8b50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb000b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e22ac691453c15 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.665+0000 7fdba17fa640 1 -- 172.21.15.16:0/935889189 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0004250 con 0x55cd3bae2dc0 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.665+0000 7fdba17fa640 1 -- 172.21.15.16:0/935889189 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb00043f0 con 0x55cd3bae2dc0 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.665+0000 7fdba17fa640 1 -- 172.21.15.16:0/935889189 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0022770 con 0x55cd3bae2dc0 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.665+0000 7fdbc4229740 1 -- 172.21.15.16:0/935889189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 msgr2=0x55cd3bae8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3bae8b50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb000b390 comp rx=0 tx=0).stop 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 -- 172.21.15.16:0/935889189 shutdown_connections 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3baed680 0x55cd3bae57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae2dc0 0x55cd3bae8b50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 --2- 172.21.15.16:0/935889189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3bae24a0 0x55cd3baf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 -- 172.21.15.16:0/935889189 >> 172.21.15.16:0/935889189 conn(0x55cd3badea30 msgr2=0x55cd3bae50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 -- 172.21.15.16:0/935889189 shutdown_connections 2024-09-12T03:06:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 -- 172.21.15.16:0/935889189 wait complete. 2024-09-12T03:06:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 -- start start 2024-09-12T03:06:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.666+0000 7fdbc4229740 1 Processor -- start 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 0x55cd3baf5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 0x55cd3baf0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baf13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc070f60 con 0x55cd3bae2dc0 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc070890 con 0x55cd3baed680 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc06dad0 con 0x55cd3bae24a0 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbc4229740 10 client.0 _ll_register_callbacks cb 0x7fdbac00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 0x55cd3baf0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baf13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 0x55cd3baf5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 0x55cd3baf0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48054/0 (socket says 172.21.15.16:48054) 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbaffd640 1 -- 172.21.15.16:0/2327033308 learned_addr learned my addr 172.21.15.16:0/2327033308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 msgr2=0x55cd3baf13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baf13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/2327033308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 msgr2=0x55cd3baf0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 0x55cd3baf0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 -- 172.21.15.16:0/2327033308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd3baf1a30 con 0x55cd3bae24a0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdbbb7fe640 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 0x55cd3baf5b80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c6a0 tx=0x7fdbac004bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faeae940bd47834e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.667+0000 7fdb9affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac09bc80 con 0x55cd3bae24a0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdbac0a4670 con 0x55cd3bae24a0 2024-09-12T03:06:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb00043f0 con 0x55cd3bae24a0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fdbac082050 con 0x55cd3bae24a0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 10 client.28912 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55cd3baf1a30 con 0x55cd3bae24a0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac0a0030 con 0x55cd3bae24a0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fdbac09b1f0 con 0x55cd3bae24a0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 10 client.28912 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 10 client.28912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 20 client.28912 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bb67f20 0x55cd3bb6a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 4 client.28912 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.668+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fdbb00043f0 con 0x55cd3bb67f20 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.669+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bb67f20 0x55cd3bb6a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.669+0000 7fdb9affd640 10 client.28912 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.669+0000 7fdbbaffd640 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bb67f20 0x55cd3bb6a310 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda9a20060f28981 server_cookie=9ed9d3c0e2556746 in_seq=0 out_seq=0 2024-09-12T03:06:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdb9affd640 1 -- 172.21.15.16:0/2327033308 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fdbb00043f0 con 0x55cd3bb67f20 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdb9affd640 10 client.28912 handle_command_reply: tid=0 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 client.28912 shutdown 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 msgr2=0x55cd3baf5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 0x55cd3baf5b80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c6a0 tx=0x7fdbac004bb0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 shutdown_connections 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cd3bb67f20 0x55cd3bb6a310 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd3baed680 0x55cd3baf13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cd3bae2dc0 0x55cd3baf0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdb9affd640 0 client.28912 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 --2- 172.21.15.16:0/2327033308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cd3bae24a0 0x55cd3baf5b80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 >> 172.21.15.16:0/2327033308 conn(0x55cd3badea30 msgr2=0x55cd3bae4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 shutdown_connections 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 1 -- 172.21.15.16:0/2327033308 wait complete. 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.672+0000 7fdbc4229740 20 client.28912 trim_cache size 0 max 16384 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout: "316e25ff-0a84-4c90-af26-e3957f9c254e": { 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:06:08.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "316e25ff-0a84-4c90-af26-e3957f9c254e", 2024-09-12T03:06:08.677 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:06:08.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:08.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:08.677 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 msgr2=0x7fdba40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 0x7fdba40a5040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009a60 tx=0x7fdbb4004a80 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 msgr2=0x7fdbbc19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00e9f0 tx=0x7fdbac00edf0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 shutdown_connections 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbbc10e500 0x7fdbbc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdbbc101440 0x7fdbbc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdba40a2b80 0x7fdba40a5040 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 --2- 172.21.15.16:0/818848127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc100b20 0x7fdbbc19a6f0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 >> 172.21.15.16:0/818848127 conn(0x7fdbbc0fc7d0 msgr2=0x7fdbbc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 shutdown_connections 2024-09-12T03:06:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:08.673+0000 7fdbc1e50640 1 -- 172.21.15.16:0/818848127 wait complete. 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:08 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:08.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:08 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:09.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 133798 26.47 MB/sec execute 190 sec latency 152.994 ms 2024-09-12T03:06:09.505 INFO:tasks.cephfs.filesystem:scrub status for tag:316e25ff-0a84-4c90-af26-e3957f9c254e - {'path': '/', 'tag': '316e25ff-0a84-4c90-af26-e3957f9c254e', 'options': 'recursive,force'} 2024-09-12T03:06:09.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:09 smithi016 ceph-mon[30925]: pgmap v884: 97 pgs: 97 active+clean; 694 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 608 op/s 2024-09-12T03:06:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:09 smithi033 ceph-mon[35092]: pgmap v884: 97 pgs: 97 active+clean; 694 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 608 op/s 2024-09-12T03:06:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:09 smithi184 ceph-mon[35321]: pgmap v884: 97 pgs: 97 active+clean; 694 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 608 op/s 2024-09-12T03:06:10.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134322 26.46 MB/sec execute 191 sec latency 130.053 ms 2024-09-12T03:06:10.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:10 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:06:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:10 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:06:10.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:10 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:06:11.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 134844 26.45 MB/sec execute 192 sec latency 160.186 ms 2024-09-12T03:06:11.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:11 smithi033 ceph-mon[35092]: pgmap v885: 97 pgs: 97 active+clean; 703 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 766 op/s 2024-09-12T03:06:11.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:11 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:06:11.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:11 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:06:11.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:11 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:06:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:11 smithi184 ceph-mon[35321]: pgmap v885: 97 pgs: 97 active+clean; 703 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 766 op/s 2024-09-12T03:06:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:11 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:06:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:11 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:06:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:11 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:06:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:11 smithi016 ceph-mon[30925]: pgmap v885: 97 pgs: 97 active+clean; 703 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 766 op/s 2024-09-12T03:06:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:11 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:06:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:11 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:06:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:11 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:06:12.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 135511 26.48 MB/sec execute 193 sec latency 225.602 ms 2024-09-12T03:06:13.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 135939 26.46 MB/sec execute 194 sec latency 299.339 ms 2024-09-12T03:06:13.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:13 smithi033 ceph-mon[35092]: pgmap v886: 97 pgs: 97 active+clean; 704 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.9 MiB/s wr, 640 op/s 2024-09-12T03:06:13.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:13 smithi184 ceph-mon[35321]: pgmap v886: 97 pgs: 97 active+clean; 704 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.9 MiB/s wr, 640 op/s 2024-09-12T03:06:13.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:13 smithi016 ceph-mon[30925]: pgmap v886: 97 pgs: 97 active+clean; 704 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.9 MiB/s wr, 640 op/s 2024-09-12T03:06:14.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 136616 26.51 MB/sec execute 195 sec latency 61.206 ms 2024-09-12T03:06:15.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 137314 26.58 MB/sec execute 196 sec latency 221.969 ms 2024-09-12T03:06:15.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:15 smithi033 ceph-mon[35092]: pgmap v887: 97 pgs: 97 active+clean; 709 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.8 MiB/s wr, 692 op/s 2024-09-12T03:06:15.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:15 smithi184 ceph-mon[35321]: pgmap v887: 97 pgs: 97 active+clean; 709 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.8 MiB/s wr, 692 op/s 2024-09-12T03:06:15.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:15 smithi016 ceph-mon[30925]: pgmap v887: 97 pgs: 97 active+clean; 709 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.8 MiB/s wr, 692 op/s 2024-09-12T03:06:16.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 137919 26.64 MB/sec execute 197 sec latency 112.136 ms 2024-09-12T03:06:17.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 138535 26.70 MB/sec execute 198 sec latency 112.840 ms 2024-09-12T03:06:17.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:17 smithi033 ceph-mon[35092]: pgmap v888: 97 pgs: 97 active+clean; 688 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 669 op/s 2024-09-12T03:06:17.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:17 smithi184 ceph-mon[35321]: pgmap v888: 97 pgs: 97 active+clean; 688 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 669 op/s 2024-09-12T03:06:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:17 smithi016 ceph-mon[30925]: pgmap v888: 97 pgs: 97 active+clean; 688 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 669 op/s 2024-09-12T03:06:18.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 138923 26.63 MB/sec execute 199 sec latency 304.715 ms 2024-09-12T03:06:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:06:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:06:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:06:18.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:06:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:06:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:06:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:06:18.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:06:18.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:06:18.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:06:18.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:06:18.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:06:19.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 139327 26.63 MB/sec execute 200 sec latency 214.460 ms 2024-09-12T03:06:19.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:19 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:06:19.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:19 smithi033 ceph-mon[35092]: pgmap v889: 97 pgs: 97 active+clean; 689 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.0 MiB/s wr, 655 op/s 2024-09-12T03:06:19.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:19 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:06:19.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:19 smithi184 ceph-mon[35321]: pgmap v889: 97 pgs: 97 active+clean; 689 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.0 MiB/s wr, 655 op/s 2024-09-12T03:06:19.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:19 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:06:19.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:19 smithi016 ceph-mon[30925]: pgmap v889: 97 pgs: 97 active+clean; 689 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.0 MiB/s wr, 655 op/s 2024-09-12T03:06:20.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 140113 26.66 MB/sec execute 201 sec latency 72.937 ms 2024-09-12T03:06:21.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 140730 26.72 MB/sec execute 202 sec latency 89.166 ms 2024-09-12T03:06:21.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:21 smithi033 ceph-mon[35092]: pgmap v890: 97 pgs: 97 active+clean; 691 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 711 op/s 2024-09-12T03:06:21.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:21 smithi184 ceph-mon[35321]: pgmap v890: 97 pgs: 97 active+clean; 691 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 711 op/s 2024-09-12T03:06:21.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:21 smithi016 ceph-mon[30925]: pgmap v890: 97 pgs: 97 active+clean; 691 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 711 op/s 2024-09-12T03:06:22.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 141204 26.70 MB/sec execute 203 sec latency 153.789 ms 2024-09-12T03:06:23.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 141771 26.72 MB/sec execute 204 sec latency 147.502 ms 2024-09-12T03:06:23.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:23 smithi033 ceph-mon[35092]: pgmap v891: 97 pgs: 97 active+clean; 699 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.0 MiB/s wr, 635 op/s 2024-09-12T03:06:23.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:23 smithi184 ceph-mon[35321]: pgmap v891: 97 pgs: 97 active+clean; 699 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.0 MiB/s wr, 635 op/s 2024-09-12T03:06:23.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:23 smithi016 ceph-mon[30925]: pgmap v891: 97 pgs: 97 active+clean; 699 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.0 MiB/s wr, 635 op/s 2024-09-12T03:06:24.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 142418 26.78 MB/sec execute 205 sec latency 97.103 ms 2024-09-12T03:06:25.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 142962 26.80 MB/sec execute 206 sec latency 73.813 ms 2024-09-12T03:06:25.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:25 smithi033 ceph-mon[35092]: pgmap v892: 97 pgs: 97 active+clean; 700 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 650 op/s 2024-09-12T03:06:25.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:25 smithi184 ceph-mon[35321]: pgmap v892: 97 pgs: 97 active+clean; 700 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 650 op/s 2024-09-12T03:06:25.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:25 smithi016 ceph-mon[30925]: pgmap v892: 97 pgs: 97 active+clean; 700 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 650 op/s 2024-09-12T03:06:26.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 143510 26.78 MB/sec execute 207 sec latency 207.051 ms 2024-09-12T03:06:27.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 144263 26.87 MB/sec execute 208 sec latency 41.068 ms 2024-09-12T03:06:27.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:27 smithi033 ceph-mon[35092]: pgmap v893: 97 pgs: 97 active+clean; 705 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 614 op/s 2024-09-12T03:06:27.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:27 smithi184 ceph-mon[35321]: pgmap v893: 97 pgs: 97 active+clean; 705 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 614 op/s 2024-09-12T03:06:27.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:27 smithi016 ceph-mon[30925]: pgmap v893: 97 pgs: 97 active+clean; 705 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 614 op/s 2024-09-12T03:06:28.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 144911 26.89 MB/sec execute 209 sec latency 180.392 ms 2024-09-12T03:06:29.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 145398 26.91 MB/sec execute 210 sec latency 170.578 ms 2024-09-12T03:06:29.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:29 smithi033 ceph-mon[35092]: pgmap v894: 97 pgs: 97 active+clean; 706 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.3 MiB/s wr, 584 op/s 2024-09-12T03:06:29.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:29 smithi184 ceph-mon[35321]: pgmap v894: 97 pgs: 97 active+clean; 706 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.3 MiB/s wr, 584 op/s 2024-09-12T03:06:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:29 smithi016 ceph-mon[30925]: pgmap v894: 97 pgs: 97 active+clean; 706 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.3 MiB/s wr, 584 op/s 2024-09-12T03:06:30.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 146106 26.97 MB/sec execute 211 sec latency 68.954 ms 2024-09-12T03:06:31.004 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T03:06:31.005 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.8 2024-09-12T03:06:31.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:06:31.020 INFO:tasks.workunit.client.0.smithi016.stdout: 10 146800 27.01 MB/sec execute 212 sec latency 41.394 ms 2024-09-12T03:06:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:31 smithi016 ceph-mon[30925]: pgmap v895: 97 pgs: 97 active+clean; 715 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-09-12T03:06:31.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:31 smithi033 ceph-mon[35092]: pgmap v895: 97 pgs: 97 active+clean; 715 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-09-12T03:06:31.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:31 smithi184 ceph-mon[35321]: pgmap v895: 97 pgs: 97 active+clean; 715 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 -- 172.21.15.16:0/768560030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 msgr2=0x7f57301006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 --2- 172.21.15.16:0/768560030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301006d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f572c009930 tx=0x7f572c02a420 comp rx=0 tx=0).stop 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 -- 172.21.15.16:0/768560030 shutdown_connections 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 --2- 172.21.15.16:0/768560030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730106f50 0x7f573010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 --2- 172.21.15.16:0/768560030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730100ca0 0x7f5730106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 --2- 172.21.15.16:0/768560030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301006d0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 -- 172.21.15.16:0/768560030 >> 172.21.15.16:0/768560030 conn(0x7f5730077210 msgr2=0x7f57300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 -- 172.21.15.16:0/768560030 shutdown_connections 2024-09-12T03:06:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.837+0000 7f57371e8640 1 -- 172.21.15.16:0/768560030 wait complete. 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 Processor -- start 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 -- start start 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 0x7f57301989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730106f50 0x7f573019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5730111ff0 con 0x7f57301002f0 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5730111e70 con 0x7f5730106f50 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.838+0000 7f57371e8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5730112170 con 0x7f5730100ca0 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54456/0 (socket says 172.21.15.16:54456) 2024-09-12T03:06:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 -- 172.21.15.16:0/1204185290 learned_addr learned my addr 172.21.15.16:0/1204185290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5735785640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730106f50 0x7f573019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5727fff640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 0x7f57301989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 -- 172.21.15.16:0/1204185290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 msgr2=0x7f57301989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 0x7f57301989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 -- 172.21.15.16:0/1204185290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730106f50 msgr2=0x7f573019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730106f50 0x7f573019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 -- 172.21.15.16:0/1204185290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573019d460 con 0x7f57301002f0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5727fff640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 0x7f57301989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5734f84640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f572c02a940 tx=0x7f572c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c03f070 con 0x7f57301002f0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572c031c90 con 0x7f57301002f0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573019d6f0 con 0x7f57301002f0 2024-09-12T03:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.839+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c035030 con 0x7f57301002f0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.840+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57301a5030 con 0x7f57301002f0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.841+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57301033b0 con 0x7f57301002f0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.843+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f572c042070 con 0x7f57301002f0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.844+0000 7f5725ffb640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 0x7f57180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.844+0000 7f5727fff640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 0x7f57180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.844+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f572c0efdd0 con 0x7f57301002f0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.844+0000 7f5727fff640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 0x7f57180a5040 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f57300ff7f0 tx=0x7f5720005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:31.848+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f572c0bc6f0 con 0x7f57301002f0 2024-09-12T03:06:32.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 147397 27.04 MB/sec execute 213 sec latency 65.804 ms 2024-09-12T03:06:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.104+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57301006d0 con 0x7f57301002f0 2024-09-12T03:06:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.105+0000 7f5725ffb640 1 -- 172.21.15.16:0/1204185290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f572c0c09f0 con 0x7f57301002f0 2024-09-12T03:06:32.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:06:32.105 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:06:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:06:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 msgr2=0x7f57180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 0x7f57180a5040 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f57300ff7f0 tx=0x7f5720005f90 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 msgr2=0x7f57301984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f572c02a940 tx=0x7f572c003b20 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 shutdown_connections 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5730106f50 0x7f573019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5730100ca0 0x7f57301989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f57180a2b80 0x7f57180a5040 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 --2- 172.21.15.16:0/1204185290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57301002f0 0x7f57301984b0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.107+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 >> 172.21.15.16:0/1204185290 conn(0x7f5730077210 msgr2=0x7f57301051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.108+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 shutdown_connections 2024-09-12T03:06:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:32.108+0000 7f57371e8640 1 -- 172.21.15.16:0/1204185290 wait complete. 2024-09-12T03:06:32.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1204185290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:32.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1204185290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:32.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1204185290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:33.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 147883 27.01 MB/sec execute 214 sec latency 216.186 ms 2024-09-12T03:06:33.518 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0' 2024-09-12T03:06:33.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0 2024-09-12T03:06:33.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:33 smithi033 ceph-mon[35092]: pgmap v896: 97 pgs: 97 active+clean; 716 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:06:33.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:33 smithi184 ceph-mon[35321]: pgmap v896: 97 pgs: 97 active+clean; 716 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:06:33.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:33 smithi016 ceph-mon[30925]: pgmap v896: 97 pgs: 97 active+clean; 716 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:06:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:34.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148444 27.01 MB/sec execute 215 sec latency 147.174 ms 2024-09-12T03:06:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 -- 172.21.15.16:0/3756186888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0101de0 msgr2=0x7fd0f01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 --2- 172.21.15.16:0/3756186888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0101de0 0x7fd0f01021c0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009900 tx=0x7fd0e802a410 comp rx=0 tx=0).stop 2024-09-12T03:06:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 -- 172.21.15.16:0/3756186888 shutdown_connections 2024-09-12T03:06:34.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 --2- 172.21.15.16:0/3756186888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0106f10 0x7fd0f010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 --2- 172.21.15.16:0/3756186888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 --2- 172.21.15.16:0/3756186888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0101de0 0x7fd0f01021c0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 -- 172.21.15.16:0/3756186888 >> 172.21.15.16:0/3756186888 conn(0x7fd0f00fd8c0 msgr2=0x7fd0f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 -- 172.21.15.16:0/3756186888 shutdown_connections 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.634+0000 7fd0f746d640 1 -- 172.21.15.16:0/3756186888 wait complete. 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 Processor -- start 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 -- start start 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 0x7fd0f0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0106f10 0x7fd0f019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0111f30 con 0x7fd0f0106f10 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0111db0 con 0x7fd0f0102790 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f746d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01120b0 con 0x7fd0f0101de0 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f5a0a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0106f10 0x7fd0f019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 0x7fd0f0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f4a08640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f4a08640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38888/0 (socket says 172.21.15.16:38888) 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 0x7fd0f0198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42226/0 (socket says 172.21.15.16:42226) 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.635+0000 7fd0f4a08640 1 -- 172.21.15.16:0/1523806947 learned_addr learned my addr 172.21.15.16:0/1523806947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 -- 172.21.15.16:0/1523806947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 msgr2=0x7fd0f0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 0x7fd0f0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 -- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0106f10 msgr2=0x7fd0f019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0106f10 0x7fd0f019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 -- 172.21.15.16:0/1523806947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f019d3b0 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9f0 tx=0x7fd0e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0ee7fc640 1 -- 172.21.15.16:0/1523806947 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400cd80 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0ee7fc640 1 -- 172.21.15.16:0/1523806947 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e4010910 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f019d6a0 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0ee7fc640 1 -- 172.21.15.16:0/1523806947 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4018990 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.636+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f01a4f80 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.638+0000 7fd0ee7fc640 1 -- 172.21.15.16:0/1523806947 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd0e4010ac0 con 0x7fd0f0102790 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 Processor -- start 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 -- start start 2024-09-12T03:06:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 0x555eca4f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 0x555eca503400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca505d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555eca45e2a0 con 0x555eca4f8760 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555eca405b10 con 0x555eca503940 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f9846740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555eca45a260 con 0x555eca4f9080 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0ee7fc640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 0x7fd0d80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 0x555eca503400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f5a0a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 0x555eca4f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 0x555eca503400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42238/0 (socket says 172.21.15.16:42238) 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f5209640 1 -- 172.21.15.16:0/2400740782 learned_addr learned my addr 172.21.15.16:0/2400740782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 0x7fd0d80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.639+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca505d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0ee7fc640 1 -- 172.21.15.16:0/1523806947 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd0e40cf3a0 con 0x7fd0f0102790 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/2400740782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 msgr2=0x555eca503400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 0x555eca503400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/2400740782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 msgr2=0x555eca505d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca505d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/2400740782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eca1cfb90 con 0x555eca4f8760 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca505d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 0x555eca4f8b40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0005fd0 tx=0x7fd0e0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8819b1b67d95705 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 0x7fd0d80a5040 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0010c40 tx=0x7fd0e0017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0d77fe640 1 -- 172.21.15.16:0/2400740782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0024070 con 0x555eca4f8760 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0d77fe640 1 -- 172.21.15.16:0/2400740782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009c50 con 0x555eca4f8760 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.640+0000 7fd0d77fe640 1 -- 172.21.15.16:0/2400740782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0005280 con 0x555eca4f8760 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 -- 172.21.15.16:0/2400740782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 msgr2=0x555eca4f8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 0x555eca4f8b40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0005fd0 tx=0x7fd0e0010740 comp rx=0 tx=0).stop 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 -- 172.21.15.16:0/2400740782 shutdown_connections 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca505d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f9080 0x555eca503400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 --2- 172.21.15.16:0/2400740782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f8760 0x555eca4f8b40 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 -- 172.21.15.16:0/2400740782 >> 172.21.15.16:0/2400740782 conn(0x555eca4f4cf0 msgr2=0x555eca4f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 -- 172.21.15.16:0/2400740782 shutdown_connections 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.641+0000 7fd0f9846740 1 -- 172.21.15.16:0/2400740782 wait complete. 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 -- start start 2024-09-12T03:06:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 Processor -- start 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 0x555eca500880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca500dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca503940 0x555eca501300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555eca506560 con 0x555eca4f9080 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555eca3e66e0 con 0x555eca4f8760 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.642+0000 7fd0f9846740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555eca5066e0 con 0x555eca503940 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 0x555eca500880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f9846740 10 client.0 _ll_register_callbacks cb 0x555eca40aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 0x555eca500880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38898/0 (socket says 172.21.15.16:38898) 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f5209640 1 -- 172.21.15.16:0/3914970198 learned_addr learned my addr 172.21.15.16:0/3914970198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca500dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca503940 0x555eca501300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 -- 172.21.15.16:0/3914970198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca503940 msgr2=0x555eca501300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca503940 0x555eca501300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 -- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 msgr2=0x555eca500880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 0x555eca500880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.643+0000 7fd0f4a08640 1 -- 172.21.15.16:0/3914970198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eca50fad0 con 0x555eca4f9080 2024-09-12T03:06:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca500dc0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40cce20 tx=0x7fd0e406aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae591ec000bc5b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4018d30 con 0x555eca4f9080 2024-09-12T03:06:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd0e409b0b0 con 0x555eca4f9080 2024-09-12T03:06:34.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e0009c50 con 0x555eca4f9080 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd0e409ff60 con 0x555eca4f9080 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0f9846740 10 client.19719 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.644+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555eca512460 con 0x555eca4f9080 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009c50 con 0x555eca4f9080 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd0e4097ad0 con 0x555eca4f9080 2024-09-12T03:06:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 10 client.19719 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 10 client.19719 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 20 client.19719 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca513870 0x555eca515c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 4 client.19719 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555eca46b9d0 con 0x555eca513870 2024-09-12T03:06:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.645+0000 7fd0f5209640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca513870 0x555eca515c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.647+0000 7fd0d4ff9640 10 client.19719 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.647+0000 7fd0f5209640 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca513870 0x555eca515c60 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8eecca207c1cde5 server_cookie=a6b5852f13a4d55 in_seq=0 out_seq=0 2024-09-12T03:06:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.647+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/3914970198 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555eca46b9d0 con 0x555eca513870 2024-09-12T03:06:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.647+0000 7fd0d4ff9640 10 client.19719 handle_command_reply: tid=0 2024-09-12T03:06:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.647+0000 7fd0f9846740 1 client.19719 shutdown 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 msgr2=0x555eca500dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca500dc0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40cce20 tx=0x7fd0e406aca0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 shutdown_connections 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca513870 0x555eca515c60 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca503940 0x555eca501300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca500dc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 --2- 172.21.15.16:0/3914970198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f8760 0x555eca500880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0d4ff9640 0 client.19719 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 >> 172.21.15.16:0/3914970198 conn(0x555eca4f4cf0 msgr2=0x555eca4f3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 shutdown_connections 2024-09-12T03:06:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 1 -- 172.21.15.16:0/3914970198 wait complete. 2024-09-12T03:06:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.648+0000 7fd0f9846740 20 client.19719 trim_cache size 0 max 16384 2024-09-12T03:06:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.666+0000 7fd0f9846740 1 Processor -- start 2024-09-12T03:06:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.666+0000 7fd0f9846740 1 -- start start 2024-09-12T03:06:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca4f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca51ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ab340 con 0x555eca4f9080 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ace80 con 0x555eca503940 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f9846740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01aba10 con 0x555eca4f8760 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f4a08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f4a08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54508/0 (socket says 172.21.15.16:54508) 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f4a08640 1 -- 172.21.15.16:0/451105744 learned_addr learned my addr 172.21.15.16:0/451105744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca51ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5209640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca4f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/451105744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 msgr2=0x555eca4f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca4f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/451105744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 msgr2=0x555eca51b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/451105744 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eca4f0260 con 0x555eca503940 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.667+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.668+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca51ba50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0012b40 tx=0x7fd0e0012e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a385d070be5a86f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.668+0000 7fd0d6ffd640 1 -- 172.21.15.16:0/451105744 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e002c030 con 0x555eca503940 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.668+0000 7fd0d6ffd640 1 -- 172.21.15.16:0/451105744 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009c50 con 0x555eca503940 2024-09-12T03:06:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.668+0000 7fd0d6ffd640 1 -- 172.21.15.16:0/451105744 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0024070 con 0x555eca503940 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 -- 172.21.15.16:0/451105744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 msgr2=0x555eca51ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca51ba50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0012b40 tx=0x7fd0e0012e70 comp rx=0 tx=0).stop 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 -- 172.21.15.16:0/451105744 shutdown_connections 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca503940 0x555eca51ba50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca4f9080 0x555eca51b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 --2- 172.21.15.16:0/451105744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca4f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 -- 172.21.15.16:0/451105744 >> 172.21.15.16:0/451105744 conn(0x555eca4f4cf0 msgr2=0x555eca4f3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 -- 172.21.15.16:0/451105744 shutdown_connections 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.669+0000 7fd0f9846740 1 -- 172.21.15.16:0/451105744 wait complete. 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 -- start start 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 Processor -- start 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 0x555eca5087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 0x555eca5764a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ac7b0 con 0x555eca503940 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ac0e0 con 0x555eca4f9080 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a9320 con 0x555eca4f8760 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f9846740 10 client.0 _ll_register_callbacks cb 0x7fd0e409bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.670+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f4a08640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 0x555eca5087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5209640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42268/0 (socket says 172.21.15.16:42268) 2024-09-12T03:06:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5209640 1 -- 172.21.15.16:0/746905646 learned_addr learned my addr 172.21.15.16:0/746905646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 0x555eca5764a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/746905646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 msgr2=0x555eca508270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 msgr2=0x555eca5087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 0x555eca5087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 -- 172.21.15.16:0/746905646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eca4f5da0 con 0x555eca503940 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5a0a640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 0x555eca5764a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00096e0 tx=0x7fd0e0009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b897175d2ca46410 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0024070 con 0x555eca503940 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f5209640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f4a08640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 0x555eca5087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd0e000ddc0 con 0x555eca503940 2024-09-12T03:06:34.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.671+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e0009c50 con 0x555eca503940 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd0e002c070 con 0x555eca503940 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 10 client.19725 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555eca57ee60 con 0x555eca503940 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009c50 con 0x555eca503940 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd0e0020430 con 0x555eca503940 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 10 client.19725 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 10 client.19725 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 20 client.19725 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca57f3e0 0x555eca5817d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 4 client.19725 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}] 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}) -- 0x555eca465d20 con 0x555eca57f3e0 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.672+0000 7fd0f5209640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca57f3e0 0x555eca5817d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:34.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.673+0000 7fd0f5209640 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca57f3e0 0x555eca5817d0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9686808356a6d675 server_cookie=efe5ee89018ad6d0 in_seq=0 out_seq=0 2024-09-12T03:06:34.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:34.673+0000 7fd0d4ff9640 10 client.19725 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:35.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148741 26.96 MB/sec execute 216 sec latency 317.747 ms 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0d4ff9640 1 -- 172.21.15.16:0/746905646 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555eca465d20 con 0x555eca57f3e0 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0d4ff9640 10 client.19725 handle_command_reply: tid=0 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 client.19725 shutdown 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 msgr2=0x555eca5764a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 0x555eca5764a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00096e0 tx=0x7fd0e0009440 comp rx=0 tx=0).stop 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 shutdown_connections 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555eca57f3e0 0x555eca5817d0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555eca503940 0x555eca5764a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555eca4f9080 0x555eca5087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 --2- 172.21.15.16:0/746905646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555eca4f8760 0x555eca508270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 >> 172.21.15.16:0/746905646 conn(0x555eca4f4cf0 msgr2=0x555eca4f3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.460+0000 7fd0d4ff9640 0 client.19725 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.461+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 shutdown_connections 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.461+0000 7fd0f9846740 1 -- 172.21.15.16:0/746905646 wait complete. 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.461+0000 7fd0f9846740 20 client.19725 trim_cache size 0 max 16384 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:06:35.462 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 66, 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "ec120ce2": { 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 66, 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:06:35.463 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:06:35.464 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:06:35.465 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:06:35.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:06:35.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:06:35.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:35.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:35.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:35.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:35.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:35.466 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.461+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 msgr2=0x7fd0d80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 0x7fd0d80a5040 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0010c40 tx=0x7fd0e0017000 comp rx=0 tx=0).stop 2024-09-12T03:06:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 msgr2=0x7fd0f0198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9f0 tx=0x7fd0e400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 shutdown_connections 2024-09-12T03:06:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd0d80a2b80 0x7fd0d80a5040 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0f0106f10 0x7fd0f019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0f0102790 0x7fd0f0198940 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 --2- 172.21.15.16:0/1523806947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f0101de0 0x7fd0f0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 >> 172.21.15.16:0/1523806947 conn(0x7fd0f00fd8c0 msgr2=0x7fd0f00fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 shutdown_connections 2024-09-12T03:06:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:35.462+0000 7fd0f746d640 1 -- 172.21.15.16:0/1523806947 wait complete. 2024-09-12T03:06:35.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:35 smithi033 ceph-mon[35092]: pgmap v897: 97 pgs: 97 active+clean; 720 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 606 op/s 2024-09-12T03:06:35.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:35 smithi184 ceph-mon[35321]: pgmap v897: 97 pgs: 97 active+clean; 720 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 606 op/s 2024-09-12T03:06:35.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:35 smithi016 ceph-mon[30925]: pgmap v897: 97 pgs: 97 active+clean; 720 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 606 op/s 2024-09-12T03:06:36.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.83 MB/sec execute 217 sec latency 1307.007 ms 2024-09-12T03:06:36.267 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ec120ce2', quiesce took 5.3 seconds. Will release after: 16.5 2024-09-12T03:06:37.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.71 MB/sec execute 218 sec latency 2307.100 ms 2024-09-12T03:06:37.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:37 smithi033 ceph-mon[35092]: pgmap v898: 97 pgs: 97 active+clean; 709 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 681 op/s 2024-09-12T03:06:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:37 smithi184 ceph-mon[35321]: pgmap v898: 97 pgs: 97 active+clean; 709 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 681 op/s 2024-09-12T03:06:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:37 smithi016 ceph-mon[30925]: pgmap v898: 97 pgs: 97 active+clean; 709 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 681 op/s 2024-09-12T03:06:38.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.59 MB/sec execute 219 sec latency 3307.148 ms 2024-09-12T03:06:39.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.47 MB/sec execute 220 sec latency 4307.188 ms 2024-09-12T03:06:39.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:06:39.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:39 smithi033 ceph-mon[35092]: pgmap v899: 97 pgs: 97 active+clean; 705 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-09-12T03:06:39.768 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:39.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:39 smithi184 ceph-mon[35321]: pgmap v899: 97 pgs: 97 active+clean; 705 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-09-12T03:06:39.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:39 smithi016 ceph-mon[30925]: pgmap v899: 97 pgs: 97 active+clean; 705 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-09-12T03:06:40.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.35 MB/sec execute 221 sec latency 5307.270 ms 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 -- 172.21.15.16:0/4045249592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c106a80 msgr2=0x7f369c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 --2- 172.21.15.16:0/4045249592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c106a80 0x7f369c106e60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f368c009930 tx=0x7f368c02a420 comp rx=0 tx=0).stop 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 -- 172.21.15.16:0/4045249592 shutdown_connections 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 --2- 172.21.15.16:0/4045249592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c101420 0x7f369c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 --2- 172.21.15.16:0/4045249592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c100a80 0x7f369c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 --2- 172.21.15.16:0/4045249592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c106a80 0x7f369c106e60 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 -- 172.21.15.16:0/4045249592 >> 172.21.15.16:0/4045249592 conn(0x7f369c0fc7a0 msgr2=0x7f369c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 -- 172.21.15.16:0/4045249592 shutdown_connections 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.237+0000 7f36a1051640 1 -- 172.21.15.16:0/4045249592 wait complete. 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 Processor -- start 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 -- start start 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c101420 0x7f369c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c106a80 0x7f369c1a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f369c110ea0 con 0x7f369c100a80 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f369c110d20 con 0x7f369c101420 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f36a1051640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f369c111020 con 0x7f369c106a80 2024-09-12T03:06:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f369ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f369ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53128/0 (socket says 172.21.15.16:53128) 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f369ad9d640 1 -- 172.21.15.16:0/2743719888 learned_addr learned my addr 172.21.15.16:0/2743719888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369b59e640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c106a80 0x7f369c1a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.238+0000 7f3693fff640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c101420 0x7f369c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 -- 172.21.15.16:0/2743719888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c106a80 msgr2=0x7f369c1a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c106a80 0x7f369c1a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 -- 172.21.15.16:0/2743719888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c101420 msgr2=0x7f369c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c101420 0x7f369c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 -- 172.21.15.16:0/2743719888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c1a48b0 con 0x7f369c100a80 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f368c009a50 tx=0x7f368c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f3698d99640 1 -- 172.21.15.16:0/2743719888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c03f030 con 0x7f369c100a80 2024-09-12T03:06:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f3698d99640 1 -- 172.21.15.16:0/2743719888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f368c003ad0 con 0x7f369c100a80 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f369c1a4a80 con 0x7f369c100a80 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f3698d99640 1 -- 172.21.15.16:0/2743719888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c035070 con 0x7f369c100a80 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.239+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f369c1a4ed0 con 0x7f369c100a80 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.241+0000 7f3698d99640 1 -- 172.21.15.16:0/2743719888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f368c042030 con 0x7f369c100a80 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 Processor -- start 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 -- start start 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a579740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a588550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56546a4df2e0 con 0x56546a588a90 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56546a486ce0 con 0x56546a579360 2024-09-12T03:06:40.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.242+0000 7f36a342a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56546a4db2a0 con 0x56546a579c80 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3698d99640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 0x7f36880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53132/0 (socket says 172.21.15.16:53132) 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 -- 172.21.15.16:0/393257917 learned_addr learned my addr 172.21.15.16:0/393257917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f369b59e640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a579740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f369b59e640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 0x7f36880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3698d99640 1 -- 172.21.15.16:0/2743719888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f368c0ef7c0 con 0x7f369c100a80 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f369ad9d640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a588550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 -- 172.21.15.16:0/393257917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 msgr2=0x56546a588550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a588550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 -- 172.21.15.16:0/393257917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 msgr2=0x56546a579740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a579740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 -- 172.21.15.16:0/393257917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546a46b440 con 0x56546a588a90 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f369b59e640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a579740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f3693fff640 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f3684009a40 tx=0x7f3684009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3972745caca1f81c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f369b59e640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 0x7f36880a5040 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3694005c40 tx=0x7f3694007520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f36917fa640 1 -- 172.21.15.16:0/393257917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684017070 con 0x56546a588a90 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f36917fa640 1 -- 172.21.15.16:0/393257917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3684013470 con 0x56546a588a90 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.243+0000 7f36917fa640 1 -- 172.21.15.16:0/393257917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684012630 con 0x56546a588a90 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 -- 172.21.15.16:0/393257917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 msgr2=0x56546a58ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f3684009a40 tx=0x7f3684009f00 comp rx=0 tx=0).stop 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 -- 172.21.15.16:0/393257917 shutdown_connections 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a58ae80 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a588550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 --2- 172.21.15.16:0/393257917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a579740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 -- 172.21.15.16:0/393257917 >> 172.21.15.16:0/393257917 conn(0x56546a575a30 msgr2=0x56546a575e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 -- 172.21.15.16:0/393257917 shutdown_connections 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.244+0000 7f36a342a740 1 -- 172.21.15.16:0/393257917 wait complete. 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 -- start start 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 Processor -- start 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579360 0x56546a588440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5856f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56546a58b660 con 0x56546a588a90 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56546a49a050 con 0x56546a579c80 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56546a58b7e0 con 0x56546a579360 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f36a342a740 10 client.0 _ll_register_callbacks cb 0x56546a4da160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53148/0 (socket says 172.21.15.16:53148) 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 -- 172.21.15.16:0/2845259752 learned_addr learned my addr 172.21.15.16:0/2845259752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f3693fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5856f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579360 0x56546a588440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 -- 172.21.15.16:0/2845259752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579360 msgr2=0x56546a588440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579360 0x56546a588440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 -- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 msgr2=0x56546a5856f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5856f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.245+0000 7f369b59e640 1 -- 172.21.15.16:0/2845259752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546a2503c0 con 0x56546a588a90 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f369b59e640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f3694011b80 tx=0x7f3694004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86302f06f2008236 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694019070 con 0x56546a588a90 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f369400f650 con 0x56546a588a90 2024-09-12T03:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3684013470 con 0x56546a588a90 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f369400f850 con 0x56546a588a90 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 10 client.19737 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56546a597160 con 0x56546a588a90 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3684013470 con 0x56546a588a90 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3694028a90 con 0x56546a588a90 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 10 client.19737 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 10 client.19737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 20 client.19737 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a5987b0 0x56546a59aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 4 client.19737 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.246+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56546a4ec850 con 0x56546a5987b0 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.247+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a5987b0 0x56546a59aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.247+0000 7f369ad9d640 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a5987b0 0x56546a59aba0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c2c22513a8cb82 server_cookie=83512660fc1b0d48 in_seq=0 out_seq=0 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.247+0000 7f36827fc640 10 client.19737 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36827fc640 1 -- 172.21.15.16:0/2845259752 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56546a4ec850 con 0x56546a5987b0 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36827fc640 10 client.19737 handle_command_reply: tid=0 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 client.19737 shutdown 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 msgr2=0x56546a585c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f3694011b80 tx=0x7f3694004a40 comp rx=0 tx=0).stop 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 shutdown_connections 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a5987b0 0x56546a59aba0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a585c90 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5856f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 --2- 172.21.15.16:0/2845259752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579360 0x56546a588440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36827fc640 0 client.19737 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:40.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 >> 172.21.15.16:0/2845259752 conn(0x56546a575a30 msgr2=0x56546a57b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 shutdown_connections 2024-09-12T03:06:40.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 1 -- 172.21.15.16:0/2845259752 wait complete. 2024-09-12T03:06:40.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.248+0000 7f36a342a740 20 client.19737 trim_cache size 0 max 16384 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.265+0000 7f36a342a740 1 Processor -- start 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.265+0000 7f36a342a740 1 -- start start 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a588a90 0x56546a5a0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1ab260 con 0x56546a579360 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1acda0 con 0x56546a579c80 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f36a342a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1ab930 con 0x56546a588a90 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53158/0 (socket says 172.21.15.16:53158) 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 -- 172.21.15.16:0/653781011 learned_addr learned my addr 172.21.15.16:0/653781011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f3693fff640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369b59e640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a588a90 0x56546a5a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 -- 172.21.15.16:0/653781011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a588a90 msgr2=0x56546a5a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a588a90 0x56546a5a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 -- 172.21.15.16:0/653781011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 msgr2=0x56546a5a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 -- 172.21.15.16:0/653781011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546a583c40 con 0x56546a579360 2024-09-12T03:06:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f3693fff640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f369ad9d640 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f368c0ed270 tx=0x7f368c0314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecdca81f61ed9e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f3690ff9640 1 -- 172.21.15.16:0/653781011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c049040 con 0x56546a579360 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f3690ff9640 1 -- 172.21.15.16:0/653781011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f368c0c4c40 con 0x56546a579360 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.266+0000 7f3690ff9640 1 -- 172.21.15.16:0/653781011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c003c80 con 0x56546a579360 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 -- 172.21.15.16:0/653781011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 msgr2=0x56546a594900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f368c0ed270 tx=0x7f368c0314b0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 -- 172.21.15.16:0/653781011 shutdown_connections 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a588a90 0x56546a5a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579c80 0x56546a5a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 --2- 172.21.15.16:0/653781011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a579360 0x56546a594900 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 -- 172.21.15.16:0/653781011 >> 172.21.15.16:0/653781011 conn(0x56546a575a30 msgr2=0x56546a583fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 -- 172.21.15.16:0/653781011 shutdown_connections 2024-09-12T03:06:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.267+0000 7f36a342a740 1 -- 172.21.15.16:0/653781011 wait complete. 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 -- start start 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 Processor -- start 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a58d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a58d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1ac6d0 con 0x56546a588a90 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1ac000 con 0x56546a579360 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1a9240 con 0x56546a579c80 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f36a342a740 10 client.0 _ll_register_callbacks cb 0x7f3694007ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f369b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f369b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53164/0 (socket says 172.21.15.16:53164) 2024-09-12T03:06:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f369b59e640 1 -- 172.21.15.16:0/1944753335 learned_addr learned my addr 172.21.15.16:0/1944753335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f369ad9d640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a58d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.268+0000 7f3693fff640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a58d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 -- 172.21.15.16:0/1944753335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 msgr2=0x56546a58d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a58d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 -- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 msgr2=0x56546a58d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a58d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 -- 172.21.15.16:0/1944753335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546a590dd0 con 0x56546a588a90 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369ad9d640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a58d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f369b59e640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3694028e00 tx=0x7f3694013580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb546b14d9368702 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36827fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694019050 con 0x56546a588a90 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f369400f450 con 0x56546a588a90 2024-09-12T03:06:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f368c0c4c40 con 0x56546a588a90 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3694004400 con 0x56546a588a90 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36a342a740 10 client.19743 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56546a603d50 con 0x56546a588a90 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.269+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f368c0c4c40 con 0x56546a588a90 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f369403b020 con 0x56546a588a90 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 10 client.19743 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 10 client.19743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 20 client.19743 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a604440 0x56546a606860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 4 client.19743 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56546a4e6d60 con 0x56546a604440 2024-09-12T03:06:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f369ad9d640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a604440 0x56546a606860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36827fc640 10 client.19743 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f369ad9d640 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a604440 0x56546a606860 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f7ca0dc607013d server_cookie=51e3022255b7a011 in_seq=0 out_seq=0 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36827fc640 1 -- 172.21.15.16:0/1944753335 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56546a4e6d60 con 0x56546a604440 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36827fc640 10 client.19743 handle_command_reply: tid=0 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.270+0000 7f36a342a740 1 client.19743 shutdown 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 msgr2=0x56546a590730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3694028e00 tx=0x7f3694013580 comp rx=0 tx=0).stop 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 shutdown_connections 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56546a604440 0x56546a606860 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36827fc640 0 client.19743 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56546a588a90 0x56546a590730 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56546a579c80 0x56546a58d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 --2- 172.21.15.16:0/1944753335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56546a579360 0x56546a58d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 >> 172.21.15.16:0/1944753335 conn(0x56546a575a30 msgr2=0x56546a57ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 shutdown_connections 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 1 -- 172.21.15.16:0/1944753335 wait complete. 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a342a740 20 client.19743 trim_cache size 0 max 16384 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:06:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:06:40.283 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 msgr2=0x7f36880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 0x7f36880a5040 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3694005c40 tx=0x7f3694007520 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 msgr2=0x7f369c1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f368c009a50 tx=0x7f368c004270 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 shutdown_connections 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f369c106a80 0x7f369c1a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f369c101420 0x7f369c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f36880a2b80 0x7f36880a5040 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 --2- 172.21.15.16:0/2743719888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f369c100a80 0x7f369c1992f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.271+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 >> 172.21.15.16:0/2743719888 conn(0x7f369c0fc7a0 msgr2=0x7f369c0fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.272+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 shutdown_connections 2024-09-12T03:06:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:40.272+0000 7f36a1051640 1 -- 172.21.15.16:0/2743719888 wait complete. 2024-09-12T03:06:41.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.23 MB/sec execute 222 sec latency 6307.360 ms 2024-09-12T03:06:41.537 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:06:41.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:06:41.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:41 smithi033 ceph-mon[35092]: pgmap v900: 97 pgs: 97 active+clean; 706 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.0 MiB/s wr, 618 op/s 2024-09-12T03:06:41.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:41 smithi184 ceph-mon[35321]: pgmap v900: 97 pgs: 97 active+clean; 706 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.0 MiB/s wr, 618 op/s 2024-09-12T03:06:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:41 smithi016 ceph-mon[30925]: pgmap v900: 97 pgs: 97 active+clean; 706 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.0 MiB/s wr, 618 op/s 2024-09-12T03:06:42.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.11 MB/sec execute 223 sec latency 7307.405 ms 2024-09-12T03:06:42.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- 172.21.15.16:0/2985103962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 msgr2=0x7fa2ac103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/2985103962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac103170 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009cd0 tx=0x7fa29c02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- 172.21.15.16:0/2985103962 shutdown_connections 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/2985103962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 0x7fa2ac10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/2985103962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 0x7fa2ac107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/2985103962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac103170 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- 172.21.15.16:0/2985103962 >> 172.21.15.16:0/2985103962 conn(0x7fa2ac0fc7d0 msgr2=0x7fa2ac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- 172.21.15.16:0/2985103962 shutdown_connections 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- 172.21.15.16:0/2985103962 wait complete. 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 Processor -- start 2024-09-12T03:06:42.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.227+0000 7fa2b41aa640 1 -- start start 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 0x7fa2ac198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 0x7fa2ac19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1120d0 con 0x7fa2ac102d90 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac111f50 con 0x7fa2ac103740 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b41aa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac112250 con 0x7fa2ac108290 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53178/0 (socket says 172.21.15.16:53178) 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 -- 172.21.15.16:0/3658115921 learned_addr learned my addr 172.21.15.16:0/3658115921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1745640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 0x7fa2ac198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b2747640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 0x7fa2ac19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 -- 172.21.15.16:0/3658115921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 msgr2=0x7fa2ac19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:42.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 0x7fa2ac19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 -- 172.21.15.16:0/3658115921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 msgr2=0x7fa2ac198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 0x7fa2ac198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 -- 172.21.15.16:0/3658115921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac19d520 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.228+0000 7fa2b1f46640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa29c02ae00 tx=0x7fa29c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.229+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c004240 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.229+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29c0305a0 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.229+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac19d7b0 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.229+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c03a660 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.229+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac1a50f0 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.230+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac06a7f0 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.231+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa29c030ac0 con 0x7fa2ac102d90 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.231+0000 7fa2aaffd640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 0x7fa2940a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.232+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa29c02e030 con 0x7fa2ac102d90 2024-09-12T03:06:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.232+0000 7fa2b1745640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 0x7fa2940a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.232+0000 7fa2b1745640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 0x7fa2940a50c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0fed80 tx=0x7fa2a000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.236+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa29c0098b0 con 0x7fa2ac102d90 2024-09-12T03:06:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.454+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2ac10a2a0 con 0x7fa2ac102d90 2024-09-12T03:06:42.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.457+0000 7fa2aaffd640 1 -- 172.21.15.16:0/3658115921 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fa29c0bd1b0 con 0x7fa2ac102d90 2024-09-12T03:06:42.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:06:42.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:06:42.457 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.459+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 msgr2=0x7fa2940a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.459+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 0x7fa2940a50c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0fed80 tx=0x7fa2a000a2d0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 msgr2=0x7fa2ac198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa29c02ae00 tx=0x7fa29c003ba0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 shutdown_connections 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2ac108290 0x7fa2ac19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2ac103740 0x7fa2ac198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa2940a2c00 0x7fa2940a50c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 --2- 172.21.15.16:0/3658115921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2ac102d90 0x7fa2ac198570 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 >> 172.21.15.16:0/3658115921 conn(0x7fa2ac0fc7d0 msgr2=0x7fa2ac106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 shutdown_connections 2024-09-12T03:06:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:42.460+0000 7fa2b41aa640 1 -- 172.21.15.16:0/3658115921 wait complete. 2024-09-12T03:06:43.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 26.00 MB/sec execute 224 sec latency 8307.443 ms 2024-09-12T03:06:43.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:06:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:43.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:43 smithi016 ceph-mon[30925]: pgmap v901: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.9 MiB/s wr, 461 op/s 2024-09-12T03:06:43.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:43 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3658115921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:43.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:43 smithi033 ceph-mon[35092]: pgmap v901: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.9 MiB/s wr, 461 op/s 2024-09-12T03:06:43.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3658115921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:43 smithi184 ceph-mon[35321]: pgmap v901: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.9 MiB/s wr, 461 op/s 2024-09-12T03:06:43.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:43 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3658115921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:44.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.88 MB/sec execute 225 sec latency 9307.488 ms 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.033+0000 7f54bc9eb640 1 -- 172.21.15.16:0/2735862261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8100e20 msgr2=0x7f54b8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.033+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/2735862261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8100e20 0x7f54b8101200 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009900 tx=0x7f54a802a480 comp rx=0 tx=0).stop 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 -- 172.21.15.16:0/2735862261 shutdown_connections 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/2735862261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8102ef0 0x7f54b810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/2735862261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/2735862261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8100e20 0x7f54b8101200 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 -- 172.21.15.16:0/2735862261 >> 172.21.15.16:0/2735862261 conn(0x7f54b80fc7d0 msgr2=0x7f54b80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 -- 172.21.15.16:0/2735862261 shutdown_connections 2024-09-12T03:06:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.034+0000 7f54bc9eb640 1 -- 172.21.15.16:0/2735862261 wait complete. 2024-09-12T03:06:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 Processor -- start 2024-09-12T03:06:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 -- start start 2024-09-12T03:06:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 0x7f54b81992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8110f50 con 0x7f54b8102ef0 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8110dd0 con 0x7f54b8102570 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54bc9eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81110d0 con 0x7f54b8100e20 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 0x7f54b81992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53208/0 (socket says 172.21.15.16:53208) 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38010/0 (socket says 172.21.15.16:38010) 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.035+0000 7f54b6d9d640 1 -- 172.21.15.16:0/1343600163 learned_addr learned my addr 172.21.15.16:0/1343600163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 -- 172.21.15.16:0/1343600163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 msgr2=0x7f54b8199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 -- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 msgr2=0x7f54b81a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 -- 172.21.15.16:0/1343600163 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b81a48a0 con 0x7f54b8100e20 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54b659c640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 0x7f54b81992e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005940 tx=0x7f54a8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54a77fe640 1 -- 172.21.15.16:0/1343600163 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a803f070 con 0x7f54b8100e20 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54a77fe640 1 -- 172.21.15.16:0/1343600163 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a8031c30 con 0x7f54b8100e20 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b81a4a70 con 0x7f54b8100e20 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54a77fe640 1 -- 172.21.15.16:0/1343600163 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8035030 con 0x7f54b8100e20 2024-09-12T03:06:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.036+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b81a4ec0 con 0x7f54b8100e20 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.038+0000 7f54a77fe640 1 -- 172.21.15.16:0/1343600163 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f54a8042070 con 0x7f54b8100e20 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 Processor -- start 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 -- start start 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bbe2182f0 con 0x560bbe2b3b40 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560bbe1bfcf0 con 0x560bbe2c1780 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54bedc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560bbe2142b0 con 0x560bbe2b25d0 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53210/0 (socket says 172.21.15.16:53210) 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54b6d9d640 1 -- 172.21.15.16:0/2292287983 learned_addr learned my addr 172.21.15.16:0/2292287983 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54b659c640 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.039+0000 7f54a77fe640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 0x7f54a00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 -- 172.21.15.16:0/2292287983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 msgr2=0x560bbe2b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 -- 172.21.15.16:0/2292287983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 msgr2=0x560bbe2c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 -- 172.21.15.16:0/2292287983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bbe1a43a0 con 0x560bbe2b3b40 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54a77fe640 1 -- 172.21.15.16:0/1343600163 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f54a80ef700 con 0x7f54b8100e20 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 0x7f54a00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f54b00118e0 tx=0x7f54b0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39065400b91f7f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54a4ff9640 1 -- 172.21.15.16:0/2292287983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0014c00 con 0x560bbe2b3b40 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54a4ff9640 1 -- 172.21.15.16:0/2292287983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54b0014da0 con 0x560bbe2b3b40 2024-09-12T03:06:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.040+0000 7f54a4ff9640 1 -- 172.21.15.16:0/2292287983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b001c670 con 0x560bbe2b3b40 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- 172.21.15.16:0/2292287983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 msgr2=0x560bbe2b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f54b00118e0 tx=0x7f54b0011da0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 0x7f54a00a5040 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f54b0022000 tx=0x7f54b001f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- 172.21.15.16:0/2292287983 shutdown_connections 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2292287983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2b2090 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- 172.21.15.16:0/2292287983 >> 172.21.15.16:0/2292287983 conn(0x560bbe27b310 msgr2=0x560bbe2aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- 172.21.15.16:0/2292287983 shutdown_connections 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- 172.21.15.16:0/2292287983 wait complete. 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.041+0000 7f54bedc4740 1 -- start start 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 Processor -- start 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2bea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bbe2c4480 con 0x560bbe2b3b40 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560bbe1d3060 con 0x560bbe2c1780 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560bbe2c4600 con 0x560bbe2b25d0 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54bedc4740 10 client.0 _ll_register_callbacks cb 0x560bbdf268c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53222/0 (socket says 172.21.15.16:53222) 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b5d9b640 1 -- 172.21.15.16:0/960584917 learned_addr learned my addr 172.21.15.16:0/960584917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 -- 172.21.15.16:0/960584917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 msgr2=0x560bbe2bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 -- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 msgr2=0x560bbe2be450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b659c640 1 -- 172.21.15.16:0/960584917 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bbdf893c0 con 0x560bbe2b25d0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.042+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54b659c640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c11a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f54a80bbc50 tx=0x7f54a80f3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb2dce660bdea8ac server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8049070 con 0x560bbe2b25d0 2024-09-12T03:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f54a803b460 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b0014da0 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f54a80bb8b0 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 10 client.28960 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560bbe2c4530 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54b0014da0 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f54a80b85c0 con 0x560bbe2b25d0 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 10 client.28960 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 10 client.28960 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 20 client.28960 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe2d1460 0x560bbe2d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 4 client.28960 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.043+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bbe225860 con 0x560bbe2d1460 2024-09-12T03:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe2d1460 0x560bbe2d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe2d1460 0x560bbe2d38b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=56d6773058fda531 server_cookie=3e2953469d902a96 in_seq=0 out_seq=0 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f549dffb640 10 client.28960 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f549dffb640 1 -- 172.21.15.16:0/960584917 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560bbe225860 con 0x560bbe2d1460 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f549dffb640 10 client.28960 handle_command_reply: tid=0 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.044+0000 7f54bedc4740 1 client.28960 shutdown 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 msgr2=0x560bbe2c11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c11a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f54a80bbc50 tx=0x7f54a80f3040 comp rx=0 tx=0).stop 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 shutdown_connections 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe2d1460 0x560bbe2d38b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2be450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 --2- 172.21.15.16:0/960584917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c11a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f549dffb640 0 client.28960 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 >> 172.21.15.16:0/960584917 conn(0x560bbe27b310 msgr2=0x560bbe2b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 shutdown_connections 2024-09-12T03:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 1 -- 172.21.15.16:0/960584917 wait complete. 2024-09-12T03:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.045+0000 7f54bedc4740 20 client.28960 trim_cache size 0 max 16384 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 Processor -- start 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 -- start start 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ab220 con 0x560bbe2b3b40 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81acd60 con 0x560bbe2c1780 2024-09-12T03:06:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54bedc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ab8f0 con 0x560bbe2b25d0 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53234/0 (socket says 172.21.15.16:53234) 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b5d9b640 1 -- 172.21.15.16:0/2468362869 learned_addr learned my addr 172.21.15.16:0/2468362869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b659c640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.058+0000 7f54b5d9b640 1 -- 172.21.15.16:0/2468362869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 msgr2=0x560bbe2b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b5d9b640 1 -- 172.21.15.16:0/2468362869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 msgr2=0x560bbe2ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b5d9b640 1 -- 172.21.15.16:0/2468362869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bbe2d3f00 con 0x560bbe2b3b40 2024-09-12T03:06:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b659c640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00b010 tx=0x7f54ac00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d40b0fe8f470cf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f549ffff640 1 -- 172.21.15.16:0/2468362869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00fcd0 con 0x560bbe2b3b40 2024-09-12T03:06:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f549ffff640 1 -- 172.21.15.16:0/2468362869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54ac00fe70 con 0x560bbe2b3b40 2024-09-12T03:06:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.059+0000 7f549ffff640 1 -- 172.21.15.16:0/2468362869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac026400 con 0x560bbe2b3b40 2024-09-12T03:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- 172.21.15.16:0/2468362869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 msgr2=0x560bbe2cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00b010 tx=0x7f54ac00a2d0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- 172.21.15.16:0/2468362869 shutdown_connections 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2cd750 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 --2- 172.21.15.16:0/2468362869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- 172.21.15.16:0/2468362869 >> 172.21.15.16:0/2468362869 conn(0x560bbe27b310 msgr2=0x560bbe2a9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- 172.21.15.16:0/2468362869 shutdown_connections 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- 172.21.15.16:0/2468362869 wait complete. 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 -- start start 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.060+0000 7f54bedc4740 1 Processor -- start 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ac690 con 0x560bbe2b3b40 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81abfc0 con 0x560bbe2c1780 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81a9220 con 0x560bbe2b25d0 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 10 client.0 _ll_register_callbacks cb 0x7f54a8009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53240/0 (socket says 172.21.15.16:53240) 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 -- 172.21.15.16:0/3266066189 learned_addr learned my addr 172.21.15.16:0/3266066189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b659c640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b6d9d640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 -- 172.21.15.16:0/3266066189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 msgr2=0x560bbe2c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 -- 172.21.15.16:0/3266066189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 msgr2=0x560bbe2c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 -- 172.21.15.16:0/3266066189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bbe2c4360 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54b5d9b640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005cd0 tx=0x7f54ac00fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b647e6017ed7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f549dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00fcc0 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f54ac004fa0 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.061+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54ac00fe70 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f54ac0051a0 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 10 client.19767 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560bbe33bbe0 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54ac00fe70 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f54ac031420 con 0x560bbe2b3b40 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 10 client.19767 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 10 client.19767 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 20 client.19767 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe33c330 0x560bbe33e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 4 client.19767 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bbe21fd70 con 0x560bbe33c330 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54b659c640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe33c330 0x560bbe33e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f549dffb640 10 client.19767 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.062+0000 7f54b659c640 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe33c330 0x560bbe33e750 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8bb822e61d182358 server_cookie=2c6c29833082b9bb in_seq=0 out_seq=0 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f549dffb640 1 -- 172.21.15.16:0/3266066189 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bbe21fd70 con 0x560bbe33c330 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f549dffb640 10 client.19767 handle_command_reply: tid=0 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 client.19767 shutdown 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 msgr2=0x560bbe2ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005cd0 tx=0x7f54ac00fad0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 shutdown_connections 2024-09-12T03:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x560bbe33c330 0x560bbe33e750 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560bbe2c1780 0x560bbe2c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bbe2b3b40 0x560bbe2ca190 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f549dffb640 0 client.19767 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 --2- 172.21.15.16:0/3266066189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560bbe2b25d0 0x560bbe2c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 >> 172.21.15.16:0/3266066189 conn(0x560bbe27b310 msgr2=0x560bbe2b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 shutdown_connections 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 1 -- 172.21.15.16:0/3266066189 wait complete. 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.063+0000 7f54bedc4740 20 client.19767 trim_cache size 0 max 16384 2024-09-12T03:06:44.069 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 msgr2=0x7f54a00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 0x7f54a00a5040 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f54b0022000 tx=0x7f54b001f000 comp rx=0 tx=0).stop 2024-09-12T03:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 msgr2=0x7f54b81992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 0x7f54b81992e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005940 tx=0x7f54a8003c10 comp rx=0 tx=0).stop 2024-09-12T03:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 shutdown_connections 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f54a00a2b80 0x7f54a00a5040 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b8102ef0 0x7f54b81a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f54b8102570 0x7f54b8199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 --2- 172.21.15.16:0/1343600163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54b8100e20 0x7f54b81992e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 >> 172.21.15.16:0/1343600163 conn(0x7f54b80fc7d0 msgr2=0x7f54b8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 shutdown_connections 2024-09-12T03:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:44.064+0000 7f54bc9eb640 1 -- 172.21.15.16:0/1343600163 wait complete. 2024-09-12T03:06:44.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:06:45.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.77 MB/sec execute 226 sec latency 10307.554 ms 2024-09-12T03:06:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:45.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:45 smithi016 ceph-mon[30925]: pgmap v902: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.0 MiB/s wr, 405 op/s 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 -- 172.21.15.16:0/1661446318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 msgr2=0x7fa3d410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/1661446318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d410da40 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009900 tx=0x7fa3cc02a480 comp rx=0 tx=0).stop 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 -- 172.21.15.16:0/1661446318 shutdown_connections 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/1661446318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d410da40 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/1661446318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3d4102570 0x7fa3d41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/1661446318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4100e20 0x7fa3d4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 -- 172.21.15.16:0/1661446318 >> 172.21.15.16:0/1661446318 conn(0x7fa3d40fc7d0 msgr2=0x7fa3d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 -- 172.21.15.16:0/1661446318 shutdown_connections 2024-09-12T03:06:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.608+0000 7fa3d8eab640 1 -- 172.21.15.16:0/1661446318 wait complete. 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 Processor -- start 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 -- start start 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3d4100e20 0x7fa3d41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4102570 0x7fa3d4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4111020 con 0x7fa3d4102ef0 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4110ea0 con 0x7fa3d4102570 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d8eab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41111a0 con 0x7fa3d4100e20 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33672/0 (socket says 172.21.15.16:33672) 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 -- 172.21.15.16:0/3959245011 learned_addr learned my addr 172.21.15.16:0/3959245011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4102570 0x7fa3d4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 -- 172.21.15.16:0/3959245011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3d4100e20 msgr2=0x7fa3d41985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3d4100e20 0x7fa3d41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 -- 172.21.15.16:0/3959245011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4102570 msgr2=0x7fa3d4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4102570 0x7fa3d4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.609+0000 7fa3d2d9d640 1 -- 172.21.15.16:0/3959245011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d419d590 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3d2d9d640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005ab0 tx=0x7fa3cc003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3c37fe640 1 -- 172.21.15.16:0/3959245011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc03f070 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d419d820 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3c37fe640 1 -- 172.21.15.16:0/3959245011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3cc031e30 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3c37fe640 1 -- 172.21.15.16:0/3959245011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc035030 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.610+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d41a5160 con 0x7fa3d4102ef0 2024-09-12T03:06:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3c37fe640 1 -- 172.21.15.16:0/3959245011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa3cc042070 con 0x7fa3d4102ef0 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 Processor -- start 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 -- start start 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509e7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d5094d2f0 con 0x557d509f2370 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d508f4cf0 con 0x557d509e75d0 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.613+0000 7fa3db284740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557d509492b0 con 0x557d509e8b40 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33686/0 (socket says 172.21.15.16:33686) 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/1711466011 learned_addr learned my addr 172.21.15.16:0/1711466011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509e7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34800/0 (socket says 172.21.15.16:34800) 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3c37fe640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 0x7fa3bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/1711466011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 msgr2=0x557d509e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509e7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/1711466011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 msgr2=0x557d509e7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509e7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/1711466011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d508d93a0 con 0x557d509f2370 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3c37fe640 1 -- 172.21.15.16:0/3959245011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3cc0ef9d0 con 0x7fa3d4102ef0 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d2d9d640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 0x7fa3bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8016530 tx=0x7fa3c800a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549f8920194a60a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3c0ff9640 1 -- 172.21.15.16:0/1711466011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8018430 con 0x557d509f2370 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3c0ff9640 1 -- 172.21.15.16:0/1711466011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c8018ab0 con 0x557d509f2370 2024-09-12T03:06:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.614+0000 7fa3c0ff9640 1 -- 172.21.15.16:0/1711466011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8017a40 con 0x557d509f2370 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 -- 172.21.15.16:0/1711466011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 msgr2=0x557d509f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3d2d9d640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 0x7fa3bc0a5040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005820 tx=0x7fa3cc02a480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8016530 tx=0x7fa3c800a2b0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 -- 172.21.15.16:0/1711466011 shutdown_connections 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d509f4760 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 --2- 172.21.15.16:0/1711466011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 -- 172.21.15.16:0/1711466011 >> 172.21.15.16:0/1711466011 conn(0x557d509b0310 msgr2=0x557d509e3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 -- 172.21.15.16:0/1711466011 shutdown_connections 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.615+0000 7fa3db284740 1 -- 172.21.15.16:0/1711466011 wait complete. 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 -- start start 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 Processor -- start 2024-09-12T03:06:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ef610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d509f4fb0 con 0x557d509e8b40 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557d50908060 con 0x557d509f2370 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557d509f5130 con 0x557d509e75d0 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3db284740 10 client.0 _ll_register_callbacks cb 0x557d5065b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33700/0 (socket says 172.21.15.16:33700) 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 -- 172.21.15.16:0/15018443 learned_addr learned my addr 172.21.15.16:0/15018443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 -- 172.21.15.16:0/15018443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 msgr2=0x557d509f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 -- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 msgr2=0x557d509ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ef610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.616+0000 7fa3d259c640 1 -- 172.21.15.16:0/15018443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d506be3c0 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3d259c640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400a9f0 tx=0x7fa3c400aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4229899fe16fb111 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4018020 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3c400ebd0 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c8018ab0 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa3c400edd0 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 10 client.19779 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557d50a00b00 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c8018ab0 con 0x557d509e8b40 2024-09-12T03:06:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa3c4014660 con 0x557d509e8b40 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 10 client.19779 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 10 client.19779 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 20 client.19779 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.617+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a02060 0x557d50a04480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.618+0000 7fa3db284740 4 client.19779 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.618+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d5095a860 con 0x557d50a02060 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.618+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a02060 0x557d50a04480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.618+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a02060 0x557d50a04480 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b29d2b028e737aa0 server_cookie=a4451ee0daca8415 in_seq=0 out_seq=0 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.618+0000 7fa3b9ffb640 10 client.19779 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/15018443 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557d5095a860 con 0x557d50a02060 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3b9ffb640 10 client.19779 handle_command_reply: tid=0 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 client.19779 shutdown 2024-09-12T03:06:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 msgr2=0x557d509ef020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400a9f0 tx=0x7fa3c400aeb0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 shutdown_connections 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a02060 0x557d50a04480 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ef020 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3b9ffb640 0 client.19779 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 --2- 172.21.15.16:0/15018443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 >> 172.21.15.16:0/15018443 conn(0x557d509b0310 msgr2=0x557d509e9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 shutdown_connections 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 1 -- 172.21.15.16:0/15018443 wait complete. 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.619+0000 7fa3db284740 20 client.19779 trim_cache size 0 max 16384 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 Processor -- start 2024-09-12T03:06:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 -- start start 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509fe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509ee020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41ad5d0 con 0x557d509f2370 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41af110 con 0x557d509e75d0 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.632+0000 7fa3db284740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41adca0 con 0x557d509e8b40 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33706/0 (socket says 172.21.15.16:33706) 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/2041552792 learned_addr learned my addr 172.21.15.16:0/2041552792 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509fe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d259c640 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/2041552792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 msgr2=0x557d509ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509ee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/2041552792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 msgr2=0x557d509fe250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509fe250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 -- 172.21.15.16:0/2041552792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d509e3de0 con 0x557d509f2370 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c800a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3859ef5a4fb4efb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3bbfff640 1 -- 172.21.15.16:0/2041552792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8034070 con 0x557d509f2370 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3bbfff640 1 -- 172.21.15.16:0/2041552792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c8018800 con 0x557d509f2370 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.633+0000 7fa3bbfff640 1 -- 172.21.15.16:0/2041552792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8026430 con 0x557d509f2370 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- 172.21.15.16:0/2041552792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 msgr2=0x557d50a09d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c800a2b0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- 172.21.15.16:0/2041552792 shutdown_connections 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509f2370 0x557d50a09d30 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e8b40 0x557d509ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 --2- 172.21.15.16:0/2041552792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509e75d0 0x557d509fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- 172.21.15.16:0/2041552792 >> 172.21.15.16:0/2041552792 conn(0x557d509b0310 msgr2=0x557d509ed910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- 172.21.15.16:0/2041552792 shutdown_connections 2024-09-12T03:06:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- 172.21.15.16:0/2041552792 wait complete. 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 -- start start 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.634+0000 7fa3db284740 1 Processor -- start 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509fab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41aea40 con 0x557d509e8b40 2024-09-12T03:06:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41ae370 con 0x557d509f2370 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41ab5d0 con 0x557d509e75d0 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3db284740 10 client.0 _ll_register_callbacks cb 0x7fa3c4009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33722/0 (socket says 172.21.15.16:33722) 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 -- 172.21.15.16:0/1334180984 learned_addr learned my addr 172.21.15.16:0/1334180984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509fab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 -- 172.21.15.16:0/1334180984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 msgr2=0x557d509fab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509fab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 -- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 msgr2=0x557d509ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 -- 172.21.15.16:0/1334180984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d509ecb40 con 0x557d509e8b40 2024-09-12T03:06:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3d259c640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4002230 tx=0x7fa3c4014690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ecae49ab9d5527 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.635+0000 7fa3b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c402d020 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3c4028690 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c8018800 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa3c4018040 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 10 client.19785 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557d50a6cd60 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c8018800 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa3c401f6c0 con 0x557d509e8b40 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 10 client.19785 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 10 client.19785 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 20 client.19785 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a6d450 0x557d50a6f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 4 client.19785 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d50954d70 con 0x557d50a6d450 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.636+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a6d450 0x557d50a6f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3d1d9b640 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a6d450 0x557d50a6f870 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d46c1a4038b20287 server_cookie=47700de7fd830cf0 in_seq=0 out_seq=0 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3b9ffb640 10 client.19785 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3b9ffb640 1 -- 172.21.15.16:0/1334180984 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d50954d70 con 0x557d50a6d450 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3b9ffb640 10 client.19785 handle_command_reply: tid=0 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 client.19785 shutdown 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 msgr2=0x557d509ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4002230 tx=0x7fa3c4014690 comp rx=0 tx=0).stop 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 shutdown_connections 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x557d50a6d450 0x557d50a6f870 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557d509f2370 0x557d509ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d509e8b40 0x557d509ebdd0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3b9ffb640 0 client.19785 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 --2- 172.21.15.16:0/1334180984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557d509e75d0 0x557d509fab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 >> 172.21.15.16:0/1334180984 conn(0x557d509b0310 msgr2=0x557d509ee130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 shutdown_connections 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 1 -- 172.21.15.16:0/1334180984 wait complete. 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.637+0000 7fa3db284740 20 client.19785 trim_cache size 0 max 16384 2024-09-12T03:06:45.653 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 msgr2=0x7fa3bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 0x7fa3bc0a5040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005820 tx=0x7fa3cc02a480 comp rx=0 tx=0).stop 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 msgr2=0x7fa3d419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005ab0 tx=0x7fa3cc003c30 comp rx=0 tx=0).stop 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 shutdown_connections 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3bc0a2b80 0x7fa3bc0a5040 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d4102ef0 0x7fa3d419ceb0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3d4102570 0x7fa3d4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 --2- 172.21.15.16:0/3959245011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3d4100e20 0x7fa3d41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 >> 172.21.15.16:0/3959245011 conn(0x7fa3d40fc7d0 msgr2=0x7fa3d406a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 shutdown_connections 2024-09-12T03:06:45.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:45.638+0000 7fa3d8eab640 1 -- 172.21.15.16:0/3959245011 wait complete. 2024-09-12T03:06:45.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:45 smithi033 ceph-mon[35092]: pgmap v902: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.0 MiB/s wr, 405 op/s 2024-09-12T03:06:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:45 smithi184 ceph-mon[35321]: pgmap v902: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.0 MiB/s wr, 405 op/s 2024-09-12T03:06:46.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.65 MB/sec execute 227 sec latency 11307.607 ms 2024-09-12T03:06:46.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:06:46.580 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:47.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.54 MB/sec execute 228 sec latency 12307.636 ms 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 -- 172.21.15.16:0/1339498392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8106f00 msgr2=0x7f26b810ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 --2- 172.21.15.16:0/1339498392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8106f00 0x7f26b810ea80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009be0 tx=0x7f26b002a6c0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 -- 172.21.15.16:0/1339498392 shutdown_connections 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 --2- 172.21.15.16:0/1339498392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8106f00 0x7f26b810ea80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 --2- 172.21.15.16:0/1339498392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8102780 0x7f26b81067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 --2- 172.21.15.16:0/1339498392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b81021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 -- 172.21.15.16:0/1339498392 >> 172.21.15.16:0/1339498392 conn(0x7f26b80fd8b0 msgr2=0x7f26b80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 -- 172.21.15.16:0/1339498392 shutdown_connections 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.040+0000 7f26bf224640 1 -- 172.21.15.16:0/1339498392 wait complete. 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 Processor -- start 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 -- start start 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8102780 0x7f26b8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8111f10 con 0x7f26b8106f00 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8111d90 con 0x7f26b8102780 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.041+0000 7f26bf224640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8112090 con 0x7f26b8101dd0 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bd7c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bd7c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33728/0 (socket says 172.21.15.16:33728) 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bd7c1640 1 -- 172.21.15.16:0/3196345615 learned_addr learned my addr 172.21.15.16:0/3196345615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26b7fff640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8102780 0x7f26b8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 -- 172.21.15.16:0/3196345615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8102780 msgr2=0x7f26b8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8102780 0x7f26b8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 -- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 msgr2=0x7f26b8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bcfc0640 1 -- 172.21.15.16:0/3196345615 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b81aade0 con 0x7f26b8101dd0 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.042+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.043+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b80744f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00e9f0 tx=0x7f26ac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.043+0000 7f26b5ffb640 1 -- 172.21.15.16:0/3196345615 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00ccc0 con 0x7f26b8101dd0 2024-09-12T03:06:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.043+0000 7f26b5ffb640 1 -- 172.21.15.16:0/3196345615 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ac010910 con 0x7f26b8101dd0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.043+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b81aafb0 con 0x7f26b8101dd0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.044+0000 7f26b5ffb640 1 -- 172.21.15.16:0/3196345615 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac018960 con 0x7f26b8101dd0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.044+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b81ab4e0 con 0x7f26b8101dd0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.046+0000 7f26c15fd740 1 Processor -- start 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 -- start start 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b3b40 0x55837f3b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55837f3182f0 con 0x55837f3b25d0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55837f2bfcf0 con 0x55837f3bd370 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26c15fd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55837f3142b0 con 0x55837f3b3b40 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34840/0 (socket says 172.21.15.16:34840) 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26b7fff640 1 -- 172.21.15.16:0/820928504 learned_addr learned my addr 172.21.15.16:0/820928504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26b5ffb640 1 -- 172.21.15.16:0/3196345615 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f26ac010430 con 0x7f26b8101dd0 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.047+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 -- 172.21.15.16:0/820928504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b3b40 msgr2=0x55837f3b2090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b3b40 0x55837f3b2090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 -- 172.21.15.16:0/820928504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 msgr2=0x55837f3b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 -- 172.21.15.16:0/820928504 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55837f2a43a0 con 0x55837f3bd370 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26b7fff640 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009c30 tx=0x7f26a4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c72a97a0608e8e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26a2ffd640 1 -- 172.21.15.16:0/820928504 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4009de0 con 0x55837f3bd370 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26a2ffd640 1 -- 172.21.15.16:0/820928504 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a40035f0 con 0x55837f3bd370 2024-09-12T03:06:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.048+0000 7f26a2ffd640 1 -- 172.21.15.16:0/820928504 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4019650 con 0x55837f3bd370 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.049+0000 7f26b5ffb640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 0x7f26a80a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.049+0000 7f26b5ffb640 1 -- 172.21.15.16:0/3196345615 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26ac0cf410 con 0x7f26b8101dd0 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.049+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 0x7f26a80a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.049+0000 7f26c15fd740 1 -- 172.21.15.16:0/820928504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 msgr2=0x55837f3bf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.049+0000 7f26c15fd740 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009c30 tx=0x7f26a4011810 comp rx=0 tx=0).stop 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 -- 172.21.15.16:0/820928504 shutdown_connections 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3bf760 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 --2- 172.21.15.16:0/820928504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b3b40 0x55837f3b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 -- 172.21.15.16:0/820928504 >> 172.21.15.16:0/820928504 conn(0x55837f37b310 msgr2=0x55837f3aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 0x7f26a80a5140 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009370 tx=0x7f26b0009860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 -- 172.21.15.16:0/820928504 shutdown_connections 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 -- 172.21.15.16:0/820928504 wait complete. 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 -- start start 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.050+0000 7f26c15fd740 1 Processor -- start 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3bcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3b9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55837f3c0070 con 0x55837f3b25d0 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55837f2d3060 con 0x55837f3b3b40 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55837f3c01f0 con 0x55837f3bd370 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26c15fd740 10 client.0 _ll_register_callbacks cb 0x55837f0268c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bcfc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3bcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47780/0 (socket says 172.21.15.16:47780) 2024-09-12T03:06:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 -- 172.21.15.16:0/564811443 learned_addr learned my addr 172.21.15.16:0/564811443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26b7fff640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3b9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 -- 172.21.15.16:0/564811443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 msgr2=0x55837f3b9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3b9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 -- 172.21.15.16:0/564811443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 msgr2=0x55837f3bcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3bcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.051+0000 7f26bd7c1640 1 -- 172.21.15.16:0/564811443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55837f0893c0 con 0x55837f3bd370 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f26b0005e70 tx=0x7f26b0009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9173f7d2e358fcbc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f2697fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b004f020 con 0x55837f3bd370 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a40035f0 con 0x55837f3bd370 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26b0004240 con 0x55837f3bd370 2024-09-12T03:06:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26b0036050 con 0x55837f3bd370 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f26c15fd740 10 client.28984 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.052+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55837f3cbad0 con 0x55837f3bd370 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a40035f0 con 0x55837f3bd370 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f26b003c040 con 0x55837f3bd370 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 10 client.28984 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 10 client.28984 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 20 client.28984 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f3cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 4 client.28984 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.053+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55837f325860 con 0x55837f3ccf90 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.054+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f3cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.054+0000 7f2697fff640 10 client.28984 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.054+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f3cf430 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0e9b4687ae3067d server_cookie=606b829ee2ec8876 in_seq=0 out_seq=0 2024-09-12T03:06:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f2697fff640 1 -- 172.21.15.16:0/564811443 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55837f325860 con 0x55837f3ccf90 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f2697fff640 10 client.28984 handle_command_reply: tid=0 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 client.28984 shutdown 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 msgr2=0x55837f3ba5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f26b0005e70 tx=0x7f26b0009500 comp rx=0 tx=0).stop 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 shutdown_connections 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f3cf430 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3ba5d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f2697fff640 0 client.28984 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3b9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 --2- 172.21.15.16:0/564811443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3bcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 >> 172.21.15.16:0/564811443 conn(0x55837f37b310 msgr2=0x55837f3b4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 shutdown_connections 2024-09-12T03:06:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 1 -- 172.21.15.16:0/564811443 wait complete. 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.055+0000 7f26c15fd740 20 client.28984 trim_cache size 0 max 16384 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 Processor -- start 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 -- start start 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3d4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b3ad0 con 0x55837f3b25d0 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b5610 con 0x55837f3b3b40 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26c15fd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b41a0 con 0x55837f3bd370 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26bcfc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26bcfc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33766/0 (socket says 172.21.15.16:33766) 2024-09-12T03:06:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26bcfc0640 1 -- 172.21.15.16:0/1313940715 learned_addr learned my addr 172.21.15.16:0/1313940715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26b7fff640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.069+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3d4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 -- 172.21.15.16:0/1313940715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 msgr2=0x55837f3d4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3d4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 -- 172.21.15.16:0/1313940715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 msgr2=0x55837f3c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 -- 172.21.15.16:0/1313940715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55837f3b8550 con 0x55837f3b25d0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3d4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26b7fff640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f26ac09bc60 tx=0x7f26ac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bff35c9f04cab3b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26a27fc640 1 -- 172.21.15.16:0/1313940715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac09bcd0 con 0x55837f3b25d0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26a27fc640 1 -- 172.21.15.16:0/1313940715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ac09ffd0 con 0x55837f3b25d0 2024-09-12T03:06:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.070+0000 7f26a27fc640 1 -- 172.21.15.16:0/1313940715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac010ac0 con 0x55837f3b25d0 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- 172.21.15.16:0/1313940715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 msgr2=0x55837f3aee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f26ac09bc60 tx=0x7f26ac00cf60 comp rx=0 tx=0).stop 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- 172.21.15.16:0/1313940715 shutdown_connections 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3bd370 0x55837f3d4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3b3b40 0x55837f3c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1313940715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b25d0 0x55837f3aee10 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- 172.21.15.16:0/1313940715 >> 172.21.15.16:0/1313940715 conn(0x55837f37b310 msgr2=0x55837f3ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- 172.21.15.16:0/1313940715 shutdown_connections 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- 172.21.15.16:0/1313940715 wait complete. 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 -- start start 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.071+0000 7f26c15fd740 1 Processor -- start 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b25d0 0x55837f3c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3c2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b4f40 con 0x55837f3b3b40 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b4870 con 0x55837f3bd370 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b81b1ab0 con 0x55837f3b25d0 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33770/0 (socket says 172.21.15.16:33770) 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 -- 172.21.15.16:0/1411577413 learned_addr learned my addr 172.21.15.16:0/1411577413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b25d0 0x55837f3c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26c15fd740 10 client.0 _ll_register_callbacks cb 0x7f26a4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26bd7c1640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 -- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b25d0 msgr2=0x55837f3c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b25d0 0x55837f3c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 -- 172.21.15.16:0/1411577413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 msgr2=0x55837f3c2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3c2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 -- 172.21.15.16:0/1411577413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55837f3c2b10 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f26b7fff640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006010 tx=0x7f26a40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0b342fb6f3bf6c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.072+0000 7f2697fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4028070 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26a4004660 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26ac09ffd0 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26a4004860 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 10 client.19803 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55837f438600 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ac09ffd0 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f26a4019af0 con 0x55837f3b3b40 2024-09-12T03:06:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 10 client.19803 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 10 client.19803 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 20 client.19803 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f43ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 4 client.19803 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.073+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55837f31fd70 con 0x55837f3ccf90 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f43ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f2697fff640 10 client.19803 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26bcfc0640 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f43ac00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=396d55d28c86448d server_cookie=c0d7e454b8429439 in_seq=0 out_seq=0 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f2697fff640 1 -- 172.21.15.16:0/1411577413 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55837f31fd70 con 0x55837f3ccf90 2024-09-12T03:06:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f2697fff640 10 client.19803 handle_command_reply: tid=0 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 client.19803 shutdown 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 msgr2=0x55837f3c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006010 tx=0x7f26a40056a0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 shutdown_connections 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55837f3bd370 0x55837f3c2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55837f3b3b40 0x55837f3c5ba0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55837f3ccf90 0x55837f43ac00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 --2- 172.21.15.16:0/1411577413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55837f3b25d0 0x55837f3c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f2697fff640 0 client.19803 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 >> 172.21.15.16:0/1411577413 conn(0x55837f37b310 msgr2=0x55837f3bb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 shutdown_connections 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.074+0000 7f26c15fd740 1 -- 172.21.15.16:0/1411577413 wait complete. 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26c15fd740 20 client.19803 trim_cache size 0 max 16384 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 msgr2=0x7f26a80a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 0x7f26a80a5140 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009370 tx=0x7f26b0009860 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 msgr2=0x7f26b80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b80744f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00e9f0 tx=0x7f26ac00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 shutdown_connections 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f26a80a2c80 0x7f26a80a5140 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26b8106f00 0x7f26b8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26b8102780 0x7f26b8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 --2- 172.21.15.16:0/3196345615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b8101dd0 0x7f26b80744f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 >> 172.21.15.16:0/3196345615 conn(0x7f26b80fd8b0 msgr2=0x7f26b80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 shutdown_connections 2024-09-12T03:06:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:47.075+0000 7f26bf224640 1 -- 172.21.15.16:0/3196345615 wait complete. 2024-09-12T03:06:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:47 smithi016 ceph-mon[30925]: pgmap v903: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 347 op/s 2024-09-12T03:06:47.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:06:47.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:47 smithi033 ceph-mon[35092]: pgmap v903: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 347 op/s 2024-09-12T03:06:47.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:47 smithi184 ceph-mon[35321]: pgmap v903: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 347 op/s 2024-09-12T03:06:48.000 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:48.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.43 MB/sec execute 229 sec latency 13307.664 ms 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.526+0000 7f4d37d04640 1 -- 172.21.15.16:0/1934329337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30100e20 msgr2=0x7f4d30101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.526+0000 7f4d37d04640 1 --2- 172.21.15.16:0/1934329337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30100e20 0x7f4d30101200 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c02a480 comp rx=0 tx=0).stop 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.526+0000 7f4d37d04640 1 -- 172.21.15.16:0/1934329337 shutdown_connections 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- 172.21.15.16:0/1934329337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30102ef0 0x7f4d3010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- 172.21.15.16:0/1934329337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 0x7f4d301029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- 172.21.15.16:0/1934329337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30100e20 0x7f4d30101200 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- 172.21.15.16:0/1934329337 >> 172.21.15.16:0/1934329337 conn(0x7f4d300fc7d0 msgr2=0x7f4d300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- 172.21.15.16:0/1934329337 shutdown_connections 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- 172.21.15.16:0/1934329337 wait complete. 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 Processor -- start 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- start start 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30100e20 0x7f4d30199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 0x7f4d30199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30110f80 con 0x7f4d30102ef0 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30110e00 con 0x7f4d30102570 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.527+0000 7f4d37d04640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111100 con 0x7f4d30100e20 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33786/0 (socket says 172.21.15.16:33786) 2024-09-12T03:06:48.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 -- 172.21.15.16:0/3443557604 learned_addr learned my addr 172.21.15.16:0/3443557604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30100e20 0x7f4d30199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d3529f640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 0x7f4d30199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 -- 172.21.15.16:0/3443557604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30100e20 msgr2=0x7f4d30199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30100e20 0x7f4d30199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 -- 172.21.15.16:0/3443557604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 msgr2=0x7f4d30199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 0x7f4d30199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 -- 172.21.15.16:0/3443557604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d301a48e0 con 0x7f4d30102ef0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800eaa0 tx=0x7f4d2800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3443557604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800cdb0 con 0x7f4d30102ef0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3443557604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d28005020 con 0x7f4d30102ef0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d301a4ab0 con 0x7f4d30102ef0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3443557604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d280116b0 con 0x7f4d30102ef0 2024-09-12T03:06:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.528+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d301a4f70 con 0x7f4d30102ef0 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.531+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3443557604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4d28005560 con 0x7f4d30102ef0 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.531+0000 7f4d3a0dd740 1 Processor -- start 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 -- start start 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff568090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff573370 0x558bff575760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bff4ce2f0 con 0x558bff5685d0 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558bff475cf0 con 0x558bff569b40 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558bff4ca2b0 con 0x558bff573370 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33800/0 (socket says 172.21.15.16:33800) 2024-09-12T03:06:48.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1561164908 learned_addr learned my addr 172.21.15.16:0/1561164908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff568090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff573370 0x558bff575760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1561164908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff573370 msgr2=0x558bff575760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff573370 0x558bff575760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1561164908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 msgr2=0x558bff568090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff568090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d26ffd640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 0x7f4d1c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.532+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1561164908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bff45a3a0 con 0x558bff5685d0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d3529f640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 0x7f4d1c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3443557604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d280cf220 con 0x7f4d30102ef0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc2f266fc657903 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d1bfff640 1 -- 172.21.15.16:0/1561164908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c035030 con 0x558bff5685d0 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d1bfff640 1 -- 172.21.15.16:0/1561164908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2c040070 con 0x558bff5685d0 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.533+0000 7f4d1bfff640 1 -- 172.21.15.16:0/1561164908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c031ad0 con 0x558bff5685d0 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1561164908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 msgr2=0x558bff568a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c004170 comp rx=0 tx=0).stop 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3529f640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 0x7f4d1c0a5040 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005d90 tx=0x7f4d2000e5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1561164908 shutdown_connections 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff573370 0x558bff575760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff568a10 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1561164908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff568090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1561164908 >> 172.21.15.16:0/1561164908 conn(0x558bff531310 msgr2=0x558bff564c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1561164908 shutdown_connections 2024-09-12T03:06:48.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1561164908 wait complete. 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 -- start start 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.534+0000 7f4d3a0dd740 1 Processor -- start 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff572cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff56ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bff576010 con 0x558bff573370 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558bff489060 con 0x558bff569b40 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558bff576190 con 0x558bff5685d0 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3a0dd740 10 client.0 _ll_register_callbacks cb 0x558bff1dc8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3529f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff56ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33816/0 (socket says 172.21.15.16:33816) 2024-09-12T03:06:48.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 -- 172.21.15.16:0/3812400832 learned_addr learned my addr 172.21.15.16:0/3812400832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff572cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 -- 172.21.15.16:0/3812400832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 msgr2=0x558bff572cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff572cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 -- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 msgr2=0x558bff56ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff56ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 -- 172.21.15.16:0/3812400832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bff23f3c0 con 0x558bff573370 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d3529f640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff56ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d362a1640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f4d28097ef0 tx=0x7f4d2809c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87844b495ec674a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.535+0000 7f4d197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2801f070 con 0x558bff573370 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d2809aee0 con 0x558bff573370 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d2c040070 con 0x558bff573370 2024-09-12T03:06:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d280a4700 con 0x558bff573370 2024-09-12T03:06:48.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 10 client.19815 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:48.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bff581b20 con 0x558bff573370 2024-09-12T03:06:48.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2c040070 con 0x558bff573370 2024-09-12T03:06:48.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4d2800f040 con 0x558bff573370 2024-09-12T03:06:48.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 10 client.19815 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:48.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 10 client.19815 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 20 client.19815 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff582fe0 0x558bff585430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 4 client.19815 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.536+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bff4db860 con 0x558bff582fe0 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.537+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff582fe0 0x558bff585430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.537+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff582fe0 0x558bff585430 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e80b085f387cc89d server_cookie=3279365c1ddb81aa in_seq=0 out_seq=0 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.537+0000 7f4d197fa640 10 client.19815 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d197fa640 1 -- 172.21.15.16:0/3812400832 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558bff4db860 con 0x558bff582fe0 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d197fa640 10 client.19815 handle_command_reply: tid=0 2024-09-12T03:06:48.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 client.19815 shutdown 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 msgr2=0x558bff570520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f4d28097ef0 tx=0x7f4d2809c2d0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 shutdown_connections 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff582fe0 0x558bff585430 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff573370 0x558bff570520 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff569b40 0x558bff56ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/3812400832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff572cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d197fa640 0 client.19815 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 >> 172.21.15.16:0/3812400832 conn(0x558bff531310 msgr2=0x558bff56a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 shutdown_connections 2024-09-12T03:06:48.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/3812400832 wait complete. 2024-09-12T03:06:48.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.538+0000 7f4d3a0dd740 20 client.19815 trim_cache size 0 max 16384 2024-09-12T03:06:48.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 Processor -- start 2024-09-12T03:06:48.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 -- start start 2024-09-12T03:06:48.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff57f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff564c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ad4c0 con 0x558bff569b40 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301af000 con 0x558bff573370 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301adb90 con 0x558bff5685d0 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3529f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3529f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33822/0 (socket says 172.21.15.16:33822) 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.551+0000 7f4d3529f640 1 -- 172.21.15.16:0/1429994138 learned_addr learned my addr 172.21.15.16:0/1429994138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff57f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d362a1640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff564c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 -- 172.21.15.16:0/1429994138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 msgr2=0x558bff57f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff57f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 -- 172.21.15.16:0/1429994138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 msgr2=0x558bff564c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff564c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 -- 172.21.15.16:0/1429994138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bff573060 con 0x558bff569b40 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff57f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d362a1640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff564c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005fd0 tx=0x7f4d20014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441e8cd652e78454 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d1b7fe640 1 -- 172.21.15.16:0/1429994138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20021d70 con 0x558bff569b40 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d1b7fe640 1 -- 172.21.15.16:0/1429994138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2001a070 con 0x558bff569b40 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.552+0000 7f4d1b7fe640 1 -- 172.21.15.16:0/1429994138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20004a00 con 0x558bff569b40 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1429994138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 msgr2=0x558bff564700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005fd0 tx=0x7f4d20014040 comp rx=0 tx=0).stop 2024-09-12T03:06:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1429994138 shutdown_connections 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff564c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff569b40 0x558bff564700 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1429994138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff5685d0 0x558bff57f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1429994138 >> 172.21.15.16:0/1429994138 conn(0x558bff531310 msgr2=0x558bff56ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1429994138 shutdown_connections 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1429994138 wait complete. 2024-09-12T03:06:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 -- start start 2024-09-12T03:06:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.553+0000 7f4d3a0dd740 1 Processor -- start 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff569b40 0x558bff56cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff56d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ae930 con 0x558bff5685d0 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ae260 con 0x558bff573370 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ab4a0 con 0x558bff569b40 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3a0dd740 10 client.0 _ll_register_callbacks cb 0x7f4d28097d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d3529f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff569b40 0x558bff56cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33832/0 (socket says 172.21.15.16:33832) 2024-09-12T03:06:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1818632771 learned_addr learned my addr 172.21.15.16:0/1818632771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d362a1640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff56d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1818632771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff569b40 msgr2=0x558bff56cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff569b40 0x558bff56cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 msgr2=0x558bff56d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff56d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d35aa0640 1 -- 172.21.15.16:0/1818632771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bff56d8c0 con 0x558bff5685d0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.554+0000 7f4d362a1640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff56d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d35aa0640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10edfa7babf7ed37 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c054030 con 0x558bff5685d0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d2c005020 con 0x558bff5685d0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d2001a070 con 0x558bff5685d0 2024-09-12T03:06:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d2c035040 con 0x558bff5685d0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 10 client.19821 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bff5edc10 con 0x558bff5685d0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2001a070 con 0x558bff5685d0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4d2c004750 con 0x558bff5685d0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 10 client.19821 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 10 client.19821 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 20 client.19821 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff5edfb0 0x558bff5f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 4 client.19821 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.555+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558bff4d5d70 con 0x558bff5edfb0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff5edfb0 0x558bff5f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d3529f640 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff5edfb0 0x558bff5f03a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7c84d651f84de7e server_cookie=1be3e70e2f1d93f4 in_seq=0 out_seq=0 2024-09-12T03:06:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d197fa640 10 client.19821 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d197fa640 1 -- 172.21.15.16:0/1818632771 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558bff4d5d70 con 0x558bff5edfb0 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d197fa640 10 client.19821 handle_command_reply: tid=0 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d3a0dd740 1 client.19821 shutdown 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 msgr2=0x558bff57ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.556+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c0058d0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 shutdown_connections 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558bff5edfb0 0x558bff5f03a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bff573370 0x558bff56d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d197fa640 0 client.19821 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bff569b40 0x558bff56cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 --2- 172.21.15.16:0/1818632771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bff5685d0 0x558bff57ba60 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 >> 172.21.15.16:0/1818632771 conn(0x558bff531310 msgr2=0x558bff5646c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 shutdown_connections 2024-09-12T03:06:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 1 -- 172.21.15.16:0/1818632771 wait complete. 2024-09-12T03:06:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d3a0dd740 20 client.19821 trim_cache size 0 max 16384 2024-09-12T03:06:48.564 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 msgr2=0x7f4d1c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 0x7f4d1c0a5040 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005d90 tx=0x7f4d2000e5f0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 msgr2=0x7f4d301a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.557+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800eaa0 tx=0x7f4d2800ef60 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 shutdown_connections 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d1c0a2b80 0x7f4d1c0a5040 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d30102ef0 0x7f4d301a43a0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30102570 0x7f4d30199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 --2- 172.21.15.16:0/3443557604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d30100e20 0x7f4d30199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 >> 172.21.15.16:0/3443557604 conn(0x7f4d300fc7d0 msgr2=0x7f4d300fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 shutdown_connections 2024-09-12T03:06:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:48.558+0000 7f4d37d04640 1 -- 172.21.15.16:0/3443557604 wait complete. 2024-09-12T03:06:49.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.32 MB/sec execute 230 sec latency 14307.715 ms 2024-09-12T03:06:49.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:06:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:49.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:49 smithi033 ceph-mon[35092]: pgmap v904: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 231 op/s 2024-09-12T03:06:49.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:49 smithi016 ceph-mon[30925]: pgmap v904: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 231 op/s 2024-09-12T03:06:49.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:49 smithi184 ceph-mon[35321]: pgmap v904: 97 pgs: 97 active+clean; 703 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 231 op/s 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 -- 172.21.15.16:0/3463317827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 msgr2=0x7f3f78103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 --2- 172.21.15.16:0/3463317827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f78103110 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009930 tx=0x7f3f6802a420 comp rx=0 tx=0).stop 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 -- 172.21.15.16:0/3463317827 shutdown_connections 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 --2- 172.21.15.16:0/3463317827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f781082a0 0x7f3f7810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 --2- 172.21.15.16:0/3463317827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 0x7f3f78107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 --2- 172.21.15.16:0/3463317827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f78103110 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 -- 172.21.15.16:0/3463317827 >> 172.21.15.16:0/3463317827 conn(0x7f3f780fc7d0 msgr2=0x7f3f780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 -- 172.21.15.16:0/3463317827 shutdown_connections 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 -- 172.21.15.16:0/3463317827 wait complete. 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.978+0000 7f3f7d695640 1 Processor -- start 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 -- start start 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 0x7f3f78198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f781082a0 0x7f3f7819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781120a0 con 0x7f3f78102d30 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78111f20 con 0x7f3f781036e0 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f7d695640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78112220 con 0x7f3f781082a0 2024-09-12T03:06:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33842/0 (socket says 172.21.15.16:33842) 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 -- 172.21.15.16:0/2673402594 learned_addr learned my addr 172.21.15.16:0/2673402594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f767fc640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 0x7f3f78198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 -- 172.21.15.16:0/2673402594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f781082a0 msgr2=0x7f3f7819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f781082a0 0x7f3f7819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 -- 172.21.15.16:0/2673402594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 msgr2=0x7f3f78198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 0x7f3f78198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.979+0000 7f3f76ffd640 1 -- 172.21.15.16:0/2673402594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7819d570 con 0x7f3f78102d30 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009230 tx=0x7f3f68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f5ffff640 1 -- 172.21.15.16:0/2673402594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6803f030 con 0x7f3f78102d30 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7819d800 con 0x7f3f78102d30 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f5ffff640 1 -- 172.21.15.16:0/2673402594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f68003ad0 con 0x7f3f78102d30 2024-09-12T03:06:49.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f5ffff640 1 -- 172.21.15.16:0/2673402594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68035070 con 0x7f3f78102d30 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.980+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f781a5140 con 0x7f3f78102d30 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.982+0000 7f3f5ffff640 1 -- 172.21.15.16:0/2673402594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3f68042030 con 0x7f3f78102d30 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.982+0000 7f3f7fa6e740 1 Processor -- start 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.982+0000 7f3f7fa6e740 1 -- start start 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec451a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 0x55f4ec462b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec3b72f0 con 0x55f4ec452b40 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec35ecf0 con 0x55f4ec460780 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec3b32b0 con 0x55f4ec4515d0 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f5ffff640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 0x7f3f600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33850/0 (socket says 172.21.15.16:33850) 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f777fe640 1 -- 172.21.15.16:0/1461934648 learned_addr learned my addr 172.21.15.16:0/1461934648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 0x55f4ec462b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f5ffff640 1 -- 172.21.15.16:0/2673402594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3f680ef520 con 0x7f3f78102d30 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec451a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 -- 172.21.15.16:0/1461934648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 msgr2=0x55f4ec451a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec451a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 -- 172.21.15.16:0/1461934648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 msgr2=0x55f4ec451090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f767fc640 1 -- 172.21.15.16:0/1461934648 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4ec3433a0 con 0x55f4ec460780 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec451a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f777fe640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.983+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 0x7f3f600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f767fc640 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 0x55f4ec462b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3f7806a370 tx=0x7f3f6c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b03a105ba72a4ee7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f5d7fa640 1 -- 172.21.15.16:0/1461934648 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c004030 con 0x55f4ec460780 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f5d7fa640 1 -- 172.21.15.16:0/1461934648 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f6c0041d0 con 0x55f4ec460780 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f5d7fa640 1 -- 172.21.15.16:0/1461934648 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c011680 con 0x55f4ec460780 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 0x7f3f600a5040 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000f420 tx=0x7f3f70004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1461934648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 msgr2=0x55f4ec462b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.984+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 0x55f4ec462b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3f7806a370 tx=0x7f3f6c0107a0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1461934648 shutdown_connections 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec460780 0x55f4ec462b70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec451a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1461934648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec452b40 0x55f4ec451090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1461934648 >> 172.21.15.16:0/1461934648 conn(0x55f4ec41a310 msgr2=0x55f4ec44dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1461934648 shutdown_connections 2024-09-12T03:06:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1461934648 wait complete. 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- start start 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 Processor -- start 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec460180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 0x55f4ec45d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.985+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec463480 con 0x55f4ec460780 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec372060 con 0x55f4ec452b40 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f4ec463600 con 0x55f4ec4515d0 2024-09-12T03:06:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f7fa6e740 10 client.0 _ll_register_callbacks cb 0x55f4ec0c58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33862/0 (socket says 172.21.15.16:33862) 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 -- 172.21.15.16:0/2521294554 learned_addr learned my addr 172.21.15.16:0/2521294554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec460180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f767fc640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 0x55f4ec45d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 -- 172.21.15.16:0/2521294554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 msgr2=0x55f4ec460180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec460180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 -- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 msgr2=0x55f4ec45d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 0x55f4ec45d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 -- 172.21.15.16:0/2521294554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4ec1283c0 con 0x55f4ec460780 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f767fc640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 0x55f4ec45d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011a30 tx=0x7f3f700073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff237c3ee914354 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70021070 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f6c0041d0 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.986+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3f70010d60 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3f7002d050 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 10 client.19833 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f4ec46eef0 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f6c0041d0 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3f7000f040 con 0x55f4ec460780 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 10 client.19833 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 10 client.19833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 20 client.19833 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4703f0 0x55f4ec472860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 4 client.19833 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4ec3c4860 con 0x55f4ec4703f0 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.987+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4703f0 0x55f4ec472860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.988+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4703f0 0x55f4ec472860 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2512bf07cf2cb7fe server_cookie=a4d499a7f01818d4 in_seq=0 out_seq=0 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.988+0000 7f3f5aff5640 10 client.19833 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2521294554 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f4ec3c4860 con 0x55f4ec4703f0 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f5aff5640 10 client.19833 handle_command_reply: tid=0 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 client.19833 shutdown 2024-09-12T03:06:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 msgr2=0x55f4ec45da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011a30 tx=0x7f3f700073f0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 shutdown_connections 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4703f0 0x55f4ec472860 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45da20 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec452b40 0x55f4ec45d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2521294554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec4515d0 0x55f4ec460180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 >> 172.21.15.16:0/2521294554 conn(0x55f4ec41a310 msgr2=0x55f4ec453960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f5aff5640 0 client.19833 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 shutdown_connections 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2521294554 wait complete. 2024-09-12T03:06:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:49.989+0000 7f3f7fa6e740 20 client.19833 trim_cache size 0 max 16384 2024-09-12T03:06:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 Processor -- start 2024-09-12T03:06:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 -- start start 2024-09-12T03:06:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec46c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec45bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781ab3e0 con 0x55f4ec460780 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781acf20 con 0x55f4ec4515d0 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781abab0 con 0x55f4ec452b40 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33872/0 (socket says 172.21.15.16:33872) 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f777fe640 1 -- 172.21.15.16:0/1720845905 learned_addr learned my addr 172.21.15.16:0/1720845905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.002+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec46c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f767fc640 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec45bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 -- 172.21.15.16:0/1720845905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 msgr2=0x55f4ec45bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec45bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 -- 172.21.15.16:0/1720845905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 msgr2=0x55f4ec46c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec46c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 -- 172.21.15.16:0/1720845905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4ec4604f0 con 0x55f4ec460780 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f777fe640 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3f70007a10 tx=0x7f3f7000fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8980dd6f309227ac server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f5cff9640 1 -- 172.21.15.16:0/1720845905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70035030 con 0x55f4ec460780 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f5cff9640 1 -- 172.21.15.16:0/1720845905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f7001b7d0 con 0x55f4ec460780 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.003+0000 7f3f5cff9640 1 -- 172.21.15.16:0/1720845905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70021050 con 0x55f4ec460780 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1720845905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 msgr2=0x55f4ec45c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3f70007a10 tx=0x7f3f7000fe00 comp rx=0 tx=0).stop 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1720845905 shutdown_connections 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec45c260 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec45bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/1720845905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec46c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1720845905 >> 172.21.15.16:0/1720845905 conn(0x55f4ec41a310 msgr2=0x55f4ec448a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1720845905 shutdown_connections 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/1720845905 wait complete. 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- start start 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 Processor -- start 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec463df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec464330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781ac850 con 0x55f4ec460780 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781ac180 con 0x55f4ec4515d0 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781a93e0 con 0x55f4ec452b40 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.004+0000 7f3f7fa6e740 10 client.0 _ll_register_callbacks cb 0x7f3f70007cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33880/0 (socket says 172.21.15.16:33880) 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f76ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec463df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 -- 172.21.15.16:0/2494192121 learned_addr learned my addr 172.21.15.16:0/2494192121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f767fc640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec464330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 -- 172.21.15.16:0/2494192121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 msgr2=0x55f4ec464330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec464330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 -- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 msgr2=0x55f4ec463df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec463df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 -- 172.21.15.16:0/2494192121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4ec4d9670 con 0x55f4ec460780 2024-09-12T03:06:50.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec463df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f777fe640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3f7001be70 tx=0x7f3f7001bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2169de2bd54b20a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7003e030 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3f70010d70 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7001b7d0 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3f70035070 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.005+0000 7f3f7fa6e740 10 client.19839 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f4ec4da970 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f7001b7d0 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3f700454d0 con 0x55f4ec460780 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 10 client.19839 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 10 client.19839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 20 client.19839 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4daf80 0x55f4ec4dd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 4 client.19839 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4ec3bed70 con 0x55f4ec4daf80 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4daf80 0x55f4ec4dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f5aff5640 10 client.19839 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.006+0000 7f3f76ffd640 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4daf80 0x55f4ec4dd370 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da391a7c4d37fce server_cookie=a5f5cc91c0be4dfb in_seq=0 out_seq=0 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f5aff5640 1 -- 172.21.15.16:0/2494192121 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4ec3bed70 con 0x55f4ec4daf80 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f5aff5640 10 client.19839 handle_command_reply: tid=0 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 client.19839 shutdown 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 msgr2=0x55f4ec4d9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3f7001be70 tx=0x7f3f7001bbb0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 shutdown_connections 2024-09-12T03:06:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f4ec4daf80 0x55f4ec4dd370 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f4ec460780 0x55f4ec4d9130 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f4ec452b40 0x55f4ec464330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 --2- 172.21.15.16:0/2494192121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4ec4515d0 0x55f4ec463df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 >> 172.21.15.16:0/2494192121 conn(0x55f4ec41a310 msgr2=0x55f4ec45bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f5aff5640 0 client.19839 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 shutdown_connections 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 1 -- 172.21.15.16:0/2494192121 wait complete. 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.007+0000 7f3f7fa6e740 20 client.19839 trim_cache size 0 max 16384 2024-09-12T03:06:50.012 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 msgr2=0x7f3f600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 0x7f3f600a5040 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000f420 tx=0x7f3f70004a40 comp rx=0 tx=0).stop 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 msgr2=0x7f3f781985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009230 tx=0x7f3f68004270 comp rx=0 tx=0).stop 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 shutdown_connections 2024-09-12T03:06:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f781082a0 0x7f3f7819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f781036e0 0x7f3f78198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3f600a2b80 0x7f3f600a5040 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 --2- 172.21.15.16:0/2673402594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f78102d30 0x7f3f781985c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 >> 172.21.15.16:0/2673402594 conn(0x7f3f780fc7d0 msgr2=0x7f3f78105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 shutdown_connections 2024-09-12T03:06:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:50.008+0000 7f3f7d695640 1 -- 172.21.15.16:0/2673402594 wait complete. 2024-09-12T03:06:50.021 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.21 MB/sec execute 231 sec latency 15307.744 ms 2024-09-12T03:06:51.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 25.10 MB/sec execute 232 sec latency 16307.785 ms 2024-09-12T03:06:51.695 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:06:51.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:06:51.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:51 smithi016 ceph-mon[30925]: pgmap v905: 97 pgs: 97 active+clean; 700 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 6.0 MiB/s wr, 245 op/s 2024-09-12T03:06:51.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:51 smithi033 ceph-mon[35092]: pgmap v905: 97 pgs: 97 active+clean; 700 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 6.0 MiB/s wr, 245 op/s 2024-09-12T03:06:51.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:51 smithi184 ceph-mon[35321]: pgmap v905: 97 pgs: 97 active+clean; 700 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 6.0 MiB/s wr, 245 op/s 2024-09-12T03:06:51.966 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:52.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 24.99 MB/sec execute 233 sec latency 17307.830 ms 2024-09-12T03:06:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 -- 172.21.15.16:0/4000974527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40102790 msgr2=0x7f4f401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 --2- 172.21.15.16:0/4000974527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40102790 0x7f4f401067e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009900 tx=0x7f4f3402a410 comp rx=0 tx=0).stop 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 -- 172.21.15.16:0/4000974527 shutdown_connections 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 --2- 172.21.15.16:0/4000974527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 0x7f4f4010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 --2- 172.21.15.16:0/4000974527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40102790 0x7f4f401067e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 --2- 172.21.15.16:0/4000974527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40101de0 0x7f4f401021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 -- 172.21.15.16:0/4000974527 >> 172.21.15.16:0/4000974527 conn(0x7f4f400fd8c0 msgr2=0x7f4f400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 -- 172.21.15.16:0/4000974527 shutdown_connections 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 -- 172.21.15.16:0/4000974527 wait complete. 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.457+0000 7f4f45bba640 1 Processor -- start 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 -- start start 2024-09-12T03:06:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40102790 0x7f4f40074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 0x7f4f40074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f40111f70 con 0x7f4f40101de0 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f40111df0 con 0x7f4f40106f10 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f45bba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401120f0 con 0x7f4f40102790 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33906/0 (socket says 172.21.15.16:33906) 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 -- 172.21.15.16:0/3174561756 learned_addr learned my addr 172.21.15.16:0/3174561756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3effd640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40102790 0x7f4f40074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 0x7f4f40074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 -- 172.21.15.16:0/3174561756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40102790 msgr2=0x7f4f40074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40102790 0x7f4f40074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 -- 172.21.15.16:0/3174561756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 msgr2=0x7f4f40074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 0x7f4f40074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 -- 172.21.15.16:0/3174561756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f401a6890 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.458+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000cce0 tx=0x7f4f3000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.459+0000 7f4f3cff9640 1 -- 172.21.15.16:0/3174561756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30004b20 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.459+0000 7f4f3cff9640 1 -- 172.21.15.16:0/3174561756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f30004cc0 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.459+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f401a6b80 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.459+0000 7f4f3cff9640 1 -- 172.21.15.16:0/3174561756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000f8c0 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.459+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f401a70c0 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3cff9640 1 -- 172.21.15.16:0/3174561756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4f3000fa60 con 0x7f4f40101de0 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 Processor -- start 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 -- start start 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559040dfe280 con 0x559040e98b60 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559040da5b10 con 0x559040ea3d40 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f47f93740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559040dfa240 con 0x559040e99480 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33914/0 (socket says 172.21.15.16:33914) 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3ffff640 1 -- 172.21.15.16:0/3550065316 learned_addr learned my addr 172.21.15.16:0/3550065316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.462+0000 7f4f3effd640 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 -- 172.21.15.16:0/3550065316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 msgr2=0x559040ea3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 -- 172.21.15.16:0/3550065316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 msgr2=0x559040ea6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 -- 172.21.15.16:0/3550065316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559040db21f0 con 0x559040e98b60 2024-09-12T03:06:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3cff9640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 0x7f4f280a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 0x7f4f280a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800cb40 tx=0x7f4f3800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5609f33bf681636 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f3cff9640 1 -- 172.21.15.16:0/3174561756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4f300cf120 con 0x7f4f40101de0 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3550065316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38005020 con 0x559040e98b60 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3550065316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f380047a0 con 0x559040e98b60 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.463+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3550065316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38010700 con 0x559040e98b60 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- 172.21.15.16:0/3550065316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 msgr2=0x559040e98f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f3f7fe640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 0x7f4f280a50c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4f300ccfa0 tx=0x7f4f30086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800cb40 tx=0x7f4f3800e350 comp rx=0 tx=0).stop 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- 172.21.15.16:0/3550065316 shutdown_connections 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 --2- 172.21.15.16:0/3550065316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e98f40 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- 172.21.15.16:0/3550065316 >> 172.21.15.16:0/3550065316 conn(0x559040e950f0 msgr2=0x559040e954d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- 172.21.15.16:0/3550065316 shutdown_connections 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- 172.21.15.16:0/3550065316 wait complete. 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.464+0000 7f4f47f93740 1 -- start start 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 Processor -- start 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e98b60 0x559040ea0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559040ea6910 con 0x559040ea3d40 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559040db8f10 con 0x559040e98b60 2024-09-12T03:06:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559040ea6a90 con 0x559040e99480 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f47f93740 10 client.0 _ll_register_callbacks cb 0x559040d81a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33930/0 (socket says 172.21.15.16:33930) 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1760737509 learned_addr learned my addr 172.21.15.16:0/1760737509 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e98b60 0x559040ea0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1760737509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 msgr2=0x559040ea10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e98b60 msgr2=0x559040ea0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e98b60 0x559040ea0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.465+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1760737509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559040b6f6a0 con 0x559040ea3d40 2024-09-12T03:06:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800f010 tx=0x7f4f38005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d61434d838fe5c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3801e020 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4f38005310 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f380047a0 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4f3802f050 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f47f93740 10 client.19851 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559040eb24e0 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f380047a0 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4f38003a60 con 0x559040ea3d40 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f47f93740 10 client.19851 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.466+0000 7f4f47f93740 10 client.19851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f47f93740 20 client.19851 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040eb5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f47f93740 4 client.19851 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559040e0b7f0 con 0x559040eb39a0 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040eb5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f1effd640 10 client.19851 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.467+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040eb5d90 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee245038eb9021e3 server_cookie=9f13dbaba528f270 in_seq=0 out_seq=0 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f1effd640 1 -- 172.21.15.16:0/1760737509 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559040e0b7f0 con 0x559040eb39a0 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f1effd640 10 client.19851 handle_command_reply: tid=0 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 client.19851 shutdown 2024-09-12T03:06:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 msgr2=0x559040ea1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800f010 tx=0x7f4f38005740 comp rx=0 tx=0).stop 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 shutdown_connections 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040eb5d90 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f1effd640 0 client.19851 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040ea3d40 0x559040ea1600 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1760737509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e98b60 0x559040ea0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 >> 172.21.15.16:0/1760737509 conn(0x559040e950f0 msgr2=0x559040e94270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 shutdown_connections 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 1 -- 172.21.15.16:0/1760737509 wait complete. 2024-09-12T03:06:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.468+0000 7f4f47f93740 20 client.19851 trim_cache size 0 max 16384 2024-09-12T03:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.486+0000 7f4f47f93740 1 Processor -- start 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.486+0000 7f4f47f93740 1 -- start start 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e99480 0x559040ebb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040ea3d40 0x559040ebba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401af700 con 0x559040e98b60 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b1240 con 0x559040e99480 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f47f93740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401afdd0 con 0x559040ea3d40 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33946/0 (socket says 172.21.15.16:33946) 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1072856992 learned_addr learned my addr 172.21.15.16:0/1072856992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e99480 0x559040ebb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040ea3d40 0x559040ebba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1072856992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040ea3d40 msgr2=0x559040ebba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040ea3d40 0x559040ebba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1072856992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e99480 msgr2=0x559040ebb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e99480 0x559040ebb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1072856992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559040e961a0 con 0x559040e98b60 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.487+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800e9d0 tx=0x7f4f38005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a2b4130d2f886d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f257fa640 1 -- 172.21.15.16:0/1072856992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3801e070 con 0x559040e98b60 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f257fa640 1 -- 172.21.15.16:0/1072856992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f38003ad0 con 0x559040e98b60 2024-09-12T03:06:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f257fa640 1 -- 172.21.15.16:0/1072856992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38004230 con 0x559040e98b60 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 -- 172.21.15.16:0/1072856992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 msgr2=0x559040e9acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800e9d0 tx=0x7f4f38005040 comp rx=0 tx=0).stop 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 -- 172.21.15.16:0/1072856992 shutdown_connections 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040ea3d40 0x559040ebba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040e99480 0x559040ebb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1072856992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040e9acc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 -- 172.21.15.16:0/1072856992 >> 172.21.15.16:0/1072856992 conn(0x559040e950f0 msgr2=0x559040e9a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.488+0000 7f4f47f93740 1 -- 172.21.15.16:0/1072856992 shutdown_connections 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 -- 172.21.15.16:0/1072856992 wait complete. 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 -- start start 2024-09-12T03:06:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 Processor -- start 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b0b70 con 0x559040e98b60 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b04a0 con 0x559040ea3d40 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401ad6e0 con 0x559040e99480 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f47f93740 10 client.0 _ll_register_callbacks cb 0x7f4f38014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33950/0 (socket says 172.21.15.16:33950) 2024-09-12T03:06:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.489+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1664889005 learned_addr learned my addr 172.21.15.16:0/1664889005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1664889005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 msgr2=0x559040ea7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 msgr2=0x559040ea8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 -- 172.21.15.16:0/1664889005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559040eb62f0 con 0x559040e98b60 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f3ffff640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4f38003f00 tx=0x7f4f38004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e9c8530d18b4df server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f1effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3803a040 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4f38003fd0 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f38003ad0 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4f3801e040 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f47f93740 10 client.19857 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559040f1e5f0 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.490+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f38003ad0 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4f38010640 con 0x559040e98b60 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 10 client.19857 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 10 client.19857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 20 client.19857 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040f20c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 4 client.19857 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559040e05d00 con 0x559040eb39a0 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040f20c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f3effd640 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040f20c10 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35078a9a301e28c server_cookie=c216cbb644d85c2f in_seq=0 out_seq=0 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.491+0000 7f4f1effd640 10 client.19857 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f1effd640 1 -- 172.21.15.16:0/1664889005 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559040e05d00 con 0x559040eb39a0 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f1effd640 10 client.19857 handle_command_reply: tid=0 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 client.19857 shutdown 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 msgr2=0x559040ea79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4f38003f00 tx=0x7f4f38004c60 comp rx=0 tx=0).stop 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 shutdown_connections 2024-09-12T03:06:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559040eb39a0 0x559040f20c10 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559040ea3d40 0x559040ea8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559040e99480 0x559040ea7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 --2- 172.21.15.16:0/1664889005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559040e98b60 0x559040ea79f0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 >> 172.21.15.16:0/1664889005 conn(0x559040e950f0 msgr2=0x559040e9fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f1effd640 0 client.19857 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 shutdown_connections 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 1 -- 172.21.15.16:0/1664889005 wait complete. 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.493+0000 7f4f47f93740 20 client.19857 trim_cache size 0 max 16384 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:52.502 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:06:52.503 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "533b8c64-e088-4f60-a66a-4275eb4bf7ec", 2024-09-12T03:06:52.503 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:06:52.503 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 msgr2=0x7f4f280a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 0x7f4f280a50c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4f300ccfa0 tx=0x7f4f30086040 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 msgr2=0x7f4f400744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000cce0 tx=0x7f4f3000a620 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 shutdown_connections 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f40106f10 0x7f4f40074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f40102790 0x7f4f40074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4f280a2c00 0x7f4f280a50c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 --2- 172.21.15.16:0/3174561756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40101de0 0x7f4f400744f0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 >> 172.21.15.16:0/3174561756 conn(0x7f4f400fd8c0 msgr2=0x7f4f400fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 shutdown_connections 2024-09-12T03:06:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:52.494+0000 7f4f45bba640 1 -- 172.21.15.16:0/3174561756 wait complete. 2024-09-12T03:06:52.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:06:53.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 24.89 MB/sec execute 234 sec latency 18307.938 ms 2024-09-12T03:06:53.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:06:53.257 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:53.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: pgmap v906: 97 pgs: 97 active+clean; 696 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 5.6 MiB/s wr, 197 op/s 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:53.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:53 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: pgmap v906: 97 pgs: 97 active+clean; 696 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 5.6 MiB/s wr, 197 op/s 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:53 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: pgmap v906: 97 pgs: 97 active+clean; 696 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 5.6 MiB/s wr, 197 op/s 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:53.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:53 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 24.78 MB/sec execute 235 sec latency 19308.000 ms 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 -- 172.21.15.16:0/2506002012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 msgr2=0x7fa33810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/2506002012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33810e610 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa330009ca0 tx=0x7fa33002a950 comp rx=0 tx=0).stop 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 -- 172.21.15.16:0/2506002012 shutdown_connections 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/2506002012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33810e610 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/2506002012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338103660 0x7fa33810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/2506002012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338102d40 0x7fa338103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 -- 172.21.15.16:0/2506002012 >> 172.21.15.16:0/2506002012 conn(0x7fa3380fc7a0 msgr2=0x7fa3380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 -- 172.21.15.16:0/2506002012 shutdown_connections 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.753+0000 7fa33d4ec640 1 -- 172.21.15.16:0/2506002012 wait complete. 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 Processor -- start 2024-09-12T03:06:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 -- start start 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338102d40 0x7fa3381983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338103660 0x7fa338198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa338111f70 con 0x7fa33810c170 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa338111df0 con 0x7fa338103660 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa33d4ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381120f0 con 0x7fa338102d40 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33954/0 (socket says 172.21.15.16:33954) 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 -- 172.21.15.16:0/831082577 learned_addr learned my addr 172.21.15.16:0/831082577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa336ffd640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338102d40 0x7fa3381983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa32e7fc640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338103660 0x7fa338198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 -- 172.21.15.16:0/831082577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338102d40 msgr2=0x7fa3381983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338102d40 0x7fa3381983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 -- 172.21.15.16:0/831082577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338103660 msgr2=0x7fa338198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338103660 0x7fa338198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.754+0000 7fa3377fe640 1 -- 172.21.15.16:0/831082577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33819d3a0 con 0x7fa33810c170 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa3377fe640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa33002ae70 tx=0x7fa33002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa330034040 con 0x7fa33810c170 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:06:54.067 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa330030920 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33819d630 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa330045480 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.755+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3381a4f70 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.756+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33806b070 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.757+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa330037020 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.758+0000 7fa334ff9640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 0x7fa3240a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.758+0000 7fa336ffd640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 0x7fa3240a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.758+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa3300f00a0 con 0x7fa33810c170 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.758+0000 7fa336ffd640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 0x7fa3240a5040 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa328005d00 tx=0x7fa32800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.761+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa3300811b0 con 0x7fa33810c170 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.972+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3381071e0 con 0x7fa33810c170 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.973+0000 7fa334ff9640 1 -- 172.21.15.16:0/831082577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fa3300bcfb0 con 0x7fa33810c170 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 msgr2=0x7fa3240a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 0x7fa3240a5040 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa328005d00 tx=0x7fa32800b410 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 msgr2=0x7fa33819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa33002ae70 tx=0x7fa33002f5b0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 shutdown_connections 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa3240a2b80 0x7fa3240a5040 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa33810c170 0x7fa33819ccc0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa338103660 0x7fa338198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 --2- 172.21.15.16:0/831082577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa338102d40 0x7fa3381983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 >> 172.21.15.16:0/831082577 conn(0x7fa3380fc7a0 msgr2=0x7fa3380fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 shutdown_connections 2024-09-12T03:06:54.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:53.975+0000 7fa33d4ec640 1 -- 172.21.15.16:0/831082577 wait complete. 2024-09-12T03:06:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:54.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:54 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:54 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:54 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:06:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:54 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:54 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/831082577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:54.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:54 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:54.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:54 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:54 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:06:54.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:54 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:54 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/831082577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:54.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:54 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:54.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:54 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:54 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:06:54.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:54 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:06:54.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:54 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/831082577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:54.941 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id ec120ce2 --release --await' 2024-09-12T03:06:54.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id ec120ce2 --release --await 2024-09-12T03:06:55.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 24.67 MB/sec execute 236 sec latency 20308.295 ms 2024-09-12T03:06:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 -- 172.21.15.16:0/2772952583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100cc0 msgr2=0x7f48cc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 --2- 172.21.15.16:0/2772952583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100cc0 0x7f48cc106800 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009900 tx=0x7f48b402a410 comp rx=0 tx=0).stop 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 -- 172.21.15.16:0/2772952583 shutdown_connections 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 --2- 172.21.15.16:0/2772952583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc106f30 0x7f48cc10eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 --2- 172.21.15.16:0/2772952583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100cc0 0x7f48cc106800 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 --2- 172.21.15.16:0/2772952583 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc100310 0x7f48cc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 -- 172.21.15.16:0/2772952583 >> 172.21.15.16:0/2772952583 conn(0x7f48cc077210 msgr2=0x7f48cc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 -- 172.21.15.16:0/2772952583 shutdown_connections 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.150+0000 7f48d0aed640 1 -- 172.21.15.16:0/2772952583 wait complete. 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 Processor -- start 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 -- start start 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc100cc0 0x7f48cc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 0x7f48cc19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc111f90 con 0x7f48cc100310 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc111e10 con 0x7f48cc100cc0 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48d0aed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc112110 con 0x7f48cc106f30 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33980/0 (socket says 172.21.15.16:33980) 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48ca59c640 1 -- 172.21.15.16:0/4003815857 learned_addr learned my addr 172.21.15.16:0/4003815857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.151+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc100cc0 0x7f48cc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48cad9d640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 0x7f48cc19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 -- 172.21.15.16:0/4003815857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 msgr2=0x7f48cc19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 0x7f48cc19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 -- 172.21.15.16:0/4003815857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc100cc0 msgr2=0x7f48cc198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc100cc0 0x7f48cc198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 -- 172.21.15.16:0/4003815857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc19d400 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48cad9d640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 0x7f48cc19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00eab0 tx=0x7f48bc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48c37fe640 1 -- 172.21.15.16:0/4003815857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc00ce40 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48c37fe640 1 -- 172.21.15.16:0/4003815857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48bc011950 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc19d6f0 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48c37fe640 1 -- 172.21.15.16:0/4003815857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc010ae0 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.152+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc1a4fd0 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48c37fe640 1 -- 172.21.15.16:0/4003815857 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f48bc011b00 con 0x7f48cc100310 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 Processor -- start 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 -- start start 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096380 0x559fca096760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca0a5a90 0x559fca0a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fc9ffc2e0 con 0x559fca096ca0 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559fc9fa3ce0 con 0x559fca096380 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559fc9ff82a0 con 0x559fca0a5a90 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.155+0000 7f48c37fe640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 0x7f48b80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48cad9d640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 0x7f48b80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48cad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096380 0x559fca096760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33988/0 (socket says 172.21.15.16:33988) 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48c37fe640 1 -- 172.21.15.16:0/4003815857 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48bc0cf700 con 0x7f48cc100310 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 -- 172.21.15.16:0/3050865013 learned_addr learned my addr 172.21.15.16:0/3050865013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca0a5a90 0x559fca0a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 -- 172.21.15.16:0/3050865013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca0a5a90 msgr2=0x559fca0a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca0a5a90 0x559fca0a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 -- 172.21.15.16:0/3050865013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096380 msgr2=0x559fca096760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096380 0x559fca096760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 -- 172.21.15.16:0/3050865013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fc9f88440 con 0x559fca096ca0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48cad9d640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 0x7f48b80a5040 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f48c400df00 tx=0x7f48c400d5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48ca59c640 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f48bc01c040 tx=0x7f48bc015e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff63da4debd08f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.156+0000 7f48c0ff9640 1 -- 172.21.15.16:0/3050865013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc08d070 con 0x559fca096ca0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48c0ff9640 1 -- 172.21.15.16:0/3050865013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48bc09c670 con 0x559fca096ca0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48c0ff9640 1 -- 172.21.15.16:0/3050865013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0985e0 con 0x559fca096ca0 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 -- 172.21.15.16:0/3050865013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 msgr2=0x559fca0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f48bc01c040 tx=0x7f48bc015e70 comp rx=0 tx=0).stop 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 -- 172.21.15.16:0/3050865013 shutdown_connections 2024-09-12T03:06:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca0a5a90 0x559fca0a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0a5550 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/3050865013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096380 0x559fca096760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 -- 172.21.15.16:0/3050865013 >> 172.21.15.16:0/3050865013 conn(0x559fca092a30 msgr2=0x559fca092e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 -- 172.21.15.16:0/3050865013 shutdown_connections 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.157+0000 7f48d2ec6740 1 -- 172.21.15.16:0/3050865013 wait complete. 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 -- start start 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 Processor -- start 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 0x559fca0a2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fca0a8660 con 0x559fca0a5a90 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559fc9fb7050 con 0x559fca096ca0 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559fca0a87e0 con 0x559fca096380 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48d2ec6740 10 client.0 _ll_register_callbacks cb 0x559fc9ff7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34002/0 (socket says 172.21.15.16:34002) 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48cad9d640 1 -- 172.21.15.16:0/2263781419 learned_addr learned my addr 172.21.15.16:0/2263781419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48ca59c640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 0x559fca0a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48cad9d640 1 -- 172.21.15.16:0/2263781419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 msgr2=0x559fca0a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.158+0000 7f48cad9d640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48cad9d640 1 -- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 msgr2=0x559fca0a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48cad9d640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 0x559fca0a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48cad9d640 1 -- 172.21.15.16:0/2263781419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fc9d6d3c0 con 0x559fca0a5a90 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48ca59c640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 0x559fca0a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48cad9d640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f48c40117a0 tx=0x7f48c40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef23221e1ac5269 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4022070 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48c4012a60 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48bc09c670 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f48c4012c60 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 10 client.19875 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559fca0b4050 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48bc09c670 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f48c400f3c0 con 0x559fca0a5a90 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 10 client.19875 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 10 client.19875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.159+0000 7f48d2ec6740 20 client.19875 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0b5510 0x559fca0b7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48d2ec6740 4 client.19875 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fca009850 con 0x559fca0b5510 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48ca59c640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0b5510 0x559fca0b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48ca59c640 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0b5510 0x559fca0b7900 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=135b57dcd65dd264 server_cookie=ed68a126aab90e49 in_seq=0 out_seq=0 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.160+0000 7f48b1ffb640 10 client.19875 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48b1ffb640 1 -- 172.21.15.16:0/2263781419 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559fca009850 con 0x559fca0b5510 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48b1ffb640 10 client.19875 handle_command_reply: tid=0 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 client.19875 shutdown 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 msgr2=0x559fca0a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f48c40117a0 tx=0x7f48c40058b0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 shutdown_connections 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0b5510 0x559fca0b7900 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca0a5a90 0x559fca0a2b70 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca096ca0 0x559fca0a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/2263781419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 >> 172.21.15.16:0/2263781419 conn(0x559fca092a30 msgr2=0x559fca098790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48b1ffb640 0 client.19875 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 shutdown_connections 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 1 -- 172.21.15.16:0/2263781419 wait complete. 2024-09-12T03:06:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.161+0000 7f48d2ec6740 20 client.19875 trim_cache size 0 max 16384 2024-09-12T03:06:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 Processor -- start 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 -- start start 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0b1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 0x559fca0bd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1ab340 con 0x559fca096380 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1ace80 con 0x559fca092a30 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1aba10 con 0x559fca096ca0 2024-09-12T03:06:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34006/0 (socket says 172.21.15.16:34006) 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48c9d9b640 1 -- 172.21.15.16:0/463445335 learned_addr learned my addr 172.21.15.16:0/463445335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48ca59c640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.180+0000 7f48cad9d640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 0x559fca0bd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 -- 172.21.15.16:0/463445335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 msgr2=0x559fca0bd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 0x559fca0bd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 -- 172.21.15.16:0/463445335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 msgr2=0x559fca0b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 -- 172.21.15.16:0/463445335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fca0a0b80 con 0x559fca096380 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48cad9d640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 0x559fca0bd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f48b402a930 tx=0x7f48b4031880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677667ab7f486c72 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48b3fff640 1 -- 172.21.15.16:0/463445335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4035040 con 0x559fca096380 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48b3fff640 1 -- 172.21.15.16:0/463445335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b4005500 con 0x559fca096380 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.181+0000 7f48b3fff640 1 -- 172.21.15.16:0/463445335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4004d00 con 0x559fca096380 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 -- 172.21.15.16:0/463445335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 msgr2=0x559fca09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f48b402a930 tx=0x7f48b4031880 comp rx=0 tx=0).stop 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 -- 172.21.15.16:0/463445335 shutdown_connections 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096ca0 0x559fca0bd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096380 0x559fca09dc00 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/463445335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 -- 172.21.15.16:0/463445335 >> 172.21.15.16:0/463445335 conn(0x559fca0a2cd0 msgr2=0x559fca09d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 -- 172.21.15.16:0/463445335 shutdown_connections 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.182+0000 7f48d2ec6740 1 -- 172.21.15.16:0/463445335 wait complete. 2024-09-12T03:06:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 -- start start 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 Processor -- start 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1ac7b0 con 0x559fca096ca0 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1ac0e0 con 0x559fca092a30 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc1a9320 con 0x559fca096380 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48d2ec6740 10 client.0 _ll_register_callbacks cb 0x7f48c4014330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:35120/0 (socket says 172.21.15.16:35120) 2024-09-12T03:06:55.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48ca59c640 1 -- 172.21.15.16:0/4020115623 learned_addr learned my addr 172.21.15.16:0/4020115623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.183+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 -- 172.21.15.16:0/4020115623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 msgr2=0x559fca0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 -- 172.21.15.16:0/4020115623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 msgr2=0x559fca0aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 -- 172.21.15.16:0/4020115623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fca11f360 con 0x559fca092a30 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48cad9d640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48ca59c640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48bc09bc70 tx=0x7f48bc0cf900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745d11e9c313ab3f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc08d040 con 0x559fca092a30 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48bc0985e0 con 0x559fca092a30 2024-09-12T03:06:55.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b4005500 con 0x559fca092a30 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f48bc0b6050 con 0x559fca092a30 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48d2ec6740 10 client.29141 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.184+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559fca1206b0 con 0x559fca092a30 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b4005500 con 0x559fca092a30 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f48bc00f040 con 0x559fca092a30 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 10 client.29141 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 10 client.29141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 20 client.29141 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0a5a90 0x559fca122a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 4 client.29141 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559fca003d60 con 0x559fca0a5a90 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.185+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0a5a90 0x559fca122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48c9d9b640 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0a5a90 0x559fca122a00 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c960f3cad2e149d server_cookie=73d15358982a2020 in_seq=0 out_seq=0 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48b1ffb640 10 client.29141 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48b1ffb640 1 -- 172.21.15.16:0/4020115623 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559fca003d60 con 0x559fca0a5a90 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48b1ffb640 10 client.29141 handle_command_reply: tid=0 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 client.29141 shutdown 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 msgr2=0x559fca0a99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48bc09bc70 tx=0x7f48bc0cf900 comp rx=0 tx=0).stop 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 shutdown_connections 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559fca0a5a90 0x559fca122a00 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fca096ca0 0x559fca0aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559fca096380 0x559fca0a9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48b1ffb640 0 client.29141 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 --2- 172.21.15.16:0/4020115623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559fca092a30 0x559fca0a99d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 >> 172.21.15.16:0/4020115623 conn(0x559fca0a2cd0 msgr2=0x559fca0a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 shutdown_connections 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 1 -- 172.21.15.16:0/4020115623 wait complete. 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.186+0000 7f48d2ec6740 20 client.29141 trim_cache size 0 max 16384 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:06:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "533b8c64-e088-4f60-a66a-4275eb4bf7ec": { 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "533b8c64-e088-4f60-a66a-4275eb4bf7ec", 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:55.193 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 msgr2=0x7f48b80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 0x7f48b80a5040 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f48c400df00 tx=0x7f48c400d5b0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 msgr2=0x7f48cc198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00eab0 tx=0x7f48bc00ef70 comp rx=0 tx=0).stop 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 shutdown_connections 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48cc106f30 0x7f48cc19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48cc100cc0 0x7f48cc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48b80a2b80 0x7f48b80a5040 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 --2- 172.21.15.16:0/4003815857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48cc100310 0x7f48cc198450 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:55.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 >> 172.21.15.16:0/4003815857 conn(0x7f48cc077210 msgr2=0x7f48cc1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:55.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 shutdown_connections 2024-09-12T03:06:55.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:55.187+0000 7f48d0aed640 1 -- 172.21.15.16:0/4003815857 wait complete. 2024-09-12T03:06:55.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:55 smithi033 ceph-mon[35092]: pgmap v907: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-09-12T03:06:55.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:55 smithi184 ceph-mon[35321]: pgmap v907: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-09-12T03:06:55.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:55 smithi016 ceph-mon[30925]: pgmap v907: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-09-12T03:06:55.955 INFO:tasks.cephfs.filesystem:scrub status for tag:533b8c64-e088-4f60-a66a-4275eb4bf7ec - {'path': '/', 'tag': '533b8c64-e088-4f60-a66a-4275eb4bf7ec', 'options': 'recursive,force'} 2024-09-12T03:06:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:56.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 148746 24.57 MB/sec execute 237 sec latency 21308.340 ms 2024-09-12T03:06:56.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 -- 172.21.15.16:0/134981016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb08106f20 msgr2=0x7fcb0810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/134981016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb08106f20 0x7fcb0810eaa0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009ca0 tx=0x7fcb0002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 -- 172.21.15.16:0/134981016 shutdown_connections 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/134981016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb08106f20 0x7fcb0810eaa0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/134981016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb081027a0 0x7fcb081067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/134981016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 0x7fcb081021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 -- 172.21.15.16:0/134981016 >> 172.21.15.16:0/134981016 conn(0x7fcb080fd8b0 msgr2=0x7fcb080ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.495+0000 7fcb0f25d640 1 -- 172.21.15.16:0/134981016 shutdown_connections 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 -- 172.21.15.16:0/134981016 wait complete. 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 Processor -- start 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 -- start start 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 0x7fcb08198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb08106f20 0x7fcb0819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08111f80 con 0x7fcb081027a0 2024-09-12T03:06:56.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08111e00 con 0x7fcb08106f20 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb0f25d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08112100 con 0x7fcb08101df0 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54134/0 (socket says 172.21.15.16:54134) 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.496+0000 7fcb07fff640 1 -- 172.21.15.16:0/610841068 learned_addr learned my addr 172.21.15.16:0/610841068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb08106f20 0x7fcb0819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 0x7fcb08198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 -- 172.21.15.16:0/610841068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 msgr2=0x7fcb08198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 0x7fcb08198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 -- 172.21.15.16:0/610841068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb08106f20 msgr2=0x7fcb0819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb08106f20 0x7fcb0819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 -- 172.21.15.16:0/610841068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0819d2f0 con 0x7fcb081027a0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb07fff640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c920 tx=0x7fcafc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb05ffb640 1 -- 172.21.15.16:0/610841068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc007c70 con 0x7fcb081027a0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb05ffb640 1 -- 172.21.15.16:0/610841068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc007e10 con 0x7fcb081027a0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0819d5e0 con 0x7fcb081027a0 2024-09-12T03:06:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.497+0000 7fcb05ffb640 1 -- 172.21.15.16:0/610841068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc018890 con 0x7fcb081027a0 2024-09-12T03:06:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.498+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081a4ec0 con 0x7fcb081027a0 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb05ffb640 1 -- 172.21.15.16:0/610841068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcafc018a70 con 0x7fcb081027a0 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 Processor -- start 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 -- start start 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939d880 0x5631d93ac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631d9302620 con 0x5631d939cf60 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5631d92a9e50 con 0x5631d93ac650 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb11636740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5631d92fe5e0 con 0x5631d939d880 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.500+0000 7fcb0d7fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54142/0 (socket says 172.21.15.16:54142) 2024-09-12T03:06:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/3282756134 learned_addr learned my addr 172.21.15.16:0/3282756134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb07fff640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb05ffb640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 0x7fcaf00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939d880 0x5631d93ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 0x7fcaf00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/3282756134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939d880 msgr2=0x5631d93ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939d880 0x5631d93ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/3282756134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 msgr2=0x5631d93aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/3282756134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d929ad40 con 0x5631d939cf60 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb07fff640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb05ffb640 1 -- 172.21.15.16:0/610841068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcafc0cf090 con 0x7fcb081027a0 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.501+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009430 tx=0x7fcb00003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696a7ef3f7c142e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 0x7fcaf00a50c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcb00048000 tx=0x7fcb0004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcaeeffd640 1 -- 172.21.15.16:0/3282756134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00045040 con 0x5631d939cf60 2024-09-12T03:06:56.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcaeeffd640 1 -- 172.21.15.16:0/3282756134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb0004d030 con 0x5631d939cf60 2024-09-12T03:06:56.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcaeeffd640 1 -- 172.21.15.16:0/3282756134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000305c0 con 0x5631d939cf60 2024-09-12T03:06:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 -- 172.21.15.16:0/3282756134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 msgr2=0x5631d939d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009430 tx=0x7fcb00003b70 comp rx=0 tx=0).stop 2024-09-12T03:06:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 -- 172.21.15.16:0/3282756134 shutdown_connections 2024-09-12T03:06:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939d880 0x5631d93ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 --2- 172.21.15.16:0/3282756134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939cf60 0x5631d939d340 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.502+0000 7fcb11636740 1 -- 172.21.15.16:0/3282756134 >> 172.21.15.16:0/3282756134 conn(0x5631d93995f0 msgr2=0x5631d93999d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 -- 172.21.15.16:0/3282756134 shutdown_connections 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 -- 172.21.15.16:0/3282756134 wait complete. 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 -- start start 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 Processor -- start 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93a9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.503+0000 7fcb11636740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb11636740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631d93af220 con 0x5631d939d880 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb11636740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5631d928af10 con 0x5631d93ac650 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb11636740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5631d93af3a0 con 0x5631d939cf60 2024-09-12T03:06:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb11636740 10 client.0 _ll_register_callbacks cb 0x5631d92aedb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54158/0 (socket says 172.21.15.16:54158) 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 -- 172.21.15.16:0/2470232130 learned_addr learned my addr 172.21.15.16:0/2470232130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 -- 172.21.15.16:0/2470232130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 msgr2=0x5631d93a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93a9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 -- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 msgr2=0x5631d93aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 -- 172.21.15.16:0/2470232130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d9011580 con 0x5631d939d880 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c8f0 tx=0x7fcafc09b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a34140bf950ae78 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcae7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc09b900 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcafc00fb30 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.504+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb0004d030 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcafc0a4570 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 10 client.19893 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5631d90db960 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb0004d030 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcafc0d10f0 con 0x5631d939d880 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 10 client.19893 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 10 client.19893 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 20 client.19893 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d93bc270 0x5631d93be660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 4 client.19893 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631d930fb90 con 0x5631d93bc270 2024-09-12T03:06:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.505+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d93bc270 0x5631d93be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.506+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d93bc270 0x5631d93be660 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc03b090bc6e0af6 server_cookie=9e00fc4dad67462a in_seq=0 out_seq=0 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.506+0000 7fcae7fff640 10 client.19893 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.506+0000 7fcae7fff640 1 -- 172.21.15.16:0/2470232130 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5631d930fb90 con 0x5631d93bc270 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.506+0000 7fcae7fff640 10 client.19893 handle_command_reply: tid=0 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 client.19893 shutdown 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 msgr2=0x5631d93a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c8f0 tx=0x7fcafc09b500 comp rx=0 tx=0).stop 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 shutdown_connections 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d93bc270 0x5631d93be660 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d93ac650 0x5631d93aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93a9b40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 --2- 172.21.15.16:0/2470232130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 >> 172.21.15.16:0/2470232130 conn(0x5631d93995f0 msgr2=0x5631d939f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcae7fff640 0 client.19893 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 shutdown_connections 2024-09-12T03:06:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 1 -- 172.21.15.16:0/2470232130 wait complete. 2024-09-12T03:06:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.507+0000 7fcb11636740 20 client.19893 trim_cache size 0 max 16384 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 Processor -- start 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 -- start start 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 0x5631d93ab570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d93ac650 0x5631d93c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ab240 con 0x5631d939d880 2024-09-12T03:06:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081acd80 con 0x5631d939cf60 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb11636740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ab910 con 0x5631d93ac650 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54162/0 (socket says 172.21.15.16:54162) 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.524+0000 7fcb07fff640 1 -- 172.21.15.16:0/2043419325 learned_addr learned my addr 172.21.15.16:0/2043419325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 0x5631d93ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d93ac650 0x5631d93c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 -- 172.21.15.16:0/2043419325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d93ac650 msgr2=0x5631d93c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d93ac650 0x5631d93c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 -- 172.21.15.16:0/2043419325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 msgr2=0x5631d93ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 0x5631d93ab570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 -- 172.21.15.16:0/2043419325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d93abf70 con 0x5631d939d880 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 0x5631d93ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcb07fff640 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0982d0 tx=0x7fcafc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cf3e715cdf37fa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcaee7fc640 1 -- 172.21.15.16:0/2043419325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc018d60 con 0x5631d939d880 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcaee7fc640 1 -- 172.21.15.16:0/2043419325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc0b1e50 con 0x5631d939d880 2024-09-12T03:06:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.525+0000 7fcaee7fc640 1 -- 172.21.15.16:0/2043419325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0a40e0 con 0x5631d939d880 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- 172.21.15.16:0/2043419325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 msgr2=0x5631d93c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0982d0 tx=0x7fcafc000bc0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- 172.21.15.16:0/2043419325 shutdown_connections 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d93ac650 0x5631d93c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d939d880 0x5631d93c3f30 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 --2- 172.21.15.16:0/2043419325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939cf60 0x5631d93ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- 172.21.15.16:0/2043419325 >> 172.21.15.16:0/2043419325 conn(0x5631d93995f0 msgr2=0x5631d93aae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- 172.21.15.16:0/2043419325 shutdown_connections 2024-09-12T03:06:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- 172.21.15.16:0/2043419325 wait complete. 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.526+0000 7fcb11636740 1 -- start start 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 Processor -- start 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 0x5631d93b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ac6b0 con 0x5631d93ac650 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081abfe0 con 0x5631d939d880 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081a9220 con 0x5631d939cf60 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb0d7fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb11636740 10 client.0 _ll_register_callbacks cb 0x7fcafc00a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb0d7fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54172/0 (socket says 172.21.15.16:54172) 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/4107170674 learned_addr learned my addr 172.21.15.16:0/4107170674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb07fff640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 0x5631d93b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.527+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/4107170674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 msgr2=0x5631d93b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 msgr2=0x5631d93b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 0x5631d93b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 -- 172.21.15.16:0/4107170674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d9426540 con 0x5631d93ac650 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb07fff640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 0x5631d93b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb0d7fa640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fcb0002ade0 tx=0x7fcb00004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f31512180c65c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcae7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0005f060 con 0x5631d93ac650 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcb000318e0 con 0x5631d93ac650 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcafc0b1e50 con 0x5631d93ac650 2024-09-12T03:06:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcb00045040 con 0x5631d93ac650 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb11636740 10 client.19899 fetch_fsmap learned FSMap version 39 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.528+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5631d9427860 con 0x5631d93ac650 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc0b1e50 con 0x5631d93ac650 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcb00048030 con 0x5631d93ac650 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 10 client.19899 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 10 client.19899 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 20 client.19899 populate_metadata read hostname 'smithi016' 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d9427b90 0x5631d9429f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 4 client.19899 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ec120ce2", "release": true, "await": true}] 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ec120ce2", "release": true, "await": true}) -- 0x5631d93aa410 con 0x5631d9427b90 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.529+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d9427b90 0x5631d9429f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.530+0000 7fcae7fff640 10 client.19899 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.530+0000 7fcb0cff9640 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d9427b90 0x5631d9429f80 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fe9b68c51fb19b0 server_cookie=7e029b15c21edfdc in_seq=0 out_seq=0 2024-09-12T03:06:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcae7fff640 1 -- 172.21.15.16:0/4107170674 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5631d93aa410 con 0x5631d9427b90 2024-09-12T03:06:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcae7fff640 10 client.19899 handle_command_reply: tid=0 2024-09-12T03:06:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 client.19899 shutdown 2024-09-12T03:06:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 msgr2=0x5631d9425f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fcb0002ade0 tx=0x7fcb00004810 comp rx=0 tx=0).stop 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 shutdown_connections 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5631d9427b90 0x5631d9429f80 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d93ac650 0x5631d9425f00 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d939d880 0x5631d93b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 --2- 172.21.15.16:0/4107170674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d939cf60 0x5631d93b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcae7fff640 0 client.19899 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 >> 172.21.15.16:0/4107170674 conn(0x5631d93995f0 msgr2=0x5631d939f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 shutdown_connections 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 1 -- 172.21.15.16:0/4107170674 wait complete. 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.586+0000 7fcb11636740 20 client.19899 trim_cache size 0 max 16384 2024-09-12T03:06:56.589 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 68, 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "ec120ce2": { 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 68, 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:06:56.590 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:56.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 msgr2=0x7fcaf00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 0x7fcaf00a50c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcb00048000 tx=0x7fcb0004a000 comp rx=0 tx=0).stop 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 msgr2=0x7fcb08198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c920 tx=0x7fcafc00cde0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 shutdown_connections 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb08106f20 0x7fcb0819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcaf00a2c00 0x7fcaf00a50c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb081027a0 0x7fcb08198880 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 --2- 172.21.15.16:0/610841068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb08101df0 0x7fcb08198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 >> 172.21.15.16:0/610841068 conn(0x7fcb080fd8b0 msgr2=0x7fcb080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 shutdown_connections 2024-09-12T03:06:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:56.587+0000 7fcb0f25d640 1 -- 172.21.15.16:0/610841068 wait complete. 2024-09-12T03:06:57.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 149076 24.56 MB/sec execute 238 sec latency 21936.604 ms 2024-09-12T03:06:57.401 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ec120ce2', total seconds elapsed: 26.4 2024-09-12T03:06:57.401 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.8 2024-09-12T03:06:57.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:06:57.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:57 smithi016 ceph-mon[30925]: pgmap v908: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 5.2 MiB/s wr, 216 op/s 2024-09-12T03:06:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:57.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:57 smithi033 ceph-mon[35092]: pgmap v908: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 5.2 MiB/s wr, 216 op/s 2024-09-12T03:06:57.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:57 smithi184 ceph-mon[35321]: pgmap v908: 97 pgs: 97 active+clean; 701 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 5.2 MiB/s wr, 216 op/s 2024-09-12T03:06:58.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 149723 24.59 MB/sec execute 239 sec latency 113.825 ms 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.204+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/3966914669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 msgr2=0x7fc5e41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.204+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/3966914669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e41021d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009900 tx=0x7fc5e002a410 comp rx=0 tx=0).stop 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/3966914669 shutdown_connections 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/3966914669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/3966914669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/3966914669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e41021d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/3966914669 >> 172.21.15.16:0/3966914669 conn(0x7fc5e40fd8f0 msgr2=0x7fc5e40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/3966914669 shutdown_connections 2024-09-12T03:06:58.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.205+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/3966914669 wait complete. 2024-09-12T03:06:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 Processor -- start 2024-09-12T03:06:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 -- start start 2024-09-12T03:06:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:58.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4111f80 con 0x7fc5e4106f20 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4111e00 con 0x7fc5e41027a0 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ec2c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4112100 con 0x7fc5e4101df0 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ea865640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5e9863640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5e9863640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:50238/0 (socket says 172.21.15.16:50238) 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5e9863640 1 -- 172.21.15.16:0/2696957621 learned_addr learned my addr 172.21.15.16:0/2696957621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.206+0000 7fc5ea064640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 -- 172.21.15.16:0/2696957621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 msgr2=0x7fc5e4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 -- 172.21.15.16:0/2696957621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 msgr2=0x7fc5e4198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 -- 172.21.15.16:0/2696957621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e419d4f0 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5e9863640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ea865640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e419ce10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00da10 tx=0x7fc5dc00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc00bd70 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5dc003e50 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e419d7e0 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0106a0 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.207+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e41a50c0 con 0x7fc5e4106f20 2024-09-12T03:06:58.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.208+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e41021d0 con 0x7fc5e4106f20 2024-09-12T03:06:58.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.210+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc5dc004b40 con 0x7fc5e4106f20 2024-09-12T03:06:58.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.210+0000 7fc5db7fe640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 0x7fc5d00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:06:58.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.211+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc5dc0cf660 con 0x7fc5e4106f20 2024-09-12T03:06:58.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.211+0000 7fc5ea064640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 0x7fc5d00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:06:58.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.211+0000 7fc5ea064640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 0x7fc5d00a50c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009230 tx=0x7fc5e00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:06:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.214+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc5dc09c650 con 0x7fc5e4106f20 2024-09-12T03:06:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.546+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5e41998a0 con 0x7fc5e4106f20 2024-09-12T03:06:58.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.547+0000 7fc5db7fe640 1 -- 172.21.15.16:0/2696957621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fc5e41998a0 con 0x7fc5e4106f20 2024-09-12T03:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:06:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 msgr2=0x7fc5d00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 0x7fc5d00a50c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009230 tx=0x7fc5e00058d0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 msgr2=0x7fc5e419ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e419ce10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00da10 tx=0x7fc5dc00ded0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 shutdown_connections 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc5d00a2c00 0x7fc5d00a50c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e4106f20 0x7fc5e419ce10 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5e41027a0 0x7fc5e4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 --2- 172.21.15.16:0/2696957621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5e4101df0 0x7fc5e4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:06:58.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 >> 172.21.15.16:0/2696957621 conn(0x7fc5e40fd8f0 msgr2=0x7fc5e40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:06:58.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 shutdown_connections 2024-09-12T03:06:58.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:06:58.549+0000 7fc5ec2c8640 1 -- 172.21.15.16:0/2696957621 wait complete. 2024-09-12T03:06:59.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150288 24.63 MB/sec execute 240 sec latency 205.082 ms 2024-09-12T03:06:59.359 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0' 2024-09-12T03:06:59.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 141.8 --await-for 45.0 2024-09-12T03:06:59.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:59 smithi016 ceph-mon[30925]: pgmap v909: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 5.2 MiB/s wr, 226 op/s 2024-09-12T03:06:59.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:06:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2696957621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:06:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:59 smithi033 ceph-mon[35092]: pgmap v909: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 5.2 MiB/s wr, 226 op/s 2024-09-12T03:06:59.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:06:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2696957621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:06:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:59 smithi184 ceph-mon[35321]: pgmap v909: 97 pgs: 97 active+clean; 702 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 5.2 MiB/s wr, 226 op/s 2024-09-12T03:06:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:06:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2696957621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:00.022 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150786 24.65 MB/sec execute 241 sec latency 112.447 ms 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 -- 172.21.15.16:0/4118110071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8106a90 msgr2=0x7f0aa8106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 --2- 172.21.15.16:0/4118110071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8106a90 0x7f0aa8106e70 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009930 tx=0x7f0aa402a420 comp rx=0 tx=0).stop 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 -- 172.21.15.16:0/4118110071 shutdown_connections 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 --2- 172.21.15.16:0/4118110071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8101430 0x7f0aa810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 --2- 172.21.15.16:0/4118110071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8100a90 0x7f0aa8100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 --2- 172.21.15.16:0/4118110071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8106a90 0x7f0aa8106e70 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 -- 172.21.15.16:0/4118110071 >> 172.21.15.16:0/4118110071 conn(0x7f0aa80fc7d0 msgr2=0x7f0aa80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 -- 172.21.15.16:0/4118110071 shutdown_connections 2024-09-12T03:07:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.163+0000 7f0aafb32640 1 -- 172.21.15.16:0/4118110071 wait complete. 2024-09-12T03:07:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 Processor -- start 2024-09-12T03:07:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 -- start start 2024-09-12T03:07:00.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8101430 0x7f0aa8102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8106a90 0x7f0aa81030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8110e90 con 0x7f0aa8100a90 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8110d10 con 0x7f0aa8101430 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.164+0000 7f0aafb32640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8111010 con 0x7f0aa8106a90 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54220/0 (socket says 172.21.15.16:54220) 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 -- 172.21.15.16:0/2169269601 learned_addr learned my addr 172.21.15.16:0/2169269601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8101430 0x7f0aa8102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8106a90 0x7f0aa81030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 -- 172.21.15.16:0/2169269601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8106a90 msgr2=0x7f0aa81030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8106a90 0x7f0aa81030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 -- 172.21.15.16:0/2169269601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8101430 msgr2=0x7f0aa8102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8101430 0x7f0aa8102ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 -- 172.21.15.16:0/2169269601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa81a6910 con 0x7f0aa8100a90 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.165+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009230 tx=0x7f0aa4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.166+0000 7f0a9effd640 1 -- 172.21.15.16:0/2169269601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa403f030 con 0x7f0aa8100a90 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.166+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa81a6ba0 con 0x7f0aa8100a90 2024-09-12T03:07:00.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.166+0000 7f0a9effd640 1 -- 172.21.15.16:0/2169269601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aa4004440 con 0x7f0aa8100a90 2024-09-12T03:07:00.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.166+0000 7f0a9effd640 1 -- 172.21.15.16:0/2169269601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4035070 con 0x7f0aa8100a90 2024-09-12T03:07:00.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.166+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa81a7080 con 0x7f0aa8100a90 2024-09-12T03:07:00.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.169+0000 7f0a9effd640 1 -- 172.21.15.16:0/2169269601 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0aa4042030 con 0x7f0aa8100a90 2024-09-12T03:07:00.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.169+0000 7f0ab1f0b740 1 Processor -- start 2024-09-12T03:07:00.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.169+0000 7f0ab1f0b740 1 -- start start 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c45dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56076c3b62a0 con 0x56076c450760 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56076c35db10 con 0x56076c451080 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56076c3b2260 con 0x56076c45b940 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54222/0 (socket says 172.21.15.16:54222) 2024-09-12T03:07:00.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2631776297 learned_addr learned my addr 172.21.15.16:0/2631776297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c45dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2631776297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 msgr2=0x56076c45dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c45dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2631776297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 msgr2=0x56076c45b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.170+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2631776297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56076c127b90 con 0x56076c450760 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009f50 tx=0x7f0aa00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7405d7fe8765029 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0a9effd640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 0x7f0a940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0a9effd640 1 -- 172.21.15.16:0/2169269601 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0aa40ef310 con 0x7f0aa8100a90 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 0x7f0a940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.171+0000 7f0a93fff640 1 -- 172.21.15.16:0/2631776297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0004d20 con 0x56076c450760 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.172+0000 7f0a93fff640 1 -- 172.21.15.16:0/2631776297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aa0004050 con 0x56076c450760 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.172+0000 7f0a93fff640 1 -- 172.21.15.16:0/2631776297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0005280 con 0x56076c450760 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.172+0000 7f0aad8ce640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 0x7f0a940a5040 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40ed190 tx=0x7f0aa4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2631776297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 msgr2=0x56076c450b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009f50 tx=0x7f0aa00118d0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2631776297 shutdown_connections 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c45dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2631776297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c450b40 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.173+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2631776297 >> 172.21.15.16:0/2631776297 conn(0x56076c44ccf0 msgr2=0x56076c44d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.174+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2631776297 shutdown_connections 2024-09-12T03:07:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.174+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2631776297 wait complete. 2024-09-12T03:07:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.174+0000 7f0ab1f0b740 1 -- start start 2024-09-12T03:07:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.174+0000 7f0ab1f0b740 1 Processor -- start 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c450760 0x56076c458810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 0x56076c459290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56076c45e510 con 0x56076c45b940 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56076c33e6e0 con 0x56076c451080 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56076c45e690 con 0x56076c450760 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aad0cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aad0cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:50272/0 (socket says 172.21.15.16:50272) 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3378250535 learned_addr learned my addr 172.21.15.16:0/3378250535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c450760 0x56076c458810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0ab1f0b740 10 client.0 _ll_register_callbacks cb 0x56076c362a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 0x56076c459290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.175+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3378250535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c450760 msgr2=0x56076c458810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c450760 0x56076c458810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3378250535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 msgr2=0x56076c459290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 0x56076c459290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3378250535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56076c467a30 con 0x56076c451080 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 0x56076c459290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0a98005890 tx=0x7f0a98012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a38fe47fed4130 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0a917fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9800faf0 con 0x56076c451080 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0a9800fc90 con 0x56076c451080 2024-09-12T03:07:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0aa0004050 con 0x56076c451080 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0a9802d050 con 0x56076c451080 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0ab1f0b740 10 client.29171 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.176+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56076c46a3c0 con 0x56076c451080 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a98021030 con 0x56076c451080 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f0a98004bf0 con 0x56076c451080 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 10 client.29171 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 10 client.29171 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 20 client.29171 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c46b890 0x56076c46dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 4 client.29171 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.177+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0aa0004050 con 0x56076c46b890 2024-09-12T03:07:00.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.178+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c46b890 0x56076c46dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.178+0000 7f0a917fa640 10 client.29171 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:00.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.179+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c46b890 0x56076c46dcb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3235bcb0034c255 server_cookie=64a28ba7c474c463 in_seq=0 out_seq=0 2024-09-12T03:07:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.179+0000 7f0a917fa640 1 -- 172.21.15.16:0/3378250535 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f0aa0004050 con 0x56076c46b890 2024-09-12T03:07:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.179+0000 7f0a917fa640 10 client.29171 handle_command_reply: tid=0 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 client.29171 shutdown 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 msgr2=0x56076c458d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0a98005890 tx=0x7f0a98012b90 comp rx=0 tx=0).stop 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 shutdown_connections 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c46b890 0x56076c46dcb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c45b940 0x56076c459290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c458d50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3378250535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c450760 0x56076c458810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0a917fa640 0 client.29171 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 >> 172.21.15.16:0/3378250535 conn(0x56076c44ccf0 msgr2=0x56076c44be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.180+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 shutdown_connections 2024-09-12T03:07:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.181+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3378250535 wait complete. 2024-09-12T03:07:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.181+0000 7f0ab1f0b740 20 client.29171 trim_cache size 0 max 16384 2024-09-12T03:07:00.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 Processor -- start 2024-09-12T03:07:00.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 -- start start 2024-09-12T03:07:00.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c4590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c45b940 0x56076c473560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81ab320 con 0x56076c450760 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81ace60 con 0x56076c45b940 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.200+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81ab9f0 con 0x56076c451080 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aae0cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c4590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42672/0 (socket says 172.21.15.16:42672) 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3622941785 learned_addr learned my addr 172.21.15.16:0/3622941785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c45b940 0x56076c473560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3622941785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c45b940 msgr2=0x56076c473560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c45b940 0x56076c473560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3622941785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 msgr2=0x56076c4590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c4590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 -- 172.21.15.16:0/3622941785 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56076c44d980 con 0x56076c451080 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c4590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a980051e0 tx=0x7f0a980059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a9ab061d92e114 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0a937fe640 1 -- 172.21.15.16:0/3622941785 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98005460 con 0x56076c451080 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0a937fe640 1 -- 172.21.15.16:0/3622941785 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a9802a020 con 0x56076c451080 2024-09-12T03:07:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.201+0000 7f0a937fe640 1 -- 172.21.15.16:0/3622941785 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a980035f0 con 0x56076c451080 2024-09-12T03:07:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3622941785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 msgr2=0x56076c44bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a980051e0 tx=0x7f0a980059b0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3622941785 shutdown_connections 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c45b940 0x56076c473560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c451080 0x56076c44bd60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/3622941785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c4590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3622941785 >> 172.21.15.16:0/3622941785 conn(0x56076c44ccf0 msgr2=0x56076c44b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3622941785 shutdown_connections 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.202+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/3622941785 wait complete. 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 -- start start 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 Processor -- start 2024-09-12T03:07:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c460330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81ac790 con 0x56076c450760 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81ac0c0 con 0x56076c451080 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa81a9340 con 0x56076c45b940 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0ab1f0b740 10 client.0 _ll_register_callbacks cb 0x7f0a98009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0aae0cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0aae0cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54250/0 (socket says 172.21.15.16:54250) 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2657646638 learned_addr learned my addr 172.21.15.16:0/2657646638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.203+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c460330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2657646638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 msgr2=0x56076c460330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c460330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 msgr2=0x56076c45fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 -- 172.21.15.16:0/2657646638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56076c4d5730 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0aae0cf640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0005cc0 tx=0x7f0aa0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1ab6498102b863 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0a917fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0004db0 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0aa0003a30 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a9802a020 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0aa0003c30 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0ab1f0b740 10 client.19920 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56076c4d6a30 con 0x56076c450760 2024-09-12T03:07:00.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.204+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a9802a020 con 0x56076c450760 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f0aa000f480 con 0x56076c450760 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 10 client.19920 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 10 client.19920 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 20 client.19920 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c4d70e0 0x56076c4d9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 4 client.19920 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}] 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.8, "await_for": 45.0}) -- 0x56076c3c39d0 con 0x56076c4d70e0 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.205+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c4d70e0 0x56076c4d9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.207+0000 7f0aad0cd640 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c4d70e0 0x56076c4d9540 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cfb017d72535ecf server_cookie=6588e303e5be19e7 in_seq=0 out_seq=0 2024-09-12T03:07:00.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.207+0000 7f0a917fa640 10 client.19920 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.983+0000 7f0a917fa640 1 -- 172.21.15.16:0/2657646638 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56076c3c39d0 con 0x56076c4d70e0 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.983+0000 7f0a917fa640 10 client.19920 handle_command_reply: tid=0 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.983+0000 7f0ab1f0b740 1 client.19920 shutdown 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.983+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 msgr2=0x56076c45f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.983+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0005cc0 tx=0x7f0aa0005fd0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 shutdown_connections 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56076c4d70e0 0x56076c4d9540 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56076c45b940 0x56076c460330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56076c451080 0x56076c45fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 --2- 172.21.15.16:0/2657646638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56076c450760 0x56076c45f8b0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 >> 172.21.15.16:0/2657646638 conn(0x56076c44ccf0 msgr2=0x56076c44be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0a917fa640 0 client.19920 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 shutdown_connections 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 1 -- 172.21.15.16:0/2657646638 wait complete. 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.984+0000 7f0ab1f0b740 20 client.19920 trim_cache size 0 max 16384 2024-09-12T03:07:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 70, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "964a08e0": { 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 70, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:07:00.987 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 msgr2=0x7f0a940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 0x7f0a940a5040 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40ed190 tx=0x7f0aa4009260 comp rx=0 tx=0).stop 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 msgr2=0x7f0aa8102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009230 tx=0x7f0aa4004040 comp rx=0 tx=0).stop 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 shutdown_connections 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0aa8106a90 0x7f0aa81030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0aa8101430 0x7f0aa8102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0a940a2b80 0x7f0a940a5040 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 --2- 172.21.15.16:0/2169269601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa8100a90 0x7f0aa8102660 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.985+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 >> 172.21.15.16:0/2169269601 conn(0x7f0aa80fc7d0 msgr2=0x7f0aa81048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.986+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 shutdown_connections 2024-09-12T03:07:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:00.986+0000 7f0aafb32640 1 -- 172.21.15.16:0/2169269601 wait complete. 2024-09-12T03:07:01.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.60 MB/sec execute 242 sec latency 741.492 ms 2024-09-12T03:07:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:01 smithi033 ceph-mon[35092]: pgmap v910: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.0 MiB/s wr, 316 op/s 2024-09-12T03:07:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:01 smithi184 ceph-mon[35321]: pgmap v910: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.0 MiB/s wr, 316 op/s 2024-09-12T03:07:01.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:01 smithi016 ceph-mon[30925]: pgmap v910: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.0 MiB/s wr, 316 op/s 2024-09-12T03:07:01.834 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '964a08e0', quiesce took 4.4 seconds. Will release after: 17.4 2024-09-12T03:07:02.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.50 MB/sec execute 243 sec latency 1741.565 ms 2024-09-12T03:07:03.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.40 MB/sec execute 244 sec latency 2741.610 ms 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: pgmap v911: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:03.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: pgmap v911: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: pgmap v911: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 364 op/s 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:04.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.30 MB/sec execute 245 sec latency 3741.646 ms 2024-09-12T03:07:05.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.20 MB/sec execute 246 sec latency 4741.697 ms 2024-09-12T03:07:05.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:05 smithi033 ceph-mon[35092]: pgmap v912: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 368 op/s 2024-09-12T03:07:05.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:05 smithi184 ceph-mon[35321]: pgmap v912: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 368 op/s 2024-09-12T03:07:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:05 smithi016 ceph-mon[30925]: pgmap v912: 97 pgs: 97 active+clean; 699 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 368 op/s 2024-09-12T03:07:06.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.10 MB/sec execute 247 sec latency 5741.749 ms 2024-09-12T03:07:07.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 24.00 MB/sec execute 248 sec latency 6741.837 ms 2024-09-12T03:07:07.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:07 smithi033 ceph-mon[35092]: pgmap v913: 97 pgs: 97 active+clean; 703 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 401 op/s 2024-09-12T03:07:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:07 smithi184 ceph-mon[35321]: pgmap v913: 97 pgs: 97 active+clean; 703 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 401 op/s 2024-09-12T03:07:07.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:07 smithi016 ceph-mon[30925]: pgmap v913: 97 pgs: 97 active+clean; 703 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 401 op/s 2024-09-12T03:07:08.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.91 MB/sec execute 249 sec latency 7741.957 ms 2024-09-12T03:07:09.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.81 MB/sec execute 250 sec latency 8742.074 ms 2024-09-12T03:07:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:09 smithi033 ceph-mon[35092]: pgmap v914: 97 pgs: 97 active+clean; 704 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 374 op/s 2024-09-12T03:07:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:09 smithi184 ceph-mon[35321]: pgmap v914: 97 pgs: 97 active+clean; 704 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 374 op/s 2024-09-12T03:07:09.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:09 smithi016 ceph-mon[30925]: pgmap v914: 97 pgs: 97 active+clean; 704 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 374 op/s 2024-09-12T03:07:10.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.72 MB/sec execute 251 sec latency 9742.191 ms 2024-09-12T03:07:11.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.62 MB/sec execute 252 sec latency 10742.263 ms 2024-09-12T03:07:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:11 smithi184 ceph-mon[35321]: pgmap v915: 97 pgs: 97 active+clean; 696 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-09-12T03:07:11.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:11 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:07:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:11 smithi016 ceph-mon[30925]: pgmap v915: 97 pgs: 97 active+clean; 696 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-09-12T03:07:11.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:11 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:07:11.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:11 smithi033 ceph-mon[35092]: pgmap v915: 97 pgs: 97 active+clean; 696 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-09-12T03:07:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:11 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:07:12.023 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.53 MB/sec execute 253 sec latency 11742.315 ms 2024-09-12T03:07:12.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:12 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:07:12.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:12 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:07:12.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:12 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:07:12.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:07:12.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:07:12.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:07:12.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:12 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:07:12.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:12 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:07:12.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:12 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:07:13.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.44 MB/sec execute 254 sec latency 12742.421 ms 2024-09-12T03:07:13.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:13 smithi016 ceph-mon[30925]: pgmap v916: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 261 op/s 2024-09-12T03:07:13.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:13 smithi033 ceph-mon[35092]: pgmap v916: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 261 op/s 2024-09-12T03:07:14.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.34 MB/sec execute 255 sec latency 13742.484 ms 2024-09-12T03:07:14.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:13 smithi184 ceph-mon[35321]: pgmap v916: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 261 op/s 2024-09-12T03:07:15.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.25 MB/sec execute 256 sec latency 14742.553 ms 2024-09-12T03:07:15.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:15 smithi016 ceph-mon[30925]: pgmap v917: 97 pgs: 97 active+clean; 691 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-09-12T03:07:15.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:15 smithi033 ceph-mon[35092]: pgmap v917: 97 pgs: 97 active+clean; 691 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-09-12T03:07:16.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.16 MB/sec execute 257 sec latency 15742.645 ms 2024-09-12T03:07:16.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:15 smithi184 ceph-mon[35321]: pgmap v917: 97 pgs: 97 active+clean; 691 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-09-12T03:07:17.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 23.07 MB/sec execute 258 sec latency 16742.753 ms 2024-09-12T03:07:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:17 smithi016 ceph-mon[30925]: pgmap v918: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-09-12T03:07:17.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:17 smithi033 ceph-mon[35092]: pgmap v918: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-09-12T03:07:18.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 22.98 MB/sec execute 259 sec latency 17742.885 ms 2024-09-12T03:07:18.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:17 smithi184 ceph-mon[35321]: pgmap v918: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-09-12T03:07:19.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 22.90 MB/sec execute 260 sec latency 18743.008 ms 2024-09-12T03:07:19.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:07:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:19.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:19 smithi016 ceph-mon[30925]: pgmap v919: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-09-12T03:07:19.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:19 smithi033 ceph-mon[35092]: pgmap v919: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-09-12T03:07:20.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 -- 172.21.15.16:0/1735191045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 msgr2=0x7f5a001021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/1735191045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a001021c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009900 tx=0x7f59f002a410 comp rx=0 tx=0).stop 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 -- 172.21.15.16:0/1735191045 shutdown_connections 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/1735191045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00106f10 0x7f5a0010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/1735191045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00102790 0x7f5a001067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/1735191045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a001021c0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 -- 172.21.15.16:0/1735191045 >> 172.21.15.16:0/1735191045 conn(0x7f5a000fd8c0 msgr2=0x7f5a000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 -- 172.21.15.16:0/1735191045 shutdown_connections 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.993+0000 7f5a05aa4640 1 -- 172.21.15.16:0/1735191045 wait complete. 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 Processor -- start 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 -- start start 2024-09-12T03:07:20.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 0x7f5a0019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00106f10 0x7f5a0019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00111f70 con 0x7f5a00101de0 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00111df0 con 0x7f5a00102790 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f5a05aa4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a001120f0 con 0x7f5a00106f10 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f59ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f59ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35338/0 (socket says 172.21.15.16:35338) 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.994+0000 7f59ff7fe640 1 -- 172.21.15.16:0/249002361 learned_addr learned my addr 172.21.15.16:0/249002361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59feffd640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 0x7f5a0019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 -- 172.21.15.16:0/249002361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00106f10 msgr2=0x7f5a0019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59fffff640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00106f10 0x7f5a0019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:20.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00106f10 0x7f5a0019eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 -- 172.21.15.16:0/249002361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 msgr2=0x7f5a0019ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 0x7f5a0019ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 -- 172.21.15.16:0/249002361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0019f5c0 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59feffd640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 0x7f5a0019ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59ff7fe640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f59f0005ab0 tx=0x7f59f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f003f030 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0019f850 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f0003ce0 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0035070 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.995+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a001a7190 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.996+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a001021c0 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.998+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f59f0042030 con 0x7f5a00101de0 2024-09-12T03:07:20.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.998+0000 7f59fcff9640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 0x7f59e80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.998+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f59f00ef9b0 con 0x7f5a00101de0 2024-09-12T03:07:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.998+0000 7f59feffd640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 0x7f59e80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:19.999+0000 7f59feffd640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 0x7f59e80a5040 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5a0010ee50 tx=0x7f59f4005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.002+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f59f00bc920 con 0x7f5a00101de0 2024-09-12T03:07:20.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 22.81 MB/sec execute 261 sec latency 19743.075 ms 2024-09-12T03:07:20.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:19 smithi184 ceph-mon[35321]: pgmap v919: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-09-12T03:07:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.215+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a001a7360 con 0x7f5a00101de0 2024-09-12T03:07:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.216+0000 7f59fcff9640 1 -- 172.21.15.16:0/249002361 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f59f00bc220 con 0x7f5a00101de0 2024-09-12T03:07:20.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:07:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:07:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 msgr2=0x7f59e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 0x7f59e80a5040 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5a0010ee50 tx=0x7f59f4005d30 comp rx=0 tx=0).stop 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 msgr2=0x7f5a0019a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f59f0005ab0 tx=0x7f59f00042f0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 shutdown_connections 2024-09-12T03:07:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a00106f10 0x7f5a0019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a00102790 0x7f5a0019ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f59e80a2b80 0x7f59e80a5040 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 --2- 172.21.15.16:0/249002361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a00101de0 0x7f5a0019a610 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 >> 172.21.15.16:0/249002361 conn(0x7f5a000fd8c0 msgr2=0x7f5a000ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 shutdown_connections 2024-09-12T03:07:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:20.218+0000 7f5a05aa4640 1 -- 172.21.15.16:0/249002361 wait complete. 2024-09-12T03:07:20.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/249002361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:20.937 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 964a08e0 --release --await' 2024-09-12T03:07:20.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 964a08e0 --release --await 2024-09-12T03:07:20.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/249002361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:21.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 150935 22.72 MB/sec execute 262 sec latency 20743.122 ms 2024-09-12T03:07:21.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/249002361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:21.215 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 -- 172.21.15.16:0/637228604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102790 msgr2=0x7f89681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 --2- 172.21.15.16:0/637228604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102790 0x7f89681067e0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8950009900 tx=0x7f895002a410 comp rx=0 tx=0).stop 2024-09-12T03:07:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 -- 172.21.15.16:0/637228604 shutdown_connections 2024-09-12T03:07:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 --2- 172.21.15.16:0/637228604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968106f10 0x7f896810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 --2- 172.21.15.16:0/637228604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102790 0x7f89681067e0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 --2- 172.21.15.16:0/637228604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968101de0 0x7f89681021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 -- 172.21.15.16:0/637228604 >> 172.21.15.16:0/637228604 conn(0x7f89680fd8c0 msgr2=0x7f89680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 -- 172.21.15.16:0/637228604 shutdown_connections 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.693+0000 7f896e951640 1 -- 172.21.15.16:0/637228604 wait complete. 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 Processor -- start 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 -- start start 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968102790 0x7f8968074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968106f10 0x7f8968074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8968111fc0 con 0x7f8968101de0 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8968111e40 con 0x7f8968102790 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896e951640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8968112140 con 0x7f8968106f10 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35364/0 (socket says 172.21.15.16:35364) 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 -- 172.21.15.16:0/2894966845 learned_addr learned my addr 172.21.15.16:0/2894966845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f896ceee640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968106f10 0x7f8968074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f895ffff640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968102790 0x7f8968074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 -- 172.21.15.16:0/2894966845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968106f10 msgr2=0x7f8968074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968106f10 0x7f8968074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 -- 172.21.15.16:0/2894966845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968102790 msgr2=0x7f8968074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968102790 0x7f8968074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.694+0000 7f8967fff640 1 -- 172.21.15.16:0/2894966845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89681a6890 con 0x7f8968101de0 2024-09-12T03:07:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f8967fff640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f895800d770 tx=0x7f895800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f8965ffb640 1 -- 172.21.15.16:0/2894966845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958017070 con 0x7f8968101de0 2024-09-12T03:07:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89681a6b80 con 0x7f8968101de0 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f8965ffb640 1 -- 172.21.15.16:0/2894966845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8958005500 con 0x7f8968101de0 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f8965ffb640 1 -- 172.21.15.16:0/2894966845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958004e00 con 0x7f8968101de0 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.695+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89681a7090 con 0x7f8968101de0 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.697+0000 7f8965ffb640 1 -- 172.21.15.16:0/2894966845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8958004fa0 con 0x7f8968101de0 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.697+0000 7f8970d2a740 1 Processor -- start 2024-09-12T03:07:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.697+0000 7f8970d2a740 1 -- start start 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b4d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 0x55f724b57d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b5a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f724ab2620 con 0x55f724b4d980 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f724a59e50 con 0x55f724b58240 2024-09-12T03:07:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8970d2a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f724aae5e0 con 0x55f724b4d060 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 0x55f724b57d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8965ffb640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 0x7f89540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 0x55f724b57d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35378/0 (socket says 172.21.15.16:35378) 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8967fff640 1 -- 172.21.15.16:0/1083019565 learned_addr learned my addr 172.21.15.16:0/1083019565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b4d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 0x7f89540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f895ffff640 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b5a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f8965ffb640 1 -- 172.21.15.16:0/2894966845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f89580ceab0 con 0x7f8968101de0 2024-09-12T03:07:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 -- 172.21.15.16:0/1083019565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 msgr2=0x55f724b5a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b5a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 -- 172.21.15.16:0/1083019565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 msgr2=0x55f724b57d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 0x55f724b57d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.698+0000 7f896ceee640 1 -- 172.21.15.16:0/1083019565 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f724a4ad40 con 0x55f724b4d060 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.699+0000 7f896ceee640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 0x7f89540a5040 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8960007fd0 tx=0x7f8960016000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.699+0000 7f896ceee640 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b4d440 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8960014dd0 tx=0x7f89600117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31bbfa9e119601fc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.699+0000 7f895e7fc640 1 -- 172.21.15.16:0/1083019565 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960024030 con 0x55f724b4d060 2024-09-12T03:07:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.699+0000 7f895e7fc640 1 -- 172.21.15.16:0/1083019565 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8960007ba0 con 0x55f724b4d060 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.699+0000 7f895e7fc640 1 -- 172.21.15.16:0/1083019565 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960004030 con 0x55f724b4d060 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- 172.21.15.16:0/1083019565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 msgr2=0x55f724b4d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b4d440 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8960014dd0 tx=0x7f89600117b0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- 172.21.15.16:0/1083019565 shutdown_connections 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b5a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d980 0x55f724b57d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1083019565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b4d440 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- 172.21.15.16:0/1083019565 >> 172.21.15.16:0/1083019565 conn(0x55f724b495f0 msgr2=0x55f724b499d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- 172.21.15.16:0/1083019565 shutdown_connections 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- 172.21.15.16:0/1083019565 wait complete. 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 -- start start 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 Processor -- start 2024-09-12T03:07:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b55620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.700+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b55b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f724b5ae10 con 0x55f724b4d060 2024-09-12T03:07:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f724a3af10 con 0x55f724b58240 2024-09-12T03:07:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f724b5af90 con 0x55f724b4d980 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 10 client.0 _ll_register_callbacks cb 0x55f724a5edb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35380/0 (socket says 172.21.15.16:35380) 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 -- 172.21.15.16:0/3863959841 learned_addr learned my addr 172.21.15.16:0/3863959841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f895ffff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b55620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f896ceee640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b55b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 -- 172.21.15.16:0/3863959841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 msgr2=0x55f724b55620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b55620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 -- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 msgr2=0x55f724b55b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b55b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 -- 172.21.15.16:0/3863959841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7247c1580 con 0x55f724b4d060 2024-09-12T03:07:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f896ceee640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b55b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8967fff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f8958097780 tx=0x7f89580048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a209464715a758ec server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f89477fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958020040 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8958099620 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8960007ba0 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f89580975c0 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 10 client.19938 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.701+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f72488b960 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8960007ba0 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f8958097990 con 0x55f724b4d060 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 10 client.19938 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 10 client.19938 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 20 client.19938 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724b67db0 0x55f724b6a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 4 client.19938 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f724abfb90 con 0x55f724b67db0 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.702+0000 7f895ffff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724b67db0 0x55f724b6a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.703+0000 7f895ffff640 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724b67db0 0x55f724b6a1a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f6f10c62bb1313f server_cookie=5f92d528c387017c in_seq=0 out_seq=0 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.703+0000 7f89477fe640 10 client.19938 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.703+0000 7f89477fe640 1 -- 172.21.15.16:0/3863959841 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f724abfb90 con 0x55f724b67db0 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.703+0000 7f89477fe640 10 client.19938 handle_command_reply: tid=0 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.703+0000 7f8970d2a740 1 client.19938 shutdown 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 msgr2=0x55f724b550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f8958097780 tx=0x7f89580048a0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 shutdown_connections 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724b67db0 0x55f724b6a1a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b58240 0x55f724b55b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b55620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 --2- 172.21.15.16:0/3863959841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b4d060 0x55f724b550e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 >> 172.21.15.16:0/3863959841 conn(0x55f724b495f0 msgr2=0x55f724b48750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f89477fe640 0 client.19938 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 shutdown_connections 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 1 -- 172.21.15.16:0/3863959841 wait complete. 2024-09-12T03:07:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.704+0000 7f8970d2a740 20 client.19938 trim_cache size 0 max 16384 2024-09-12T03:07:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 Processor -- start 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 -- start start 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d060 0x55f724b6fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b6ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89681af650 con 0x55f724b58240 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89681b1190 con 0x55f724b4d060 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8970d2a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89681afd20 con 0x55f724b4d980 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f896ceee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f896ceee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35386/0 (socket says 172.21.15.16:35386) 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f896ceee640 1 -- 172.21.15.16:0/1160093202 learned_addr learned my addr 172.21.15.16:0/1160093202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.721+0000 7f8967fff640 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d060 0x55f724b6fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f895ffff640 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b6ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 -- 172.21.15.16:0/1160093202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 msgr2=0x55f724b6ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b6ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 -- 172.21.15.16:0/1160093202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d060 msgr2=0x55f724b6fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d060 0x55f724b6fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 -- 172.21.15.16:0/1160093202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f724b4a2c0 con 0x55f724b58240 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f896ceee640 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f8960007e80 tx=0x7f8960011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce4fad4c68f561d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f895dffb640 1 -- 172.21.15.16:0/1160093202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896000cbf0 con 0x55f724b58240 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f895dffb640 1 -- 172.21.15.16:0/1160093202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f896000cd90 con 0x55f724b58240 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.722+0000 7f895dffb640 1 -- 172.21.15.16:0/1160093202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89600057e0 con 0x55f724b58240 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- 172.21.15.16:0/1160093202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 msgr2=0x55f724b52380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f8960007e80 tx=0x7f8960011d60 comp rx=0 tx=0).stop 2024-09-12T03:07:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- 172.21.15.16:0/1160093202 shutdown_connections 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b52380 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d980 0x55f724b6ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 --2- 172.21.15.16:0/1160093202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d060 0x55f724b6fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- 172.21.15.16:0/1160093202 >> 172.21.15.16:0/1160093202 conn(0x55f724b495f0 msgr2=0x55f724b47f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- 172.21.15.16:0/1160093202 shutdown_connections 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- 172.21.15.16:0/1160093202 wait complete. 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 -- start start 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.723+0000 7f8970d2a740 1 Processor -- start 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 0x55f724b60b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89681b0ac0 con 0x55f724b58240 2024-09-12T03:07:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f89681b03f0 con 0x55f724b4d980 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89681ad630 con 0x55f724b4d060 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8970d2a740 10 client.0 _ll_register_callbacks cb 0x7f895809ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f8967fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35394/0 (socket says 172.21.15.16:35394) 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 -- 172.21.15.16:0/2495949641 learned_addr learned my addr 172.21.15.16:0/2495949641 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f895ffff640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 0x55f724b60b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 -- 172.21.15.16:0/2495949641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 msgr2=0x55f724b605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 -- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 msgr2=0x55f724b60b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 0x55f724b60b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 -- 172.21.15.16:0/2495949641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f724b5dbe0 con 0x55f724b58240 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f895ffff640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 0x55f724b60b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f896ceee640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f8960007a70 tx=0x7f8960007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf00ad8aaf036e5b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.724+0000 7f89477fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960039070 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8960003b80 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f896000cd90 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8960024040 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 10 client.19944 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f724bd3710 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f896000cd90 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f896000f690 con 0x55f724b58240 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 10 client.19944 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 10 client.19944 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 20 client.19944 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724bd3c30 0x55f724bd6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 4 client.19944 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "964a08e0", "release": true, "await": true}] 2024-09-12T03:07:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "964a08e0", "release": true, "await": true}) -- 0x55f724aba0a0 con 0x55f724bd3c30 2024-09-12T03:07:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.725+0000 7f8967fff640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724bd3c30 0x55f724bd6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.726+0000 7f8967fff640 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724bd3c30 0x55f724bd6020 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5983f4666662557 server_cookie=f4a3da489013ff88 in_seq=0 out_seq=0 2024-09-12T03:07:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.726+0000 7f89477fe640 10 client.19944 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f89477fe640 1 -- 172.21.15.16:0/2495949641 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f724aba0a0 con 0x55f724bd3c30 2024-09-12T03:07:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f89477fe640 10 client.19944 handle_command_reply: tid=0 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 client.19944 shutdown 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 msgr2=0x55f724b5d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f8960007a70 tx=0x7f8960007d10 comp rx=0 tx=0).stop 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 shutdown_connections 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f724bd3c30 0x55f724bd6020 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f724b58240 0x55f724b5d3b0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f724b4d980 0x55f724b60b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f89477fe640 0 client.19944 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 --2- 172.21.15.16:0/2495949641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f724b4d060 0x55f724b605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 >> 172.21.15.16:0/2495949641 conn(0x55f724b495f0 msgr2=0x55f724b550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 shutdown_connections 2024-09-12T03:07:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 1 -- 172.21.15.16:0/2495949641 wait complete. 2024-09-12T03:07:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.781+0000 7f8970d2a740 20 client.19944 trim_cache size 0 max 16384 2024-09-12T03:07:21.786 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:07:21.786 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:07:21.786 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:07:21.787 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 72, 2024-09-12T03:07:21.787 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:07:21.787 INFO:teuthology.orchestra.run.smithi016.stdout: "964a08e0": { 2024-09-12T03:07:21.787 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 72, 2024-09-12T03:07:21.787 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:07:21.788 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.8, 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:07:21.789 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:21.790 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 msgr2=0x7f89540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 0x7f89540a5040 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8960007fd0 tx=0x7f8960016000 comp rx=0 tx=0).stop 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 msgr2=0x7f89680744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f895800d770 tx=0x7f895800dc30 comp rx=0 tx=0).stop 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 shutdown_connections 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8968106f10 0x7f8968074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8968102790 0x7f8968074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89540a2b80 0x7f89540a5040 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 --2- 172.21.15.16:0/2894966845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968101de0 0x7f89680744f0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 >> 172.21.15.16:0/2894966845 conn(0x7f89680fd8c0 msgr2=0x7f89680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 shutdown_connections 2024-09-12T03:07:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:21.782+0000 7f896e951640 1 -- 172.21.15.16:0/2894966845 wait complete. 2024-09-12T03:07:21.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:21 smithi016 ceph-mon[30925]: pgmap v920: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 464 KiB/s wr, 52 op/s 2024-09-12T03:07:21.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:21 smithi033 ceph-mon[35092]: pgmap v920: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 464 KiB/s wr, 52 op/s 2024-09-12T03:07:22.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 151068 22.67 MB/sec execute 263 sec latency 21511.166 ms 2024-09-12T03:07:22.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:21 smithi184 ceph-mon[35321]: pgmap v920: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 464 KiB/s wr, 52 op/s 2024-09-12T03:07:22.469 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '964a08e0', total seconds elapsed: 25.1 2024-09-12T03:07:23.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 151681 22.69 MB/sec execute 264 sec latency 55.842 ms 2024-09-12T03:07:23.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:23 smithi033 ceph-mon[35092]: pgmap v921: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 268 KiB/s wr, 27 op/s 2024-09-12T03:07:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 152252 22.71 MB/sec execute 265 sec latency 47.866 ms 2024-09-12T03:07:24.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:23 smithi184 ceph-mon[35321]: pgmap v921: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 268 KiB/s wr, 27 op/s 2024-09-12T03:07:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:23 smithi016 ceph-mon[30925]: pgmap v921: 97 pgs: 97 active+clean; 690 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 268 KiB/s wr, 27 op/s 2024-09-12T03:07:25.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 152967 22.76 MB/sec execute 266 sec latency 40.061 ms 2024-09-12T03:07:25.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:07:25.985 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:25 smithi016 ceph-mon[30925]: pgmap v922: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 236 KiB/s wr, 55 op/s 2024-09-12T03:07:25.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:25 smithi033 ceph-mon[35092]: pgmap v922: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 236 KiB/s wr, 55 op/s 2024-09-12T03:07:26.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 153709 22.82 MB/sec execute 267 sec latency 38.783 ms 2024-09-12T03:07:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:25 smithi184 ceph-mon[35321]: pgmap v922: 97 pgs: 97 active+clean; 692 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 236 KiB/s wr, 55 op/s 2024-09-12T03:07:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 -- 172.21.15.16:0/2943614797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102570 msgr2=0x7f7ca81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 --2- 172.21.15.16:0/2943614797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102570 0x7f7ca81029b0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009900 tx=0x7f7c9802a480 comp rx=0 tx=0).stop 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 -- 172.21.15.16:0/2943614797 shutdown_connections 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 --2- 172.21.15.16:0/2943614797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102ef0 0x7f7ca810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 --2- 172.21.15.16:0/2943614797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102570 0x7f7ca81029b0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 --2- 172.21.15.16:0/2943614797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 -- 172.21.15.16:0/2943614797 >> 172.21.15.16:0/2943614797 conn(0x7f7ca80fc7d0 msgr2=0x7f7ca80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 -- 172.21.15.16:0/2943614797 shutdown_connections 2024-09-12T03:07:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.795+0000 7f7cafc59640 1 -- 172.21.15.16:0/2943614797 wait complete. 2024-09-12T03:07:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 Processor -- start 2024-09-12T03:07:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 -- start start 2024-09-12T03:07:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca8110f50 con 0x7f7ca8102ef0 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca8110dd0 con 0x7f7ca8102570 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cafc59640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81110d0 con 0x7f7ca8100e20 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38854/0 (socket says 172.21.15.16:38854) 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cad1f4640 1 -- 172.21.15.16:0/1227545452 learned_addr learned my addr 172.21.15.16:0/1227545452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42728/0 (socket says 172.21.15.16:42728) 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.796+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 -- 172.21.15.16:0/1227545452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 msgr2=0x7f7ca81985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca81985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 -- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 msgr2=0x7f7ca819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 -- 172.21.15.16:0/1227545452 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca819d550 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cae1f6640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7c98006010 tx=0x7f7c98003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7c9effd640 1 -- 172.21.15.16:0/1227545452 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9803f070 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca819d7e0 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7c9effd640 1 -- 172.21.15.16:0/1227545452 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c98031c30 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7c9effd640 1 -- 172.21.15.16:0/1227545452 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98035030 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.797+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81a5120 con 0x7f7ca8102570 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 Processor -- start 2024-09-12T03:07:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 -- start start 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7c9effd640 1 -- 172.21.15.16:0/1227545452 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7c98042070 con 0x7f7ca8102570 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b61550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b63e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5593f0ab82e0 con 0x5593f0b523a0 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5593f0a5fce0 con 0x5593f0b61a90 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.800+0000 7f7cb2032740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5593f0ab42a0 con 0x5593f0b52cc0 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b63e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42740/0 (socket says 172.21.15.16:42740) 2024-09-12T03:07:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 -- 172.21.15.16:0/173614674 learned_addr learned my addr 172.21.15.16:0/173614674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b61550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 -- 172.21.15.16:0/173614674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 msgr2=0x5593f0b61550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b61550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 -- 172.21.15.16:0/173614674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 msgr2=0x5593f0b63e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b63e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cae1f6640 1 -- 172.21.15.16:0/173614674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f0a44440 con 0x5593f0b523a0 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b63e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.801+0000 7f7c9effd640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 0x7f7c940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7cae1f6640 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0011790 tx=0x7f7ca0011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d7a376bcfb7f82 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 0x7f7c940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7c9effd640 1 -- 172.21.15.16:0/1227545452 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c980ef320 con 0x7f7ca8102570 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7c93fff640 1 -- 172.21.15.16:0/173614674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0003c30 con 0x5593f0b523a0 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7c93fff640 1 -- 172.21.15.16:0/173614674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca000fe80 con 0x5593f0b523a0 2024-09-12T03:07:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.802+0000 7f7c93fff640 1 -- 172.21.15.16:0/173614674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca00193e0 con 0x5593f0b523a0 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 -- 172.21.15.16:0/173614674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 msgr2=0x5593f0b52780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 0x7f7c940a5040 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x5593f0b55090 tx=0x7f7ca4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0011790 tx=0x7f7ca0011c50 comp rx=0 tx=0).stop 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 -- 172.21.15.16:0/173614674 shutdown_connections 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b63e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b61550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 --2- 172.21.15.16:0/173614674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b52780 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.803+0000 7f7cb2032740 1 -- 172.21.15.16:0/173614674 >> 172.21.15.16:0/173614674 conn(0x5593f0b4ea30 msgr2=0x5593f0b4ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.804+0000 7f7cb2032740 1 -- 172.21.15.16:0/173614674 shutdown_connections 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.804+0000 7f7cb2032740 1 -- 172.21.15.16:0/173614674 wait complete. 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.804+0000 7f7cb2032740 1 -- start start 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.804+0000 7f7cb2032740 1 Processor -- start 2024-09-12T03:07:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b5e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b5ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5593f0b646c0 con 0x5593f0b523a0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5593f0a73050 con 0x5593f0b61a90 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5593f0b64840 con 0x5593f0b52cc0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cb2032740 10 client.0 _ll_register_callbacks cb 0x5593f0ab3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42748/0 (socket says 172.21.15.16:42748) 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b5ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b5e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 -- 172.21.15.16:0/374403075 learned_addr learned my addr 172.21.15.16:0/374403075 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 -- 172.21.15.16:0/374403075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 msgr2=0x5593f0b5e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b5e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 -- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 msgr2=0x5593f0b5ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b5ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.805+0000 7f7cad9f5640 1 -- 172.21.15.16:0/374403075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f08293c0 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7c917fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009560 tx=0x7f7ca4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5883d6e28d13665d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4004020 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7ca4018030 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5593f0ac5850 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7ca40041c0 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7cb2032740 10 client.19956 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5593f0b55140 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.806+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5593f0ac5850 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7ca4003710 con 0x5593f0b523a0 2024-09-12T03:07:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 10 client.19956 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 10 client.19956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 20 client.19956 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0b56600 0x5593f0b58a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 4 client.19956 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593f0abfd60 con 0x5593f0b56600 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0b56600 0x5593f0b58a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0b56600 0x5593f0b58a30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f04975ed0cac686 server_cookie=fc2c646f1c3deb89 in_seq=0 out_seq=0 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.807+0000 7f7c917fa640 10 client.19956 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7c917fa640 1 -- 172.21.15.16:0/374403075 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593f0abfd60 con 0x5593f0b56600 2024-09-12T03:07:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7c917fa640 10 client.19956 handle_command_reply: tid=0 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 client.19956 shutdown 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 msgr2=0x5593f0b613f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009560 tx=0x7f7ca4004730 comp rx=0 tx=0).stop 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 shutdown_connections 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0b56600 0x5593f0b58a30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b61a90 0x5593f0b5ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b52cc0 0x5593f0b5e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 --2- 172.21.15.16:0/374403075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b613f0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7c917fa640 0 client.19956 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.809+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 >> 172.21.15.16:0/374403075 conn(0x5593f0b4ea30 msgr2=0x5593f0b54760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.810+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 shutdown_connections 2024-09-12T03:07:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.810+0000 7f7cb2032740 1 -- 172.21.15.16:0/374403075 wait complete. 2024-09-12T03:07:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.810+0000 7f7cb2032740 20 client.19956 trim_cache size 0 max 16384 2024-09-12T03:07:26.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.830+0000 7f7cb2032740 1 Processor -- start 2024-09-12T03:07:26.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.830+0000 7f7cb2032740 1 -- start start 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 0x5593f0b75550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b5b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81ab440 con 0x5593f0b523a0 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81acf80 con 0x5593f0b52cc0 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cb2032740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81abb10 con 0x5593f0b61a90 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 0x5593f0b75550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42756/0 (socket says 172.21.15.16:42756) 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 -- 172.21.15.16:0/2472754541 learned_addr learned my addr 172.21.15.16:0/2472754541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cae1f6640 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b5b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 -- 172.21.15.16:0/2472754541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 msgr2=0x5593f0b5b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b5b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 -- 172.21.15.16:0/2472754541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 msgr2=0x5593f0b75550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 0x5593f0b75550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 -- 172.21.15.16:0/2472754541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f0b5f1e0 con 0x5593f0b523a0 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 0x5593f0b75550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.831+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400ffd0 tx=0x7f7ca4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedab65af68c07da server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.832+0000 7f7c937fe640 1 -- 172.21.15.16:0/2472754541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4038070 con 0x5593f0b523a0 2024-09-12T03:07:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.832+0000 7f7c937fe640 1 -- 172.21.15.16:0/2472754541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca4003df0 con 0x5593f0b523a0 2024-09-12T03:07:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.832+0000 7f7c937fe640 1 -- 172.21.15.16:0/2472754541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4005020 con 0x5593f0b523a0 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.832+0000 7f7cb2032740 1 -- 172.21.15.16:0/2472754541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 msgr2=0x5593f0b75010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.832+0000 7f7cb2032740 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400ffd0 tx=0x7f7ca4009f50 comp rx=0 tx=0).stop 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 -- 172.21.15.16:0/2472754541 shutdown_connections 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b5b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b52cc0 0x5593f0b75550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 --2- 172.21.15.16:0/2472754541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b523a0 0x5593f0b75010 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 -- 172.21.15.16:0/2472754541 >> 172.21.15.16:0/2472754541 conn(0x5593f0b4ea30 msgr2=0x5593f0b546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 -- 172.21.15.16:0/2472754541 shutdown_connections 2024-09-12T03:07:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 -- 172.21.15.16:0/2472754541 wait complete. 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 -- start start 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 Processor -- start 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.833+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 0x5593f0b734c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b73a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81ac8b0 con 0x5593f0b52cc0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81ac1e0 con 0x5593f0b523a0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca81a9420 con 0x5593f0b61a90 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cb2032740 10 client.0 _ll_register_callbacks cb 0x7f7ca4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad1f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 0x5593f0b734c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b73a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38902/0 (socket says 172.21.15.16:38902) 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 -- 172.21.15.16:0/500017846 learned_addr learned my addr 172.21.15.16:0/500017846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cae1f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b73a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:56918/0 (socket says 172.21.15.16:56918) 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 -- 172.21.15.16:0/500017846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 msgr2=0x5593f0b73a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b73a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 -- 172.21.15.16:0/500017846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 msgr2=0x5593f0b734c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 0x5593f0b734c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad9f5640 1 -- 172.21.15.16:0/500017846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f0bd7240 con 0x5593f0b523a0 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.834+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 0x5593f0b734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7cad9f5640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4019040 tx=0x7f7ca4004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaae95362cd47fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7c917fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4038040 con 0x5593f0b523a0 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca4003df0 con 0x5593f0b523a0 2024-09-12T03:07:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7ca40121b0 con 0x5593f0b523a0 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7ca400f910 con 0x5593f0b523a0 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7cb2032740 10 client.29201 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5593f0bd84f0 con 0x5593f0b523a0 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.835+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca4003df0 con 0x5593f0b523a0 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7ca4004ac0 con 0x5593f0b523a0 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 10 client.29201 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 10 client.29201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 20 client.29201 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0bd8af0 0x5593f0bdaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 4 client.29201 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593f0a3c210 con 0x5593f0bd8af0 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.836+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0bd8af0 0x5593f0bdaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7cad1f4640 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0bd8af0 0x5593f0bdaee0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3cba1ec2cbb57d5 server_cookie=834dea80d387dcdf in_seq=0 out_seq=0 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7c917fa640 10 client.29201 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7c917fa640 1 -- 172.21.15.16:0/500017846 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5593f0a3c210 con 0x5593f0bd8af0 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7c917fa640 10 client.29201 handle_command_reply: tid=0 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7cb2032740 1 client.29201 shutdown 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 msgr2=0x5593f0b72f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.837+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4019040 tx=0x7f7ca4004590 comp rx=0 tx=0).stop 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 shutdown_connections 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5593f0bd8af0 0x5593f0bdaee0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5593f0b61a90 0x5593f0b73a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593f0b52cc0 0x5593f0b734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7c917fa640 0 client.29201 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 --2- 172.21.15.16:0/500017846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5593f0b523a0 0x5593f0b72f80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 >> 172.21.15.16:0/500017846 conn(0x5593f0b4ea30 msgr2=0x5593f0b53ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 shutdown_connections 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 1 -- 172.21.15.16:0/500017846 wait complete. 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cb2032740 20 client.29201 trim_cache size 0 max 16384 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:07:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:07:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:07:26.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 msgr2=0x7f7c940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 0x7f7c940a5040 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x5593f0b55090 tx=0x7f7ca4005f90 comp rx=0 tx=0).stop 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 msgr2=0x7f7ca8198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.838+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7c98006010 tx=0x7f7c98003c10 comp rx=0 tx=0).stop 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 shutdown_connections 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c940a2b80 0x7f7c940a5040 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ca8102ef0 0x7f7ca819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ca8102570 0x7f7ca8198ae0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 --2- 172.21.15.16:0/1227545452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ca8100e20 0x7f7ca81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 >> 172.21.15.16:0/1227545452 conn(0x7f7ca80fc7d0 msgr2=0x7f7ca806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 shutdown_connections 2024-09-12T03:07:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:26.839+0000 7f7cafc59640 1 -- 172.21.15.16:0/1227545452 wait complete. 2024-09-12T03:07:27.024 INFO:tasks.workunit.client.0.smithi016.stdout: 10 154489 22.90 MB/sec execute 268 sec latency 49.987 ms 2024-09-12T03:07:27.700 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:07:27.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:07:27.767 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:27 smithi016 ceph-mon[30925]: pgmap v923: 97 pgs: 97 active+clean; 698 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 885 KiB/s wr, 133 op/s 2024-09-12T03:07:27.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:27 smithi033 ceph-mon[35092]: pgmap v923: 97 pgs: 97 active+clean; 698 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 885 KiB/s wr, 133 op/s 2024-09-12T03:07:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:28.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155226 22.99 MB/sec execute 269 sec latency 67.815 ms 2024-09-12T03:07:28.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:27 smithi184 ceph-mon[35321]: pgmap v923: 97 pgs: 97 active+clean; 698 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 885 KiB/s wr, 133 op/s 2024-09-12T03:07:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 -- 172.21.15.16:0/4154232066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60102790 msgr2=0x7f0d601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/4154232066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60102790 0x7f0d601067e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009900 tx=0x7f0d4c02a410 comp rx=0 tx=0).stop 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 -- 172.21.15.16:0/4154232066 shutdown_connections 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/4154232066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60106f10 0x7f0d6010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/4154232066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60102790 0x7f0d601067e0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/4154232066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d601021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 -- 172.21.15.16:0/4154232066 >> 172.21.15.16:0/4154232066 conn(0x7f0d600fd8e0 msgr2=0x7f0d600ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 -- 172.21.15.16:0/4154232066 shutdown_connections 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 -- 172.21.15.16:0/4154232066 wait complete. 2024-09-12T03:07:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.568+0000 7f0d66cdf640 1 Processor -- start 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 -- start start 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60106f10 0x7f0d60074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60111fc0 con 0x7f0d60106f10 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60111e40 con 0x7f0d60101de0 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d66cdf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60112140 con 0x7f0d60102790 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d5ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d64a7b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d64a7b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38916/0 (socket says 172.21.15.16:38916) 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d64a7b640 1 -- 172.21.15.16:0/1364950649 learned_addr learned my addr 172.21.15.16:0/1364950649 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d5ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:56936/0 (socket says 172.21.15.16:56936) 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.569+0000 7f0d6527c640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60106f10 0x7f0d60074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 -- 172.21.15.16:0/1364950649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 msgr2=0x7f0d60074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 -- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60106f10 msgr2=0x7f0d60074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60106f10 0x7f0d60074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 -- 172.21.15.16:0/1364950649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d601aaef0 con 0x7f0d60101de0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d5ffff640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d64a7b640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400e9f0 tx=0x7f0d5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5400cd80 con 0x7f0d60101de0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d601ab120 con 0x7f0d60101de0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54010910 con 0x7f0d60101de0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d540188f0 con 0x7f0d60101de0 2024-09-12T03:07:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.570+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d601ab680 con 0x7f0d60101de0 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.572+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d601021c0 con 0x7f0d60101de0 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.574+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0d54010ac0 con 0x7f0d60101de0 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.574+0000 7f0d5dffb640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 0x7f0d500a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.575+0000 7f0d5ffff640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 0x7f0d500a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.575+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0d540cf720 con 0x7f0d60101de0 2024-09-12T03:07:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.575+0000 7f0d5ffff640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 0x7f0d500a50c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000c00 tx=0x7f0d4c031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.578+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0d5409c040 con 0x7f0d60101de0 2024-09-12T03:07:28.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.816+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d6010a0f0 con 0x7f0d60101de0 2024-09-12T03:07:28.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.817+0000 7f0d5dffb640 1 -- 172.21.15.16:0/1364950649 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f0d540a0340 con 0x7f0d60101de0 2024-09-12T03:07:28.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:07:28.817 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:07:28.818 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:07:28.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 msgr2=0x7f0d500a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:28.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 0x7f0d500a50c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000c00 tx=0x7f0d4c031510 comp rx=0 tx=0).stop 2024-09-12T03:07:28.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 msgr2=0x7f0d600744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400e9f0 tx=0x7f0d5400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 shutdown_connections 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d500a2c00 0x7f0d500a50c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d60106f10 0x7f0d60074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d60102790 0x7f0d60074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 --2- 172.21.15.16:0/1364950649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d60101de0 0x7f0d600744f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 >> 172.21.15.16:0/1364950649 conn(0x7f0d600fd8e0 msgr2=0x7f0d600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.819+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 shutdown_connections 2024-09-12T03:07:28.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:28.820+0000 7f0d66cdf640 1 -- 172.21.15.16:0/1364950649 wait complete. 2024-09-12T03:07:29.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 155764 23.03 MB/sec execute 270 sec latency 91.315 ms 2024-09-12T03:07:29.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:07:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:29 smithi016 ceph-mon[30925]: pgmap v924: 97 pgs: 97 active+clean; 700 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-09-12T03:07:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:29 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1364950649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:29.983 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:29.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:29 smithi033 ceph-mon[35092]: pgmap v924: 97 pgs: 97 active+clean; 700 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-09-12T03:07:29.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:29 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1364950649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:30.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156218 23.01 MB/sec execute 271 sec latency 154.755 ms 2024-09-12T03:07:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:29 smithi184 ceph-mon[35321]: pgmap v924: 97 pgs: 97 active+clean; 700 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-09-12T03:07:30.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:29 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1364950649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 -- 172.21.15.16:0/3035169179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102570 msgr2=0x7f51dc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/3035169179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102570 0x7f51dc1029b0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009900 tx=0x7f51c402a480 comp rx=0 tx=0).stop 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 -- 172.21.15.16:0/3035169179 shutdown_connections 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/3035169179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102ef0 0x7f51dc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/3035169179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102570 0x7f51dc1029b0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/3035169179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc100e20 0x7f51dc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 -- 172.21.15.16:0/3035169179 >> 172.21.15.16:0/3035169179 conn(0x7f51dc0fc7d0 msgr2=0x7f51dc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 -- 172.21.15.16:0/3035169179 shutdown_connections 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.982+0000 7f51e1e3e640 1 -- 172.21.15.16:0/3035169179 wait complete. 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 Processor -- start 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 -- start start 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102570 0x7f51dc198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc102ef0 0x7f51dc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc110f50 con 0x7f51dc100e20 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc110dd0 con 0x7f51dc102570 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51e1e3e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc1110d0 con 0x7f51dc102ef0 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42802/0 (socket says 172.21.15.16:42802) 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51db7fe640 1 -- 172.21.15.16:0/173193799 learned_addr learned my addr 172.21.15.16:0/173193799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51d3fff640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102570 0x7f51dc198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.983+0000 7f51dbfff640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc102ef0 0x7f51dc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 -- 172.21.15.16:0/173193799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc102ef0 msgr2=0x7f51dc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc102ef0 0x7f51dc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 -- 172.21.15.16:0/173193799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102570 msgr2=0x7f51dc198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102570 0x7f51dc198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 -- 172.21.15.16:0/173193799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc19d550 con 0x7f51dc100e20 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51db7fe640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00e9f0 tx=0x7f51cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51d97fa640 1 -- 172.21.15.16:0/173193799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00cd80 con 0x7f51dc100e20 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51dc19d840 con 0x7f51dc100e20 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51dc1a5120 con 0x7f51dc100e20 2024-09-12T03:07:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51d97fa640 1 -- 172.21.15.16:0/173193799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51cc010910 con 0x7f51dc100e20 2024-09-12T03:07:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.984+0000 7f51d97fa640 1 -- 172.21.15.16:0/173193799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc018990 con 0x7f51dc100e20 2024-09-12T03:07:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.987+0000 7f51d97fa640 1 -- 172.21.15.16:0/173193799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f51cc018b30 con 0x7f51dc100e20 2024-09-12T03:07:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51d97fa640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 0x7f51c80a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 Processor -- start 2024-09-12T03:07:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51d3fff640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 0x7f51c80a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 -- start start 2024-09-12T03:07:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51d97fa640 1 -- 172.21.15.16:0/173193799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51cc0cf290 con 0x7f51dc100e20 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a324a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 0x55e52a331760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e52a28a2f0 con 0x55e52a325b40 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e52a231cf0 con 0x55e52a3245d0 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51e4217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e52a2862b0 con 0x55e52a32f370 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51db7fe640 1 -- 172.21.15.16:0/4155305343 learned_addr learned my addr 172.21.15.16:0/4155305343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51d3fff640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a324a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.989+0000 7f51dbfff640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 0x55e52a331760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51d3fff640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 0x7f51c80a5140 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009230 tx=0x7f51c40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 -- 172.21.15.16:0/4155305343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 msgr2=0x55e52a331760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 0x55e52a331760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 -- 172.21.15.16:0/4155305343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 msgr2=0x55e52a324a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a324a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 -- 172.21.15.16:0/4155305343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e52a2163a0 con 0x55e52a325b40 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51dbfff640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 0x55e52a331760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f51cc097fb0 tx=0x7f51cc09b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd814fd47363948 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51d1ffb640 1 -- 172.21.15.16:0/4155305343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0b4070 con 0x55e52a325b40 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51d1ffb640 1 -- 172.21.15.16:0/4155305343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51cc0cce50 con 0x55e52a325b40 2024-09-12T03:07:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.990+0000 7f51d1ffb640 1 -- 172.21.15.16:0/4155305343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc097b00 con 0x55e52a325b40 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 -- 172.21.15.16:0/4155305343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 msgr2=0x55e52a324090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f51cc097fb0 tx=0x7f51cc09b5b0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 -- 172.21.15.16:0/4155305343 shutdown_connections 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a32f370 0x55e52a331760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a324a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 --2- 172.21.15.16:0/4155305343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a325b40 0x55e52a324090 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 -- 172.21.15.16:0/4155305343 >> 172.21.15.16:0/4155305343 conn(0x55e52a2ed310 msgr2=0x55e52a320c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.991+0000 7f51e4217740 1 -- 172.21.15.16:0/4155305343 shutdown_connections 2024-09-12T03:07:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.992+0000 7f51e4217740 1 -- 172.21.15.16:0/4155305343 wait complete. 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.992+0000 7f51e4217740 1 -- start start 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.992+0000 7f51e4217740 1 Processor -- start 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a3245d0 0x55e52a32ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e52a336480 con 0x55e52a3245d0 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e52a245060 con 0x55e52a32f370 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e52a336600 con 0x55e52a325b40 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51e4217740 10 client.0 _ll_register_callbacks cb 0x55e529f988c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51dbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51dbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38930/0 (socket says 172.21.15.16:38930) 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51dbfff640 1 -- 172.21.15.16:0/4148934884 learned_addr learned my addr 172.21.15.16:0/4148934884 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.993+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a3245d0 0x55e52a32ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 -- 172.21.15.16:0/4148934884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 msgr2=0x55e52a32c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 -- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a3245d0 msgr2=0x55e52a32ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a3245d0 0x55e52a32ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 -- 172.21.15.16:0/4148934884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e529ffb3c0 con 0x55e52a32f370 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51d3fff640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51dbfff640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f51d4011d10 tx=0x7f51d400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7c078ebdbc327c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51baffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d401f060 con 0x55e52a32f370 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51d4003970 con 0x55e52a32f370 2024-09-12T03:07:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.994+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51cc0cce50 con 0x55e52a32f370 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f51d400fe60 con 0x55e52a32f370 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 10 client.29219 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e52a341ff0 con 0x55e52a32f370 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51cc0cce50 con 0x55e52a32f370 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f51d4019030 con 0x55e52a32f370 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 10 client.29219 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 10 client.29219 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 20 client.29219 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3434b0 0x55e52a3458e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 4 client.29219 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.995+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e52a297860 con 0x55e52a3434b0 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.996+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3434b0 0x55e52a3458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.996+0000 7f51db7fe640 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3434b0 0x55e52a3458e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1fb916b662205672 server_cookie=6280bea45ac6b4b0 in_seq=0 out_seq=0 2024-09-12T03:07:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.996+0000 7f51baffd640 10 client.29219 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:07:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.997+0000 7f51baffd640 1 -- 172.21.15.16:0/4148934884 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e52a297860 con 0x55e52a3434b0 2024-09-12T03:07:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.997+0000 7f51baffd640 10 client.29219 handle_command_reply: tid=0 2024-09-12T03:07:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.997+0000 7f51e4217740 1 client.29219 shutdown 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 msgr2=0x55e52a32c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f51d4011d10 tx=0x7f51d400f790 comp rx=0 tx=0).stop 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 shutdown_connections 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3434b0 0x55e52a3458e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a32f370 0x55e52a32c640 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 --2- 172.21.15.16:0/4148934884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a3245d0 0x55e52a32ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51baffd640 0 client.29219 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 >> 172.21.15.16:0/4148934884 conn(0x55e52a2ed310 msgr2=0x55e52a326930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.998+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 shutdown_connections 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.999+0000 7f51e4217740 1 -- 172.21.15.16:0/4148934884 wait complete. 2024-09-12T03:07:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:30.999+0000 7f51e4217740 20 client.29219 trim_cache size 0 max 16384 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 Processor -- start 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 -- start start 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a31bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a33f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc00e4c0 con 0x55e52a32f370 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc010000 con 0x55e52a3245d0 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51e4217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc00eb90 con 0x55e52a325b40 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:56984/0 (socket says 172.21.15.16:56984) 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a33f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51d3fff640 1 -- 172.21.15.16:0/3749914971 learned_addr learned my addr 172.21.15.16:0/3749914971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.015+0000 7f51db7fe640 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a31bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 -- 172.21.15.16:0/3749914971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 msgr2=0x55e52a31bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a31bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 -- 172.21.15.16:0/3749914971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 msgr2=0x55e52a33f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a33f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 -- 172.21.15.16:0/3749914971 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e52a32f0f0 con 0x55e52a325b40 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51dbfff640 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a33f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51c402a9a0 tx=0x7f51c4043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b798b4ee36d78bc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d17fa640 1 -- 172.21.15.16:0/3749914971 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4057070 con 0x55e52a325b40 2024-09-12T03:07:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d17fa640 1 -- 172.21.15.16:0/3749914971 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c4003e00 con 0x55e52a325b40 2024-09-12T03:07:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.016+0000 7f51d17fa640 1 -- 172.21.15.16:0/3749914971 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4035050 con 0x55e52a325b40 2024-09-12T03:07:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 -- 172.21.15.16:0/3749914971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 msgr2=0x55e52a32c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51c402a9a0 tx=0x7f51c4043000 comp rx=0 tx=0).stop 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 -- 172.21.15.16:0/3749914971 shutdown_connections 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a33f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a32c000 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 --2- 172.21.15.16:0/3749914971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a31bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 -- 172.21.15.16:0/3749914971 >> 172.21.15.16:0/3749914971 conn(0x55e52a2ed310 msgr2=0x55e52a32b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 -- 172.21.15.16:0/3749914971 shutdown_connections 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.017+0000 7f51e4217740 1 -- 172.21.15.16:0/3749914971 wait complete. 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 -- start start 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 Processor -- start 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a337920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a337e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a3383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc00f930 con 0x55e52a32f370 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc00f260 con 0x55e52a3245d0 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.018+0000 7f51e4217740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc00c4a0 con 0x55e52a325b40 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51e4217740 10 client.0 _ll_register_callbacks cb 0x7f51d4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a3383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51db7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a337920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a3383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42850/0 (socket says 172.21.15.16:42850) 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51dbfff640 1 -- 172.21.15.16:0/3387003539 learned_addr learned my addr 172.21.15.16:0/3387003539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a337e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 -- 172.21.15.16:0/3387003539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 msgr2=0x55e52a337920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a337920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 -- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 msgr2=0x55e52a3383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a3383c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 -- 172.21.15.16:0/3387003539 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e52a3ad490 con 0x55e52a325b40 2024-09-12T03:07:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51d3fff640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a337e60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55e52a324ee0 tx=0x7f51c403c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08a71de004cda29 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:31.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.019+0000 7f51baffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4057030 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51c4003c00 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c4003e00 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f51c4004260 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51e4217740 10 client.29146 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e52a3ae790 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.020+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c4003e00 con 0x55e52a325b40 2024-09-12T03:07:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f51c4051070 con 0x55e52a325b40 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 10 client.29146 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 10 client.29146 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 20 client.29146 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3aee60 0x55e52a3b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 4 client.29146 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e52a291d70 con 0x55e52a3aee60 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51db7fe640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3aee60 0x55e52a3b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51db7fe640 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3aee60 0x55e52a3b1280 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=28e5a8ba57baa103 server_cookie=ba02fdd14cc25f0d in_seq=0 out_seq=0 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.021+0000 7f51baffd640 10 client.29146 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51baffd640 1 -- 172.21.15.16:0/3387003539 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e52a291d70 con 0x55e52a3aee60 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51baffd640 10 client.29146 handle_command_reply: tid=0 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 client.29146 shutdown 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 msgr2=0x55e52a337e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a337e60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55e52a324ee0 tx=0x7f51c403c440 comp rx=0 tx=0).stop 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 shutdown_connections 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e52a32f370 0x55e52a3383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e52a325b40 0x55e52a337e60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55e52a3aee60 0x55e52a3b1280 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 --2- 172.21.15.16:0/3387003539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e52a3245d0 0x55e52a337920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 >> 172.21.15.16:0/3387003539 conn(0x55e52a2ed310 msgr2=0x55e52a32e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51baffd640 0 client.29146 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 shutdown_connections 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 1 -- 172.21.15.16:0/3387003539 wait complete. 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.022+0000 7f51e4217740 20 client.29146 trim_cache size 0 max 16384 2024-09-12T03:07:31.026 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:07:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 msgr2=0x7f51c80a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 0x7f51c80a5140 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009230 tx=0x7f51c40058d0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 msgr2=0x7f51dc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00e9f0 tx=0x7f51cc00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 shutdown_connections 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51dc102ef0 0x7f51dc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51dc102570 0x7f51dc198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51c80a2c80 0x7f51c80a5140 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 --2- 172.21.15.16:0/173193799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100e20 0x7f51dc1985a0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.023+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 >> 172.21.15.16:0/173193799 conn(0x7f51dc0fc7d0 msgr2=0x7f51dc06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.024+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 shutdown_connections 2024-09-12T03:07:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:31.024+0000 7f51e1e3e640 1 -- 172.21.15.16:0/173193799 wait complete. 2024-09-12T03:07:31.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 156767 23.01 MB/sec execute 272 sec latency 121.569 ms 2024-09-12T03:07:31.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:31 smithi016 ceph-mon[30925]: pgmap v925: 97 pgs: 97 active+clean; 710 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-09-12T03:07:31.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:07:31.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:31 smithi033 ceph-mon[35092]: pgmap v925: 97 pgs: 97 active+clean; 710 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-09-12T03:07:32.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 157437 23.07 MB/sec execute 273 sec latency 48.149 ms 2024-09-12T03:07:32.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:31 smithi184 ceph-mon[35321]: pgmap v925: 97 pgs: 97 active+clean; 710 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-09-12T03:07:32.128 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:32.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 -- 172.21.15.16:0/482013642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124106f20 msgr2=0x7fb12410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 --2- 172.21.15.16:0/482013642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124106f20 0x7fb12410eaa0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ca0 tx=0x7fb11c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 -- 172.21.15.16:0/482013642 shutdown_connections 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 --2- 172.21.15.16:0/482013642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124106f20 0x7fb12410eaa0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 --2- 172.21.15.16:0/482013642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 0x7fb1241067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 --2- 172.21.15.16:0/482013642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124101df0 0x7fb1241021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.681+0000 7fb12a65d640 1 -- 172.21.15.16:0/482013642 >> 172.21.15.16:0/482013642 conn(0x7fb1240fd8b0 msgr2=0x7fb1240ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.682+0000 7fb12a65d640 1 -- 172.21.15.16:0/482013642 shutdown_connections 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.682+0000 7fb12a65d640 1 -- 172.21.15.16:0/482013642 wait complete. 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.682+0000 7fb12a65d640 1 Processor -- start 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.682+0000 7fb12a65d640 1 -- start start 2024-09-12T03:07:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 0x7fb124074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124106f20 0x7fb124074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb124111f80 con 0x7fb124101df0 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb124111e00 con 0x7fb1241027a0 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb12a65d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb124112100 con 0x7fb124106f20 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb1237fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 0x7fb124074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42878/0 (socket says 172.21.15.16:42878) 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 -- 172.21.15.16:0/3146844156 learned_addr learned my addr 172.21.15.16:0/3146844156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb128bfa640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124106f20 0x7fb124074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 -- 172.21.15.16:0/3146844156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124106f20 msgr2=0x7fb124074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124106f20 0x7fb124074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 -- 172.21.15.16:0/3146844156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 msgr2=0x7fb124074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 0x7fb124074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.683+0000 7fb123fff640 1 -- 172.21.15.16:0/3146844156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1241a6880 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb123fff640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb11400e9f0 tx=0x7fb11400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb1217fa640 1 -- 172.21.15.16:0/3146844156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11400cd80 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1241a6b70 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb1217fa640 1 -- 172.21.15.16:0/3146844156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb114005500 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb1217fa640 1 -- 172.21.15.16:0/3146844156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114010690 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.684+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1241a70b0 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.686+0000 7fb1217fa640 1 -- 172.21.15.16:0/3146844156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb1140046c0 con 0x7fb124101df0 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.686+0000 7fb12ca36740 1 Processor -- start 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 -- start start 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301621ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 0x563016229760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5630161822f0 con 0x563016227370 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563016129cf0 con 0x56301621c5d0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb12ca36740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56301617e2b0 con 0x56301621db40 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1217fa640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 0x7fb10c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 0x563016229760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb128bfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb123fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301621ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb128bfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57006/0 (socket says 172.21.15.16:57006) 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb128bfa640 1 -- 172.21.15.16:0/3058217777 learned_addr learned my addr 172.21.15.16:0/3058217777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb128bfa640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 0x7fb10c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1217fa640 1 -- 172.21.15.16:0/3146844156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1140cf450 con 0x7fb124101df0 2024-09-12T03:07:32.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 -- 172.21.15.16:0/3058217777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 msgr2=0x56301621c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 -- 172.21.15.16:0/3058217777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 msgr2=0x56301621ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301621ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.687+0000 7fb1237fe640 1 -- 172.21.15.16:0/3058217777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301610e3a0 con 0x563016227370 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.688+0000 7fb1237fe640 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 0x563016229760 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb118009bf0 tx=0x7fb118011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15992c499ff0c18d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.688+0000 7fb128bfa640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 0x7fb10c0a5040 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb11c005fe0 tx=0x7fb11c004a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.688+0000 7fb10a7fc640 1 -- 172.21.15.16:0/3058217777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118004050 con 0x563016227370 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.688+0000 7fb10a7fc640 1 -- 172.21.15.16:0/3058217777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb118005500 con 0x563016227370 2024-09-12T03:07:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.688+0000 7fb10a7fc640 1 -- 172.21.15.16:0/3058217777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118004d90 con 0x563016227370 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 -- 172.21.15.16:0/3058217777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 msgr2=0x563016229760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 0x563016229760 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb118009bf0 tx=0x7fb118011810 comp rx=0 tx=0).stop 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 -- 172.21.15.16:0/3058217777 shutdown_connections 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563016227370 0x563016229760 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301621ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3058217777 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 -- 172.21.15.16:0/3058217777 >> 172.21.15.16:0/3058217777 conn(0x5630161e5310 msgr2=0x563016218c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 -- 172.21.15.16:0/3058217777 shutdown_connections 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.689+0000 7fb12ca36740 1 -- 172.21.15.16:0/3058217777 wait complete. 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 -- start start 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 Processor -- start 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x563016226ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x563016223f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56301622a070 con 0x56301621db40 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56301613d060 con 0x56301621c5d0 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56301622a1f0 con 0x563016227370 2024-09-12T03:07:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb1237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x563016223f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb12ca36740 10 client.0 _ll_register_callbacks cb 0x563015e908c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb128bfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb128bfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57014/0 (socket says 172.21.15.16:57014) 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb128bfa640 1 -- 172.21.15.16:0/3099733786 learned_addr learned my addr 172.21.15.16:0/3099733786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.690+0000 7fb123fff640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x563016226ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 -- 172.21.15.16:0/3099733786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 msgr2=0x563016224490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 -- 172.21.15.16:0/3099733786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 msgr2=0x563016223f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x563016223f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 -- 172.21.15.16:0/3099733786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563015ef33c0 con 0x56301621c5d0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb128bfa640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb123fff640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x563016226ca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb1140ccf60 tx=0x7fb11406ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec46311cb76f8c3b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb1037fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1140cf2b0 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb118005500 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.691+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1140a44a0 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb11409c3c0 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 10 client.29237 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563016235ae0 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb118005500 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb11409b1a0 con 0x56301621c5d0 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 10 client.29237 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 10 client.29237 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 20 client.29237 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563016237210 0x563016239600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 4 client.29237 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.692+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56301618f860 con 0x563016237210 2024-09-12T03:07:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.694+0000 7fb1237fe640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563016237210 0x563016239600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.694+0000 7fb1237fe640 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563016237210 0x563016239600 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b5d847933d27f2db server_cookie=ff95906dea27e219 in_seq=0 out_seq=0 2024-09-12T03:07:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.694+0000 7fb1037fe640 10 client.29237 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb1037fe640 1 -- 172.21.15.16:0/3099733786 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56301618f860 con 0x563016237210 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb1037fe640 10 client.29237 handle_command_reply: tid=0 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 client.29237 shutdown 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 msgr2=0x563016226ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x563016226ca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb1140ccf60 tx=0x7fb11406ad00 comp rx=0 tx=0).stop 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 shutdown_connections 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563016237210 0x563016239600 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016224490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x563016223f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3099733786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x563016226ca0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb1037fe640 0 client.29237 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:07:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 >> 172.21.15.16:0/3099733786 conn(0x5630161e5310 msgr2=0x56301621e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 shutdown_connections 2024-09-12T03:07:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 1 -- 172.21.15.16:0/3099733786 wait complete. 2024-09-12T03:07:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.700+0000 7fb12ca36740 20 client.29237 trim_cache size 0 max 16384 2024-09-12T03:07:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.716+0000 7fb12ca36740 1 Processor -- start 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 -- start start 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563016227370 0x56301623ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241af660 con 0x56301621c5d0 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b11a0 con 0x563016227370 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb12ca36740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241afd30 con 0x56301621db40 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb1237fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42900/0 (socket says 172.21.15.16:42900) 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 -- 172.21.15.16:0/3292751620 learned_addr learned my addr 172.21.15.16:0/3292751620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb128bfa640 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563016227370 0x56301623ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 -- 172.21.15.16:0/3292751620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 msgr2=0x56301621be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 -- 172.21.15.16:0/3292751620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563016227370 msgr2=0x56301623ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563016227370 0x56301623ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.717+0000 7fb123fff640 1 -- 172.21.15.16:0/3292751620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630162224a0 con 0x56301621c5d0 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.718+0000 7fb123fff640 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb11400c6a0 tx=0x7fb11409a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f35586f71ae590c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.718+0000 7fb109ffb640 1 -- 172.21.15.16:0/3292751620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11409b800 con 0x56301621c5d0 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.718+0000 7fb109ffb640 1 -- 172.21.15.16:0/3292751620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11409bcc0 con 0x56301621c5d0 2024-09-12T03:07:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.718+0000 7fb109ffb640 1 -- 172.21.15.16:0/3292751620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1140a44a0 con 0x56301621c5d0 2024-09-12T03:07:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- 172.21.15.16:0/3292751620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 msgr2=0x563016233340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb11400c6a0 tx=0x7fb11409a1d0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- 172.21.15.16:0/3292751620 shutdown_connections 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563016227370 0x56301623ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56301621db40 0x56301621be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 --2- 172.21.15.16:0/3292751620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621c5d0 0x563016233340 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- 172.21.15.16:0/3292751620 >> 172.21.15.16:0/3292751620 conn(0x5630161e5310 msgr2=0x563016222820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- 172.21.15.16:0/3292751620 shutdown_connections 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- 172.21.15.16:0/3292751620 wait complete. 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 -- start start 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.719+0000 7fb12ca36740 1 Processor -- start 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301622f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x56301622fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016220900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b0ad0 con 0x56301621db40 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b0400 con 0x56301621c5d0 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ad640 con 0x563016227370 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb12ca36740 10 client.0 _ll_register_callbacks cb 0x7fb11400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb1237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x56301622fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb123fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301622f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb123fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301622f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38974/0 (socket says 172.21.15.16:38974) 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb123fff640 1 -- 172.21.15.16:0/916243379 learned_addr learned my addr 172.21.15.16:0/916243379 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb128bfa640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016220900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb1237fe640 1 -- 172.21.15.16:0/916243379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 msgr2=0x563016220900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb1237fe640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016220900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb1237fe640 1 -- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 msgr2=0x56301622f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.720+0000 7fb1237fe640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301622f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1237fe640 1 -- 172.21.15.16:0/916243379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563016221130 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1237fe640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x56301622fc80 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb118004650 tx=0x7fb118005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e214d8f6851094cb server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1037fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118034060 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb118004230 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb11409bcc0 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb118011d10 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb12ca36740 10 client.20004 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5630162a17e0 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.721+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11409bcc0 con 0x56301621db40 2024-09-12T03:07:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb11802e520 con 0x56301621db40 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 10 client.20004 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 10 client.20004 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 20 client.20004 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5630162a1d10 0x5630162a4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 4 client.20004 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563016189d70 con 0x5630162a1d10 2024-09-12T03:07:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb123fff640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5630162a1d10 0x5630162a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb123fff640 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5630162a1d10 0x5630162a4100 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3203ccfc66ff2d17 server_cookie=768479330c51e38e in_seq=0 out_seq=0 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.722+0000 7fb1037fe640 10 client.20004 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb1037fe640 1 -- 172.21.15.16:0/916243379 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563016189d70 con 0x5630162a1d10 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb1037fe640 10 client.20004 handle_command_reply: tid=0 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 client.20004 shutdown 2024-09-12T03:07:32.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 msgr2=0x56301622fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x56301622fc80 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb118004650 tx=0x7fb118005d90 comp rx=0 tx=0).stop 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 shutdown_connections 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5630162a1d10 0x5630162a4100 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb1037fe640 0 client.20004 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563016227370 0x563016220900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56301621db40 0x56301622fc80 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 --2- 172.21.15.16:0/916243379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56301621c5d0 0x56301622f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 >> 172.21.15.16:0/916243379 conn(0x5630161e5310 msgr2=0x5630162174e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 shutdown_connections 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 1 -- 172.21.15.16:0/916243379 wait complete. 2024-09-12T03:07:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.723+0000 7fb12ca36740 20 client.20004 trim_cache size 0 max 16384 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 msgr2=0x7fb10c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 0x7fb10c0a5040 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb11c005fe0 tx=0x7fb11c004a80 comp rx=0 tx=0).stop 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 msgr2=0x7fb1240744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb11400e9f0 tx=0x7fb11400edf0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 shutdown_connections 2024-09-12T03:07:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb124106f20 0x7fb124074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1241027a0 0x7fb124074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb10c0a2b80 0x7fb10c0a5040 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 --2- 172.21.15.16:0/3146844156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb124101df0 0x7fb1240744f0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 >> 172.21.15.16:0/3146844156 conn(0x7fb1240fd8b0 msgr2=0x7fb1240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 shutdown_connections 2024-09-12T03:07:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:32.724+0000 7fb12a65d640 1 -- 172.21.15.16:0/3146844156 wait complete. 2024-09-12T03:07:33.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 158127 23.09 MB/sec execute 274 sec latency 115.669 ms 2024-09-12T03:07:33.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:07:33.732 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:33.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:33 smithi033 ceph-mon[35092]: pgmap v926: 97 pgs: 97 active+clean; 711 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 370 op/s 2024-09-12T03:07:34.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:33 smithi016 ceph-mon[30925]: pgmap v926: 97 pgs: 97 active+clean; 711 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 370 op/s 2024-09-12T03:07:34.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 158965 23.17 MB/sec execute 275 sec latency 77.261 ms 2024-09-12T03:07:34.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:33 smithi184 ceph-mon[35321]: pgmap v926: 97 pgs: 97 active+clean; 711 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 370 op/s 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.255+0000 7fa1051d5640 1 -- 172.21.15.16:0/2328976934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100104980 msgr2=0x7fa1001089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.255+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2328976934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100104980 0x7fa1001089d0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009900 tx=0x7fa0e802a3f0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.255+0000 7fa1051d5640 1 -- 172.21.15.16:0/2328976934 shutdown_connections 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.255+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2328976934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100109100 0x7fa100110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.256+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2328976934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100104980 0x7fa1001089d0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.256+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2328976934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa1001043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.256+0000 7fa1051d5640 1 -- 172.21.15.16:0/2328976934 >> 172.21.15.16:0/2328976934 conn(0x7fa1000ffa30 msgr2=0x7fa100101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.256+0000 7fa1051d5640 1 -- 172.21.15.16:0/2328976934 shutdown_connections 2024-09-12T03:07:34.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.256+0000 7fa1051d5640 1 -- 172.21.15.16:0/2328976934 wait complete. 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.257+0000 7fa1051d5640 1 Processor -- start 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.257+0000 7fa1051d5640 1 -- start start 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 0x7fa10019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 0x7fa10019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa100114100 con 0x7fa100109100 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa100113f80 con 0x7fa100103fd0 2024-09-12T03:07:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa1051d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa100114280 con 0x7fa100104980 2024-09-12T03:07:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38980/0 (socket says 172.21.15.16:38980) 2024-09-12T03:07:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2241580833 learned_addr learned my addr 172.21.15.16:0/2241580833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 0x7fa10019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.258+0000 7fa0f7fff640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 0x7fa10019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2241580833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 msgr2=0x7fa10019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 0x7fa10019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 msgr2=0x7fa10019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 0x7fa10019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2241580833 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10019f450 con 0x7fa100103fd0 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0f7fff640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 0x7fa10019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 0x7fa10019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000e9f0 tx=0x7fa0f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fcd99640 1 -- 172.21.15.16:0/2241580833 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000cd80 con 0x7fa100103fd0 2024-09-12T03:07:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fcd99640 1 -- 172.21.15.16:0/2241580833 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f0010910 con 0x7fa100103fd0 2024-09-12T03:07:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa0fcd99640 1 -- 172.21.15.16:0/2241580833 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0018990 con 0x7fa100103fd0 2024-09-12T03:07:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10019f740 con 0x7fa100103fd0 2024-09-12T03:07:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.259+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1001a7020 con 0x7fa100103fd0 2024-09-12T03:07:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.262+0000 7fa0fcd99640 1 -- 172.21.15.16:0/2241580833 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa0f0010ac0 con 0x7fa100103fd0 2024-09-12T03:07:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.262+0000 7fa1075ae740 1 Processor -- start 2024-09-12T03:07:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.262+0000 7fa1075ae740 1 -- start start 2024-09-12T03:07:34.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c5b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c67f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5616f7bb82f0 con 0x5616f7c65b80 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5616f7b5fcf0 con 0x5616f7c5ade0 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa1075ae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5616f7bb42b0 con 0x5616f7c5c350 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa0fcd99640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 0x7fa0ec0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.263+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57050/0 (socket says 172.21.15.16:57050) 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2186217712 learned_addr learned my addr 172.21.15.16:0/2186217712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c5b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 0x7fa0ec0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2186217712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 msgr2=0x5616f7c5b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c5b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2186217712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 msgr2=0x5616f7c67f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c67f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2186217712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616f7b443a0 con 0x5616f7c5c350 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0fcd99640 1 -- 172.21.15.16:0/2241580833 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa0f00cf450 con 0x7fa100103fd0 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.264+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8011cc0 tx=0x7fa0f800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718de0b6ed02569f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.265+0000 7fa0f57fa640 1 -- 172.21.15.16:0/2186217712 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8022020 con 0x5616f7c5c350 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.265+0000 7fa0f57fa640 1 -- 172.21.15.16:0/2186217712 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8011820 con 0x5616f7c5c350 2024-09-12T03:07:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.265+0000 7fa0f57fa640 1 -- 172.21.15.16:0/2186217712 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800e3c0 con 0x5616f7c5c350 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- 172.21.15.16:0/2186217712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 msgr2=0x5616f7c5a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 0x7fa0ec0a5040 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8023000 tx=0x7fa0f8021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8011cc0 tx=0x7fa0f800f790 comp rx=0 tx=0).stop 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- 172.21.15.16:0/2186217712 shutdown_connections 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c67f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c5b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2186217712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c5a8a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- 172.21.15.16:0/2186217712 >> 172.21.15.16:0/2186217712 conn(0x5616f7c1b310 msgr2=0x5616f7c57470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- 172.21.15.16:0/2186217712 shutdown_connections 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- 172.21.15.16:0/2186217712 wait complete. 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- start start 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 Processor -- start 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c62830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c62e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5616f7c68880 con 0x5616f7c65b80 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5616f7b73060 con 0x5616f7c5ade0 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.266+0000 7fa1075ae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5616f7c68a00 con 0x5616f7c5c350 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa1075ae740 10 client.0 _ll_register_callbacks cb 0x5616f78c68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c62e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c62830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c62e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42940/0 (socket says 172.21.15.16:42940) 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38992/0 (socket says 172.21.15.16:38992) 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2379830950 learned_addr learned my addr 172.21.15.16:0/2379830950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 msgr2=0x5616f7c62830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c62830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2379830950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 msgr2=0x5616f7c62e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c62e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 -- 172.21.15.16:0/2379830950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616f79293c0 con 0x5616f7c5ade0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa0f009b960 tx=0x7fa0f009b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd356b8447795532 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f001de90 con 0x5616f7c5ade0 2024-09-12T03:07:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.267+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa0f00cd010 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f8011820 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa0f009c360 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 10 client.29255 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5616f7c74380 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f00b1070 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa0f009b0b0 con 0x5616f7c5ade0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 10 client.29255 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 10 client.29255 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 20 client.29255 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7c756d0 0x5616f7c77ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 4 client.29255 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8011820 con 0x5616f7c756d0 2024-09-12T03:07:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.268+0000 7fa0f7fff640 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7c756d0 0x5616f7c77ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.269+0000 7fa0f7fff640 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7c756d0 0x5616f7c77ac0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=836ec7f6ae5b431d server_cookie=221dd632213c26bf in_seq=0 out_seq=0 2024-09-12T03:07:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.269+0000 7fa0e67fc640 10 client.29255 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.271+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2379830950 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fa0f8011820 con 0x5616f7c756d0 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.271+0000 7fa0e67fc640 10 client.29255 handle_command_reply: tid=0 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.271+0000 7fa1075ae740 1 client.29255 shutdown 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.271+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 msgr2=0x5616f7c65580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.271+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa0f009b960 tx=0x7fa0f009b7a0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 shutdown_connections 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c65b80 0x5616f7c62e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7c756d0 0x5616f7c77ac0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c62830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2379830950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c5ade0 0x5616f7c65580 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 >> 172.21.15.16:0/2379830950 conn(0x5616f7c1b310 msgr2=0x5616f7c5d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa0e67fc640 0 client.29255 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 shutdown_connections 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 1 -- 172.21.15.16:0/2379830950 wait complete. 2024-09-12T03:07:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.272+0000 7fa1075ae740 20 client.29255 trim_cache size 0 max 16384 2024-09-12T03:07:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.286+0000 7fa1075ae740 1 Processor -- start 2024-09-12T03:07:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 -- start start 2024-09-12T03:07:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 0x5616f7c71b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c61100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001ab4d0 con 0x5616f7c5ade0 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001ad010 con 0x5616f7c65b80 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa1075ae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001abba0 con 0x5616f7c5c350 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa0ff59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:38994/0 (socket says 172.21.15.16:38994) 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2503939091 learned_addr learned my addr 172.21.15.16:0/2503939091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c61100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.287+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 0x5616f7c71b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2503939091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 msgr2=0x5616f7c61100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c61100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2503939091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 msgr2=0x5616f7c71b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 0x5616f7c71b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2503939091 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616f7c5bbb0 con 0x5616f7c65b80 2024-09-12T03:07:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 0x5616f7c71b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009700 tx=0x7fa0f8028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=329b6f4eaae40f66 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0f4ff9640 1 -- 172.21.15.16:0/2503939091 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8022070 con 0x5616f7c65b80 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0f4ff9640 1 -- 172.21.15.16:0/2503939091 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8036070 con 0x5616f7c65b80 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.288+0000 7fa0f4ff9640 1 -- 172.21.15.16:0/2503939091 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f801d910 con 0x5616f7c65b80 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.289+0000 7fa1075ae740 1 -- 172.21.15.16:0/2503939091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 msgr2=0x5616f7c61640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.289+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009700 tx=0x7fa0f8028000 comp rx=0 tx=0).stop 2024-09-12T03:07:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 -- 172.21.15.16:0/2503939091 shutdown_connections 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c61640 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5c350 0x5616f7c61100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2503939091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5ade0 0x5616f7c71b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 -- 172.21.15.16:0/2503939091 >> 172.21.15.16:0/2503939091 conn(0x5616f7c1b310 msgr2=0x5616f7c776e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 -- 172.21.15.16:0/2503939091 shutdown_connections 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 -- 172.21.15.16:0/2503939091 wait complete. 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 -- start start 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 Processor -- start 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5ade0 0x5616f7c6df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.290+0000 7fa1075ae740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c5f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa1075ae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001ac940 con 0x5616f7c5c350 2024-09-12T03:07:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa1075ae740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001ac270 con 0x5616f7c65b80 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa1075ae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a94f0 con 0x5616f7c5ade0 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa1075ae740 10 client.0 _ll_register_callbacks cb 0x7fa0f000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42970/0 (socket says 172.21.15.16:42970) 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0f7fff640 1 -- 172.21.15.16:0/2888317534 learned_addr learned my addr 172.21.15.16:0/2888317534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c5f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5ade0 0x5616f7c6df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5ade0 msgr2=0x5616f7c6df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5ade0 0x5616f7c6df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2888317534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 msgr2=0x5616f7c6e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0ff59e640 1 -- 172.21.15.16:0/2888317534 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616f7c58170 con 0x5616f7c65b80 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.291+0000 7fa0f7fff640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa0ff59e640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c5f1e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800fe10 tx=0x7fa0f80118a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2a5f3af0fc82bd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa0e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8022070 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f8036070 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa0f801d690 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa0f8049050 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa1075ae740 10 client.29261 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.292+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5616f7cdfc70 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8036070 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa0f800ecb0 con 0x5616f7c65b80 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 10 client.29261 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 10 client.29261 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 20 client.29261 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7ce0190 0x5616f7ce2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 4 client.29261 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:07:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616f7bc5860 con 0x5616f7ce0190 2024-09-12T03:07:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7ce0190 0x5616f7ce2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa0fed9d640 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7ce0190 0x5616f7ce2580 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=962bb306454078f6 server_cookie=8f712a96df83f354 in_seq=0 out_seq=0 2024-09-12T03:07:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.293+0000 7fa0e67fc640 10 client.29261 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:07:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2888317534 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616f7bc5860 con 0x5616f7ce0190 2024-09-12T03:07:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa0e67fc640 10 client.29261 handle_command_reply: tid=0 2024-09-12T03:07:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 client.29261 shutdown 2024-09-12T03:07:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 msgr2=0x5616f7c5f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c5f1e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800fe10 tx=0x7fa0f80118a0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 shutdown_connections 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5616f7c65b80 0x5616f7c5f1e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5616f7c5c350 0x5616f7c6e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5616f7ce0190 0x5616f7ce2580 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 --2- 172.21.15.16:0/2888317534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5616f7c5ade0 0x5616f7c6df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 >> 172.21.15.16:0/2888317534 conn(0x5616f7c1b310 msgr2=0x5616f7c610c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa0e67fc640 0 client.29261 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 shutdown_connections 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 1 -- 172.21.15.16:0/2888317534 wait complete. 2024-09-12T03:07:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.296+0000 7fa1075ae740 20 client.29261 trim_cache size 0 max 16384 2024-09-12T03:07:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 msgr2=0x7fa0ec0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 0x7fa0ec0a5040 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8023000 tx=0x7fa0f8021000 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 msgr2=0x7fa10019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000e9f0 tx=0x7fa0f000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 shutdown_connections 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0ec0a2b80 0x7fa0ec0a5040 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa100109100 0x7fa10019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa100104980 0x7fa10019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 --2- 172.21.15.16:0/2241580833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa100103fd0 0x7fa10019a4a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 >> 172.21.15.16:0/2241580833 conn(0x7fa1000ffa30 msgr2=0x7fa100100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 shutdown_connections 2024-09-12T03:07:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:34.297+0000 7fa1051d5640 1 -- 172.21.15.16:0/2241580833 wait complete. 2024-09-12T03:07:35.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 159567 23.24 MB/sec execute 276 sec latency 100.535 ms 2024-09-12T03:07:35.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:07:35.431 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:35.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:35 smithi016 ceph-mon[30925]: pgmap v927: 97 pgs: 97 active+clean; 719 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 475 op/s 2024-09-12T03:07:35.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:35 smithi033 ceph-mon[35092]: pgmap v927: 97 pgs: 97 active+clean; 719 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 475 op/s 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 -- 172.21.15.16:0/1152368728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c102ef0 msgr2=0x7f091c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 --2- 172.21.15.16:0/1152368728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c102ef0 0x7f091c10da40 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0914009ca0 tx=0x7f091402a950 comp rx=0 tx=0).stop 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 -- 172.21.15.16:0/1152368728 shutdown_connections 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 --2- 172.21.15.16:0/1152368728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c102ef0 0x7f091c10da40 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 --2- 172.21.15.16:0/1152368728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102570 0x7f091c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 --2- 172.21.15.16:0/1152368728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c100e20 0x7f091c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 -- 172.21.15.16:0/1152368728 >> 172.21.15.16:0/1152368728 conn(0x7f091c0fc7d0 msgr2=0x7f091c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 -- 172.21.15.16:0/1152368728 shutdown_connections 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:35.999+0000 7f0920f78640 1 -- 172.21.15.16:0/1152368728 wait complete. 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 Processor -- start 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 -- start start 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c102570 0x7f091c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102ef0 0x7f091c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f091c110f80 con 0x7f091c100e20 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f091c110e00 con 0x7f091c102ef0 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0920f78640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f091c111100 con 0x7f091c102570 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f091a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f091a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40376/0 (socket says 172.21.15.16:40376) 2024-09-12T03:07:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f091a59c640 1 -- 172.21.15.16:0/1018527184 learned_addr learned my addr 172.21.15.16:0/1018527184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.000+0000 7f0919d9b640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c102570 0x7f091c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091ad9d640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102ef0 0x7f091c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 -- 172.21.15.16:0/1018527184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c102570 msgr2=0x7f091c19abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c102570 0x7f091c19abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 -- 172.21.15.16:0/1018527184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102ef0 msgr2=0x7f091c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102ef0 0x7f091c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 -- 172.21.15.16:0/1018527184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c19f630 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f091a59c640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e9f0 tx=0x7f090c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f09137fe640 1 -- 172.21.15.16:0/1018527184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c00cd80 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091c19f920 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f09137fe640 1 -- 172.21.15.16:0/1018527184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f090c005500 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f09137fe640 1 -- 172.21.15.16:0/1018527184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c010690 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.001+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091c1a7200 con 0x7f091c100e20 2024-09-12T03:07:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.004+0000 7f0923351740 1 Processor -- start 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.004+0000 7f09137fe640 1 -- 172.21.15.16:0/1018527184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f090c0046c0 con 0x7f091c100e20 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.004+0000 7f0923351740 1 -- start start 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b14f2f0 con 0x55bd3b1eab40 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b0f6cf0 con 0x55bd3b1f4370 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0923351740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b14b2b0 con 0x55bd3b1e95d0 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40384/0 (socket says 172.21.15.16:40384) 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 -- 172.21.15.16:0/3824619006 learned_addr learned my addr 172.21.15.16:0/3824619006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f0919d9b640 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 -- 172.21.15.16:0/3824619006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 msgr2=0x55bd3b1e9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1e9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 -- 172.21.15.16:0/3824619006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 msgr2=0x55bd3b1f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 -- 172.21.15.16:0/3824619006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd3b0db3a0 con 0x55bd3b1eab40 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091a59c640 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1e9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f09137fe640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 0x7f09080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091a59c640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 0x7f09080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.005+0000 7f091ad9d640 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f0914030020 tx=0x7f0914009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc7d79753193752 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.006+0000 7f09137fe640 1 -- 172.21.15.16:0/1018527184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f090c0cf4c0 con 0x7f091c100e20 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.006+0000 7f0910ff9640 1 -- 172.21.15.16:0/3824619006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914034040 con 0x55bd3b1eab40 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.006+0000 7f0910ff9640 1 -- 172.21.15.16:0/3824619006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f091402fb30 con 0x55bd3b1eab40 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.006+0000 7f0910ff9640 1 -- 172.21.15.16:0/3824619006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914004460 con 0x55bd3b1eab40 2024-09-12T03:07:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- 172.21.15.16:0/3824619006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 msgr2=0x55bd3b1e9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f091a59c640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 0x7f09080a5040 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f090c0cd340 tx=0x7f090c05e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f0914030020 tx=0x7f0914009360 comp rx=0 tx=0).stop 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- 172.21.15.16:0/3824619006 shutdown_connections 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1e9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 --2- 172.21.15.16:0/3824619006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1e9090 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- 172.21.15.16:0/3824619006 >> 172.21.15.16:0/3824619006 conn(0x55bd3b1b2310 msgr2=0x55bd3b1e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- 172.21.15.16:0/3824619006 shutdown_connections 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- 172.21.15.16:0/3824619006 wait complete. 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 -- start start 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.007+0000 7f0923351740 1 Processor -- start 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b1f7010 con 0x55bd3b1eab40 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b10a060 con 0x55bd3b1f4370 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b1f7190 con 0x55bd3b1e95d0 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0923351740 10 client.0 _ll_register_callbacks cb 0x55bd3ae5d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60802/0 (socket says 172.21.15.16:60802) 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f091ad9d640 1 -- 172.21.15.16:0/2124265273 learned_addr learned my addr 172.21.15.16:0/2124265273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40390/0 (socket says 172.21.15.16:40390) 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f091ad9d640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 -- 172.21.15.16:0/2124265273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 msgr2=0x55bd3b1f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 -- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 msgr2=0x55bd3b1f15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 -- 172.21.15.16:0/2124265273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3aec03c0 con 0x55bd3b1eab40 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f091ad9d640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0919d9b640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f091c10ddf0 tx=0x7f09040118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8789bc7fa94811ed server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.008+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904004020 con 0x55bd3b1eab40 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09040041c0 con 0x55bd3b1eab40 2024-09-12T03:07:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f091402fb30 con 0x55bd3b1eab40 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f09040043c0 con 0x55bd3b1eab40 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 10 client.20031 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55bd3b202a70 con 0x55bd3b1eab40 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f091402fb30 con 0x55bd3b1eab40 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f09040107c0 con 0x55bd3b1eab40 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 10 client.20031 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 10 client.20031 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 20 client.20031 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b203ed0 0x55bd3b2062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 4 client.20031 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd3b15c860 con 0x55bd3b203ed0 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.009+0000 7f091ad9d640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b203ed0 0x55bd3b2062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.011+0000 7f091ad9d640 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b203ed0 0x55bd3b2062f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2d7d95c06a19cd2 server_cookie=58a1f69600016adf in_seq=0 out_seq=0 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.011+0000 7f0901ffb640 10 client.20031 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0901ffb640 1 -- 172.21.15.16:0/2124265273 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd3b15c860 con 0x55bd3b203ed0 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0901ffb640 10 client.20031 handle_command_reply: tid=0 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 client.20031 shutdown 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 msgr2=0x55bd3b1f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f091c10ddf0 tx=0x7f09040118d0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 shutdown_connections 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b203ed0 0x55bd3b2062f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1f0fe0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 --2- 172.21.15.16:0/2124265273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0901ffb640 0 client.20031 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 >> 172.21.15.16:0/2124265273 conn(0x55bd3b1b2310 msgr2=0x55bd3b1eb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 shutdown_connections 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 1 -- 172.21.15.16:0/2124265273 wait complete. 2024-09-12T03:07:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.015+0000 7f0923351740 20 client.20031 trim_cache size 0 max 16384 2024-09-12T03:07:36.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 160176 23.31 MB/sec execute 277 sec latency 49.914 ms 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.029+0000 7f0923351740 1 Processor -- start 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.029+0000 7f0923351740 1 -- start start 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b2002b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1af740 con 0x55bd3b1eab40 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1b1280 con 0x55bd3b1f4370 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0923351740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1afe10 con 0x55bd3b1e95d0 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40402/0 (socket says 172.21.15.16:40402) 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 -- 172.21.15.16:0/3123323213 learned_addr learned my addr 172.21.15.16:0/3123323213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f091ad9d640 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b2002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 -- 172.21.15.16:0/3123323213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 msgr2=0x55bd3b2002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b2002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 -- 172.21.15.16:0/3123323213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 msgr2=0x55bd3b1ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 -- 172.21.15.16:0/3123323213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd3b206950 con 0x55bd3b1eab40 2024-09-12T03:07:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f091ad9d640 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b2002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.030+0000 7f0919d9b640 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0904010be0 tx=0x7f090400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae237ad67e54140 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.031+0000 7f0903fff640 1 -- 172.21.15.16:0/3123323213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904005850 con 0x55bd3b1eab40 2024-09-12T03:07:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.031+0000 7f0903fff640 1 -- 172.21.15.16:0/3123323213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0904010e50 con 0x55bd3b1eab40 2024-09-12T03:07:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.031+0000 7f0903fff640 1 -- 172.21.15.16:0/3123323213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09040049e0 con 0x55bd3b1eab40 2024-09-12T03:07:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- 172.21.15.16:0/3123323213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 msgr2=0x55bd3b1ea300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0904010be0 tx=0x7f090400b540 comp rx=0 tx=0).stop 2024-09-12T03:07:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- 172.21.15.16:0/3123323213 shutdown_connections 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1f4370 0x55bd3b1ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1eab40 0x55bd3b1ea300 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 --2- 172.21.15.16:0/3123323213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b2002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- 172.21.15.16:0/3123323213 >> 172.21.15.16:0/3123323213 conn(0x55bd3b1b2310 msgr2=0x55bd3b1e0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- 172.21.15.16:0/3123323213 shutdown_connections 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- 172.21.15.16:0/3123323213 wait complete. 2024-09-12T03:07:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 -- start start 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.032+0000 7f0923351740 1 Processor -- start 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1fc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 0x55bd3b1fcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1b0bb0 con 0x55bd3b1f4370 2024-09-12T03:07:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1b04e0 con 0x55bd3b1eab40 2024-09-12T03:07:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1ad720 con 0x55bd3b1e95d0 2024-09-12T03:07:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0923351740 10 client.0 _ll_register_callbacks cb 0x7f090400bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40406/0 (socket says 172.21.15.16:40406) 2024-09-12T03:07:36.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 -- 172.21.15.16:0/1408040837 learned_addr learned my addr 172.21.15.16:0/1408040837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1fc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0919d9b640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 0x55bd3b1fcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 -- 172.21.15.16:0/1408040837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 msgr2=0x55bd3b1fc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1fc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 -- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 msgr2=0x55bd3b1fcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 0x55bd3b1fcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f091ad9d640 1 -- 172.21.15.16:0/1408040837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3b1f81b0 con 0x55bd3b1f4370 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.033+0000 7f0919d9b640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 0x55bd3b1fcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f091ad9d640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f09140049d0 tx=0x7f091402a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424ec14177abfaf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:36.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091402fab0 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f091402f3a0 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0904010e50 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f091402f5a0 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 10 client.20037 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55bd3b26e610 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0904010e50 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f091404f040 con 0x55bd3b1f4370 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 10 client.20037 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 10 client.20037 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 20 client.20037 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:36.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.034+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b26e890 0x55bd3b270c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:36.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.035+0000 7f0923351740 4 client.20037 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:07:36.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.035+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd3b156d70 con 0x55bd3b26e890 2024-09-12T03:07:36.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.035+0000 7f0919d9b640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b26e890 0x55bd3b270c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:36.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.035+0000 7f0919d9b640 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b26e890 0x55bd3b270c80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=51136f7661a93584 server_cookie=7c98cf05361d7024 in_seq=0 out_seq=0 2024-09-12T03:07:36.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.035+0000 7f0901ffb640 10 client.20037 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0901ffb640 1 -- 172.21.15.16:0/1408040837 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd3b156d70 con 0x55bd3b26e890 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0901ffb640 10 client.20037 handle_command_reply: tid=0 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 client.20037 shutdown 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 msgr2=0x55bd3b1f7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f09140049d0 tx=0x7f091402a950 comp rx=0 tx=0).stop 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 shutdown_connections 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55bd3b26e890 0x55bd3b270c80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd3b1f4370 0x55bd3b1f7980 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd3b1eab40 0x55bd3b1fcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 --2- 172.21.15.16:0/1408040837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bd3b1e95d0 0x55bd3b1fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 >> 172.21.15.16:0/1408040837 conn(0x55bd3b1b2310 msgr2=0x55bd3b1ef740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0901ffb640 0 client.20037 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 shutdown_connections 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.037+0000 7f0923351740 1 -- 172.21.15.16:0/1408040837 wait complete. 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.038+0000 7f0923351740 20 client.20037 trim_cache size 0 max 16384 2024-09-12T03:07:36.045 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.038+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 msgr2=0x7f09080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.038+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 0x7f09080a5040 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f090c0cd340 tx=0x7f090c05e040 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.038+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 msgr2=0x7f091c19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.038+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e9f0 tx=0x7f090c00edf0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 shutdown_connections 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f091c102ef0 0x7f091c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f091c102570 0x7f091c19abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09080a2b80 0x7f09080a5040 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 --2- 172.21.15.16:0/1018527184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c100e20 0x7f091c19a680 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:36.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 >> 172.21.15.16:0/1018527184 conn(0x7f091c0fc7d0 msgr2=0x7f091c069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 shutdown_connections 2024-09-12T03:07:36.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:36.039+0000 7f0920f78640 1 -- 172.21.15.16:0/1018527184 wait complete. 2024-09-12T03:07:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:35 smithi184 ceph-mon[35321]: pgmap v927: 97 pgs: 97 active+clean; 719 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 475 op/s 2024-09-12T03:07:36.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:07:37.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 160649 23.30 MB/sec execute 278 sec latency 268.165 ms 2024-09-12T03:07:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 -- 172.21.15.16:0/4184613816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914101df0 msgr2=0x7f79141021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 --2- 172.21.15.16:0/4184613816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914101df0 0x7f79141021d0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f7904009900 tx=0x7f790402a410 comp rx=0 tx=0).stop 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 -- 172.21.15.16:0/4184613816 shutdown_connections 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 --2- 172.21.15.16:0/4184613816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914106f20 0x7f791410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 --2- 172.21.15.16:0/4184613816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f79141067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 --2- 172.21.15.16:0/4184613816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914101df0 0x7f79141021d0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.784+0000 7f791a40e640 1 -- 172.21.15.16:0/4184613816 >> 172.21.15.16:0/4184613816 conn(0x7f79140fd8f0 msgr2=0x7f79140ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.785+0000 7f791a40e640 1 -- 172.21.15.16:0/4184613816 shutdown_connections 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.785+0000 7f791a40e640 1 -- 172.21.15.16:0/4184613816 wait complete. 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 Processor -- start 2024-09-12T03:07:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 -- start start 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914101df0 0x7f79141983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7914111fd0 con 0x7f7914106f20 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7914111e50 con 0x7f7914101df0 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f791a40e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7914112150 con 0x7f79141027a0 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f79189ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f79137fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f79189ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40428/0 (socket says 172.21.15.16:40428) 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f79137fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60822/0 (socket says 172.21.15.16:60822) 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f79189ab640 1 -- 172.21.15.16:0/2934295154 learned_addr learned my addr 172.21.15.16:0/2934295154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.786+0000 7f7913fff640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914101df0 0x7f79141983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 -- 172.21.15.16:0/2934295154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 msgr2=0x7f7914198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 -- 172.21.15.16:0/2934295154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914101df0 msgr2=0x7f79141983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914101df0 0x7f79141983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 -- 172.21.15.16:0/2934295154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791419d3a0 con 0x7f7914106f20 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79137fe640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79189ab640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f790c00cce0 tx=0x7f790c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79117fa640 1 -- 172.21.15.16:0/2934295154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c00f450 con 0x7f7914106f20 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79117fa640 1 -- 172.21.15.16:0/2934295154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f790c00fad0 con 0x7f7914106f20 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791419d690 con 0x7f7914106f20 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f79117fa640 1 -- 172.21.15.16:0/2934295154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c018c10 con 0x7f7914106f20 2024-09-12T03:07:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.787+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79141a4f70 con 0x7f7914106f20 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f79117fa640 1 -- 172.21.15.16:0/2934295154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f790c00f5f0 con 0x7f7914106f20 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 Processor -- start 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 -- start start 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004d97090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004da4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a004cfd2f0 con 0x55a004d975d0 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a004ca4cf0 con 0x55a004d98b40 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.790+0000 7f791c7e7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a004cf92b0 con 0x55a004da2370 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40440/0 (socket says 172.21.15.16:40440) 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 -- 172.21.15.16:0/4263678468 learned_addr learned my addr 172.21.15.16:0/4263678468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79137fe640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004da4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79137fe640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004d97090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 -- 172.21.15.16:0/4263678468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 msgr2=0x55a004da4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004da4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 -- 172.21.15.16:0/4263678468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 msgr2=0x55a004d97090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004d97090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 -- 172.21.15.16:0/4263678468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a004c893a0 con 0x55a004d975d0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79137fe640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004da4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79117fa640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 0x7f78fc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f7913fff640 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7904009230 tx=0x7f79040040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb999818f33e238f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79117fa640 1 -- 172.21.15.16:0/2934295154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f790c0cf5b0 con 0x7f7914106f20 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f79137fe640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 0x7f78fc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.791+0000 7f78fa7fc640 1 -- 172.21.15.16:0/4263678468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904035040 con 0x55a004d975d0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.792+0000 7f78fa7fc640 1 -- 172.21.15.16:0/4263678468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904005520 con 0x55a004d975d0 2024-09-12T03:07:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.792+0000 7f78fa7fc640 1 -- 172.21.15.16:0/4263678468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904003850 con 0x55a004d975d0 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- 172.21.15.16:0/4263678468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 msgr2=0x55a004d97a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f79137fe640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 0x7f78fc0a5040 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55a004d98070 tx=0x7f7908005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7904009230 tx=0x7f79040040f0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- 172.21.15.16:0/4263678468 shutdown_connections 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004da4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d97a10 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 --2- 172.21.15.16:0/4263678468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004d97090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- 172.21.15.16:0/4263678468 >> 172.21.15.16:0/4263678468 conn(0x55a004d60310 msgr2=0x55a004d93c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- 172.21.15.16:0/4263678468 shutdown_connections 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- 172.21.15.16:0/4263678468 wait complete. 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.793+0000 7f791c7e7740 1 -- start start 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 Processor -- start 2024-09-12T03:07:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d975d0 0x55a004da1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004d9eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a004da50b0 con 0x55a004da2370 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a004cb8060 con 0x55a004d975d0 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a004da5230 con 0x55a004d98b40 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f791c7e7740 10 client.0 _ll_register_callbacks cb 0x55a004a0b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f79137fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004d9eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f79189ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f79189ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40446/0 (socket says 172.21.15.16:40446) 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f79189ab640 1 -- 172.21.15.16:0/606341375 learned_addr learned my addr 172.21.15.16:0/606341375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f7913fff640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d975d0 0x55a004da1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.794+0000 7f79137fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004d9eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60830/0 (socket says 172.21.15.16:60830) 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 -- 172.21.15.16:0/606341375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 msgr2=0x55a004d9eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004d9eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 -- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d975d0 msgr2=0x55a004da1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d975d0 0x55a004da1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 -- 172.21.15.16:0/606341375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a004a6e3c0 con 0x55a004da2370 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f79189ab640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f790c09bdb0 tx=0x7f790c09b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d2fac2c550aa98 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f78f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0a01d0 con 0x55a004da2370 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7904005520 con 0x55a004da2370 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f790c0a4600 con 0x55a004da2370 2024-09-12T03:07:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f790c0a4860 con 0x55a004da2370 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f791c7e7740 10 client.20049 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.795+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a004db93e0 con 0x55a004da2370 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904005520 con 0x55a004da2370 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f790c097b70 con 0x55a004da2370 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 10 client.20049 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 10 client.20049 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 20 client.20049 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004dba8a0 0x55a004dbccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 4 client.20049 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a004d0a860 con 0x55a004dba8a0 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.796+0000 7f7913fff640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004dba8a0 0x55a004dbccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.797+0000 7f78f37fe640 10 client.20049 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.797+0000 7f7913fff640 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004dba8a0 0x55a004dbccf0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0d0e3e2e1027fb6 server_cookie=ae2f547711ea0f42 in_seq=0 out_seq=0 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f78f37fe640 1 -- 172.21.15.16:0/606341375 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a004d0a860 con 0x55a004dba8a0 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f78f37fe640 10 client.20049 handle_command_reply: tid=0 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 client.20049 shutdown 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 msgr2=0x55a004d9f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f790c09bdb0 tx=0x7f790c09b780 comp rx=0 tx=0).stop 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 shutdown_connections 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004dba8a0 0x55a004dbccf0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004da2370 0x55a004d9f5e0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004d9eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 --2- 172.21.15.16:0/606341375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d975d0 0x55a004da1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f78f37fe640 0 client.20049 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 >> 172.21.15.16:0/606341375 conn(0x55a004d60310 msgr2=0x55a004d99930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 shutdown_connections 2024-09-12T03:07:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 1 -- 172.21.15.16:0/606341375 wait complete. 2024-09-12T03:07:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.799+0000 7f791c7e7740 20 client.20049 trim_cache size 0 max 16384 2024-09-12T03:07:37.812 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:37 smithi016 ceph-mon[30925]: pgmap v928: 97 pgs: 97 active+clean; 724 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.6 MiB/s wr, 531 op/s 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 Processor -- start 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 -- start start 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004db6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004da2370 0x55a004d9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ab1d0 con 0x55a004d975d0 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79141acd10 con 0x55a004da2370 2024-09-12T03:07:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f791c7e7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ab8a0 con 0x55a004d98b40 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40458/0 (socket says 172.21.15.16:40458) 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f7913fff640 1 -- 172.21.15.16:0/989992632 learned_addr learned my addr 172.21.15.16:0/989992632 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.814+0000 7f79137fe640 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004db6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f79189ab640 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004da2370 0x55a004d9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 -- 172.21.15.16:0/989992632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 msgr2=0x55a004db6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004db6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 -- 172.21.15.16:0/989992632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004da2370 msgr2=0x55a004d9e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004da2370 0x55a004d9e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 -- 172.21.15.16:0/989992632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a004d9dfa0 con 0x55a004d975d0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f7913fff640 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7904009230 tx=0x7f7904004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2673da7a67a265 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f78f9ffb640 1 -- 172.21.15.16:0/989992632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904035040 con 0x55a004d975d0 2024-09-12T03:07:37.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f78f9ffb640 1 -- 172.21.15.16:0/989992632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79040048c0 con 0x55a004d975d0 2024-09-12T03:07:37.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.815+0000 7f78f9ffb640 1 -- 172.21.15.16:0/989992632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904031ae0 con 0x55a004d975d0 2024-09-12T03:07:37.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- 172.21.15.16:0/989992632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 msgr2=0x55a004d93e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7904009230 tx=0x7f7904004d60 comp rx=0 tx=0).stop 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- 172.21.15.16:0/989992632 shutdown_connections 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004da2370 0x55a004d9e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004d98b40 0x55a004db6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 --2- 172.21.15.16:0/989992632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004d93e10 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- 172.21.15.16:0/989992632 >> 172.21.15.16:0/989992632 conn(0x55a004d60310 msgr2=0x55a004d93700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- 172.21.15.16:0/989992632 shutdown_connections 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- 172.21.15.16:0/989992632 wait complete. 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.816+0000 7f791c7e7740 1 -- start start 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 Processor -- start 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004dae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004e23580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ac640 con 0x55a004d975d0 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79141abf70 con 0x55a004d98b40 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f79141a91d0 con 0x55a004da2370 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f791c7e7740 10 client.0 _ll_register_callbacks cb 0x7f790c0077a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40464/0 (socket says 172.21.15.16:40464) 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 -- 172.21.15.16:0/1970238169 learned_addr learned my addr 172.21.15.16:0/1970238169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f79189ab640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004e23580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 -- 172.21.15.16:0/1970238169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 msgr2=0x55a004e23580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004e23580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 -- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 msgr2=0x55a004dae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004dae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.817+0000 7f7913fff640 1 -- 172.21.15.16:0/1970238169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a004e23b40 con 0x55a004d975d0 2024-09-12T03:07:37.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f7913fff640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f790402aa00 tx=0x7f790402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac3d444923c1f0f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904035070 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7904031a70 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79040048c0 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7904068050 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 10 client.20055 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a004e24e20 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79040048c0 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7904057070 con 0x55a004d975d0 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 10 client.20055 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 10 client.20055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 20 client.20055 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:37.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004e25050 0x55a004e27440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 4 client.20055 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.818+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a004d04d70 con 0x55a004e25050 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f79137fe640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004e25050 0x55a004e27440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f78f37fe640 10 client.20055 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f79137fe640 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004e25050 0x55a004e27440 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=575089bff3c671a9 server_cookie=7a6bb27e7f7a19e2 in_seq=0 out_seq=0 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f78f37fe640 1 -- 172.21.15.16:0/1970238169 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a004d04d70 con 0x55a004e25050 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f78f37fe640 10 client.20055 handle_command_reply: tid=0 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.819+0000 7f791c7e7740 1 client.20055 shutdown 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 msgr2=0x55a004dae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f790402aa00 tx=0x7f790402a410 comp rx=0 tx=0).stop 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 shutdown_connections 2024-09-12T03:07:37.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a004e25050 0x55a004e27440 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a004da2370 0x55a004e23580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a004d98b40 0x55a004dae7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 --2- 172.21.15.16:0/1970238169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a004d975d0 0x55a004dae2b0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 >> 172.21.15.16:0/1970238169 conn(0x55a004d60310 msgr2=0x55a004d92500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f78f37fe640 0 client.20055 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 shutdown_connections 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 1 -- 172.21.15.16:0/1970238169 wait complete. 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.820+0000 7f791c7e7740 20 client.20055 trim_cache size 0 max 16384 2024-09-12T03:07:37.825 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 msgr2=0x7f78fc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 0x7f78fc0a5040 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55a004d98070 tx=0x7f7908005f90 comp rx=0 tx=0).stop 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 msgr2=0x7f791419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f790c00cce0 tx=0x7f790c00a620 comp rx=0 tx=0).stop 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 shutdown_connections 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f78fc0a2b80 0x7f78fc0a5040 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914106f20 0x7f791419ccc0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79141027a0 0x7f7914198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 --2- 172.21.15.16:0/2934295154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7914101df0 0x7f79141983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 >> 172.21.15.16:0/2934295154 conn(0x7f79140fd8f0 msgr2=0x7f79140fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 shutdown_connections 2024-09-12T03:07:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:37.821+0000 7f791a40e640 1 -- 172.21.15.16:0/2934295154 wait complete. 2024-09-12T03:07:37.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:37 smithi033 ceph-mon[35092]: pgmap v928: 97 pgs: 97 active+clean; 724 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.6 MiB/s wr, 531 op/s 2024-09-12T03:07:38.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 161054 23.26 MB/sec execute 279 sec latency 307.783 ms 2024-09-12T03:07:38.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:37 smithi184 ceph-mon[35321]: pgmap v928: 97 pgs: 97 active+clean; 724 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.6 MiB/s wr, 531 op/s 2024-09-12T03:07:39.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 161671 23.29 MB/sec execute 280 sec latency 150.584 ms 2024-09-12T03:07:39.523 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:07:39.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:07:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:39 smithi016 ceph-mon[30925]: pgmap v929: 97 pgs: 97 active+clean; 725 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 470 op/s 2024-09-12T03:07:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:39 smithi033 ceph-mon[35092]: pgmap v929: 97 pgs: 97 active+clean; 725 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 470 op/s 2024-09-12T03:07:40.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 162164 23.32 MB/sec execute 281 sec latency 291.309 ms 2024-09-12T03:07:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:39 smithi184 ceph-mon[35321]: pgmap v929: 97 pgs: 97 active+clean; 725 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 470 op/s 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 -- 172.21.15.16:0/2895994954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb880106b80 msgr2=0x7fb880106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 --2- 172.21.15.16:0/2895994954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb880106b80 0x7fb880106f60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009860 tx=0x7fb87c02a340 comp rx=0 tx=0).stop 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 -- 172.21.15.16:0/2895994954 shutdown_connections 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 --2- 172.21.15.16:0/2895994954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8801005a0 0x7fb88010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 --2- 172.21.15.16:0/2895994954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8800ffc00 0x7fb880100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 --2- 172.21.15.16:0/2895994954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb880106b80 0x7fb880106f60 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.288+0000 7fb887cc3640 1 -- 172.21.15.16:0/2895994954 >> 172.21.15.16:0/2895994954 conn(0x7fb8800fba90 msgr2=0x7fb8800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.289+0000 7fb887cc3640 1 -- 172.21.15.16:0/2895994954 shutdown_connections 2024-09-12T03:07:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.289+0000 7fb887cc3640 1 -- 172.21.15.16:0/2895994954 wait complete. 2024-09-12T03:07:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.289+0000 7fb887cc3640 1 Processor -- start 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.289+0000 7fb887cc3640 1 -- start start 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8801005a0 0x7fb88019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb880106b80 0x7fb88019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb880111fa0 con 0x7fb8800ffc00 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb880111e20 con 0x7fb880106b80 2024-09-12T03:07:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb887cc3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb880112120 con 0x7fb8801005a0 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40486/0 (socket says 172.21.15.16:40486) 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 -- 172.21.15.16:0/1533058338 learned_addr learned my addr 172.21.15.16:0/1533058338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb88525e640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8801005a0 0x7fb88019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb886260640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb880106b80 0x7fb88019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 -- 172.21.15.16:0/1533058338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8801005a0 msgr2=0x7fb88019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8801005a0 0x7fb88019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 -- 172.21.15.16:0/1533058338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb880106b80 msgr2=0x7fb88019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb880106b80 0x7fb88019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.290+0000 7fb885a5f640 1 -- 172.21.15.16:0/1533058338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88019f570 con 0x7fb8800ffc00 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb885a5f640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fb87c035040 tx=0x7fb87c033b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb876ffd640 1 -- 172.21.15.16:0/1533058338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03f020 con 0x7fb8800ffc00 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88019f800 con 0x7fb8800ffc00 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb876ffd640 1 -- 172.21.15.16:0/1533058338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87c004040 con 0x7fb8800ffc00 2024-09-12T03:07:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb876ffd640 1 -- 172.21.15.16:0/1533058338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c037040 con 0x7fb8800ffc00 2024-09-12T03:07:40.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.291+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8801a7140 con 0x7fb8800ffc00 2024-09-12T03:07:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 Processor -- start 2024-09-12T03:07:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 -- start start 2024-09-12T03:07:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd68480 0x55e73dd72800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd75130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e73dccd280 con 0x55e73dd67b60 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e73dc74b10 con 0x55e73dd68480 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.294+0000 7fb88a09c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e73dcc9240 con 0x55e73dd72d40 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb876ffd640 1 -- 172.21.15.16:0/1533058338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb87c042020 con 0x7fb8800ffc00 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb88525e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd75130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40494/0 (socket says 172.21.15.16:40494) 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 -- 172.21.15.16:0/2541484853 learned_addr learned my addr 172.21.15.16:0/2541484853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb885a5f640 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd68480 0x55e73dd72800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 -- 172.21.15.16:0/2541484853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 msgr2=0x55e73dd75130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd75130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 -- 172.21.15.16:0/2541484853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd68480 msgr2=0x55e73dd72800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd68480 0x55e73dd72800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 -- 172.21.15.16:0/2541484853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e73dc811f0 con 0x55e73dd67b60 2024-09-12T03:07:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.295+0000 7fb886260640 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fb878009f90 tx=0x7fb8780118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d0a8a6149e0a03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb86bfff640 1 -- 172.21.15.16:0/2541484853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878004d20 con 0x55e73dd67b60 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb86bfff640 1 -- 172.21.15.16:0/2541484853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb878004050 con 0x55e73dd67b60 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb86bfff640 1 -- 172.21.15.16:0/2541484853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878005280 con 0x55e73dd67b60 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb876ffd640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 0x7fb86c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb885a5f640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 0x7fb86c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb876ffd640 1 -- 172.21.15.16:0/1533058338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb87c0ef100 con 0x7fb8800ffc00 2024-09-12T03:07:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb885a5f640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 0x7fb86c0a5040 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0ecf80 tx=0x7fb87c005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 -- 172.21.15.16:0/2541484853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 msgr2=0x55e73dd67f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fb878009f90 tx=0x7fb8780118d0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 -- 172.21.15.16:0/2541484853 shutdown_connections 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd75130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd68480 0x55e73dd72800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 --2- 172.21.15.16:0/2541484853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd67b60 0x55e73dd67f40 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.296+0000 7fb88a09c740 1 -- 172.21.15.16:0/2541484853 >> 172.21.15.16:0/2541484853 conn(0x55e73dd640f0 msgr2=0x55e73dd644d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.297+0000 7fb88a09c740 1 -- 172.21.15.16:0/2541484853 shutdown_connections 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.297+0000 7fb88a09c740 1 -- 172.21.15.16:0/2541484853 wait complete. 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.297+0000 7fb88a09c740 1 -- start start 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.297+0000 7fb88a09c740 1 Processor -- start 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd70600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e73dd75910 con 0x55e73dd68480 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60880/0 (socket says 172.21.15.16:60880) 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 -- 172.21.15.16:0/3492639820 learned_addr learned my addr 172.21.15.16:0/3492639820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e73dc87f10 con 0x55e73dd72d40 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e73dd75a90 con 0x55e73dd67b60 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd70600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88525e640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 -- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 msgr2=0x55e73dd70600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd70600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 -- 172.21.15.16:0/3492639820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 msgr2=0x55e73dd700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb886260640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.298+0000 7fb88a09c740 10 client.0 _ll_register_callbacks cb 0x55e73dc50a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb886260640 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e73da3e6a0 con 0x55e73dd67b60 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb88525e640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb878004050 con 0x55e73dd67b60 2024-09-12T03:07:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb886260640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb87801b040 tx=0x7fb878005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d33b946a8c4ac32 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb8697fa640 10 client.29230 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb8697fa640 1 -- 172.21.15.16:0/3492639820 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878004520 con 0x55e73dd67b60 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb8697fa640 1 -- 172.21.15.16:0/3492639820 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb87800f4e0 con 0x55e73dd67b60 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb8697fa640 1 -- 172.21.15.16:0/3492639820 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb87800f6e0 con 0x55e73dd67b60 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb88a09c740 10 client.29230 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.299+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55e73dd812e0 con 0x55e73dd67b60 2024-09-12T03:07:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb8697fa640 1 -- 172.21.15.16:0/3492639820 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb87800f880 con 0x55e73dd67b60 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 10 client.29230 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 10 client.29230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 20 client.29230 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73dd82a80 0x55e73dd84f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 4 client.29230 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e73dcda7f0 con 0x55e73dd82a80 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88525e640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73dd82a80 0x55e73dd84f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb88525e640 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73dd82a80 0x55e73dd84f20 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b96e3c0102a69a2 server_cookie=1734a013232057f2 in_seq=0 out_seq=0 2024-09-12T03:07:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.300+0000 7fb8697fa640 10 client.29230 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:40.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb8697fa640 1 -- 172.21.15.16:0/3492639820 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e73dcda7f0 con 0x55e73dd82a80 2024-09-12T03:07:40.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb8697fa640 10 client.29230 handle_command_reply: tid=0 2024-09-12T03:07:40.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 client.29230 shutdown 2024-09-12T03:07:40.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 msgr2=0x55e73dd6fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb87801b040 tx=0x7fb878005bd0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 shutdown_connections 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73dd82a80 0x55e73dd84f20 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd70600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb8697fa640 0 client.29230 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3492639820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd6fb80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 >> 172.21.15.16:0/3492639820 conn(0x55e73dd640f0 msgr2=0x55e73dd69e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 shutdown_connections 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 1 -- 172.21.15.16:0/3492639820 wait complete. 2024-09-12T03:07:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.302+0000 7fb88a09c740 20 client.29230 trim_cache size 0 max 16384 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.322+0000 7fb88a09c740 1 Processor -- start 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.322+0000 7fb88a09c740 1 -- start start 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd8ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801af6e0 con 0x55e73dd68480 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801b1220 con 0x55e73dd72d40 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88a09c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801afdb0 con 0x55e73dd67b60 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60886/0 (socket says 172.21.15.16:60886) 2024-09-12T03:07:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 -- 172.21.15.16:0/3033573418 learned_addr learned my addr 172.21.15.16:0/3033573418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88525e640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd8ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 -- 172.21.15.16:0/3033573418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 msgr2=0x55e73dd8ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd8ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 -- 172.21.15.16:0/3033573418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 msgr2=0x55e73dd8a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd8a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 -- 172.21.15.16:0/3033573418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e73dd63f20 con 0x55e73dd67b60 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb88525e640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd8a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.323+0000 7fb886260640 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb878011e50 tx=0x7fb878014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a5f1ba36aa45d1e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb86b7fe640 1 -- 172.21.15.16:0/3033573418 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878005bc0 con 0x55e73dd67b60 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb86b7fe640 1 -- 172.21.15.16:0/3033573418 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb878004050 con 0x55e73dd67b60 2024-09-12T03:07:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb86b7fe640 1 -- 172.21.15.16:0/3033573418 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878032b90 con 0x55e73dd67b60 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 -- 172.21.15.16:0/3033573418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 msgr2=0x55e73dd719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb878011e50 tx=0x7fb878014040 comp rx=0 tx=0).stop 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 -- 172.21.15.16:0/3033573418 shutdown_connections 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd72d40 0x55e73dd8ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd8a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3033573418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd67b60 0x55e73dd719e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.324+0000 7fb88a09c740 1 -- 172.21.15.16:0/3033573418 >> 172.21.15.16:0/3033573418 conn(0x55e73dd640f0 msgr2=0x55e73dd712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.325+0000 7fb88a09c740 1 -- 172.21.15.16:0/3033573418 shutdown_connections 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.325+0000 7fb88a09c740 1 -- 172.21.15.16:0/3033573418 wait complete. 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.325+0000 7fb88a09c740 1 -- start start 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.325+0000 7fb88a09c740 1 Processor -- start 2024-09-12T03:07:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 0x55e73dd6cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd6d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801b0b50 con 0x55e73dd68480 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801b0480 con 0x55e73dd67b60 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb8801ad6c0 con 0x55e73dd72d40 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88a09c740 10 client.0 _ll_register_callbacks cb 0x7fb870009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88525e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 0x55e73dd6cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88525e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40518/0 (socket says 172.21.15.16:40518) 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88525e640 1 -- 172.21.15.16:0/3576593218 learned_addr learned my addr 172.21.15.16:0/3576593218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd6d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 -- 172.21.15.16:0/3576593218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 msgr2=0x55e73dd6d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd6d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 -- 172.21.15.16:0/3576593218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 msgr2=0x55e73dd6d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 -- 172.21.15.16:0/3576593218 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e73ddec8b0 con 0x55e73dd67b60 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb88525e640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb886260640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 0x55e73dd6cd00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb878009320 tx=0x7fb878005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ecea20de4a9cc5d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb8697fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.326+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878005bc0 con 0x55e73dd67b60 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb878004790 con 0x55e73dd67b60 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb878004050 con 0x55e73dd67b60 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb878009be0 con 0x55e73dd67b60 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 10 client.29303 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e73ddedbb0 con 0x55e73dd67b60 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb878004050 con 0x55e73dd67b60 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb87803a070 con 0x55e73dd67b60 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 10 client.29303 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 10 client.29303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 20 client.29303 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73ddee1c0 0x55e73ddf05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 4 client.29303 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e73dcd4d00 con 0x55e73ddee1c0 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.327+0000 7fb88525e640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73ddee1c0 0x55e73ddf05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.328+0000 7fb88525e640 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73ddee1c0 0x55e73ddf05b0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cccd3723b68c7d53 server_cookie=56a20f26789a7417 in_seq=0 out_seq=0 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.328+0000 7fb8697fa640 10 client.29303 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb8697fa640 1 -- 172.21.15.16:0/3576593218 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e73dcd4d00 con 0x55e73ddee1c0 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb8697fa640 10 client.29303 handle_command_reply: tid=0 2024-09-12T03:07:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 client.29303 shutdown 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 msgr2=0x55e73dd6cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 0x55e73dd6cd00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb878009320 tx=0x7fb878005f10 comp rx=0 tx=0).stop 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 shutdown_connections 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e73ddee1c0 0x55e73ddf05b0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e73dd72d40 0x55e73dd6d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e73dd68480 0x55e73dd6d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 --2- 172.21.15.16:0/3576593218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e73dd67b60 0x55e73dd6cd00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 >> 172.21.15.16:0/3576593218 conn(0x55e73dd640f0 msgr2=0x55e73dd7e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.329+0000 7fb8697fa640 0 client.29303 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 shutdown_connections 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb88a09c740 1 -- 172.21.15.16:0/3576593218 wait complete. 2024-09-12T03:07:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb88a09c740 20 client.29303 trim_cache size 0 max 16384 2024-09-12T03:07:40.336 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:07:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:07:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d44d1a11-fc00-4634-b3f0-01644823b528", 2024-09-12T03:07:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:07:40.336 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 msgr2=0x7fb86c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 0x7fb86c0a5040 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0ecf80 tx=0x7fb87c005dc0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 msgr2=0x7fb88019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fb87c035040 tx=0x7fb87c033b60 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 shutdown_connections 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb880106b80 0x7fb88019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8801005a0 0x7fb88019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb86c0a2b80 0x7fb86c0a5040 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.330+0000 7fb887cc3640 1 --2- 172.21.15.16:0/1533058338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8800ffc00 0x7fb88019a5c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.331+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 >> 172.21.15.16:0/1533058338 conn(0x7fb8800fba90 msgr2=0x7fb88010cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.331+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 shutdown_connections 2024-09-12T03:07:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:40.331+0000 7fb887cc3640 1 -- 172.21.15.16:0/1533058338 wait complete. 2024-09-12T03:07:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:40 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:40.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:40 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:40.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:40 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:40.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:40 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:40.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:40 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:40.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:40 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:40.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:40 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:40.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:40 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 162798 23.34 MB/sec execute 282 sec latency 55.805 ms 2024-09-12T03:07:41.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:07:41.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:40 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:41.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:40 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:40 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:41.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:40 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.327 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:07:41.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.873+0000 7fe42a46c640 1 -- 172.21.15.16:0/1621608068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424100a80 msgr2=0x7fe424100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.873+0000 7fe42a46c640 1 --2- 172.21.15.16:0/1621608068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424100a80 0x7fe424100ee0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009930 tx=0x7fe40c02a420 comp rx=0 tx=0).stop 2024-09-12T03:07:41.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 -- 172.21.15.16:0/1621608068 shutdown_connections 2024-09-12T03:07:41.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 --2- 172.21.15.16:0/1621608068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424101420 0x7fe42410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 --2- 172.21.15.16:0/1621608068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424100a80 0x7fe424100ee0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 --2- 172.21.15.16:0/1621608068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe424106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 -- 172.21.15.16:0/1621608068 >> 172.21.15.16:0/1621608068 conn(0x7fe4240fc7a0 msgr2=0x7fe4240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 -- 172.21.15.16:0/1621608068 shutdown_connections 2024-09-12T03:07:41.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.874+0000 7fe42a46c640 1 -- 172.21.15.16:0/1621608068 wait complete. 2024-09-12T03:07:41.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 Processor -- start 2024-09-12T03:07:41.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 -- start start 2024-09-12T03:07:41.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424100a80 0x7fe4240fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101420 0x7fe4240fd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110e40 con 0x7fe424101420 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110cc0 con 0x7fe424100a80 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe42a46c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110fc0 con 0x7fe424106a80 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe428a09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe423fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424100a80 0x7fe4240fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe428a09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60904/0 (socket says 172.21.15.16:60904) 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.875+0000 7fe428a09640 1 -- 172.21.15.16:0/3495396784 learned_addr learned my addr 172.21.15.16:0/3495396784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe41bfff640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101420 0x7fe4240fd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 -- 172.21.15.16:0/3495396784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424100a80 msgr2=0x7fe4240fcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424100a80 0x7fe4240fcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 -- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101420 msgr2=0x7fe4240fd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101420 0x7fe4240fd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 -- 172.21.15.16:0/3495396784 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241aadb0 con 0x7fe424106a80 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe428a09640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00eb10 tx=0x7fe41c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.876+0000 7fe421ffb640 1 -- 172.21.15.16:0/3495396784 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c004b20 con 0x7fe424106a80 2024-09-12T03:07:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.877+0000 7fe421ffb640 1 -- 172.21.15.16:0/3495396784 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c004cc0 con 0x7fe424106a80 2024-09-12T03:07:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.877+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241aafe0 con 0x7fe424106a80 2024-09-12T03:07:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.877+0000 7fe421ffb640 1 -- 172.21.15.16:0/3495396784 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c010790 con 0x7fe424106a80 2024-09-12T03:07:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.877+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241ab440 con 0x7fe424106a80 2024-09-12T03:07:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.879+0000 7fe421ffb640 1 -- 172.21.15.16:0/3495396784 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe41c005020 con 0x7fe424106a80 2024-09-12T03:07:41.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 Processor -- start 2024-09-12T03:07:41.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 -- start start 2024-09-12T03:07:41.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe421ffb640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 0x7fe4100a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44b6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44c1680 0x55b6a44c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6a441c2e0 con 0x55b6a44b64a0 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6a43c3ce0 con 0x55b6a44b6dc0 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe42c845740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b6a44182a0 con 0x55b6a44c1680 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe421ffb640 1 -- 172.21.15.16:0/3495396784 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe41c0cf560 con 0x7fe424106a80 2024-09-12T03:07:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe41bfff640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 0x7fe4100a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.880+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41654/0 (socket says 172.21.15.16:41654) 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe41bfff640 1 -- 172.21.15.16:0/1449129987 learned_addr learned my addr 172.21.15.16:0/1449129987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe428a09640 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44c1680 0x55b6a44c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe423fff640 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe41bfff640 1 -- 172.21.15.16:0/1449129987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44c1680 msgr2=0x55b6a44c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe41bfff640 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44c1680 0x55b6a44c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe41bfff640 1 -- 172.21.15.16:0/1449129987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 msgr2=0x55b6a44b6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.881+0000 7fe41bfff640 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44b6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.882+0000 7fe41bfff640 1 -- 172.21.15.16:0/1449129987 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a43a8440 con 0x55b6a44b6dc0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.882+0000 7fe41bfff640 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe40c02a940 tx=0x7fe40c0318e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a13d0878d70f040 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.882+0000 7fe41bfff640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 0x7fe4100a50c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009930 tx=0x7fe40c043000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.883+0000 7fe41a7fc640 1 -- 172.21.15.16:0/1449129987 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c032d70 con 0x55b6a44b6dc0 2024-09-12T03:07:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.883+0000 7fe41a7fc640 1 -- 172.21.15.16:0/1449129987 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40c003cd0 con 0x55b6a44b6dc0 2024-09-12T03:07:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.884+0000 7fe41a7fc640 1 -- 172.21.15.16:0/1449129987 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c03da00 con 0x55b6a44b6dc0 2024-09-12T03:07:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- 172.21.15.16:0/1449129987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 msgr2=0x55b6a44c1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe40c02a940 tx=0x7fe40c0318e0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- 172.21.15.16:0/1449129987 shutdown_connections 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44c1680 0x55b6a44c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c1140 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 --2- 172.21.15.16:0/1449129987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44b6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- 172.21.15.16:0/1449129987 >> 172.21.15.16:0/1449129987 conn(0x55b6a44b2a30 msgr2=0x55b6a44b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- 172.21.15.16:0/1449129987 shutdown_connections 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- 172.21.15.16:0/1449129987 wait complete. 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.885+0000 7fe42c845740 1 -- start start 2024-09-12T03:07:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 Processor -- start 2024-09-12T03:07:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44be2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44be840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6a44c4250 con 0x55b6a44b64a0 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6a43d7050 con 0x55b6a44c1680 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b6a44c43d0 con 0x55b6a44b6dc0 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe42c845740 10 client.0 _ll_register_callbacks cb 0x55b6a4417160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe423fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe423fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40552/0 (socket says 172.21.15.16:40552) 2024-09-12T03:07:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe423fff640 1 -- 172.21.15.16:0/2472064647 learned_addr learned my addr 172.21.15.16:0/2472064647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.886+0000 7fe428a09640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe41bfff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44be2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 -- 172.21.15.16:0/2472064647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 msgr2=0x55b6a44be2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44be2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 -- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 msgr2=0x55b6a44be840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44be840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 -- 172.21.15.16:0/2472064647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a418d3c0 con 0x55b6a44b64a0 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe423fff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe414012740 tx=0x7fe414012b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f78862985801124 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe4037fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41400f920 con 0x55b6a44b64a0 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe41400fac0 con 0x55b6a44b64a0 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.887+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe40c003cd0 con 0x55b6a44b64a0 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe41400fcc0 con 0x55b6a44b64a0 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 10 client.20085 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b6a44cfcf0 con 0x55b6a44b64a0 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40c003cd0 con 0x55b6a44b64a0 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe41401c420 con 0x55b6a44b64a0 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 10 client.20085 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 10 client.20085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 20 client.20085 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a44d1530 0x55b6a44d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 4 client.20085 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.888+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6a4429850 con 0x55b6a44d1530 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.889+0000 7fe41bfff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a44d1530 0x55b6a44d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.889+0000 7fe4037fe640 10 client.20085 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.889+0000 7fe41bfff640 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a44d1530 0x55b6a44d3990 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc814f8fa91a7e2d server_cookie=18f76b60d378084d in_seq=0 out_seq=0 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.890+0000 7fe4037fe640 1 -- 172.21.15.16:0/2472064647 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b6a4429850 con 0x55b6a44d1530 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.890+0000 7fe4037fe640 10 client.20085 handle_command_reply: tid=0 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.890+0000 7fe42c845740 1 client.20085 shutdown 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 msgr2=0x55b6a44c1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe414012740 tx=0x7fe414012b40 comp rx=0 tx=0).stop 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 shutdown_connections 2024-09-12T03:07:41.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a44d1530 0x55b6a44d3990 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44be840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44be2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 --2- 172.21.15.16:0/2472064647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44b64a0 0x55b6a44c1010 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 >> 172.21.15.16:0/2472064647 conn(0x55b6a44b2a30 msgr2=0x55b6a44b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe4037fe640 0 client.20085 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 shutdown_connections 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.891+0000 7fe42c845740 1 -- 172.21.15.16:0/2472064647 wait complete. 2024-09-12T03:07:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.892+0000 7fe42c845740 20 client.20085 trim_cache size 0 max 16384 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 Processor -- start 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 -- start start 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b3ab0 con 0x55b6a44bddb0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b55f0 con 0x55b6a44c1680 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe42c845740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b4180 con 0x55b6a44b6dc0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe428a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe428a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40560/0 (socket says 172.21.15.16:40560) 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe428a09640 1 -- 172.21.15.16:0/1657272278 learned_addr learned my addr 172.21.15.16:0/1657272278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe423fff640 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.918+0000 7fe41bfff640 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 -- 172.21.15.16:0/1657272278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 msgr2=0x55b6a44cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 -- 172.21.15.16:0/1657272278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 msgr2=0x55b6a44b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 -- 172.21.15.16:0/1657272278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a44bd270 con 0x55b6a44bddb0 2024-09-12T03:07:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe428a09640 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cf90 tx=0x7fe41c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c600ec5eb53c0a68 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe419ffb640 1 -- 172.21.15.16:0/1657272278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c09f070 con 0x55b6a44bddb0 2024-09-12T03:07:41.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe419ffb640 1 -- 172.21.15.16:0/1657272278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c09bdd0 con 0x55b6a44bddb0 2024-09-12T03:07:41.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.919+0000 7fe419ffb640 1 -- 172.21.15.16:0/1657272278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0a4810 con 0x55b6a44bddb0 2024-09-12T03:07:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 -- 172.21.15.16:0/1657272278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 msgr2=0x55b6a44d92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cf90 tx=0x7fe41c01a040 comp rx=0 tx=0).stop 2024-09-12T03:07:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 -- 172.21.15.16:0/1657272278 shutdown_connections 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44bddb0 0x55b6a44d92a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44c1680 0x55b6a44b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 --2- 172.21.15.16:0/1657272278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.920+0000 7fe42c845740 1 -- 172.21.15.16:0/1657272278 >> 172.21.15.16:0/1657272278 conn(0x55b6a44b2a30 msgr2=0x55b6a44b9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.921+0000 7fe42c845740 1 -- 172.21.15.16:0/1657272278 shutdown_connections 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.921+0000 7fe42c845740 1 -- 172.21.15.16:0/1657272278 wait complete. 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.921+0000 7fe42c845740 1 -- start start 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 Processor -- start 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 0x55b6a44c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b4f20 con 0x55b6a44c1680 2024-09-12T03:07:41.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b4850 con 0x55b6a44bddb0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b1a90 con 0x55b6a44b6dc0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe42c845740 10 client.0 _ll_register_callbacks cb 0x7fe414009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe423fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 0x55b6a44c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe428a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 0x55b6a44c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41680/0 (socket says 172.21.15.16:41680) 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe423fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:60936/0 (socket says 172.21.15.16:60936) 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe428a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40572/0 (socket says 172.21.15.16:40572) 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.922+0000 7fe41bfff640 1 -- 172.21.15.16:0/2682221776 learned_addr learned my addr 172.21.15.16:0/2682221776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 -- 172.21.15.16:0/2682221776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 msgr2=0x55b6a44c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 -- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 msgr2=0x55b6a44c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 0x55b6a44c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 -- 172.21.15.16:0/2682221776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a453b270 con 0x55b6a44c1680 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe423fff640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe428a09640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe41c09bad0 tx=0x7fe41c0a0180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b536eb597cb664 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe4037fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c07e020 con 0x55b6a44c1680 2024-09-12T03:07:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe41c0a47b0 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.923+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41c09bdd0 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe41c09f070 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 10 client.20091 fetch_fsmap learned FSMap version 39 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b6a453c4a0 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c085120 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe41c0b1070 con 0x55b6a44c1680 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 10 client.20091 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 10 client.20091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 20 client.20091 populate_metadata read hostname 'smithi016' 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.924+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a453cbc0 0x55b6a453efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.925+0000 7fe42c845740 4 client.20091 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.925+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe41c09bdd0 con 0x55b6a453cbc0 2024-09-12T03:07:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.925+0000 7fe423fff640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a453cbc0 0x55b6a453efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:07:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.925+0000 7fe423fff640 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a453cbc0 0x55b6a453efb0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2301f12a9cf91cfe server_cookie=cb6f6b9a52488b53 in_seq=0 out_seq=0 2024-09-12T03:07:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.925+0000 7fe4037fe640 10 client.20091 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.927+0000 7fe4037fe640 1 -- 172.21.15.16:0/2682221776 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fe41c09bdd0 con 0x55b6a453cbc0 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.927+0000 7fe4037fe640 10 client.20091 handle_command_reply: tid=0 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.927+0000 7fe42c845740 1 client.20091 shutdown 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 msgr2=0x55b6a44c5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe41c09bad0 tx=0x7fe41c0a0180 comp rx=0 tx=0).stop 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 shutdown_connections 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55b6a453cbc0 0x55b6a453efb0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6a44c1680 0x55b6a44c5d30 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe4037fe640 0 client.20091 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:07:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6a44bddb0 0x55b6a44c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 --2- 172.21.15.16:0/2682221776 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b6a44b6dc0 0x55b6a44c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 >> 172.21.15.16:0/2682221776 conn(0x55b6a44b2a30 msgr2=0x55b6a44bfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 shutdown_connections 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 1 -- 172.21.15.16:0/2682221776 wait complete. 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.928+0000 7fe42c845740 20 client.20091 trim_cache size 0 max 16384 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stdout: "d44d1a11-fc00-4634-b3f0-01644823b528": { 2024-09-12T03:07:41.930 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:07:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d44d1a11-fc00-4634-b3f0-01644823b528", 2024-09-12T03:07:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:07:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:07:41.931 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 msgr2=0x7fe4100a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 0x7fe4100a50c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009930 tx=0x7fe40c043000 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 msgr2=0x7fe4240fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00eb10 tx=0x7fe41c00c2d0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 shutdown_connections 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe424106a80 0x7fe4240fd7c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4100a2c00 0x7fe4100a50c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101420 0x7fe4240fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 --2- 172.21.15.16:0/3495396784 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe424100a80 0x7fe4240fcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 >> 172.21.15.16:0/3495396784 conn(0x7fe4240fc7a0 msgr2=0x7fe42406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 shutdown_connections 2024-09-12T03:07:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:07:41.929+0000 7fe42a46c640 1 -- 172.21.15.16:0/3495396784 wait complete. 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: pgmap v930: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 620 op/s 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:41.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:41 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 163499 23.38 MB/sec execute 283 sec latency 117.019 ms 2024-09-12T03:07:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:07:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: pgmap v930: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 620 op/s 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:41 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:07:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: pgmap v930: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 620 op/s 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:07:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:41 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:07:42.738 INFO:tasks.cephfs.filesystem:scrub status for tag:d44d1a11-fc00-4634-b3f0-01644823b528 - {'path': '/', 'tag': 'd44d1a11-fc00-4634-b3f0-01644823b528', 'options': 'recursive,force'} 2024-09-12T03:07:43.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 164327 23.47 MB/sec execute 284 sec latency 111.107 ms 2024-09-12T03:07:43.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:43 smithi033 ceph-mon[35092]: pgmap v931: 97 pgs: 97 active+clean; 711 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.1 MiB/s wr, 479 op/s 2024-09-12T03:07:44.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 164931 23.56 MB/sec execute 285 sec latency 89.964 ms 2024-09-12T03:07:44.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:43 smithi184 ceph-mon[35321]: pgmap v931: 97 pgs: 97 active+clean; 711 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.1 MiB/s wr, 479 op/s 2024-09-12T03:07:44.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:43 smithi016 ceph-mon[30925]: pgmap v931: 97 pgs: 97 active+clean; 711 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.1 MiB/s wr, 479 op/s 2024-09-12T03:07:45.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 165337 23.52 MB/sec execute 286 sec latency 110.606 ms 2024-09-12T03:07:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:45 smithi016 ceph-mon[30925]: pgmap v932: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-09-12T03:07:45.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:45 smithi033 ceph-mon[35092]: pgmap v932: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-09-12T03:07:46.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 166028 23.54 MB/sec execute 287 sec latency 77.337 ms 2024-09-12T03:07:46.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:45 smithi184 ceph-mon[35321]: pgmap v932: 97 pgs: 97 active+clean; 717 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-09-12T03:07:47.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 167024 23.64 MB/sec execute 288 sec latency 54.810 ms 2024-09-12T03:07:47.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:47 smithi033 ceph-mon[35092]: pgmap v933: 97 pgs: 97 active+clean; 725 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 532 op/s 2024-09-12T03:07:48.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 167669 23.69 MB/sec execute 289 sec latency 41.623 ms 2024-09-12T03:07:48.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:47 smithi184 ceph-mon[35321]: pgmap v933: 97 pgs: 97 active+clean; 725 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 532 op/s 2024-09-12T03:07:48.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:47 smithi016 ceph-mon[30925]: pgmap v933: 97 pgs: 97 active+clean; 725 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 532 op/s 2024-09-12T03:07:49.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 168400 23.76 MB/sec execute 290 sec latency 37.031 ms 2024-09-12T03:07:49.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:49 smithi033 ceph-mon[35092]: pgmap v934: 97 pgs: 97 active+clean; 726 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 457 op/s 2024-09-12T03:07:50.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 169138 23.81 MB/sec execute 291 sec latency 58.477 ms 2024-09-12T03:07:50.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:49 smithi184 ceph-mon[35321]: pgmap v934: 97 pgs: 97 active+clean; 726 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 457 op/s 2024-09-12T03:07:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:49 smithi016 ceph-mon[30925]: pgmap v934: 97 pgs: 97 active+clean; 726 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 457 op/s 2024-09-12T03:07:51.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 169655 23.85 MB/sec execute 292 sec latency 77.471 ms 2024-09-12T03:07:51.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:51 smithi033 ceph-mon[35092]: pgmap v935: 97 pgs: 97 active+clean; 739 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 608 op/s 2024-09-12T03:07:52.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 170231 23.84 MB/sec execute 293 sec latency 56.780 ms 2024-09-12T03:07:52.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:51 smithi184 ceph-mon[35321]: pgmap v935: 97 pgs: 97 active+clean; 739 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 608 op/s 2024-09-12T03:07:52.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:51 smithi016 ceph-mon[30925]: pgmap v935: 97 pgs: 97 active+clean; 739 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 608 op/s 2024-09-12T03:07:53.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 171062 23.90 MB/sec execute 294 sec latency 39.630 ms 2024-09-12T03:07:53.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: pgmap v936: 97 pgs: 97 active+clean; 742 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.6 MiB/s wr, 480 op/s 2024-09-12T03:07:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:53.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:53 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:07:54.025 INFO:tasks.workunit.client.0.smithi016.stdout: 10 171956 23.99 MB/sec execute 295 sec latency 58.280 ms 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: pgmap v936: 97 pgs: 97 active+clean; 742 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.6 MiB/s wr, 480 op/s 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:54.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:53 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:07:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: pgmap v936: 97 pgs: 97 active+clean; 742 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.6 MiB/s wr, 480 op/s 2024-09-12T03:07:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:53 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:07:55.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 172596 24.06 MB/sec execute 296 sec latency 136.648 ms 2024-09-12T03:07:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:55 smithi033 ceph-mon[35092]: pgmap v937: 97 pgs: 97 active+clean; 746 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 479 op/s 2024-09-12T03:07:56.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 173113 24.08 MB/sec execute 297 sec latency 56.702 ms 2024-09-12T03:07:56.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:55 smithi184 ceph-mon[35321]: pgmap v937: 97 pgs: 97 active+clean; 746 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 479 op/s 2024-09-12T03:07:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:55 smithi016 ceph-mon[30925]: pgmap v937: 97 pgs: 97 active+clean; 746 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 479 op/s 2024-09-12T03:07:57.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 173680 24.08 MB/sec execute 298 sec latency 64.977 ms 2024-09-12T03:07:57.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:57 smithi033 ceph-mon[35092]: pgmap v938: 97 pgs: 97 active+clean; 760 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-09-12T03:07:58.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 174336 24.11 MB/sec execute 299 sec latency 72.587 ms 2024-09-12T03:07:58.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:57 smithi184 ceph-mon[35321]: pgmap v938: 97 pgs: 97 active+clean; 760 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-09-12T03:07:58.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:57 smithi016 ceph-mon[30925]: pgmap v938: 97 pgs: 97 active+clean; 760 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-09-12T03:07:59.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 175039 24.15 MB/sec execute 300 sec latency 160.479 ms 2024-09-12T03:07:59.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:07:59 smithi033 ceph-mon[35092]: pgmap v939: 97 pgs: 97 active+clean; 761 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 479 op/s 2024-09-12T03:08:00.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 175546 24.16 MB/sec execute 301 sec latency 234.721 ms 2024-09-12T03:08:00.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:07:59 smithi184 ceph-mon[35321]: pgmap v939: 97 pgs: 97 active+clean; 761 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 479 op/s 2024-09-12T03:08:00.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:07:59 smithi016 ceph-mon[30925]: pgmap v939: 97 pgs: 97 active+clean; 761 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 479 op/s 2024-09-12T03:08:01.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 176114 24.17 MB/sec execute 302 sec latency 254.011 ms 2024-09-12T03:08:01.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:01 smithi033 ceph-mon[35092]: pgmap v940: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.1 MiB/s wr, 598 op/s 2024-09-12T03:08:02.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 176961 24.26 MB/sec execute 303 sec latency 63.005 ms 2024-09-12T03:08:02.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:01 smithi184 ceph-mon[35321]: pgmap v940: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.1 MiB/s wr, 598 op/s 2024-09-12T03:08:02.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:01 smithi016 ceph-mon[30925]: pgmap v940: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.1 MiB/s wr, 598 op/s 2024-09-12T03:08:03.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 177589 24.32 MB/sec execute 304 sec latency 52.477 ms 2024-09-12T03:08:03.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:03 smithi033 ceph-mon[35092]: pgmap v941: 97 pgs: 97 active+clean; 774 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 562 op/s 2024-09-12T03:08:04.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 178003 24.31 MB/sec execute 305 sec latency 58.491 ms 2024-09-12T03:08:04.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:03 smithi184 ceph-mon[35321]: pgmap v941: 97 pgs: 97 active+clean; 774 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 562 op/s 2024-09-12T03:08:04.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:03 smithi016 ceph-mon[30925]: pgmap v941: 97 pgs: 97 active+clean; 774 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 562 op/s 2024-09-12T03:08:05.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 178511 24.28 MB/sec execute 306 sec latency 167.318 ms 2024-09-12T03:08:05.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:05 smithi033 ceph-mon[35092]: pgmap v942: 97 pgs: 97 active+clean; 762 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.9 MiB/s wr, 570 op/s 2024-09-12T03:08:06.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 179089 24.29 MB/sec execute 307 sec latency 256.317 ms 2024-09-12T03:08:06.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:05 smithi184 ceph-mon[35321]: pgmap v942: 97 pgs: 97 active+clean; 762 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.9 MiB/s wr, 570 op/s 2024-09-12T03:08:06.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:05 smithi016 ceph-mon[30925]: pgmap v942: 97 pgs: 97 active+clean; 762 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.9 MiB/s wr, 570 op/s 2024-09-12T03:08:07.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 179880 24.36 MB/sec execute 308 sec latency 96.836 ms 2024-09-12T03:08:07.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:07 smithi033 ceph-mon[35092]: pgmap v943: 97 pgs: 97 active+clean; 759 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.4 MiB/s wr, 687 op/s 2024-09-12T03:08:08.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 180609 24.41 MB/sec execute 309 sec latency 36.120 ms 2024-09-12T03:08:08.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:07 smithi184 ceph-mon[35321]: pgmap v943: 97 pgs: 97 active+clean; 759 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.4 MiB/s wr, 687 op/s 2024-09-12T03:08:08.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:07 smithi016 ceph-mon[30925]: pgmap v943: 97 pgs: 97 active+clean; 759 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.4 MiB/s wr, 687 op/s 2024-09-12T03:08:09.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 181521 24.51 MB/sec execute 310 sec latency 81.592 ms 2024-09-12T03:08:09.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:09 smithi033 ceph-mon[35092]: pgmap v944: 97 pgs: 97 active+clean; 758 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 567 op/s 2024-09-12T03:08:10.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 182149 24.57 MB/sec execute 311 sec latency 61.807 ms 2024-09-12T03:08:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:09 smithi184 ceph-mon[35321]: pgmap v944: 97 pgs: 97 active+clean; 758 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 567 op/s 2024-09-12T03:08:10.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:09 smithi016 ceph-mon[30925]: pgmap v944: 97 pgs: 97 active+clean; 758 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 567 op/s 2024-09-12T03:08:11.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 182477 24.54 MB/sec execute 312 sec latency 132.700 ms 2024-09-12T03:08:11.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:11 smithi033 ceph-mon[35092]: pgmap v945: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:08:12.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 183011 24.52 MB/sec execute 313 sec latency 55.600 ms 2024-09-12T03:08:12.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:11 smithi184 ceph-mon[35321]: pgmap v945: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:08:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:11 smithi016 ceph-mon[30925]: pgmap v945: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 735 op/s 2024-09-12T03:08:12.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:08:12.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:12 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:08:12.987 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:12 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:08:13.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 183814 24.56 MB/sec execute 314 sec latency 67.976 ms 2024-09-12T03:08:13.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:12 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:08:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:13.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:13 smithi033 ceph-mon[35092]: pgmap v946: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 662 op/s 2024-09-12T03:08:14.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 184531 24.61 MB/sec execute 315 sec latency 74.871 ms 2024-09-12T03:08:14.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:13 smithi016 ceph-mon[30925]: pgmap v946: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 662 op/s 2024-09-12T03:08:14.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:13 smithi184 ceph-mon[35321]: pgmap v946: 97 pgs: 97 active+clean; 768 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 662 op/s 2024-09-12T03:08:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 -- 172.21.15.16:0/1035134346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 msgr2=0x7fd60810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 --2- 172.21.15.16:0/1035134346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60810da40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd600009ca0 tx=0x7fd60002a950 comp rx=0 tx=0).stop 2024-09-12T03:08:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 -- 172.21.15.16:0/1035134346 shutdown_connections 2024-09-12T03:08:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 --2- 172.21.15.16:0/1035134346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60810da40 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 --2- 172.21.15.16:0/1035134346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608102570 0x7fd6081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 --2- 172.21.15.16:0/1035134346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608100e20 0x7fd608101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 -- 172.21.15.16:0/1035134346 >> 172.21.15.16:0/1035134346 conn(0x7fd6080fc7d0 msgr2=0x7fd6080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 -- 172.21.15.16:0/1035134346 shutdown_connections 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.137+0000 7fd60f081640 1 -- 172.21.15.16:0/1035134346 wait complete. 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 Processor -- start 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 -- start start 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608100e20 0x7fd608197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608102570 0x7fd6081978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd608110f80 con 0x7fd608102ef0 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd608110e00 con 0x7fd608102570 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60f081640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd608111100 con 0x7fd608100e20 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47418/0 (socket says 172.21.15.16:47418) 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 -- 172.21.15.16:0/1005928118 learned_addr learned my addr 172.21.15.16:0/1005928118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608100e20 0x7fd608197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd607fff640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608102570 0x7fd6081978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 -- 172.21.15.16:0/1005928118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608100e20 msgr2=0x7fd608197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608100e20 0x7fd608197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.138+0000 7fd60d61e640 1 -- 172.21.15.16:0/1005928118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608102570 msgr2=0x7fd6081978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd60d61e640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608102570 0x7fd6081978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd60d61e640 1 -- 172.21.15.16:0/1005928118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60819c310 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd60d61e640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd600004af0 tx=0x7fd60002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd605ffb640 1 -- 172.21.15.16:0/1005928118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600034040 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd605ffb640 1 -- 172.21.15.16:0/1005928118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd600030920 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60819c5a0 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd605ffb640 1 -- 172.21.15.16:0/1005928118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600004820 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.139+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6081a3ee0 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.141+0000 7fd605ffb640 1 -- 172.21.15.16:0/1005928118 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd600037020 con 0x7fd608102ef0 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 Processor -- start 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 -- start start 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becfdc0 0x55663beda140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bedca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55663be352e0 con 0x55663beda680 2024-09-12T03:08:14.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55663bddcce0 con 0x55663becf4a0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd61145a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55663be312a0 con 0x55663becfdc0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd605ffb640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 0x7fd5f80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd607fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd607fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40586/0 (socket says 172.21.15.16:40586) 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd607fff640 1 -- 172.21.15.16:0/1527074143 learned_addr learned my addr 172.21.15.16:0/1527074143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd60ce1d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becfdc0 0x55663beda140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd607fff640 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bedca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.142+0000 7fd60d61e640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 0x7fd5f80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 -- 172.21.15.16:0/1527074143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becfdc0 msgr2=0x55663beda140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becfdc0 0x55663beda140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 -- 172.21.15.16:0/1527074143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 msgr2=0x55663bedca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bedca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 -- 172.21.15.16:0/1527074143 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55663bdc1440 con 0x55663becf4a0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd607fff640 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400b6c0 tx=0x7fd5f40104f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2590775848d53144 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd605ffb640 1 -- 172.21.15.16:0/1005928118 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6000efe20 con 0x7fd608102ef0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd60d61e640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 0x7fd5f80a5040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd6000943d0 tx=0x7fd60008c540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd5f2ffd640 1 -- 172.21.15.16:0/1527074143 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40120d0 con 0x55663becf4a0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd5f2ffd640 1 -- 172.21.15.16:0/1527074143 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f4012270 con 0x55663becf4a0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.143+0000 7fd5f2ffd640 1 -- 172.21.15.16:0/1527074143 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400dde0 con 0x55663becf4a0 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- 172.21.15.16:0/1527074143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 msgr2=0x55663becf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400b6c0 tx=0x7fd5f40104f0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- 172.21.15.16:0/1527074143 shutdown_connections 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bedca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becfdc0 0x55663beda140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 --2- 172.21.15.16:0/1527074143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becf4a0 0x55663becf880 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- 172.21.15.16:0/1527074143 >> 172.21.15.16:0/1527074143 conn(0x55663becba30 msgr2=0x55663becbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- 172.21.15.16:0/1527074143 shutdown_connections 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- 172.21.15.16:0/1527074143 wait complete. 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 -- start start 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.144+0000 7fd61145a740 1 Processor -- start 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bed72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663beda680 0x55663bed7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55663bedd250 con 0x55663becf4a0 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55663bdf0050 con 0x55663becfdc0 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55663bedd3d0 con 0x55663beda680 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd61145a740 10 client.0 _ll_register_callbacks cb 0x55663be30160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47444/0 (socket says 172.21.15.16:47444) 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 -- 172.21.15.16:0/2799432439 learned_addr learned my addr 172.21.15.16:0/2799432439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663beda680 0x55663bed7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bed72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 -- 172.21.15.16:0/2799432439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663beda680 msgr2=0x55663bed7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663beda680 0x55663bed7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 -- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 msgr2=0x55663bed72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bed72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.145+0000 7fd607fff640 1 -- 172.21.15.16:0/2799432439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55663bba63c0 con 0x55663becf4a0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd607fff640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400b510 tx=0x7fd5f400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b80196fc036fc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4016030 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5f4012270 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd5f4017070 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5f400d720 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 10 client.20103 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55663bee8c40 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f4012270 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd5f402f4d0 con 0x55663becf4a0 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 10 client.20103 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 10 client.20103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.146+0000 7fd61145a740 20 client.20103 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663beec380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd61145a740 4 client.20103 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55663be42850 con 0x55663bee9f90 2024-09-12T03:08:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663beec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663beec380 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8666af4765976a3 server_cookie=267d60edc8d837ee in_seq=0 out_seq=0 2024-09-12T03:08:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.147+0000 7fd5e7fff640 10 client.20103 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd5e7fff640 1 -- 172.21.15.16:0/2799432439 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55663be42850 con 0x55663bee9f90 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd5e7fff640 10 client.20103 handle_command_reply: tid=0 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 client.20103 shutdown 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 msgr2=0x55663beda000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400b510 tx=0x7fd5f400d9e0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 shutdown_connections 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663beec380 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd5e7fff640 0 client.20103 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663beda680 0x55663bed7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bed72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 --2- 172.21.15.16:0/2799432439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becf4a0 0x55663beda000 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 >> 172.21.15.16:0/2799432439 conn(0x55663becba30 msgr2=0x55663bed1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 shutdown_connections 2024-09-12T03:08:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 1 -- 172.21.15.16:0/2799432439 wait complete. 2024-09-12T03:08:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.148+0000 7fd61145a740 20 client.20103 trim_cache size 0 max 16384 2024-09-12T03:08:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.172+0000 7fd61145a740 1 Processor -- start 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.172+0000 7fd61145a740 1 -- start start 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bef1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bef2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bed4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081ab1a0 con 0x55663beda680 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081acce0 con 0x55663becfdc0 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd61145a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081ab870 con 0x55663becf4a0 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bef1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bef1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47082/0 (socket says 172.21.15.16:47082) 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 -- 172.21.15.16:0/3221635144 learned_addr learned my addr 172.21.15.16:0/3221635144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bed4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bef2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 -- 172.21.15.16:0/3221635144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 msgr2=0x55663bef1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bef1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 -- 172.21.15.16:0/3221635144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 msgr2=0x55663bef2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.173+0000 7fd607fff640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bef2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd607fff640 1 -- 172.21.15.16:0/3221635144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55663bed6260 con 0x55663beda680 2024-09-12T03:08:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bef2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd607fff640 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bed4510 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4005d00 tx=0x7fd5f400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35053728febfba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd5f27fc640 1 -- 172.21.15.16:0/3221635144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4016040 con 0x55663beda680 2024-09-12T03:08:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd5f27fc640 1 -- 172.21.15.16:0/3221635144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f400b540 con 0x55663beda680 2024-09-12T03:08:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.174+0000 7fd5f27fc640 1 -- 172.21.15.16:0/3221635144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f401f460 con 0x55663beda680 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.175+0000 7fd61145a740 1 -- 172.21.15.16:0/3221635144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 msgr2=0x55663bed4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.175+0000 7fd61145a740 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bed4510 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4005d00 tx=0x7fd5f400d6b0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.175+0000 7fd61145a740 1 -- 172.21.15.16:0/3221635144 shutdown_connections 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663beda680 0x55663bed4510 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663becfdc0 0x55663bef2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 --2- 172.21.15.16:0/3221635144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bef1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 -- 172.21.15.16:0/3221635144 >> 172.21.15.16:0/3221635144 conn(0x55663becba30 msgr2=0x55663becab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 -- 172.21.15.16:0/3221635144 shutdown_connections 2024-09-12T03:08:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 -- 172.21.15.16:0/3221635144 wait complete. 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 -- start start 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.176+0000 7fd61145a740 1 Processor -- start 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bee2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 0x55663bee00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663beda680 0x55663bee0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd607fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bee2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd607fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bee2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47098/0 (socket says 172.21.15.16:47098) 2024-09-12T03:08:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081ac610 con 0x55663becfdc0 2024-09-12T03:08:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081abf40 con 0x55663beda680 2024-09-12T03:08:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.177+0000 7fd61145a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081a91a0 con 0x55663becf4a0 2024-09-12T03:08:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.178+0000 7fd61145a740 10 client.0 _ll_register_callbacks cb 0x7fd5f400bd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.178+0000 7fd607fff640 1 -- 172.21.15.16:0/1525027573 learned_addr learned my addr 172.21.15.16:0/1525027573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.178+0000 7fd607fff640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663beda680 0x55663bee0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.178+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 0x55663bee00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.179+0000 7fd60ce1d640 1 -- 172.21.15.16:0/1525027573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 msgr2=0x55663bee2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.179+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bee2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.179+0000 7fd60ce1d640 1 -- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663beda680 msgr2=0x55663bee0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.179+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663beda680 0x55663bee0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.179+0000 7fd60ce1d640 1 -- 172.21.15.16:0/1525027573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55663bee0e70 con 0x55663becfdc0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd60ce1d640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 0x55663bee00e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x55663beeca60 tx=0x7fd5fc013f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425707fe36409411 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd5e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc013430 con 0x55663becfdc0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd5fc0135d0 con 0x55663becfdc0 2024-09-12T03:08:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5f400b540 con 0x55663becfdc0 2024-09-12T03:08:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.180+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd5fc0097a0 con 0x55663becfdc0 2024-09-12T03:08:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.181+0000 7fd61145a740 10 client.20109 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.181+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55663bf54ae0 con 0x55663becfdc0 2024-09-12T03:08:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.181+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f400b540 con 0x55663becfdc0 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd5fc015040 con 0x55663becfdc0 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 10 client.20109 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 10 client.20109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 20 client.20109 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663bf57220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 4 client.20109 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:08:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55663be3cd60 con 0x55663bee9f90 2024-09-12T03:08:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.182+0000 7fd607fff640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663bf57220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd5e7fff640 10 client.20109 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd607fff640 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663bf57220 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=873cdf82bdfab97c server_cookie=74fa2c1454e0fba9 in_seq=0 out_seq=0 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd5e7fff640 1 -- 172.21.15.16:0/1525027573 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55663be3cd60 con 0x55663bee9f90 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd5e7fff640 10 client.20109 handle_command_reply: tid=0 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd61145a740 1 client.20109 shutdown 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 msgr2=0x55663bee00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.183+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 0x55663bee00e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x55663beeca60 tx=0x7fd5fc013f80 comp rx=0 tx=0).stop 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 shutdown_connections 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55663bee9f90 0x55663bf57220 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55663beda680 0x55663bee0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55663becfdc0 0x55663bee00e0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 --2- 172.21.15.16:0/1525027573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55663becf4a0 0x55663bee2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd5e7fff640 0 client.20109 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 >> 172.21.15.16:0/1525027573 conn(0x55663becba30 msgr2=0x55663bed8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 shutdown_connections 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 1 -- 172.21.15.16:0/1525027573 wait complete. 2024-09-12T03:08:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.184+0000 7fd61145a740 20 client.20109 trim_cache size 0 max 16384 2024-09-12T03:08:14.187 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:08:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:08:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:08:14.187 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:08:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 msgr2=0x7fd5f80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 0x7fd5f80a5040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd6000943d0 tx=0x7fd60008c540 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 msgr2=0x7fd60819bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd600004af0 tx=0x7fd60002f5b0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 shutdown_connections 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd5f80a2b80 0x7fd5f80a5040 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd608102ef0 0x7fd60819bc30 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd608102570 0x7fd6081978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 --2- 172.21.15.16:0/1005928118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd608100e20 0x7fd608197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 >> 172.21.15.16:0/1005928118 conn(0x7fd6080fc7d0 msgr2=0x7fd608106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.185+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 shutdown_connections 2024-09-12T03:08:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:14.186+0000 7fd60f081640 1 -- 172.21.15.16:0/1005928118 wait complete. 2024-09-12T03:08:15.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 184998 24.61 MB/sec execute 316 sec latency 131.716 ms 2024-09-12T03:08:15.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:14 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:14 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:14 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:15.270 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:08:15.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:08:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:16.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 185509 24.63 MB/sec execute 317 sec latency 242.593 ms 2024-09-12T03:08:16.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:15 smithi016 ceph-mon[30925]: pgmap v947: 97 pgs: 97 active+clean; 771 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.4 MiB/s wr, 606 op/s 2024-09-12T03:08:16.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:15 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 -- 172.21.15.16:0/2943328683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808106f00 msgr2=0x7f180810ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 --2- 172.21.15.16:0/2943328683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808106f00 0x7f180810ea80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1800009ca0 tx=0x7f180002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 -- 172.21.15.16:0/2943328683 shutdown_connections 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 --2- 172.21.15.16:0/2943328683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808106f00 0x7f180810ea80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 --2- 172.21.15.16:0/2943328683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f18081067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 --2- 172.21.15.16:0/2943328683 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808101dd0 0x7f18081021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 -- 172.21.15.16:0/2943328683 >> 172.21.15.16:0/2943328683 conn(0x7f18080fd8b0 msgr2=0x7f18080ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 -- 172.21.15.16:0/2943328683 shutdown_connections 2024-09-12T03:08:16.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.203+0000 7f180cbbc640 1 -- 172.21.15.16:0/2943328683 wait complete. 2024-09-12T03:08:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.205+0000 7f180cbbc640 1 Processor -- start 2024-09-12T03:08:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.205+0000 7f180cbbc640 1 -- start start 2024-09-12T03:08:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.205+0000 7f180cbbc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180cbbc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f1808198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:16.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180cbbc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180cbbc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1808111f60 con 0x7f1808101dd0 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180cbbc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1808111de0 con 0x7f1808102780 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180cbbc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18081120e0 con 0x7f1808106f00 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51100/0 (socket says 172.21.15.16:51100) 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58730/0 (socket says 172.21.15.16:58730) 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f180659c640 1 -- 172.21.15.16:0/618571061 learned_addr learned my addr 172.21.15.16:0/618571061 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1805d9b640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f1808198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:16.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1806d9d640 1 -- 172.21.15.16:0/618571061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 msgr2=0x7f1808198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1806d9d640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f1808198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.206+0000 7f1806d9d640 1 -- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 msgr2=0x7f1808198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.207+0000 7f1806d9d640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.207+0000 7f1806d9d640 1 -- 172.21.15.16:0/618571061 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180819d3f0 con 0x7f1808106f00 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.207+0000 7f1805d9b640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f1808198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:16.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.207+0000 7f180659c640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f1806d9d640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f180002ade0 tx=0x7f18000319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800003b70 con 0x7f1808106f00 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180819d680 con 0x7f1808106f00 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1800003d10 con 0x7f1808106f00 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081a4fc0 con 0x7f1808106f00 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.208+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180002fc40 con 0x7f1808106f00 2024-09-12T03:08:16.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.209+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e8005290 con 0x7f1808106f00 2024-09-12T03:08:16.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.211+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1800004050 con 0x7f1808106f00 2024-09-12T03:08:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.212+0000 7f17f77fe640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 0x7f17f00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.212+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f18000f00f0 con 0x7f1808106f00 2024-09-12T03:08:16.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.213+0000 7f180659c640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 0x7f17f00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:16.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.214+0000 7f180659c640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 0x7f17f00a50c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f17f80095f0 tx=0x7f17f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.218+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18000bd060 con 0x7f1808106f00 2024-09-12T03:08:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:15 smithi033 ceph-mon[35092]: pgmap v947: 97 pgs: 97 active+clean; 771 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.4 MiB/s wr, 606 op/s 2024-09-12T03:08:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:15 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:15 smithi184 ceph-mon[35321]: pgmap v947: 97 pgs: 97 active+clean; 771 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.4 MiB/s wr, 606 op/s 2024-09-12T03:08:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:15 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.501+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17e8005580 con 0x7f1808106f00 2024-09-12T03:08:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.502+0000 7f17f77fe640 1 -- 172.21.15.16:0/618571061 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f18000bc960 con 0x7f1808106f00 2024-09-12T03:08:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:08:16.504 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:08:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:08:16.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 msgr2=0x7f17f00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:16.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 0x7f17f00a50c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f17f80095f0 tx=0x7f17f8005f90 comp rx=0 tx=0).stop 2024-09-12T03:08:16.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 msgr2=0x7f180819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:16.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f180002ade0 tx=0x7f18000319f0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 shutdown_connections 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1808106f00 0x7f180819cd10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1808102780 0x7f1808198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f17f00a2c00 0x7f17f00a50c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 --2- 172.21.15.16:0/618571061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1808101dd0 0x7f1808198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 >> 172.21.15.16:0/618571061 conn(0x7f18080fd8b0 msgr2=0x7f18080fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 shutdown_connections 2024-09-12T03:08:16.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:16.505+0000 7f180cbbc640 1 -- 172.21.15.16:0/618571061 wait complete. 2024-09-12T03:08:17.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 186313 24.70 MB/sec execute 318 sec latency 60.806 ms 2024-09-12T03:08:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:08:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:08:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:17.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:16 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/618571061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:17.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:08:17.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:08:17.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:08:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:17.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:16 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/618571061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:08:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:08:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:08:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:16 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/618571061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:17 smithi016 ceph-mon[30925]: pgmap v948: 97 pgs: 97 active+clean; 779 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.2 MiB/s wr, 694 op/s 2024-09-12T03:08:18.026 INFO:tasks.workunit.client.0.smithi016.stdout: 10 186853 24.75 MB/sec execute 319 sec latency 75.752 ms 2024-09-12T03:08:18.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 -- 172.21.15.16:0/2134256592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2410bfc0 msgr2=0x7fad2410e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 --2- 172.21.15.16:0/2134256592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2410bfc0 0x7fad2410e460 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009be0 tx=0x7fad1c02a750 comp rx=0 tx=0).stop 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 -- 172.21.15.16:0/2134256592 shutdown_connections 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 --2- 172.21.15.16:0/2134256592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2410bfc0 0x7fad2410e460 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 --2- 172.21.15.16:0/2134256592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad241034b0 0x7fad2410b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 --2- 172.21.15.16:0/2134256592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad24102b90 0x7fad24102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.036+0000 7fad2af12640 1 -- 172.21.15.16:0/2134256592 >> 172.21.15.16:0/2134256592 conn(0x7fad240fc610 msgr2=0x7fad240fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.037+0000 7fad2af12640 1 -- 172.21.15.16:0/2134256592 shutdown_connections 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.037+0000 7fad2af12640 1 -- 172.21.15.16:0/2134256592 wait complete. 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.037+0000 7fad2af12640 1 Processor -- start 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.037+0000 7fad2af12640 1 -- start start 2024-09-12T03:08:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.037+0000 7fad2af12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2af12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad241034b0 0x7fad24198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2af12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410bfc0 0x7fad2419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2af12640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111d40 con 0x7fad241034b0 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2af12640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111bc0 con 0x7fad24102b90 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2af12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111ec0 con 0x7fad2410bfc0 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2970f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad241034b0 0x7fad24198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52846/0 (socket says 172.21.15.16:52846) 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 -- 172.21.15.16:0/572431237 learned_addr learned my addr 172.21.15.16:0/572431237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad2a711640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410bfc0 0x7fad2419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 -- 172.21.15.16:0/572431237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410bfc0 msgr2=0x7fad2419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410bfc0 0x7fad2419caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 -- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad241034b0 msgr2=0x7fad24198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad241034b0 0x7fad24198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 -- 172.21.15.16:0/572431237 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2419d1d0 con 0x7fad24102b90 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad29f10640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9c0 tx=0x7fad2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.038+0000 7fad1affd640 1 -- 172.21.15.16:0/572431237 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000ccc0 con 0x7fad24102b90 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.039+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2419d4c0 con 0x7fad24102b90 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.039+0000 7fad1affd640 1 -- 172.21.15.16:0/572431237 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad20005500 con 0x7fad24102b90 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.039+0000 7fad1affd640 1 -- 172.21.15.16:0/572431237 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20010430 con 0x7fad24102b90 2024-09-12T03:08:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.039+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241a4da0 con 0x7fad24102b90 2024-09-12T03:08:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 Processor -- start 2024-09-12T03:08:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 -- start start 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad1affd640 1 -- 172.21.15.16:0/572431237 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fad200046c0 con 0x7fad24102b90 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771bc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624771152f0 con 0x5624771af7b0 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5624770bccf0 con 0x5624771b0d20 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.042+0000 7fad2e54d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624771112b0 con 0x5624771ba550 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad29f10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52854/0 (socket says 172.21.15.16:52854) 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 -- 172.21.15.16:0/3240945257 learned_addr learned my addr 172.21.15.16:0/3240945257 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2970f640 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 -- 172.21.15.16:0/3240945257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 msgr2=0x5624771bc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771bc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 -- 172.21.15.16:0/3240945257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 msgr2=0x5624771afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 -- 172.21.15.16:0/3240945257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624770a13a0 con 0x5624771b0d20 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad29f10640 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad2a711640 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005c40 tx=0x7fad1c005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa77d5d3b94395f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.043+0000 7fad1affd640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 0x7fad100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad29f10640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 0x7fad100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad1affd640 1 -- 172.21.15.16:0/572431237 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad200cf320 con 0x7fad24102b90 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad0ffff640 1 -- 172.21.15.16:0/3240945257 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c042030 con 0x5624771b0d20 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad0ffff640 1 -- 172.21.15.16:0/3240945257 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c037040 con 0x5624771b0d20 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad0ffff640 1 -- 172.21.15.16:0/3240945257 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c036070 con 0x5624771b0d20 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad2e54d740 1 -- 172.21.15.16:0/3240945257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 msgr2=0x5624771af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad29f10640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 0x7fad100a5040 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad200cd1a0 tx=0x7fad200d2e00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.044+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005c40 tx=0x7fad1c005c70 comp rx=0 tx=0).stop 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 -- 172.21.15.16:0/3240945257 shutdown_connections 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771bc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3240945257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771af270 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 -- 172.21.15.16:0/3240945257 >> 172.21.15.16:0/3240945257 conn(0x562477178310 msgr2=0x5624771abe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 -- 172.21.15.16:0/3240945257 shutdown_connections 2024-09-12T03:08:18.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 -- 172.21.15.16:0/3240945257 wait complete. 2024-09-12T03:08:18.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 -- start start 2024-09-12T03:08:18.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.045+0000 7fad2e54d740 1 Processor -- start 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771b7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624771bd1f0 con 0x5624771af7b0 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5624770d0060 con 0x5624771b0d20 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624771bd370 con 0x5624771ba550 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2e54d740 10 client.0 _ll_register_callbacks cb 0x562476e238c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2a711640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2a711640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51134/0 (socket says 172.21.15.16:51134) 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2a711640 1 -- 172.21.15.16:0/187061266 learned_addr learned my addr 172.21.15.16:0/187061266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2a711640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 -- 172.21.15.16:0/187061266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 msgr2=0x5624771b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771b7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 -- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 msgr2=0x5624771b9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2970f640 1 -- 172.21.15.16:0/187061266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562476e863c0 con 0x5624771b0d20 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.046+0000 7fad2a711640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad2970f640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b71b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fad240fec80 tx=0x7fad14012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382a654b6e17a73c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004030 con 0x5624771b0d20 2024-09-12T03:08:18.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad140041d0 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c037040 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fad140043d0 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad2e54d740 10 client.29354 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5624771cd0e0 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c037040 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fad14003a80 con 0x5624771b0d20 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.047+0000 7fad2e54d740 10 client.29354 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2e54d740 10 client.29354 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2e54d740 20 client.29354 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5624771ce5f0 0x5624771d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2e54d740 4 client.29354 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562477122860 con 0x5624771ce5f0 2024-09-12T03:08:18.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2a711640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5624771ce5f0 0x5624771d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad2a711640 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5624771ce5f0 0x5624771d0a10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a39127310f061c22 server_cookie=ff4045c92c4e20b5 in_seq=0 out_seq=0 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.048+0000 7fad0d7fa640 10 client.29354 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad0d7fa640 1 -- 172.21.15.16:0/187061266 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562477122860 con 0x5624771ce5f0 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad0d7fa640 10 client.29354 handle_command_reply: tid=0 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 client.29354 shutdown 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 msgr2=0x5624771b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b71b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fad240fec80 tx=0x7fad14012b60 comp rx=0 tx=0).stop 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 shutdown_connections 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771ba550 0x5624771b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5624771ce5f0 0x5624771d0a10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b71b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 --2- 172.21.15.16:0/187061266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771af7b0 0x5624771b9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad0d7fa640 0 client.29354 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:08:18.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 >> 172.21.15.16:0/187061266 conn(0x562477178310 msgr2=0x5624771b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 shutdown_connections 2024-09-12T03:08:18.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 1 -- 172.21.15.16:0/187061266 wait complete. 2024-09-12T03:08:18.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.049+0000 7fad2e54d740 20 client.29354 trim_cache size 0 max 16384 2024-09-12T03:08:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 Processor -- start 2024-09-12T03:08:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 -- start start 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ab120 con 0x5624771ba550 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad241acc60 con 0x5624771b0d20 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2e54d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ab7f0 con 0x5624771af7b0 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2a711640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2a711640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51146/0 (socket says 172.21.15.16:51146) 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2a711640 1 -- 172.21.15.16:0/3968782706 learned_addr learned my addr 172.21.15.16:0/3968782706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2a711640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.065+0000 7fad2970f640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 -- 172.21.15.16:0/3968782706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 msgr2=0x5624771b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 -- 172.21.15.16:0/3968782706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 msgr2=0x5624771b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 -- 172.21.15.16:0/3968782706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624771af4b0 con 0x5624771ba550 2024-09-12T03:08:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2970f640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad2a711640 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000f80 tx=0x7fad1c040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbf3069630b5365 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad0f7fe640 1 -- 172.21.15.16:0/3968782706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c042040 con 0x5624771ba550 2024-09-12T03:08:18.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad0f7fe640 1 -- 172.21.15.16:0/3968782706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c043040 con 0x5624771ba550 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.066+0000 7fad0f7fe640 1 -- 172.21.15.16:0/3968782706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c036040 con 0x5624771ba550 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- 172.21.15.16:0/3968782706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 msgr2=0x5624771b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000f80 tx=0x7fad1c040040 comp rx=0 tx=0).stop 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- 172.21.15.16:0/3968782706 shutdown_connections 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771ba550 0x5624771b2300 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771b0d20 0x5624771b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 --2- 172.21.15.16:0/3968782706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- 172.21.15.16:0/3968782706 >> 172.21.15.16:0/3968782706 conn(0x562477178310 msgr2=0x5624771b8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- 172.21.15.16:0/3968782706 shutdown_connections 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- 172.21.15.16:0/3968782706 wait complete. 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 -- start start 2024-09-12T03:08:18.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.067+0000 7fad2e54d740 1 Processor -- start 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 0x5624771b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ac590 con 0x5624771b0d20 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad241abec0 con 0x5624771ba550 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad241a9100 con 0x5624771af7b0 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2e54d740 10 client.0 _ll_register_callbacks cb 0x7fad14009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51152/0 (socket says 172.21.15.16:51152) 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 -- 172.21.15.16:0/48081882 learned_addr learned my addr 172.21.15.16:0/48081882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2a711640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2a711640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 0x5624771b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 -- 172.21.15.16:0/48081882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 msgr2=0x5624771c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 -- 172.21.15.16:0/48081882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 msgr2=0x5624771b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 0x5624771b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2970f640 1 -- 172.21.15.16:0/48081882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624771b4ab0 con 0x5624771b0d20 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.068+0000 7fad2a711640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 0x5624771b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2970f640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fad14012420 tx=0x7fad140095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2faeab402dc4e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004210 con 0x5624771b0d20 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fad1400f920 con 0x5624771b0d20 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c043040 con 0x5624771b0d20 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fad14036050 con 0x5624771b0d20 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 10 client.20133 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:18.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562477239190 con 0x5624771b0d20 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c043040 con 0x5624771b0d20 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fad1402e020 con 0x5624771b0d20 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 10 client.20133 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 10 client.20133 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.069+0000 7fad2e54d740 20 client.20133 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562477239960 0x56247723bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 4 client.20133 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56247711cd70 con 0x562477239960 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2a711640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562477239960 0x56247723bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad0d7fa640 10 client.20133 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2a711640 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562477239960 0x56247723bd80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d29476d157baf58d server_cookie=1439bb65e395b3e2 in_seq=0 out_seq=0 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad0d7fa640 1 -- 172.21.15.16:0/48081882 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56247711cd70 con 0x562477239960 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad0d7fa640 10 client.20133 handle_command_reply: tid=0 2024-09-12T03:08:18.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 client.20133 shutdown 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 msgr2=0x5624771b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fad14012420 tx=0x7fad140095b0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 shutdown_connections 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x562477239960 0x56247723bd80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624771ba550 0x5624771b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624771b0d20 0x5624771b3e80 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 --2- 172.21.15.16:0/48081882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624771af7b0 0x5624771c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.070+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 >> 172.21.15.16:0/48081882 conn(0x562477178310 msgr2=0x5624771b9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad0d7fa640 0 client.20133 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 shutdown_connections 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2e54d740 1 -- 172.21.15.16:0/48081882 wait complete. 2024-09-12T03:08:18.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2e54d740 20 client.20133 trim_cache size 0 max 16384 2024-09-12T03:08:18.077 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:08:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 msgr2=0x7fad100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 0x7fad100a5040 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad200cd1a0 tx=0x7fad200d2e00 comp rx=0 tx=0).stop 2024-09-12T03:08:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 msgr2=0x7fad24198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9c0 tx=0x7fad2000ee80 comp rx=0 tx=0).stop 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 shutdown_connections 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad2410bfc0 0x7fad2419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fad100a2b80 0x7fad100a5040 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad241034b0 0x7fad24198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 --2- 172.21.15.16:0/572431237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad24102b90 0x7fad24198220 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.071+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 >> 172.21.15.16:0/572431237 conn(0x7fad240fc610 msgr2=0x7fad240fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.072+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 shutdown_connections 2024-09-12T03:08:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:18.072+0000 7fad2af12640 1 -- 172.21.15.16:0/572431237 wait complete. 2024-09-12T03:08:18.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:17 smithi033 ceph-mon[35092]: pgmap v948: 97 pgs: 97 active+clean; 779 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.2 MiB/s wr, 694 op/s 2024-09-12T03:08:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:17 smithi184 ceph-mon[35321]: pgmap v948: 97 pgs: 97 active+clean; 779 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.2 MiB/s wr, 694 op/s 2024-09-12T03:08:19.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 187246 24.72 MB/sec execute 320 sec latency 76.660 ms 2024-09-12T03:08:19.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:08:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:19.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:19 smithi016 ceph-mon[30925]: pgmap v949: 97 pgs: 97 active+clean; 779 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.2 MiB/s wr, 578 op/s 2024-09-12T03:08:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 -- 172.21.15.16:0/3809511690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724100a80 msgr2=0x7f2724100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 --2- 172.21.15.16:0/3809511690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724100a80 0x7f2724100ee0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f2714009930 tx=0x7f271402a420 comp rx=0 tx=0).stop 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 -- 172.21.15.16:0/3809511690 shutdown_connections 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 --2- 172.21.15.16:0/3809511690 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724101420 0x7f272410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 --2- 172.21.15.16:0/3809511690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724100a80 0x7f2724100ee0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 --2- 172.21.15.16:0/3809511690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 0x7f2724106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 -- 172.21.15.16:0/3809511690 >> 172.21.15.16:0/3809511690 conn(0x7f27240fc7a0 msgr2=0x7f27240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 -- 172.21.15.16:0/3809511690 shutdown_connections 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.992+0000 7f272c37b640 1 -- 172.21.15.16:0/3809511690 wait complete. 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 Processor -- start 2024-09-12T03:08:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 -- start start 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 0x7f27240fce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 0x7f27240fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2724110e80 con 0x7f2724101420 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2724110d00 con 0x7f2724100a80 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272c37b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2724111000 con 0x7f2724106a80 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272a117640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 0x7f27240fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51166/0 (socket says 172.21.15.16:51166) 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 -- 172.21.15.16:0/1896986053 learned_addr learned my addr 172.21.15.16:0/1896986053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f272a918640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 0x7f27240fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 -- 172.21.15.16:0/1896986053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 msgr2=0x7f27240fd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 0x7f27240fd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 -- 172.21.15.16:0/1896986053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 msgr2=0x7f27240fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.993+0000 7f2729916640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 0x7f27240fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f2729916640 1 -- 172.21.15.16:0/1896986053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27241a6930 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f272a117640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 0x7f27240fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f2729916640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2714009230 tx=0x7f2714004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f271b7fe640 1 -- 172.21.15.16:0/1896986053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271403f030 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f271b7fe640 1 -- 172.21.15.16:0/1896986053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2714004440 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27241a6bc0 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f271b7fe640 1 -- 172.21.15.16:0/1896986053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714035070 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.994+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27241a70f0 con 0x7f2724101420 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.997+0000 7f272e754740 1 Processor -- start 2024-09-12T03:08:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.997+0000 7f272e754740 1 -- start start 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.997+0000 7f271b7fe640 1 -- 172.21.15.16:0/1896986053 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2714042030 con 0x7f2724101420 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 0x563737303a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x563737310760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5637372692f0 con 0x5637373035d0 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563737210cf0 con 0x563737304b40 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272e754740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5637372652b0 con 0x56373730e370 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a117640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 0x563737303a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:52912/0 (socket says 172.21.15.16:52912) 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 -- 172.21.15.16:0/110102378 learned_addr learned my addr 172.21.15.16:0/110102378 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f2729916640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x563737310760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 -- 172.21.15.16:0/110102378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 msgr2=0x563737310760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x563737310760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 -- 172.21.15.16:0/110102378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 msgr2=0x563737303a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 0x563737303a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a918640 1 -- 172.21.15.16:0/110102378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637371f53a0 con 0x563737304b40 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f2729916640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x563737310760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.998+0000 7f272a117640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 0x563737303a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f271b7fe640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f272a918640 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f271c009fd0 tx=0x7f271c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1dcbbe1fe02188 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f272a117640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f271b7fe640 1 -- 172.21.15.16:0/1896986053 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27140ef260 con 0x7f2724101420 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f2718dea640 1 -- 172.21.15.16:0/110102378 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c004d20 con 0x563737304b40 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f2718dea640 1 -- 172.21.15.16:0/110102378 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f271c004050 con 0x563737304b40 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:19.999+0000 7f2718dea640 1 -- 172.21.15.16:0/110102378 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c005240 con 0x563737304b40 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- 172.21.15.16:0/110102378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 msgr2=0x563737303090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272a117640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2720009fd0 tx=0x7f2720005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f271c009fd0 tx=0x7f271c0118d0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- 172.21.15.16:0/110102378 shutdown_connections 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x563737310760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637373035d0 0x563737303a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 --2- 172.21.15.16:0/110102378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563737304b40 0x563737303090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- 172.21.15.16:0/110102378 >> 172.21.15.16:0/110102378 conn(0x5637372cc310 msgr2=0x5637372ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- 172.21.15.16:0/110102378 shutdown_connections 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- 172.21.15.16:0/110102378 wait complete. 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 -- start start 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.000+0000 7f272e754740 1 Processor -- start 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x56373730b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563737311070 con 0x563737304b40 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563737224060 con 0x5637373035d0 2024-09-12T03:08:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5637373111f0 con 0x56373730e370 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272e754740 10 client.0 _ll_register_callbacks cb 0x563736f778c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272a117640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51192/0 (socket says 172.21.15.16:51192) 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 -- 172.21.15.16:0/53907706 learned_addr learned my addr 172.21.15.16:0/53907706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f272a918640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x56373730b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 -- 172.21.15.16:0/53907706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 msgr2=0x56373730b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x56373730b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 -- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 msgr2=0x56373730dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 -- 172.21.15.16:0/53907706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563736fda3c0 con 0x563737304b40 2024-09-12T03:08:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f2729916640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f27140ecd70 tx=0x7f2714005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515a39d2afacc864 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.001+0000 7f270dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714049070 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27140b81f0 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f271c004050 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f27140bb950 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 10 client.20145 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56373731caf0 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f271c004050 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f2714031c60 con 0x563737304b40 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 10 client.20145 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 10 client.20145 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 20 client.20145 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56373731df90 0x563737320380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 4 client.20145 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.002+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563737276860 con 0x56373731df90 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.003+0000 7f272a117640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56373731df90 0x563737320380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.003+0000 7f270dffb640 10 client.20145 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.003+0000 7f272a117640 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56373731df90 0x563737320380 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5bc6b174343f4dea server_cookie=46869cb04518a9d0 in_seq=0 out_seq=0 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.003+0000 7f270dffb640 1 -- 172.21.15.16:0/53907706 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563737276860 con 0x56373731df90 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.003+0000 7f270dffb640 10 client.20145 handle_command_reply: tid=0 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 client.20145 shutdown 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 msgr2=0x56373730afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f27140ecd70 tx=0x7f2714005b50 comp rx=0 tx=0).stop 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 shutdown_connections 2024-09-12T03:08:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x56373731df90 0x563737320380 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373730e370 0x56373730b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f270dffb640 0 client.20145 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563737304b40 0x56373730afd0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 --2- 172.21.15.16:0/53907706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 >> 172.21.15.16:0/53907706 conn(0x5637372cc310 msgr2=0x563737305930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 shutdown_connections 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 1 -- 172.21.15.16:0/53907706 wait complete. 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.004+0000 7f272e754740 20 client.20145 trim_cache size 0 max 16384 2024-09-12T03:08:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 Processor -- start 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 -- start start 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x5637372ff4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27241af680 con 0x56373730e370 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27241b11c0 con 0x5637373035d0 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.021+0000 7f272e754740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27241afd50 con 0x56373731df90 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51206/0 (socket says 172.21.15.16:51206) 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 -- 172.21.15.16:0/586638952 learned_addr learned my addr 172.21.15.16:0/586638952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f272a918640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x5637372ff4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f272a117640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 -- 172.21.15.16:0/586638952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 msgr2=0x5637372ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x5637372ff4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 -- 172.21.15.16:0/586638952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 msgr2=0x56373730a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 -- 172.21.15.16:0/586638952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56373730b8e0 con 0x56373730e370 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f272a117640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f2729916640 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f27140ed0e0 tx=0x7f27140ba090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78de89f3d0ed80c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f270ffff640 1 -- 172.21.15.16:0/586638952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714049040 con 0x56373730e370 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f270ffff640 1 -- 172.21.15.16:0/586638952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27140c4520 con 0x56373730e370 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.022+0000 7f270ffff640 1 -- 172.21.15.16:0/586638952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27140bb4e0 con 0x56373730e370 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 -- 172.21.15.16:0/586638952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 msgr2=0x56373730cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f27140ed0e0 tx=0x7f27140ba090 comp rx=0 tx=0).stop 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 -- 172.21.15.16:0/586638952 shutdown_connections 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x5637372ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x56373730cf40 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 --2- 172.21.15.16:0/586638952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x56373730a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 -- 172.21.15.16:0/586638952 >> 172.21.15.16:0/586638952 conn(0x5637372cc310 msgr2=0x56373730c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 -- 172.21.15.16:0/586638952 shutdown_connections 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.023+0000 7f272e754740 1 -- 172.21.15.16:0/586638952 wait complete. 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 -- start start 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 Processor -- start 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x563737316be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x563737312450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27241b0af0 con 0x56373730e370 2024-09-12T03:08:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27241b0420 con 0x5637373035d0 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27241ad660 con 0x56373731df90 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272e754740 10 client.0 _ll_register_callbacks cb 0x7f2714009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272a117640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x563737316be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f2729916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51208/0 (socket says 172.21.15.16:51208) 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f2729916640 1 -- 172.21.15.16:0/1895932117 learned_addr learned my addr 172.21.15.16:0/1895932117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.024+0000 7f272a918640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x563737312450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 -- 172.21.15.16:0/1895932117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 msgr2=0x563737312450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x563737312450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 -- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 msgr2=0x563737316be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x563737316be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 -- 172.21.15.16:0/1895932117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563737312c80 con 0x56373730e370 2024-09-12T03:08:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f2729916640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f27140b7fe0 tx=0x7f27140bb3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b95d8c4e73f3ef5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:20.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.025+0000 7f270dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:20.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714049070 con 0x56373730e370 2024-09-12T03:08:20.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 187859 24.72 MB/sec execute 321 sec latency 74.073 ms 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563737270d70 con 0x56373730e370 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f27140da2b0 con 0x56373730e370 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f27140bb6c0 con 0x56373730e370 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f272e754740 10 client.20151 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563737312c80 con 0x56373730e370 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.026+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563737270d70 con 0x56373730e370 2024-09-12T03:08:20.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f27140baa40 con 0x56373730e370 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 10 client.20151 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 10 client.20151 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 20 client.20151 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563737388f00 0x56373738b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 4 client.20151 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637371ed170 con 0x563737388f00 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.027+0000 7f272a117640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563737388f00 0x56373738b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f270dffb640 10 client.20151 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272a117640 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563737388f00 0x56373738b2f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4938fdd8d4607bc9 server_cookie=31ea9416f7974a26 in_seq=0 out_seq=0 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f270dffb640 1 -- 172.21.15.16:0/1895932117 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637371ed170 con 0x563737388f00 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f270dffb640 10 client.20151 handle_command_reply: tid=0 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 client.20151 shutdown 2024-09-12T03:08:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 msgr2=0x563737311f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f27140b7fe0 tx=0x7f27140bb3a0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 shutdown_connections 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x563737388f00 0x56373738b2f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56373731df90 0x563737312450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56373730e370 0x563737311f10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f270dffb640 0 client.20151 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 --2- 172.21.15.16:0/1895932117 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5637373035d0 0x563737316be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 >> 172.21.15.16:0/1895932117 conn(0x5637372cc310 msgr2=0x563737305110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 shutdown_connections 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.028+0000 7f272e754740 1 -- 172.21.15.16:0/1895932117 wait complete. 2024-09-12T03:08:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.029+0000 7f272e754740 20 client.20151 trim_cache size 0 max 16384 2024-09-12T03:08:20.048 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:08:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.029+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 msgr2=0x7f27100a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.029+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2720009fd0 tx=0x7f2720005f90 comp rx=0 tx=0).stop 2024-09-12T03:08:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.029+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 msgr2=0x7f27240fd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.029+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2714009230 tx=0x7f2714004040 comp rx=0 tx=0).stop 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 shutdown_connections 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2724106a80 0x7f27240fd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f27100a2c00 0x7f27100a50c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2724101420 0x7f27240fd360 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 --2- 172.21.15.16:0/1896986053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2724100a80 0x7f27240fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 >> 172.21.15.16:0/1896986053 conn(0x7f27240fc7a0 msgr2=0x7f272406aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 shutdown_connections 2024-09-12T03:08:20.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:20.030+0000 7f272c37b640 1 -- 172.21.15.16:0/1896986053 wait complete. 2024-09-12T03:08:20.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:19 smithi033 ceph-mon[35092]: pgmap v949: 97 pgs: 97 active+clean; 779 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.2 MiB/s wr, 578 op/s 2024-09-12T03:08:20.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:19 smithi184 ceph-mon[35321]: pgmap v949: 97 pgs: 97 active+clean; 779 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.2 MiB/s wr, 578 op/s 2024-09-12T03:08:20.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:08:21.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 188354 24.72 MB/sec execute 322 sec latency 254.953 ms 2024-09-12T03:08:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 -- 172.21.15.16:0/3465507158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7410e2d0 msgr2=0x7ffa741106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 --2- 172.21.15.16:0/3465507158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7410e2d0 0x7ffa741106c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009ca0 tx=0x7ffa6c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 -- 172.21.15.16:0/3465507158 shutdown_connections 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 --2- 172.21.15.16:0/3465507158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7410e2d0 0x7ffa741106c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 --2- 172.21.15.16:0/3465507158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa741010d0 0x7ffa7410dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 --2- 172.21.15.16:0/3465507158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa74100720 0x7ffa74100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 -- 172.21.15.16:0/3465507158 >> 172.21.15.16:0/3465507158 conn(0x7ffa740fc610 msgr2=0x7ffa740fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 -- 172.21.15.16:0/3465507158 shutdown_connections 2024-09-12T03:08:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.692+0000 7ffa7a659640 1 -- 172.21.15.16:0/3465507158 wait complete. 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 Processor -- start 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 -- start start 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa741010d0 0x7ffa74103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 0x7ffa74107f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa79657640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74113fd0 con 0x7ffa74100720 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74113e50 con 0x7ffa7410e2d0 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.693+0000 7ffa7a659640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74114150 con 0x7ffa741010d0 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51218/0 (socket says 172.21.15.16:51218) 2024-09-12T03:08:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 -- 172.21.15.16:0/432623732 learned_addr learned my addr 172.21.15.16:0/432623732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79e58640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 0x7ffa74107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 -- 172.21.15.16:0/432623732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa741010d0 msgr2=0x7ffa74103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa741010d0 0x7ffa74103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 -- 172.21.15.16:0/432623732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 msgr2=0x7ffa74107f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 0x7ffa74107f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79657640 1 -- 172.21.15.16:0/432623732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa74108610 con 0x7ffa74100720 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.694+0000 7ffa79e58640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 0x7ffa74107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.695+0000 7ffa79657640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000e9f0 tx=0x7ffa7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.695+0000 7ffa6a7fc640 1 -- 172.21.15.16:0/432623732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7000cd80 con 0x7ffa74100720 2024-09-12T03:08:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.695+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa74108900 con 0x7ffa74100720 2024-09-12T03:08:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.695+0000 7ffa6a7fc640 1 -- 172.21.15.16:0/432623732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa70010910 con 0x7ffa74100720 2024-09-12T03:08:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.696+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7410d0c0 con 0x7ffa74100720 2024-09-12T03:08:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.696+0000 7ffa6a7fc640 1 -- 172.21.15.16:0/432623732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700188f0 con 0x7ffa74100720 2024-09-12T03:08:21.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.699+0000 7ffa6a7fc640 1 -- 172.21.15.16:0/432623732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ffa70010ac0 con 0x7ffa74100720 2024-09-12T03:08:21.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 Processor -- start 2024-09-12T03:08:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 -- start start 2024-09-12T03:08:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a252bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a2538910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594a24912f0 con 0x5594a252ccf0 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5594a2438cf0 con 0x5594a2536520 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594a248d2b0 con 0x5594a252b780 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51234/0 (socket says 172.21.15.16:51234) 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa79e58640 1 -- 172.21.15.16:0/2925306053 learned_addr learned my addr 172.21.15.16:0/2925306053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.700+0000 7ffa78e56640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a2538910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa79657640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a252bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa6a7fc640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 0x7ffa600a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 -- 172.21.15.16:0/2925306053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 msgr2=0x5594a252bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a252bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 -- 172.21.15.16:0/2925306053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 msgr2=0x5594a252b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 -- 172.21.15.16:0/2925306053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a241d3a0 con 0x5594a2536520 2024-09-12T03:08:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa6a7fc640 1 -- 172.21.15.16:0/432623732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffa70018690 con 0x7ffa74100720 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa78e56640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a2538910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffa741a1bf0 tx=0x7ffa6400be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab8da5eb130db3f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.701+0000 7ffa79e58640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.702+0000 7ffa79657640 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a252bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.702+0000 7ffa5f7fe640 1 -- 172.21.15.16:0/2925306053 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64016070 con 0x5594a2536520 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.702+0000 7ffa5f7fe640 1 -- 172.21.15.16:0/2925306053 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa6400de30 con 0x5594a2536520 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.702+0000 7ffa79e58640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 0x7ffa600a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.703+0000 7ffa5f7fe640 1 -- 172.21.15.16:0/2925306053 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64003e10 con 0x5594a2536520 2024-09-12T03:08:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.703+0000 7ffa7dc94740 1 -- 172.21.15.16:0/2925306053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 msgr2=0x5594a2538910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.703+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a2538910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffa741a1bf0 tx=0x7ffa6400be30 comp rx=0 tx=0).stop 2024-09-12T03:08:21.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.703+0000 7ffa79e58640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 0x7ffa600a5290 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009a60 tx=0x7ffa6c040040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.704+0000 7ffa7dc94740 1 -- 172.21.15.16:0/2925306053 shutdown_connections 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.704+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a2538910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.704+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a252bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.704+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/2925306053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a252b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.704+0000 7ffa7dc94740 1 -- 172.21.15.16:0/2925306053 >> 172.21.15.16:0/2925306053 conn(0x5594a24f4310 msgr2=0x5594a2527e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.705+0000 7ffa7dc94740 1 -- 172.21.15.16:0/2925306053 shutdown_connections 2024-09-12T03:08:21.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.705+0000 7ffa7dc94740 1 -- 172.21.15.16:0/2925306053 wait complete. 2024-09-12T03:08:21.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.706+0000 7ffa7dc94740 1 -- start start 2024-09-12T03:08:21.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.706+0000 7ffa7dc94740 1 Processor -- start 2024-09-12T03:08:21.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a2535dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a2533070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5594a25392c0 con 0x5594a2536520 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5594a244c060 con 0x5594a252ccf0 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594a2539440 con 0x5594a252b780 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51240/0 (socket says 172.21.15.16:51240) 2024-09-12T03:08:21.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.707+0000 7ffa79e58640 1 -- 172.21.15.16:0/397145551 learned_addr learned my addr 172.21.15.16:0/397145551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:21.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa78e56640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a2533070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa7dc94740 10 client.0 _ll_register_callbacks cb 0x5594a219f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa79e58640 1 -- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 msgr2=0x5594a2535dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa79e58640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a2535dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa79e58640 1 -- 172.21.15.16:0/397145551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 msgr2=0x5594a2533070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa79e58640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a2533070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.708+0000 7ffa79e58640 1 -- 172.21.15.16:0/397145551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a22023c0 con 0x5594a2536520 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa79e58640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009bf0 tx=0x7ffa6c02adc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce14f7aa4cf5705 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0307c0 con 0x5594a2536520 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa6400de30 con 0x5594a2536520 2024-09-12T03:08:21.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffa6c030960 con 0x5594a2536520 2024-09-12T03:08:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffa6c030b60 con 0x5594a2536520 2024-09-12T03:08:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa7dc94740 10 client.20163 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.709+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5594a2544be0 con 0x5594a2536520 2024-09-12T03:08:21.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.710+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa6400de30 con 0x5594a2536520 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ffa6c05b070 con 0x5594a2536520 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 10 client.20163 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 10 client.20163 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 20 client.20163 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a2546080 0x5594a2548520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 4 client.20163 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594a249e860 con 0x5594a2546080 2024-09-12T03:08:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa79657640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a2546080 0x5594a2548520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.711+0000 7ffa79657640 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a2546080 0x5594a2548520 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ee044bdf002e34f server_cookie=306120682fa94afc in_seq=0 out_seq=0 2024-09-12T03:08:21.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.712+0000 7ffa5cff9640 10 client.20163 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:08:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.714+0000 7ffa5cff9640 1 -- 172.21.15.16:0/397145551 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5594a249e860 con 0x5594a2546080 2024-09-12T03:08:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.714+0000 7ffa5cff9640 10 client.20163 handle_command_reply: tid=0 2024-09-12T03:08:21.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.714+0000 7ffa7dc94740 1 client.20163 shutdown 2024-09-12T03:08:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 msgr2=0x5594a25335b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009bf0 tx=0x7ffa6c02adc0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 shutdown_connections 2024-09-12T03:08:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a25335b0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a2533070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a2546080 0x5594a2548520 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/397145551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a2535dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 >> 172.21.15.16:0/397145551 conn(0x5594a24f4310 msgr2=0x5594a252db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa5cff9640 0 client.20163 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 shutdown_connections 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.715+0000 7ffa7dc94740 1 -- 172.21.15.16:0/397145551 wait complete. 2024-09-12T03:08:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.716+0000 7ffa7dc94740 20 client.20163 trim_cache size 0 max 16384 2024-09-12T03:08:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 Processor -- start 2024-09-12T03:08:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 -- start start 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a2527fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a254de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741bc280 con 0x5594a252ccf0 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741bddc0 con 0x5594a2536520 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741bc950 con 0x5594a252b780 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa78e56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a2527fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58898/0 (socket says 172.21.15.16:58898) 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 -- 172.21.15.16:0/1476607978 learned_addr learned my addr 172.21.15.16:0/1476607978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79e58640 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a254de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 -- 172.21.15.16:0/1476607978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 msgr2=0x5594a254de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a254de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 -- 172.21.15.16:0/1476607978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 msgr2=0x5594a2527fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a2527fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa79657640 1 -- 172.21.15.16:0/1476607978 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a25315c0 con 0x5594a252b780 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.737+0000 7ffa78e56640 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a2527fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.738+0000 7ffa79657640 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009380 tx=0x7ffa70009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=600c7dcaa7fca6cd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.738+0000 7ffa5effd640 1 -- 172.21.15.16:0/1476607978 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700a4760 con 0x5594a252b780 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.738+0000 7ffa5effd640 1 -- 172.21.15.16:0/1476607978 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa70099b90 con 0x5594a252b780 2024-09-12T03:08:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.738+0000 7ffa5effd640 1 -- 172.21.15.16:0/1476607978 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700d10b0 con 0x5594a252b780 2024-09-12T03:08:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- 172.21.15.16:0/1476607978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 msgr2=0x5594a25425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009380 tx=0x7ffa70009310 comp rx=0 tx=0).stop 2024-09-12T03:08:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- 172.21.15.16:0/1476607978 shutdown_connections 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a2536520 0x5594a254de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a252ccf0 0x5594a2527fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/1476607978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a25425b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- 172.21.15.16:0/1476607978 >> 172.21.15.16:0/1476607978 conn(0x5594a24f4310 msgr2=0x5594a25278b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- 172.21.15.16:0/1476607978 shutdown_connections 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- 172.21.15.16:0/1476607978 wait complete. 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 -- start start 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.739+0000 7ffa7dc94740 1 Processor -- start 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a253ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a252fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741bd6f0 con 0x5594a2536520 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741bd020 con 0x5594a252ccf0 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741ba260 con 0x5594a252b780 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa7dc94740 10 client.0 _ll_register_callbacks cb 0x7ffa6c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51254/0 (socket says 172.21.15.16:51254) 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 -- 172.21.15.16:0/894197840 learned_addr learned my addr 172.21.15.16:0/894197840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79657640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a253ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa78e56640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a252fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 -- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 msgr2=0x5594a253ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a253ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 -- 172.21.15.16:0/894197840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 msgr2=0x5594a252fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a252fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 -- 172.21.15.16:0/894197840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a2530bb0 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79657640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a253ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa79e58640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x5594a25364f0 tx=0x7ffa6c004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b34f6ac578ae71b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.740+0000 7ffa5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c03c5a0 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffa6c03c740 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa70099b90 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffa6c03c940 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa7dc94740 10 client.20169 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5594a25b0e90 con 0x5594a2536520 2024-09-12T03:08:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa70099b90 con 0x5594a2536520 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.741+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ffa6c065030 con 0x5594a2536520 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 10 client.20169 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 10 client.20169 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 20 client.20169 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a25b1540 0x5594a25b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 4 client.20169 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594a2498d70 con 0x5594a25b1540 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.742+0000 7ffa79657640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a25b1540 0x5594a25b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.743+0000 7ffa79657640 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a25b1540 0x5594a25b3960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0b8ba677ccc4f6a server_cookie=315ce812cf9d846d in_seq=0 out_seq=0 2024-09-12T03:08:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.743+0000 7ffa5cff9640 10 client.20169 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa5cff9640 1 -- 172.21.15.16:0/894197840 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594a2498d70 con 0x5594a25b1540 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa5cff9640 10 client.20169 handle_command_reply: tid=0 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 client.20169 shutdown 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 msgr2=0x5594a2530380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x5594a25364f0 tx=0x7ffa6c004df0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 shutdown_connections 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5594a2536520 0x5594a2530380 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5594a252ccf0 0x5594a252fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x5594a25b1540 0x5594a25b3960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa5cff9640 0 client.20169 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:08:21.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 --2- 172.21.15.16:0/894197840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594a252b780 0x5594a253ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 >> 172.21.15.16:0/894197840 conn(0x5594a24f4310 msgr2=0x5594a25348d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 shutdown_connections 2024-09-12T03:08:21.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 1 -- 172.21.15.16:0/894197840 wait complete. 2024-09-12T03:08:21.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.746+0000 7ffa7dc94740 20 client.20169 trim_cache size 0 max 16384 2024-09-12T03:08:21.751 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.747+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 msgr2=0x7ffa600a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.747+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 0x7ffa600a5290 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009a60 tx=0x7ffa6c040040 comp rx=0 tx=0).stop 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.747+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 msgr2=0x7ffa74102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.747+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000e9f0 tx=0x7ffa7000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 shutdown_connections 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa7410e2d0 0x7ffa74107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffa741010d0 0x7ffa74103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffa600a2dd0 0x7ffa600a5290 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 --2- 172.21.15.16:0/432623732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa74100720 0x7ffa74102b00 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:21.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 >> 172.21.15.16:0/432623732 conn(0x7ffa740fc610 msgr2=0x7ffa74110410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:21.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 shutdown_connections 2024-09-12T03:08:21.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:21.748+0000 7ffa7a659640 1 -- 172.21.15.16:0/432623732 wait complete. 2024-09-12T03:08:22.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 188981 24.76 MB/sec execute 323 sec latency 102.080 ms 2024-09-12T03:08:22.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:21 smithi016 ceph-mon[30925]: pgmap v950: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 702 op/s 2024-09-12T03:08:22.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:21 smithi033 ceph-mon[35092]: pgmap v950: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 702 op/s 2024-09-12T03:08:22.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:21 smithi184 ceph-mon[35321]: pgmap v950: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 702 op/s 2024-09-12T03:08:22.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:08:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:23.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 189763 24.80 MB/sec execute 324 sec latency 69.859 ms 2024-09-12T03:08:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 -- 172.21.15.16:0/3670026853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102ef0 msgr2=0x7fd68410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/3670026853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102ef0 0x7fd68410da40 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009900 tx=0x7fd67c02a480 comp rx=0 tx=0).stop 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 -- 172.21.15.16:0/3670026853 shutdown_connections 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/3670026853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102ef0 0x7fd68410da40 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/3670026853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684102570 0x7fd6841029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/3670026853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684100e20 0x7fd684101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 -- 172.21.15.16:0/3670026853 >> 172.21.15.16:0/3670026853 conn(0x7fd6840fc7d0 msgr2=0x7fd6840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 -- 172.21.15.16:0/3670026853 shutdown_connections 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.345+0000 7fd68b8a9640 1 -- 172.21.15.16:0/3670026853 wait complete. 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 Processor -- start 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 -- start start 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 0x7fd6841984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684102ef0 0x7fd68419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd684110fd0 con 0x7fd684102570 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd684110e50 con 0x7fd684100e20 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd68b8a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd684111150 con 0x7fd684102ef0 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.346+0000 7fd688e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd688e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51274/0 (socket says 172.21.15.16:51274) 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd688e44640 1 -- 172.21.15.16:0/460158950 learned_addr learned my addr 172.21.15.16:0/460158950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 0x7fd6841984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689e46640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684102ef0 0x7fd68419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 -- 172.21.15.16:0/460158950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684102ef0 msgr2=0x7fd68419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684102ef0 0x7fd68419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 -- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 msgr2=0x7fd684198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd689645640 1 -- 172.21.15.16:0/460158950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68419d480 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.347+0000 7fd688e44640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.348+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 0x7fd6841984d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd67400dab0 tx=0x7fd67400df70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.348+0000 7fd6827fc640 1 -- 172.21.15.16:0/460158950 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67400be30 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.348+0000 7fd6827fc640 1 -- 172.21.15.16:0/460158950 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd674003e50 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.348+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68419d770 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.348+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6841a5050 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.349+0000 7fd6827fc640 1 -- 172.21.15.16:0/460158950 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674010770 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.352+0000 7fd68dc82740 1 Processor -- start 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.352+0000 7fd68dc82740 1 -- start start 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd6827fc640 1 -- 172.21.15.16:0/460158950 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd674004000 con 0x7fd684100e20 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ec094a2f0 con 0x555ec09f3780 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555ec08f1cf0 con 0x555ec09e45d0 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd68dc82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555ec09462b0 con 0x555ec09e5b40 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58946/0 (socket says 172.21.15.16:58946) 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 -- 172.21.15.16:0/225278322 learned_addr learned my addr 172.21.15.16:0/225278322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689645640 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 -- 172.21.15.16:0/225278322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 msgr2=0x555ec09e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 -- 172.21.15.16:0/225278322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 msgr2=0x555ec09f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689e46640 1 -- 172.21.15.16:0/225278322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ec08d63a0 con 0x555ec09e5b40 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.353+0000 7fd689645640 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd689e46640 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009900 tx=0x7fd67c005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d213c830592c177f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd6827fc640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 0x7fd66c0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd66b7fe640 1 -- 172.21.15.16:0/225278322 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c040070 con 0x555ec09e5b40 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd66b7fe640 1 -- 172.21.15.16:0/225278322 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67c004400 con 0x555ec09e5b40 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd66b7fe640 1 -- 172.21.15.16:0/225278322 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c035040 con 0x555ec09e5b40 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd6827fc640 1 -- 172.21.15.16:0/460158950 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd674010510 con 0x7fd684100e20 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.354+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 0x7fd66c0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 -- 172.21.15.16:0/225278322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 msgr2=0x555ec09e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd689645640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 0x7fd66c0a52c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd67409b990 tx=0x7fd674004770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009900 tx=0x7fd67c005c00 comp rx=0 tx=0).stop 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 -- 172.21.15.16:0/225278322 shutdown_connections 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 --2- 172.21.15.16:0/225278322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e4090 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.355+0000 7fd68dc82740 1 -- 172.21.15.16:0/225278322 >> 172.21.15.16:0/225278322 conn(0x555ec09ad310 msgr2=0x555ec09e0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.356+0000 7fd68dc82740 1 -- 172.21.15.16:0/225278322 shutdown_connections 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.356+0000 7fd68dc82740 1 -- 172.21.15.16:0/225278322 wait complete. 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.356+0000 7fd68dc82740 1 -- start start 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.356+0000 7fd68dc82740 1 Processor -- start 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09e5b40 0x555ec09f0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09f3780 0x555ec09f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ec09f6420 con 0x555ec09e5b40 2024-09-12T03:08:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555ec0905060 con 0x555ec09e45d0 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555ec09f65a0 con 0x555ec09f3780 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd68dc82740 10 client.0 _ll_register_callbacks cb 0x555ec06588c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53032/0 (socket says 172.21.15.16:53032) 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 -- 172.21.15.16:0/548676184 learned_addr learned my addr 172.21.15.16:0/548676184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09f3780 0x555ec09f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd688e44640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09e5b40 0x555ec09f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 -- 172.21.15.16:0/548676184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09f3780 msgr2=0x555ec09f0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09f3780 0x555ec09f0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 -- 172.21.15.16:0/548676184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09e5b40 msgr2=0x555ec09f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09e5b40 0x555ec09f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.357+0000 7fd689e46640 1 -- 172.21.15.16:0/548676184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ec06bb3c0 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd689e46640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd67c004b70 tx=0x7fd67c004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f6c22238ababcb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd668ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c040070 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd67c005000 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd67c004400 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd67c035040 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd68dc82740 10 client.29405 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555ec0a01e90 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.358+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67c003800 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd67c031540 con 0x555ec09e45d0 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 10 client.29405 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 10 client.29405 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 20 client.29405 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:23.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a031e0 0x555ec0a055d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 4 client.29405 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd67c004400 con 0x555ec0a031e0 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.359+0000 7fd688e44640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a031e0 0x555ec0a055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.360+0000 7fd688e44640 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a031e0 0x555ec0a055d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dfda04d1c58050cf server_cookie=257e43c5f32e92fe in_seq=0 out_seq=0 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.360+0000 7fd668ff9640 10 client.29405 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd668ff9640 1 -- 172.21.15.16:0/548676184 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fd67c004400 con 0x555ec0a031e0 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd668ff9640 10 client.29405 handle_command_reply: tid=0 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 client.29405 shutdown 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 msgr2=0x555ec09f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd67c004b70 tx=0x7fd67c004c70 comp rx=0 tx=0).stop 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 shutdown_connections 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a031e0 0x555ec0a055d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09f3780 0x555ec09f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09e5b40 0x555ec09f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 --2- 172.21.15.16:0/548676184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09f3190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd668ff9640 0 client.29405 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 >> 172.21.15.16:0/548676184 conn(0x555ec09ad310 msgr2=0x555ec09e6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 shutdown_connections 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 1 -- 172.21.15.16:0/548676184 wait complete. 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.361+0000 7fd68dc82740 20 client.29405 trim_cache size 0 max 16384 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 Processor -- start 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 -- start start 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09e0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841ab3b0 con 0x555ec09f3780 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841acef0 con 0x555ec09e45d0 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd68dc82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841aba80 con 0x555ec09e5b40 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09e0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd688e44640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53044/0 (socket says 172.21.15.16:53044) 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd688e44640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58970/0 (socket says 172.21.15.16:58970) 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 -- 172.21.15.16:0/506823565 learned_addr learned my addr 172.21.15.16:0/506823565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 -- 172.21.15.16:0/506823565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 msgr2=0x555ec09e04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.382+0000 7fd689e46640 1 -- 172.21.15.16:0/506823565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 msgr2=0x555ec09e0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd689e46640 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09e0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd689e46640 1 -- 172.21.15.16:0/506823565 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ec09f13e0 con 0x555ec09e45d0 2024-09-12T03:08:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd689e46640 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd67c031940 tx=0x7fd67c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b533c4a6de9b1455 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd66affd640 1 -- 172.21.15.16:0/506823565 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c040040 con 0x555ec09e45d0 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd66affd640 1 -- 172.21.15.16:0/506823565 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67c004380 con 0x555ec09e45d0 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.383+0000 7fd66affd640 1 -- 172.21.15.16:0/506823565 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c031b30 con 0x555ec09e45d0 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 -- 172.21.15.16:0/506823565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 msgr2=0x555ec09eed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd67c031940 tx=0x7fd67c000bc0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 -- 172.21.15.16:0/506823565 shutdown_connections 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09e0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e5b40 0x555ec09e04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 --2- 172.21.15.16:0/506823565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e45d0 0x555ec09eed10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 -- 172.21.15.16:0/506823565 >> 172.21.15.16:0/506823565 conn(0x555ec09ad310 msgr2=0x555ec09e6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 -- 172.21.15.16:0/506823565 shutdown_connections 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.384+0000 7fd68dc82740 1 -- 172.21.15.16:0/506823565 wait complete. 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.385+0000 7fd68dc82740 1 -- start start 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.385+0000 7fd68dc82740 1 Processor -- start 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 0x555ec09ea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841ac820 con 0x555ec09f3780 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841ac150 con 0x555ec09e5b40 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6841a9390 con 0x555ec09e45d0 2024-09-12T03:08:23.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd68dc82740 10 client.0 _ll_register_callbacks cb 0x7fd67c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd689e46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd689e46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58982/0 (socket says 172.21.15.16:58982) 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd689e46640 1 -- 172.21.15.16:0/1324245202 learned_addr learned my addr 172.21.15.16:0/1324245202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.386+0000 7fd688e44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 0x555ec09ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 -- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 msgr2=0x555ec09ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 0x555ec09ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 -- 172.21.15.16:0/1324245202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 msgr2=0x555ec09eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 -- 172.21.15.16:0/1324245202 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ec0a06980 con 0x555ec09e45d0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd688e44640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 0x555ec09ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd689e46640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd67c005d00 tx=0x7fd67c0045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=338f24b0a727a4b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd668ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:08:23.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c063060 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.387+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd67c031b30 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd67c004380 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd67c049070 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 10 client.29341 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555ec0a72480 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67c004380 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd67c003800 con 0x555ec09e45d0 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 10 client.29341 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 10 client.29341 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 20 client.29341 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.388+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a72b40 0x555ec0a74f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.389+0000 7fd68dc82740 4 client.29341 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.389+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555ec0957860 con 0x555ec0a72b40 2024-09-12T03:08:23.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.389+0000 7fd688e44640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a72b40 0x555ec0a74f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd688e44640 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a72b40 0x555ec0a74f30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=189e580724d1a393 server_cookie=d26a7aa14f8c448 in_seq=0 out_seq=0 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd668ff9640 10 client.29341 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd668ff9640 1 -- 172.21.15.16:0/1324245202 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555ec0957860 con 0x555ec0a72b40 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd668ff9640 10 client.29341 handle_command_reply: tid=0 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 client.29341 shutdown 2024-09-12T03:08:23.459 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 msgr2=0x555ec0a08f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd67c005d00 tx=0x7fd67c0045d0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 shutdown_connections 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x555ec0a72b40 0x555ec0a74f30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ec09f3780 0x555ec09eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555ec09e5b40 0x555ec09ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 --2- 172.21.15.16:0/1324245202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555ec09e45d0 0x555ec0a08f00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 >> 172.21.15.16:0/1324245202 conn(0x555ec09ad310 msgr2=0x555ec09fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd668ff9640 0 client.29341 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.391+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 shutdown_connections 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.392+0000 7fd68dc82740 1 -- 172.21.15.16:0/1324245202 wait complete. 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.392+0000 7fd68dc82740 20 client.29341 trim_cache size 0 max 16384 2024-09-12T03:08:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 msgr2=0x7fd66c0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 0x7fd66c0a52c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd67409b990 tx=0x7fd674004770 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 msgr2=0x7fd6841984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 0x7fd6841984d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd67400dab0 tx=0x7fd67400df70 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 shutdown_connections 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd684102ef0 0x7fd68419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd66c0a2e00 0x7fd66c0a52c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd684102570 0x7fd684198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 --2- 172.21.15.16:0/460158950 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd684100e20 0x7fd6841984d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 >> 172.21.15.16:0/460158950 conn(0x7fd6840fc7d0 msgr2=0x7fd68406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 shutdown_connections 2024-09-12T03:08:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:23.393+0000 7fd68b8a9640 1 -- 172.21.15.16:0/460158950 wait complete. 2024-09-12T03:08:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 190647 24.88 MB/sec execute 325 sec latency 65.586 ms 2024-09-12T03:08:24.074 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:23 smithi016 ceph-mon[30925]: pgmap v951: 97 pgs: 97 active+clean; 791 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 578 op/s 2024-09-12T03:08:24.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:08:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:23 smithi033 ceph-mon[35092]: pgmap v951: 97 pgs: 97 active+clean; 791 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 578 op/s 2024-09-12T03:08:24.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:23 smithi184 ceph-mon[35321]: pgmap v951: 97 pgs: 97 active+clean; 791 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 578 op/s 2024-09-12T03:08:24.438 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- 172.21.15.16:0/1106595849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01036e0 msgr2=0x7fa8e0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/1106595849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01036e0 0x7fa8e0107b90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009930 tx=0x7fa8cc02a420 comp rx=0 tx=0).stop 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- 172.21.15.16:0/1106595849 shutdown_connections 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/1106595849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e01082a0 0x7fa8e010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/1106595849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01036e0 0x7fa8e0107b90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/1106595849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e0102d30 0x7fa8e0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- 172.21.15.16:0/1106595849 >> 172.21.15.16:0/1106595849 conn(0x7fa8e00fc7d0 msgr2=0x7fa8e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- 172.21.15.16:0/1106595849 shutdown_connections 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- 172.21.15.16:0/1106595849 wait complete. 2024-09-12T03:08:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 Processor -- start 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.946+0000 7fa8e70bf640 1 -- start start 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e0102d30 0x7fa8e0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e01036e0 0x7fa8e0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01120a0 con 0x7fa8e01082a0 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0111f20 con 0x7fa8e01036e0 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e70bf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0112220 con 0x7fa8e0102d30 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51328/0 (socket says 172.21.15.16:51328) 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e4e5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e0102d30 0x7fa8e0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 -- 172.21.15.16:0/2577972845 learned_addr learned my addr 172.21.15.16:0/2577972845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8dffff640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e01036e0 0x7fa8e0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 -- 172.21.15.16:0/2577972845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e0102d30 msgr2=0x7fa8e0100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e0102d30 0x7fa8e0100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 -- 172.21.15.16:0/2577972845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e01036e0 msgr2=0x7fa8e0100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e01036e0 0x7fa8e0100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 -- 172.21.15.16:0/2577972845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e01a6b00 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8e565c640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800eb10 tx=0x7fa8d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.947+0000 7fa8ddffb640 1 -- 172.21.15.16:0/2577972845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8004b20 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.948+0000 7fa8ddffb640 1 -- 172.21.15.16:0/2577972845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d8004cc0 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.948+0000 7fa8ddffb640 1 -- 172.21.15.16:0/2577972845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8010760 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.948+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e01a6df0 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.948+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e01a72b0 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.950+0000 7fa8ddffb640 1 -- 172.21.15.16:0/2577972845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa8d8005020 con 0x7fa8e01082a0 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 Processor -- start 2024-09-12T03:08:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 -- start start 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eaaffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab0c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604eaa652f0 con 0x5604eaaff5d0 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5604eaa0ccf0 con 0x5604eab00b40 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e9498740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5604eaa612b0 con 0x5604eab0a370 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8ddffb640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 0x7fa8d00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e4e5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eaaffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53066/0 (socket says 172.21.15.16:53066) 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8dffff640 1 -- 172.21.15.16:0/3062813687 learned_addr learned my addr 172.21.15.16:0/3062813687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e4e5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eaaffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51332/0 (socket says 172.21.15.16:51332) 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8dffff640 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab0c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.951+0000 7fa8e565c640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 0x7fa8d00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 -- 172.21.15.16:0/3062813687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 msgr2=0x5604eab0c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab0c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 -- 172.21.15.16:0/3062813687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 msgr2=0x5604eaaffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8ddffb640 1 -- 172.21.15.16:0/2577972845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8d80cf600 con 0x7fa8e01082a0 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eaaffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 -- 172.21.15.16:0/3062813687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ea9f13a0 con 0x5604eab00b40 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8dffff640 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc005ec0 tx=0x7fa8cc005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed4f5fc837003d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8e565c640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 0x7fa8d00a50c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000f80 tx=0x7fa8d806ef30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8caffd640 1 -- 172.21.15.16:0/3062813687 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc035070 con 0x5604eab00b40 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8caffd640 1 -- 172.21.15.16:0/3062813687 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc0318f0 con 0x5604eab00b40 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.952+0000 7fa8caffd640 1 -- 172.21.15.16:0/3062813687 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc003c80 con 0x5604eab00b40 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 -- 172.21.15.16:0/3062813687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 msgr2=0x5604eaaff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc005ec0 tx=0x7fa8cc005780 comp rx=0 tx=0).stop 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 -- 172.21.15.16:0/3062813687 shutdown_connections 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab0c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eaaffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 --2- 172.21.15.16:0/3062813687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eaaff090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 -- 172.21.15.16:0/3062813687 >> 172.21.15.16:0/3062813687 conn(0x5604eaac8310 msgr2=0x5604eaafbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 -- 172.21.15.16:0/3062813687 shutdown_connections 2024-09-12T03:08:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.953+0000 7fa8e9498740 1 -- 172.21.15.16:0/3062813687 wait complete. 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 -- start start 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 Processor -- start 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 0x5604eab07060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab077f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604eab0d010 con 0x5604eab00b40 2024-09-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5604eaa20060 con 0x5604eaaff5d0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.954+0000 7fa8e9498740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5604eab0d190 con 0x5604eab0a370 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8e9498740 10 client.0 _ll_register_callbacks cb 0x5604ea7738c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:53076/0 (socket says 172.21.15.16:53076) 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 -- 172.21.15.16:0/1073216555 learned_addr learned my addr 172.21.15.16:0/1073216555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab077f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 0x5604eab07060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 -- 172.21.15.16:0/1073216555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 msgr2=0x5604eab077f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab077f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 -- 172.21.15.16:0/1073216555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 msgr2=0x5604eab07060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 0x5604eab07060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 -- 172.21.15.16:0/1073216555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ea7d63c0 con 0x5604eaaff5d0 2024-09-12T03:08:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 0x5604eab07060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009900 tx=0x7fa8cc041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23567ab377c9f9ef server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.955+0000 7fa8bffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc04e030 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8cc0318f0 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8cc04a440 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc035070 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 10 client.29423 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5604eab21310 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc0318f0 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa8cc04a7d0 con 0x5604eaaff5d0 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 10 client.29423 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 10 client.29423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 20 client.29423 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab22660 0x5604eab24a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.956+0000 7fa8e9498740 4 client.29423 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.957+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604eaa72860 con 0x5604eab22660 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.957+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab22660 0x5604eab24a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.957+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab22660 0x5604eab24a50 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ff68a0c65faa30 server_cookie=fa83502a9bff0088 in_seq=0 out_seq=0 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.957+0000 7fa8bffff640 10 client.29423 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8bffff640 1 -- 172.21.15.16:0/1073216555 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604eaa72860 con 0x5604eab22660 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8bffff640 10 client.29423 handle_command_reply: tid=0 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 client.29423 shutdown 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 msgr2=0x5604eab09db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009900 tx=0x7fa8cc041040 comp rx=0 tx=0).stop 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 shutdown_connections 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab22660 0x5604eab24a50 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eab00b40 0x5604eab07060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1073216555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eaaff5d0 0x5604eab09db0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 >> 172.21.15.16:0/1073216555 conn(0x5604eaac8310 msgr2=0x5604eab019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8bffff640 0 client.29423 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 shutdown_connections 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 1 -- 172.21.15.16:0/1073216555 wait complete. 2024-09-12T03:08:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.960+0000 7fa8e9498740 20 client.29423 trim_cache size 0 max 16384 2024-09-12T03:08:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.974+0000 7fa8e9498740 1 Processor -- start 2024-09-12T03:08:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 -- start start 2024-09-12T03:08:24.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab00b40 0x5604eab02120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab0a370 0x5604eab02660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01ab3b0 con 0x5604eaaff5d0 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01acef0 con 0x5604eab0a370 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e9498740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01aba80 con 0x5604eab00b40 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51362/0 (socket says 172.21.15.16:51362) 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 -- 172.21.15.16:0/531167384 learned_addr learned my addr 172.21.15.16:0/531167384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab00b40 0x5604eab02120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab0a370 0x5604eab02660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 -- 172.21.15.16:0/531167384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab00b40 msgr2=0x5604eab02120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab00b40 0x5604eab02120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 -- 172.21.15.16:0/531167384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab0a370 msgr2=0x5604eab02660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab0a370 0x5604eab02660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.975+0000 7fa8dffff640 1 -- 172.21.15.16:0/531167384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604eab02ba0 con 0x5604eaaff5d0 2024-09-12T03:08:24.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.976+0000 7fa8dffff640 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc04d040 tx=0x7fa8cc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d891ed7b46b846d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.976+0000 7fa8ca7fc640 1 -- 172.21.15.16:0/531167384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc04e040 con 0x5604eaaff5d0 2024-09-12T03:08:24.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.976+0000 7fa8ca7fc640 1 -- 172.21.15.16:0/531167384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc061070 con 0x5604eaaff5d0 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.976+0000 7fa8ca7fc640 1 -- 172.21.15.16:0/531167384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc004770 con 0x5604eaaff5d0 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- 172.21.15.16:0/531167384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 msgr2=0x5604eab1e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc04d040 tx=0x7fa8cc000bc0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- 172.21.15.16:0/531167384 shutdown_connections 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab0a370 0x5604eab02660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab00b40 0x5604eab02120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 --2- 172.21.15.16:0/531167384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab1e3c0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- 172.21.15.16:0/531167384 >> 172.21.15.16:0/531167384 conn(0x5604eaac8310 msgr2=0x5604eab1dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- 172.21.15.16:0/531167384 shutdown_connections 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- 172.21.15.16:0/531167384 wait complete. 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 -- start start 2024-09-12T03:08:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.977+0000 7fa8e9498740 1 Processor -- start 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eab84870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab24c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01ac820 con 0x5604eaaff5d0 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01ac150 con 0x5604eab00b40 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01a93b0 con 0x5604eab0a370 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e9498740 10 client.0 _ll_register_callbacks cb 0x7fa8d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eab84870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51372/0 (socket says 172.21.15.16:51372) 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8dffff640 1 -- 172.21.15.16:0/1226659022 learned_addr learned my addr 172.21.15.16:0/1226659022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab24c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 -- 172.21.15.16:0/1226659022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 msgr2=0x5604eab24c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab24c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 -- 172.21.15.16:0/1226659022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 msgr2=0x5604eab84330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8e4e5b640 1 -- 172.21.15.16:0/1226659022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604eab25480 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.978+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8e4e5b640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eab84870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x5604eab0d1e0 tx=0x7fa8d40125f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4022b591ccf338f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8bffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d402a060 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa8d4025440 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8cc061070 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa8d4025660 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8e9498740 10 client.29429 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5604eab8cb30 con 0x5604eab00b40 2024-09-12T03:08:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.979+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc061070 con 0x5604eab00b40 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa8d4010830 con 0x5604eab00b40 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 10 client.29429 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 10 client.29429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 20 client.29429 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab8d2b0 0x5604eab8f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 4 client.29429 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604eaa6cd70 con 0x5604eab8d2b0 2024-09-12T03:08:24.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab8d2b0 0x5604eab8f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:24.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8bffff640 10 client.29429 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:24.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.980+0000 7fa8dffff640 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab8d2b0 0x5604eab8f6d0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50426878aea93541 server_cookie=132505477a677275 in_seq=0 out_seq=0 2024-09-12T03:08:24.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8bffff640 1 -- 172.21.15.16:0/1226659022 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604eaa6cd70 con 0x5604eab8d2b0 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8bffff640 10 client.29429 handle_command_reply: tid=0 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 client.29429 shutdown 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 msgr2=0x5604eab84870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eab84870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x5604eab0d1e0 tx=0x7fa8d40125f0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 shutdown_connections 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5604eab8d2b0 0x5604eab8f6d0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5604eab0a370 0x5604eab24c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604eab00b40 0x5604eab84870 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8bffff640 0 client.29429 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 --2- 172.21.15.16:0/1226659022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604eaaff5d0 0x5604eab84330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 >> 172.21.15.16:0/1226659022 conn(0x5604eaac8310 msgr2=0x5604eab1dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 shutdown_connections 2024-09-12T03:08:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.981+0000 7fa8e9498740 1 -- 172.21.15.16:0/1226659022 wait complete. 2024-09-12T03:08:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e9498740 20 client.29429 trim_cache size 0 max 16384 2024-09-12T03:08:24.987 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:08:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 msgr2=0x7fa8d00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 0x7fa8d00a50c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000f80 tx=0x7fa8d806ef30 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 msgr2=0x7fa8e0101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800eb10 tx=0x7fa8d800c2d0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 shutdown_connections 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa8d00a2c00 0x7fa8d00a50c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8e01082a0 0x7fa8e0101410 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8e01036e0 0x7fa8e0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 --2- 172.21.15.16:0/2577972845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa8e0102d30 0x7fa8e0100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 >> 172.21.15.16:0/2577972845 conn(0x7fa8e00fc7d0 msgr2=0x7fa8e0106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 shutdown_connections 2024-09-12T03:08:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:24.982+0000 7fa8e70bf640 1 -- 172.21.15.16:0/2577972845 wait complete. 2024-09-12T03:08:25.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191276 24.94 MB/sec execute 326 sec latency 51.187 ms 2024-09-12T03:08:25.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:25 smithi016 ceph-mon[30925]: pgmap v952: 97 pgs: 97 active+clean; 795 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.3 MiB/s wr, 599 op/s 2024-09-12T03:08:26.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 191624 24.92 MB/sec execute 327 sec latency 241.271 ms 2024-09-12T03:08:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:25 smithi033 ceph-mon[35092]: pgmap v952: 97 pgs: 97 active+clean; 795 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.3 MiB/s wr, 599 op/s 2024-09-12T03:08:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:25 smithi184 ceph-mon[35321]: pgmap v952: 97 pgs: 97 active+clean; 795 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.3 MiB/s wr, 599 op/s 2024-09-12T03:08:26.683 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:08:26.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:08:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:27.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 192092 24.90 MB/sec execute 328 sec latency 102.430 ms 2024-09-12T03:08:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.572+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/2452932032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38102570 msgr2=0x7f2f381029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.572+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/2452932032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38102570 0x7f2f381029b0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009860 tx=0x7f2f2002a3d0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/2452932032 shutdown_connections 2024-09-12T03:08:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/2452932032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102ef0 0x7f2f3810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/2452932032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38102570 0x7f2f381029b0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/2452932032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38100e20 0x7f2f38101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/2452932032 >> 172.21.15.16:0/2452932032 conn(0x7f2f380fc7d0 msgr2=0x7f2f380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/2452932032 shutdown_connections 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/2452932032 wait complete. 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 Processor -- start 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.573+0000 7f2f3d5fd640 1 -- start start 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102570 0x7f2f3819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38102ef0 0x7f2f3819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38110f50 con 0x7f2f38100e20 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38110dd0 con 0x7f2f38102ef0 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f3d5fd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381110d0 con 0x7f2f38102570 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58748/0 (socket says 172.21.15.16:58748) 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 -- 172.21.15.16:0/204143207 learned_addr learned my addr 172.21.15.16:0/204143207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102570 0x7f2f3819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f377fe640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38102ef0 0x7f2f3819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 -- 172.21.15.16:0/204143207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102570 msgr2=0x7f2f3819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102570 0x7f2f3819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 -- 172.21.15.16:0/204143207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38102ef0 msgr2=0x7f2f3819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38102ef0 0x7f2f3819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.574+0000 7f2f36ffd640 1 -- 172.21.15.16:0/204143207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3819f690 con 0x7f2f38100e20 2024-09-12T03:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9c0 tx=0x7f2f2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f34ff9640 1 -- 172.21.15.16:0/204143207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ccc0 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f34ff9640 1 -- 172.21.15.16:0/204143207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f28005500 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3819f980 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f34ff9640 1 -- 172.21.15.16:0/204143207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f28010430 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.575+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381a7260 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.577+0000 7f2f34ff9640 1 -- 172.21.15.16:0/204143207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2f280046c0 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.578+0000 7f2f34ff9640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 0x7f2f240a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.578+0000 7f2f3f9d6740 1 Processor -- start 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.578+0000 7f2f3f9d6740 1 -- start start 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.578+0000 7f2f34ff9640 1 -- 172.21.15.16:0/204143207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2f280cf270 con 0x7f2f38100e20 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.578+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 0x7f2f240a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd2480 0x55e807ddc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 0x55e807ddf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e807d37280 con 0x55e807dd1b60 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e807cdeb10 con 0x55e807ddcd40 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e807d33240 con 0x55e807dd2480 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58756/0 (socket says 172.21.15.16:58756) 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 -- 172.21.15.16:0/4129008842 learned_addr learned my addr 172.21.15.16:0/4129008842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd2480 0x55e807ddc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 0x7f2f240a50c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2f38104c00 tx=0x7f2f2002f300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 -- 172.21.15.16:0/4129008842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd2480 msgr2=0x55e807ddc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd2480 0x55e807ddc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 -- 172.21.15.16:0/4129008842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 msgr2=0x55e807ddf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f377fe640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 0x55e807ddf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.579+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 0x55e807ddf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f36ffd640 1 -- 172.21.15.16:0/4129008842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e807ceb1f0 con 0x55e807dd1b60 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f377fe640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 0x55e807ddf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2f280a4220 tx=0x7f2f28004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b767dab7a8f42a82 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/4129008842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280b3020 con 0x55e807dd1b60 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/4129008842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2809bae0 con 0x55e807dd1b60 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.580+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/4129008842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280a4520 con 0x55e807dd1b60 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/4129008842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 msgr2=0x55e807dd1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2f280a4220 tx=0x7f2f28004b00 comp rx=0 tx=0).stop 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/4129008842 shutdown_connections 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807ddcd40 0x55e807ddf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd2480 0x55e807ddc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/4129008842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807dd1f40 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/4129008842 >> 172.21.15.16:0/4129008842 conn(0x55e807dce0f0 msgr2=0x55e807dce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/4129008842 shutdown_connections 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/4129008842 wait complete. 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 -- start start 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.581+0000 7f2f3f9d6740 1 Processor -- start 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807dd9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807dda150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e807ddf910 con 0x55e807ddcd40 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e807cf1f10 con 0x55e807dd2480 2024-09-12T03:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e807ddfa90 con 0x55e807dd1b60 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 10 client.0 _ll_register_callbacks cb 0x55e807cbaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58760/0 (socket says 172.21.15.16:58760) 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 -- 172.21.15.16:0/587098941 learned_addr learned my addr 172.21.15.16:0/587098941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807dd9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807dda150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 -- 172.21.15.16:0/587098941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 msgr2=0x55e807dd9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807dd9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 -- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 msgr2=0x55e807dda150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807dda150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 -- 172.21.15.16:0/587098941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e807aa86a0 con 0x55e807ddcd40 2024-09-12T03:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807dda150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f377fe640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f2f30012d50 tx=0x7f2f300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63a2d76f218d81a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30004d20 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2f30003970 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.582+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f2809bae0 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f2f30003b70 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 10 client.20208 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e807deb4e0 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2809bae0 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f2f30019030 con 0x55e807ddcd40 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 10 client.20208 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 10 client.20208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 20 client.20208 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807dec9a0 0x55e807deee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 4 client.20208 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e807d447f0 con 0x55e807dec9a0 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.583+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807dec9a0 0x55e807deee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.584+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807dec9a0 0x55e807deee00 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f785105aa542f2 server_cookie=6edd16c2ae8fd5af in_seq=0 out_seq=0 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.584+0000 7f2f1e7fc640 10 client.20208 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/587098941 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e807d447f0 con 0x55e807dec9a0 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f1e7fc640 10 client.20208 handle_command_reply: tid=0 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 client.20208 shutdown 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 msgr2=0x55e807dda690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f2f30012d50 tx=0x7f2f300107a0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 shutdown_connections 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807dec9a0 0x55e807deee00 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807ddcd40 0x55e807dda690 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807dda150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/587098941 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807dd9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 >> 172.21.15.16:0/587098941 conn(0x55e807dce0f0 msgr2=0x55e807dcd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f1e7fc640 0 client.20208 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 shutdown_connections 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.585+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/587098941 wait complete. 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.586+0000 7f2f3f9d6740 20 client.20208 trim_cache size 0 max 16384 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.611+0000 7f2f3f9d6740 1 Processor -- start 2024-09-12T03:08:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.611+0000 7f2f3f9d6740 1 -- start start 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 0x55e807dd45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807ddba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807df4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381af770 con 0x55e807dd1b60 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381b12b0 con 0x55e807dd2480 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381afe40 con 0x55e807dce0f0 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 0x55e807dd45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807ddba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807ddba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58776/0 (socket says 172.21.15.16:58776) 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f2ffff640 1 -- 172.21.15.16:0/3691658581 learned_addr learned my addr 172.21.15.16:0/3691658581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 -- 172.21.15.16:0/3691658581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 msgr2=0x55e807df4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807df4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 -- 172.21.15.16:0/3691658581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 msgr2=0x55e807ddba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807ddba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.612+0000 7f2f36ffd640 1 -- 172.21.15.16:0/3691658581 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e807ddf470 con 0x55e807dce0f0 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.613+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 0x55e807dd45c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2f280a4220 tx=0x7f2f28009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8713403f657366f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.613+0000 7f2f2cff9640 1 -- 172.21.15.16:0/3691658581 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280b3040 con 0x55e807dce0f0 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.613+0000 7f2f2cff9640 1 -- 172.21.15.16:0/3691658581 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2807e070 con 0x55e807dce0f0 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.613+0000 7f2f2cff9640 1 -- 172.21.15.16:0/3691658581 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280a42c0 con 0x55e807dce0f0 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/3691658581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 msgr2=0x55e807dd45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 0x55e807dd45c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2f280a4220 tx=0x7f2f28009f90 comp rx=0 tx=0).stop 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/3691658581 shutdown_connections 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dd2480 0x55e807df4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd1b60 0x55e807ddba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/3691658581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dce0f0 0x55e807dd45c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/3691658581 >> 172.21.15.16:0/3691658581 conn(0x55e807dd5600 msgr2=0x55e807ddb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/3691658581 shutdown_connections 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.614+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/3691658581 wait complete. 2024-09-12T03:08:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 -- start start 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 Processor -- start 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dce0f0 0x55e807de0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 0x55e807de15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381b0be0 con 0x55e807dd2480 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381b0510 con 0x55e807dce0f0 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381ad750 con 0x55e807dd1b60 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.615+0000 7f2f3f9d6740 10 client.0 _ll_register_callbacks cb 0x7f2f30009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52748/0 (socket says 172.21.15.16:52748) 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f2ffff640 1 -- 172.21.15.16:0/421946907 learned_addr learned my addr 172.21.15.16:0/421946907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 0x55e807de15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dce0f0 0x55e807de0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 -- 172.21.15.16:0/421946907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 msgr2=0x55e807de1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 -- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dce0f0 msgr2=0x55e807de0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dce0f0 0x55e807de0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f377fe640 1 -- 172.21.15.16:0/421946907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e807e56850 con 0x55e807dd2480 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.616+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f377fe640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 0x55e807de15d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f2f3001a6b0 tx=0x7f2f3001aa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0477c70c98d74f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30029070 con 0x55e807dd2480 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f2f3001ab00 con 0x55e807dd2480 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f2807e070 con 0x55e807dd2480 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f2f30033050 con 0x55e807dd2480 2024-09-12T03:08:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f3f9d6740 10 client.20214 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.617+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e807e57b10 con 0x55e807dd2480 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2807e070 con 0x55e807dd2480 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f2f30019040 con 0x55e807dd2480 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 10 client.20214 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 10 client.20214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 20 client.20214 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807ddcd40 0x55e807e5a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 4 client.20214 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e807d3ed00 con 0x55e807ddcd40 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.618+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807ddcd40 0x55e807e5a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.619+0000 7f2f1e7fc640 10 client.20214 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.619+0000 7f2f36ffd640 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807ddcd40 0x55e807e5a2f0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb803827cfd087ad server_cookie=bf4c8fd779bfac35 in_seq=0 out_seq=0 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.620+0000 7f2f1e7fc640 1 -- 172.21.15.16:0/421946907 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e807d3ed00 con 0x55e807ddcd40 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f1e7fc640 10 client.20214 handle_command_reply: tid=0 2024-09-12T03:08:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 client.20214 shutdown 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 msgr2=0x55e807de15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 0x55e807de15d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f2f3001a6b0 tx=0x7f2f3001aa40 comp rx=0 tx=0).stop 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 shutdown_connections 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55e807ddcd40 0x55e807e5a2f0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e807dd2480 0x55e807de15d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e807dd1b60 0x55e807de1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 --2- 172.21.15.16:0/421946907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e807dce0f0 0x55e807de0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 >> 172.21.15.16:0/421946907 conn(0x55e807dd5600 msgr2=0x55e807dcd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f1e7fc640 0 client.20214 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 shutdown_connections 2024-09-12T03:08:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 1 -- 172.21.15.16:0/421946907 wait complete. 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "089318c0-9767-4494-b950-6102ee2e3885", 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.621+0000 7f2f3f9d6740 20 client.20214 trim_cache size 0 max 16384 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 msgr2=0x7f2f240a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 0x7f2f240a50c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2f38104c00 tx=0x7f2f2002f300 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 msgr2=0x7f2f3819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9c0 tx=0x7f2f2800ee80 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 shutdown_connections 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f38102ef0 0x7f2f3819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f38102570 0x7f2f3819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f2f240a2c00 0x7f2f240a50c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 --2- 172.21.15.16:0/204143207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f38100e20 0x7f2f3819a6e0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 >> 172.21.15.16:0/204143207 conn(0x7f2f380fc7d0 msgr2=0x7f2f3806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 shutdown_connections 2024-09-12T03:08:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:27.622+0000 7f2f3d5fd640 1 -- 172.21.15.16:0/204143207 wait complete. 2024-09-12T03:08:28.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 192695 24.92 MB/sec execute 329 sec latency 82.592 ms 2024-09-12T03:08:28.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:27 smithi016 ceph-mon[30925]: pgmap v953: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 693 op/s 2024-09-12T03:08:28.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:27 smithi033 ceph-mon[35092]: pgmap v953: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 693 op/s 2024-09-12T03:08:28.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:27 smithi184 ceph-mon[35321]: pgmap v953: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.5 MiB/s wr, 693 op/s 2024-09-12T03:08:28.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:08:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:08:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:28.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:08:28.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:28 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 193344 24.95 MB/sec execute 330 sec latency 50.306 ms 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:08:29.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:28 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:08:29.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:28 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 -- 172.21.15.16:0/1845537182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102570 msgr2=0x7f10a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/1845537182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102570 0x7f10a01029b0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f108c009900 tx=0x7f108c02a480 comp rx=0 tx=0).stop 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 -- 172.21.15.16:0/1845537182 shutdown_connections 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/1845537182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102ef0 0x7f10a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/1845537182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102570 0x7f10a01029b0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/1845537182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 -- 172.21.15.16:0/1845537182 >> 172.21.15.16:0/1845537182 conn(0x7f10a00fc7d0 msgr2=0x7f10a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 -- 172.21.15.16:0/1845537182 shutdown_connections 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.772+0000 7f10a6dbc640 1 -- 172.21.15.16:0/1845537182 wait complete. 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.773+0000 7f10a6dbc640 1 Processor -- start 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.773+0000 7f10a6dbc640 1 -- start start 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102570 0x7f10a0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0110f50 con 0x7f10a0102ef0 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0110dd0 con 0x7f10a0102570 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a6dbc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01110d0 con 0x7f10a0100e20 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58808/0 (socket says 172.21.15.16:58808) 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f109ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102570 0x7f10a0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 -- 172.21.15.16:0/2732497620 learned_addr learned my addr 172.21.15.16:0/2732497620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a01985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52768/0 (socket says 172.21.15.16:52768) 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 -- 172.21.15.16:0/2732497620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 msgr2=0x7f10a01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 -- 172.21.15.16:0/2732497620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102570 msgr2=0x7f10a0198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102570 0x7f10a0198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 -- 172.21.15.16:0/2732497620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a019d550 con 0x7f10a0102ef0 2024-09-12T03:08:29.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.774+0000 7f10a5359640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f109800eb10 tx=0x7f109800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.775+0000 7f109dffb640 1 -- 172.21.15.16:0/2732497620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098004b20 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.775+0000 7f109dffb640 1 -- 172.21.15.16:0/2732497620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1098004cc0 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.775+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a019d840 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.775+0000 7f109dffb640 1 -- 172.21.15.16:0/2732497620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098010760 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.775+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a01a5120 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.778+0000 7f10a9195740 1 Processor -- start 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.778+0000 7f109dffb640 1 -- 172.21.15.16:0/2732497620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1098005020 con 0x7f10a0102ef0 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.778+0000 7f10a9195740 1 -- start start 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b1d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b2aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db66a832e0 con 0x55db66b28680 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55db66a2ace0 con 0x55db66b1d4a0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a9195740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55db66a7f2a0 con 0x55db66b1ddc0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52782/0 (socket says 172.21.15.16:52782) 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 -- 172.21.15.16:0/2104428192 learned_addr learned my addr 172.21.15.16:0/2104428192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f109ffff640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b1d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f109ffff640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b2aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 -- 172.21.15.16:0/2104428192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 msgr2=0x55db66b1d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b1d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 -- 172.21.15.16:0/2104428192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 msgr2=0x55db66b2aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b2aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f10a4b58640 1 -- 172.21.15.16:0/2104428192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db66a0f440 con 0x55db66b1ddc0 2024-09-12T03:08:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f109ffff640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b2aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.779+0000 7f109dffb640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 0x7f10900a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f109dffb640 1 -- 172.21.15.16:0/2732497620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f10980cf4c0 con 0x7f10a0102ef0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f109ffff640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 0x7f10900a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f10a4b58640 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f109400bd10 tx=0x7f10940107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3394b00b7f117e08 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f108affd640 1 -- 172.21.15.16:0/2104428192 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094004030 con 0x55db66b1ddc0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f108affd640 1 -- 172.21.15.16:0/2104428192 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10940041d0 con 0x55db66b1ddc0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.780+0000 7f108affd640 1 -- 172.21.15.16:0/2104428192 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094011820 con 0x55db66b1ddc0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- 172.21.15.16:0/2104428192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 msgr2=0x55db66b28140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f109ffff640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 0x7f10900a50c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f108c009230 tx=0x7f108c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f109400bd10 tx=0x7f10940107a0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- 172.21.15.16:0/2104428192 shutdown_connections 2024-09-12T03:08:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b2aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b28140 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 --2- 172.21.15.16:0/2104428192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b1d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- 172.21.15.16:0/2104428192 >> 172.21.15.16:0/2104428192 conn(0x55db66b19a30 msgr2=0x55db66b19e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- 172.21.15.16:0/2104428192 shutdown_connections 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- 172.21.15.16:0/2104428192 wait complete. 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.781+0000 7f10a9195740 1 -- start start 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 Processor -- start 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b27fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b25280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55db66b2b250 con 0x55db66b28680 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55db66a3e050 con 0x55db66b1d4a0 2024-09-12T03:08:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55db66b2b3d0 con 0x55db66b1ddc0 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a9195740 10 client.0 _ll_register_callbacks cb 0x55db66a7e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58824/0 (socket says 172.21.15.16:58824) 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a5359640 1 -- 172.21.15.16:0/1066049603 learned_addr learned my addr 172.21.15.16:0/1066049603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f10a4b58640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b27fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.782+0000 7f109ffff640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b25280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 -- 172.21.15.16:0/1066049603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 msgr2=0x55db66b25280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b25280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 -- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 msgr2=0x55db66b27fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b27fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 -- 172.21.15.16:0/1066049603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db667f43c0 con 0x55db66b28680 2024-09-12T03:08:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a4b58640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b27fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:08:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a5359640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f10980a4470 tx=0x7f109809b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab60817dec7581 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10980cf2e0 con 0x55db66b28680 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f10980a4510 con 0x55db66b28680 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10940041d0 con 0x55db66b28680 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f10980a00e0 con 0x55db66b28680 2024-09-12T03:08:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a9195740 10 client.20223 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55db66b36b90 con 0x55db66b28680 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10940041d0 con 0x55db66b28680 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.783+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f10980b2070 con 0x55db66b28680 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 10 client.20223 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 10 client.20223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 20 client.20223 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66b38050 0x55db66b3a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 4 client.20223 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db66a90850 con 0x55db66b38050 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a4b58640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66b38050 0x55db66b3a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f10a4b58640 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66b38050 0x55db66b3a4c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=137a1d471e482127 server_cookie=f01d8bd4871f053 in_seq=0 out_seq=0 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.784+0000 7f107ffff640 10 client.20223 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f107ffff640 1 -- 172.21.15.16:0/1066049603 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55db66a90850 con 0x55db66b38050 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f107ffff640 10 client.20223 handle_command_reply: tid=0 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 client.20223 shutdown 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 msgr2=0x55db66b257e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f10980a4470 tx=0x7f109809b8f0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 shutdown_connections 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66b38050 0x55db66b3a4c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b257e0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1ddc0 0x55db66b25280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f107ffff640 0 client.20223 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 --2- 172.21.15.16:0/1066049603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1d4a0 0x55db66b27fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.785+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 >> 172.21.15.16:0/1066049603 conn(0x55db66b19a30 msgr2=0x55db66b1f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.786+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 shutdown_connections 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.786+0000 7f10a9195740 1 -- 172.21.15.16:0/1066049603 wait complete. 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.786+0000 7f10a9195740 20 client.20223 trim_cache size 0 max 16384 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.805+0000 7f10a9195740 1 Processor -- start 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.805+0000 7f10a9195740 1 -- start start 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b28680 0x55db66b3fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01ad540 con 0x55db66b1d4a0 2024-09-12T03:08:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01af080 con 0x55db66b1ddc0 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a9195740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01adc10 con 0x55db66b28680 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58832/0 (socket says 172.21.15.16:58832) 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f109ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 -- 172.21.15.16:0/894689861 learned_addr learned my addr 172.21.15.16:0/894689861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a5359640 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b28680 0x55db66b3fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 -- 172.21.15.16:0/894689861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b28680 msgr2=0x55db66b3fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b28680 0x55db66b3fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 -- 172.21.15.16:0/894689861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 msgr2=0x55db66b207f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b207f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 -- 172.21.15.16:0/894689861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db66b2add0 con 0x55db66b1d4a0 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.806+0000 7f10a4b58640 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f109400bc30 tx=0x7f1094011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15ef613bd62d572 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f108a7fc640 1 -- 172.21.15.16:0/894689861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094005240 con 0x55db66b1d4a0 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f108a7fc640 1 -- 172.21.15.16:0/894689861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10940053e0 con 0x55db66b1d4a0 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f108a7fc640 1 -- 172.21.15.16:0/894689861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094011870 con 0x55db66b1d4a0 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 -- 172.21.15.16:0/894689861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 msgr2=0x55db66b344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f109400bc30 tx=0x7f1094011440 comp rx=0 tx=0).stop 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 -- 172.21.15.16:0/894689861 shutdown_connections 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b28680 0x55db66b3fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b207f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 --2- 172.21.15.16:0/894689861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b1d4a0 0x55db66b344b0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 -- 172.21.15.16:0/894689861 >> 172.21.15.16:0/894689861 conn(0x55db66b19a30 msgr2=0x55db66b200e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 -- 172.21.15.16:0/894689861 shutdown_connections 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.807+0000 7f10a9195740 1 -- 172.21.15.16:0/894689861 wait complete. 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 -- start start 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 Processor -- start 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1d4a0 0x55db66b30720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b30c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01ae9b0 con 0x55db66b28680 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01ae2e0 con 0x55db66b1ddc0 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01ab520 con 0x55db66b1d4a0 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a9195740 10 client.0 _ll_register_callbacks cb 0x7f109809bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f109ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b30c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a5359640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58844/0 (socket says 172.21.15.16:58844) 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a5359640 1 -- 172.21.15.16:0/3704856126 learned_addr learned my addr 172.21.15.16:0/3704856126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f109ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b30c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:55452/0 (socket says 172.21.15.16:55452) 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.808+0000 7f10a4b58640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1d4a0 0x55db66b30720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 -- 172.21.15.16:0/3704856126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1d4a0 msgr2=0x55db66b30720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1d4a0 0x55db66b30720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 -- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 msgr2=0x55db66b30c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b30c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 -- 172.21.15.16:0/3704856126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db66b9bc20 con 0x55db66b28680 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a5359640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f10980cd340 tx=0x7f109807add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afefcd1811475298 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f107ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10980cf320 con 0x55db66b28680 2024-09-12T03:08:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f10980a4710 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10940053e0 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f10980a49a0 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a9195740 10 client.20229 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.809+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55db66b2e240 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10940053e0 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f10980b2070 con 0x55db66b28680 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 10 client.20229 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 10 client.20229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 20 client.20229 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66ba7460 0x55db66ba9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 4 client.20229 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db66a8ad60 con 0x55db66ba7460 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.810+0000 7f10a4b58640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66ba7460 0x55db66ba9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a4b58640 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66ba7460 0x55db66ba9850 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b0650600afeb97 server_cookie=b1594c7ef5f7dcd1 in_seq=0 out_seq=0 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f107ffff640 10 client.20229 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f107ffff640 1 -- 172.21.15.16:0/3704856126 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55db66a8ad60 con 0x55db66ba7460 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f107ffff640 10 client.20229 handle_command_reply: tid=0 2024-09-12T03:08:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 client.20229 shutdown 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "089318c0-9767-4494-b950-6102ee2e3885": { 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "089318c0-9767-4494-b950-6102ee2e3885", 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:29.899 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 msgr2=0x55db66b9b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f10980cd340 tx=0x7f109807add0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 shutdown_connections 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55db66ba7460 0x55db66ba9850 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55db66b28680 0x55db66b9b560 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55db66b1ddc0 0x55db66b30c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f107ffff640 0 client.20229 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 --2- 172.21.15.16:0/3704856126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55db66b1d4a0 0x55db66b30720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 >> 172.21.15.16:0/3704856126 conn(0x55db66b19a30 msgr2=0x55db66b23710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 shutdown_connections 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 1 -- 172.21.15.16:0/3704856126 wait complete. 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.811+0000 7f10a9195740 20 client.20229 trim_cache size 0 max 16384 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 msgr2=0x7f10900a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 0x7f10900a50c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f108c009230 tx=0x7f108c0058d0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 msgr2=0x7f10a019ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f109800eb10 tx=0x7f109800c2d0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 shutdown_connections 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f10900a2c00 0x7f10900a50c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a0102ef0 0x7f10a019ce70 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10a0102570 0x7f10a0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 --2- 172.21.15.16:0/2732497620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10a0100e20 0x7f10a01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 >> 172.21.15.16:0/2732497620 conn(0x7f10a00fc7d0 msgr2=0x7f10a006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 shutdown_connections 2024-09-12T03:08:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:29.812+0000 7f10a6dbc640 1 -- 172.21.15.16:0/2732497620 wait complete. 2024-09-12T03:08:30.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 194022 24.98 MB/sec execute 331 sec latency 54.192 ms 2024-09-12T03:08:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:29 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:30.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:29 smithi016 ceph-mon[30925]: pgmap v954: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 599 op/s 2024-09-12T03:08:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:29 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:30.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:29 smithi033 ceph-mon[35092]: pgmap v954: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 599 op/s 2024-09-12T03:08:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:29 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:08:30.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:29 smithi184 ceph-mon[35321]: pgmap v954: 97 pgs: 97 active+clean; 803 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 599 op/s 2024-09-12T03:08:30.928 INFO:tasks.cephfs.filesystem:scrub status for tag:089318c0-9767-4494-b950-6102ee2e3885 - {'path': '/', 'tag': '089318c0-9767-4494-b950-6102ee2e3885', 'options': 'recursive,force'} 2024-09-12T03:08:31.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 194586 24.98 MB/sec execute 332 sec latency 212.882 ms 2024-09-12T03:08:32.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 195309 25.02 MB/sec execute 333 sec latency 66.533 ms 2024-09-12T03:08:32.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:31 smithi016 ceph-mon[30925]: pgmap v955: 97 pgs: 97 active+clean; 793 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 789 op/s 2024-09-12T03:08:32.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:31 smithi033 ceph-mon[35092]: pgmap v955: 97 pgs: 97 active+clean; 793 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 789 op/s 2024-09-12T03:08:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:31 smithi184 ceph-mon[35321]: pgmap v955: 97 pgs: 97 active+clean; 793 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 789 op/s 2024-09-12T03:08:33.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 195961 25.10 MB/sec execute 334 sec latency 54.797 ms 2024-09-12T03:08:34.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 196430 25.10 MB/sec execute 335 sec latency 42.154 ms 2024-09-12T03:08:34.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:33 smithi016 ceph-mon[30925]: pgmap v956: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 691 op/s 2024-09-12T03:08:34.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:33 smithi033 ceph-mon[35092]: pgmap v956: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 691 op/s 2024-09-12T03:08:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:33 smithi184 ceph-mon[35321]: pgmap v956: 97 pgs: 97 active+clean; 786 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 691 op/s 2024-09-12T03:08:35.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 197148 25.12 MB/sec execute 336 sec latency 50.568 ms 2024-09-12T03:08:36.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 197681 25.12 MB/sec execute 337 sec latency 255.944 ms 2024-09-12T03:08:36.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:35 smithi016 ceph-mon[30925]: pgmap v957: 97 pgs: 97 active+clean; 789 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 699 op/s 2024-09-12T03:08:36.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:35 smithi033 ceph-mon[35092]: pgmap v957: 97 pgs: 97 active+clean; 789 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 699 op/s 2024-09-12T03:08:36.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:35 smithi184 ceph-mon[35321]: pgmap v957: 97 pgs: 97 active+clean; 789 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 699 op/s 2024-09-12T03:08:37.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 198177 25.13 MB/sec execute 338 sec latency 196.874 ms 2024-09-12T03:08:38.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 198931 25.17 MB/sec execute 339 sec latency 45.716 ms 2024-09-12T03:08:38.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:37 smithi016 ceph-mon[30925]: pgmap v958: 97 pgs: 97 active+clean; 796 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 735 op/s 2024-09-12T03:08:38.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:37 smithi033 ceph-mon[35092]: pgmap v958: 97 pgs: 97 active+clean; 796 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 735 op/s 2024-09-12T03:08:38.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:37 smithi184 ceph-mon[35321]: pgmap v958: 97 pgs: 97 active+clean; 796 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 735 op/s 2024-09-12T03:08:39.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 199817 25.22 MB/sec execute 340 sec latency 58.705 ms 2024-09-12T03:08:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:08:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:08:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:08:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:08:39.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:08:39.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:08:39.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:08:39.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:08:39.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:08:39.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:08:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:08:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:08:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:08:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:08:39.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:08:40.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 200395 25.27 MB/sec execute 341 sec latency 79.304 ms 2024-09-12T03:08:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:39 smithi016 ceph-mon[30925]: pgmap v959: 97 pgs: 97 active+clean; 798 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.1 MiB/s wr, 597 op/s 2024-09-12T03:08:40.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:39 smithi033 ceph-mon[35092]: pgmap v959: 97 pgs: 97 active+clean; 798 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.1 MiB/s wr, 597 op/s 2024-09-12T03:08:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:39 smithi184 ceph-mon[35321]: pgmap v959: 97 pgs: 97 active+clean; 798 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 8.1 MiB/s wr, 597 op/s 2024-09-12T03:08:41.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 200770 25.26 MB/sec execute 342 sec latency 337.070 ms 2024-09-12T03:08:42.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 201287 25.27 MB/sec execute 343 sec latency 354.529 ms 2024-09-12T03:08:42.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:41 smithi016 ceph-mon[30925]: pgmap v960: 97 pgs: 97 active+clean; 808 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 762 op/s 2024-09-12T03:08:42.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:41 smithi033 ceph-mon[35092]: pgmap v960: 97 pgs: 97 active+clean; 808 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 762 op/s 2024-09-12T03:08:42.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:41 smithi184 ceph-mon[35321]: pgmap v960: 97 pgs: 97 active+clean; 808 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 762 op/s 2024-09-12T03:08:43.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 201955 25.29 MB/sec execute 344 sec latency 75.971 ms 2024-09-12T03:08:43.442 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.9 2024-09-12T03:08:43.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:08:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:44.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 202443 25.29 MB/sec execute 345 sec latency 90.611 ms 2024-09-12T03:08:44.133 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:43 smithi016 ceph-mon[30925]: pgmap v961: 97 pgs: 97 active+clean; 807 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.3 MiB/s wr, 592 op/s 2024-09-12T03:08:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:43 smithi033 ceph-mon[35092]: pgmap v961: 97 pgs: 97 active+clean; 807 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.3 MiB/s wr, 592 op/s 2024-09-12T03:08:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:43 smithi184 ceph-mon[35321]: pgmap v961: 97 pgs: 97 active+clean; 807 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.3 MiB/s wr, 592 op/s 2024-09-12T03:08:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 -- 172.21.15.16:0/1640027587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd81027a0 msgr2=0x7fafd81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 --2- 172.21.15.16:0/1640027587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd81027a0 0x7fafd81067f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009900 tx=0x7fafcc02a410 comp rx=0 tx=0).stop 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 -- 172.21.15.16:0/1640027587 shutdown_connections 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 --2- 172.21.15.16:0/1640027587 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 0x7fafd810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 --2- 172.21.15.16:0/1640027587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd81027a0 0x7fafd81067f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 --2- 172.21.15.16:0/1640027587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd8101df0 0x7fafd81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 -- 172.21.15.16:0/1640027587 >> 172.21.15.16:0/1640027587 conn(0x7fafd80fd8b0 msgr2=0x7fafd80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 -- 172.21.15.16:0/1640027587 shutdown_connections 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.384+0000 7fafddbf1640 1 -- 172.21.15.16:0/1640027587 wait complete. 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 Processor -- start 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 -- start start 2024-09-12T03:08:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd81027a0 0x7fafd8074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 0x7fafd80753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111f80 con 0x7fafd8101df0 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111e00 con 0x7fafd81027a0 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafddbf1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8112100 con 0x7fafd8106f20 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44862/0 (socket says 172.21.15.16:44862) 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafd77fe640 1 -- 172.21.15.16:0/2184123856 learned_addr learned my addr 172.21.15.16:0/2184123856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafd6ffd640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd81027a0 0x7fafd8074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.385+0000 7fafd7fff640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 0x7fafd80753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 -- 172.21.15.16:0/2184123856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 msgr2=0x7fafd80753d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 0x7fafd80753d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 -- 172.21.15.16:0/2184123856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd81027a0 msgr2=0x7fafd8074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd81027a0 0x7fafd8074e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 -- 172.21.15.16:0/2184123856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd81aac60 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd77fe640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9f0 tx=0x7fafc800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc800cd80 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8010910 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81aaf50 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8018990 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.386+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81ab3a0 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.387+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd81021d0 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.389+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fafc8010ac0 con 0x7fafd8101df0 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.390+0000 7fafd4ff9640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 0x7fafc00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.390+0000 7fafd6ffd640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 0x7fafc00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.391+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fafc80cf6f0 con 0x7fafd8101df0 2024-09-12T03:08:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.391+0000 7fafd6ffd640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 0x7fafc00a5040 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009230 tx=0x7fafcc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.395+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fafc809c010 con 0x7fafd8101df0 2024-09-12T03:08:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.686+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd802d7c0 con 0x7fafd8101df0 2024-09-12T03:08:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.687+0000 7fafd4ff9640 1 -- 172.21.15.16:0/2184123856 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fafc80a0310 con 0x7fafd8101df0 2024-09-12T03:08:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:08:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:08:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 msgr2=0x7fafc00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 0x7fafc00a5040 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009230 tx=0x7fafcc0058d0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 msgr2=0x7fafd8074950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9f0 tx=0x7fafc800eeb0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 shutdown_connections 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.690+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd8106f20 0x7fafd80753d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd81027a0 0x7fafd8074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fafc00a2b80 0x7fafc00a5040 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 --2- 172.21.15.16:0/2184123856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd8101df0 0x7fafd8074950 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 >> 172.21.15.16:0/2184123856 conn(0x7fafd80fd8b0 msgr2=0x7fafd80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 shutdown_connections 2024-09-12T03:08:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:44.691+0000 7fafddbf1640 1 -- 172.21.15.16:0/2184123856 wait complete. 2024-09-12T03:08:45.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 203035 25.31 MB/sec execute 346 sec latency 75.039 ms 2024-09-12T03:08:45.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:44 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2184123856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:45.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:44 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2184123856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:45.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:44 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2184123856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:08:45.869 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0' 2024-09-12T03:08:45.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0 2024-09-12T03:08:46.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 203683 25.33 MB/sec execute 347 sec latency 195.764 ms 2024-09-12T03:08:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:45 smithi016 ceph-mon[30925]: pgmap v962: 97 pgs: 97 active+clean; 815 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.1 MiB/s wr, 657 op/s 2024-09-12T03:08:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:08:46.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:45 smithi033 ceph-mon[35092]: pgmap v962: 97 pgs: 97 active+clean; 815 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.1 MiB/s wr, 657 op/s 2024-09-12T03:08:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:45 smithi184 ceph-mon[35321]: pgmap v962: 97 pgs: 97 active+clean; 815 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.1 MiB/s wr, 657 op/s 2024-09-12T03:08:47.027 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204269 25.36 MB/sec execute 348 sec latency 231.157 ms 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 -- 172.21.15.16:0/509089183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670106f10 msgr2=0x7fa67010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 --2- 172.21.15.16:0/509089183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670106f10 0x7fa67010ea90 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa668009ca0 tx=0x7fa66802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 -- 172.21.15.16:0/509089183 shutdown_connections 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 --2- 172.21.15.16:0/509089183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670106f10 0x7fa67010ea90 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 --2- 172.21.15.16:0/509089183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670102790 0x7fa6701067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 --2- 172.21.15.16:0/509089183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670101de0 0x7fa6701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.908+0000 7fa676f9f640 1 -- 172.21.15.16:0/509089183 >> 172.21.15.16:0/509089183 conn(0x7fa6700fd8c0 msgr2=0x7fa6700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.909+0000 7fa676f9f640 1 -- 172.21.15.16:0/509089183 shutdown_connections 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.909+0000 7fa676f9f640 1 -- 172.21.15.16:0/509089183 wait complete. 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 Processor -- start 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 -- start start 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 0x7fa67010e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670102790 0x7fa67010ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 0x7fa67010bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa670075fc0 con 0x7fa670102790 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa670075e40 con 0x7fa670101de0 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.910+0000 7fa676f9f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa670076140 con 0x7fa670106f10 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa67553c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 0x7fa67010bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa67553c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 0x7fa67010bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36792/0 (socket says 172.21.15.16:36792) 2024-09-12T03:08:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa67553c640 1 -- 172.21.15.16:0/3464082907 learned_addr learned my addr 172.21.15.16:0/3464082907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa66ffff640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670102790 0x7fa67010ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 0x7fa67010e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 -- 172.21.15.16:0/3464082907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 msgr2=0x7fa67010bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 0x7fa67010bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 -- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670102790 msgr2=0x7fa67010ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670102790 0x7fa67010ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.911+0000 7fa674d3b640 1 -- 172.21.15.16:0/3464082907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6701aad60 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 0x7fa67010e9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c970 tx=0x7fa66400cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa66dffb640 1 -- 172.21.15.16:0/3464082907 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664007bb0 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6701aaf30 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa66dffb640 1 -- 172.21.15.16:0/3464082907 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66400ae40 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa66dffb640 1 -- 172.21.15.16:0/3464082907 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664004e00 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.913+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6701ab380 con 0x7fa670101de0 2024-09-12T03:08:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 Processor -- start 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 -- start start 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed686400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed688d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ed5e12a0 con 0x5589ed67b760 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.917+0000 7fa679378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589ed588b10 con 0x5589ed67c080 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa679378740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5589ed5dd260 con 0x5589ed686940 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa66dffb640 1 -- 172.21.15.16:0/3464082907 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa664005020 con 0x7fa670101de0 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa674d3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed686400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50362/0 (socket says 172.21.15.16:50362) 2024-09-12T03:08:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 -- 172.21.15.16:0/2178956529 learned_addr learned my addr 172.21.15.16:0/2178956529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 -- 172.21.15.16:0/2178956529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 msgr2=0x5589ed688d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed688d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 -- 172.21.15.16:0/2178956529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 msgr2=0x5589ed686400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed686400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.918+0000 7fa67553c640 1 -- 172.21.15.16:0/2178956529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ed352b90 con 0x5589ed67b760 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.919+0000 7fa67553c640 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa668031be0 tx=0x7fa668031c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3372c8ce1bf02d9e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.919+0000 7fa65affd640 1 -- 172.21.15.16:0/2178956529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668003b70 con 0x5589ed67b760 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.919+0000 7fa65affd640 1 -- 172.21.15.16:0/2178956529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa668036040 con 0x5589ed67b760 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.919+0000 7fa65affd640 1 -- 172.21.15.16:0/2178956529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6680045e0 con 0x5589ed67b760 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa66dffb640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 0x7fa6600a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa66dffb640 1 -- 172.21.15.16:0/3464082907 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa6640cebf0 con 0x7fa670101de0 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 -- 172.21.15.16:0/2178956529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 msgr2=0x5589ed67bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa668031be0 tx=0x7fa668031c10 comp rx=0 tx=0).stop 2024-09-12T03:08:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 -- 172.21.15.16:0/2178956529 shutdown_connections 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed688d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed686400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 --2- 172.21.15.16:0/2178956529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed67bb40 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.920+0000 7fa679378740 1 -- 172.21.15.16:0/2178956529 >> 172.21.15.16:0/2178956529 conn(0x5589ed677cf0 msgr2=0x5589ed6780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa679378740 1 -- 172.21.15.16:0/2178956529 shutdown_connections 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 0x7fa6600a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa679378740 1 -- 172.21.15.16:0/2178956529 wait complete. 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa679378740 1 -- start start 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa674d3b640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 0x7fa6600a5140 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa664097dd0 tx=0x7fa6640bb040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.921+0000 7fa679378740 1 Processor -- start 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed683c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50376/0 (socket says 172.21.15.16:50376) 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa67553c640 1 -- 172.21.15.16:0/147890357 learned_addr learned my addr 172.21.15.16:0/147890357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ed69a3a0 con 0x5589ed67b760 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589ed5696e0 con 0x5589ed67c080 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5589ed69a520 con 0x5589ed686940 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.922+0000 7fa66ffff640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed683c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa679378740 10 client.0 _ll_register_callbacks cb 0x5589ed58da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 -- 172.21.15.16:0/147890357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 msgr2=0x5589ed6841c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6841c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 -- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 msgr2=0x5589ed683c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed683c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ed6a3870 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa67553c640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa668004940 tx=0x7fa6680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30ab2a79a860f59 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa64ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668053060 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa668003970 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.923+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa668036040 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa668003b70 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 10 client.20247 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5589ed69a3a0 con 0x5589ed67b760 2024-09-12T03:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa668036040 con 0x5589ed67b760 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa668043020 con 0x5589ed67b760 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 10 client.20247 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 10 client.20247 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 20 client.20247 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6a72e0 0x5589ed6a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 4 client.20247 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589ed5ee9d0 con 0x5589ed6a72e0 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.924+0000 7fa66ffff640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6a72e0 0x5589ed6a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.925+0000 7fa64ffff640 10 client.20247 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.925+0000 7fa66ffff640 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6a72e0 0x5589ed6a96d0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd783825498296f server_cookie=ebc0558f1164ac4b in_seq=0 out_seq=0 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.926+0000 7fa64ffff640 1 -- 172.21.15.16:0/147890357 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5589ed5ee9d0 con 0x5589ed6a72e0 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.926+0000 7fa64ffff640 10 client.20247 handle_command_reply: tid=0 2024-09-12T03:08:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.926+0000 7fa679378740 1 client.20247 shutdown 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 msgr2=0x5589ed683740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa668004940 tx=0x7fa6680095c0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 shutdown_connections 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6a72e0 0x5589ed6a96d0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6841c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed683c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 --2- 172.21.15.16:0/147890357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed683740 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa64ffff640 0 client.20247 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 >> 172.21.15.16:0/147890357 conn(0x5589ed677cf0 msgr2=0x5589ed676e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 shutdown_connections 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 1 -- 172.21.15.16:0/147890357 wait complete. 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.927+0000 7fa679378740 20 client.20247 trim_cache size 0 max 16384 2024-09-12T03:08:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 Processor -- start 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 -- start start 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 0x5589ed682800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed684010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 0x5589ed67d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701ad360 con 0x5589ed686940 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa67011cb00 con 0x5589ed67c080 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa679378740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701acc90 con 0x5589ed67b760 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa67553c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 0x5589ed682800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 0x5589ed67d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa66ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed684010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 0x5589ed67d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50380/0 (socket says 172.21.15.16:50380) 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.955+0000 7fa67553c640 1 -- 172.21.15.16:0/2876152436 learned_addr learned my addr 172.21.15.16:0/2876152436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 -- 172.21.15.16:0/2876152436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 msgr2=0x5589ed684010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed684010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 -- 172.21.15.16:0/2876152436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 msgr2=0x5589ed67d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 0x5589ed67d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 -- 172.21.15.16:0/2876152436 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ed677b20 con 0x5589ed67b760 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa67553c640 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 0x5589ed682800 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa668004970 tx=0x7fa66802b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=725b1f0daf3ca00 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa65a7fc640 1 -- 172.21.15.16:0/2876152436 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66805e040 con 0x5589ed67b760 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa65a7fc640 1 -- 172.21.15.16:0/2876152436 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa668036030 con 0x5589ed67b760 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.956+0000 7fa65a7fc640 1 -- 172.21.15.16:0/2876152436 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668053040 con 0x5589ed67b760 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 -- 172.21.15.16:0/2876152436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 msgr2=0x5589ed682800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 0x5589ed682800 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa668004970 tx=0x7fa66802b040 comp rx=0 tx=0).stop 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 -- 172.21.15.16:0/2876152436 shutdown_connections 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed686940 0x5589ed67d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed684010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 --2- 172.21.15.16:0/2876152436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed67b760 0x5589ed682800 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.957+0000 7fa679378740 1 -- 172.21.15.16:0/2876152436 >> 172.21.15.16:0/2876152436 conn(0x5589ed677cf0 msgr2=0x5589ed67d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.958+0000 7fa679378740 1 -- 172.21.15.16:0/2876152436 shutdown_connections 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.958+0000 7fa679378740 1 -- 172.21.15.16:0/2876152436 wait complete. 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.958+0000 7fa679378740 1 -- start start 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.958+0000 7fa679378740 1 Processor -- start 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed680bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa67553c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50390/0 (socket says 172.21.15.16:50390) 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa67553c640 1 -- 172.21.15.16:0/3858109922 learned_addr learned my addr 172.21.15.16:0/3858109922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701abef0 con 0x5589ed67b760 2024-09-12T03:08:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701ab820 con 0x5589ed67c080 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701ac5c0 con 0x5589ed686940 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa679378740 10 client.0 _ll_register_callbacks cb 0x7fa6680046a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa66ffff640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed680bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.959+0000 7fa67553c640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 -- 172.21.15.16:0/3858109922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 msgr2=0x5589ed6810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6810f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 -- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 msgr2=0x5589ed680bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed680bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ed67e5d0 con 0x5589ed67b760 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa67553c640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fa66802b0b0 tx=0x7fa668004010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d26b8d0ddc6179f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.960+0000 7fa64ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66805e070 con 0x5589ed67b760 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa668004be0 con 0x5589ed67b760 2024-09-12T03:08:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa668036030 con 0x5589ed67b760 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa668053040 con 0x5589ed67b760 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa679378740 10 client.20253 fetch_fsmap learned FSMap version 39 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.961+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5589ed673260 con 0x5589ed67b760 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66806c030 con 0x5589ed67b760 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa66804b040 con 0x5589ed67b760 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 10 client.20253 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 10 client.20253 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 20 client.20253 populate_metadata read hostname 'smithi016' 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6915d0 0x5589ed6939f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 4 client.20253 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}] 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.962+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}) -- 0x7fa668036030 con 0x5589ed6915d0 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.963+0000 7fa66ffff640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6915d0 0x5589ed6939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.963+0000 7fa64ffff640 10 client.20253 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:47.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:46.963+0000 7fa66ffff640 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6915d0 0x5589ed6939f0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cd4d92980a7c431 server_cookie=5dedbfdbcfe9773 in_seq=0 out_seq=0 2024-09-12T03:08:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa64ffff640 1 -- 172.21.15.16:0/3858109922 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7fa668036030 con 0x5589ed6915d0 2024-09-12T03:08:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa64ffff640 10 client.20253 handle_command_reply: tid=0 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa679378740 1 client.20253 shutdown 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 msgr2=0x5589ed680670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fa66802b0b0 tx=0x7fa668004010 comp rx=0 tx=0).stop 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 shutdown_connections 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5589ed6915d0 0x5589ed6939f0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.687+0000 7fa64ffff640 0 client.20253 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5589ed686940 0x5589ed6810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ed67c080 0x5589ed680bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 --2- 172.21.15.16:0/3858109922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ed67b760 0x5589ed680670 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 >> 172.21.15.16:0/3858109922 conn(0x5589ed677cf0 msgr2=0x5589ed67d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 shutdown_connections 2024-09-12T03:08:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 1 -- 172.21.15.16:0/3858109922 wait complete. 2024-09-12T03:08:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.688+0000 7fa679378740 20 client.20253 trim_cache size 0 max 16384 2024-09-12T03:08:47.692 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:08:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 74, 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "39bb926f": { 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 74, 2024-09-12T03:08:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.9, 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:08:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:08:47.695 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.689+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 msgr2=0x7fa6600a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.689+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 0x7fa6600a5140 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa664097dd0 tx=0x7fa6640bb040 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.689+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 msgr2=0x7fa67010e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.689+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 0x7fa67010e9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c970 tx=0x7fa66400cd70 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 shutdown_connections 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa670106f10 0x7fa67010bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6600a2c80 0x7fa6600a5140 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa670102790 0x7fa67010ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 --2- 172.21.15.16:0/3464082907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa670101de0 0x7fa67010e9a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 >> 172.21.15.16:0/3464082907 conn(0x7fa6700fd8c0 msgr2=0x7fa6700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 shutdown_connections 2024-09-12T03:08:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:08:47.690+0000 7fa676f9f640 1 -- 172.21.15.16:0/3464082907 wait complete. 2024-09-12T03:08:48.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 25.28 MB/sec execute 349 sec latency 973.067 ms 2024-09-12T03:08:48.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:47 smithi016 ceph-mon[30925]: pgmap v963: 97 pgs: 97 active+clean; 817 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 10 MiB/s wr, 677 op/s 2024-09-12T03:08:48.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:47 smithi033 ceph-mon[35092]: pgmap v963: 97 pgs: 97 active+clean; 817 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 10 MiB/s wr, 677 op/s 2024-09-12T03:08:48.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:47 smithi184 ceph-mon[35321]: pgmap v963: 97 pgs: 97 active+clean; 817 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 10 MiB/s wr, 677 op/s 2024-09-12T03:08:48.524 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '39bb926f', quiesce took 5.1 seconds. Will release after: 14.799999999999999 2024-09-12T03:08:49.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 25.21 MB/sec execute 350 sec latency 1973.189 ms 2024-09-12T03:08:50.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 25.14 MB/sec execute 351 sec latency 2973.248 ms 2024-09-12T03:08:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:49 smithi016 ceph-mon[30925]: pgmap v964: 97 pgs: 97 active+clean; 818 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 9.1 MiB/s wr, 601 op/s 2024-09-12T03:08:50.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:49 smithi033 ceph-mon[35092]: pgmap v964: 97 pgs: 97 active+clean; 818 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 9.1 MiB/s wr, 601 op/s 2024-09-12T03:08:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:49 smithi184 ceph-mon[35321]: pgmap v964: 97 pgs: 97 active+clean; 818 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 9.1 MiB/s wr, 601 op/s 2024-09-12T03:08:51.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 25.07 MB/sec execute 352 sec latency 3973.304 ms 2024-09-12T03:08:52.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 25.00 MB/sec execute 353 sec latency 4973.376 ms 2024-09-12T03:08:52.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:51 smithi016 ceph-mon[30925]: pgmap v965: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.9 MiB/s wr, 713 op/s 2024-09-12T03:08:52.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:51 smithi033 ceph-mon[35092]: pgmap v965: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.9 MiB/s wr, 713 op/s 2024-09-12T03:08:52.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:51 smithi184 ceph-mon[35321]: pgmap v965: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.9 MiB/s wr, 713 op/s 2024-09-12T03:08:53.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.93 MB/sec execute 354 sec latency 5973.441 ms 2024-09-12T03:08:54.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.86 MB/sec execute 355 sec latency 6973.482 ms 2024-09-12T03:08:54.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:53 smithi016 ceph-mon[30925]: pgmap v966: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.2 MiB/s wr, 526 op/s 2024-09-12T03:08:54.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:53 smithi033 ceph-mon[35092]: pgmap v966: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.2 MiB/s wr, 526 op/s 2024-09-12T03:08:54.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:53 smithi184 ceph-mon[35321]: pgmap v966: 97 pgs: 97 active+clean; 800 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.2 MiB/s wr, 526 op/s 2024-09-12T03:08:55.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.79 MB/sec execute 356 sec latency 7973.574 ms 2024-09-12T03:08:56.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.72 MB/sec execute 357 sec latency 8973.646 ms 2024-09-12T03:08:56.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:55 smithi016 ceph-mon[30925]: pgmap v967: 97 pgs: 97 active+clean; 799 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 482 op/s 2024-09-12T03:08:56.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:55 smithi033 ceph-mon[35092]: pgmap v967: 97 pgs: 97 active+clean; 799 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 482 op/s 2024-09-12T03:08:56.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:55 smithi184 ceph-mon[35321]: pgmap v967: 97 pgs: 97 active+clean; 799 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 482 op/s 2024-09-12T03:08:57.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.65 MB/sec execute 358 sec latency 9973.708 ms 2024-09-12T03:08:58.028 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.58 MB/sec execute 359 sec latency 10973.815 ms 2024-09-12T03:08:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:57 smithi033 ceph-mon[35092]: pgmap v968: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-09-12T03:08:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:57 smithi184 ceph-mon[35321]: pgmap v968: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-09-12T03:08:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:57 smithi016 ceph-mon[30925]: pgmap v968: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 376 op/s 2024-09-12T03:08:59.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.51 MB/sec execute 360 sec latency 11973.938 ms 2024-09-12T03:09:00.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.44 MB/sec execute 361 sec latency 12974.050 ms 2024-09-12T03:09:00.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:08:59 smithi033 ceph-mon[35092]: pgmap v969: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-09-12T03:09:00.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:08:59 smithi184 ceph-mon[35321]: pgmap v969: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-09-12T03:09:00.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:08:59 smithi016 ceph-mon[30925]: pgmap v969: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-09-12T03:09:00.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:09:01.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.37 MB/sec execute 362 sec latency 13974.098 ms 2024-09-12T03:09:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 -- 172.21.15.16:0/1073589958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 msgr2=0x7f2300100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 --2- 172.21.15.16:0/1073589958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300100ee0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009930 tx=0x7f22f402a420 comp rx=0 tx=0).stop 2024-09-12T03:09:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 -- 172.21.15.16:0/1073589958 shutdown_connections 2024-09-12T03:09:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 --2- 172.21.15.16:0/1073589958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f230010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 --2- 172.21.15.16:0/1073589958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300100ee0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 --2- 172.21.15.16:0/1073589958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 0x7f2300106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 -- 172.21.15.16:0/1073589958 >> 172.21.15.16:0/1073589958 conn(0x7f23000fc7a0 msgr2=0x7f23000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 -- 172.21.15.16:0/1073589958 shutdown_connections 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.765+0000 7f230749b640 1 -- 172.21.15.16:0/1073589958 wait complete. 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 Processor -- start 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 -- start start 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f2300198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 0x7f230019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2300110e80 con 0x7f2300100a80 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2300110d00 con 0x7f2300106a80 2024-09-12T03:09:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f230749b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2300111000 con 0x7f2300101420 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50794/0 (socket says 172.21.15.16:50794) 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f2305237640 1 -- 172.21.15.16:0/3931408917 learned_addr learned my addr 172.21.15.16:0/3931408917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f2305a38640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 0x7f230019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.766+0000 7f2304a36640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f2300198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 -- 172.21.15.16:0/3931408917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 msgr2=0x7f2300198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f2300198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 -- 172.21.15.16:0/3931408917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 msgr2=0x7f230019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 0x7f230019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 -- 172.21.15.16:0/3931408917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230019d3e0 con 0x7f2300100a80 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2304a36640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f2300198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f2305237640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00d950 tx=0x7f22fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f22f27fc640 1 -- 172.21.15.16:0/3931408917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc017030 con 0x7f2300100a80 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230019d6d0 con 0x7f2300100a80 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f22f27fc640 1 -- 172.21.15.16:0/3931408917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22fc004020 con 0x7f2300100a80 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f22f27fc640 1 -- 172.21.15.16:0/3931408917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0047a0 con 0x7f2300100a80 2024-09-12T03:09:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.767+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23001a4fb0 con 0x7f2300100a80 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.769+0000 7f22f27fc640 1 -- 172.21.15.16:0/3931408917 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f22fc0041d0 con 0x7f2300100a80 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 Processor -- start 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 -- start start 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e6dc0 0x555f302f1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302f1680 0x555f302f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f3024c2e0 con 0x555f302f1680 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f301f3ce0 con 0x555f302e64a0 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2309874740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555f302482a0 con 0x555f302e6dc0 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2304a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302f1680 0x555f302f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60812/0 (socket says 172.21.15.16:60812) 2024-09-12T03:09:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 -- 172.21.15.16:0/122059875 learned_addr learned my addr 172.21.15.16:0/122059875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305237640 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e6dc0 0x555f302f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f22f27fc640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 0x7f22ec0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 -- 172.21.15.16:0/122059875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e6dc0 msgr2=0x555f302f1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e6dc0 0x555f302f1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 -- 172.21.15.16:0/122059875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302f1680 msgr2=0x555f302f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302f1680 0x555f302f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.770+0000 7f2305a38640 1 -- 172.21.15.16:0/122059875 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f301d8440 con 0x555f302e64a0 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f22f27fc640 1 -- 172.21.15.16:0/3931408917 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22fc0ce9c0 con 0x7f2300100a80 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f2305a38640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 0x7f22ec0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f2305a38640 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22f8010370 tx=0x7f22f8010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b487243a569d2ec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f22eb7fe640 1 -- 172.21.15.16:0/122059875 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8022070 con 0x555f302e64a0 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f22eb7fe640 1 -- 172.21.15.16:0/122059875 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f80113e0 con 0x555f302e64a0 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.771+0000 7f22eb7fe640 1 -- 172.21.15.16:0/122059875 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8016070 con 0x555f302e64a0 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- 172.21.15.16:0/122059875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 msgr2=0x555f302e6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22f8010370 tx=0x7f22f8010830 comp rx=0 tx=0).stop 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2305a38640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 0x7f22ec0a5040 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f22f801f000 tx=0x7f22f8022000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- 172.21.15.16:0/122059875 shutdown_connections 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302f1680 0x555f302f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e6dc0 0x555f302f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 --2- 172.21.15.16:0/122059875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e6880 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- 172.21.15.16:0/122059875 >> 172.21.15.16:0/122059875 conn(0x555f302e2a30 msgr2=0x555f302e2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- 172.21.15.16:0/122059875 shutdown_connections 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- 172.21.15.16:0/122059875 wait complete. 2024-09-12T03:09:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 -- start start 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.772+0000 7f2309874740 1 Processor -- start 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302ee220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302f1680 0x555f302ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f302f4250 con 0x555f302e64a0 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f30207050 con 0x555f302e6dc0 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555f302f43d0 con 0x555f302f1680 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2309874740 10 client.0 _ll_register_callbacks cb 0x555f30247160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50806/0 (socket says 172.21.15.16:50806) 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 -- 172.21.15.16:0/903779582 learned_addr learned my addr 172.21.15.16:0/903779582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2304a36640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 -- 172.21.15.16:0/903779582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302f1680 msgr2=0x555f302ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302f1680 0x555f302ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 -- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 msgr2=0x555f302ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 -- 172.21.15.16:0/903779582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2ffbd3c0 con 0x555f302e64a0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2304a36640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f2305237640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009610 tx=0x7f22fc09b0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b14374ccc24d119 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.773+0000 7f22e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc09b230 con 0x555f302e64a0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22fc0bb0c0 con 0x555f302e64a0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f80113e0 con 0x555f302e64a0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f22fc09f5e0 con 0x555f302e64a0 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 10 client.20265 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555f302ffbc0 con 0x555f302e64a0 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f80113e0 con 0x555f302e64a0 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f22fc09a680 con 0x555f302e64a0 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 10 client.20265 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 10 client.20265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 20 client.20265 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f30300fd0 0x555f30303420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 4 client.20265 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f30259850 con 0x555f30300fd0 2024-09-12T03:09:01.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.774+0000 7f2304a36640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f30300fd0 0x555f30303420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.775+0000 7f2304a36640 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f30300fd0 0x555f30303420 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f9b629c069f7aa server_cookie=3cb8b3d6916bc2bc in_seq=0 out_seq=0 2024-09-12T03:09:01.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.775+0000 7f22e8ff9640 10 client.20265 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:01.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.775+0000 7f22e8ff9640 1 -- 172.21.15.16:0/903779582 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555f30259850 con 0x555f30300fd0 2024-09-12T03:09:01.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.775+0000 7f22e8ff9640 10 client.20265 handle_command_reply: tid=0 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.775+0000 7f2309874740 1 client.20265 shutdown 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 msgr2=0x555f302f0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009610 tx=0x7f22fc09b0c0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 shutdown_connections 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f30300fd0 0x555f30303420 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302f1680 0x555f302ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 --2- 172.21.15.16:0/903779582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e64a0 0x555f302f0f70 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f22e8ff9640 0 client.20265 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 >> 172.21.15.16:0/903779582 conn(0x555f302e2a30 msgr2=0x555f302e8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 shutdown_connections 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 1 -- 172.21.15.16:0/903779582 wait complete. 2024-09-12T03:09:01.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.776+0000 7f2309874740 20 client.20265 trim_cache size 0 max 16384 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 Processor -- start 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 -- start start 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e2a30 0x555f302fd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 0x555f30308c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23001ad5f0 con 0x555f302e6dc0 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23001af130 con 0x555f302e64a0 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.793+0000 7f2309874740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23001adcc0 con 0x555f302e2a30 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2305a38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 0x555f30308c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60832/0 (socket says 172.21.15.16:60832) 2024-09-12T03:09:01.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 -- 172.21.15.16:0/1491103620 learned_addr learned my addr 172.21.15.16:0/1491103620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2305a38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 0x555f30308c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50822/0 (socket says 172.21.15.16:50822) 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2305237640 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e2a30 0x555f302fd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 -- 172.21.15.16:0/1491103620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e2a30 msgr2=0x555f302fd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e2a30 0x555f302fd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 -- 172.21.15.16:0/1491103620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 msgr2=0x555f30308c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 0x555f30308c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 -- 172.21.15.16:0/1491103620 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f302e2860 con 0x555f302e64a0 2024-09-12T03:09:01.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f2304a36640 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009900 tx=0x7f22f4009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5dbbd32c3593599 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f22eaffd640 1 -- 172.21.15.16:0/1491103620 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4035040 con 0x555f302e64a0 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f22eaffd640 1 -- 172.21.15.16:0/1491103620 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f4049070 con 0x555f302e64a0 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.794+0000 7f22eaffd640 1 -- 172.21.15.16:0/1491103620 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004b50 con 0x555f302e64a0 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- 172.21.15.16:0/1491103620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 msgr2=0x555f302e97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009900 tx=0x7f22f4009660 comp rx=0 tx=0).stop 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- 172.21.15.16:0/1491103620 shutdown_connections 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e6dc0 0x555f30308c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e64a0 0x555f302e97f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 --2- 172.21.15.16:0/1491103620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e2a30 0x555f302fd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- 172.21.15.16:0/1491103620 >> 172.21.15.16:0/1491103620 conn(0x555f302ee8c0 msgr2=0x555f302e90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- 172.21.15.16:0/1491103620 shutdown_connections 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- 172.21.15.16:0/1491103620 wait complete. 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 -- start start 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.795+0000 7f2309874740 1 Processor -- start 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 0x555f302f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302f5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23001aea60 con 0x555f302e2a30 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23001ae390 con 0x555f302e6dc0 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23001ab5d0 con 0x555f302e64a0 2024-09-12T03:09:01.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2309874740 10 client.0 _ll_register_callbacks cb 0x7f22fc005d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50826/0 (socket says 172.21.15.16:50826) 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 -- 172.21.15.16:0/1514125754 learned_addr learned my addr 172.21.15.16:0/1514125754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305a38640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2304a36640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 0x555f302f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 -- 172.21.15.16:0/1514125754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 msgr2=0x555f302f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 0x555f302f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 -- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 msgr2=0x555f302f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 -- 172.21.15.16:0/1514125754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f302f5a70 con 0x555f302e2a30 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2304a36640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 0x555f302f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f2305237640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f22fc097690 tx=0x7f22fc09a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce542180b19bf01 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:01.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.796+0000 7f22e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc09b230 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f22fc0bb2b0 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f4049070 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f22fc09a490 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 10 client.20271 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555f3036bac0 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f4049070 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f22fc09f0f0 con 0x555f302e2a30 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 10 client.20271 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 10 client.20271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 20 client.20271 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f302f1680 0x555f3036e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 4 client.20271 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.797+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f30253d60 con 0x555f302f1680 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2304a36640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f302f1680 0x555f3036e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f22e8ff9640 10 client.20271 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2304a36640 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f302f1680 0x555f3036e010 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f04b3513f20a6a8 server_cookie=8eaf3b05e9dd7fa3 in_seq=0 out_seq=0 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1514125754 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555f30253d60 con 0x555f302f1680 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f22e8ff9640 10 client.20271 handle_command_reply: tid=0 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 client.20271 shutdown 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 msgr2=0x555f302f9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f22fc097690 tx=0x7f22fc09a8b0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 shutdown_connections 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x555f302f1680 0x555f3036e010 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f302e6dc0 0x555f302f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f22e8ff9640 0 client.20271 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555f302e64a0 0x555f302f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 --2- 172.21.15.16:0/1514125754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f302e2a30 0x555f302f9bc0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 >> 172.21.15.16:0/1514125754 conn(0x555f302ee8c0 msgr2=0x555f302e1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 shutdown_connections 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.798+0000 7f2309874740 1 -- 172.21.15.16:0/1514125754 wait complete. 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f2309874740 20 client.20271 trim_cache size 0 max 16384 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:09:01.804 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 msgr2=0x7f22ec0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 0x7f22ec0a5040 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f22f801f000 tx=0x7f22f8022000 comp rx=0 tx=0).stop 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 msgr2=0x7f2300198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00d950 tx=0x7f22fc00de10 comp rx=0 tx=0).stop 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 shutdown_connections 2024-09-12T03:09:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2300106a80 0x7f230019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2300101420 0x7f2300198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f22ec0a2b80 0x7f22ec0a5040 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 --2- 172.21.15.16:0/3931408917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2300100a80 0x7f2300198430 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 >> 172.21.15.16:0/3931408917 conn(0x7f23000fc7a0 msgr2=0x7f230006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 shutdown_connections 2024-09-12T03:09:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:01.799+0000 7f230749b640 1 -- 172.21.15.16:0/3931408917 wait complete. 2024-09-12T03:09:02.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.31 MB/sec execute 363 sec latency 14974.170 ms 2024-09-12T03:09:02.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:01 smithi016 ceph-mon[30925]: pgmap v970: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 303 op/s 2024-09-12T03:09:02.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:01 smithi033 ceph-mon[35092]: pgmap v970: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 303 op/s 2024-09-12T03:09:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:01 smithi184 ceph-mon[35321]: pgmap v970: 97 pgs: 97 active+clean; 798 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 303 op/s 2024-09-12T03:09:02.532 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:09:02.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:09:02.789 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:03.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.24 MB/sec execute 364 sec latency 15974.253 ms 2024-09-12T03:09:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 -- 172.21.15.16:0/3920258433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 msgr2=0x7ff06c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 --2- 172.21.15.16:0/3920258433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c10e610 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff064009ca0 tx=0x7ff06402a950 comp rx=0 tx=0).stop 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 -- 172.21.15.16:0/3920258433 shutdown_connections 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 --2- 172.21.15.16:0/3920258433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c10e610 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 --2- 172.21.15.16:0/3920258433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 0x7ff06c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 --2- 172.21.15.16:0/3920258433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 -- 172.21.15.16:0/3920258433 >> 172.21.15.16:0/3920258433 conn(0x7ff06c0fc7a0 msgr2=0x7ff06c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 -- 172.21.15.16:0/3920258433 shutdown_connections 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.282+0000 7ff072065640 1 -- 172.21.15.16:0/3920258433 wait complete. 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 Processor -- start 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 -- start start 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 0x7ff06c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c111f70 con 0x7ff06c10c170 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c111df0 con 0x7ff06c103660 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff072065640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c1120f0 con 0x7ff06c102d40 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff06bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff06bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50844/0 (socket says 172.21.15.16:50844) 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff06bfff640 1 -- 172.21.15.16:0/2491296929 learned_addr learned my addr 172.21.15.16:0/2491296929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff06affd640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 0x7ff06c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.283+0000 7ff06b7fe640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 -- 172.21.15.16:0/2491296929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 msgr2=0x7ff06c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 -- 172.21.15.16:0/2491296929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 msgr2=0x7ff06c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 0x7ff06c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 -- 172.21.15.16:0/2491296929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c19f540 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06b7fe640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff06bfff640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff064006010 tx=0x7ff06402f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff064034040 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff064003e00 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c19f7d0 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0640456e0 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.284+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c1a7110 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.285+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c103120 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.286+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff064037020 con 0x7ff06c10c170 2024-09-12T03:09:03.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.287+0000 7ff068ff9640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 0x7ff0540a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:03.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.287+0000 7ff06b7fe640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 0x7ff0540a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:03.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.287+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff0640f01d0 con 0x7ff06c10c170 2024-09-12T03:09:03.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.288+0000 7ff06b7fe640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 0x7ff0540a50c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005fd0 tx=0x7ff05c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:03.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.290+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff0640bd0e0 con 0x7ff06c10c170 2024-09-12T03:09:03.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:09:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.504+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff06c1a72e0 con 0x7ff06c10c170 2024-09-12T03:09:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.505+0000 7ff068ff9640 1 -- 172.21.15.16:0/2491296929 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7ff0640bc9e0 con 0x7ff06c10c170 2024-09-12T03:09:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:09:03.506 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:09:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 msgr2=0x7ff0540a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 0x7ff0540a50c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005fd0 tx=0x7ff05c005ea0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 msgr2=0x7ff06c19ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff064006010 tx=0x7ff06402f710 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 shutdown_connections 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff0540a2c00 0x7ff0540a50c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff06c10c170 0x7ff06c19ee60 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff06c103660 0x7ff06c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 --2- 172.21.15.16:0/2491296929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff06c102d40 0x7ff06c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 >> 172.21.15.16:0/2491296929 conn(0x7ff06c0fc7a0 msgr2=0x7ff06c10d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 shutdown_connections 2024-09-12T03:09:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:03.508+0000 7ff072065640 1 -- 172.21.15.16:0/2491296929 wait complete. 2024-09-12T03:09:04.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.17 MB/sec execute 365 sec latency 16974.354 ms 2024-09-12T03:09:04.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:03 smithi016 ceph-mon[30925]: pgmap v971: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 5.2 MiB/s wr, 183 op/s 2024-09-12T03:09:04.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:03 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2491296929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:04.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:09:04.223 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:03 smithi033 ceph-mon[35092]: pgmap v971: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 5.2 MiB/s wr, 183 op/s 2024-09-12T03:09:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:03 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2491296929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:03 smithi184 ceph-mon[35321]: pgmap v971: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 5.2 MiB/s wr, 183 op/s 2024-09-12T03:09:04.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:03 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2491296929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.870+0000 7f51bf4bd640 1 -- 172.21.15.16:0/3629739976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b81081f0 msgr2=0x7f51b8108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.870+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/3629739976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b81081f0 0x7f51b8108610 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009930 tx=0x7f51a802a420 comp rx=0 tx=0).stop 2024-09-12T03:09:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 -- 172.21.15.16:0/3629739976 shutdown_connections 2024-09-12T03:09:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/3629739976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b8108b50 0x7f51b8115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/3629739976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b81081f0 0x7f51b8108610 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/3629739976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 -- 172.21.15.16:0/3629739976 >> 172.21.15.16:0/3629739976 conn(0x7f51b8077bb0 msgr2=0x7f51b8079fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 -- 172.21.15.16:0/3629739976 shutdown_connections 2024-09-12T03:09:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.872+0000 7f51bf4bd640 1 -- 172.21.15.16:0/3629739976 wait complete. 2024-09-12T03:09:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 Processor -- start 2024-09-12T03:09:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 -- start start 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 0x7f51b81a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8118580 con 0x7f51b8108b50 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8118400 con 0x7f51b81081f0 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.873+0000 7f51bf4bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8118700 con 0x7f51b810e230 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bd259640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 0x7f51b81a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bca58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bca58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50876/0 (socket says 172.21.15.16:50876) 2024-09-12T03:09:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bca58640 1 -- 172.21.15.16:0/4257189432 learned_addr learned my addr 172.21.15.16:0/4257189432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bd259640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 0x7f51b81a46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60886/0 (socket says 172.21.15.16:60886) 2024-09-12T03:09:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 -- 172.21.15.16:0/4257189432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 msgr2=0x7f51b81a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 0x7f51b81a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 -- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 msgr2=0x7f51b81a4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bda5a640 1 -- 172.21.15.16:0/4257189432 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b819efe0 con 0x7f51b810e230 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.874+0000 7f51bca58640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51bda5a640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b819e8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b000cd80 con 0x7f51b810e230 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b0005020 con 0x7f51b810e230 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0010720 con 0x7f51b810e230 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b819f2d0 con 0x7f51b810e230 2024-09-12T03:09:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.875+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b81ab5c0 con 0x7f51b810e230 2024-09-12T03:09:04.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.876+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b8108610 con 0x7f51b810e230 2024-09-12T03:09:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.878+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f51b00051d0 con 0x7f51b810e230 2024-09-12T03:09:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.879+0000 7f51ae7fc640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 0x7f51a40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.879+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f51b00cf640 con 0x7f51b810e230 2024-09-12T03:09:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.880+0000 7f51bd259640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 0x7f51a40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.880+0000 7f51bd259640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 0x7f51a40a5040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f51b819f9f0 tx=0x7f51b4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:04.884+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f51b009c5b0 con 0x7f51b810e230 2024-09-12T03:09:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:05.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.11 MB/sec execute 366 sec latency 17974.402 ms 2024-09-12T03:09:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.102+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51b802d7c0 con 0x7f51b810e230 2024-09-12T03:09:05.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.103+0000 7f51ae7fc640 1 -- 172.21.15.16:0/4257189432 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f51b009beb0 con 0x7f51b810e230 2024-09-12T03:09:05.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:09:05.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:09:05.104 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:09:05.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 msgr2=0x7f51a40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:05.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 0x7f51a40a5040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f51b819f9f0 tx=0x7f51b4005f90 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 msgr2=0x7f51b819e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b819e8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b000ee60 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 shutdown_connections 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51b810e230 0x7f51b819e8b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f51a40a2b80 0x7f51a40a5040 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51b8108b50 0x7f51b81a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 --2- 172.21.15.16:0/4257189432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51b81081f0 0x7f51b81a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.105+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 >> 172.21.15.16:0/4257189432 conn(0x7f51b8077bb0 msgr2=0x7f51b80793e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.106+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 shutdown_connections 2024-09-12T03:09:05.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:05.106+0000 7f51bf4bd640 1 -- 172.21.15.16:0/4257189432 wait complete. 2024-09-12T03:09:05.891 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 39bb926f --release --await' 2024-09-12T03:09:05.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id 39bb926f --release --await 2024-09-12T03:09:06.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 24.04 MB/sec execute 367 sec latency 18974.450 ms 2024-09-12T03:09:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/585147615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 msgr2=0x7f8a041021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/585147615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a041021d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009900 tx=0x7f8a0002a410 comp rx=0 tx=0).stop 2024-09-12T03:09:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/585147615 shutdown_connections 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/585147615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 0x7f8a0410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/585147615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/585147615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a041021d0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/585147615 >> 172.21.15.16:0/585147615 conn(0x7f8a040fd8f0 msgr2=0x7f8a040ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/585147615 shutdown_connections 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.088+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/585147615 wait complete. 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 Processor -- start 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 -- start start 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 0x7f8a0419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a04111f80 con 0x7f8a04101df0 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a04111e00 con 0x7f8a041027a0 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0c5f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a04112100 con 0x7f8a04106f20 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.089+0000 7f8a0a38e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39960/0 (socket says 172.21.15.16:39960) 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 -- 172.21.15.16:0/1353125862 learned_addr learned my addr 172.21.15.16:0/1353125862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 0x7f8a0419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 -- 172.21.15.16:0/1353125862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 msgr2=0x7f8a0419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 0x7f8a0419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 -- 172.21.15.16:0/1353125862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 msgr2=0x7f8a041989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 -- 172.21.15.16:0/1353125862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0419d410 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.090+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8a00006010 tx=0x7f8a000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.091+0000 7f89fb7fe640 1 -- 172.21.15.16:0/1353125862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0003f030 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.091+0000 7f89fb7fe640 1 -- 172.21.15.16:0/1353125862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a00003ce0 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.091+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0419d6a0 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.091+0000 7f89fb7fe640 1 -- 172.21.15.16:0/1353125862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00035070 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.091+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a04074420 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.093+0000 7f89fb7fe640 1 -- 172.21.15.16:0/1353125862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8a00042030 con 0x7f8a04101df0 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.093+0000 7f8a0e9cb740 1 Processor -- start 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.093+0000 7f8a0e9cb740 1 -- start start 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.093+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126ef6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b126e5c2f0 con 0x55b126f01370 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b126e03cf0 con 0x55b126ef65d0 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b126e582b0 con 0x55b126ef7b40 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39964/0 (socket says 172.21.15.16:39964) 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 -- 172.21.15.16:0/794705669 learned_addr learned my addr 172.21.15.16:0/794705669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f89fb7fe640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126ef6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 -- 172.21.15.16:0/794705669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 msgr2=0x55b126ef6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 -- 172.21.15.16:0/794705669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 msgr2=0x55b126ef6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126ef6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 -- 172.21.15.16:0/794705669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b126de83a0 con 0x55b126f01370 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126ef6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f89fb7fe640 1 -- 172.21.15.16:0/1353125862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8a000ef1a0 con 0x7f8a04101df0 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f89f400bd30 tx=0x7f89f400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b472067484aa61 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.094+0000 7f89f8ff9640 1 -- 172.21.15.16:0/794705669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4015070 con 0x55b126f01370 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009fd0 tx=0x7f89fc004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f89f8ff9640 1 -- 172.21.15.16:0/794705669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f400be20 con 0x55b126f01370 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f89f8ff9640 1 -- 172.21.15.16:0/794705669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f40107f0 con 0x55b126f01370 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/794705669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 msgr2=0x55b126f03760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f89f400bd30 tx=0x7f89f400f730 comp rx=0 tx=0).stop 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/794705669 shutdown_connections 2024-09-12T03:09:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126f03760 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126ef6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/794705669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/794705669 >> 172.21.15.16:0/794705669 conn(0x55b126ebf310 msgr2=0x55b126ef2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.095+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/794705669 shutdown_connections 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/794705669 wait complete. 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 -- start start 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 Processor -- start 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f00d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126efe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b126f04070 con 0x55b126ef7b40 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b126e17060 con 0x55b126f01370 2024-09-12T03:09:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b126f041f0 con 0x55b126ef65d0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0e9cb740 10 client.0 _ll_register_callbacks cb 0x55b126b6a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39970/0 (socket says 172.21.15.16:39970) 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a09b8d640 1 -- 172.21.15.16:0/3309986345 learned_addr learned my addr 172.21.15.16:0/3309986345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f00d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.096+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126efe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 -- 172.21.15.16:0/3309986345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 msgr2=0x55b126f00d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f00d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 -- 172.21.15.16:0/3309986345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 msgr2=0x55b126efe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126efe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 -- 172.21.15.16:0/3309986345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b126bcd3c0 con 0x55b126ef7b40 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f89f400fc30 tx=0x7f89f4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2238e241803991d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f89edffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4029060 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f89f40035f0 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f400be20 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f89f4015050 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a0e9cb740 10 client.20295 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b126f0fac0 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.097+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f400be20 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f89f4010a30 con 0x55b126ef7b40 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 10 client.20295 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 10 client.20295 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 20 client.20295 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f10ec0 0x55b126f132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 4 client.20295 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b126e69860 con 0x55b126f10ec0 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f10ec0 0x55b126f132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f89edffb640 10 client.20295 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.098+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f10ec0 0x55b126f132b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18fed54920ce9f0a server_cookie=e4668a41fe67797f in_seq=0 out_seq=0 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f89edffb640 1 -- 172.21.15.16:0/3309986345 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b126e69860 con 0x55b126f10ec0 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f89edffb640 10 client.20295 handle_command_reply: tid=0 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 client.20295 shutdown 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 msgr2=0x55b126efdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f89f400fc30 tx=0x7f89f4004040 comp rx=0 tx=0).stop 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 shutdown_connections 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f10ec0 0x55b126f132b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126efe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126efdfd0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/3309986345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f00d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 >> 172.21.15.16:0/3309986345 conn(0x55b126ebf310 msgr2=0x55b126ef8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f89edffb640 0 client.20295 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 shutdown_connections 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/3309986345 wait complete. 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.099+0000 7f8a0e9cb740 20 client.20295 trim_cache size 0 max 16384 2024-09-12T03:09:06.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.112+0000 7f8a0e9cb740 1 Processor -- start 2024-09-12T03:09:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 -- start start 2024-09-12T03:09:06.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:05 smithi016 ceph-mon[30925]: pgmap v972: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 175 op/s 2024-09-12T03:09:06.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:05 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4257189432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126efc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041af7f0 con 0x55b126f01370 2024-09-12T03:09:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041b1330 con 0x55b126ef65d0 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0e9cb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041afec0 con 0x55b126ef7b40 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0ab8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0ab8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39984/0 (socket says 172.21.15.16:39984) 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0ab8f640 1 -- 172.21.15.16:0/1945930926 learned_addr learned my addr 172.21.15.16:0/1945930926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126efc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.113+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 -- 172.21.15.16:0/1945930926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 msgr2=0x55b126ef9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 -- 172.21.15.16:0/1945930926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 msgr2=0x55b126efc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126efc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 -- 172.21.15.16:0/1945930926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b126efe8e0 con 0x55b126f01370 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126efc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f89fc013400 tx=0x7f89fc009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37282f2301bd44fe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f89effff640 1 -- 172.21.15.16:0/1945930926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc022bd0 con 0x55b126f01370 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f89effff640 1 -- 172.21.15.16:0/1945930926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc022d70 con 0x55b126f01370 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.114+0000 7f89effff640 1 -- 172.21.15.16:0/1945930926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc02a760 con 0x55b126f01370 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1945930926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 msgr2=0x55b126ef9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f89fc013400 tx=0x7f89fc009c80 comp rx=0 tx=0).stop 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1945930926 shutdown_connections 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126f01370 0x55b126ef9660 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef7b40 0x55b126ef9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1945930926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126ef65d0 0x55b126efc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1945930926 >> 172.21.15.16:0/1945930926 conn(0x55b126ebf310 msgr2=0x55b126f00050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1945930926 shutdown_connections 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1945930926 wait complete. 2024-09-12T03:09:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.115+0000 7f8a0e9cb740 1 -- start start 2024-09-12T03:09:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 Processor -- start 2024-09-12T03:09:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f05530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126f83060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a09b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39994/0 (socket says 172.21.15.16:39994) 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a09b8d640 1 -- 172.21.15.16:0/1618034747 learned_addr learned my addr 172.21.15.16:0/1618034747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041b0c60 con 0x55b126ef7b40 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041b0590 con 0x55b126f01370 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041ad7d0 con 0x55b126ef65d0 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0e9cb740 10 client.0 _ll_register_callbacks cb 0x7f89fc009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0ab8f640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126f83060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.116+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f05530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 -- 172.21.15.16:0/1618034747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 msgr2=0x55b126f05530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f05530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 -- 172.21.15.16:0/1618034747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 msgr2=0x55b126f83060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126f83060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b126f06320 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a09b8d640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f89f40059f0 tx=0x7f89f4005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cc298bee470295 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f89edffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4034020 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f89f4011ad0 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89fc022d70 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f89f4029040 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a0e9cb740 10 client.20301 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.117+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b126f84660 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc022d70 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f89f40109c0 con 0x55b126ef7b40 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 10 client.20301 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 10 client.20301 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 20 client.20301 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f84cf0 0x55b126f87110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 4 client.20301 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b126e63d70 con 0x55b126f84cf0 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.118+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f84cf0 0x55b126f87110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0a38e640 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f84cf0 0x55b126f87110 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7a812a9382218f0a server_cookie=d5846752792880da in_seq=0 out_seq=0 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f89edffb640 10 client.20301 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f89edffb640 1 -- 172.21.15.16:0/1618034747 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b126e63d70 con 0x55b126f84cf0 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f89edffb640 10 client.20301 handle_command_reply: tid=0 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 client.20301 shutdown 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 msgr2=0x55b126f05a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f89f40059f0 tx=0x7f89f4005690 comp rx=0 tx=0).stop 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 shutdown_connections 2024-09-12T03:09:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x55b126f84cf0 0x55b126f87110 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f89edffb640 0 client.20301 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b126f01370 0x55b126f83060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b126ef7b40 0x55b126f05a70 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 --2- 172.21.15.16:0/1618034747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b126ef65d0 0x55b126f05530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.119+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 >> 172.21.15.16:0/1618034747 conn(0x55b126ebf310 msgr2=0x55b126ef14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 shutdown_connections 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0e9cb740 1 -- 172.21.15.16:0/1618034747 wait complete. 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0e9cb740 20 client.20301 trim_cache size 0 max 16384 2024-09-12T03:09:06.124 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 msgr2=0x7f89f00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009fd0 tx=0x7f89fc004ab0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 msgr2=0x7f8a04198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.120+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8a00006010 tx=0x7f8a000042f0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 shutdown_connections 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a04106f20 0x7f8a0419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a041027a0 0x7f8a041989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f89f00a2b80 0x7f89f00a5040 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 --2- 172.21.15.16:0/1353125862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a04101df0 0x7f8a04198460 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 >> 172.21.15.16:0/1353125862 conn(0x7f8a040fd8f0 msgr2=0x7f8a040fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 shutdown_connections 2024-09-12T03:09:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:06.121+0000 7f8a0c5f2640 1 -- 172.21.15.16:0/1353125862 wait complete. 2024-09-12T03:09:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:05 smithi033 ceph-mon[35092]: pgmap v972: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 175 op/s 2024-09-12T03:09:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:05 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4257189432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:05 smithi184 ceph-mon[35321]: pgmap v972: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 175 op/s 2024-09-12T03:09:06.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:05 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4257189432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:06.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:09:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:07.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204273 23.98 MB/sec execute 368 sec latency 19974.495 ms 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- 172.21.15.16:0/760131558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730102ef0 msgr2=0x7f673010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 --2- 172.21.15.16:0/760131558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730102ef0 0x7f673010da40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f6728009ca0 tx=0x7f672802a950 comp rx=0 tx=0).stop 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- 172.21.15.16:0/760131558 shutdown_connections 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 --2- 172.21.15.16:0/760131558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730102ef0 0x7f673010da40 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 --2- 172.21.15.16:0/760131558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f67301029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 --2- 172.21.15.16:0/760131558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730100e20 0x7f6730101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- 172.21.15.16:0/760131558 >> 172.21.15.16:0/760131558 conn(0x7f67300fc7d0 msgr2=0x7f67300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- 172.21.15.16:0/760131558 shutdown_connections 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- 172.21.15.16:0/760131558 wait complete. 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 Processor -- start 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.399+0000 7f6736b74640 1 -- start start 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f6730199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730102ef0 0x7f67301a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6730110f80 con 0x7f6730100e20 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6730110e00 con 0x7f6730102570 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6730111100 con 0x7f6730102ef0 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40006/0 (socket says 172.21.15.16:40006) 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 -- 172.21.15.16:0/2036208010 learned_addr learned my addr 172.21.15.16:0/2036208010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6735111640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730102ef0 0x7f67301a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f672ffff640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f6730199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 -- 172.21.15.16:0/2036208010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730102ef0 msgr2=0x7f67301a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730102ef0 0x7f67301a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 -- 172.21.15.16:0/2036208010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 msgr2=0x7f6730199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f6730199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 -- 172.21.15.16:0/2036208010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67301a48b0 con 0x7f6730100e20 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f672ffff640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f6730199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6734910640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f672000c970 tx=0x7f672000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f672dffb640 1 -- 172.21.15.16:0/2036208010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720007bb0 con 0x7f6730100e20 2024-09-12T03:09:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67301a4a80 con 0x7f6730100e20 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f672dffb640 1 -- 172.21.15.16:0/2036208010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6720007d50 con 0x7f6730100e20 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.400+0000 7f672dffb640 1 -- 172.21.15.16:0/2036208010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672000f450 con 0x7f6730100e20 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.401+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67301a4f40 con 0x7f6730100e20 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.403+0000 7f672dffb640 1 -- 172.21.15.16:0/2036208010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6720005020 con 0x7f6730100e20 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 Processor -- start 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 -- start start 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121d3980 0x5633121ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121de240 0x5633121e0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563312138620 con 0x5633121d3060 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5633120dfe50 con 0x5633121d3980 2024-09-12T03:09:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6738f4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5633121345e0 con 0x5633121de240 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6735111640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f672dffb640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 0x7f67180a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6735111640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40012/0 (socket says 172.21.15.16:40012) 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6735111640 1 -- 172.21.15.16:0/258739153 learned_addr learned my addr 172.21.15.16:0/258739153 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6734910640 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121d3980 0x5633121ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f672ffff640 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121de240 0x5633121e0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f672dffb640 1 -- 172.21.15.16:0/2036208010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f67200cee70 con 0x7f6730100e20 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.404+0000 7f6735111640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 0x7f67180a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 -- 172.21.15.16:0/258739153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121de240 msgr2=0x5633121e0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121de240 0x5633121e0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 -- 172.21.15.16:0/258739153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121d3980 msgr2=0x5633121ddd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121d3980 0x5633121ddd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 -- 172.21.15.16:0/258739153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633120d0d40 con 0x5633121d3060 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f6728035000 tx=0x7f672802f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113dcd7e9c057567 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6716ffd640 1 -- 172.21.15.16:0/258739153 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728043050 con 0x5633121d3060 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6716ffd640 1 -- 172.21.15.16:0/258739153 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728044040 con 0x5633121d3060 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6716ffd640 1 -- 172.21.15.16:0/258739153 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672804d020 con 0x5633121d3060 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.405+0000 7f6735111640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 0x7f67180a5040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f672804e000 tx=0x7f672804c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- 172.21.15.16:0/258739153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 msgr2=0x5633121d3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f6728035000 tx=0x7f672802f420 comp rx=0 tx=0).stop 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- 172.21.15.16:0/258739153 shutdown_connections 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121de240 0x5633121e0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121d3980 0x5633121ddd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 --2- 172.21.15.16:0/258739153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3060 0x5633121d3440 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- 172.21.15.16:0/258739153 >> 172.21.15.16:0/258739153 conn(0x5633121cf5f0 msgr2=0x5633121cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- 172.21.15.16:0/258739153 shutdown_connections 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- 172.21.15.16:0/258739153 wait complete. 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 -- start start 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.406+0000 7f6738f4d740 1 Processor -- start 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121de240 0x5633121dbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5633121e0e10 con 0x5633121d3980 2024-09-12T03:09:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5633120c0f10 con 0x5633121de240 2024-09-12T03:09:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5633121e0f90 con 0x5633121d3060 2024-09-12T03:09:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6738f4d740 10 client.0 _ll_register_callbacks cb 0x5633120e4db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40026/0 (socket says 172.21.15.16:40026) 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 -- 172.21.15.16:0/668790149 learned_addr learned my addr 172.21.15.16:0/668790149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6735111640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121de240 0x5633121dbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6734910640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 -- 172.21.15.16:0/668790149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 msgr2=0x5633121db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121db090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 -- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121de240 msgr2=0x5633121dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121de240 0x5633121dbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 -- 172.21.15.16:0/668790149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563311e47580 con 0x5633121d3980 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f6734910640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f672ffff640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f6724009ec0 tx=0x7f67240117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f6ca3336cf279f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.407+0000 7f670ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6724012860 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6724012a00 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728044040 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6724012c00 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 10 client.20313 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563311f11960 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728044040 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f672400f040 con 0x5633121d3980 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 10 client.20313 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 10 client.20313 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 20 client.20313 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121eddd0 0x5633121f0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 4 client.20313 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.408+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563312145b90 con 0x5633121eddd0 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.409+0000 7f6734910640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121eddd0 0x5633121f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.409+0000 7f670ffff640 10 client.20313 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.409+0000 7f6734910640 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121eddd0 0x5633121f0270 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879561a1a22633cd server_cookie=1dce18881421512f in_seq=0 out_seq=0 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f670ffff640 1 -- 172.21.15.16:0/668790149 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563312145b90 con 0x5633121eddd0 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f670ffff640 10 client.20313 handle_command_reply: tid=0 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 client.20313 shutdown 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 msgr2=0x5633121db5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f6724009ec0 tx=0x7f67240117b0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 shutdown_connections 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121eddd0 0x5633121f0270 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121de240 0x5633121dbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121db5d0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 --2- 172.21.15.16:0/668790149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.410+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 >> 172.21.15.16:0/668790149 conn(0x5633121cf5f0 msgr2=0x5633121ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.411+0000 7f670ffff640 0 client.20313 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.412+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 shutdown_connections 2024-09-12T03:09:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.412+0000 7f6738f4d740 1 -- 172.21.15.16:0/668790149 wait complete. 2024-09-12T03:09:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.412+0000 7f6738f4d740 20 client.20313 trim_cache size 0 max 16384 2024-09-12T03:09:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 Processor -- start 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 -- start start 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121f6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ad520 con 0x5633121d3980 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67301af060 con 0x5633121cf5f0 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.434+0000 7f6738f4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f67301adbf0 con 0x5633121d3060 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33976/0 (socket says 172.21.15.16:33976) 2024-09-12T03:09:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 -- 172.21.15.16:0/414765187 learned_addr learned my addr 172.21.15.16:0/414765187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f672ffff640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6735111640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 -- 172.21.15.16:0/414765187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 msgr2=0x5633121f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 -- 172.21.15.16:0/414765187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 msgr2=0x5633121f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 -- 172.21.15.16:0/414765187 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633121cf420 con 0x5633121cf5f0 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6735111640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f6734910640 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6720007590 tx=0x7f672006e740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6949a824a056fc4b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f67167fc640 1 -- 172.21.15.16:0/414765187 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672001d040 con 0x5633121cf5f0 2024-09-12T03:09:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f67167fc640 1 -- 172.21.15.16:0/414765187 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f672009fa30 con 0x5633121cf5f0 2024-09-12T03:09:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.435+0000 7f67167fc640 1 -- 172.21.15.16:0/414765187 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720097cf0 con 0x5633121cf5f0 2024-09-12T03:09:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 -- 172.21.15.16:0/414765187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 msgr2=0x5633121ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6720007590 tx=0x7f672006e740 comp rx=0 tx=0).stop 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 -- 172.21.15.16:0/414765187 shutdown_connections 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x5633121f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 --2- 172.21.15.16:0/414765187 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121ce640 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 -- 172.21.15.16:0/414765187 >> 172.21.15.16:0/414765187 conn(0x5633121d6a80 msgr2=0x5633121cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 -- 172.21.15.16:0/414765187 shutdown_connections 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.436+0000 7f6738f4d740 1 -- 172.21.15.16:0/414765187 wait complete. 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 -- start start 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 Processor -- start 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121e2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x563312257b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ae990 con 0x5633121d3980 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ae2c0 con 0x5633121cf5f0 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f6738f4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ab500 con 0x5633121d3060 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.437+0000 7f672ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f6738f4d740 10 client.0 _ll_register_callbacks cb 0x7f6724009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35462/0 (socket says 172.21.15.16:35462) 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 -- 172.21.15.16:0/6040638 learned_addr learned my addr 172.21.15.16:0/6040638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f6735111640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x563312257b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f6734910640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 -- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 msgr2=0x5633121e2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121e2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 -- 172.21.15.16:0/6040638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 msgr2=0x563312257b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x563312257b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 -- 172.21.15.16:0/6040638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633121e31d0 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f672ffff640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5633121dc6d0 tx=0x7f6724005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c59cecf86cc51fd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f670ffff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6724034060 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.438+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f672009fa30 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.439+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6724019ac0 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.439+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6724029040 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.439+0000 7f6738f4d740 10 client.29464 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.439+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563312259160 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.439+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f672009fa30 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f67240053e0 con 0x5633121d3060 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 10 client.29464 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 10 client.29464 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:09:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 20 client.29464 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121de240 0x56331225b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 4 client.29464 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "39bb926f", "release": true, "await": true}] 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "39bb926f", "release": true, "await": true}) -- 0x5633121400a0 con 0x5633121de240 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6734910640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121de240 0x56331225b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f6734910640 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121de240 0x56331225b950 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86b454a63773bfb3 server_cookie=7ca32f6b08713e05 in_seq=0 out_seq=0 2024-09-12T03:09:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.440+0000 7f670ffff640 10 client.29464 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:07.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f670ffff640 1 -- 172.21.15.16:0/6040638 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f672006a260 con 0x5633121de240 2024-09-12T03:09:07.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f670ffff640 10 client.29464 handle_command_reply: tid=0 2024-09-12T03:09:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 client.29464 shutdown 2024-09-12T03:09:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 msgr2=0x5633121e28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5633121dc6d0 tx=0x7f6724005e50 comp rx=0 tx=0).stop 2024-09-12T03:09:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 shutdown_connections 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5633121de240 0x56331225b950 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5633121d3980 0x563312257b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5633121d3060 0x5633121e28d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 --2- 172.21.15.16:0/6040638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5633121cf5f0 0x5633121e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f670ffff640 0 client.29464 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 >> 172.21.15.16:0/6040638 conn(0x5633121d6a80 msgr2=0x5633121ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 shutdown_connections 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 1 -- 172.21.15.16:0/6040638 wait complete. 2024-09-12T03:09:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.495+0000 7f6738f4d740 20 client.29464 trim_cache size 0 max 16384 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 76, 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "39bb926f": { 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 76, 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:09:07.499 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.9, 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:09:07.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:07.501 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 msgr2=0x7f67180a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 0x7f67180a5040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f672804e000 tx=0x7f672804c000 comp rx=0 tx=0).stop 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 msgr2=0x7f67301992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f672000c970 tx=0x7f672000ce30 comp rx=0 tx=0).stop 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 shutdown_connections 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6730102ef0 0x7f67301a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6730102570 0x7f6730199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f67180a2b80 0x7f67180a5040 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 --2- 172.21.15.16:0/2036208010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6730100e20 0x7f67301992f0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 >> 172.21.15.16:0/2036208010 conn(0x7f67300fc7d0 msgr2=0x7f6730105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 shutdown_connections 2024-09-12T03:09:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:07.496+0000 7f6736b74640 1 -- 172.21.15.16:0/2036208010 wait complete. 2024-09-12T03:09:08.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 204531 23.93 MB/sec execute 369 sec latency 20493.713 ms 2024-09-12T03:09:08.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:07 smithi016 ceph-mon[30925]: pgmap v973: 97 pgs: 97 active+clean; 793 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 5.0 MiB/s wr, 195 op/s 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 -- 172.21.15.16:0/2356635638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581036e0 msgr2=0x7fe958107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 --2- 172.21.15.16:0/2356635638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581036e0 0x7fe958107b90 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe944009930 tx=0x7fe94402a420 comp rx=0 tx=0).stop 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 -- 172.21.15.16:0/2356635638 shutdown_connections 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 --2- 172.21.15.16:0/2356635638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581082a0 0x7fe95810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 --2- 172.21.15.16:0/2356635638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581036e0 0x7fe958107b90 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 --2- 172.21.15.16:0/2356635638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe958102d30 0x7fe958103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 -- 172.21.15.16:0/2356635638 >> 172.21.15.16:0/2356635638 conn(0x7fe9580fc7d0 msgr2=0x7fe9580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.179+0000 7fe95ed29640 1 -- 172.21.15.16:0/2356635638 shutdown_connections 2024-09-12T03:09:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.180+0000 7fe95ed29640 1 -- 172.21.15.16:0/2356635638 wait complete. 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.180+0000 7fe95ed29640 1 Processor -- start 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.180+0000 7fe95ed29640 1 -- start start 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe958102d30 0x7fe958198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581082a0 0x7fe95819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581120a0 con 0x7fe9581082a0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe958111f20 con 0x7fe958102d30 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95ed29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe958112220 con 0x7fe9581036e0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe95d2c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581082a0 0x7fe95819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35482/0 (socket says 172.21.15.16:35482) 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 -- 172.21.15.16:0/602308445 learned_addr learned my addr 172.21.15.16:0/602308445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 -- 172.21.15.16:0/602308445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe958102d30 msgr2=0x7fe958198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe958102d30 0x7fe958198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 -- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581082a0 msgr2=0x7fe95819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581082a0 0x7fe95819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.181+0000 7fe957fff640 1 -- 172.21.15.16:0/602308445 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95819d5f0 con 0x7fe9581036e0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe957fff640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe944005940 tx=0x7fe944031b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe955ffb640 1 -- 172.21.15.16:0/602308445 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94403f030 con 0x7fe9581036e0 2024-09-12T03:09:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95819d880 con 0x7fe9581036e0 2024-09-12T03:09:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe955ffb640 1 -- 172.21.15.16:0/602308445 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9440043d0 con 0x7fe9581036e0 2024-09-12T03:09:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe955ffb640 1 -- 172.21.15.16:0/602308445 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944035070 con 0x7fe9581036e0 2024-09-12T03:09:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.182+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9581a51c0 con 0x7fe9581036e0 2024-09-12T03:09:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.185+0000 7fe961102740 1 Processor -- start 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 -- start start 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe955ffb640 1 -- 172.21.15.16:0/602308445 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe944042030 con 0x7fe9581036e0 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 0x55f94a882a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a893b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f94a7e82f0 con 0x55f94a891780 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f94a78fcf0 con 0x55f94a8825d0 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.186+0000 7fe961102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f94a7e42b0 con 0x55f94a883b40 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35496/0 (socket says 172.21.15.16:35496) 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2371648010 learned_addr learned my addr 172.21.15.16:0/2371648010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 0x55f94a882a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe957fff640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a893b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2371648010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 msgr2=0x55f94a882a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 0x55f94a882a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2371648010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 msgr2=0x55f94a893b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a893b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2371648010 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94a7743a0 con 0x55f94a883b40 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.187+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 0x55f94a882a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe950009f50 tx=0x7fe9500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d4334713945581b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe942ffd640 1 -- 172.21.15.16:0/2371648010 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950004d20 con 0x55f94a883b40 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe942ffd640 1 -- 172.21.15.16:0/2371648010 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe950004050 con 0x55f94a883b40 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe942ffd640 1 -- 172.21.15.16:0/2371648010 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950005280 con 0x55f94a883b40 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe955ffb640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 0x7fe9480a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe955ffb640 1 -- 172.21.15.16:0/602308445 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9440ef010 con 0x7fe9581036e0 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.188+0000 7fe95cac5640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 0x7fe9480a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 -- 172.21.15.16:0/2371648010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 msgr2=0x55f94a882090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe950009f50 tx=0x7fe9500118d0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 -- 172.21.15.16:0/2371648010 shutdown_connections 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a893b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a8825d0 0x55f94a882a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 --2- 172.21.15.16:0/2371648010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a882090 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe961102740 1 -- 172.21.15.16:0/2371648010 >> 172.21.15.16:0/2371648010 conn(0x55f94a84b310 msgr2=0x55f94a87ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.189+0000 7fe95cac5640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 0x7fe9480a5040 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55f94a87fac0 tx=0x7fe94c009620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.190+0000 7fe961102740 1 -- 172.21.15.16:0/2371648010 shutdown_connections 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.190+0000 7fe961102740 1 -- 172.21.15.16:0/2371648010 wait complete. 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.190+0000 7fe961102740 1 -- start start 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.190+0000 7fe961102740 1 Processor -- start 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a88e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a891780 0x55f94a88e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f94a894480 con 0x55f94a8825d0 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f94a7a3060 con 0x55f94a891780 2024-09-12T03:09:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f94a894600 con 0x55f94a883b40 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe961102740 10 client.0 _ll_register_callbacks cb 0x55f94a4f68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe95cac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a88e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe95cac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40078/0 (socket says 172.21.15.16:40078) 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe95cac5640 1 -- 172.21.15.16:0/944940341 learned_addr learned my addr 172.21.15.16:0/944940341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a891780 0x55f94a88e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 -- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a891780 msgr2=0x55f94a88e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a891780 0x55f94a88e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 -- 172.21.15.16:0/944940341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 msgr2=0x55f94a891160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe957fff640 1 -- 172.21.15.16:0/944940341 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94a5593c0 con 0x55f94a883b40 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.191+0000 7fe95cac5640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe957fff640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a88e410 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe9440ecb70 tx=0x7fe9440b7400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34172c8bea7f3d33 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe937fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944049050 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9440b7ef0 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe950004050 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe9440bb3c0 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe961102740 10 client.29476 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f94a8944e0 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.192+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe950004050 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe94404c060 con 0x55f94a883b40 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 10 client.29476 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 10 client.29476 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 20 client.29476 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a8a3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 4 client.29476 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f94a7f5860 con 0x55f94a8a1380 2024-09-12T03:09:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.193+0000 7fe95cac5640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a8a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.194+0000 7fe95cac5640 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a8a3790 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eee1572fdcc48b0b server_cookie=dc2c5e32d90ac613 in_seq=0 out_seq=0 2024-09-12T03:09:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.194+0000 7fe937fff640 10 client.29476 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.196+0000 7fe937fff640 1 -- 172.21.15.16:0/944940341 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f94a7f5860 con 0x55f94a8a1380 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.196+0000 7fe937fff640 10 client.29476 handle_command_reply: tid=0 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.196+0000 7fe961102740 1 client.29476 shutdown 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 msgr2=0x55f94a88e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a88e410 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe9440ecb70 tx=0x7fe9440b7400 comp rx=0 tx=0).stop 2024-09-12T03:09:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 shutdown_connections 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a8a3790 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a891780 0x55f94a88e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a883b40 0x55f94a88e410 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe937fff640 0 client.29476 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 --2- 172.21.15.16:0/944940341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a891160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 >> 172.21.15.16:0/944940341 conn(0x55f94a84b310 msgr2=0x55f94a884960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 shutdown_connections 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 1 -- 172.21.15.16:0/944940341 wait complete. 2024-09-12T03:09:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.197+0000 7fe961102740 20 client.29476 trim_cache size 0 max 16384 2024-09-12T03:09:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 Processor -- start 2024-09-12T03:09:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 -- start start 2024-09-12T03:09:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a8a9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a891780 0x55f94a8a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581ab320 con 0x55f94a8825d0 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581ace60 con 0x55f94a883b40 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe961102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581ab9f0 con 0x55f94a891780 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe95cac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe95cac5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40086/0 (socket says 172.21.15.16:40086) 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe95cac5640 1 -- 172.21.15.16:0/2133568775 learned_addr learned my addr 172.21.15.16:0/2133568775 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe957fff640 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a8a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.218+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a891780 0x55f94a8a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 -- 172.21.15.16:0/2133568775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a891780 msgr2=0x55f94a8a95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a891780 0x55f94a8a95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 -- 172.21.15.16:0/2133568775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 msgr2=0x55f94a8a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a8a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 -- 172.21.15.16:0/2133568775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94a8914d0 con 0x55f94a8825d0 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe94c011ca0 tx=0x7fe94c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddec6bc22184848 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe9427fc640 1 -- 172.21.15.16:0/2133568775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c02e030 con 0x55f94a8825d0 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe9427fc640 1 -- 172.21.15.16:0/2133568775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe94c005510 con 0x55f94a8825d0 2024-09-12T03:09:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.219+0000 7fe9427fc640 1 -- 172.21.15.16:0/2133568775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c02b690 con 0x55f94a8825d0 2024-09-12T03:09:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 -- 172.21.15.16:0/2133568775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 msgr2=0x55f94a89d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe94c011ca0 tx=0x7fe94c018000 comp rx=0 tx=0).stop 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 -- 172.21.15.16:0/2133568775 shutdown_connections 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a891780 0x55f94a8a95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a8a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 --2- 172.21.15.16:0/2133568775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a8825d0 0x55f94a89d720 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 -- 172.21.15.16:0/2133568775 >> 172.21.15.16:0/2133568775 conn(0x55f94a84b310 msgr2=0x55f94a88cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 -- 172.21.15.16:0/2133568775 shutdown_connections 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.220+0000 7fe961102740 1 -- 172.21.15.16:0/2133568775 wait complete. 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 -- start start 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 Processor -- start 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 0x55f94a899990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a899ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581ac790 con 0x55f94a891780 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581ac0c0 con 0x55f94a883b40 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581a9300 con 0x55f94a8825d0 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe961102740 10 client.0 _ll_register_callbacks cb 0x7fe944005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe95d2c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe95cac5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 0x55f94a899990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe95d2c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40100/0 (socket says 172.21.15.16:40100) 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2893688720 learned_addr learned my addr 172.21.15.16:0/2893688720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.221+0000 7fe957fff640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a899ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2893688720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 msgr2=0x55f94a899990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 0x55f94a899990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 msgr2=0x55f94a899ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a899ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 -- 172.21.15.16:0/2893688720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94a8866e0 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 0x55f94a899990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe95d2c6640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe950009ef0 tx=0x7fe950004050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbbfb3315e30a06 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.222+0000 7fe937fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950004770 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe950004910 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe94c005510 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe950004b30 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 10 client.20334 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f94a897520 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe94c005510 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe95000f480 con 0x55f94a891780 2024-09-12T03:09:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 10 client.20334 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 10 client.20334 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 20 client.20334 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a9129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 4 client.20334 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.223+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f94a7efd70 con 0x55f94a8a1380 2024-09-12T03:09:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.226+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a9129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.227+0000 7fe95cac5640 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a9129f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=515df4ef19807425 server_cookie=f14eb4db8750639b in_seq=0 out_seq=0 2024-09-12T03:09:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.227+0000 7fe937fff640 10 client.20334 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe937fff640 1 -- 172.21.15.16:0/2893688720 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f94a7efd70 con 0x55f94a8a1380 2024-09-12T03:09:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe937fff640 10 client.20334 handle_command_reply: tid=0 2024-09-12T03:09:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 client.20334 shutdown 2024-09-12T03:09:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 msgr2=0x55f94a885f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe950009ef0 tx=0x7fe950004050 comp rx=0 tx=0).stop 2024-09-12T03:09:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 shutdown_connections 2024-09-12T03:09:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55f94a8a1380 0x55f94a9129f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f94a891780 0x55f94a885f70 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe937fff640 0 client.20334 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.229+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f94a883b40 0x55f94a899ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.230+0000 7fe961102740 1 --2- 172.21.15.16:0/2893688720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f94a8825d0 0x55f94a899990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.230+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 >> 172.21.15.16:0/2893688720 conn(0x55f94a84b310 msgr2=0x55f94a87d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.230+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 shutdown_connections 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.230+0000 7fe961102740 1 -- 172.21.15.16:0/2893688720 wait complete. 2024-09-12T03:09:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.230+0000 7fe961102740 20 client.20334 trim_cache size 0 max 16384 2024-09-12T03:09:08.234 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.231+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 msgr2=0x7fe9480a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.231+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 0x7fe9480a5040 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55f94a87fac0 tx=0x7fe94c009620 comp rx=0 tx=0).stop 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.231+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 msgr2=0x7fe958198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.231+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe944005940 tx=0x7fe944031b50 comp rx=0 tx=0).stop 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 shutdown_connections 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9480a2b80 0x7fe9480a5040 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9581082a0 0x7fe95819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9581036e0 0x7fe958198b80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 --2- 172.21.15.16:0/602308445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe958102d30 0x7fe958198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 >> 172.21.15.16:0/602308445 conn(0x7fe9580fc7d0 msgr2=0x7fe958106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 shutdown_connections 2024-09-12T03:09:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:08.232+0000 7fe95ed29640 1 -- 172.21.15.16:0/602308445 wait complete. 2024-09-12T03:09:08.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:07 smithi033 ceph-mon[35092]: pgmap v973: 97 pgs: 97 active+clean; 793 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 5.0 MiB/s wr, 195 op/s 2024-09-12T03:09:08.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:07 smithi184 ceph-mon[35321]: pgmap v973: 97 pgs: 97 active+clean; 793 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 5.0 MiB/s wr, 195 op/s 2024-09-12T03:09:08.855 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '39bb926f', total seconds elapsed: 25.4 2024-09-12T03:09:09.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 205074 23.96 MB/sec execute 370 sec latency 264.513 ms 2024-09-12T03:09:09.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:09:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:10.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 205822 24.01 MB/sec execute 371 sec latency 299.779 ms 2024-09-12T03:09:10.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:09 smithi016 ceph-mon[30925]: pgmap v974: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 183 op/s 2024-09-12T03:09:10.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:09 smithi033 ceph-mon[35092]: pgmap v974: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 183 op/s 2024-09-12T03:09:10.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:09 smithi184 ceph-mon[35321]: pgmap v974: 97 pgs: 97 active+clean; 795 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 183 op/s 2024-09-12T03:09:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 -- 172.21.15.16:0/1940759558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc10cf30 msgr2=0x7f5afc119410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1940759558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc10cf30 0x7f5afc119410 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009c10 tx=0x7f5af402a6e0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 -- 172.21.15.16:0/1940759558 shutdown_connections 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1940759558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc10cf30 0x7f5afc119410 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1940759558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10c590 0x7f5afc10c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1940759558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc112640 0x7f5afc112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 -- 172.21.15.16:0/1940759558 >> 172.21.15.16:0/1940759558 conn(0x7f5afc06cdc0 msgr2=0x7f5afc06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.483+0000 7f5b01c33640 1 -- 172.21.15.16:0/1940759558 shutdown_connections 2024-09-12T03:09:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.484+0000 7f5b01c33640 1 -- 172.21.15.16:0/1940759558 wait complete. 2024-09-12T03:09:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 Processor -- start 2024-09-12T03:09:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 -- start start 2024-09-12T03:09:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10cf30 0x7f5afc1a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 0x7f5afc1a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc11c950 con 0x7f5afc10c590 2024-09-12T03:09:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc11c7d0 con 0x7f5afc10cf30 2024-09-12T03:09:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5b01c33640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc11cad0 con 0x7f5afc112640 2024-09-12T03:09:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.485+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40122/0 (socket says 172.21.15.16:40122) 2024-09-12T03:09:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 -- 172.21.15.16:0/1699949020 learned_addr learned my addr 172.21.15.16:0/1699949020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afbfff640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 0x7f5afc1a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afaffd640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10cf30 0x7f5afc1a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 -- 172.21.15.16:0/1699949020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 msgr2=0x7f5afc1a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 0x7f5afc1a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 -- 172.21.15.16:0/1699949020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10cf30 msgr2=0x7f5afc1a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10cf30 0x7f5afc1a32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 -- 172.21.15.16:0/1699949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc1a7d30 con 0x7f5afc10c590 2024-09-12T03:09:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afbfff640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 0x7f5afc1a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e9c0 tx=0x7f5aec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.486+0000 7f5af8ff9640 1 -- 172.21.15.16:0/1699949020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00ccc0 con 0x7f5afc10c590 2024-09-12T03:09:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.487+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5afc1a8020 con 0x7f5afc10c590 2024-09-12T03:09:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.487+0000 7f5af8ff9640 1 -- 172.21.15.16:0/1699949020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aec005500 con 0x7f5afc10c590 2024-09-12T03:09:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.487+0000 7f5af8ff9640 1 -- 172.21.15.16:0/1699949020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec010430 con 0x7f5afc10c590 2024-09-12T03:09:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.487+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5afc1af900 con 0x7f5afc10c590 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.489+0000 7f5af8ff9640 1 -- 172.21.15.16:0/1699949020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5aec0046c0 con 0x7f5afc10c590 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.489+0000 7f5b0400c740 1 Processor -- start 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.489+0000 7f5b0400c740 1 -- start start 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb147a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb154760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fdb0ad2f0 con 0x558fdb1475d0 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558fdb054cf0 con 0x558fdb148b40 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5b0400c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558fdb0a92b0 con 0x558fdb152370 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5af8ff9640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 0x7f5ae40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb147a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40130/0 (socket says 172.21.15.16:40130) 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afb7fe640 1 -- 172.21.15.16:0/2507566001 learned_addr learned my addr 172.21.15.16:0/2507566001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34054/0 (socket says 172.21.15.16:34054) 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5af8ff9640 1 -- 172.21.15.16:0/1699949020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5aec0cf320 con 0x7f5afc10c590 2024-09-12T03:09:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 -- 172.21.15.16:0/2507566001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 msgr2=0x558fdb147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 -- 172.21.15.16:0/2507566001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 msgr2=0x558fdb147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afaffd640 1 -- 172.21.15.16:0/2507566001 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdb0393a0 con 0x558fdb152370 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.490+0000 7f5afbfff640 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.491+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb154760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009bd0 tx=0x7f5af0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82b87e55b1fc96d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.491+0000 7f5afbfff640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 0x7f5ae40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.491+0000 7f5ae1ffb640 1 -- 172.21.15.16:0/2507566001 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0004230 con 0x558fdb152370 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.491+0000 7f5ae1ffb640 1 -- 172.21.15.16:0/2507566001 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5af0009e40 con 0x558fdb152370 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.491+0000 7f5ae1ffb640 1 -- 172.21.15.16:0/2507566001 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0004db0 con 0x558fdb152370 2024-09-12T03:09:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- 172.21.15.16:0/2507566001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 msgr2=0x558fdb154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb154760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009bd0 tx=0x7f5af0011810 comp rx=0 tx=0).stop 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5afbfff640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 0x7f5ae40a5040 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5af4004af0 tx=0x7f5af4005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- 172.21.15.16:0/2507566001 shutdown_connections 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb154760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2507566001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- 172.21.15.16:0/2507566001 >> 172.21.15.16:0/2507566001 conn(0x558fdb110310 msgr2=0x558fdb143c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- 172.21.15.16:0/2507566001 shutdown_connections 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- 172.21.15.16:0/2507566001 wait complete. 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 -- start start 2024-09-12T03:09:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.492+0000 7f5b0400c740 1 Processor -- start 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fdb155070 con 0x558fdb1475d0 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558fdb068060 con 0x558fdb148b40 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558fdb1551f0 con 0x558fdb152370 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5b0400c740 10 client.0 _ll_register_callbacks cb 0x558fdadbb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40140/0 (socket says 172.21.15.16:40140) 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 -- 172.21.15.16:0/190040976 learned_addr learned my addr 172.21.15.16:0/190040976 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afaffd640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afbfff640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 -- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 msgr2=0x558fdb14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 -- 172.21.15.16:0/190040976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 msgr2=0x558fdb14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 -- 172.21.15.16:0/190040976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdae1e3c0 con 0x558fdb1475d0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000c00 tx=0x7f5aec097f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ab79d756ed4862 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.493+0000 7f5adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0056b0 con 0x558fdb1475d0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5aec0962b0 con 0x558fdb1475d0 2024-09-12T03:09:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558fdb0ba860 con 0x558fdb1475d0 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5aec09c290 con 0x558fdb1475d0 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5b0400c740 10 client.20346 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558fdb169330 con 0x558fdb1475d0 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.494+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558fdb0ba860 con 0x558fdb1475d0 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5aec0ce070 con 0x558fdb1475d0 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 10 client.20346 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 10 client.20346 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 20 client.20346 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb16d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 4 client.20346 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fdb0b4d70 con 0x558fdb16abc0 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5afaffd640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb16d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5adaffd640 10 client.20346 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.495+0000 7f5afaffd640 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb16d030 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a6fdb3bc1872b3e server_cookie=f46ac7f14b32b08 in_seq=0 out_seq=0 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5adaffd640 1 -- 172.21.15.16:0/190040976 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558fdb0b4d70 con 0x558fdb16abc0 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5adaffd640 10 client.20346 handle_command_reply: tid=0 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 client.20346 shutdown 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 msgr2=0x558fdb151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000c00 tx=0x7f5aec097f30 comp rx=0 tx=0).stop 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 shutdown_connections 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb16d030 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5adaffd640 0 client.20346 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 --2- 172.21.15.16:0/190040976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb151d20 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 >> 172.21.15.16:0/190040976 conn(0x558fdb110310 msgr2=0x558fdb149960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 shutdown_connections 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 1 -- 172.21.15.16:0/190040976 wait complete. 2024-09-12T03:09:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.497+0000 7f5b0400c740 20 client.20346 trim_cache size 0 max 16384 2024-09-12T03:09:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.517+0000 7f5b0400c740 1 Processor -- start 2024-09-12T03:09:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.517+0000 7f5b0400c740 1 -- start start 2024-09-12T03:09:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b3a60 con 0x558fdb1475d0 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b55a0 con 0x558fdb148b40 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5b0400c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b4130 con 0x558fdb152370 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34070/0 (socket says 172.21.15.16:34070) 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 -- 172.21.15.16:0/2935839168 learned_addr learned my addr 172.21.15.16:0/2935839168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afbfff640 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 -- 172.21.15.16:0/2935839168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 msgr2=0x558fdb14a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 -- 172.21.15.16:0/2935839168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 msgr2=0x558fdb166ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb166ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.518+0000 7f5afaffd640 1 -- 172.21.15.16:0/2935839168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdb154ad0 con 0x558fdb148b40 2024-09-12T03:09:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.519+0000 7f5afaffd640 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5af0004920 tx=0x7f5af0009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1d34fb968e47040 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.519+0000 7f5ae17fa640 1 -- 172.21.15.16:0/2935839168 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af001e040 con 0x558fdb148b40 2024-09-12T03:09:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.519+0000 7f5ae17fa640 1 -- 172.21.15.16:0/2935839168 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5af00043e0 con 0x558fdb148b40 2024-09-12T03:09:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.519+0000 7f5ae17fa640 1 -- 172.21.15.16:0/2935839168 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0024860 con 0x558fdb148b40 2024-09-12T03:09:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 -- 172.21.15.16:0/2935839168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 msgr2=0x558fdb146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5af0004920 tx=0x7f5af0009b00 comp rx=0 tx=0).stop 2024-09-12T03:09:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 -- 172.21.15.16:0/2935839168 shutdown_connections 2024-09-12T03:09:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb14a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb148b40 0x558fdb146ed0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 --2- 172.21.15.16:0/2935839168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb1475d0 0x558fdb166ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.520+0000 7f5b0400c740 1 -- 172.21.15.16:0/2935839168 >> 172.21.15.16:0/2935839168 conn(0x558fdb110310 msgr2=0x558fdb14e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.521+0000 7f5b0400c740 1 -- 172.21.15.16:0/2935839168 shutdown_connections 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.521+0000 7f5b0400c740 1 -- 172.21.15.16:0/2935839168 wait complete. 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.521+0000 7f5b0400c740 1 -- start start 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.522+0000 7f5b0400c740 1 Processor -- start 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.522+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.522+0000 7f5b0400c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 0x558fdb15e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34078/0 (socket says 172.21.15.16:34078) 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afb7fe640 1 -- 172.21.15.16:0/344029090 learned_addr learned my addr 172.21.15.16:0/344029090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb1d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b4ed0 con 0x558fdb148b40 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afaffd640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 0x558fdb15e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b4800 con 0x558fdb1475d0 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc1b1a60 con 0x558fdb152370 2024-09-12T03:09:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afbfff640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb1d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5b0400c740 10 client.0 _ll_register_callbacks cb 0x7f5aec004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.523+0000 7f5afb7fe640 1 -- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 msgr2=0x558fdb1d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb1d38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afb7fe640 1 -- 172.21.15.16:0/344029090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 msgr2=0x558fdb15e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 0x558fdb15e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afb7fe640 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdb1d3e90 con 0x558fdb1475d0 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afaffd640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 0x558fdb15e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5afb7fe640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aec004ab0 tx=0x7f5aec005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89187718a6064ec0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0056b0 con 0x558fdb1475d0 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5aec005200 con 0x558fdb1475d0 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.524+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5af00043e0 con 0x558fdb1475d0 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.525+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5aec09b6d0 con 0x558fdb1475d0 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.525+0000 7f5b0400c740 10 client.29573 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.525+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558fdb1d5130 con 0x558fdb1475d0 2024-09-12T03:09:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.525+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aec09bb90 con 0x558fdb1475d0 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5aec0ce040 con 0x558fdb1475d0 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 10 client.29573 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 10 client.29573 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 20 client.29573 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb1d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 4 client.29573 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f5af00043e0 con 0x558fdb16abc0 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.526+0000 7f5afaffd640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb1d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.527+0000 7f5afaffd640 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb1d77f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1829f53474cddf9 server_cookie=597d4f1bfdb5947c in_seq=0 out_seq=0 2024-09-12T03:09:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.527+0000 7f5adaffd640 10 client.29573 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.531+0000 7f5adaffd640 1 -- 172.21.15.16:0/344029090 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f5af00043e0 con 0x558fdb16abc0 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.531+0000 7f5adaffd640 10 client.29573 handle_command_reply: tid=0 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.531+0000 7f5b0400c740 1 client.29573 shutdown 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 msgr2=0x558fdb15e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5aec004ab0 tx=0x7f5aec005b70 comp rx=0 tx=0).stop 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 shutdown_connections 2024-09-12T03:09:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x558fdb16abc0 0x558fdb1d77f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5adaffd640 0 client.29573 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558fdb152370 0x558fdb1d38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fdb148b40 0x558fdb15e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 --2- 172.21.15.16:0/344029090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558fdb1475d0 0x558fdb15e270 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 >> 172.21.15.16:0/344029090 conn(0x558fdb110310 msgr2=0x558fdb150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 shutdown_connections 2024-09-12T03:09:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 1 -- 172.21.15.16:0/344029090 wait complete. 2024-09-12T03:09:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.532+0000 7f5b0400c740 20 client.29573 trim_cache size 0 max 16384 2024-09-12T03:09:10.535 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 msgr2=0x7f5ae40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 0x7f5ae40a5040 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5af4004af0 tx=0x7f5af4005bf0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 msgr2=0x7f5afc1a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e9c0 tx=0x7f5aec00ee80 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 shutdown_connections 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc112640 0x7f5afc1a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5afc10cf30 0x7f5afc1a32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5ae40a2b80 0x7f5ae40a5040 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 --2- 172.21.15.16:0/1699949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5afc10c590 0x7f5afc1a2d80 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 >> 172.21.15.16:0/1699949020 conn(0x7f5afc06cdc0 msgr2=0x7f5afc10b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:10.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 shutdown_connections 2024-09-12T03:09:10.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:10.533+0000 7f5b01c33640 1 -- 172.21.15.16:0/1699949020 wait complete. 2024-09-12T03:09:11.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 206441 24.04 MB/sec execute 372 sec latency 62.740 ms 2024-09-12T03:09:11.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:09:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:12.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 206820 24.02 MB/sec execute 373 sec latency 290.884 ms 2024-09-12T03:09:12.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:11 smithi016 ceph-mon[30925]: pgmap v975: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 301 op/s 2024-09-12T03:09:12.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:11 smithi033 ceph-mon[35092]: pgmap v975: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 301 op/s 2024-09-12T03:09:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 -- 172.21.15.16:0/156176258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab98102d30 msgr2=0x7fab98103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/156176258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab98102d30 0x7fab98103110 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fab88009930 tx=0x7fab8802a420 comp rx=0 tx=0).stop 2024-09-12T03:09:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 -- 172.21.15.16:0/156176258 shutdown_connections 2024-09-12T03:09:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/156176258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab981082a0 0x7fab9810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/156176258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/156176258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab98102d30 0x7fab98103110 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 -- 172.21.15.16:0/156176258 >> 172.21.15.16:0/156176258 conn(0x7fab980fc7d0 msgr2=0x7fab980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 -- 172.21.15.16:0/156176258 shutdown_connections 2024-09-12T03:09:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.274+0000 7fab9c8dd640 1 -- 172.21.15.16:0/156176258 wait complete. 2024-09-12T03:09:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.275+0000 7fab9c8dd640 1 Processor -- start 2024-09-12T03:09:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.275+0000 7fab9c8dd640 1 -- start start 2024-09-12T03:09:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab98102d30 0x7fab98100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 0x7fab98101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab981120e0 con 0x7fab981082a0 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab98111f60 con 0x7fab981036e0 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9c8dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab98112260 con 0x7fab98102d30 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 0x7fab98101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34098/0 (socket says 172.21.15.16:34098) 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab8ffff640 1 -- 172.21.15.16:0/3945264341 learned_addr learned my addr 172.21.15.16:0/3945264341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab9659c640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab98102d30 0x7fab98100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 -- 172.21.15.16:0/3945264341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab98102d30 msgr2=0x7fab98100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab98102d30 0x7fab98100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 -- 172.21.15.16:0/3945264341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 msgr2=0x7fab98100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 -- 172.21.15.16:0/3945264341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab981aaf50 con 0x7fab981082a0 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab8ffff640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.276+0000 7fab96d9d640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 0x7fab98101410 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fab90009fd0 tx=0x7fab9000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.277+0000 7fab8f7fe640 1 -- 172.21.15.16:0/3945264341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9000ce20 con 0x7fab981082a0 2024-09-12T03:09:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.277+0000 7fab8f7fe640 1 -- 172.21.15.16:0/3945264341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab90005020 con 0x7fab981082a0 2024-09-12T03:09:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.277+0000 7fab8f7fe640 1 -- 172.21.15.16:0/3945264341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90010720 con 0x7fab981082a0 2024-09-12T03:09:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.277+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab981ab180 con 0x7fab981082a0 2024-09-12T03:09:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.277+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981ab6c0 con 0x7fab981082a0 2024-09-12T03:09:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.280+0000 7fab8f7fe640 1 -- 172.21.15.16:0/3945264341 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fab90005540 con 0x7fab981082a0 2024-09-12T03:09:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.280+0000 7fab9ecb6740 1 Processor -- start 2024-09-12T03:09:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.280+0000 7fab9ecb6740 1 -- start start 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6bab40 0x55c66d6b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6c6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c66d61f2f0 con 0x55c66d6bab40 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c66d5c6cf0 con 0x55c66d6c4370 2024-09-12T03:09:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c66d61b2b0 con 0x55c66d6b95d0 2024-09-12T03:09:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6bab40 0x55c66d6b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab8f7fe640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 0x7fab840a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35594/0 (socket says 172.21.15.16:35594) 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 -- 172.21.15.16:0/2620118283 learned_addr learned my addr 172.21.15.16:0/2620118283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab8ffff640 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab8f7fe640 1 -- 172.21.15.16:0/3945264341 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fab900cf6f0 con 0x7fab981082a0 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 -- 172.21.15.16:0/2620118283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 msgr2=0x55c66d6c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 -- 172.21.15.16:0/2620118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6bab40 msgr2=0x55c66d6b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6bab40 0x55c66d6b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab9659c640 1 -- 172.21.15.16:0/2620118283 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c66d5ab3a0 con 0x55c66d6b95d0 2024-09-12T03:09:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.281+0000 7fab96d9d640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 0x7fab840a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.282+0000 7fab9659c640 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab88009930 tx=0x7fab88005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc620b38a0193e7f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.282+0000 7fab96d9d640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 0x7fab840a50c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fab900988d0 tx=0x7fab90005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.282+0000 7fab8cff9640 1 -- 172.21.15.16:0/2620118283 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88035040 con 0x55c66d6b95d0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.282+0000 7fab8cff9640 1 -- 172.21.15.16:0/2620118283 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab88005520 con 0x55c66d6b95d0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.282+0000 7fab8cff9640 1 -- 172.21.15.16:0/2620118283 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880035f0 con 0x55c66d6b95d0 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2620118283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 msgr2=0x55c66d6b9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab88009930 tx=0x7fab88005de0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2620118283 shutdown_connections 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6b9a10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2620118283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6bab40 0x55c66d6b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2620118283 >> 172.21.15.16:0/2620118283 conn(0x55c66d682310 msgr2=0x55c66d6b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2620118283 shutdown_connections 2024-09-12T03:09:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.283+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2620118283 wait complete. 2024-09-12T03:09:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 -- start start 2024-09-12T03:09:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 Processor -- start 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6c4370 0x55c66d6c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.284+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c66d6c7010 con 0x55c66d6c4370 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c66d5da060 con 0x55c66d6bab40 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c66d6c7190 con 0x55c66d6b95d0 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35606/0 (socket says 172.21.15.16:35606) 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 -- 172.21.15.16:0/2466376995 learned_addr learned my addr 172.21.15.16:0/2466376995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9659c640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6c4370 0x55c66d6c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9ecb6740 10 client.0 _ll_register_callbacks cb 0x55c66d32d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 -- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 msgr2=0x55c66d6c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 -- 172.21.15.16:0/2466376995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6c4370 msgr2=0x55c66d6c1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6c4370 0x55c66d6c1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 -- 172.21.15.16:0/2466376995 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c66d3903c0 con 0x55c66d6b95d0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9659c640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab88005520 con 0x55c66d6b95d0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab8ffff640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fab9819c6e0 tx=0x7fab80015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10866e519152fe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab7dffb640 10 client.29512 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab7dffb640 1 -- 172.21.15.16:0/2466376995 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab80004020 con 0x55c66d6b95d0 2024-09-12T03:09:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.285+0000 7fab7dffb640 1 -- 172.21.15.16:0/2466376995 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fab800041c0 con 0x55c66d6b95d0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab7dffb640 1 -- 172.21.15.16:0/2466376995 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fab800043c0 con 0x55c66d6b95d0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 10 client.29512 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55c66d6d2a40 con 0x55c66d6b95d0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab7dffb640 1 -- 172.21.15.16:0/2466376995 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fab8000f8c0 con 0x55c66d6b95d0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 10 client.29512 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 10 client.29512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 20 client.29512 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d6d62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 4 client.29512 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c66d62c860 con 0x55c66d6d3ee0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.286+0000 7fab9659c640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d6d62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.287+0000 7fab9659c640 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d6d62d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54ea552b75533ec6 server_cookie=f8202ac6af34032d in_seq=0 out_seq=0 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.287+0000 7fab7dffb640 10 client.29512 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.287+0000 7fab7dffb640 1 -- 172.21.15.16:0/2466376995 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c66d62c860 con 0x55c66d6d3ee0 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.287+0000 7fab7dffb640 10 client.29512 handle_command_reply: tid=0 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 client.29512 shutdown 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 msgr2=0x55c66d6c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fab9819c6e0 tx=0x7fab80015040 comp rx=0 tx=0).stop 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 shutdown_connections 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d6d62d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6c4370 0x55c66d6c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab7dffb640 0 client.29512 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/2466376995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6b95d0 0x55c66d6c3d90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 >> 172.21.15.16:0/2466376995 conn(0x55c66d682310 msgr2=0x55c66d6bb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 shutdown_connections 2024-09-12T03:09:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 1 -- 172.21.15.16:0/2466376995 wait complete. 2024-09-12T03:09:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.288+0000 7fab9ecb6740 20 client.29512 trim_cache size 0 max 16384 2024-09-12T03:09:12.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:11 smithi184 ceph-mon[35321]: pgmap v975: 97 pgs: 97 active+clean; 796 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 301 op/s 2024-09-12T03:09:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 Processor -- start 2024-09-12T03:09:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 -- start start 2024-09-12T03:09:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6c4370 0x55c66d6dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b3b20 con 0x55c66d6b95d0 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b5660 con 0x55c66d6bab40 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b41f0 con 0x55c66d6c4370 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34136/0 (socket says 172.21.15.16:34136) 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab9659c640 1 -- 172.21.15.16:0/1443443161 learned_addr learned my addr 172.21.15.16:0/1443443161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6c4370 0x55c66d6dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.304+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 -- 172.21.15.16:0/1443443161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6c4370 msgr2=0x55c66d6dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6c4370 0x55c66d6dba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 -- 172.21.15.16:0/1443443161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 msgr2=0x55c66d6b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 -- 172.21.15.16:0/1443443161 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c66d6c1c30 con 0x55c66d6bab40 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab9659c640 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab88004a60 tx=0x7fab88005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef44194682a22971 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab7ffff640 1 -- 172.21.15.16:0/1443443161 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88035040 con 0x55c66d6bab40 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab7ffff640 1 -- 172.21.15.16:0/1443443161 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab88004380 con 0x55c66d6bab40 2024-09-12T03:09:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.305+0000 7fab7ffff640 1 -- 172.21.15.16:0/1443443161 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88031810 con 0x55c66d6bab40 2024-09-12T03:09:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1443443161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 msgr2=0x55c66d6d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab88004a60 tx=0x7fab88005fd0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1443443161 shutdown_connections 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6c4370 0x55c66d6dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6bab40 0x55c66d6d02b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1443443161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.306+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1443443161 >> 172.21.15.16:0/1443443161 conn(0x55c66d682310 msgr2=0x55c66d6b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.307+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1443443161 shutdown_connections 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.307+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1443443161 wait complete. 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.307+0000 7fab9ecb6740 1 -- start start 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.307+0000 7fab9ecb6740 1 Processor -- start 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.307+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b4f90 con 0x55c66d6b95d0 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b48c0 con 0x55c66d6c4370 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b1b00 con 0x55c66d6bab40 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35624/0 (socket says 172.21.15.16:35624) 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 -- 172.21.15.16:0/1011048732 learned_addr learned my addr 172.21.15.16:0/1011048732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9ecb6740 10 client.0 _ll_register_callbacks cb 0x7fab80013550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 -- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 msgr2=0x55c66d6cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 -- 172.21.15.16:0/1011048732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 msgr2=0x55c66d6cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.308+0000 7fab9659c640 1 -- 172.21.15.16:0/1011048732 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c66d6bc120 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab9659c640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fab88031550 tx=0x7fab8802a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7034b2a966aa1cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880315b0 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab88004380 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fab88031750 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fab88068050 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab9ecb6740 10 client.29518 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c66d6b6dd0 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.309+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab88004380 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fab880035f0 con 0x55c66d6bab40 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 10 client.29518 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 10 client.29518 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 20 client.29518 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d751db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 4 client.29518 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c66d626d70 con 0x55c66d6d3ee0 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d751db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab7dffb640 10 client.29518 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.310+0000 7fab8ffff640 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d751db0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1fb3c253a6678a74 server_cookie=73b15a7cb293de0d in_seq=0 out_seq=0 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab7dffb640 1 -- 172.21.15.16:0/1011048732 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c66d626d70 con 0x55c66d6d3ee0 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab7dffb640 10 client.29518 handle_command_reply: tid=0 2024-09-12T03:09:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 client.29518 shutdown 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 msgr2=0x55c66d6cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fab88031550 tx=0x7fab8802a420 comp rx=0 tx=0).stop 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 shutdown_connections 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55c66d6d3ee0 0x55c66d751db0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c66d6c4370 0x55c66d6cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c66d6bab40 0x55c66d6cb960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 --2- 172.21.15.16:0/1011048732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c66d6b95d0 0x55c66d6cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 >> 172.21.15.16:0/1011048732 conn(0x55c66d682310 msgr2=0x55c66d6bb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab7dffb640 0 client.29518 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 shutdown_connections 2024-09-12T03:09:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 1 -- 172.21.15.16:0/1011048732 wait complete. 2024-09-12T03:09:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.311+0000 7fab9ecb6740 20 client.29518 trim_cache size 0 max 16384 2024-09-12T03:09:12.319 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 msgr2=0x7fab840a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 0x7fab840a50c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fab900988d0 tx=0x7fab90005940 comp rx=0 tx=0).stop 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 msgr2=0x7fab98101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 0x7fab98101410 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fab90009fd0 tx=0x7fab9000ef10 comp rx=0 tx=0).stop 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 shutdown_connections 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab840a2c00 0x7fab840a50c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab981082a0 0x7fab98101410 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab981036e0 0x7fab98100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 --2- 172.21.15.16:0/3945264341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab98102d30 0x7fab98100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 >> 172.21.15.16:0/3945264341 conn(0x7fab980fc7d0 msgr2=0x7fab98106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 shutdown_connections 2024-09-12T03:09:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:12.312+0000 7fab9c8dd640 1 -- 172.21.15.16:0/3945264341 wait complete. 2024-09-12T03:09:13.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 207293 24.02 MB/sec execute 374 sec latency 235.108 ms 2024-09-12T03:09:13.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:09:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:14.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 208019 24.06 MB/sec execute 375 sec latency 86.373 ms 2024-09-12T03:09:14.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:13 smithi016 ceph-mon[30925]: pgmap v976: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.225+0000 7f7c20e57640 1 -- 172.21.15.16:0/424751004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c106a70 msgr2=0x7f7c1c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.225+0000 7f7c20e57640 1 --2- 172.21.15.16:0/424751004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c106a70 0x7f7c1c106e50 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009930 tx=0x7f7c0c02a420 comp rx=0 tx=0).stop 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 -- 172.21.15.16:0/424751004 shutdown_connections 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 --2- 172.21.15.16:0/424751004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c101410 0x7f7c1c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 --2- 172.21.15.16:0/424751004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c100a70 0x7f7c1c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 --2- 172.21.15.16:0/424751004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c106a70 0x7f7c1c106e50 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 -- 172.21.15.16:0/424751004 >> 172.21.15.16:0/424751004 conn(0x7f7c1c0fc790 msgr2=0x7f7c1c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 -- 172.21.15.16:0/424751004 shutdown_connections 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.226+0000 7f7c20e57640 1 -- 172.21.15.16:0/424751004 wait complete. 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 Processor -- start 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 -- start start 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 0x7f7c1c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c106a70 0x7f7c1c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c110e70 con 0x7f7c1c100a70 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c110cf0 con 0x7f7c1c106a70 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c20e57640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c110ff0 con 0x7f7c1c101410 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40236/0 (socket says 172.21.15.16:40236) 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 -- 172.21.15.16:0/1983396068 learned_addr learned my addr 172.21.15.16:0/1983396068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 0x7f7c1c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c106a70 0x7f7c1c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 -- 172.21.15.16:0/1983396068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 msgr2=0x7f7c1c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 0x7f7c1c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 -- 172.21.15.16:0/1983396068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c106a70 msgr2=0x7f7c1c19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c106a70 0x7f7c1c19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.227+0000 7f7c1a59c640 1 -- 172.21.15.16:0/1983396068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c19d450 con 0x7f7c1c100a70 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 0x7f7c1c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:14.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009230 tx=0x7f7c0c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/1983396068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c03f030 con 0x7f7c1c100a70 2024-09-12T03:09:14.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/1983396068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c004440 con 0x7f7c1c100a70 2024-09-12T03:09:14.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/1983396068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c035070 con 0x7f7c1c100a70 2024-09-12T03:09:14.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c19d6e0 con 0x7f7c1c100a70 2024-09-12T03:09:14.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.228+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1a5020 con 0x7f7c1c100a70 2024-09-12T03:09:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.232+0000 7f7c23230740 1 Processor -- start 2024-09-12T03:09:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.232+0000 7f7c23230740 1 -- start start 2024-09-12T03:09:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.232+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/1983396068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7c0c042030 con 0x7f7c1c100a70 2024-09-12T03:09:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.232+0000 7f7c23230740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc58f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.232+0000 7f7c23230740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c23230740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc59c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c23230740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc4f52f0 con 0x55a7bc59a370 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c23230740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc49ccf0 con 0x55a7bc590b40 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c23230740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc4f12b0 con 0x55a7bc58f5d0 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc58f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:35656/0 (socket says 172.21.15.16:35656) 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c19d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc59c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 -- 172.21.15.16:0/2622240634 learned_addr learned my addr 172.21.15.16:0/2622240634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc58f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34168/0 (socket says 172.21.15.16:34168) 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 -- 172.21.15.16:0/2622240634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 msgr2=0x55a7bc58f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc58f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 -- 172.21.15.16:0/2622240634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 msgr2=0x55a7bc59c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc59c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.233+0000 7f7c1a59c640 1 -- 172.21.15.16:0/2622240634 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7bc4813a0 con 0x55a7bc58f5d0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0f7480 tx=0x7f7c0c0f74b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99eca82f187b55aa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c0b7fe640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 0x7f7c040a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 0x7f7c040a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c08ff9640 1 -- 172.21.15.16:0/2622240634 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0b7d90 con 0x55a7bc58f5d0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c0b7fe640 1 -- 172.21.15.16:0/1983396068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c0c0ef1d0 con 0x7f7c1c100a70 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c08ff9640 1 -- 172.21.15.16:0/2622240634 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c0fc030 con 0x55a7bc58f5d0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.234+0000 7f7c08ff9640 1 -- 172.21.15.16:0/2622240634 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0baf40 con 0x55a7bc58f5d0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 -- 172.21.15.16:0/2622240634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 msgr2=0x55a7bc58fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0f7480 tx=0x7f7c0c0f74b0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 0x7f7c040a5040 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7c14005fd0 tx=0x7f7c14009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 -- 172.21.15.16:0/2622240634 shutdown_connections 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc59c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58fa10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 --2- 172.21.15.16:0/2622240634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc58f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 -- 172.21.15.16:0/2622240634 >> 172.21.15.16:0/2622240634 conn(0x55a7bc558310 msgr2=0x55a7bc58bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 -- 172.21.15.16:0/2622240634 shutdown_connections 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.235+0000 7f7c23230740 1 -- 172.21.15.16:0/2622240634 wait complete. 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.236+0000 7f7c23230740 1 -- start start 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.236+0000 7f7c23230740 1 Processor -- start 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc599d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc597610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc59ce30 con 0x55a7bc59a370 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc4b0060 con 0x55a7bc590b40 2024-09-12T03:09:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a7bc59cfb0 con 0x55a7bc58f5d0 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c23230740 10 client.0 _ll_register_callbacks cb 0x55a7bc2038c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34172/0 (socket says 172.21.15.16:34172) 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 -- 172.21.15.16:0/3372461964 learned_addr learned my addr 172.21.15.16:0/3372461964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc597610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc599d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 -- 172.21.15.16:0/3372461964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 msgr2=0x55a7bc599d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc599d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 -- 172.21.15.16:0/3372461964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 msgr2=0x55a7bc597610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc597610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.237+0000 7f7c19d9b640 1 -- 172.21.15.16:0/3372461964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7bc2663c0 con 0x55a7bc590b40 2024-09-12T03:09:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc597610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7c10012b10 tx=0x7f7c1000f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b0884bdec6458c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1000fb40 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c0c0fc030 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c1000fce0 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7c10004050 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c23230740 10 client.29603 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.238+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a7bc5a88c0 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c0fc030 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7c10010430 con 0x55a7bc590b40 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 10 client.29603 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 10 client.29603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:14.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 20 client.29603 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc5aa140 0x55a7bc5ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 4 client.29603 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7bc502860 con 0x55a7bc5aa140 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.239+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc5aa140 0x55a7bc5ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.240+0000 7f7c01ffb640 10 client.29603 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.240+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc5aa140 0x55a7bc5ac580 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b603591053b096c server_cookie=9fa482fbd4974024 in_seq=0 out_seq=0 2024-09-12T03:09:14.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:13 smithi033 ceph-mon[35092]: pgmap v976: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-12T03:09:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.241+0000 7f7c01ffb640 1 -- 172.21.15.16:0/3372461964 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a7bc502860 con 0x55a7bc5aa140 2024-09-12T03:09:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.241+0000 7f7c01ffb640 10 client.29603 handle_command_reply: tid=0 2024-09-12T03:09:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.241+0000 7f7c23230740 1 client.29603 shutdown 2024-09-12T03:09:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 msgr2=0x55a7bc597020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7c10012b10 tx=0x7f7c1000f3c0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 shutdown_connections 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc5aa140 0x55a7bc5ac580 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc597610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc597020 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 --2- 172.21.15.16:0/3372461964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc58f5d0 0x55a7bc599d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 >> 172.21.15.16:0/3372461964 conn(0x55a7bc558310 msgr2=0x55a7bc591960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c01ffb640 0 client.29603 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 shutdown_connections 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 1 -- 172.21.15.16:0/3372461964 wait complete. 2024-09-12T03:09:14.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.242+0000 7f7c23230740 20 client.29603 trim_cache size 0 max 16384 2024-09-12T03:09:14.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 Processor -- start 2024-09-12T03:09:14.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 -- start start 2024-09-12T03:09:14.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 0x55a7bc5a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc5968e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1ad5d0 con 0x55a7bc59a370 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1af110 con 0x55a7bc58f5d0 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c23230740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1adca0 con 0x55a7bc590b40 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34176/0 (socket says 172.21.15.16:34176) 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.257+0000 7f7c1a59c640 1 -- 172.21.15.16:0/832158308 learned_addr learned my addr 172.21.15.16:0/832158308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc5968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 0x55a7bc5a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 -- 172.21.15.16:0/832158308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 msgr2=0x55a7bc5a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 0x55a7bc5a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 -- 172.21.15.16:0/832158308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 msgr2=0x55a7bc5968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc5968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 -- 172.21.15.16:0/832158308 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7bc595ff0 con 0x55a7bc58f5d0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 0x55a7bc5a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0ed050 tx=0x7f7c0c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec9d3774d8dfa56 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c03fff640 1 -- 172.21.15.16:0/832158308 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0bb270 con 0x55a7bc58f5d0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c03fff640 1 -- 172.21.15.16:0/832158308 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c0fc070 con 0x55a7bc58f5d0 2024-09-12T03:09:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.258+0000 7f7c03fff640 1 -- 172.21.15.16:0/832158308 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0bb450 con 0x55a7bc58f5d0 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 -- 172.21.15.16:0/832158308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 msgr2=0x55a7bc58be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0ed050 tx=0x7f7c0c009260 comp rx=0 tx=0).stop 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 -- 172.21.15.16:0/832158308 shutdown_connections 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc59a370 0x55a7bc5968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc590b40 0x55a7bc5a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 --2- 172.21.15.16:0/832158308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc58f5d0 0x55a7bc58be10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 -- 172.21.15.16:0/832158308 >> 172.21.15.16:0/832158308 conn(0x55a7bc558310 msgr2=0x55a7bc58b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 -- 172.21.15.16:0/832158308 shutdown_connections 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.259+0000 7f7c23230740 1 -- 172.21.15.16:0/832158308 wait complete. 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.260+0000 7f7c23230740 1 -- start start 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.260+0000 7f7c23230740 1 Processor -- start 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc5a2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 0x55a7bc5a2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1aea40 con 0x55a7bc58f5d0 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1ae370 con 0x55a7bc590b40 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1ab5b0 con 0x55a7bc59a370 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c23230740 10 client.0 _ll_register_callbacks cb 0x7f7c10009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40270/0 (socket says 172.21.15.16:40270) 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 -- 172.21.15.16:0/261896700 learned_addr learned my addr 172.21.15.16:0/261896700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 -- 172.21.15.16:0/261896700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 msgr2=0x55a7bc5a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 0x55a7bc5a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 0x55a7bc5a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 -- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 msgr2=0x55a7bc5a2200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc5a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 -- 172.21.15.16:0/261896700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7bc5a2c80 con 0x55a7bc58f5d0 2024-09-12T03:09:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1ad9d640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 0x55a7bc5a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.261+0000 7f7c1a59c640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c03d040 tx=0x7f7c0c0ef5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d37ad064fc9a13 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0bb630 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7c0c0b9b90 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c0c0fc070 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7c0c0bb8c0 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c23230740 10 client.20385 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.262+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a7bc5a2c80 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c0da020 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7c0c0b8160 con 0x55a7bc58f5d0 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 10 client.20385 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 10 client.20385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 20 client.20385 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc61da60 0x55a7bc61fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 4 client.20385 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7c0c0fc070 con 0x55a7bc61da60 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.263+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc61da60 0x55a7bc61fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c01ffb640 10 client.20385 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c19d9b640 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc61da60 0x55a7bc61fe50 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f48cf354514660e9 server_cookie=87bbab96c50ff3c6 in_seq=0 out_seq=0 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c01ffb640 1 -- 172.21.15.16:0/261896700 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7c0c0fc070 con 0x55a7bc61da60 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c01ffb640 10 client.20385 handle_command_reply: tid=0 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c23230740 1 client.20385 shutdown 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.264+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 msgr2=0x55a7bc610840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c03d040 tx=0x7f7c0c0ef5a0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 shutdown_connections 2024-09-12T03:09:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a7bc61da60 0x55a7bc61fe50 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a7bc59a370 0x55a7bc5a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a7bc590b40 0x55a7bc5a2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c01ffb640 0 client.20385 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 --2- 172.21.15.16:0/261896700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7bc58f5d0 0x55a7bc610840 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 >> 172.21.15.16:0/261896700 conn(0x55a7bc558310 msgr2=0x55a7bc58ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 shutdown_connections 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 1 -- 172.21.15.16:0/261896700 wait complete. 2024-09-12T03:09:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.265+0000 7f7c23230740 20 client.20385 trim_cache size 0 max 16384 2024-09-12T03:09:14.273 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 msgr2=0x7f7c040a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 0x7f7c040a5040 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7c14005fd0 tx=0x7f7c14009500 comp rx=0 tx=0).stop 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 msgr2=0x7f7c1c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009230 tx=0x7f7c0c004040 comp rx=0 tx=0).stop 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 shutdown_connections 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c106a70 0x7f7c1c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c101410 0x7f7c1c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7c040a2b80 0x7f7c040a5040 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 --2- 172.21.15.16:0/1983396068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c100a70 0x7f7c1c1984a0 secure :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009230 tx=0x7f7c0c004040 comp rx=0 tx=0).stop 2024-09-12T03:09:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.266+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 >> 172.21.15.16:0/1983396068 conn(0x7f7c1c0fc790 msgr2=0x7f7c1c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.267+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 shutdown_connections 2024-09-12T03:09:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:14.267+0000 7f7c20e57640 1 -- 172.21.15.16:0/1983396068 wait complete. 2024-09-12T03:09:14.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:13 smithi184 ceph-mon[35321]: pgmap v976: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.8 MiB/s wr, 305 op/s 2024-09-12T03:09:15.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 208590 24.10 MB/sec execute 376 sec latency 47.556 ms 2024-09-12T03:09:16.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 209326 24.10 MB/sec execute 377 sec latency 94.909 ms 2024-09-12T03:09:16.099 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:09:16.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:09:16.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:15 smithi033 ceph-mon[35092]: pgmap v977: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.0 MiB/s wr, 318 op/s 2024-09-12T03:09:16.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:15 smithi184 ceph-mon[35321]: pgmap v977: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.0 MiB/s wr, 318 op/s 2024-09-12T03:09:16.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:15 smithi016 ceph-mon[30925]: pgmap v977: 97 pgs: 97 active+clean; 790 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.0 MiB/s wr, 318 op/s 2024-09-12T03:09:16.419 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:17.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 209789 24.13 MB/sec execute 378 sec latency 286.451 ms 2024-09-12T03:09:17.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:16 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/1201485025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0102ac0 msgr2=0x7fd2e01000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/1201485025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0102ac0 0x7fd2e01000b0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009900 tx=0x7fd2d402a480 comp rx=0 tx=0).stop 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/1201485025 shutdown_connections 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/1201485025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0100c10 0x7fd2e0110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/1201485025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e01005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/1201485025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0102ac0 0x7fd2e01000b0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.213+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/1201485025 >> 172.21.15.16:0/1201485025 conn(0x7fd2e0077210 msgr2=0x7fd2e00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.214+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/1201485025 shutdown_connections 2024-09-12T03:09:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.214+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/1201485025 wait complete. 2024-09-12T03:09:17.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.214+0000 7fd2e6dbb640 1 Processor -- start 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 -- start start 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0100c10 0x7fd2e019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 0x7fd2e019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e01141c0 con 0x7fd2e0100c10 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0114040 con 0x7fd2e0102ac0 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e6dbb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0114340 con 0x7fd2e0102ea0 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e5358640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e5358640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54634/0 (socket says 172.21.15.16:54634) 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2e5358640 1 -- 172.21.15.16:0/2396535939 learned_addr learned my addr 172.21.15.16:0/2396535939 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.215+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 0x7fd2e019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0100c10 0x7fd2e019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 -- 172.21.15.16:0/2396535939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 msgr2=0x7fd2e019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 -- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0100c10 msgr2=0x7fd2e019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0100c10 0x7fd2e019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 -- 172.21.15.16:0/2396535939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e019f650 con 0x7fd2e0102ac0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.216+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 0x7fd2e019abe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00d720 tx=0x7fd2cc00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.217+0000 7fd2ddffb640 1 -- 172.21.15.16:0/2396535939 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc017070 con 0x7fd2e0102ac0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.217+0000 7fd2ddffb640 1 -- 172.21.15.16:0/2396535939 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2cc00de30 con 0x7fd2e0102ac0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.217+0000 7fd2ddffb640 1 -- 172.21.15.16:0/2396535939 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc004d90 con 0x7fd2e0102ac0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.217+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e019f940 con 0x7fd2e0102ac0 2024-09-12T03:09:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.217+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e019fd70 con 0x7fd2e0102ac0 2024-09-12T03:09:17.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.221+0000 7fd2ddffb640 1 -- 172.21.15.16:0/2396535939 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd2cc005020 con 0x7fd2e0102ac0 2024-09-12T03:09:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.223+0000 7fd2ddffb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 0x7fd2d00a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.223+0000 7fd2ddffb640 1 -- 172.21.15.16:0/2396535939 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd2cc0ce8b0 con 0x7fd2e0102ac0 2024-09-12T03:09:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.223+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 0x7fd2d00a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.223+0000 7fd2e9194740 1 Processor -- start 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.223+0000 7fd2e9194740 1 -- start start 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861e2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 0x7fd2d00a5210 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4000c00 tx=0x7fd2d4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564a8613a280 con 0x564a861d4b60 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33260/0 (socket says 172.21.15.16:33260) 2024-09-12T03:09:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e4b57640 1 -- 172.21.15.16:0/1013948558 learned_addr learned my addr 172.21.15.16:0/1013948558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564a860e1b10 con 0x564a861dfd40 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564a86136240 con 0x564a861d5480 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2dffff640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e5358640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e5358640 1 -- 172.21.15.16:0/1013948558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 msgr2=0x564a861df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.224+0000 7fd2e5358640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2e5358640 1 -- 172.21.15.16:0/1013948558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 msgr2=0x564a861d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2e5358640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2e5358640 1 -- 172.21.15.16:0/1013948558 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a860ee1f0 con 0x564a861dfd40 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2e5358640 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861e2130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005c40 tx=0x7fd2d80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fa0de39763911cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2caffd640 1 -- 172.21.15.16:0/1013948558 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8004b20 con 0x564a861dfd40 2024-09-12T03:09:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2caffd640 1 -- 172.21.15.16:0/1013948558 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d8004cc0 con 0x564a861dfd40 2024-09-12T03:09:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.225+0000 7fd2caffd640 1 -- 172.21.15.16:0/1013948558 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80053a0 con 0x564a861dfd40 2024-09-12T03:09:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.226+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 msgr2=0x564a861e2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.226+0000 7fd2e9194740 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861e2130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005c40 tx=0x7fd2d80118d0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 shutdown_connections 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861e2130 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 --2- 172.21.15.16:0/1013948558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 >> 172.21.15.16:0/1013948558 conn(0x564a861d10f0 msgr2=0x564a861d14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 shutdown_connections 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.227+0000 7fd2e9194740 1 -- 172.21.15.16:0/1013948558 wait complete. 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.228+0000 7fd2e9194740 1 -- start start 2024-09-12T03:09:17.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.228+0000 7fd2e9194740 1 Processor -- start 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 0x564a861df780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d5480 0x564a861dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564a861e2910 con 0x564a861d5480 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564a860f4f10 con 0x564a861d4b60 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564a861e2a90 con 0x564a861dfd40 2024-09-12T03:09:17.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e9194740 10 client.0 _ll_register_callbacks cb 0x564a860bda40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:17.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 0x564a861df780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 0x564a861df780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46806/0 (socket says 172.21.15.16:46806) 2024-09-12T03:09:17.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e5358640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.229+0000 7fd2e5358640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54648/0 (socket says 172.21.15.16:54648) 2024-09-12T03:09:17.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 -- 172.21.15.16:0/693953775 learned_addr learned my addr 172.21.15.16:0/693953775 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2dffff640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d5480 0x564a861dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 -- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 msgr2=0x564a861df780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 0x564a861df780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 -- 172.21.15.16:0/693953775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d5480 msgr2=0x564a861dcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d5480 0x564a861dcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.230+0000 7fd2e5358640 1 -- 172.21.15.16:0/693953775 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a85eab6a0 con 0x564a861dfd40 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.231+0000 7fd2e5358640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8004c70 tx=0x7fd2d800fd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f4199135a4d04ea server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.231+0000 7fd2bffff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:17.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.231+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8028070 con 0x564a861dfd40 2024-09-12T03:09:17.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd2d8024650 con 0x564a861dfd40 2024-09-12T03:09:17.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2d8004cc0 con 0x564a861dfd40 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd2d8034050 con 0x564a861dfd40 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2e9194740 10 client.29545 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564a861ee3f0 con 0x564a861dfd40 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.232+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d8004cc0 con 0x564a861dfd40 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd2d8022a90 con 0x564a861dfd40 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 10 client.29545 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 10 client.29545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 20 client.29545 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a861ef7a0 0x564a861f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 4 client.29545 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.234+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a861477f0 con 0x564a861ef7a0 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.235+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a861ef7a0 0x564a861f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.236+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a861ef7a0 0x564a861f1b90 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f39b01b4bd3f36 server_cookie=f70573d031dafb4a in_seq=0 out_seq=0 2024-09-12T03:09:17.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.236+0000 7fd2bffff640 10 client.29545 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:17.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.237+0000 7fd2bffff640 1 -- 172.21.15.16:0/693953775 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564a861477f0 con 0x564a861ef7a0 2024-09-12T03:09:17.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.237+0000 7fd2bffff640 10 client.29545 handle_command_reply: tid=0 2024-09-12T03:09:17.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.237+0000 7fd2e9194740 1 client.29545 shutdown 2024-09-12T03:09:17.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 msgr2=0x564a861dd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8004c70 tx=0x7fd2d800fd80 comp rx=0 tx=0).stop 2024-09-12T03:09:17.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 shutdown_connections 2024-09-12T03:09:17.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a861ef7a0 0x564a861f1b90 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861dd260 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d5480 0x564a861dcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 --2- 172.21.15.16:0/693953775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d4b60 0x564a861df780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 >> 172.21.15.16:0/693953775 conn(0x564a861d10f0 msgr2=0x564a861d6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.238+0000 7fd2bffff640 0 client.29545 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.239+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 shutdown_connections 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.239+0000 7fd2e9194740 1 -- 172.21.15.16:0/693953775 wait complete. 2024-09-12T03:09:17.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.239+0000 7fd2e9194740 20 client.29545 trim_cache size 0 max 16384 2024-09-12T03:09:17.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:16 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:09:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.270+0000 7fd2e9194740 1 Processor -- start 2024-09-12T03:09:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.270+0000 7fd2e9194740 1 -- start start 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861dc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 0x564a861de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861f7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0105840 con 0x564a861d4b60 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0107380 con 0x564a861d5480 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e9194740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0105f10 con 0x564a861dfd40 2024-09-12T03:09:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861dc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e4b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861dc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33266/0 (socket says 172.21.15.16:33266) 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2e4b57640 1 -- 172.21.15.16:0/4132757991 learned_addr learned my addr 172.21.15.16:0/4132757991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.271+0000 7fd2dffff640 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 0x564a861de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2e5358640 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2dffff640 1 -- 172.21.15.16:0/4132757991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 msgr2=0x564a861f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2dffff640 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861f7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2dffff640 1 -- 172.21.15.16:0/4132757991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 msgr2=0x564a861dc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2dffff640 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861dc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.272+0000 7fd2dffff640 1 -- 172.21.15.16:0/4132757991 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a861e2470 con 0x564a861d5480 2024-09-12T03:09:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.273+0000 7fd2dffff640 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 0x564a861de930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005fd0 tx=0x7fd2cc0bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891fa53854b3c262 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.274+0000 7fd2ca7fc640 1 -- 172.21.15.16:0/4132757991 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc020040 con 0x564a861d5480 2024-09-12T03:09:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.274+0000 7fd2ca7fc640 1 -- 172.21.15.16:0/4132757991 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2cc09f470 con 0x564a861d5480 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2ca7fc640 1 -- 172.21.15.16:0/4132757991 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0bb0b0 con 0x564a861d5480 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 -- 172.21.15.16:0/4132757991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 msgr2=0x564a861de930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 0x564a861de930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005fd0 tx=0x7fd2cc0bb040 comp rx=0 tx=0).stop 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 -- 172.21.15.16:0/4132757991 shutdown_connections 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861dfd40 0x564a861f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861d5480 0x564a861de930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 --2- 172.21.15.16:0/4132757991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861dc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 -- 172.21.15.16:0/4132757991 >> 172.21.15.16:0/4132757991 conn(0x564a861d10f0 msgr2=0x564a861de220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 -- 172.21.15.16:0/4132757991 shutdown_connections 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.275+0000 7fd2e9194740 1 -- 172.21.15.16:0/4132757991 wait complete. 2024-09-12T03:09:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 -- start start 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 Processor -- start 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861da0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.276+0000 7fd2e9194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0106cb0 con 0x564a861d4b60 2024-09-12T03:09:17.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.277+0000 7fd2e9194740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e01065e0 con 0x564a861dfd40 2024-09-12T03:09:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.277+0000 7fd2e9194740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e0103820 con 0x564a861d5480 2024-09-12T03:09:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.277+0000 7fd2e9194740 10 client.0 _ll_register_callbacks cb 0x7fd2d8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.278+0000 7fd2dffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.278+0000 7fd2dffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54662/0 (socket says 172.21.15.16:54662) 2024-09-12T03:09:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.278+0000 7fd2dffff640 1 -- 172.21.15.16:0/2312490425 learned_addr learned my addr 172.21.15.16:0/2312490425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.278+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.278+0000 7fd2e5358640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861da0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2dffff640 1 -- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 msgr2=0x564a861da0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861da0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2dffff640 1 -- 172.21.15.16:0/2312490425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 msgr2=0x564a861e85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861e85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2dffff640 1 -- 172.21.15.16:0/2312490425 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a861e60f0 con 0x564a861d5480 2024-09-12T03:09:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.279+0000 7fd2e5358640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.280+0000 7fd2dffff640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0cc3b0 tx=0x7fd2cc097450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=928528dafe684224 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.280+0000 7fd2bffff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.280+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0a7020 con 0x564a861d5480 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.280+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd2cc0bb0b0 con 0x564a861d5480 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.280+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2cc09f470 con 0x564a861d5480 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.281+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd2cc020040 con 0x564a861d5480 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.281+0000 7fd2e9194740 10 client.29551 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.281+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564a8626b410 con 0x564a861d5480 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.281+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2cc097730 con 0x564a861d5480 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.281+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd2cc0af170 con 0x564a861d5480 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 10 client.29551 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 10 client.29551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 20 client.29551 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a8626ba10 0x564a8626de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 4 client.29551 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:09:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.282+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fd2cc09f470 con 0x564a8626ba10 2024-09-12T03:09:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.283+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a8626ba10 0x564a8626de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.283+0000 7fd2bffff640 10 client.29551 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.283+0000 7fd2e4b57640 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a8626ba10 0x564a8626de30 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ba1a83cf5e4094 server_cookie=961386a0df8bb306 in_seq=0 out_seq=0 2024-09-12T03:09:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.287+0000 7fd2bffff640 1 -- 172.21.15.16:0/2312490425 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fd2cc09f470 con 0x564a8626ba10 2024-09-12T03:09:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.287+0000 7fd2bffff640 10 client.29551 handle_command_reply: tid=0 2024-09-12T03:09:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.287+0000 7fd2e9194740 1 client.29551 shutdown 2024-09-12T03:09:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 msgr2=0x564a861d9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0cc3b0 tx=0x7fd2cc097450 comp rx=0 tx=0).stop 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 shutdown_connections 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564a8626ba10 0x564a8626de30 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564a861dfd40 0x564a861da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a861d5480 0x564a861d9970 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 --2- 172.21.15.16:0/2312490425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a861d4b60 0x564a861e85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 >> 172.21.15.16:0/2312490425 conn(0x564a861d10f0 msgr2=0x564a861d6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2bffff640 0 client.29551 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 shutdown_connections 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.288+0000 7fd2e9194740 1 -- 172.21.15.16:0/2312490425 wait complete. 2024-09-12T03:09:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.289+0000 7fd2e9194740 20 client.29551 trim_cache size 0 max 16384 2024-09-12T03:09:17.291 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:09:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b6f6bffd-76f2-446a-ae96-aad44c71ef5e", 2024-09-12T03:09:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:09:17.291 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.289+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 msgr2=0x7fd2d00a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.289+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 0x7fd2d00a5210 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4000c00 tx=0x7fd2d4005a20 comp rx=0 tx=0).stop 2024-09-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 msgr2=0x7fd2e019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 0x7fd2e019abe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00d720 tx=0x7fd2cc00dbe0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 shutdown_connections 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2e0102ea0 0x7fd2e019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e0102ac0 0x7fd2e019abe0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2d00a2d50 0x7fd2d00a5210 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 --2- 172.21.15.16:0/2396535939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2e0100c10 0x7fd2e019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 >> 172.21.15.16:0/2396535939 conn(0x7fd2e0077210 msgr2=0x7fd2e00fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.290+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 shutdown_connections 2024-09-12T03:09:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:17.291+0000 7fd2e6dbb640 1 -- 172.21.15.16:0/2396535939 wait complete. 2024-09-12T03:09:17.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:16 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:09:18.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 210404 24.16 MB/sec execute 379 sec latency 62.547 ms 2024-09-12T03:09:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: pgmap v978: 97 pgs: 97 active+clean; 801 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 492 op/s 2024-09-12T03:09:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:18 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:09:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: pgmap v978: 97 pgs: 97 active+clean; 801 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 492 op/s 2024-09-12T03:09:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:09:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:18 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: pgmap v978: 97 pgs: 97 active+clean; 801 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 492 op/s 2024-09-12T03:09:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:09:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:18.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:18 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:19.029 INFO:tasks.workunit.client.0.smithi016.stdout: 10 211023 24.18 MB/sec execute 380 sec latency 97.993 ms 2024-09-12T03:09:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.067+0000 7f7fe94d4640 1 -- 172.21.15.16:0/3920049412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 msgr2=0x7f7fe410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.067+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/3920049412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe410eaa0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009ca0 tx=0x7f7fdc02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 -- 172.21.15.16:0/3920049412 shutdown_connections 2024-09-12T03:09:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/3920049412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe410eaa0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/3920049412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe41027a0 0x7f7fe41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/3920049412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe4101df0 0x7f7fe41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 -- 172.21.15.16:0/3920049412 >> 172.21.15.16:0/3920049412 conn(0x7f7fe40fd8f0 msgr2=0x7f7fe40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 -- 172.21.15.16:0/3920049412 shutdown_connections 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 -- 172.21.15.16:0/3920049412 wait complete. 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 Processor -- start 2024-09-12T03:09:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.068+0000 7f7fe94d4640 1 -- start start 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 0x7f7fe40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe4111f80 con 0x7f7fe4106f20 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe4111e00 con 0x7f7fe41027a0 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe94d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe4112100 con 0x7f7fe4101df0 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 0x7f7fe40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46836/0 (socket says 172.21.15.16:46836) 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 -- 172.21.15.16:0/899059782 learned_addr learned my addr 172.21.15.16:0/899059782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 -- 172.21.15.16:0/899059782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 msgr2=0x7f7fe40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 0x7f7fe40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 -- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 msgr2=0x7f7fe4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fdbfff640 1 -- 172.21.15.16:0/899059782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe41a6950 con 0x7f7fe41027a0 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.069+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 0x7f7fe40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e9f0 tx=0x7f7fcc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fe0ff9640 1 -- 172.21.15.16:0/899059782 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc00cd80 con 0x7f7fe41027a0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe41a6c40 con 0x7f7fe41027a0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fe0ff9640 1 -- 172.21.15.16:0/899059782 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fcc005500 con 0x7f7fe41027a0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fe0ff9640 1 -- 172.21.15.16:0/899059782 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc010690 con 0x7f7fe41027a0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.070+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe41a7180 con 0x7f7fe41027a0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.073+0000 7f7feb8ad740 1 Processor -- start 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.073+0000 7f7feb8ad740 1 -- start start 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.073+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.073+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54688/0 (socket says 172.21.15.16:54688) 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33294/0 (socket says 172.21.15.16:33294) 2024-09-12T03:09:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbab6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fffba0f2e0 con 0x55fffbaa9dc0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 -- 172.21.15.16:0/4104679578 learned_addr learned my addr 172.21.15.16:0/4104679578 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fffb9b6ce0 con 0x55fffbab4680 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fffba0b2a0 con 0x55fffbaa94a0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 -- 172.21.15.16:0/4104679578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 msgr2=0x55fffbab6a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbab6a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 -- 172.21.15.16:0/4104679578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 msgr2=0x55fffbab4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 -- 172.21.15.16:0/4104679578 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fffb99b440 con 0x55fffbaa94a0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.074+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc031f80 tx=0x7f7fdc005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf034c87e754d8c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.075+0000 7f7fe0ff9640 1 -- 172.21.15.16:0/899059782 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7fcc004f60 con 0x7f7fe41027a0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.075+0000 7f7fd97fa640 1 -- 172.21.15.16:0/4104679578 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc003b70 con 0x55fffbaa94a0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.075+0000 7f7fd97fa640 1 -- 172.21.15.16:0/4104679578 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fdc036040 con 0x55fffbaa94a0 2024-09-12T03:09:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.075+0000 7f7fd97fa640 1 -- 172.21.15.16:0/4104679578 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0045b0 con 0x55fffbaa94a0 2024-09-12T03:09:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7fe0ff9640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 0x7f7fd00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 0x7f7fd00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7fe0ff9640 1 -- 172.21.15.16:0/899059782 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fcc0cf3b0 con 0x7f7fe41027a0 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 msgr2=0x55fffbaa9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc031f80 tx=0x7f7fdc005ef0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 shutdown_connections 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbab6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/4104679578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbaa9880 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 >> 172.21.15.16:0/4104679578 conn(0x55fffbaa5a30 msgr2=0x55fffbaa5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 shutdown_connections 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7feb8ad740 1 -- 172.21.15.16:0/4104679578 wait complete. 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.076+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 0x7f7fd00a5040 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55fffbaa6890 tx=0x7f7fd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 -- start start 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 Processor -- start 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbab4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa9dc0 0x55fffbab1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fffbab70d0 con 0x55fffbab4680 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fffb9ca050 con 0x55fffbaa9dc0 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fffbab7250 con 0x55fffbaa94a0 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7feb8ad740 10 client.0 _ll_register_callbacks cb 0x55fffba0a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33304/0 (socket says 172.21.15.16:33304) 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7fe37fe640 1 -- 172.21.15.16:0/3667865977 learned_addr learned my addr 172.21.15.16:0/3667865977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbab4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.077+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa9dc0 0x55fffbab1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 -- 172.21.15.16:0/3667865977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 msgr2=0x55fffbab4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbab4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 -- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa9dc0 msgr2=0x55fffbab1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa9dc0 0x55fffbab1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 -- 172.21.15.16:0/3667865977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fffb7803c0 con 0x55fffbab4680 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc004af0 tx=0x7f7fdc02f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7999248483f6a42 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc031720 con 0x55fffbab4680 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fdc036040 con 0x55fffbab4680 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fdc0318c0 con 0x55fffbab4680 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7fdc003970 con 0x55fffbab4680 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7feb8ad740 10 client.20415 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:19.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.078+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fffbab70d0 con 0x55fffbab4680 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fdc036040 con 0x55fffbab4680 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7fdc043070 con 0x55fffbab4680 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 10 client.20415 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 10 client.20415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 20 client.20415 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbac3fa0 0x55fffbac6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 4 client.20415 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fffba1c850 con 0x55fffbac3fa0 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbac3fa0 0x55fffbac6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7fca7fc640 10 client.20415 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.079+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbac3fa0 0x55fffbac6390 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53471561727e6eb4 server_cookie=b60126fa334a3bcb in_seq=0 out_seq=0 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.080+0000 7f7fca7fc640 1 -- 172.21.15.16:0/3667865977 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fffba1c850 con 0x55fffbac3fa0 2024-09-12T03:09:19.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.080+0000 7f7fca7fc640 10 client.20415 handle_command_reply: tid=0 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.080+0000 7f7feb8ad740 1 client.20415 shutdown 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.080+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 msgr2=0x55fffbab1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.080+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc004af0 tx=0x7f7fdc02f390 comp rx=0 tx=0).stop 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 shutdown_connections 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbac3fa0 0x55fffbac6390 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab1920 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa9dc0 0x55fffbab1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/3667865977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbab4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 >> 172.21.15.16:0/3667865977 conn(0x55fffbaa5a30 msgr2=0x55fffbaab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7fca7fc640 0 client.20415 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 shutdown_connections 2024-09-12T03:09:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 1 -- 172.21.15.16:0/3667865977 wait complete. 2024-09-12T03:09:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.081+0000 7f7feb8ad740 20 client.20415 trim_cache size 0 max 16384 2024-09-12T03:09:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.101+0000 7f7feb8ad740 1 Processor -- start 2024-09-12T03:09:19.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.101+0000 7f7feb8ad740 1 -- start start 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbacbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41af600 con 0x55fffbaa9dc0 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41b1140 con 0x55fffbab4680 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41afcd0 con 0x55fffbaa94a0 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbacbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:54718/0 (socket says 172.21.15.16:54718) 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe2ffd640 1 -- 172.21.15.16:0/541211991 learned_addr learned my addr 172.21.15.16:0/541211991 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33318/0 (socket says 172.21.15.16:33318) 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 -- 172.21.15.16:0/541211991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 msgr2=0x55fffbac0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 -- 172.21.15.16:0/541211991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 msgr2=0x55fffbacbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbacbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fdbfff640 1 -- 172.21.15.16:0/541211991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fffbab3a40 con 0x55fffbaa9dc0 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.102+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbacbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.103+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x55fffbaa6db0 tx=0x7f7fcc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23955cc537a647be server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.103+0000 7f7fd8ff9640 1 -- 172.21.15.16:0/541211991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc0b9070 con 0x55fffbaa9dc0 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.103+0000 7f7fd8ff9640 1 -- 172.21.15.16:0/541211991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fcc09c320 con 0x55fffbaa9dc0 2024-09-12T03:09:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.103+0000 7f7fd8ff9640 1 -- 172.21.15.16:0/541211991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc098290 con 0x55fffbaa9dc0 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- 172.21.15.16:0/541211991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 msgr2=0x55fffbab2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x55fffbaa6db0 tx=0x7f7fcc00cf20 comp rx=0 tx=0).stop 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- 172.21.15.16:0/541211991 shutdown_connections 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbab4680 0x55fffbacbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbaa9dc0 0x55fffbab2b90 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/541211991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa94a0 0x55fffbac0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- 172.21.15.16:0/541211991 >> 172.21.15.16:0/541211991 conn(0x55fffbaa5a30 msgr2=0x55fffbaac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- 172.21.15.16:0/541211991 shutdown_connections 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- 172.21.15.16:0/541211991 wait complete. 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 -- start start 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.104+0000 7f7feb8ad740 1 Processor -- start 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa9dc0 0x55fffbab87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41b0a70 con 0x55fffbab4680 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa9dc0 0x55fffbab87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7fe2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:46844/0 (socket says 172.21.15.16:46844) 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7fe2ffd640 1 -- 172.21.15.16:0/551679781 learned_addr learned my addr 172.21.15.16:0/551679781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41b03a0 con 0x55fffbaa94a0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe41ad5e0 con 0x55fffbaa9dc0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7fe37fe640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.105+0000 7f7feb8ad740 10 client.0 _ll_register_callbacks cb 0x7f7fd4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 -- 172.21.15.16:0/551679781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa9dc0 msgr2=0x55fffbab87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa9dc0 0x55fffbab87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 -- 172.21.15.16:0/551679781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 msgr2=0x55fffbab8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 -- 172.21.15.16:0/551679781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fffbac6700 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fe2ffd640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400fb50 tx=0x7f7fd4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb8c5795a76632fe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd400fdc0 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7fd4004020 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fcc09c320 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7fd4004220 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7feb8ad740 10 client.29639 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.106+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fffbab43f0 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fcc09c320 con 0x55fffbaa94a0 2024-09-12T03:09:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7fd401dbf0 con 0x55fffbaa94a0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 10 client.29639 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 10 client.29639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 20 client.29639 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbb2f3b0 0x55fffbb317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 4 client.29639 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fffba16d60 con 0x55fffbb2f3b0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.107+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbb2f3b0 0x55fffbb317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7fdbfff640 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbb2f3b0 0x55fffbb317d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1faaccc18ea2b5 server_cookie=e6a0945f565b3f29 in_seq=0 out_seq=0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7fca7fc640 10 client.29639 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7fca7fc640 1 -- 172.21.15.16:0/551679781 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fffba16d60 con 0x55fffbb2f3b0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7fca7fc640 10 client.29639 handle_command_reply: tid=0 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 client.29639 shutdown 2024-09-12T03:09:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 msgr2=0x55fffbab82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400fb50 tx=0x7f7fd4017000 comp rx=0 tx=0).stop 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 shutdown_connections 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55fffbb2f3b0 0x55fffbb317d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7fca7fc640 0 client.29639 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fffbab4680 0x55fffbab8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fffbaa9dc0 0x55fffbab87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 --2- 172.21.15.16:0/551679781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fffbaa94a0 0x55fffbab82b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.108+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 >> 172.21.15.16:0/551679781 conn(0x55fffbaa5a30 msgr2=0x55fffbaaae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.109+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 shutdown_connections 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.109+0000 7f7feb8ad740 1 -- 172.21.15.16:0/551679781 wait complete. 2024-09-12T03:09:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.109+0000 7f7feb8ad740 20 client.29639 trim_cache size 0 max 16384 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "b6f6bffd-76f2-446a-ae96-aad44c71ef5e": { 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b6f6bffd-76f2-446a-ae96-aad44c71ef5e", 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 msgr2=0x7f7fd00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 0x7f7fd00a5040 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55fffbaa6890 tx=0x7f7fd4005f90 comp rx=0 tx=0).stop 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 msgr2=0x7f7fe4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e9f0 tx=0x7f7fcc00edf0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 shutdown_connections 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7fd00a2b80 0x7f7fd00a5040 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe4106f20 0x7f7fe4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7fe41027a0 0x7f7fe4074a30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 --2- 172.21.15.16:0/899059782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe4101df0 0x7f7fe40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 >> 172.21.15.16:0/899059782 conn(0x7f7fe40fd8f0 msgr2=0x7f7fe40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 shutdown_connections 2024-09-12T03:09:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:19.110+0000 7f7fe94d4640 1 -- 172.21.15.16:0/899059782 wait complete. 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:09:19.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:09:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:09:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:09:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:09:19.957 INFO:tasks.cephfs.filesystem:scrub status for tag:b6f6bffd-76f2-446a-ae96-aad44c71ef5e - {'path': '/', 'tag': 'b6f6bffd-76f2-446a-ae96-aad44c71ef5e', 'options': 'recursive,force'} 2024-09-12T03:09:20.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 211688 24.20 MB/sec execute 381 sec latency 50.479 ms 2024-09-12T03:09:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:20 smithi184 ceph-mon[35321]: pgmap v979: 97 pgs: 97 active+clean; 804 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-09-12T03:09:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:20 smithi016 ceph-mon[30925]: pgmap v979: 97 pgs: 97 active+clean; 804 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-09-12T03:09:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:20 smithi033 ceph-mon[35092]: pgmap v979: 97 pgs: 97 active+clean; 804 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-09-12T03:09:21.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 212387 24.25 MB/sec execute 382 sec latency 76.810 ms 2024-09-12T03:09:22.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 212944 24.27 MB/sec execute 383 sec latency 66.069 ms 2024-09-12T03:09:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:22 smithi016 ceph-mon[30925]: pgmap v980: 97 pgs: 97 active+clean; 809 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 655 op/s 2024-09-12T03:09:22.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:22 smithi033 ceph-mon[35092]: pgmap v980: 97 pgs: 97 active+clean; 809 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 655 op/s 2024-09-12T03:09:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:22 smithi184 ceph-mon[35321]: pgmap v980: 97 pgs: 97 active+clean; 809 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 655 op/s 2024-09-12T03:09:23.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 213589 24.29 MB/sec execute 384 sec latency 90.740 ms 2024-09-12T03:09:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 214230 24.30 MB/sec execute 385 sec latency 249.048 ms 2024-09-12T03:09:24.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:24 smithi016 ceph-mon[30925]: pgmap v981: 97 pgs: 97 active+clean; 806 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 619 op/s 2024-09-12T03:09:24.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:24 smithi033 ceph-mon[35092]: pgmap v981: 97 pgs: 97 active+clean; 806 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 619 op/s 2024-09-12T03:09:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:24 smithi184 ceph-mon[35321]: pgmap v981: 97 pgs: 97 active+clean; 806 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 619 op/s 2024-09-12T03:09:25.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 214967 24.35 MB/sec execute 386 sec latency 107.625 ms 2024-09-12T03:09:26.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 215713 24.39 MB/sec execute 387 sec latency 55.716 ms 2024-09-12T03:09:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:26 smithi016 ceph-mon[30925]: pgmap v982: 97 pgs: 97 active+clean; 814 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 673 op/s 2024-09-12T03:09:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:26 smithi033 ceph-mon[35092]: pgmap v982: 97 pgs: 97 active+clean; 814 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 673 op/s 2024-09-12T03:09:26.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:26 smithi184 ceph-mon[35321]: pgmap v982: 97 pgs: 97 active+clean; 814 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 673 op/s 2024-09-12T03:09:27.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 216485 24.45 MB/sec execute 388 sec latency 57.494 ms 2024-09-12T03:09:28.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 217146 24.48 MB/sec execute 389 sec latency 48.979 ms 2024-09-12T03:09:28.346 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:28 smithi184 ceph-mon[35321]: pgmap v983: 97 pgs: 97 active+clean; 820 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 713 op/s 2024-09-12T03:09:28.346 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:28 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:09:28.346 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:28 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:09:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:28 smithi016 ceph-mon[30925]: pgmap v983: 97 pgs: 97 active+clean; 820 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 713 op/s 2024-09-12T03:09:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:28 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:09:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:28 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:09:28.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:28 smithi033 ceph-mon[35092]: pgmap v983: 97 pgs: 97 active+clean; 820 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 713 op/s 2024-09-12T03:09:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:28 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:09:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:28 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:09:29.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 217696 24.49 MB/sec execute 390 sec latency 60.608 ms 2024-09-12T03:09:29.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:29 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:09:29.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:29 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:09:29.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:29 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:09:29.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:29 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:09:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:29 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:09:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:29 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:09:29.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:29 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:09:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:29 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:09:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:29 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:09:30.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 218393 24.50 MB/sec execute 391 sec latency 164.224 ms 2024-09-12T03:09:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:30 smithi016 ceph-mon[30925]: pgmap v984: 97 pgs: 97 active+clean; 823 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 574 op/s 2024-09-12T03:09:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:30 smithi033 ceph-mon[35092]: pgmap v984: 97 pgs: 97 active+clean; 823 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 574 op/s 2024-09-12T03:09:30.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:30 smithi184 ceph-mon[35321]: pgmap v984: 97 pgs: 97 active+clean; 823 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 574 op/s 2024-09-12T03:09:31.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 219062 24.54 MB/sec execute 392 sec latency 58.231 ms 2024-09-12T03:09:32.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 219650 24.54 MB/sec execute 393 sec latency 117.969 ms 2024-09-12T03:09:32.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:32 smithi016 ceph-mon[30925]: pgmap v985: 97 pgs: 97 active+clean; 827 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 658 op/s 2024-09-12T03:09:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:32 smithi033 ceph-mon[35092]: pgmap v985: 97 pgs: 97 active+clean; 827 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 658 op/s 2024-09-12T03:09:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:32 smithi184 ceph-mon[35321]: pgmap v985: 97 pgs: 97 active+clean; 827 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 658 op/s 2024-09-12T03:09:33.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 220282 24.58 MB/sec execute 394 sec latency 196.115 ms 2024-09-12T03:09:34.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 221011 24.62 MB/sec execute 395 sec latency 56.626 ms 2024-09-12T03:09:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:34 smithi016 ceph-mon[30925]: pgmap v986: 97 pgs: 97 active+clean; 817 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 554 op/s 2024-09-12T03:09:34.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:34 smithi033 ceph-mon[35092]: pgmap v986: 97 pgs: 97 active+clean; 817 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 554 op/s 2024-09-12T03:09:34.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:34 smithi184 ceph-mon[35321]: pgmap v986: 97 pgs: 97 active+clean; 817 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.6 MiB/s wr, 554 op/s 2024-09-12T03:09:35.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 221525 24.64 MB/sec execute 396 sec latency 197.121 ms 2024-09-12T03:09:36.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 222090 24.65 MB/sec execute 397 sec latency 72.076 ms 2024-09-12T03:09:36.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:36 smithi016 ceph-mon[30925]: pgmap v987: 97 pgs: 97 active+clean; 813 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:09:36.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:36 smithi033 ceph-mon[35092]: pgmap v987: 97 pgs: 97 active+clean; 813 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:09:36.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:36 smithi184 ceph-mon[35321]: pgmap v987: 97 pgs: 97 active+clean; 813 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.9 MiB/s wr, 496 op/s 2024-09-12T03:09:37.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 222693 24.67 MB/sec execute 398 sec latency 88.281 ms 2024-09-12T03:09:38.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 223305 24.68 MB/sec execute 399 sec latency 82.419 ms 2024-09-12T03:09:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:38 smithi016 ceph-mon[30925]: pgmap v988: 97 pgs: 97 active+clean; 819 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 482 op/s 2024-09-12T03:09:38.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:38 smithi033 ceph-mon[35092]: pgmap v988: 97 pgs: 97 active+clean; 819 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 482 op/s 2024-09-12T03:09:38.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:38 smithi184 ceph-mon[35321]: pgmap v988: 97 pgs: 97 active+clean; 819 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 482 op/s 2024-09-12T03:09:39.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 223966 24.71 MB/sec execute 400 sec latency 87.194 ms 2024-09-12T03:09:40.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 224715 24.72 MB/sec execute 401 sec latency 55.212 ms 2024-09-12T03:09:40.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:40 smithi016 ceph-mon[30925]: pgmap v989: 97 pgs: 97 active+clean; 822 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 427 op/s 2024-09-12T03:09:40.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:40 smithi033 ceph-mon[35092]: pgmap v989: 97 pgs: 97 active+clean; 822 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 427 op/s 2024-09-12T03:09:40.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:40 smithi184 ceph-mon[35321]: pgmap v989: 97 pgs: 97 active+clean; 822 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 427 op/s 2024-09-12T03:09:41.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 225460 24.80 MB/sec execute 402 sec latency 59.068 ms 2024-09-12T03:09:42.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 225850 24.78 MB/sec execute 403 sec latency 388.580 ms 2024-09-12T03:09:42.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:42 smithi016 ceph-mon[30925]: pgmap v990: 97 pgs: 97 active+clean; 833 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 527 op/s 2024-09-12T03:09:42.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:42 smithi033 ceph-mon[35092]: pgmap v990: 97 pgs: 97 active+clean; 833 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 527 op/s 2024-09-12T03:09:42.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:42 smithi184 ceph-mon[35321]: pgmap v990: 97 pgs: 97 active+clean; 833 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 527 op/s 2024-09-12T03:09:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 226163 24.77 MB/sec execute 404 sec latency 742.457 ms 2024-09-12T03:09:44.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 226771 24.79 MB/sec execute 405 sec latency 91.538 ms 2024-09-12T03:09:44.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:44 smithi016 ceph-mon[30925]: pgmap v991: 97 pgs: 97 active+clean; 839 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 466 op/s 2024-09-12T03:09:44.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:44 smithi033 ceph-mon[35092]: pgmap v991: 97 pgs: 97 active+clean; 839 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 466 op/s 2024-09-12T03:09:44.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:44 smithi184 ceph-mon[35321]: pgmap v991: 97 pgs: 97 active+clean; 839 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 466 op/s 2024-09-12T03:09:45.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 227389 24.81 MB/sec execute 406 sec latency 95.916 ms 2024-09-12T03:09:45.813 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-12T03:09:45.813 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-09-12T03:09:45.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:09:46.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 227981 24.82 MB/sec execute 407 sec latency 47.572 ms 2024-09-12T03:09:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:46.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:46 smithi016 ceph-mon[30925]: pgmap v992: 97 pgs: 97 active+clean; 841 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:09:46.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:46 smithi033 ceph-mon[35092]: pgmap v992: 97 pgs: 97 active+clean; 841 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:09:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:46 smithi184 ceph-mon[35321]: pgmap v992: 97 pgs: 97 active+clean; 841 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 443 op/s 2024-09-12T03:09:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3943058764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c102ef0 msgr2=0x7f0e9c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3943058764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c10da40 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009ca0 tx=0x7f0e9402a950 comp rx=0 tx=0).stop 2024-09-12T03:09:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3943058764 shutdown_connections 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3943058764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c10da40 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3943058764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102570 0x7f0e9c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3943058764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c100e20 0x7f0e9c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3943058764 >> 172.21.15.16:0/3943058764 conn(0x7f0e9c0fc7d0 msgr2=0x7f0e9c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3943058764 shutdown_connections 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.610+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3943058764 wait complete. 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.611+0000 7f0ea3a08640 1 Processor -- start 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.611+0000 7f0ea3a08640 1 -- start start 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.611+0000 7f0ea3a08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.611+0000 7f0ea3a08640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 0x7f0e9c1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.611+0000 7f0ea3a08640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea3a08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c110f80 con 0x7f0e9c100e20 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea3a08640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c110e00 con 0x7f0e9c102570 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea3a08640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c111100 con 0x7f0e9c102ef0 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea17a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea17a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34976/0 (socket says 172.21.15.16:34976) 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea17a4640 1 -- 172.21.15.16:0/3161288455 learned_addr learned my addr 172.21.15.16:0/3161288455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 0x7f0e9c1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea1fa5640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 -- 172.21.15.16:0/3161288455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102ef0 msgr2=0x7f0e9c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 -- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 msgr2=0x7f0e9c197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea0fa3640 1 -- 172.21.15.16:0/3161288455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c1a69d0 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.612+0000 7f0ea17a4640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.613+0000 7f0ea0fa3640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 0x7f0e9c1976a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00e9f0 tx=0x7f0e8c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.613+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00ccc0 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.613+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e8c005020 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.613+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9c1a6cc0 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.613+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c004b20 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.614+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9c1a71b0 con 0x7f0e9c102570 2024-09-12T03:09:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.615+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c101200 con 0x7f0e9c102570 2024-09-12T03:09:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.616+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0e8c004cc0 con 0x7f0e9c102570 2024-09-12T03:09:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.617+0000 7f0e927fc640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 0x7f0e880a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.617+0000 7f0ea17a4640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 0x7f0e880a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.617+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0e8c0cef00 con 0x7f0e9c102570 2024-09-12T03:09:46.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.618+0000 7f0ea17a4640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 0x7f0e880a50c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0e980095f0 tx=0x7f0e98005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:46.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.623+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0e8c09bea0 con 0x7f0e9c102570 2024-09-12T03:09:46.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.882+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e9c06a560 con 0x7f0e9c102570 2024-09-12T03:09:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.883+0000 7f0e927fc640 1 -- 172.21.15.16:0/3161288455 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f0e8c09b7a0 con 0x7f0e9c102570 2024-09-12T03:09:46.883 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:09:46.884 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:09:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:09:46.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 msgr2=0x7f0e880a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:46.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 0x7f0e880a50c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0e980095f0 tx=0x7f0e98005f90 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 msgr2=0x7f0e9c1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 0x7f0e9c1976a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00e9f0 tx=0x7f0e8c00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 shutdown_connections 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e9c102ef0 0x7f0e9c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c102570 0x7f0e9c1976a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0e880a2c00 0x7f0e880a50c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 --2- 172.21.15.16:0/3161288455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c100e20 0x7f0e9c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.886+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 >> 172.21.15.16:0/3161288455 conn(0x7f0e9c0fc7d0 msgr2=0x7f0e9c106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.887+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 shutdown_connections 2024-09-12T03:09:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:46.887+0000 7f0ea3a08640 1 -- 172.21.15.16:0/3161288455 wait complete. 2024-09-12T03:09:47.030 INFO:tasks.workunit.client.0.smithi016.stdout: 10 228682 24.85 MB/sec execute 408 sec latency 67.334 ms 2024-09-12T03:09:47.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:47 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3161288455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:47.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:47 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3161288455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:47.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:47 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3161288455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:47.886 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-09-12T03:09:47.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-09-12T03:09:48.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229267 24.85 MB/sec execute 409 sec latency 219.259 ms 2024-09-12T03:09:48.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:48 smithi016 ceph-mon[30925]: pgmap v993: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-09-12T03:09:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:48.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:48 smithi033 ceph-mon[35092]: pgmap v993: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-09-12T03:09:48.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:48 smithi184 ceph-mon[35321]: pgmap v993: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-09-12T03:09:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 -- 172.21.15.16:0/86699675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 msgr2=0x7fa978107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 --2- 172.21.15.16:0/86699675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978107b90 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009930 tx=0x7fa96c02a420 comp rx=0 tx=0).stop 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 -- 172.21.15.16:0/86699675 shutdown_connections 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 --2- 172.21.15.16:0/86699675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9781082a0 0x7fa97810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 --2- 172.21.15.16:0/86699675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978107b90 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 --2- 172.21.15.16:0/86699675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa978102d30 0x7fa978103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 -- 172.21.15.16:0/86699675 >> 172.21.15.16:0/86699675 conn(0x7fa9780fc7d0 msgr2=0x7fa9780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 -- 172.21.15.16:0/86699675 shutdown_connections 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.818+0000 7fa97e606640 1 -- 172.21.15.16:0/86699675 wait complete. 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 Processor -- start 2024-09-12T03:09:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 -- start start 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 0x7fa978100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 0x7fa978101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781120b0 con 0x7fa9781036e0 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa978111f30 con 0x7fa978102d30 2024-09-12T03:09:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97e606640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa978112230 con 0x7fa9781082a0 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa977fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 0x7fa978100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34986/0 (socket says 172.21.15.16:34986) 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa97cba3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 0x7fa978101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.819+0000 7fa9777fe640 1 -- 172.21.15.16:0/2757594839 learned_addr learned my addr 172.21.15.16:0/2757594839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 -- 172.21.15.16:0/2757594839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 msgr2=0x7fa978100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 0x7fa978100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 -- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 msgr2=0x7fa978100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 -- 172.21.15.16:0/2757594839 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9781a6a70 con 0x7fa9781082a0 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa977fff640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 0x7fa978100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 0x7fa978101410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa970009fd0 tx=0x7fa97000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa9757fa640 1 -- 172.21.15.16:0/2757594839 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97000ce20 con 0x7fa9781082a0 2024-09-12T03:09:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa9757fa640 1 -- 172.21.15.16:0/2757594839 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa970005020 con 0x7fa9781082a0 2024-09-12T03:09:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9781a6d60 con 0x7fa9781082a0 2024-09-12T03:09:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa9757fa640 1 -- 172.21.15.16:0/2757594839 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970010720 con 0x7fa9781082a0 2024-09-12T03:09:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.820+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9781a72a0 con 0x7fa9781082a0 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.823+0000 7fa9809df740 1 Processor -- start 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.823+0000 7fa9757fa640 1 -- 172.21.15.16:0/2757594839 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa970005540 con 0x7fa9781082a0 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.823+0000 7fa9809df740 1 -- start start 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962ed4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a962e2d2a0 con 0x55a962ec7760 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a962dd4b10 con 0x55a962ed2940 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9809df740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a962e29260 con 0x55a962ec8080 2024-09-12T03:09:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34990/0 (socket says 172.21.15.16:34990) 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9777fe640 1 -- 172.21.15.16:0/2324973304 learned_addr learned my addr 172.21.15.16:0/2324973304 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962ed4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 -- 172.21.15.16:0/2324973304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 msgr2=0x55a962ed4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962ed4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 -- 172.21.15.16:0/2324973304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 msgr2=0x55a962ec7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa977fff640 1 -- 172.21.15.16:0/2324973304 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a962b9eb90 con 0x55a962ec8080 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.824+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa9757fa640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 0x7fa9600a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa977fff640 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed2400 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa968009fd0 tx=0x7fa9680118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f030f2ae81f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 0x7fa9600a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa9757fa640 1 -- 172.21.15.16:0/2757594839 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa9700cf5f0 con 0x7fa9781082a0 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa95e7fc640 1 -- 172.21.15.16:0/2324973304 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968004310 con 0x55a962ec8080 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa95e7fc640 1 -- 172.21.15.16:0/2324973304 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9680044b0 con 0x55a962ec8080 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.825+0000 7fa95e7fc640 1 -- 172.21.15.16:0/2324973304 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968003650 con 0x55a962ec8080 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- 172.21.15.16:0/2324973304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 msgr2=0x55a962ed2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed2400 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa968009fd0 tx=0x7fa9680118d0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 0x7fa9600a50c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa96c004650 tx=0x7fa96c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- 172.21.15.16:0/2324973304 shutdown_connections 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962ed4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed2400 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 --2- 172.21.15.16:0/2324973304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ec7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- 172.21.15.16:0/2324973304 >> 172.21.15.16:0/2324973304 conn(0x55a962ec3cf0 msgr2=0x55a962ec40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- 172.21.15.16:0/2324973304 shutdown_connections 2024-09-12T03:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- 172.21.15.16:0/2324973304 wait complete. 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.826+0000 7fa9809df740 1 -- start start 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 Processor -- start 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ecf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ecfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a962ed5510 con 0x55a962ec7760 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a962db56e0 con 0x55a962ec8080 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a962ed5690 con 0x55a962ed2940 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9809df740 10 client.0 _ll_register_callbacks cb 0x55a962dd9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ecfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa97cba3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ecf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa97cba3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48788/0 (socket says 172.21.15.16:48788) 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.827+0000 7fa97cba3640 1 -- 172.21.15.16:0/326658881 learned_addr learned my addr 172.21.15.16:0/326658881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 -- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 msgr2=0x55a962ecfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ecfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 -- 172.21.15.16:0/326658881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 msgr2=0x55a962ecf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ecf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 -- 172.21.15.16:0/326658881 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a962ede9b0 con 0x55a962ed2940 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa97cba3640 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa970098830 tx=0x7fa97009c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afa894c80b04020 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa9577fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97009c0c0 con 0x55a962ed2940 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9680044b0 con 0x55a962ed2940 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.828+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa97009c260 con 0x55a962ed2940 2024-09-12T03:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa9700a4640 con 0x55a962ed2940 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 10 client.29587 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a962ee13f0 con 0x55a962ed2940 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9680044b0 con 0x55a962ed2940 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa970097bb0 con 0x55a962ed2940 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 10 client.29587 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 10 client.29587 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 20 client.29587 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962ee2740 0x55a962ee4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 4 client.29587 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a962e3a9d0 con 0x55a962ee2740 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.829+0000 7fa977fff640 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962ee2740 0x55a962ee4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.830+0000 7fa9577fe640 10 client.29587 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.830+0000 7fa977fff640 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962ee2740 0x55a962ee4b90 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20959f26fbb15959 server_cookie=d3ec2f799e944f7d in_seq=0 out_seq=0 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9577fe640 1 -- 172.21.15.16:0/326658881 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a962e3a9d0 con 0x55a962ee2740 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9577fe640 10 client.29587 handle_command_reply: tid=0 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 client.29587 shutdown 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 msgr2=0x55a962ed0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa970098830 tx=0x7fa97009c6a0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 shutdown_connections 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962ee2740 0x55a962ee4b90 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ed2940 0x55a962ed0220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9577fe640 0 client.29587 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ecfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 --2- 172.21.15.16:0/326658881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ecf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.832+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 >> 172.21.15.16:0/326658881 conn(0x55a962ec3cf0 msgr2=0x55a962ec2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.833+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 shutdown_connections 2024-09-12T03:09:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.833+0000 7fa9809df740 1 -- 172.21.15.16:0/326658881 wait complete. 2024-09-12T03:09:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.833+0000 7fa9809df740 20 client.29587 trim_cache size 0 max 16384 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 Processor -- start 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 -- start start 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ec2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 0x55a962eea480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b3b40 con 0x55a962ed2940 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b5680 con 0x55a962ec8080 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.856+0000 7fa9809df740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b4210 con 0x55a962ec7760 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48800/0 (socket says 172.21.15.16:48800) 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ec2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 -- 172.21.15.16:0/2746305511 learned_addr learned my addr 172.21.15.16:0/2746305511 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 0x55a962eea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 -- 172.21.15.16:0/2746305511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 msgr2=0x55a962ec2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ec2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 -- 172.21.15.16:0/2746305511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 msgr2=0x55a962eea480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 0x55a962eea480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 -- 172.21.15.16:0/2746305511 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a962ebf260 con 0x55a962ec7760 2024-09-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 0x55a962eea480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.857+0000 7fa977fff640 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa968014040 tx=0x7fa968004c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327b736043ba727a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.858+0000 7fa95dffb640 1 -- 172.21.15.16:0/2746305511 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968033070 con 0x55a962ec7760 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.858+0000 7fa95dffb640 1 -- 172.21.15.16:0/2746305511 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9680042b0 con 0x55a962ec7760 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.858+0000 7fa95dffb640 1 -- 172.21.15.16:0/2746305511 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968003830 con 0x55a962ec7760 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- 172.21.15.16:0/2746305511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 msgr2=0x55a962eca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa968014040 tx=0x7fa968004c10 comp rx=0 tx=0).stop 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- 172.21.15.16:0/2746305511 shutdown_connections 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ed2940 0x55a962eea480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ec8080 0x55a962ec2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 --2- 172.21.15.16:0/2746305511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec7760 0x55a962eca1c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- 172.21.15.16:0/2746305511 >> 172.21.15.16:0/2746305511 conn(0x55a962ec3cf0 msgr2=0x55a962ec2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- 172.21.15.16:0/2746305511 shutdown_connections 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- 172.21.15.16:0/2746305511 wait complete. 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.859+0000 7fa9809df740 1 -- start start 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 Processor -- start 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ed6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962f4c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b4fb0 con 0x55a962ec7760 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b48e0 con 0x55a962ed2940 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781b1b40 con 0x55a962ec8080 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9809df740 10 client.0 _ll_register_callbacks cb 0x7fa97009b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:48.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48806/0 (socket says 172.21.15.16:48806) 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa9777fe640 1 -- 172.21.15.16:0/2091519638 learned_addr learned my addr 172.21.15.16:0/2091519638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa977fff640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ed6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.860+0000 7fa97cba3640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962f4c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 -- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 msgr2=0x55a962f4c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962f4c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 -- 172.21.15.16:0/2091519638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 msgr2=0x55a962ed6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ed6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 -- 172.21.15.16:0/2091519638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a962f4c990 con 0x55a962ec8080 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9777fe640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009900 tx=0x7fa96c02a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e7906f3dd8b9180 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9577fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.861+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c035040 con 0x55a962ec8080 2024-09-12T03:09:48.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.862+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa96c0325c0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.862+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9680042b0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.862+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa96c0327c0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.862+0000 7fa9809df740 10 client.29593 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.862+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a962f4dad0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9680042b0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa96c003ca0 con 0x55a962ec8080 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 10 client.29593 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 10 client.29593 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 20 client.29593 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962f4e2f0 0x55a962f50710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 4 client.29593 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55a962e34d20 con 0x55a962f4e2f0 2024-09-12T03:09:48.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.863+0000 7fa977fff640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962f4e2f0 0x55a962f50710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.864+0000 7fa9577fe640 10 client.29593 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:48.864+0000 7fa977fff640 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962f4e2f0 0x55a962f50710 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8707ae8b768c2ce0 server_cookie=401103ad39cddf52 in_seq=0 out_seq=0 2024-09-12T03:09:49.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229823 24.88 MB/sec execute 410 sec latency 166.274 ms 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.607+0000 7fa9577fe640 1 -- 172.21.15.16:0/2091519638 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a962e34d20 con 0x55a962f4e2f0 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.607+0000 7fa9577fe640 10 client.29593 handle_command_reply: tid=0 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.607+0000 7fa9809df740 1 client.29593 shutdown 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 msgr2=0x55a962ed7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009900 tx=0x7fa96c02a420 comp rx=0 tx=0).stop 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 shutdown_connections 2024-09-12T03:09:49.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a962f4e2f0 0x55a962f50710 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a962ed2940 0x55a962f4c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a962ec8080 0x55a962ed7000 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9577fe640 0 client.29593 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 --2- 172.21.15.16:0/2091519638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a962ec7760 0x55a962ed6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 >> 172.21.15.16:0/2091519638 conn(0x55a962ec3cf0 msgr2=0x55a962ec2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 shutdown_connections 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 1 -- 172.21.15.16:0/2091519638 wait complete. 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.608+0000 7fa9809df740 20 client.29593 trim_cache size 0 max 16384 2024-09-12T03:09:49.609 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 78, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "a19b8590": { 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 78, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:09:49.610 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:49.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:49.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:09:49.612 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:49.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 msgr2=0x7fa9600a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:49.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 0x7fa9600a50c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa96c004650 tx=0x7fa96c005b50 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 msgr2=0x7fa978101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 0x7fa978101410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa970009fd0 tx=0x7fa97000ef10 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 shutdown_connections 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9781082a0 0x7fa978101410 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa9600a2c00 0x7fa9600a50c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9781036e0 0x7fa978100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 --2- 172.21.15.16:0/2757594839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa978102d30 0x7fa978100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 >> 172.21.15.16:0/2757594839 conn(0x7fa9780fc7d0 msgr2=0x7fa978106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 shutdown_connections 2024-09-12T03:09:49.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:49.609+0000 7fa97e606640 1 -- 172.21.15.16:0/2757594839 wait complete. 2024-09-12T03:09:49.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:09:50.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.82 MB/sec execute 411 sec latency 1166.329 ms 2024-09-12T03:09:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:50 smithi016 ceph-mon[30925]: pgmap v994: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.4 MiB/s wr, 481 op/s 2024-09-12T03:09:50.258 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a19b8590', quiesce took 4.4 seconds. Will release after: 25.6 2024-09-12T03:09:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:50.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:50 smithi033 ceph-mon[35092]: pgmap v994: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.4 MiB/s wr, 481 op/s 2024-09-12T03:09:50.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:50 smithi184 ceph-mon[35321]: pgmap v994: 97 pgs: 97 active+clean; 849 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.4 MiB/s wr, 481 op/s 2024-09-12T03:09:50.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.767+0000 7f5b90a98640 1 -- 172.21.15.16:0/3977126346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c102d90 msgr2=0x7f5b8c103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3977126346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c102d90 0x7f5b8c103170 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009930 tx=0x7f5b7c02a420 comp rx=0 tx=0).stop 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 -- 172.21.15.16:0/3977126346 shutdown_connections 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3977126346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c108290 0x7f5b8c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3977126346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 0x7f5b8c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3977126346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c102d90 0x7f5b8c103170 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 -- 172.21.15.16:0/3977126346 >> 172.21.15.16:0/3977126346 conn(0x7f5b8c0fc7d0 msgr2=0x7f5b8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 -- 172.21.15.16:0/3977126346 shutdown_connections 2024-09-12T03:09:50.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.768+0000 7f5b90a98640 1 -- 172.21.15.16:0/3977126346 wait complete. 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 Processor -- start 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 -- start start 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c102d90 0x7f5b8c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 0x7f5b8c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c112150 con 0x7f5b8c108290 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c111fd0 con 0x7f5b8c102d90 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b90a98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1122d0 con 0x7f5b8c103740 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35012/0 (socket says 172.21.15.16:35012) 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 -- 172.21.15.16:0/3893107045 learned_addr learned my addr 172.21.15.16:0/3893107045 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b83fff640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 0x7f5b8c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c102d90 0x7f5b8c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 -- 172.21.15.16:0/3893107045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 msgr2=0x7f5b8c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 0x7f5b8c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 -- 172.21.15.16:0/3893107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c102d90 msgr2=0x7f5b8c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c102d90 0x7f5b8c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.769+0000 7f5b8ad9d640 1 -- 172.21.15.16:0/3893107045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c19d6c0 con 0x7f5b8c108290 2024-09-12T03:09:50.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b8ad9d640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b970 tx=0x7f5b8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b837fe640 1 -- 172.21.15.16:0/3893107045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84019070 con 0x7f5b8c108290 2024-09-12T03:09:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c19d9b0 con 0x7f5b8c108290 2024-09-12T03:09:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b837fe640 1 -- 172.21.15.16:0/3893107045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b84014440 con 0x7f5b8c108290 2024-09-12T03:09:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b837fe640 1 -- 172.21.15.16:0/3893107045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b840075d0 con 0x7f5b8c108290 2024-09-12T03:09:50.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.770+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c1a5290 con 0x7f5b8c108290 2024-09-12T03:09:50.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.773+0000 7f5b837fe640 1 -- 172.21.15.16:0/3893107045 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5b8401c070 con 0x7f5b8c108290 2024-09-12T03:09:50.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.773+0000 7f5b92e71740 1 Processor -- start 2024-09-12T03:09:50.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.773+0000 7f5b92e71740 1 -- start start 2024-09-12T03:09:50.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d44760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44ca0 0x562164d53550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562164caa2e0 con 0x562164d53a90 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562164c51ce0 con 0x562164d44380 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b92e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562164ca62a0 con 0x562164d44ca0 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35024/0 (socket says 172.21.15.16:35024) 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 -- 172.21.15.16:0/2744148232 learned_addr learned my addr 172.21.15.16:0/2744148232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d44760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44ca0 0x562164d53550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b837fe640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 0x7f5b780a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 -- 172.21.15.16:0/2744148232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44ca0 msgr2=0x562164d53550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44ca0 0x562164d53550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 -- 172.21.15.16:0/2744148232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 msgr2=0x562164d44760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d44760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 -- 172.21.15.16:0/2744148232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562164c36440 con 0x562164d53a90 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b8ad9d640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 0x7f5b780a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b837fe640 1 -- 172.21.15.16:0/3893107045 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5b840ce5d0 con 0x7f5b8c108290 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.774+0000 7f5b83fff640 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009fa0 tx=0x7f5b740094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf0aab17aecf9c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b80ff9640 1 -- 172.21.15.16:0/2744148232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74017470 con 0x562164d53a90 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b80ff9640 1 -- 172.21.15.16:0/2744148232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7401c070 con 0x562164d53a90 2024-09-12T03:09:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b80ff9640 1 -- 172.21.15.16:0/2744148232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74015a30 con 0x562164d53a90 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 -- 172.21.15.16:0/2744148232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 msgr2=0x562164d55e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009fa0 tx=0x7f5b740094d0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b8ad9d640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 0x7f5b780a5040 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400bfd0 tx=0x7f5b84060040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 -- 172.21.15.16:0/2744148232 shutdown_connections 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d53a90 0x562164d55e80 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44ca0 0x562164d53550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 --2- 172.21.15.16:0/2744148232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d44760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.775+0000 7f5b92e71740 1 -- 172.21.15.16:0/2744148232 >> 172.21.15.16:0/2744148232 conn(0x562164d40a30 msgr2=0x562164d40e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- 172.21.15.16:0/2744148232 shutdown_connections 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- 172.21.15.16:0/2744148232 wait complete. 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- start start 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 Processor -- start 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 0x562164d50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d50bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562164d566b0 con 0x562164d44380 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562164c65050 con 0x562164d44ca0 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562164d56830 con 0x562164d53a90 2024-09-12T03:09:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b92e71740 10 client.0 _ll_register_callbacks cb 0x562164ca5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b83fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35038/0 (socket says 172.21.15.16:35038) 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b83fff640 1 -- 172.21.15.16:0/1507648498 learned_addr learned my addr 172.21.15.16:0/1507648498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.776+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 0x562164d50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d50bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 -- 172.21.15.16:0/1507648498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 msgr2=0x562164d50bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d50bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 -- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 msgr2=0x562164d50680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 0x562164d50680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 -- 172.21.15.16:0/1507648498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562164a1b3c0 con 0x562164d44380 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 0x562164d50680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b83fff640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f5b74015710 tx=0x7f5b74015610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cba2a73f18c3f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74017530 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b7401c070 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5b740176d0 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5b740178d0 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b92e71740 10 client.20457 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562164d620a0 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.777+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7401c070 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5b740061c0 con 0x562164d44380 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 10 client.20457 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 10 client.20457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 20 client.20457 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164d63830 0x562164d65c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 4 client.20457 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562164cb7850 con 0x562164d63830 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164d63830 0x562164d65c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164d63830 0x562164d65c40 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a9c9278305c40e server_cookie=c4e6e42991828af5 in_seq=0 out_seq=0 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.778+0000 7f5b71ffb640 10 client.20457 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b71ffb640 1 -- 172.21.15.16:0/1507648498 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562164cb7850 con 0x562164d63830 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b71ffb640 10 client.20457 handle_command_reply: tid=0 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 client.20457 shutdown 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 msgr2=0x562164d533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f5b74015710 tx=0x7f5b74015610 comp rx=0 tx=0).stop 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 shutdown_connections 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164d63830 0x562164d65c40 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d50bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44ca0 0x562164d50680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 --2- 172.21.15.16:0/1507648498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44380 0x562164d533d0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 >> 172.21.15.16:0/1507648498 conn(0x562164d40a30 msgr2=0x562164d46790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.779+0000 7f5b71ffb640 0 client.20457 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.780+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 shutdown_connections 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.780+0000 7f5b92e71740 1 -- 172.21.15.16:0/1507648498 wait complete. 2024-09-12T03:09:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.780+0000 7f5b92e71740 20 client.20457 trim_cache size 0 max 16384 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 Processor -- start 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 -- start start 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44380 0x562164d5f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d53a90 0x562164d3fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ab490 con 0x562164d44ca0 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1acfd0 con 0x562164d53a90 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b92e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1abb60 con 0x562164d44380 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35040/0 (socket says 172.21.15.16:35040) 2024-09-12T03:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 -- 172.21.15.16:0/3241998940 learned_addr learned my addr 172.21.15.16:0/3241998940 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b83fff640 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44380 0x562164d5f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 -- 172.21.15.16:0/3241998940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44380 msgr2=0x562164d5f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44380 0x562164d5f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b83fff640 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d53a90 0x562164d3fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 -- 172.21.15.16:0/3241998940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d53a90 msgr2=0x562164d3fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d53a90 0x562164d3fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.797+0000 7f5b8a59c640 1 -- 172.21.15.16:0/3241998940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562164d4f630 con 0x562164d44ca0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009900 tx=0x7f5b7c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6631879ce2df77b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b73fff640 1 -- 172.21.15.16:0/3241998940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c049070 con 0x562164d44ca0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b73fff640 1 -- 172.21.15.16:0/3241998940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c0043b0 con 0x562164d44ca0 2024-09-12T03:09:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b73fff640 1 -- 172.21.15.16:0/3241998940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c035040 con 0x562164d44ca0 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 -- 172.21.15.16:0/3241998940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 msgr2=0x562164d4ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009900 tx=0x7f5b7c0058d0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 -- 172.21.15.16:0/3241998940 shutdown_connections 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d53a90 0x562164d3fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d4ef50 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 --2- 172.21.15.16:0/3241998940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d44380 0x562164d5f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 -- 172.21.15.16:0/3241998940 >> 172.21.15.16:0/3241998940 conn(0x562164d40a30 msgr2=0x562164d3f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 -- 172.21.15.16:0/3241998940 shutdown_connections 2024-09-12T03:09:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.798+0000 7f5b92e71740 1 -- 172.21.15.16:0/3241998940 wait complete. 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 -- start start 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 Processor -- start 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d5c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d57e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ac900 con 0x562164d44ca0 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ac230 con 0x562164d44380 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1a9470 con 0x562164d53a90 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b92e71740 10 client.0 _ll_register_callbacks cb 0x7f5b74009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b8a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d5c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b8a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35048/0 (socket says 172.21.15.16:35048) 2024-09-12T03:09:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b8a59c640 1 -- 172.21.15.16:0/698831863 learned_addr learned my addr 172.21.15.16:0/698831863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.799+0000 7f5b83fff640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d57e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 -- 172.21.15.16:0/698831863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 msgr2=0x562164d57e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d57e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 -- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 msgr2=0x562164d5c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d5c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 -- 172.21.15.16:0/698831863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562164d44ad0 con 0x562164d44ca0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b83fff640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d57e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b8a59c640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009850 tx=0x7f5b7c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9ba91018841648 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c049050 con 0x562164d44ca0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b7c0043b0 con 0x562164d44ca0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5b7c0047a0 con 0x562164d44ca0 2024-09-12T03:09:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5b7c035040 con 0x562164d44ca0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b92e71740 10 client.20463 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562164dce4d0 con 0x562164d44ca0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c0043b0 con 0x562164d44ca0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.800+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f5b7c05f020 con 0x562164d44ca0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 10 client.20463 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 10 client.20463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 20 client.20463 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164dcec20 0x562164dd1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 4 client.20463 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562164cb1d60 con 0x562164dcec20 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b83fff640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164dcec20 0x562164dd1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b83fff640 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164dcec20 0x562164dd1040 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e020596e5720dbd4 server_cookie=eb0be0858556fa53 in_seq=0 out_seq=0 2024-09-12T03:09:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b71ffb640 10 client.20463 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b71ffb640 1 -- 172.21.15.16:0/698831863 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562164cb1d60 con 0x562164dcec20 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b71ffb640 10 client.20463 handle_command_reply: tid=0 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.801+0000 7f5b92e71740 1 client.20463 shutdown 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 msgr2=0x562164d576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009850 tx=0x7f5b7c02a420 comp rx=0 tx=0).stop 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 shutdown_connections 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562164dcec20 0x562164dd1040 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562164d53a90 0x562164d57e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b71ffb640 0 client.20463 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562164d44ca0 0x562164d576d0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 --2- 172.21.15.16:0/698831863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562164d44380 0x562164d5c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 >> 172.21.15.16:0/698831863 conn(0x562164d40a30 msgr2=0x562164d3f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 shutdown_connections 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 1 -- 172.21.15.16:0/698831863 wait complete. 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b92e71740 20 client.20463 trim_cache size 0 max 16384 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:09:50.808 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 msgr2=0x7f5b780a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 0x7f5b780a5040 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400bfd0 tx=0x7f5b84060040 comp rx=0 tx=0).stop 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 msgr2=0x7f5b8c19cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b970 tx=0x7f5b8400bd70 comp rx=0 tx=0).stop 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 shutdown_connections 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f5b780a2b80 0x7f5b780a5040 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b8c108290 0x7f5b8c19cfe0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c103740 0x7f5b8c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 --2- 172.21.15.16:0/3893107045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b8c102d90 0x7f5b8c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.802+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 >> 172.21.15.16:0/3893107045 conn(0x7f5b8c0fc7d0 msgr2=0x7f5b8c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.803+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 shutdown_connections 2024-09-12T03:09:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:50.803+0000 7f5b90a98640 1 -- 172.21.15.16:0/3893107045 wait complete. 2024-09-12T03:09:51.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.76 MB/sec execute 412 sec latency 2166.410 ms 2024-09-12T03:09:51.592 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:09:51.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:09:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:52.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.70 MB/sec execute 413 sec latency 3166.454 ms 2024-09-12T03:09:52.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:52 smithi016 ceph-mon[30925]: pgmap v995: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 649 op/s 2024-09-12T03:09:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 -- 172.21.15.16:0/161894571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 msgr2=0x7fd2941021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/161894571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd2941021c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009900 tx=0x7fd27c02a410 comp rx=0 tx=0).stop 2024-09-12T03:09:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 -- 172.21.15.16:0/161894571 shutdown_connections 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/161894571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/161894571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 0x7fd2941067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/161894571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd2941021c0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 -- 172.21.15.16:0/161894571 >> 172.21.15.16:0/161894571 conn(0x7fd2940fd8c0 msgr2=0x7fd2940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 -- 172.21.15.16:0/161894571 shutdown_connections 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.308+0000 7fd29b1c3640 1 -- 172.21.15.16:0/161894571 wait complete. 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 Processor -- start 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 -- start start 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 0x7fd2941989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd294111f70 con 0x7fd294101de0 2024-09-12T03:09:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd294111df0 con 0x7fd294106f10 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd29b1c3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2941120f0 con 0x7fd294102790 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35062/0 (socket says 172.21.15.16:35062) 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 -- 172.21.15.16:0/4184263986 learned_addr learned my addr 172.21.15.16:0/4184263986 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd293fff640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 0x7fd2941989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd299760640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 -- 172.21.15.16:0/4184263986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 msgr2=0x7fd2941989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 0x7fd2941989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 -- 172.21.15.16:0/4184263986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 msgr2=0x7fd29419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd298f5f640 1 -- 172.21.15.16:0/4184263986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29419d420 con 0x7fd294101de0 2024-09-12T03:09:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.309+0000 7fd299760640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd298f5f640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005ab0 tx=0x7fd27c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c03f030 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd27c003ce0 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29419d6b0 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c035070 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.310+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2941a4ff0 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.311+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2941032c0 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.312+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd27c042030 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.313+0000 7fd291ffb640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 0x7fd2800a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.313+0000 7fd293fff640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 0x7fd2800a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.313+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd27c0ef9a0 con 0x7fd294101de0 2024-09-12T03:09:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.313+0000 7fd293fff640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 0x7fd2800a5040 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd2940fdd50 tx=0x7fd28400a570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.317+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd27c0bc8b0 con 0x7fd294101de0 2024-09-12T03:09:52.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:52 smithi033 ceph-mon[35092]: pgmap v995: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 649 op/s 2024-09-12T03:09:52.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:52 smithi184 ceph-mon[35321]: pgmap v995: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.9 MiB/s wr, 649 op/s 2024-09-12T03:09:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.642+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd29410a0f0 con 0x7fd294101de0 2024-09-12T03:09:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.643+0000 7fd291ffb640 1 -- 172.21.15.16:0/4184263986 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fd27c0bc1b0 con 0x7fd294101de0 2024-09-12T03:09:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:09:52.644 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:09:52.644 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 msgr2=0x7fd2800a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 0x7fd2800a5040 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd2940fdd50 tx=0x7fd28400a570 comp rx=0 tx=0).stop 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 msgr2=0x7fd294198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005ab0 tx=0x7fd27c0042f0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 shutdown_connections 2024-09-12T03:09:52.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd294106f10 0x7fd29419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd294102790 0x7fd2941989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd2800a2b80 0x7fd2800a5040 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 --2- 172.21.15.16:0/4184263986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd294101de0 0x7fd294198470 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 >> 172.21.15.16:0/4184263986 conn(0x7fd2940fd8c0 msgr2=0x7fd2940fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 shutdown_connections 2024-09-12T03:09:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:52.646+0000 7fd29b1c3640 1 -- 172.21.15.16:0/4184263986 wait complete. 2024-09-12T03:09:53.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.64 MB/sec execute 414 sec latency 4166.561 ms 2024-09-12T03:09:53.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4184263986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:53.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4184263986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:53.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:09:53.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4184263986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:09:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:54.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.58 MB/sec execute 415 sec latency 5166.607 ms 2024-09-12T03:09:54.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:54 smithi016 ceph-mon[30925]: pgmap v996: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-12T03:09:54.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 -- 172.21.15.16:0/2995481263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 msgr2=0x7f1998103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 --2- 172.21.15.16:0/2995481263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998103120 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1988009ca0 tx=0x7f198802a950 comp rx=0 tx=0).stop 2024-09-12T03:09:54.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 -- 172.21.15.16:0/2995481263 shutdown_connections 2024-09-12T03:09:54.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 --2- 172.21.15.16:0/2995481263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f199810c170 0x7f199810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 --2- 172.21.15.16:0/2995481263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1998103660 0x7f199810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 --2- 172.21.15.16:0/2995481263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998103120 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 -- 172.21.15.16:0/2995481263 >> 172.21.15.16:0/2995481263 conn(0x7f19980fc7a0 msgr2=0x7f19980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 -- 172.21.15.16:0/2995481263 shutdown_connections 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.320+0000 7f199c939640 1 -- 172.21.15.16:0/2995481263 wait complete. 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 Processor -- start 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 -- start start 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 0x7f19981989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 0x7f199819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1998111f70 con 0x7f1998102d40 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1998111df0 con 0x7f1998103660 2024-09-12T03:09:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199c939640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f19981120f0 con 0x7f199810c170 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35082/0 (socket says 172.21.15.16:35082) 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 -- 172.21.15.16:0/2796553813 learned_addr learned my addr 172.21.15.16:0/2796553813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f1995d9b640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 0x7f19981989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f1996d9d640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 0x7f199819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 -- 172.21.15.16:0/2796553813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 msgr2=0x7f199819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 0x7f199819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 -- 172.21.15.16:0/2796553813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 msgr2=0x7f19981989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 0x7f19981989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f199659c640 1 -- 172.21.15.16:0/2796553813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199819d420 con 0x7f1998102d40 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f1995d9b640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 0x7f19981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:54.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.321+0000 7f1996d9d640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 0x7f199819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f199659c640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1988005cc0 tx=0x7f198802f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f198f7fe640 1 -- 172.21.15.16:0/2796553813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988034040 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f198f7fe640 1 -- 172.21.15.16:0/2796553813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1988030920 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199819d6b0 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f198f7fe640 1 -- 172.21.15.16:0/2796553813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988004780 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.322+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19981a4ff0 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.324+0000 7f198f7fe640 1 -- 172.21.15.16:0/2796553813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1988037020 con 0x7f1998102d40 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.324+0000 7f199ed12740 1 Processor -- start 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 -- start start 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 0x555e46e51a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e5c370 0x555e46e5e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555e46db72f0 con 0x555e46e52b40 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555e46d5ecf0 con 0x555e46e5c370 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199ed12740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555e46db32b0 con 0x555e46e515d0 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1996d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 0x555e46e51a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1996d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35088/0 (socket says 172.21.15.16:35088) 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1996d9d640 1 -- 172.21.15.16:0/3030338381 learned_addr learned my addr 172.21.15.16:0/3030338381 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f198f7fe640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 0x7f19840a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e5c370 0x555e46e5e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 -- 172.21.15.16:0/3030338381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e5c370 msgr2=0x555e46e5e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e5c370 0x555e46e5e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 -- 172.21.15.16:0/3030338381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 msgr2=0x555e46e51090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f199659c640 1 -- 172.21.15.16:0/3030338381 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e46d433a0 con 0x555e46e515d0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1996d9d640 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f1996d9d640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 0x7f19840a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.325+0000 7f198f7fe640 1 -- 172.21.15.16:0/2796553813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f19880eff30 con 0x7f1998102d40 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.326+0000 7f199659c640 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 0x555e46e51a10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f19880eda90 tx=0x7f198807c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26806abed7bf015b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.326+0000 7f1996d9d640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 0x7f19840a50c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1990005940 tx=0x7f19900046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.326+0000 7f198cff9640 1 -- 172.21.15.16:0/3030338381 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198802c040 con 0x555e46e515d0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.326+0000 7f198cff9640 1 -- 172.21.15.16:0/3030338381 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1988003e10 con 0x555e46e515d0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.326+0000 7f198cff9640 1 -- 172.21.15.16:0/3030338381 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880c5410 con 0x555e46e515d0 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- 172.21.15.16:0/3030338381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 msgr2=0x555e46e51a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 0x555e46e51a10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f19880eda90 tx=0x7f198807c890 comp rx=0 tx=0).stop 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- 172.21.15.16:0/3030338381 shutdown_connections 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e5c370 0x555e46e5e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e515d0 0x555e46e51a10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- 172.21.15.16:0/3030338381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e51090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- 172.21.15.16:0/3030338381 >> 172.21.15.16:0/3030338381 conn(0x555e46e1a310 msgr2=0x555e46e4dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- 172.21.15.16:0/3030338381 shutdown_connections 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- 172.21.15.16:0/3030338381 wait complete. 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- start start 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 Processor -- start 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e5bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555e46e5ee30 con 0x555e46e52b40 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555e46d72060 con 0x555e46e515d0 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555e46e5efb0 con 0x555e46e5c370 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.327+0000 7f199ed12740 10 client.0 _ll_register_callbacks cb 0x555e46ac58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35090/0 (socket says 172.21.15.16:35090) 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 -- 172.21.15.16:0/3534318287 learned_addr learned my addr 172.21.15.16:0/3534318287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1996d9d640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f199659c640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e5bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 -- 172.21.15.16:0/3534318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 msgr2=0x555e46e597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 -- 172.21.15.16:0/3534318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 msgr2=0x555e46e5bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e5bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 -- 172.21.15.16:0/3534318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e46b283c0 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1996d9d640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f1980012720 tx=0x7f1980012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b92bcc86f20247 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f197dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980011bb0 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1980011d50 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555e46dc4860 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.328+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1980005020 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 10 client.20481 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555e46e6a860 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555e46dc4860 con 0x555e46e52b40 2024-09-12T03:09:54.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f198001c470 con 0x555e46e52b40 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 10 client.20481 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 10 client.20481 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 20 client.20481 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46e6bd70 0x555e46e6e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 4 client.20481 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e46dbed70 con 0x555e46e6bd70 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199659c640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46e6bd70 0x555e46e6e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f197dffb640 10 client.20481 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.329+0000 7f199659c640 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46e6bd70 0x555e46e6e1b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=239b51f945d8c4cc server_cookie=a9ccca1ce0a17b8 in_seq=0 out_seq=0 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f197dffb640 1 -- 172.21.15.16:0/3534318287 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555e46dbed70 con 0x555e46e6bd70 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f197dffb640 10 client.20481 handle_command_reply: tid=0 2024-09-12T03:09:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 client.20481 shutdown 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 msgr2=0x555e46e59050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f1980012720 tx=0x7f1980012be0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 shutdown_connections 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e52b40 0x555e46e59050 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46e6bd70 0x555e46e6e1b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 --2- 172.21.15.16:0/3534318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e5bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 >> 172.21.15.16:0/3534318287 conn(0x555e46e1a310 msgr2=0x555e46e53960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f197dffb640 0 client.20481 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 shutdown_connections 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.330+0000 7f199ed12740 1 -- 172.21.15.16:0/3534318287 wait complete. 2024-09-12T03:09:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.331+0000 7f199ed12740 20 client.20481 trim_cache size 0 max 16384 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.343+0000 7f199ed12740 1 Processor -- start 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.343+0000 7f199ed12740 1 -- start start 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 0x555e46e54660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e73a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ad450 con 0x555e46e515d0 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19981aef90 con 0x555e46e52b40 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199ed12740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f19981adb20 con 0x555e46e5c370 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35092/0 (socket says 172.21.15.16:35092) 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 -- 172.21.15.16:0/2514552111 learned_addr learned my addr 172.21.15.16:0/2514552111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f1996d9d640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e73a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f1995d9b640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 0x555e46e54660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 -- 172.21.15.16:0/2514552111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 msgr2=0x555e46e73a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e73a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 -- 172.21.15.16:0/2514552111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 msgr2=0x555e46e54660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 0x555e46e54660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 -- 172.21.15.16:0/2514552111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e46e684a0 con 0x555e46e515d0 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f1995d9b640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 0x555e46e54660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f199659c640 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f198802fde0 tx=0x7f19880c0b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ced161d0ee0fa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.344+0000 7f197ffff640 1 -- 172.21.15.16:0/2514552111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880bcea0 con 0x555e46e515d0 2024-09-12T03:09:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f197ffff640 1 -- 172.21.15.16:0/2514552111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19880c5410 con 0x555e46e515d0 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f197ffff640 1 -- 172.21.15.16:0/2514552111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880db0d0 con 0x555e46e515d0 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 -- 172.21.15.16:0/2514552111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 msgr2=0x555e46e54120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f198802fde0 tx=0x7f19880c0b50 comp rx=0 tx=0).stop 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 -- 172.21.15.16:0/2514552111 shutdown_connections 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e5c370 0x555e46e73a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e52b40 0x555e46e54660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 --2- 172.21.15.16:0/2514552111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e515d0 0x555e46e54120 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 -- 172.21.15.16:0/2514552111 >> 172.21.15.16:0/2514552111 conn(0x555e46e1a310 msgr2=0x555e46e58140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 -- 172.21.15.16:0/2514552111 shutdown_connections 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.345+0000 7f199ed12740 1 -- 172.21.15.16:0/2514552111 wait complete. 2024-09-12T03:09:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 -- start start 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 Processor -- start 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e52b40 0x555e46e5fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 0x555e46e600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ae8c0 con 0x555e46e5c370 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ae1f0 con 0x555e46e515d0 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ab430 con 0x555e46e52b40 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199ed12740 10 client.0 _ll_register_callbacks cb 0x7f1980009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f1996d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 0x555e46e600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f1995d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e52b40 0x555e46e5fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:48138/0 (socket says 172.21.15.16:48138) 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f1996d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 0x555e46e600f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35106/0 (socket says 172.21.15.16:35106) 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 -- 172.21.15.16:0/3166945728 learned_addr learned my addr 172.21.15.16:0/3166945728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 -- 172.21.15.16:0/3166945728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e52b40 msgr2=0x555e46e5fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e52b40 0x555e46e5fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 -- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 msgr2=0x555e46e600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 0x555e46e600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.346+0000 7f199659c640 1 -- 172.21.15.16:0/3166945728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e46e60920 con 0x555e46e515d0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199659c640 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1988004af0 tx=0x7f198802ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755f758346bc0327 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880b8a40 con 0x555e46e515d0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19880c5410 con 0x555e46e515d0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f19880a70c0 con 0x555e46e515d0 2024-09-12T03:09:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1988004040 con 0x555e46e515d0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 10 client.29690 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555e46edae70 con 0x555e46e515d0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19880c5410 con 0x555e46e515d0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f198804d030 con 0x555e46e515d0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 10 client.29690 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 10 client.29690 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 20 client.29690 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46edb160 0x555e46edd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 4 client.29690 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.347+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e46d3b170 con 0x555e46edb160 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46edb160 0x555e46edd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f1995d9b640 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46edb160 0x555e46edd550 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=81c09cf6120191bf server_cookie=7978543bafe4d230 in_seq=0 out_seq=0 2024-09-12T03:09:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f197dffb640 10 client.29690 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f197dffb640 1 -- 172.21.15.16:0/3166945728 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e46d3b170 con 0x555e46edb160 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f197dffb640 10 client.29690 handle_command_reply: tid=0 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 client.29690 shutdown 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 msgr2=0x555e46e64840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1988004af0 tx=0x7f198802ff50 comp rx=0 tx=0).stop 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 shutdown_connections 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555e46e5c370 0x555e46e600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e46e52b40 0x555e46e5fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x555e46edb160 0x555e46edd550 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 --2- 172.21.15.16:0/3166945728 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555e46e515d0 0x555e46e64840 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f197dffb640 0 client.29690 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 >> 172.21.15.16:0/3166945728 conn(0x555e46e1a310 msgr2=0x555e46e4c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.348+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 shutdown_connections 2024-09-12T03:09:54.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199ed12740 1 -- 172.21.15.16:0/3166945728 wait complete. 2024-09-12T03:09:54.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199ed12740 20 client.29690 trim_cache size 0 max 16384 2024-09-12T03:09:54.357 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 msgr2=0x7f19840a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 0x7f19840a50c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1990005940 tx=0x7f19900046c0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 msgr2=0x7f1998198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1988005cc0 tx=0x7f198802f630 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 shutdown_connections 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f199810c170 0x7f199819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1998103660 0x7f19981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f19840a2c00 0x7f19840a50c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 --2- 172.21.15.16:0/2796553813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1998102d40 0x7f1998198470 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 >> 172.21.15.16:0/2796553813 conn(0x7f19980fc7a0 msgr2=0x7f19980fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 shutdown_connections 2024-09-12T03:09:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:54.349+0000 7f199c939640 1 -- 172.21.15.16:0/2796553813 wait complete. 2024-09-12T03:09:54.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:54 smithi033 ceph-mon[35092]: pgmap v996: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-12T03:09:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:54 smithi184 ceph-mon[35321]: pgmap v996: 97 pgs: 97 active+clean; 857 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.5 MiB/s wr, 554 op/s 2024-09-12T03:09:55.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.52 MB/sec execute 416 sec latency 6166.712 ms 2024-09-12T03:09:55.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:09:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 -- 172.21.15.16:0/3512591588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70102570 msgr2=0x7f1c701029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/3512591588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70102570 0x7f1c701029b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009900 tx=0x7f1c5802a480 comp rx=0 tx=0).stop 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 -- 172.21.15.16:0/3512591588 shutdown_connections 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/3512591588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/3512591588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70102570 0x7f1c701029b0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/3512591588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70100e20 0x7f1c70101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 -- 172.21.15.16:0/3512591588 >> 172.21.15.16:0/3512591588 conn(0x7f1c700fc7d0 msgr2=0x7f1c700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 -- 172.21.15.16:0/3512591588 shutdown_connections 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.861+0000 7f1c74fa6640 1 -- 172.21.15.16:0/3512591588 wait complete. 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 Processor -- start 2024-09-12T03:09:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 -- start start 2024-09-12T03:09:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70102570 0x7f1c70198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70110f80 con 0x7f1c70100e20 2024-09-12T03:09:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70110e00 con 0x7f1c70102570 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c74fa6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70111100 con 0x7f1c70102ef0 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34902/0 (socket says 172.21.15.16:34902) 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c6e59c640 1 -- 172.21.15.16:0/1171368230 learned_addr learned my addr 172.21.15.16:0/1171368230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.862+0000 7f1c67fff640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70102570 0x7f1c70198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 -- 172.21.15.16:0/1171368230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 msgr2=0x7f1c7019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 -- 172.21.15.16:0/1171368230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70102570 msgr2=0x7f1c70198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70102570 0x7f1c70198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 -- 172.21.15.16:0/1171368230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7019d3e0 con 0x7f1c70100e20 2024-09-12T03:09:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000c920 tx=0x7f1c6000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c677fe640 1 -- 172.21.15.16:0/1171368230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60007c70 con 0x7f1c70100e20 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7019d6d0 con 0x7f1c70100e20 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c677fe640 1 -- 172.21.15.16:0/1171368230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c60007e10 con 0x7f1c70100e20 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c677fe640 1 -- 172.21.15.16:0/1171368230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60018890 con 0x7f1c70100e20 2024-09-12T03:09:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.863+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c701a4fb0 con 0x7f1c70100e20 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c677fe640 1 -- 172.21.15.16:0/1171368230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1c60018a70 con 0x7f1c70100e20 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 Processor -- start 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 -- start start 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x56468942da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x56468943a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5646893932f0 con 0x56468942eb40 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56468933acf0 con 0x564689438370 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c7737f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56468938f2b0 con 0x56468942d5d0 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.866+0000 7f1c677fe640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 0x7f1c5c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34906/0 (socket says 172.21.15.16:34906) 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/922055575 learned_addr learned my addr 172.21.15.16:0/922055575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 0x7f1c5c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x56468942da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c677fe640 1 -- 172.21.15.16:0/1171368230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1c600cf1e0 con 0x7f1c70100e20 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c67fff640 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x56468943a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/922055575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 msgr2=0x56468942da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x56468942da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/922055575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 msgr2=0x56468943a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x56468943a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/922055575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56468931f3a0 con 0x56468942eb40 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f1c68012d70 tx=0x7f1c68010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8641c378f60bc13c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 0x7f1c5c0a5040 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1c68010c40 tx=0x7f1c68017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c64ff9640 1 -- 172.21.15.16:0/922055575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68023070 con 0x56468942eb40 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.867+0000 7f1c64ff9640 1 -- 172.21.15.16:0/922055575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c68009d20 con 0x56468942eb40 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c64ff9640 1 -- 172.21.15.16:0/922055575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68005260 con 0x56468942eb40 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 -- 172.21.15.16:0/922055575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 msgr2=0x56468942d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f1c68012d70 tx=0x7f1c68010740 comp rx=0 tx=0).stop 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 -- 172.21.15.16:0/922055575 shutdown_connections 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x56468943a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x56468942da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 --2- 172.21.15.16:0/922055575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x56468942d090 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 -- 172.21.15.16:0/922055575 >> 172.21.15.16:0/922055575 conn(0x5646893f6310 msgr2=0x564689429c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 -- 172.21.15.16:0/922055575 shutdown_connections 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.868+0000 7f1c7737f740 1 -- 172.21.15.16:0/922055575 wait complete. 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 -- start start 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 Processor -- start 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x564689437da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x5646894357e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56468943b010 con 0x56468942eb40 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56468934e060 con 0x564689438370 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56468943b190 con 0x56468942d5d0 2024-09-12T03:09:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c7737f740 10 client.0 _ll_register_callbacks cb 0x5646890a18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34910/0 (socket says 172.21.15.16:34910) 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c67fff640 1 -- 172.21.15.16:0/2686775032 learned_addr learned my addr 172.21.15.16:0/2686775032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x564689437da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.869+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x5646894357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 -- 172.21.15.16:0/2686775032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 msgr2=0x564689437da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x564689437da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 -- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 msgr2=0x5646894357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x5646894357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 -- 172.21.15.16:0/2686775032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646891043c0 con 0x56468942eb40 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1c5802a9a0 tx=0x7f1c580043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca8e4d4f5edf347 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58035030 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1c58003a00 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5646893a0860 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1c58003c00 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c7737f740 10 client.20499 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564689446af0 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.870+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5646893a0860 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1c5803c440 con 0x56468942eb40 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 10 client.20499 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 10 client.20499 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 20 client.20499 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x564689447fb0 0x56468944a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 4 client.20499 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56468939ad70 con 0x564689447fb0 2024-09-12T03:09:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x564689447fb0 0x56468944a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x564689447fb0 0x56468944a3f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ec3e5fe3b51ce6b9 server_cookie=906acabfe4a13b2e in_seq=0 out_seq=0 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.871+0000 7f1c55ffb640 10 client.20499 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2686775032 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56468939ad70 con 0x564689447fb0 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c55ffb640 10 client.20499 handle_command_reply: tid=0 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 client.20499 shutdown 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 msgr2=0x564689435050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1c5802a9a0 tx=0x7f1c580043b0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 shutdown_connections 2024-09-12T03:09:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x564689447fb0 0x56468944a3f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564689438370 0x5646894357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689435050 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c55ffb640 0 client.20499 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2686775032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942d5d0 0x564689437da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 >> 172.21.15.16:0/2686775032 conn(0x5646893f6310 msgr2=0x56468942f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 shutdown_connections 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.872+0000 7f1c7737f740 1 -- 172.21.15.16:0/2686775032 wait complete. 2024-09-12T03:09:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.873+0000 7f1c7737f740 20 client.20499 trim_cache size 0 max 16384 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.885+0000 7f1c7737f740 1 Processor -- start 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.885+0000 7f1c7737f740 1 -- start start 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689434fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942eb40 0x564689434090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701ad470 con 0x564689438370 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701aefb0 con 0x56468942d5d0 2024-09-12T03:09:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c7737f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701adb40 con 0x56468942eb40 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34918/0 (socket says 172.21.15.16:34918) 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/2910106618 learned_addr learned my addr 172.21.15.16:0/2910106618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942eb40 0x564689434090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689434fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/2910106618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942eb40 msgr2=0x564689434090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942eb40 0x564689434090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/2910106618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 msgr2=0x564689434fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689434fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 -- 172.21.15.16:0/2910106618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56468942a960 con 0x564689438370 2024-09-12T03:09:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.886+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ce90 tx=0x7f1c6800ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cdde1c3a2b23cf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c57fff640 1 -- 172.21.15.16:0/2910106618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68023040 con 0x564689438370 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c57fff640 1 -- 172.21.15.16:0/2910106618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6800cb40 con 0x564689438370 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c57fff640 1 -- 172.21.15.16:0/2910106618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c680056c0 con 0x564689438370 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- 172.21.15.16:0/2910106618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 msgr2=0x564689440800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ce90 tx=0x7f1c6800ce20 comp rx=0 tx=0).stop 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- 172.21.15.16:0/2910106618 shutdown_connections 2024-09-12T03:09:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564689438370 0x564689440800 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56468942eb40 0x564689434090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2910106618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689434fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- 172.21.15.16:0/2910106618 >> 172.21.15.16:0/2910106618 conn(0x5646893f6310 msgr2=0x564689433980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- 172.21.15.16:0/2910106618 shutdown_connections 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- 172.21.15.16:0/2910106618 wait complete. 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.887+0000 7f1c7737f740 1 -- start start 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 Processor -- start 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689431320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 0x564689431da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701ae8e0 con 0x56468942eb40 2024-09-12T03:09:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701ae210 con 0x56468942d5d0 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701ab450 con 0x564689438370 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c7737f740 10 client.0 _ll_register_callbacks cb 0x7f1c58009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34926/0 (socket says 172.21.15.16:34926) 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689431320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 -- 172.21.15.16:0/2586811212 learned_addr learned my addr 172.21.15.16:0/2586811212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 0x564689431da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 -- 172.21.15.16:0/2586811212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 msgr2=0x564689431da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 0x564689431da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 -- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 msgr2=0x564689431320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689431320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c67fff640 1 -- 172.21.15.16:0/2586811212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564689430120 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.888+0000 7f1c6ed9d640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 0x564689431da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c67fff640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c58031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828ae632cff5ba3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58035070 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1c58003800 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6800cb40 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1c5803cc90 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 10 client.20505 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5646894b6e90 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6800cb40 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1c580049a0 con 0x56468942eb40 2024-09-12T03:09:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 10 client.20505 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 10 client.20505 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 20 client.20505 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5646894b75b0 0x5646894b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 4 client.20505 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.889+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564689317170 con 0x5646894b75b0 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5646894b75b0 0x5646894b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c6e59c640 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5646894b75b0 0x5646894b99a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=38949f0fe85be7f6 server_cookie=d9a10d765274f77a in_seq=0 out_seq=0 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c55ffb640 10 client.20505 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c55ffb640 1 -- 172.21.15.16:0/2586811212 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564689317170 con 0x5646894b75b0 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c55ffb640 10 client.20505 handle_command_reply: tid=0 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 client.20505 shutdown 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 msgr2=0x564689431860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c58031400 comp rx=0 tx=0).stop 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 shutdown_connections 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5646894b75b0 0x5646894b99a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564689438370 0x564689431da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56468942eb40 0x564689431860 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 --2- 172.21.15.16:0/2586811212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56468942d5d0 0x564689431320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 >> 172.21.15.16:0/2586811212 conn(0x5646893f6310 msgr2=0x5646894341a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c55ffb640 0 client.20505 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 shutdown_connections 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.890+0000 7f1c7737f740 1 -- 172.21.15.16:0/2586811212 wait complete. 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c7737f740 20 client.20505 trim_cache size 0 max 16384 2024-09-12T03:09:55.896 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 msgr2=0x7f1c5c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 0x7f1c5c0a5040 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1c68010c40 tx=0x7f1c68017000 comp rx=0 tx=0).stop 2024-09-12T03:09:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 msgr2=0x7f1c70198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000c920 tx=0x7f1c6000cde0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 shutdown_connections 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c70102ef0 0x7f1c7019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c70102570 0x7f1c70198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1c5c0a2b80 0x7f1c5c0a5040 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 --2- 172.21.15.16:0/1171368230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c70100e20 0x7f1c70198430 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 >> 172.21.15.16:0/1171368230 conn(0x7f1c700fc7d0 msgr2=0x7f1c7006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 shutdown_connections 2024-09-12T03:09:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:55.891+0000 7f1c74fa6640 1 -- 172.21.15.16:0/1171368230 wait complete. 2024-09-12T03:09:56.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.47 MB/sec execute 417 sec latency 7166.813 ms 2024-09-12T03:09:56.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:56 smithi016 ceph-mon[30925]: pgmap v997: 97 pgs: 97 active+clean; 856 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 512 op/s 2024-09-12T03:09:56.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:56 smithi033 ceph-mon[35092]: pgmap v997: 97 pgs: 97 active+clean; 856 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 512 op/s 2024-09-12T03:09:56.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:56 smithi184 ceph-mon[35321]: pgmap v997: 97 pgs: 97 active+clean; 856 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 512 op/s 2024-09-12T03:09:56.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:09:57.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.41 MB/sec execute 418 sec latency 8166.867 ms 2024-09-12T03:09:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:57.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 -- 172.21.15.16:0/967193462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 msgr2=0x7f99f010d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 --2- 172.21.15.16:0/967193462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f010d890 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009ca0 tx=0x7f99e802a950 comp rx=0 tx=0).stop 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 -- 172.21.15.16:0/967193462 shutdown_connections 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 --2- 172.21.15.16:0/967193462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f010d890 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 --2- 172.21.15.16:0/967193462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0101790 0x7f99f0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 --2- 172.21.15.16:0/967193462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0105ba0 0x7f99f0105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 -- 172.21.15.16:0/967193462 >> 172.21.15.16:0/967193462 conn(0x7f99f0077210 msgr2=0x7f99f0077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 -- 172.21.15.16:0/967193462 shutdown_connections 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.754+0000 7f99f6774640 1 -- 172.21.15.16:0/967193462 wait complete. 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 Processor -- start 2024-09-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 -- start start 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 0x7f99f01991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0105ba0 0x7f99f01a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110e60 con 0x7f99f0102110 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110ce0 con 0x7f99f0101790 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f6774640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110fe0 con 0x7f99f0105ba0 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34934/0 (socket says 172.21.15.16:34934) 2024-09-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99ef7fe640 1 -- 172.21.15.16:0/1506272990 learned_addr learned my addr 172.21.15.16:0/1506272990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99effff640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 0x7f99f01991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.755+0000 7f99f4d11640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0105ba0 0x7f99f01a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 -- 172.21.15.16:0/1506272990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0105ba0 msgr2=0x7f99f01a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0105ba0 0x7f99f01a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 -- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 msgr2=0x7f99f01996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 -- 172.21.15.16:0/1506272990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f01a4770 con 0x7f99f0101790 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99effff640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 0x7f99f01991b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99ed7fa640 1 -- 172.21.15.16:0/1506272990 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0018070 con 0x7f99f0101790 2024-09-12T03:09:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99ed7fa640 1 -- 172.21.15.16:0/1506272990 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0009c30 con 0x7f99f0101790 2024-09-12T03:09:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f0196fa0 con 0x7f99f0101790 2024-09-12T03:09:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99ed7fa640 1 -- 172.21.15.16:0/1506272990 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00035f0 con 0x7f99f0101790 2024-09-12T03:09:57.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.756+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f01974e0 con 0x7f99f0101790 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 Processor -- start 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 -- start start 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99ed7fa640 1 -- 172.21.15.16:0/1506272990 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f99e0025440 con 0x7f99f0101790 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55de9dd682f0 con 0x55de9de03b40 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55de9dd0fcf0 con 0x55de9de0d370 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55de9dd642b0 con 0x55de9de025d0 2024-09-12T03:09:57.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99f4d11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.760+0000 7f99effff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58688/0 (socket says 172.21.15.16:58688) 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99f4d11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de02090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34936/0 (socket says 172.21.15.16:34936) 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 -- 172.21.15.16:0/1802776361 learned_addr learned my addr 172.21.15.16:0/1802776361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 -- 172.21.15.16:0/1802776361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 msgr2=0x55de9de0f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de0f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 -- 172.21.15.16:0/1802776361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 msgr2=0x55de9de02090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de02090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 -- 172.21.15.16:0/1802776361 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de9dcf43a0 con 0x55de9de025d0 2024-09-12T03:09:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99effff640 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f99e00d6b70 tx=0x7f99e00d6c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f2260835753f492 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99ed7fa640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 0x7f99d80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99f4d11640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 0x7f99d80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.761+0000 7f99d67fc640 1 -- 172.21.15.16:0/1802776361 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e009b670 con 0x55de9de025d0 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99ed7fa640 1 -- 172.21.15.16:0/1506272990 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f99e00d2b60 con 0x7f99f0101790 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99d67fc640 1 -- 172.21.15.16:0/1802776361 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e009fad0 con 0x55de9de025d0 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99d67fc640 1 -- 172.21.15.16:0/1802776361 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00a6b00 con 0x55de9de025d0 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1802776361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 msgr2=0x55de9de02a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f4d11640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 0x7f99d80a5040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f99e8005e30 tx=0x7f99e8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f99e00d6b70 tx=0x7f99e00d6c60 comp rx=0 tx=0).stop 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1802776361 shutdown_connections 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de02a10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1802776361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de02090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1802776361 >> 172.21.15.16:0/1802776361 conn(0x55de9ddcb310 msgr2=0x55de9ddfec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.762+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1802776361 shutdown_connections 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.763+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1802776361 wait complete. 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.763+0000 7f99f8b4d740 1 -- start start 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.763+0000 7f99f8b4d740 1 Processor -- start 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de0a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55de9de10070 con 0x55de9de025d0 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55de9dd23060 con 0x55de9de03b40 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55de9de101f0 con 0x55de9de0d370 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f4d11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de0a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f4d11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de0a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58700/0 (socket says 172.21.15.16:58700) 2024-09-12T03:09:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f4d11640 1 -- 172.21.15.16:0/1117808239 learned_addr learned my addr 172.21.15.16:0/1117808239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34556/0 (socket says 172.21.15.16:34556) 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99f8b4d740 10 client.0 _ll_register_callbacks cb 0x55de9da768c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 -- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 msgr2=0x55de9de0a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de0a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 -- 172.21.15.16:0/1117808239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 msgr2=0x55de9de0cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99ef7fe640 1 -- 172.21.15.16:0/1117808239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de9dad93c0 con 0x55de9de03b40 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.764+0000 7f99effff640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f99f0101210 tx=0x7f99e400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25bf5693529ce982 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e40120d0 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f99e4010430 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e009fad0 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f99e4012270 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99f8b4d740 10 client.29714 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55de9de243c0 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e009fad0 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.765+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f99e401c7b0 con 0x55de9de03b40 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 10 client.29714 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 10 client.29714 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 20 client.29714 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de257d0 0x55de9de27c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 4 client.29714 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de9dd75860 con 0x55de9de257d0 2024-09-12T03:09:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99effff640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de257d0 0x55de9de27c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99effff640 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de257d0 0x55de9de27c50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5de2fd3bcb344e3f server_cookie=de709dbd6d4905ef in_seq=0 out_seq=0 2024-09-12T03:09:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.766+0000 7f99cf7fe640 10 client.29714 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.767+0000 7f99cf7fe640 1 -- 172.21.15.16:0/1117808239 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de9dd75860 con 0x55de9de257d0 2024-09-12T03:09:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.767+0000 7f99cf7fe640 10 client.29714 handle_command_reply: tid=0 2024-09-12T03:09:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.767+0000 7f99f8b4d740 1 client.29714 shutdown 2024-09-12T03:09:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 msgr2=0x55de9de0a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f99f0101210 tx=0x7f99e400f790 comp rx=0 tx=0).stop 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 shutdown_connections 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de257d0 0x55de9de27c50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99cf7fe640 0 client.29714 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de0a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de03b40 0x55de9de0a050 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/1117808239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 >> 172.21.15.16:0/1117808239 conn(0x55de9ddcb310 msgr2=0x55de9de04960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 shutdown_connections 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 1 -- 172.21.15.16:0/1117808239 wait complete. 2024-09-12T03:09:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.768+0000 7f99f8b4d740 20 client.29714 trim_cache size 0 max 16384 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.781+0000 7f99f8b4d740 1 Processor -- start 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.781+0000 7f99f8b4d740 1 -- start start 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.781+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de05120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01ab3d0 con 0x55de9de03b40 2024-09-12T03:09:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01acf10 con 0x55de9de0d370 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01abaa0 con 0x55de9de025d0 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34954/0 (socket says 172.21.15.16:34954) 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 -- 172.21.15.16:0/151713419 learned_addr learned my addr 172.21.15.16:0/151713419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99f4d11640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de05120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99effff640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 -- 172.21.15.16:0/151713419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 msgr2=0x55de9de0cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 -- 172.21.15.16:0/151713419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 msgr2=0x55de9de05120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de05120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 -- 172.21.15.16:0/151713419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de9de22000 con 0x55de9de03b40 2024-09-12T03:09:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99effff640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f99e401cdc0 tx=0x7f99e400fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf18c06187d50ea server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99d5ffb640 1 -- 172.21.15.16:0/151713419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e401c420 con 0x55de9de03b40 2024-09-12T03:09:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99d5ffb640 1 -- 172.21.15.16:0/151713419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e402b020 con 0x55de9de03b40 2024-09-12T03:09:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.782+0000 7f99d5ffb640 1 -- 172.21.15.16:0/151713419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4004770 con 0x55de9de03b40 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- 172.21.15.16:0/151713419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 msgr2=0x55de9de03300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f99e401cdc0 tx=0x7f99e400fc90 comp rx=0 tx=0).stop 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- 172.21.15.16:0/151713419 shutdown_connections 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de0d370 0x55de9de05120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de03300 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/151713419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de025d0 0x55de9de0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- 172.21.15.16:0/151713419 >> 172.21.15.16:0/151713419 conn(0x55de9ddcb310 msgr2=0x55de9ddf9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- 172.21.15.16:0/151713419 shutdown_connections 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- 172.21.15.16:0/151713419 wait complete. 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.783+0000 7f99f8b4d740 1 -- start start 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 Processor -- start 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de1e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de19620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01ac840 con 0x55de9de03b40 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01ac170 con 0x55de9de025d0 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01a93b0 con 0x55de9de0d370 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f8b4d740 10 client.0 _ll_register_callbacks cb 0x7f99e4009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99effff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de1e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99effff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34578/0 (socket says 172.21.15.16:34578) 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99effff640 1 -- 172.21.15.16:0/95256312 learned_addr learned my addr 172.21.15.16:0/95256312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99f4d11640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de19620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99ef7fe640 1 -- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 msgr2=0x55de9de19620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de19620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.784+0000 7f99ef7fe640 1 -- 172.21.15.16:0/95256312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 msgr2=0x55de9de1e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99ef7fe640 1 -- 172.21.15.16:0/95256312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de9de19e50 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99effff640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99ef7fe640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de1e5e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f99e400a010 tx=0x7f99e401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1dbaecb7406289 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4035060 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f99e4018040 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e402b020 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f99e4010430 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99f8b4d740 10 client.20523 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55de9de900c0 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.785+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e402b020 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f99e4030520 con 0x55de9de03b40 2024-09-12T03:09:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 10 client.20523 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 10 client.20523 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 20 client.20523 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de90670 0x55de9de92a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 4 client.20523 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de9dd6fd70 con 0x55de9de90670 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99effff640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de90670 0x55de9de92a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99cf7fe640 10 client.20523 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99effff640 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de90670 0x55de9de92a90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70ff52ef514043e2 server_cookie=55918f90b88c1149 in_seq=0 out_seq=0 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99cf7fe640 1 -- 172.21.15.16:0/95256312 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de9dd6fd70 con 0x55de9de90670 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99cf7fe640 10 client.20523 handle_command_reply: tid=0 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 1 client.20523 shutdown 2024-09-12T03:09:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 msgr2=0x55de9de1e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.786+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de1e5e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f99e400a010 tx=0x7f99e401c9b0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 shutdown_connections 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55de9de90670 0x55de9de92a90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55de9de0d370 0x55de9de19620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de9de03b40 0x55de9de1e5e0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 --2- 172.21.15.16:0/95256312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55de9de025d0 0x55de9de1e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 >> 172.21.15.16:0/95256312 conn(0x55de9ddcb310 msgr2=0x55de9de030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99cf7fe640 0 client.20523 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 shutdown_connections 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 1 -- 172.21.15.16:0/95256312 wait complete. 2024-09-12T03:09:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f8b4d740 20 client.20523 trim_cache size 0 max 16384 2024-09-12T03:09:57.794 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 msgr2=0x7f99d80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 0x7f99d80a5040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f99e8005e30 tx=0x7f99e8005c50 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 msgr2=0x7f99f01991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 0x7f99f01991b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000dd20 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 shutdown_connections 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99f0105ba0 0x7f99f01a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f99d80a2b80 0x7f99d80a5040 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99f0102110 0x7f99f01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 --2- 172.21.15.16:0/1506272990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f99f0101790 0x7f99f01991b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.787+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 >> 172.21.15.16:0/1506272990 conn(0x7f99f0077210 msgr2=0x7f99f00ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.788+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 shutdown_connections 2024-09-12T03:09:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:57.788+0000 7f99f6774640 1 -- 172.21.15.16:0/1506272990 wait complete. 2024-09-12T03:09:58.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.35 MB/sec execute 419 sec latency 9166.974 ms 2024-09-12T03:09:58.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:09:58 smithi016 ceph-mon[30925]: pgmap v998: 97 pgs: 97 active+clean; 859 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.2 MiB/s wr, 499 op/s 2024-09-12T03:09:58.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:09:58 smithi033 ceph-mon[35092]: pgmap v998: 97 pgs: 97 active+clean; 859 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.2 MiB/s wr, 499 op/s 2024-09-12T03:09:58.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:09:58.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:09:58 smithi184 ceph-mon[35321]: pgmap v998: 97 pgs: 97 active+clean; 859 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.2 MiB/s wr, 499 op/s 2024-09-12T03:09:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:09:59.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.29 MB/sec execute 420 sec latency 10167.023 ms 2024-09-12T03:09:59.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- 172.21.15.16:0/2241230027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 msgr2=0x7f762810d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 --2- 172.21.15.16:0/2241230027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f762810d970 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f7620009ca0 tx=0x7f762002a8c0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- 172.21.15.16:0/2241230027 shutdown_connections 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 --2- 172.21.15.16:0/2241230027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f762810d970 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 --2- 172.21.15.16:0/2241230027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76281011c0 0x7f7628101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 --2- 172.21.15.16:0/2241230027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7628105ad0 0x7f7628105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- 172.21.15.16:0/2241230027 >> 172.21.15.16:0/2241230027 conn(0x7f7628077210 msgr2=0x7f76280fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- 172.21.15.16:0/2241230027 shutdown_connections 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- 172.21.15.16:0/2241230027 wait complete. 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 Processor -- start 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.299+0000 7f762de56640 1 -- start start 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76281011c0 0x7f76281983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7628105ad0 0x7f762819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7628110ef0 con 0x7f7628101b60 2024-09-12T03:09:59.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7628110d70 con 0x7f76281011c0 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f762de56640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7628111070 con 0x7f7628105ad0 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34992/0 (socket says 172.21.15.16:34992) 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 -- 172.21.15.16:0/4076960924 learned_addr learned my addr 172.21.15.16:0/4076960924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7627fff640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7628105ad0 0x7f762819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f76277fe640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76281011c0 0x7f76281983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 -- 172.21.15.16:0/4076960924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7628105ad0 msgr2=0x7f762819ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7628105ad0 0x7f762819ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 -- 172.21.15.16:0/4076960924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76281011c0 msgr2=0x7f76281983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76281011c0 0x7f76281983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 -- 172.21.15.16:0/4076960924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762819d390 con 0x7f7628101b60 2024-09-12T03:09:59.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.300+0000 7f7626ffd640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f761c00d770 tx=0x7f761c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.301+0000 7f7624ff9640 1 -- 172.21.15.16:0/4076960924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c017070 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.301+0000 7f7624ff9640 1 -- 172.21.15.16:0/4076960924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f761c009820 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.301+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762819d680 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.301+0000 7f7624ff9640 1 -- 172.21.15.16:0/4076960924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c004cd0 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.301+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76281a4f60 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f7624ff9640 1 -- 172.21.15.16:0/4076960924 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f761c0099d0 con 0x7f7628101b60 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f763022f740 1 Processor -- start 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f763022f740 1 -- start start 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f763022f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f860b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f763022f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f860ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.303+0000 7f763022f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f763022f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5568f85712f0 con 0x5568f861a780 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f763022f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568f8518cf0 con 0x5568f860cb40 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f763022f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5568f856d2b0 con 0x5568f860b5d0 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7624ff9640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 0x7f76100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34994/0 (socket says 172.21.15.16:34994) 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 -- 172.21.15.16:0/246876795 learned_addr learned my addr 172.21.15.16:0/246876795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7627fff640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f860b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f76277fe640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f860ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7627fff640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 0x7f76100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7624ff9640 1 -- 172.21.15.16:0/4076960924 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f761c0cea20 con 0x7f7628101b60 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 -- 172.21.15.16:0/246876795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 msgr2=0x5568f860ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f860ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 -- 172.21.15.16:0/246876795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 msgr2=0x5568f860b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f860b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 -- 172.21.15.16:0/246876795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f84fd3a0 con 0x5568f861a780 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7627fff640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f860b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7626ffd640 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f761c0cc4d0 tx=0x7f761c096dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff3d86224397400 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f760dffb640 1 -- 172.21.15.16:0/246876795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c020070 con 0x5568f861a780 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.304+0000 7f7627fff640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 0x7f76100a5040 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7620005e30 tx=0x7f7620009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f760dffb640 1 -- 172.21.15.16:0/246876795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f761c09b930 con 0x5568f861a780 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f760dffb640 1 -- 172.21.15.16:0/246876795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c0bb0f0 con 0x5568f861a780 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 -- 172.21.15.16:0/246876795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 msgr2=0x5568f861cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f761c0cc4d0 tx=0x7f761c096dd0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 -- 172.21.15.16:0/246876795 shutdown_connections 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f861cb70 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f860ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 --2- 172.21.15.16:0/246876795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f860b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 -- 172.21.15.16:0/246876795 >> 172.21.15.16:0/246876795 conn(0x5568f85d4310 msgr2=0x5568f8607c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 -- 172.21.15.16:0/246876795 shutdown_connections 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.305+0000 7f763022f740 1 -- 172.21.15.16:0/246876795 wait complete. 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.306+0000 7f763022f740 1 -- start start 2024-09-12T03:09:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.306+0000 7f763022f740 1 Processor -- start 2024-09-12T03:09:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 0x5568f8617bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5568f861d420 con 0x5568f860cb40 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5568f852c060 con 0x5568f861a780 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5568f861d5a0 con 0x5568f860b5d0 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f763022f740 10 client.0 _ll_register_callbacks cb 0x5568f827f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f76277fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 0x5568f8617bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7626ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34996/0 (socket says 172.21.15.16:34996) 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f76277fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:58758/0 (socket says 172.21.15.16:58758) 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7626ffd640 1 -- 172.21.15.16:0/4116282970 learned_addr learned my addr 172.21.15.16:0/4116282970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 -- 172.21.15.16:0/4116282970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 msgr2=0x5568f861a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 -- 172.21.15.16:0/4116282970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 msgr2=0x5568f8617460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7627fff640 1 -- 172.21.15.16:0/4116282970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f82e23c0 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f7626ffd640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.307+0000 7f76277fe640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7627fff640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 0x5568f8617bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7620006010 tx=0x7f76200095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b8ed42b2de3754 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620004240 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f76200043e0 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f761c09b930 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f76200045e0 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f763022f740 10 client.29729 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5568f8628f00 con 0x5568f861a780 2024-09-12T03:09:59.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f761c09b930 con 0x5568f861a780 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7620047070 con 0x5568f861a780 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f763022f740 10 client.29729 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.308+0000 7f763022f740 10 client.29729 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f763022f740 20 client.29729 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f862a2f0 0x5568f862c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f763022f740 4 client.29729 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f857e860 con 0x5568f862a2f0 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f76277fe640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f862a2f0 0x5568f862c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f76277fe640 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f862a2f0 0x5568f862c700 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=801542cc675601e4 server_cookie=12b2e7bcf44d0877 in_seq=0 out_seq=0 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.309+0000 7f7606ffd640 10 client.29729 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f7606ffd640 1 -- 172.21.15.16:0/4116282970 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568f857e860 con 0x5568f862a2f0 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f7606ffd640 10 client.29729 handle_command_reply: tid=0 2024-09-12T03:09:59.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 client.29729 shutdown 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 msgr2=0x5568f8617bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 0x5568f8617bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7620006010 tx=0x7f76200095c0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 shutdown_connections 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f862a2f0 0x5568f862c700 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f861a780 0x5568f8617bf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860cb40 0x5568f8617460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f7606ffd640 0 client.29729 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 --2- 172.21.15.16:0/4116282970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f861a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 >> 172.21.15.16:0/4116282970 conn(0x5568f85d4310 msgr2=0x5568f860d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 shutdown_connections 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 1 -- 172.21.15.16:0/4116282970 wait complete. 2024-09-12T03:09:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.310+0000 7f763022f740 20 client.29729 trim_cache size 0 max 16384 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 Processor -- start 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 -- start start 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f86193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f8632010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76281ab3c0 con 0x5568f861a780 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76281acf00 con 0x5568f860cb40 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.323+0000 7f763022f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76281aba90 con 0x5568f860b5d0 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35000/0 (socket says 172.21.15.16:35000) 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 -- 172.21.15.16:0/1606179730 learned_addr learned my addr 172.21.15.16:0/1606179730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f76277fe640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f86193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f8632010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 -- 172.21.15.16:0/1606179730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 msgr2=0x5568f86193d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f86193d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 -- 172.21.15.16:0/1606179730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 msgr2=0x5568f8632010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f8632010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 -- 172.21.15.16:0/1606179730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f862cd80 con 0x5568f861a780 2024-09-12T03:09:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f8632010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:09:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f7627fff640 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f7620009c80 tx=0x7f7620005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b113831f2bbf86c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f760d7fa640 1 -- 172.21.15.16:0/1606179730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620062070 con 0x5568f861a780 2024-09-12T03:09:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f760d7fa640 1 -- 172.21.15.16:0/1606179730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7620059020 con 0x5568f861a780 2024-09-12T03:09:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.324+0000 7f760d7fa640 1 -- 172.21.15.16:0/1606179730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200305c0 con 0x5568f861a780 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- 172.21.15.16:0/1606179730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 msgr2=0x5568f8632550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f7620009c80 tx=0x7f7620005b50 comp rx=0 tx=0).stop 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- 172.21.15.16:0/1606179730 shutdown_connections 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f861a780 0x5568f8632550 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f8632010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 --2- 172.21.15.16:0/1606179730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f860b5d0 0x5568f86193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- 172.21.15.16:0/1606179730 >> 172.21.15.16:0/1606179730 conn(0x5568f85d4310 msgr2=0x5568f8618cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- 172.21.15.16:0/1606179730 shutdown_connections 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- 172.21.15.16:0/1606179730 wait complete. 2024-09-12T03:09:59.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.325+0000 7f763022f740 1 -- start start 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 Processor -- start 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f861f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f861a780 0x5568f8693fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76281ac830 con 0x5568f860b5d0 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76281ac160 con 0x5568f860cb40 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76281a93a0 con 0x5568f861a780 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35014/0 (socket says 172.21.15.16:35014) 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 -- 172.21.15.16:0/1176853889 learned_addr learned my addr 172.21.15.16:0/1176853889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f763022f740 10 client.0 _ll_register_callbacks cb 0x7f76200099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f7627fff640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f861a780 0x5568f8693fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f861f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 -- 172.21.15.16:0/1176853889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f861a780 msgr2=0x5568f8693fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f861a780 0x5568f8693fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 -- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 msgr2=0x5568f861f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f861f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f76277fe640 1 -- 172.21.15.16:0/1176853889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f8694520 con 0x5568f860b5d0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.326+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f861f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f76277fe640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x5568f862cf80 tx=0x7f7618006410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ccb55519f99ec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618005e50 con 0x5568f860b5d0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f76180146a0 con 0x5568f860b5d0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7620059020 con 0x5568f860b5d0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f76180148a0 con 0x5568f860b5d0 2024-09-12T03:09:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 10 client.20541 fetch_fsmap learned FSMap version 39 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5568f86956e0 con 0x5568f860b5d0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7620059020 con 0x5568f860b5d0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7618015cc0 con 0x5568f860b5d0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 10 client.20541 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 10 client.20541 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.327+0000 7f763022f740 20 client.20541 populate_metadata read hostname 'smithi016' 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f8695db0 0x5568f86981a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 4 client.20541 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568f8578d70 con 0x5568f8695db0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f8695db0 0x5568f86981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f7606ffd640 10 client.20541 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f7626ffd640 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f8695db0 0x5568f86981a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18818ecee1878fa0 server_cookie=1dbdf3224c534fa2 in_seq=0 out_seq=0 2024-09-12T03:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f7606ffd640 1 -- 172.21.15.16:0/1176853889 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568f8578d70 con 0x5568f8695db0 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f7606ffd640 10 client.20541 handle_command_reply: tid=0 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 1 client.20541 shutdown 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 msgr2=0x5568f861ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.328+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x5568f862cf80 tx=0x7f7618006410 comp rx=0 tx=0).stop 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 shutdown_connections 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5568f8695db0 0x5568f86981a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f7606ffd640 0 client.20541 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5568f861a780 0x5568f8693fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5568f860cb40 0x5568f861f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 --2- 172.21.15.16:0/1176853889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568f860b5d0 0x5568f861ebe0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 >> 172.21.15.16:0/1176853889 conn(0x5568f85d4310 msgr2=0x5568f86194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 shutdown_connections 2024-09-12T03:09:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 1 -- 172.21.15.16:0/1176853889 wait complete. 2024-09-12T03:09:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f763022f740 20 client.20541 trim_cache size 0 max 16384 2024-09-12T03:09:59.335 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 msgr2=0x7f76100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 0x7f76100a5040 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7620005e30 tx=0x7f7620009360 comp rx=0 tx=0).stop 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 msgr2=0x7f7628198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f761c00d770 tx=0x7f761c00db70 comp rx=0 tx=0).stop 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 shutdown_connections 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7628105ad0 0x7f762819ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f76100a2b80 0x7f76100a5040 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628101b60 0x7f7628198920 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 --2- 172.21.15.16:0/4076960924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76281011c0 0x7f76281983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:09:59.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 >> 172.21.15.16:0/4076960924 conn(0x7f7628077210 msgr2=0x7f76281002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:09:59.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.329+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 shutdown_connections 2024-09-12T03:09:59.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:09:59.330+0000 7f762de56640 1 -- 172.21.15.16:0/4076960924 wait complete. 2024-09-12T03:10:00.031 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.23 MB/sec execute 421 sec latency 11167.130 ms 2024-09-12T03:10:00.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:10:00.346 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:00.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:00 smithi016 ceph-mon[30925]: pgmap v999: 97 pgs: 97 active+clean; 860 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-09-12T03:10:00.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:00 smithi016 ceph-mon[30925]: overall HEALTH_OK 2024-09-12T03:10:00.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:00 smithi033 ceph-mon[35092]: pgmap v999: 97 pgs: 97 active+clean; 860 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-09-12T03:10:00.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:00 smithi033 ceph-mon[35092]: overall HEALTH_OK 2024-09-12T03:10:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:00 smithi184 ceph-mon[35321]: pgmap v999: 97 pgs: 97 active+clean; 860 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-09-12T03:10:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:00 smithi184 ceph-mon[35321]: overall HEALTH_OK 2024-09-12T03:10:00.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 -- 172.21.15.16:0/470627247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 msgr2=0x7fced410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/470627247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced410ea90 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009ca0 tx=0x7fcecc02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 -- 172.21.15.16:0/470627247 shutdown_connections 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/470627247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced410ea90 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/470627247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4102790 0x7fced41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/470627247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4101de0 0x7fced41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 -- 172.21.15.16:0/470627247 >> 172.21.15.16:0/470627247 conn(0x7fced40fd8c0 msgr2=0x7fced40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 -- 172.21.15.16:0/470627247 shutdown_connections 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 -- 172.21.15.16:0/470627247 wait complete. 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.800+0000 7fcedc1ae640 1 Processor -- start 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 -- start start 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4101de0 0x7fced41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4102790 0x7fced4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fced4111fc0 con 0x7fced4106f10 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fced4111e40 con 0x7fced4101de0 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fcedc1ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fced4112140 con 0x7fced4102790 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fced9f4a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4101de0 0x7fced41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35042/0 (socket says 172.21.15.16:35042) 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 -- 172.21.15.16:0/1675740115 learned_addr learned my addr 172.21.15.16:0/1675740115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fced9749640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4102790 0x7fced4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 -- 172.21.15.16:0/1675740115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4102790 msgr2=0x7fced4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4102790 0x7fced4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 -- 172.21.15.16:0/1675740115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4101de0 msgr2=0x7fced41983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4101de0 0x7fced41983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.801+0000 7fceda74b640 1 -- 172.21.15.16:0/1675740115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced419d380 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fceda74b640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fcecc003f10 tx=0x7fcecc003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fcecaffd640 1 -- 172.21.15.16:0/1675740115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc031710 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fcecaffd640 1 -- 172.21.15.16:0/1675740115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcecc0318b0 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced419d610 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fcecaffd640 1 -- 172.21.15.16:0/1675740115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc02fb50 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.802+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced41a4f50 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.804+0000 7fcecaffd640 1 -- 172.21.15.16:0/1675740115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcecc02fd30 con 0x7fced4106f10 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.804+0000 7fcede587740 1 Processor -- start 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 -- start start 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c45760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bf8b9e2f0 con 0x558bf8c43370 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558bf8b45cf0 con 0x558bf8c39b40 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcede587740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558bf8b9a2b0 con 0x558bf8c385d0 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcecaffd640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 0x7fcec00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c45760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34646/0 (socket says 172.21.15.16:34646) 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 -- 172.21.15.16:0/368288522 learned_addr learned my addr 172.21.15.16:0/368288522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9f4a640 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fceda74b640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 0x7fcec00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fcecaffd640 1 -- 172.21.15.16:0/1675740115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcecc0f0580 con 0x7fced4106f10 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 -- 172.21.15.16:0/368288522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 msgr2=0x558bf8c38a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c38a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 -- 172.21.15.16:0/368288522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 msgr2=0x558bf8c45760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c45760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.805+0000 7fced9749640 1 -- 172.21.15.16:0/368288522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf8b2a3a0 con 0x558bf8c39b40 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.806+0000 7fced9749640 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcec400bfd0 tx=0x7fcec400b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ead717fb73766439 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.806+0000 7fceda74b640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 0x7fcec00a5040 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006010 tx=0x7fcecc087a10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.806+0000 7fcebffff640 1 -- 172.21.15.16:0/368288522 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4004020 con 0x558bf8c39b40 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.806+0000 7fcebffff640 1 -- 172.21.15.16:0/368288522 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec400fc10 con 0x558bf8c39b40 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.806+0000 7fcebffff640 1 -- 172.21.15.16:0/368288522 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec40038a0 con 0x558bf8c39b40 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- 172.21.15.16:0/368288522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 msgr2=0x558bf8c38090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcec400bfd0 tx=0x7fcec400b410 comp rx=0 tx=0).stop 2024-09-12T03:10:00.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- 172.21.15.16:0/368288522 shutdown_connections 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c45760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c38a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 --2- 172.21.15.16:0/368288522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c38090 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- 172.21.15.16:0/368288522 >> 172.21.15.16:0/368288522 conn(0x558bf8c01310 msgr2=0x558bf8c34c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- 172.21.15.16:0/368288522 shutdown_connections 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- 172.21.15.16:0/368288522 wait complete. 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 -- start start 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.807+0000 7fcede587740 1 Processor -- start 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c42da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c39b40 0x558bf8c40050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bf8c46070 con 0x558bf8c43370 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558bf8b59060 con 0x558bf8c385d0 2024-09-12T03:10:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558bf8c461f0 con 0x558bf8c39b40 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcede587740 10 client.0 _ll_register_callbacks cb 0x558bf88ac8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35064/0 (socket says 172.21.15.16:35064) 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 -- 172.21.15.16:0/1750912012 learned_addr learned my addr 172.21.15.16:0/1750912012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9f4a640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c39b40 0x558bf8c40050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c42da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 -- 172.21.15.16:0/1750912012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c39b40 msgr2=0x558bf8c40050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c39b40 0x558bf8c40050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 -- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 msgr2=0x558bf8c42da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c42da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 -- 172.21.15.16:0/1750912012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf890f3c0 con 0x558bf8c43370 2024-09-12T03:10:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fced9749640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fcec4005890 tx=0x7fcec4017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ed3b7ef4073ee1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcebd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.808+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4012020 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcec40121c0 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec400fc10 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcec400f730 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 10 client.20553 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bf8c51b30 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec400fc10 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcec4004840 con 0x558bf8c43370 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 10 client.20553 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 10 client.20553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 20 client.20553 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8c52ee0 0x558bf8c552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 4 client.20553 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.809+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bf8bab860 con 0x558bf8c52ee0 2024-09-12T03:10:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.810+0000 7fced9f4a640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8c52ee0 0x558bf8c552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.810+0000 7fced9f4a640 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8c52ee0 0x558bf8c552d0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdf380762322e7da server_cookie=3d5ffe64dcedfa73 in_seq=0 out_seq=0 2024-09-12T03:10:00.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.810+0000 7fcebd7fa640 10 client.20553 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcebd7fa640 1 -- 172.21.15.16:0/1750912012 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558bf8bab860 con 0x558bf8c52ee0 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcebd7fa640 10 client.20553 handle_command_reply: tid=0 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 client.20553 shutdown 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 msgr2=0x558bf8c407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fcec4005890 tx=0x7fcec4017040 comp rx=0 tx=0).stop 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 shutdown_connections 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8c52ee0 0x558bf8c552d0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c407e0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcebd7fa640 0 client.20553 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c39b40 0x558bf8c40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 --2- 172.21.15.16:0/1750912012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c42da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 >> 172.21.15.16:0/1750912012 conn(0x558bf8c01310 msgr2=0x558bf8c3a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 shutdown_connections 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 1 -- 172.21.15.16:0/1750912012 wait complete. 2024-09-12T03:10:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.811+0000 7fcede587740 20 client.20553 trim_cache size 0 max 16384 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 Processor -- start 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 -- start start 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c39300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c3b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ad520 con 0x558bf8c43370 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fced41af060 con 0x558bf8c39b40 2024-09-12T03:10:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fcede587740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fced41adbf0 con 0x558bf8c385d0 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fced9749640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fced9749640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35078/0 (socket says 172.21.15.16:35078) 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fced9749640 1 -- 172.21.15.16:0/1445635812 learned_addr learned my addr 172.21.15.16:0/1445635812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fced9f4a640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c3b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.824+0000 7fced9749640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c39300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 -- 172.21.15.16:0/1445635812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 msgr2=0x558bf8c39300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c39300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 -- 172.21.15.16:0/1445635812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 msgr2=0x558bf8c3b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c3b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 -- 172.21.15.16:0/1445635812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf8c55730 con 0x558bf8c43370 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9f4a640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c3b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fced9749640 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fcec4004540 tx=0x7fcec400f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7130459eeb96394f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fcebf7fe640 1 -- 172.21.15.16:0/1445635812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4011430 con 0x558bf8c43370 2024-09-12T03:10:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fcebf7fe640 1 -- 172.21.15.16:0/1445635812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec400ba20 con 0x558bf8c43370 2024-09-12T03:10:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.825+0000 7fcebf7fe640 1 -- 172.21.15.16:0/1445635812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4004f70 con 0x558bf8c43370 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- 172.21.15.16:0/1445635812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 msgr2=0x558bf8c3b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fcec4004540 tx=0x7fcec400f8d0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- 172.21.15.16:0/1445635812 shutdown_connections 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c43370 0x558bf8c3b660 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c39b40 0x558bf8c3b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 --2- 172.21.15.16:0/1445635812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c385d0 0x558bf8c39300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- 172.21.15.16:0/1445635812 >> 172.21.15.16:0/1445635812 conn(0x558bf8c01310 msgr2=0x558bf8c2fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- 172.21.15.16:0/1445635812 shutdown_connections 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- 172.21.15.16:0/1445635812 wait complete. 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 -- start start 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.826+0000 7fcede587740 1 Processor -- start 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c3d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 0x558bf8c3dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ae990 con 0x558bf8c39b40 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ae2c0 con 0x558bf8c385d0 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ab500 con 0x558bf8c43370 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcede587740 10 client.0 _ll_register_callbacks cb 0x7fcec400be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35094/0 (socket says 172.21.15.16:35094) 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 -- 172.21.15.16:0/4180904273 learned_addr learned my addr 172.21.15.16:0/4180904273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9749640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c3d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9749640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 0x558bf8c3dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 -- 172.21.15.16:0/4180904273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 msgr2=0x558bf8c3dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 0x558bf8c3dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 -- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 msgr2=0x558bf8c3d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c3d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 -- 172.21.15.16:0/4180904273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf8cbc9c0 con 0x558bf8c39b40 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9749640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 0x558bf8c3dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fced9f4a640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x558bf8c41690 tx=0x7fced000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4605506b75b18fff server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcebd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.827+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced002b060 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fced0011640 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec400ba20 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fced0011840 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 10 client.20559 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558bf8cbdd10 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec400ba20 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fced001dbb0 con 0x558bf8c39b40 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 10 client.20559 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 10 client.20559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 20 client.20559 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8cbe4c0 0x558bf8cc08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 4 client.20559 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.828+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558bf8ba5d70 con 0x558bf8cbe4c0 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fced9749640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8cbe4c0 0x558bf8cc08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcebd7fa640 10 client.20559 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fced9749640 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8cbe4c0 0x558bf8cc08e0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8caca7f703de4e3d server_cookie=4e953d5f158cb1aa in_seq=0 out_seq=0 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcebd7fa640 1 -- 172.21.15.16:0/4180904273 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558bf8ba5d70 con 0x558bf8cbe4c0 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcebd7fa640 10 client.20559 handle_command_reply: tid=0 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 client.20559 shutdown 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 msgr2=0x558bf8c3d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x558bf8c41690 tx=0x7fced000bf70 comp rx=0 tx=0).stop 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 shutdown_connections 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558bf8cbe4c0 0x558bf8cc08e0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558bf8c43370 0x558bf8c3dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcebd7fa640 0 client.20559 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf8c39b40 0x558bf8c3d5a0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 --2- 172.21.15.16:0/4180904273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558bf8c385d0 0x558bf8c3d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 >> 172.21.15.16:0/4180904273 conn(0x558bf8c01310 msgr2=0x558bf8c418b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.829+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 shutdown_connections 2024-09-12T03:10:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcede587740 1 -- 172.21.15.16:0/4180904273 wait complete. 2024-09-12T03:10:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcede587740 20 client.20559 trim_cache size 0 max 16384 2024-09-12T03:10:00.835 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 msgr2=0x7fcec00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 0x7fcec00a5040 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006010 tx=0x7fcecc087a10 comp rx=0 tx=0).stop 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 msgr2=0x7fced419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fcecc003f10 tx=0x7fcecc003f40 comp rx=0 tx=0).stop 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 shutdown_connections 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcec00a2b80 0x7fcec00a5040 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4106f10 0x7fced419cca0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fced4102790 0x7fced4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 --2- 172.21.15.16:0/1675740115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fced4101de0 0x7fced41983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 >> 172.21.15.16:0/1675740115 conn(0x7fced40fd8c0 msgr2=0x7fced40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 shutdown_connections 2024-09-12T03:10:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:00.830+0000 7fcedc1ae640 1 -- 172.21.15.16:0/1675740115 wait complete. 2024-09-12T03:10:01.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.18 MB/sec execute 422 sec latency 12167.228 ms 2024-09-12T03:10:02.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.12 MB/sec execute 423 sec latency 13167.306 ms 2024-09-12T03:10:02.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:02 smithi016 ceph-mon[30925]: pgmap v1000: 97 pgs: 97 active+clean; 857 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 398 op/s 2024-09-12T03:10:02.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:02 smithi033 ceph-mon[35092]: pgmap v1000: 97 pgs: 97 active+clean; 857 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 398 op/s 2024-09-12T03:10:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:02 smithi184 ceph-mon[35321]: pgmap v1000: 97 pgs: 97 active+clean; 857 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 398 op/s 2024-09-12T03:10:02.600 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:10:02.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:10:02.979 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:03.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.06 MB/sec execute 424 sec latency 14167.401 ms 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 -- 172.21.15.16:0/492744499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 msgr2=0x7f28d4108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 --2- 172.21.15.16:0/492744499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d4108a10 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009900 tx=0x7f28c802a410 comp rx=0 tx=0).stop 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 -- 172.21.15.16:0/492744499 shutdown_connections 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 --2- 172.21.15.16:0/492744499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4109140 0x7f28d4110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 --2- 172.21.15.16:0/492744499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d4108a10 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 --2- 172.21.15.16:0/492744499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4104010 0x7f28d41043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 -- 172.21.15.16:0/492744499 >> 172.21.15.16:0/492744499 conn(0x7f28d40ffa50 msgr2=0x7f28d4101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 -- 172.21.15.16:0/492744499 shutdown_connections 2024-09-12T03:10:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.445+0000 7f28da696640 1 -- 172.21.15.16:0/492744499 wait complete. 2024-09-12T03:10:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 Processor -- start 2024-09-12T03:10:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 -- start start 2024-09-12T03:10:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4104010 0x7f28d419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4109140 0x7f28d419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4114140 con 0x7f28d41049c0 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4113fc0 con 0x7f28d4109140 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.446+0000 7f28da696640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41142c0 con 0x7f28d4104010 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35110/0 (socket says 172.21.15.16:35110) 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 -- 172.21.15.16:0/3342127287 learned_addr learned my addr 172.21.15.16:0/3342127287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d3fff640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4104010 0x7f28d419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d8c33640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4109140 0x7f28d419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 -- 172.21.15.16:0/3342127287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4104010 msgr2=0x7f28d419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4104010 0x7f28d419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 -- 172.21.15.16:0/3342127287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4109140 msgr2=0x7f28d419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4109140 0x7f28d419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 -- 172.21.15.16:0/3342127287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d419f5d0 con 0x7f28d41049c0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f28c8006010 tx=0x7f28c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d17fa640 1 -- 172.21.15.16:0/3342127287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c803f030 con 0x7f28d41049c0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d17fa640 1 -- 172.21.15.16:0/3342127287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c8003ce0 con 0x7f28d41049c0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28d17fa640 1 -- 172.21.15.16:0/3342127287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8035070 con 0x7f28d41049c0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d419f860 con 0x7f28d41049c0 2024-09-12T03:10:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.447+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d41a71a0 con 0x7f28d41049c0 2024-09-12T03:10:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.450+0000 7f28d17fa640 1 -- 172.21.15.16:0/3342127287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f28c8042030 con 0x7f28d41049c0 2024-09-12T03:10:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.450+0000 7f28dca6f740 1 Processor -- start 2024-09-12T03:10:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 -- start start 2024-09-12T03:10:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 0x564766498f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 0x5647664a6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647663fe280 con 0x5647664a3d40 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647663a5b10 con 0x564766498b60 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28dca6f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647663fa240 con 0x564766499480 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 0x5647664a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 0x5647664a6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35120/0 (socket says 172.21.15.16:35120) 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d8c33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 0x564766498f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d37fe640 1 -- 172.21.15.16:0/4109519053 learned_addr learned my addr 172.21.15.16:0/4109519053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d3fff640 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d17fa640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 0x7f28bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d8c33640 1 -- 172.21.15.16:0/4109519053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 msgr2=0x5647664a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d8c33640 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d8c33640 1 -- 172.21.15.16:0/4109519053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 msgr2=0x5647664a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.451+0000 7f28d8c33640 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 0x5647664a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d8c33640 1 -- 172.21.15.16:0/4109519053 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647663b21f0 con 0x564766498b60 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d3fff640 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d17fa640 1 -- 172.21.15.16:0/3342127287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f28c80ef6c0 con 0x7f28d41049c0 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d8c33640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 0x7f28bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d8c33640 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 0x564766498f40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f28cc007e50 tx=0x7f28cc00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47f1f9a39eb22428 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28ba7fc640 1 -- 172.21.15.16:0/4109519053 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc022070 con 0x564766498b60 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28ba7fc640 1 -- 172.21.15.16:0/4109519053 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28cc01c470 con 0x564766498b60 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28d8c33640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 0x7f28bc0a5040 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f28cc01f000 tx=0x7f28cc020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.452+0000 7f28ba7fc640 1 -- 172.21.15.16:0/4109519053 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc00ee30 con 0x564766498b60 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- 172.21.15.16:0/4109519053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 msgr2=0x564766498f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 0x564766498f40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f28cc007e50 tx=0x7f28cc00e720 comp rx=0 tx=0).stop 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- 172.21.15.16:0/4109519053 shutdown_connections 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647664a3d40 0x5647664a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 --2- 172.21.15.16:0/4109519053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766498b60 0x564766498f40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- 172.21.15.16:0/4109519053 >> 172.21.15.16:0/4109519053 conn(0x5647664950f0 msgr2=0x5647664954d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- 172.21.15.16:0/4109519053 shutdown_connections 2024-09-12T03:10:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- 172.21.15.16:0/4109519053 wait complete. 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 -- start start 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.453+0000 7f28dca6f740 1 Processor -- start 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 0x5647664a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x5647664a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647664a6910 con 0x564766499480 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5647663b8f10 con 0x5647664a3d40 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5647664a6a90 con 0x564766498b60 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28dca6f740 10 client.0 _ll_register_callbacks cb 0x564766381a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35132/0 (socket says 172.21.15.16:35132) 2024-09-12T03:10:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 -- 172.21.15.16:0/3955400450 learned_addr learned my addr 172.21.15.16:0/3955400450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d3fff640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 0x5647664a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d8c33640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x5647664a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 -- 172.21.15.16:0/3955400450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 msgr2=0x5647664a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 0x5647664a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 -- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 msgr2=0x5647664a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x5647664a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 -- 172.21.15.16:0/3955400450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56476616f6a0 con 0x564766499480 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d3fff640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 0x5647664a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d8c33640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x5647664a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28d37fe640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f28c8005ab0 tx=0x7f28c80057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33755e2cef1ef817 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.454+0000 7f28b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8049070 con 0x564766499480 2024-09-12T03:10:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f28c80bc250 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28cc01c470 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f28c80bbdc0 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 10 client.20571 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5647664b2550 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28cc01c470 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f28c80c47d0 con 0x564766499480 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 10 client.20571 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 10 client.20571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 20 client.20571 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647664b3940 0x5647664b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 4 client.20571 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.455+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56476640b7f0 con 0x5647664b3940 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.456+0000 7f28d3fff640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647664b3940 0x5647664b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.456+0000 7f28d3fff640 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647664b3940 0x5647664b5d90 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15f3bff413afb0b3 server_cookie=c41850a2204d8e76 in_seq=0 out_seq=0 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.456+0000 7f28b37fe640 10 client.20571 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28b37fe640 1 -- 172.21.15.16:0/3955400450 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56476640b7f0 con 0x5647664b3940 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28b37fe640 10 client.20571 handle_command_reply: tid=0 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 client.20571 shutdown 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 msgr2=0x5647664a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f28c8005ab0 tx=0x7f28c80057d0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 shutdown_connections 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5647664b3940 0x5647664b5d90 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x5647664a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766499480 0x5647664a11a0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 --2- 172.21.15.16:0/3955400450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766498b60 0x5647664a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 >> 172.21.15.16:0/3955400450 conn(0x5647664950f0 msgr2=0x56476649add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28b37fe640 0 client.20571 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 shutdown_connections 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 1 -- 172.21.15.16:0/3955400450 wait complete. 2024-09-12T03:10:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.457+0000 7f28dca6f740 20 client.20571 trim_cache size 0 max 16384 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.474+0000 7f28dca6f740 1 Processor -- start 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 -- start start 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 0x5647664bb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647664a3d40 0x5647664bbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41af6c0 con 0x564766498b60 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b1200 con 0x564766499480 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28dca6f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41afd90 con 0x5647664a3d40 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35142/0 (socket says 172.21.15.16:35142) 2024-09-12T03:10:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 -- 172.21.15.16:0/2980249548 learned_addr learned my addr 172.21.15.16:0/2980249548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d8c33640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647664a3d40 0x5647664bbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d37fe640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 0x5647664bb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 -- 172.21.15.16:0/2980249548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647664a3d40 msgr2=0x5647664bbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647664a3d40 0x5647664bbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 -- 172.21.15.16:0/2980249548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 msgr2=0x5647664bb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 0x5647664bb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 -- 172.21.15.16:0/2980249548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647664992b0 con 0x564766498b60 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d37fe640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 0x5647664bb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.475+0000 7f28d3fff640 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x5647664b65e0 tx=0x7f28c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6daaee822e40dc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28b9ffb640 1 -- 172.21.15.16:0/2980249548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4003e20 con 0x564766498b60 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28b9ffb640 1 -- 172.21.15.16:0/2980249548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4012930 con 0x564766498b60 2024-09-12T03:10:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28b9ffb640 1 -- 172.21.15.16:0/2980249548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401fa50 con 0x564766498b60 2024-09-12T03:10:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 -- 172.21.15.16:0/2980249548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 msgr2=0x5647664a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x5647664b65e0 tx=0x7f28c4005f90 comp rx=0 tx=0).stop 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 -- 172.21.15.16:0/2980249548 shutdown_connections 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5647664a3d40 0x5647664bbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564766499480 0x5647664bb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 --2- 172.21.15.16:0/2980249548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a2ac0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.476+0000 7f28dca6f740 1 -- 172.21.15.16:0/2980249548 >> 172.21.15.16:0/2980249548 conn(0x5647664950f0 msgr2=0x5647664a23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- 172.21.15.16:0/2980249548 shutdown_connections 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- 172.21.15.16:0/2980249548 wait complete. 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- start start 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 Processor -- start 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x56476651d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b0b30 con 0x564766498b60 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b0460 con 0x5647664a3d40 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.477+0000 7f28dca6f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41ad6a0 con 0x564766499480 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28dca6f740 10 client.0 _ll_register_callbacks cb 0x7f28c80f5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35148/0 (socket says 172.21.15.16:35148) 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 -- 172.21.15.16:0/820529719 learned_addr learned my addr 172.21.15.16:0/820529719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d37fe640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d8c33640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x56476651d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 -- 172.21.15.16:0/820529719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 msgr2=0x5647664a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 -- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 msgr2=0x56476651d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x56476651d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 -- 172.21.15.16:0/820529719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56476651dbb0 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28d3fff640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f28c4004650 tx=0x7f28c401f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c1079ff8b4d962 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4003c20 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f28c4003dc0 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.478+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c4012930 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f28c4012450 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 10 client.20577 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56476651ee00 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4012930 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f28c401f7d0 con 0x564766498b60 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 10 client.20577 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 10 client.20577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 20 client.20577 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56476651f460 0x564766521880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 4 client.20577 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564766405d00 con 0x56476651f460 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.479+0000 7f28d37fe640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56476651f460 0x564766521880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.480+0000 7f28d37fe640 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56476651f460 0x564766521880 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae3d2df9ac38b689 server_cookie=6a59319a622c1b7e in_seq=0 out_seq=0 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.480+0000 7f28b37fe640 10 client.20577 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28b37fe640 1 -- 172.21.15.16:0/820529719 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564766405d00 con 0x56476651f460 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28b37fe640 10 client.20577 handle_command_reply: tid=0 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 client.20577 shutdown 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 msgr2=0x5647664a8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f28c4004650 tx=0x7f28c401f400 comp rx=0 tx=0).stop 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 shutdown_connections 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56476651f460 0x564766521880 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5647664a3d40 0x56476651d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564766499480 0x5647664a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 --2- 172.21.15.16:0/820529719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564766498b60 0x5647664a8090 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 >> 172.21.15.16:0/820529719 conn(0x5647664950f0 msgr2=0x56476649add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28b37fe640 0 client.20577 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 shutdown_connections 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 1 -- 172.21.15.16:0/820529719 wait complete. 2024-09-12T03:10:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.481+0000 7f28dca6f740 20 client.20577 trim_cache size 0 max 16384 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "966d541e-88aa-43df-b8ef-5544ebb7fd08", 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 msgr2=0x7f28bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 0x7f28bc0a5040 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f28cc01f000 tx=0x7f28cc020000 comp rx=0 tx=0).stop 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 msgr2=0x7f28d419ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f28c8006010 tx=0x7f28c80042f0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 shutdown_connections 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28d4109140 0x7f28d419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f28bc0a2b80 0x7f28bc0a5040 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d41049c0 0x7f28d419ab60 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 --2- 172.21.15.16:0/3342127287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28d4104010 0x7f28d419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 >> 172.21.15.16:0/3342127287 conn(0x7f28d40ffa50 msgr2=0x7f28d4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 shutdown_connections 2024-09-12T03:10:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:03.482+0000 7f28da696640 1 -- 172.21.15.16:0/3342127287 wait complete. 2024-09-12T03:10:04.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 24.01 MB/sec execute 425 sec latency 15167.449 ms 2024-09-12T03:10:04.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: pgmap v1001: 97 pgs: 97 active+clean; 855 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 5.8 MiB/s wr, 235 op/s 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:04 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: pgmap v1001: 97 pgs: 97 active+clean; 855 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 5.8 MiB/s wr, 235 op/s 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.536 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:04.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:04 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:10:04.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: pgmap v1001: 97 pgs: 97 active+clean; 855 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 5.8 MiB/s wr, 235 op/s 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:04.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:04 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:05.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.95 MB/sec execute 426 sec latency 16167.485 ms 2024-09-12T03:10:05.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:05 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:05.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:05 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:05.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:05 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:05.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:05 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 -- 172.21.15.16:0/2101458087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 msgr2=0x7f26741021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 --2- 172.21.15.16:0/2101458087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f26741021c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f2664009900 tx=0x7f266402a410 comp rx=0 tx=0).stop 2024-09-12T03:10:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 -- 172.21.15.16:0/2101458087 shutdown_connections 2024-09-12T03:10:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 --2- 172.21.15.16:0/2101458087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674106f10 0x7f267410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 --2- 172.21.15.16:0/2101458087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674102790 0x7f26741067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 --2- 172.21.15.16:0/2101458087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f26741021c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 -- 172.21.15.16:0/2101458087 >> 172.21.15.16:0/2101458087 conn(0x7f26740fd8c0 msgr2=0x7f26740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 -- 172.21.15.16:0/2101458087 shutdown_connections 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.270+0000 7f267a67d640 1 -- 172.21.15.16:0/2101458087 wait complete. 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 Processor -- start 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 -- start start 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674102790 0x7f26741989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674106f10 0x7f267419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2674111f90 con 0x7f2674101de0 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2674111e10 con 0x7f2674106f10 2024-09-12T03:10:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.271+0000 7f267a67d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2674112110 con 0x7f2674102790 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35152/0 (socket says 172.21.15.16:35152) 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f26737fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674102790 0x7f26741989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 -- 172.21.15.16:0/1326620723 learned_addr learned my addr 172.21.15.16:0/1326620723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674106f10 0x7f267419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 -- 172.21.15.16:0/1326620723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674102790 msgr2=0x7f26741989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674102790 0x7f26741989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 -- 172.21.15.16:0/1326620723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674106f10 msgr2=0x7f267419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674106f10 0x7f267419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 -- 172.21.15.16:0/1326620723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267419d440 con 0x7f2674101de0 2024-09-12T03:10:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f2673fff640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2664006010 tx=0x7f26640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f26717fa640 1 -- 172.21.15.16:0/1326620723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266403f030 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267419d6d0 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f26717fa640 1 -- 172.21.15.16:0/1326620723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2664003ce0 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.272+0000 7f26717fa640 1 -- 172.21.15.16:0/1326620723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664035070 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.273+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26741a5010 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f26717fa640 1 -- 172.21.15.16:0/1326620723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2664042030 con 0x7f2674101de0 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 Processor -- start 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 -- start start 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x55640006b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x556400078a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5563fffd12e0 con 0x55640006bdc0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5563fff78ce0 con 0x556400076680 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f267ca56740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5563fffcd2a0 con 0x55640006b4a0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44174/0 (socket says 172.21.15.16:44174) 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f2673fff640 1 -- 172.21.15.16:0/1664170320 learned_addr learned my addr 172.21.15.16:0/1664170320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.275+0000 7f26717fa640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 0x7f265c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f26737fe640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x556400078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x55640006b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f26717fa640 1 -- 172.21.15.16:0/1326620723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26640ef640 con 0x7f2674101de0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 0x7f265c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 -- 172.21.15.16:0/1664170320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 msgr2=0x55640006b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x55640006b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 -- 172.21.15.16:0/1664170320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 msgr2=0x556400078a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x556400078a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 -- 172.21.15.16:0/1664170320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563fff5d440 con 0x55640006bdc0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x55640006b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2673fff640 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f26640bbfb0 tx=0x7f26640a6ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feecc60ed9cec4f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f265a7fc640 1 -- 172.21.15.16:0/1664170320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664049070 con 0x55640006bdc0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 0x7f265c0a5040 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f266c009900 tx=0x7f266c009650 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f265a7fc640 1 -- 172.21.15.16:0/1664170320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f266403b580 con 0x55640006bdc0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.276+0000 7f265a7fc640 1 -- 172.21.15.16:0/1664170320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640c0320 con 0x55640006bdc0 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- 172.21.15.16:0/1664170320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 msgr2=0x556400076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f26640bbfb0 tx=0x7f26640a6ba0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- 172.21.15.16:0/1664170320 shutdown_connections 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x556400078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400076140 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 --2- 172.21.15.16:0/1664170320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x55640006b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- 172.21.15.16:0/1664170320 >> 172.21.15.16:0/1664170320 conn(0x556400067a30 msgr2=0x556400067e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- 172.21.15.16:0/1664170320 shutdown_connections 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- 172.21.15.16:0/1664170320 wait complete. 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.277+0000 7f267ca56740 1 -- start start 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 Processor -- start 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x556400075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x5564000737e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556400079010 con 0x55640006bdc0 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5563fff8c050 con 0x556400076680 2024-09-12T03:10:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556400079190 con 0x55640006b4a0 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f267ca56740 10 client.0 _ll_register_callbacks cb 0x5563fffcc160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f26737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f2673fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x556400075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f26737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44186/0 (socket says 172.21.15.16:44186) 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f26737fe640 1 -- 172.21.15.16:0/797626779 learned_addr learned my addr 172.21.15.16:0/797626779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.278+0000 7f2678c1a640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x5564000737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 -- 172.21.15.16:0/797626779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 msgr2=0x556400075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x556400075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 -- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 msgr2=0x5564000737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x5564000737e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 -- 172.21.15.16:0/797626779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ffd423c0 con 0x55640006bdc0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f2678c1a640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x5564000737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26737fe640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f266800bad0 tx=0x7f266800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f6fa6a03894efc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2668011440 con 0x55640006bdc0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26680115e0 con 0x55640006bdc0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5563fffde850 con 0x55640006bdc0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26680117e0 con 0x55640006bdc0 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f267ca56740 10 client.20589 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5564000848e0 con 0x55640006bdc0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5563fffde850 con 0x55640006bdc0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.279+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f266801bad0 con 0x55640006bdc0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 10 client.20589 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 10 client.20589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 20 client.20589 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556400085db0 0x5564000881f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 4 client.20589 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563fffd8d60 con 0x556400085db0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f2673fff640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556400085db0 0x5564000881f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f26537fe640 10 client.20589 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.280+0000 7f2673fff640 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556400085db0 0x5564000881f0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63f4a7289643b9fb server_cookie=fa870a65055e834f in_seq=0 out_seq=0 2024-09-12T03:10:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f26537fe640 1 -- 172.21.15.16:0/797626779 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563fffd8d60 con 0x556400085db0 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f26537fe640 10 client.20589 handle_command_reply: tid=0 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 client.20589 shutdown 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 msgr2=0x556400073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f266800bad0 tx=0x7f266800bf90 comp rx=0 tx=0).stop 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 shutdown_connections 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556400085db0 0x5564000881f0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556400076680 0x5564000737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006bdc0 0x556400073280 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 --2- 172.21.15.16:0/797626779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006b4a0 0x556400075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f26537fe640 0 client.20589 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 >> 172.21.15.16:0/797626779 conn(0x556400067a30 msgr2=0x55640006d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 shutdown_connections 2024-09-12T03:10:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 1 -- 172.21.15.16:0/797626779 wait complete. 2024-09-12T03:10:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.281+0000 7f267ca56740 20 client.20589 trim_cache size 0 max 16384 2024-09-12T03:10:05.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.298+0000 7f267ca56740 1 Processor -- start 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 -- start start 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006b4a0 0x5564000821a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006bdc0 0x55640008dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26741ab3b0 con 0x556400076680 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26741acef0 con 0x55640006b4a0 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f267ca56740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26741aba80 con 0x55640006bdc0 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44202/0 (socket says 172.21.15.16:44202) 2024-09-12T03:10:05.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 -- 172.21.15.16:0/2299666368 learned_addr learned my addr 172.21.15.16:0/2299666368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f26737fe640 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006bdc0 0x55640008dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2673fff640 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006b4a0 0x5564000821a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 -- 172.21.15.16:0/2299666368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006bdc0 msgr2=0x55640008dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006bdc0 0x55640008dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 -- 172.21.15.16:0/2299666368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006b4a0 msgr2=0x5564000821a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006b4a0 0x5564000821a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 -- 172.21.15.16:0/2299666368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556400073b50 con 0x556400076680 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.299+0000 7f2678c1a640 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f266c00bf20 tx=0x7f266c00d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b0e4ec1881586 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f2659ffb640 1 -- 172.21.15.16:0/2299666368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c02a070 con 0x556400076680 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f2659ffb640 1 -- 172.21.15.16:0/2299666368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f266c01adf0 con 0x556400076680 2024-09-12T03:10:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f2659ffb640 1 -- 172.21.15.16:0/2299666368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c014040 con 0x556400076680 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 -- 172.21.15.16:0/2299666368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 msgr2=0x55640008e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f266c00bf20 tx=0x7f266c00d040 comp rx=0 tx=0).stop 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 -- 172.21.15.16:0/2299666368 shutdown_connections 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556400076680 0x55640008e000 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55640006bdc0 0x55640008dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 --2- 172.21.15.16:0/2299666368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006b4a0 0x5564000821a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 -- 172.21.15.16:0/2299666368 >> 172.21.15.16:0/2299666368 conn(0x556400067a30 msgr2=0x55640006e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 -- 172.21.15.16:0/2299666368 shutdown_connections 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.300+0000 7f267ca56740 1 -- 172.21.15.16:0/2299666368 wait complete. 2024-09-12T03:10:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 -- start start 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 Processor -- start 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 0x55640007eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556400076680 0x556400079970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26741ac820 con 0x55640006b4a0 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26741ac150 con 0x55640006bdc0 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26741a9390 con 0x556400076680 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f267ca56740 10 client.0 _ll_register_callbacks cb 0x7f2668004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f2673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44214/0 (socket says 172.21.15.16:44214) 2024-09-12T03:10:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f2673fff640 1 -- 172.21.15.16:0/1860704351 learned_addr learned my addr 172.21.15.16:0/1860704351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.301+0000 7f26737fe640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 0x55640007eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2678c1a640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556400076680 0x556400079970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 -- 172.21.15.16:0/1860704351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556400076680 msgr2=0x556400079970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556400076680 0x556400079970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 -- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 msgr2=0x55640007eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 0x55640007eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 -- 172.21.15.16:0/1860704351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640007a1a0 con 0x55640006b4a0 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f26737fe640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 0x55640007eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f2673fff640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f26640ed1a0 tx=0x7f2664009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376fecee4c99449b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:05.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f26537fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664049040 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f26640ba320 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f266c01adf0 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f26640bbb60 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f267ca56740 10 client.20595 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.302+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5564000f0680 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f266c01adf0 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f26640c4750 con 0x55640006b4a0 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 10 client.20595 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 10 client.20595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 20 client.20595 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564000f0b70 0x5564000f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 4 client.20595 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5563fff55210 con 0x5564000f0b70 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f26737fe640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564000f0b70 0x5564000f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f26737fe640 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564000f0b70 0x5564000f2f60 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95afd99a41315b3f server_cookie=f6e5a96f641ad81f in_seq=0 out_seq=0 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.303+0000 7f26537fe640 10 client.20595 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f26537fe640 1 -- 172.21.15.16:0/1860704351 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5563fff55210 con 0x5564000f0b70 2024-09-12T03:10:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f26537fe640 10 client.20595 handle_command_reply: tid=0 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 client.20595 shutdown 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 msgr2=0x55640007e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f26640ed1a0 tx=0x7f2664009880 comp rx=0 tx=0).stop 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 shutdown_connections 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5564000f0b70 0x5564000f2f60 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556400076680 0x556400079970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55640006bdc0 0x55640007eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 --2- 172.21.15.16:0/1860704351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55640006b4a0 0x55640007e5e0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 >> 172.21.15.16:0/1860704351 conn(0x556400067a30 msgr2=0x556400074ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f26537fe640 0 client.20595 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 shutdown_connections 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 1 -- 172.21.15.16:0/1860704351 wait complete. 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.304+0000 7f267ca56740 20 client.20595 trim_cache size 0 max 16384 2024-09-12T03:10:05.312 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "966d541e-88aa-43df-b8ef-5544ebb7fd08": { 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "966d541e-88aa-43df-b8ef-5544ebb7fd08", 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:05.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:05.314 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 msgr2=0x7f265c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 0x7f265c0a5040 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f266c009900 tx=0x7f266c009650 comp rx=0 tx=0).stop 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 msgr2=0x7f2674198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2664006010 tx=0x7f26640042f0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 shutdown_connections 2024-09-12T03:10:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2674106f10 0x7f267419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2674102790 0x7f26741989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f265c0a2b80 0x7f265c0a5040 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 --2- 172.21.15.16:0/1326620723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2674101de0 0x7f2674198490 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 >> 172.21.15.16:0/1326620723 conn(0x7f26740fd8c0 msgr2=0x7f26740fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 shutdown_connections 2024-09-12T03:10:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:05.305+0000 7f267a67d640 1 -- 172.21.15.16:0/1326620723 wait complete. 2024-09-12T03:10:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:05 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:05 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:05 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:05.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:05 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:05 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:05 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:05 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:05 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:06.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.89 MB/sec execute 427 sec latency 17167.561 ms 2024-09-12T03:10:06.176 INFO:tasks.cephfs.filesystem:scrub status for tag:966d541e-88aa-43df-b8ef-5544ebb7fd08 - {'path': '/', 'tag': '966d541e-88aa-43df-b8ef-5544ebb7fd08', 'options': 'recursive,force'} 2024-09-12T03:10:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:06 smithi016 ceph-mon[30925]: pgmap v1002: 97 pgs: 97 active+clean; 853 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 5.7 MiB/s wr, 218 op/s 2024-09-12T03:10:06.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:06 smithi033 ceph-mon[35092]: pgmap v1002: 97 pgs: 97 active+clean; 853 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 5.7 MiB/s wr, 218 op/s 2024-09-12T03:10:06.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:06 smithi184 ceph-mon[35321]: pgmap v1002: 97 pgs: 97 active+clean; 853 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 5.7 MiB/s wr, 218 op/s 2024-09-12T03:10:07.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.84 MB/sec execute 428 sec latency 18167.654 ms 2024-09-12T03:10:08.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.78 MB/sec execute 429 sec latency 19167.759 ms 2024-09-12T03:10:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:08 smithi016 ceph-mon[30925]: pgmap v1003: 97 pgs: 97 active+clean; 852 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 6.0 MiB/s wr, 247 op/s 2024-09-12T03:10:08.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:08 smithi033 ceph-mon[35092]: pgmap v1003: 97 pgs: 97 active+clean; 852 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 6.0 MiB/s wr, 247 op/s 2024-09-12T03:10:08.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:08 smithi184 ceph-mon[35321]: pgmap v1003: 97 pgs: 97 active+clean; 852 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 6.0 MiB/s wr, 247 op/s 2024-09-12T03:10:09.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.73 MB/sec execute 430 sec latency 20167.810 ms 2024-09-12T03:10:10.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.67 MB/sec execute 431 sec latency 21167.912 ms 2024-09-12T03:10:10.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:10 smithi016 ceph-mon[30925]: pgmap v1004: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-09-12T03:10:10.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:10 smithi033 ceph-mon[35092]: pgmap v1004: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-09-12T03:10:10.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:10 smithi184 ceph-mon[35321]: pgmap v1004: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-09-12T03:10:11.032 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.62 MB/sec execute 432 sec latency 22168.033 ms 2024-09-12T03:10:12.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.56 MB/sec execute 433 sec latency 23168.143 ms 2024-09-12T03:10:12.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:12 smithi016 ceph-mon[30925]: pgmap v1005: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-09-12T03:10:12.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:12 smithi033 ceph-mon[35092]: pgmap v1005: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-09-12T03:10:12.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:12 smithi184 ceph-mon[35321]: pgmap v1005: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-09-12T03:10:13.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.51 MB/sec execute 434 sec latency 24168.178 ms 2024-09-12T03:10:14.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.45 MB/sec execute 435 sec latency 25168.217 ms 2024-09-12T03:10:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:14 smithi016 ceph-mon[30925]: pgmap v1006: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-09-12T03:10:14.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:14 smithi033 ceph-mon[35092]: pgmap v1006: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-09-12T03:10:14.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:14 smithi184 ceph-mon[35321]: pgmap v1006: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-09-12T03:10:15.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.40 MB/sec execute 436 sec latency 26168.307 ms 2024-09-12T03:10:15.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:10:16.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.35 MB/sec execute 437 sec latency 27168.372 ms 2024-09-12T03:10:16.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:16.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:16 smithi016 ceph-mon[30925]: pgmap v1007: 97 pgs: 97 active+clean; 845 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-09-12T03:10:16.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:16 smithi033 ceph-mon[35092]: pgmap v1007: 97 pgs: 97 active+clean; 845 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-09-12T03:10:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:16 smithi184 ceph-mon[35321]: pgmap v1007: 97 pgs: 97 active+clean; 845 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-09-12T03:10:16.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 -- 172.21.15.16:0/360153288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148102790 msgr2=0x7f21481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 --2- 172.21.15.16:0/360153288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148102790 0x7f21481067e0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2138009900 tx=0x7f213802a410 comp rx=0 tx=0).stop 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 -- 172.21.15.16:0/360153288 shutdown_connections 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 --2- 172.21.15.16:0/360153288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 0x7f214810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 --2- 172.21.15.16:0/360153288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148102790 0x7f21481067e0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 --2- 172.21.15.16:0/360153288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148101de0 0x7f21481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 -- 172.21.15.16:0/360153288 >> 172.21.15.16:0/360153288 conn(0x7f21480fd8c0 msgr2=0x7f21480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 -- 172.21.15.16:0/360153288 shutdown_connections 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.634+0000 7f214ee75640 1 -- 172.21.15.16:0/360153288 wait complete. 2024-09-12T03:10:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 Processor -- start 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 -- start start 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148102790 0x7f21481989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 0x7f214819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2148111f70 con 0x7f2148101de0 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2148111df0 con 0x7f2148106f10 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214ee75640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21481120f0 con 0x7f2148102790 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214cc11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214cc11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55554/0 (socket says 172.21.15.16:55554) 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214cc11640 1 -- 172.21.15.16:0/3522659342 learned_addr learned my addr 172.21.15.16:0/3522659342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f214d412640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 0x7f214819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.635+0000 7f213ffff640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148102790 0x7f21481989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:16.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 -- 172.21.15.16:0/3522659342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148102790 msgr2=0x7f21481989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148102790 0x7f21481989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 -- 172.21.15.16:0/3522659342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 msgr2=0x7f214819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 0x7f214819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 -- 172.21.15.16:0/3522659342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214819d440 con 0x7f2148101de0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214cc11640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f214400d950 tx=0x7f214400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144017070 con 0x7f2148101de0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f214819d730 con 0x7f2148101de0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2144009c30 con 0x7f2148101de0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214400f6b0 con 0x7f2148101de0 2024-09-12T03:10:16.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.636+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21481a5010 con 0x7f2148101de0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.637+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21481021c0 con 0x7f2148101de0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.638+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f21440109d0 con 0x7f2148101de0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.639+0000 7f213dffb640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 0x7f21300a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.639+0000 7f213ffff640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 0x7f21300a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.639+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f21440cebf0 con 0x7f2148101de0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.640+0000 7f213ffff640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 0x7f21300a5040 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2138009230 tx=0x7f21380058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:16.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.644+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2144009de0 con 0x7f2148101de0 2024-09-12T03:10:16.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.861+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f214810a150 con 0x7f2148101de0 2024-09-12T03:10:16.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.862+0000 7f213dffb640 1 -- 172.21.15.16:0/3522659342 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f214409b510 con 0x7f2148101de0 2024-09-12T03:10:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:10:16.863 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:10:16.864 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:10:16.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 msgr2=0x7f21300a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 0x7f21300a5040 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2138009230 tx=0x7f21380058d0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 msgr2=0x7f2148198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f214400d950 tx=0x7f214400de10 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 shutdown_connections 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2148106f10 0x7f214819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2148102790 0x7f21481989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21300a2b80 0x7f21300a5040 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 --2- 172.21.15.16:0/3522659342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2148101de0 0x7f2148198490 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.864+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 >> 172.21.15.16:0/3522659342 conn(0x7f21480fd8c0 msgr2=0x7f21480fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.865+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 shutdown_connections 2024-09-12T03:10:16.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:16.865+0000 7f214ee75640 1 -- 172.21.15.16:0/3522659342 wait complete. 2024-09-12T03:10:17.033 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.29 MB/sec execute 438 sec latency 28168.476 ms 2024-09-12T03:10:17.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:17 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3522659342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:17.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:17 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3522659342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:17.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:17 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3522659342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:17.795 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id a19b8590 --release --await' 2024-09-12T03:10:17.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id a19b8590 --release --await 2024-09-12T03:10:18.035 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229838 23.24 MB/sec execute 439 sec latency 29170.972 ms 2024-09-12T03:10:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:18.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:18 smithi016 ceph-mon[30925]: pgmap v1008: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 5.0 MiB/s wr, 227 op/s 2024-09-12T03:10:18.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:10:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:10:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:18 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:10:18.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:18 smithi033 ceph-mon[35092]: pgmap v1008: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 5.0 MiB/s wr, 227 op/s 2024-09-12T03:10:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:10:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:10:18.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:18 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:10:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:18 smithi184 ceph-mon[35321]: pgmap v1008: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 5.0 MiB/s wr, 227 op/s 2024-09-12T03:10:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:10:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:10:18.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:18 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:10:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 -- 172.21.15.16:0/1951007858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c103780 msgr2=0x7f353c107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 --2- 172.21.15.16:0/1951007858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c103780 0x7f353c107bc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3530009860 tx=0x7f353002a330 comp rx=0 tx=0).stop 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 -- 172.21.15.16:0/1951007858 shutdown_connections 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 --2- 172.21.15.16:0/1951007858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c108300 0x7f353c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 --2- 172.21.15.16:0/1951007858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c103780 0x7f353c107bc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 --2- 172.21.15.16:0/1951007858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c102dd0 0x7f353c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 -- 172.21.15.16:0/1951007858 >> 172.21.15.16:0/1951007858 conn(0x7f353c0fc7d0 msgr2=0x7f353c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 -- 172.21.15.16:0/1951007858 shutdown_connections 2024-09-12T03:10:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.681+0000 7f3541282640 1 -- 172.21.15.16:0/1951007858 wait complete. 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 Processor -- start 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 -- start start 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c102dd0 0x7f353c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c103780 0x7f353c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f353c112050 con 0x7f353c103780 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f353c111ed0 con 0x7f353c108300 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.682+0000 7f3541282640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1121d0 con 0x7f353c102dd0 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c102dd0 0x7f353c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c103780 0x7f353c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54122/0 (socket says 172.21.15.16:54122) 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 -- 172.21.15.16:0/3994092772 learned_addr learned my addr 172.21.15.16:0/3994092772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 -- 172.21.15.16:0/3994092772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c102dd0 msgr2=0x7f353c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c102dd0 0x7f353c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 -- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c103780 msgr2=0x7f353c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c103780 0x7f353c198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 -- 172.21.15.16:0/3994092772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c19d540 con 0x7f353c108300 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.683+0000 7f353b7fe640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3534009fd0 tx=0x7f353400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.684+0000 7f3523fff640 1 -- 172.21.15.16:0/3994092772 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534004b20 con 0x7f353c108300 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.684+0000 7f3523fff640 1 -- 172.21.15.16:0/3994092772 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3534003e50 con 0x7f353c108300 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.684+0000 7f3523fff640 1 -- 172.21.15.16:0/3994092772 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353400b630 con 0x7f353c108300 2024-09-12T03:10:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.684+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353c19d830 con 0x7f353c108300 2024-09-12T03:10:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.684+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353c1a5110 con 0x7f353c108300 2024-09-12T03:10:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.686+0000 7f3523fff640 1 -- 172.21.15.16:0/3994092772 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f353401d020 con 0x7f353c108300 2024-09-12T03:10:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f3523fff640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 0x7f35240a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f3523fff640 1 -- 172.21.15.16:0/3994092772 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f35340cef20 con 0x7f353c108300 2024-09-12T03:10:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f353affd640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 0x7f35240a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 Processor -- start 2024-09-12T03:10:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 -- start start 2024-09-12T03:10:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc8712c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8713b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc8713da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc87091620 con 0x55dc8712c840 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dc87038e50 con 0x55dc8713b650 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.687+0000 7f354365b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dc8708d5e0 con 0x55dc8712bf20 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 0x7f35240a50c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f352c009bd0 tx=0x7f352c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8713b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc8712c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8713b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55586/0 (socket says 172.21.15.16:55586) 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353a7fc640 1 -- 172.21.15.16:0/406614550 learned_addr learned my addr 172.21.15.16:0/406614550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353b7fe640 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc8713da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 -- 172.21.15.16:0/406614550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 msgr2=0x55dc8713da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc8713da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 -- 172.21.15.16:0/406614550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 msgr2=0x55dc8713b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8713b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 -- 172.21.15.16:0/406614550 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc87029d40 con 0x55dc8712bf20 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f353affd640 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc8712c300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f352c009fd0 tx=0x7f352c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1085c684bb418266 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.688+0000 7f35217fa640 1 -- 172.21.15.16:0/406614550 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c004030 con 0x55dc8712bf20 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f35217fa640 1 -- 172.21.15.16:0/406614550 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352c0041d0 con 0x55dc8712bf20 2024-09-12T03:10:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f35217fa640 1 -- 172.21.15.16:0/406614550 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0266f0 con 0x55dc8712bf20 2024-09-12T03:10:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 -- 172.21.15.16:0/406614550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 msgr2=0x55dc8712c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc8712c300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f352c009fd0 tx=0x7f352c014000 comp rx=0 tx=0).stop 2024-09-12T03:10:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 -- 172.21.15.16:0/406614550 shutdown_connections 2024-09-12T03:10:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc8713da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8713b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 --2- 172.21.15.16:0/406614550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc8712c300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 -- 172.21.15.16:0/406614550 >> 172.21.15.16:0/406614550 conn(0x55dc871285f0 msgr2=0x55dc871289d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 -- 172.21.15.16:0/406614550 shutdown_connections 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.689+0000 7f354365b740 1 -- 172.21.15.16:0/406614550 wait complete. 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 -- start start 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 Processor -- start 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 0x55dc87138b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 0x55dc87139080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc8713e220 con 0x55dc8713b650 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dc87019f10 con 0x55dc8712c840 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dc8713e3a0 con 0x55dc8712bf20 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f354365b740 10 client.0 _ll_register_callbacks cb 0x55dc8703ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f353affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 0x55dc87138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f353b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 0x55dc87139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f353affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33364/0 (socket says 172.21.15.16:33364) 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.690+0000 7f353affd640 1 -- 172.21.15.16:0/2916039595 learned_addr learned my addr 172.21.15.16:0/2916039595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 -- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 msgr2=0x55dc87138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 0x55dc87138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 -- 172.21.15.16:0/2916039595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 msgr2=0x55dc87139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 0x55dc87139080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 -- 172.21.15.16:0/2916039595 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc86da0580 con 0x55dc8712bf20 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353a7fc640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 0x55dc87138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353b7fe640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 0x55dc87139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f353affd640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f352c003e00 tx=0x7f352c004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4c7fd39f88870a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f351eff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:10:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c026730 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352c0041d0 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f352c0269f0 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f352c026bf0 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f354365b740 10 client.29740 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55dc86e6a960 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.691+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352c0041d0 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f352c0046c0 con 0x55dc8712bf20 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 10 client.29740 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 10 client.29740 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 20 client.29740 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc8714b1b0 0x55dc8714d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 4 client.29740 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc8709eb90 con 0x55dc8714b1b0 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.692+0000 7f353a7fc640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc8714b1b0 0x55dc8714d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f353a7fc640 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc8714b1b0 0x55dc8714d5a0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcaf6084891926a8 server_cookie=53b8c902c2480058 in_seq=0 out_seq=0 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f351eff5640 10 client.29740 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f351eff5640 1 -- 172.21.15.16:0/2916039595 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc8709eb90 con 0x55dc8714b1b0 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f351eff5640 10 client.29740 handle_command_reply: tid=0 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 client.29740 shutdown 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 msgr2=0x55dc87138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f352c003e00 tx=0x7f352c004400 comp rx=0 tx=0).stop 2024-09-12T03:10:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 shutdown_connections 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc8714b1b0 0x55dc8714d5a0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8713b650 0x55dc87139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712c840 0x55dc87138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 --2- 172.21.15.16:0/2916039595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712bf20 0x55dc87138600 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 >> 172.21.15.16:0/2916039595 conn(0x55dc871285f0 msgr2=0x55dc87127750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.693+0000 7f351eff5640 0 client.29740 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.694+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 shutdown_connections 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.694+0000 7f354365b740 1 -- 172.21.15.16:0/2916039595 wait complete. 2024-09-12T03:10:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.694+0000 7f354365b740 20 client.29740 trim_cache size 0 max 16384 2024-09-12T03:10:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 Processor -- start 2024-09-12T03:10:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 -- start start 2024-09-12T03:10:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 0x55dc871374b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc87150720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1ab450 con 0x55dc8712bf20 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1acf90 con 0x55dc8713b650 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f354365b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1abb20 con 0x55dc8712c840 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33376/0 (socket says 172.21.15.16:33376) 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f353a7fc640 1 -- 172.21.15.16:0/3010322530 learned_addr learned my addr 172.21.15.16:0/3010322530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f353affd640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 0x55dc871374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.716+0000 7f353b7fe640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc87150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 -- 172.21.15.16:0/3010322530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 msgr2=0x55dc87150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc87150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 -- 172.21.15.16:0/3010322530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 msgr2=0x55dc871374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 0x55dc871374b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 -- 172.21.15.16:0/3010322530 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc8712c670 con 0x55dc8712c840 2024-09-12T03:10:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353b7fe640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc87150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353affd640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 0x55dc871374b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.717+0000 7f353a7fc640 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3530035040 tx=0x7f35300058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bec3b0f0c81f4988 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.718+0000 7f3520ff9640 1 -- 172.21.15.16:0/3010322530 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353004c070 con 0x55dc8712c840 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.718+0000 7f3520ff9640 1 -- 172.21.15.16:0/3010322530 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530030770 con 0x55dc8712c840 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.718+0000 7f3520ff9640 1 -- 172.21.15.16:0/3010322530 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530037040 con 0x55dc8712c840 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 -- 172.21.15.16:0/3010322530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 msgr2=0x55dc871501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3530035040 tx=0x7f35300058d0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 -- 172.21.15.16:0/3010322530 shutdown_connections 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8713b650 0x55dc87150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8712c840 0x55dc871501e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 --2- 172.21.15.16:0/3010322530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712bf20 0x55dc871374b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 -- 172.21.15.16:0/3010322530 >> 172.21.15.16:0/3010322530 conn(0x55dc871285f0 msgr2=0x55dc87136da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 -- 172.21.15.16:0/3010322530 shutdown_connections 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.719+0000 7f354365b740 1 -- 172.21.15.16:0/3010322530 wait complete. 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 -- start start 2024-09-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 Processor -- start 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 0x55dc8714df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8713b650 0x55dc8714e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1ac8c0 con 0x55dc8712c840 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1ac1f0 con 0x55dc8712bf20 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1a9430 con 0x55dc8713b650 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f354365b740 10 client.0 _ll_register_callbacks cb 0x7f352c004090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f353a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55608/0 (socket says 172.21.15.16:55608) 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f353a7fc640 1 -- 172.21.15.16:0/1221855024 learned_addr learned my addr 172.21.15.16:0/1221855024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f353affd640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 0x55dc8714df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.720+0000 7f353b7fe640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8713b650 0x55dc8714e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 -- 172.21.15.16:0/1221855024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8713b650 msgr2=0x55dc8714e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8713b650 0x55dc8714e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 -- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 msgr2=0x55dc8714df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 0x55dc8714df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 -- 172.21.15.16:0/1221855024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc871b4af0 con 0x55dc8712c840 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353affd640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 0x55dc8714df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f353a7fc640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3530009780 tx=0x7f3530030920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8915e75555b7b5ba server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f351eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353004c030 con 0x55dc8712c840 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3530044040 con 0x55dc8712c840 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3530030770 con 0x55dc8712c840 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3530037040 con 0x55dc8712c840 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f354365b740 10 client.20619 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.721+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55dc871b5e40 con 0x55dc8712c840 2024-09-12T03:10:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530030770 con 0x55dc8712c840 2024-09-12T03:10:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f3530032440 con 0x55dc8712c840 2024-09-12T03:10:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 10 client.20619 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 10 client.20619 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 20 client.20619 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc871b65a0 0x55dc871b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 4 client.20619 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a19b8590", "release": true, "await": true}] 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a19b8590", "release": true, "await": true}) -- 0x55dc870990a0 con 0x55dc871b65a0 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.722+0000 7f353affd640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc871b65a0 0x55dc871b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.723+0000 7f351eff5640 10 client.20619 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.723+0000 7f353affd640 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc871b65a0 0x55dc871b89c0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae8514700beff894 server_cookie=70877b3b5b3acd5f in_seq=0 out_seq=0 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.785+0000 7f351eff5640 1 -- 172.21.15.16:0/1221855024 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55dc870990a0 con 0x55dc871b65a0 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.785+0000 7f351eff5640 10 client.20619 handle_command_reply: tid=0 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.785+0000 7f354365b740 1 client.20619 shutdown 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 msgr2=0x55dc8714e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3530009780 tx=0x7f3530030920 comp rx=0 tx=0).stop 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 shutdown_connections 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55dc871b65a0 0x55dc871b89c0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dc8713b650 0x55dc8714e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f351eff5640 0 client.20619 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc8712c840 0x55dc8714e4b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 --2- 172.21.15.16:0/1221855024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dc8712bf20 0x55dc8714df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 >> 172.21.15.16:0/1221855024 conn(0x55dc871285f0 msgr2=0x55dc87127750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 shutdown_connections 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 1 -- 172.21.15.16:0/1221855024 wait complete. 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.786+0000 7f354365b740 20 client.20619 trim_cache size 0 max 16384 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 80, 2024-09-12T03:10:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "a19b8590": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 80, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 msgr2=0x7f35240a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 0x7f35240a50c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f352c009bd0 tx=0x7f352c005f90 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 msgr2=0x7f353c19ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3534009fd0 tx=0x7f353400c800 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 shutdown_connections 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f353c108300 0x7f353c19ce60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35240a2c00 0x7f35240a50c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f353c103780 0x7f353c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.787+0000 7f3541282640 1 --2- 172.21.15.16:0/3994092772 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f353c102dd0 0x7f353c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.788+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 >> 172.21.15.16:0/3994092772 conn(0x7f353c0fc7d0 msgr2=0x7f353c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.788+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 shutdown_connections 2024-09-12T03:10:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:18.788+0000 7f3541282640 1 -- 172.21.15.16:0/3994092772 wait complete. 2024-09-12T03:10:19.035 INFO:tasks.workunit.client.0.smithi016.stdout: 10 229957 23.20 MB/sec execute 440 sec latency 29942.904 ms 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:19 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:10:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:10:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:10:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:19 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:19 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:19.640 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a19b8590', total seconds elapsed: 33.8 2024-09-12T03:10:19.640 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-09-12T03:10:19.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:20.035 INFO:tasks.workunit.client.0.smithi016.stdout: 10 230484 23.22 MB/sec execute 441 sec latency 57.651 ms 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: pgmap v1009: 97 pgs: 97 active+clean; 848 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 4.9 MiB/s wr, 194 op/s 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:10:20.384 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: pgmap v1009: 97 pgs: 97 active+clean; 848 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 4.9 MiB/s wr, 194 op/s 2024-09-12T03:10:20.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:10:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2465697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 msgr2=0x7fd6d0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2465697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0100ee0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00d940 tx=0x7fd6cc02a190 comp rx=0 tx=0).stop 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2465697426 shutdown_connections 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2465697426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 0x7fd6d010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2465697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0100ee0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2465697426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2465697426 >> 172.21.15.16:0/2465697426 conn(0x7fd6d00fc7a0 msgr2=0x7fd6d00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.494+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2465697426 shutdown_connections 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.495+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2465697426 wait complete. 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.495+0000 7fd6d7bed640 1 Processor -- start 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.495+0000 7fd6d7bed640 1 -- start start 2024-09-12T03:10:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 0x7fd6d01967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010fd70 con 0x7fd6d0106a80 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010fbf0 con 0x7fd6d0101420 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d7bed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010fef0 con 0x7fd6d0100a80 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d5989640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d618a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d618a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55626/0 (socket says 172.21.15.16:55626) 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d618a640 1 -- 172.21.15.16:0/2046139582 learned_addr learned my addr 172.21.15.16:0/2046139582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.496+0000 7fd6d5188640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 0x7fd6d01967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 -- 172.21.15.16:0/2046139582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 msgr2=0x7fd6d0196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 -- 172.21.15.16:0/2046139582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 msgr2=0x7fd6d01967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 0x7fd6d01967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 -- 172.21.15.16:0/2046139582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d019b220 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.497+0000 7fd6d618a640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800eb10 tx=0x7fd6c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.498+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8004b20 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.498+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c8004cc0 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.498+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8010760 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.498+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d019b510 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.498+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d01a2df0 con 0x7fd6d0106a80 2024-09-12T03:10:20.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.499+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d0100ee0 con 0x7fd6d0106a80 2024-09-12T03:10:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.501+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd6c8005020 con 0x7fd6d0106a80 2024-09-12T03:10:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.502+0000 7fd6c6ffd640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 0x7fd6bc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.502+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6c80cf5f0 con 0x7fd6d0106a80 2024-09-12T03:10:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.502+0000 7fd6d5989640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 0x7fd6bc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.503+0000 7fd6d5989640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 0x7fd6bc0a50c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000fed0 tx=0x7fd6c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.510+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6c809c500 con 0x7fd6d0106a80 2024-09-12T03:10:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: pgmap v1009: 97 pgs: 97 active+clean; 848 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 4.9 MiB/s wr, 194 op/s 2024-09-12T03:10:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:10:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:10:20.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:10:20.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.824+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d0109040 con 0x7fd6d0106a80 2024-09-12T03:10:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.825+0000 7fd6c6ffd640 1 -- 172.21.15.16:0/2046139582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fd6c809be00 con 0x7fd6d0106a80 2024-09-12T03:10:20.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:10:20.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:10:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:10:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 msgr2=0x7fd6bc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 0x7fd6bc0a50c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000fed0 tx=0x7fd6c0005f90 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 msgr2=0x7fd6d019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800eb10 tx=0x7fd6c800c2d0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 shutdown_connections 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6bc0a2c00 0x7fd6bc0a50c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0106a80 0x7fd6d019ab40 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6d0101420 0x7fd6d01967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 --2- 172.21.15.16:0/2046139582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6d0100a80 0x7fd6d0196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 >> 172.21.15.16:0/2046139582 conn(0x7fd6d00fc7a0 msgr2=0x7fd6d00fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.827+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 shutdown_connections 2024-09-12T03:10:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:20.828+0000 7fd6d7bed640 1 -- 172.21.15.16:0/2046139582 wait complete. 2024-09-12T03:10:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231059 23.24 MB/sec execute 442 sec latency 63.274 ms 2024-09-12T03:10:21.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:21 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2046139582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:21.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:21 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2046139582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:21.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:21 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2046139582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:21.665 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-09-12T03:10:21.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-09-12T03:10:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:22.035 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231621 23.25 MB/sec execute 443 sec latency 54.870 ms 2024-09-12T03:10:22.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:22 smithi016 ceph-mon[30925]: pgmap v1010: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 239 op/s 2024-09-12T03:10:22.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:22 smithi033 ceph-mon[35092]: pgmap v1010: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 239 op/s 2024-09-12T03:10:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:22 smithi184 ceph-mon[35321]: pgmap v1010: 97 pgs: 97 active+clean; 850 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 239 op/s 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.544+0000 7f166c28b640 1 -- 172.21.15.16:0/2139770938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 msgr2=0x7f16641108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.544+0000 7f166c28b640 1 --2- 172.21.15.16:0/2139770938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f16641108e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f165c009ca0 tx=0x7f165c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.545+0000 7f166c28b640 1 -- 172.21.15.16:0/2139770938 shutdown_connections 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.545+0000 7f166c28b640 1 --2- 172.21.15.16:0/2139770938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f16641108e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.545+0000 7f166c28b640 1 --2- 172.21.15.16:0/2139770938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16641012f0 0x7f166410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.545+0000 7f166c28b640 1 --2- 172.21.15.16:0/2139770938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1664100940 0x7f1664100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.545+0000 7f166c28b640 1 -- 172.21.15.16:0/2139770938 >> 172.21.15.16:0/2139770938 conn(0x7f16640fc7d0 msgr2=0x7f16640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.546+0000 7f166c28b640 1 -- 172.21.15.16:0/2139770938 shutdown_connections 2024-09-12T03:10:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.546+0000 7f166c28b640 1 -- 172.21.15.16:0/2139770938 wait complete. 2024-09-12T03:10:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 Processor -- start 2024-09-12T03:10:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 -- start start 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 0x7f166419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16641012f0 0x7f166419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1664114160 con 0x7f166410e4f0 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1664113fe0 con 0x7f16641012f0 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166c28b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16641142e0 con 0x7f1664100940 2024-09-12T03:10:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166a828640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.547+0000 7f166a828640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55636/0 (socket says 172.21.15.16:55636) 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 -- 172.21.15.16:0/3740532580 learned_addr learned my addr 172.21.15.16:0/3740532580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a027640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 0x7f166419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 -- 172.21.15.16:0/3740532580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 msgr2=0x7f166419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 0x7f166419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 -- 172.21.15.16:0/3740532580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16641012f0 msgr2=0x7f166419ab60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16641012f0 0x7f166419ab60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 -- 172.21.15.16:0/3740532580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166419f5d0 con 0x7f166410e4f0 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a027640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 0x7f166419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.548+0000 7f166a828640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f165c0319f0 tx=0x7f165c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.549+0000 7f165b7fe640 1 -- 172.21.15.16:0/3740532580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c003b70 con 0x7f166410e4f0 2024-09-12T03:10:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.549+0000 7f165b7fe640 1 -- 172.21.15.16:0/3740532580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165c003d10 con 0x7f166410e4f0 2024-09-12T03:10:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.549+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166419f860 con 0x7f166410e4f0 2024-09-12T03:10:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.549+0000 7f165b7fe640 1 -- 172.21.15.16:0/3740532580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c02fc10 con 0x7f166410e4f0 2024-09-12T03:10:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.549+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16641a71a0 con 0x7f166410e4f0 2024-09-12T03:10:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.552+0000 7f166e664740 1 Processor -- start 2024-09-12T03:10:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.552+0000 7f165b7fe640 1 -- 172.21.15.16:0/3740532580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f165c0305c0 con 0x7f166410e4f0 2024-09-12T03:10:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 -- start start 2024-09-12T03:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d210080 0x55f36d21a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d21cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f36d1752a0 con 0x55f36d21a940 2024-09-12T03:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f36d11cb10 con 0x55f36d20f760 2024-09-12T03:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f166e664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f36d171260 con 0x55f36d210080 2024-09-12T03:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f1669826640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f1669826640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54208/0 (socket says 172.21.15.16:54208) 2024-09-12T03:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.553+0000 7f1669826640 1 -- 172.21.15.16:0/2647254585 learned_addr learned my addr 172.21.15.16:0/2647254585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.554+0000 7f166a027640 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d210080 0x55f36d21a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d21cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 -- 172.21.15.16:0/2647254585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d210080 msgr2=0x55f36d21a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d210080 0x55f36d21a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 -- 172.21.15.16:0/2647254585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 msgr2=0x55f36d21cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d21cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.555+0000 7f1669826640 1 -- 172.21.15.16:0/2647254585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f36cee6b90 con 0x55f36d20f760 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.556+0000 7f165b7fe640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 0x7f16500a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.556+0000 7f165b7fe640 1 -- 172.21.15.16:0/3740532580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f165c0f03b0 con 0x7f166410e4f0 2024-09-12T03:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.557+0000 7f166a027640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 0x7f16500a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.557+0000 7f1669826640 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1654009fa0 tx=0x7f1654009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd2378bfd30efd4b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.558+0000 7f1658dea640 1 -- 172.21.15.16:0/2647254585 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654004020 con 0x55f36d20f760 2024-09-12T03:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.558+0000 7f1658dea640 1 -- 172.21.15.16:0/2647254585 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1654005500 con 0x55f36d20f760 2024-09-12T03:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.558+0000 7f1658dea640 1 -- 172.21.15.16:0/2647254585 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654004d90 con 0x55f36d20f760 2024-09-12T03:10:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.559+0000 7f166e664740 1 -- 172.21.15.16:0/2647254585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 msgr2=0x55f36d20fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.559+0000 7f166e664740 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1654009fa0 tx=0x7f1654009640 comp rx=0 tx=0).stop 2024-09-12T03:10:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.559+0000 7f166a027640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 0x7f16500a5140 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x55f36d20cb50 tx=0x7f1660005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 -- 172.21.15.16:0/2647254585 shutdown_connections 2024-09-12T03:10:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d21cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d210080 0x55f36d21a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 --2- 172.21.15.16:0/2647254585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d20f760 0x55f36d20fb40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 -- 172.21.15.16:0/2647254585 >> 172.21.15.16:0/2647254585 conn(0x55f36d20bcf0 msgr2=0x55f36d20c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.560+0000 7f166e664740 1 -- 172.21.15.16:0/2647254585 shutdown_connections 2024-09-12T03:10:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.561+0000 7f166e664740 1 -- 172.21.15.16:0/2647254585 wait complete. 2024-09-12T03:10:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.561+0000 7f166e664740 1 -- start start 2024-09-12T03:10:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.561+0000 7f166e664740 1 Processor -- start 2024-09-12T03:10:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.561+0000 7f166e664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d217860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.561+0000 7f166e664740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 0x55f36d217da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166e664740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 0x55f36d2182e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166e664740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f36d22e520 con 0x55f36d210080 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166e664740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f36d0fd6e0 con 0x55f36d21a940 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166e664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f36d22e6a0 con 0x55f36d20f760 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166e664740 10 client.0 _ll_register_callbacks cb 0x55f36d121a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166a027640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d217860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166a027640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d217860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33452/0 (socket says 172.21.15.16:33452) 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166a027640 1 -- 172.21.15.16:0/1162902820 learned_addr learned my addr 172.21.15.16:0/1162902820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 0x55f36d217da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f166a828640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 0x55f36d2182e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 -- 172.21.15.16:0/1162902820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 msgr2=0x55f36d217860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d217860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 -- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 msgr2=0x55f36d2182e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 0x55f36d2182e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.562+0000 7f1669826640 1 -- 172.21.15.16:0/1162902820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f36d237a40 con 0x55f36d210080 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f166a828640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 0x55f36d2182e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f1669826640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 0x55f36d217da0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f165400a010 tx=0x7f1654009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9cda0c1cebce2b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f164dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654009c40 con 0x55f36d210080 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1654005500 con 0x55f36d210080 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.563+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1654004020 con 0x55f36d210080 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f16540046c0 con 0x55f36d210080 2024-09-12T03:10:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 10 client.20637 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f36d23a3d0 con 0x55f36d210080 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1654004220 con 0x55f36d210080 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f165402d480 con 0x55f36d210080 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 10 client.20637 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 10 client.20637 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 20 client.20637 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d23bbc0 0x55f36d23dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 4 client.20637 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1654005500 con 0x55f36d23bbc0 2024-09-12T03:10:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.564+0000 7f166a027640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d23bbc0 0x55f36d23dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.566+0000 7f166a027640 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d23bbc0 0x55f36d23dfe0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5bd6cc2f76c00fa server_cookie=57fe35a184ee860a in_seq=0 out_seq=0 2024-09-12T03:10:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.566+0000 7f164dffb640 10 client.20637 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f164dffb640 1 -- 172.21.15.16:0/1162902820 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f1654005500 con 0x55f36d23bbc0 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f164dffb640 10 client.20637 handle_command_reply: tid=0 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 client.20637 shutdown 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 msgr2=0x55f36d217da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 0x55f36d217da0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f165400a010 tx=0x7f1654009530 comp rx=0 tx=0).stop 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 shutdown_connections 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d23bbc0 0x55f36d23dfe0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d21a940 0x55f36d2182e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d210080 0x55f36d217da0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 --2- 172.21.15.16:0/1162902820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d217860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 >> 172.21.15.16:0/1162902820 conn(0x55f36d20bcf0 msgr2=0x55f36d20ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.567+0000 7f164dffb640 0 client.20637 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.568+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 shutdown_connections 2024-09-12T03:10:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.568+0000 7f166e664740 1 -- 172.21.15.16:0/1162902820 wait complete. 2024-09-12T03:10:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.568+0000 7f166e664740 20 client.20637 trim_cache size 0 max 16384 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 Processor -- start 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 -- start start 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d219730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d218130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f166406fc30 con 0x55f36d21a940 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1664071770 con 0x55f36d210080 2024-09-12T03:10:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166e664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1664070300 con 0x55f36d20f760 2024-09-12T03:10:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166a027640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166a027640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:33464/0 (socket says 172.21.15.16:33464) 2024-09-12T03:10:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.591+0000 7f166a027640 1 -- 172.21.15.16:0/926766996 learned_addr learned my addr 172.21.15.16:0/926766996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f1669826640 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d219730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 -- 172.21.15.16:0/926766996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 msgr2=0x55f36d219730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d219730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 -- 172.21.15.16:0/926766996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 msgr2=0x55f36d218130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d218130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 -- 172.21.15.16:0/926766996 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f36d20bb20 con 0x55f36d20f760 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f166a027640 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f166001b790 tx=0x7f1660015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa07cfc445976ee8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f164ffff640 1 -- 172.21.15.16:0/926766996 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166001e0d0 con 0x55f36d20f760 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f164ffff640 1 -- 172.21.15.16:0/926766996 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166001e910 con 0x55f36d20f760 2024-09-12T03:10:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.592+0000 7f164ffff640 1 -- 172.21.15.16:0/926766996 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16600039f0 con 0x55f36d20f760 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 -- 172.21.15.16:0/926766996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 msgr2=0x55f36d216b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f166001b790 tx=0x7f1660015000 comp rx=0 tx=0).stop 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 -- 172.21.15.16:0/926766996 shutdown_connections 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d21a940 0x55f36d218130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d219730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 --2- 172.21.15.16:0/926766996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d20f760 0x55f36d216b80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 -- 172.21.15.16:0/926766996 >> 172.21.15.16:0/926766996 conn(0x55f36d20bcf0 msgr2=0x55f36d20a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 -- 172.21.15.16:0/926766996 shutdown_connections 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.593+0000 7f166e664740 1 -- 172.21.15.16:0/926766996 wait complete. 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 -- start start 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 Processor -- start 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d214400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d21a940 0x55f36d214940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16640710a0 con 0x55f36d20f760 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16640709d0 con 0x55f36d210080 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f166406dbf0 con 0x55f36d21a940 2024-09-12T03:10:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166e664740 10 client.0 _ll_register_callbacks cb 0x7f16540053e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166a027640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166a027640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55672/0 (socket says 172.21.15.16:55672) 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f166a027640 1 -- 172.21.15.16:0/3985777993 learned_addr learned my addr 172.21.15.16:0/3985777993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.594+0000 7f1669826640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d214400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a828640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d21a940 0x55f36d214940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 -- 172.21.15.16:0/3985777993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d21a940 msgr2=0x55f36d214940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d21a940 0x55f36d214940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 -- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 msgr2=0x55f36d214400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d214400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 -- 172.21.15.16:0/3985777993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f36d214fb0 con 0x55f36d20f760 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166a027640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f166001b760 tx=0x7f166001e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8772fd1aff630e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f164dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166001be50 con 0x55f36d20f760 2024-09-12T03:10:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f166001e910 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1660004020 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1660004220 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166e664740 10 client.20640 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.595+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f36d2a66f0 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166001e910 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1660019040 con 0x55f36d20f760 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 10 client.20640 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 10 client.20640 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 20 client.20640 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d2a6ec0 0x55f36d2a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 4 client.20640 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-09-12T03:10:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55f36d1829d0 con 0x55f36d2a6ec0 2024-09-12T03:10:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.596+0000 7f1669826640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d2a6ec0 0x55f36d2a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.597+0000 7f164dffb640 10 client.20640 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:22.597+0000 7f1669826640 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d2a6ec0 0x55f36d2a92e0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b3f761e9a02262 server_cookie=ce2be15c0d04195 in_seq=0 out_seq=0 2024-09-12T03:10:23.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231928 23.23 MB/sec execute 444 sec latency 399.352 ms 2024-09-12T03:10:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f164dffb640 1 -- 172.21.15.16:0/3985777993 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f36d1829d0 con 0x55f36d2a6ec0 2024-09-12T03:10:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f164dffb640 10 client.20640 handle_command_reply: tid=0 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 client.20640 shutdown 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 msgr2=0x55f36d233ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f166001b760 tx=0x7f166001e140 comp rx=0 tx=0).stop 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 shutdown_connections 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f36d2a6ec0 0x55f36d2a92e0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f164dffb640 0 client.20640 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f36d21a940 0x55f36d214940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f36d210080 0x55f36d214400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 --2- 172.21.15.16:0/3985777993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f36d20f760 0x55f36d233ea0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 >> 172.21.15.16:0/3985777993 conn(0x55f36d20bcf0 msgr2=0x55f36d20ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 shutdown_connections 2024-09-12T03:10:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 1 -- 172.21.15.16:0/3985777993 wait complete. 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.341+0000 7f166e664740 20 client.20640 trim_cache size 0 max 16384 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 82, 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "b83f3b93": { 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 82, 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:10:23.344 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:23.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:23.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:23.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:23.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:23.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:23.346 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 msgr2=0x7f16500a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 0x7f16500a5140 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x55f36d20cb50 tx=0x7f1660005f90 comp rx=0 tx=0).stop 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 msgr2=0x7f166419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f165c0319f0 tx=0x7f165c031a20 comp rx=0 tx=0).stop 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 shutdown_connections 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f16500a2c80 0x7f16500a5140 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f166410e4f0 0x7f166419eef0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f16641012f0 0x7f166419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 --2- 172.21.15.16:0/3740532580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1664100940 0x7f166419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 >> 172.21.15.16:0/3740532580 conn(0x7f16640fc7d0 msgr2=0x7f16640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 shutdown_connections 2024-09-12T03:10:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:23.342+0000 7f166c28b640 1 -- 172.21.15.16:0/3740532580 wait complete. 2024-09-12T03:10:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 23.18 MB/sec execute 445 sec latency 1388.642 ms 2024-09-12T03:10:24.064 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b83f3b93', quiesce took 4.4 seconds. Will release after: 25.6 2024-09-12T03:10:24.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:24 smithi016 ceph-mon[30925]: pgmap v1011: 97 pgs: 97 active+clean; 846 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-09-12T03:10:24.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:24 smithi033 ceph-mon[35092]: pgmap v1011: 97 pgs: 97 active+clean; 846 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-09-12T03:10:24.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:24 smithi184 ceph-mon[35321]: pgmap v1011: 97 pgs: 97 active+clean; 846 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-09-12T03:10:25.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 23.12 MB/sec execute 446 sec latency 2388.705 ms 2024-09-12T03:10:26.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 23.07 MB/sec execute 447 sec latency 3388.757 ms 2024-09-12T03:10:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:26 smithi033 ceph-mon[35092]: pgmap v1012: 97 pgs: 97 active+clean; 837 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-09-12T03:10:26.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:26 smithi184 ceph-mon[35321]: pgmap v1012: 97 pgs: 97 active+clean; 837 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-09-12T03:10:26.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:26 smithi016 ceph-mon[30925]: pgmap v1012: 97 pgs: 97 active+clean; 837 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-09-12T03:10:27.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 23.02 MB/sec execute 448 sec latency 4388.867 ms 2024-09-12T03:10:28.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.97 MB/sec execute 449 sec latency 5388.997 ms 2024-09-12T03:10:28.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:28 smithi033 ceph-mon[35092]: pgmap v1013: 97 pgs: 97 active+clean; 816 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-09-12T03:10:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:28 smithi184 ceph-mon[35321]: pgmap v1013: 97 pgs: 97 active+clean; 816 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-09-12T03:10:28.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:28 smithi016 ceph-mon[30925]: pgmap v1013: 97 pgs: 97 active+clean; 816 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-09-12T03:10:29.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.92 MB/sec execute 450 sec latency 6389.103 ms 2024-09-12T03:10:30.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.87 MB/sec execute 451 sec latency 7389.219 ms 2024-09-12T03:10:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:30 smithi033 ceph-mon[35092]: pgmap v1014: 97 pgs: 97 active+clean; 815 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.7 MiB/s wr, 320 op/s 2024-09-12T03:10:30.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:30 smithi184 ceph-mon[35321]: pgmap v1014: 97 pgs: 97 active+clean; 815 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.7 MiB/s wr, 320 op/s 2024-09-12T03:10:30.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:30 smithi016 ceph-mon[30925]: pgmap v1014: 97 pgs: 97 active+clean; 815 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.7 MiB/s wr, 320 op/s 2024-09-12T03:10:31.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.82 MB/sec execute 452 sec latency 8389.336 ms 2024-09-12T03:10:32.036 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.77 MB/sec execute 453 sec latency 9389.444 ms 2024-09-12T03:10:32.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:32 smithi033 ceph-mon[35092]: pgmap v1015: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-09-12T03:10:32.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:32 smithi184 ceph-mon[35321]: pgmap v1015: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-09-12T03:10:32.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:32 smithi016 ceph-mon[30925]: pgmap v1015: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-09-12T03:10:33.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.72 MB/sec execute 454 sec latency 10389.561 ms 2024-09-12T03:10:34.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.67 MB/sec execute 455 sec latency 11389.692 ms 2024-09-12T03:10:34.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:34 smithi033 ceph-mon[35092]: pgmap v1016: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 227 op/s 2024-09-12T03:10:34.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:34 smithi184 ceph-mon[35321]: pgmap v1016: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 227 op/s 2024-09-12T03:10:34.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:34 smithi016 ceph-mon[30925]: pgmap v1016: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 227 op/s 2024-09-12T03:10:35.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.62 MB/sec execute 456 sec latency 12389.798 ms 2024-09-12T03:10:36.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.57 MB/sec execute 457 sec latency 13389.893 ms 2024-09-12T03:10:36.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:10:36.448 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:36.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:36 smithi033 ceph-mon[35092]: pgmap v1017: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-09-12T03:10:36.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:36 smithi184 ceph-mon[35321]: pgmap v1017: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-09-12T03:10:36.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:36 smithi016 ceph-mon[30925]: pgmap v1017: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 -- 172.21.15.16:0/2605805914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 msgr2=0x7fb80c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 --2- 172.21.15.16:0/2605805914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c10ea90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fb804009ca0 tx=0x7fb80402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 -- 172.21.15.16:0/2605805914 shutdown_connections 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 --2- 172.21.15.16:0/2605805914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c10ea90 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 --2- 172.21.15.16:0/2605805914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c102790 0x7fb80c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 --2- 172.21.15.16:0/2605805914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c101de0 0x7fb80c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 -- 172.21.15.16:0/2605805914 >> 172.21.15.16:0/2605805914 conn(0x7fb80c0fd8c0 msgr2=0x7fb80c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 -- 172.21.15.16:0/2605805914 shutdown_connections 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 -- 172.21.15.16:0/2605805914 wait complete. 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.014+0000 7fb8147a4640 1 Processor -- start 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 -- start start 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c101de0 0x7fb80c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c102790 0x7fb80c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c111f70 con 0x7fb80c106f10 2024-09-12T03:10:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c111df0 con 0x7fb80c101de0 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb8147a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1120f0 con 0x7fb80c102790 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35738/0 (socket says 172.21.15.16:35738) 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 -- 172.21.15.16:0/1486189930 learned_addr learned my addr 172.21.15.16:0/1486189930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb811d3f640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c102790 0x7fb80c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812540640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c101de0 0x7fb80c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 -- 172.21.15.16:0/1486189930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c102790 msgr2=0x7fb80c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c102790 0x7fb80c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 -- 172.21.15.16:0/1486189930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c101de0 msgr2=0x7fb80c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c101de0 0x7fb80c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.015+0000 7fb812d41640 1 -- 172.21.15.16:0/1486189930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c1a6800 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb812d41640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fb804031a20 tx=0x7fb804031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb8037fe640 1 -- 172.21.15.16:0/1486189930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb804003b70 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c1a6a90 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb8037fe640 1 -- 172.21.15.16:0/1486189930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb804003d10 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb8037fe640 1 -- 172.21.15.16:0/1486189930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80402fc80 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.016+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c1a6fd0 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.018+0000 7fb8037fe640 1 -- 172.21.15.16:0/1486189930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb8040046a0 con 0x7fb80c106f10 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.018+0000 7fb816b7d740 1 Processor -- start 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.018+0000 7fb816b7d740 1 -- start start 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c194140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c194680 0x563c5c196a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563c5c0ef2e0 con 0x563c5c1894a0 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c5c096ce0 con 0x563c5c189dc0 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb816b7d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563c5c0eb2a0 con 0x563c5c194680 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35754/0 (socket says 172.21.15.16:35754) 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 -- 172.21.15.16:0/3563819108 learned_addr learned my addr 172.21.15.16:0/3563819108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb8037fe640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 0x7fb7f80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c194680 0x563c5c196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb812540640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c194140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 -- 172.21.15.16:0/3563819108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c194680 msgr2=0x563c5c196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c194680 0x563c5c196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 -- 172.21.15.16:0/3563819108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 msgr2=0x563c5c194140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb812d41640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 0x7fb7f80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c194140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 -- 172.21.15.16:0/3563819108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c5c07b440 con 0x563c5c1894a0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb812540640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c194140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb8037fe640 1 -- 172.21.15.16:0/1486189930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb8040f01d0 con 0x7fb80c106f10 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.019+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00af60 tx=0x7fb7fc00a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bd65df1be2963a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb800ff9640 1 -- 172.21.15.16:0/3563819108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc019030 con 0x563c5c1894a0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb812d41640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 0x7fb7f80a5040 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb8040bc7d0 tx=0x7fb8040876a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb800ff9640 1 -- 172.21.15.16:0/3563819108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc0110d0 con 0x563c5c1894a0 2024-09-12T03:10:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb800ff9640 1 -- 172.21.15.16:0/3563819108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00f8e0 con 0x563c5c1894a0 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 -- 172.21.15.16:0/3563819108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 msgr2=0x563c5c189880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00af60 tx=0x7fb7fc00a780 comp rx=0 tx=0).stop 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 -- 172.21.15.16:0/3563819108 shutdown_connections 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c194680 0x563c5c196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c194140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3563819108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c1894a0 0x563c5c189880 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.020+0000 7fb816b7d740 1 -- 172.21.15.16:0/3563819108 >> 172.21.15.16:0/3563819108 conn(0x563c5c185a30 msgr2=0x563c5c185e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- 172.21.15.16:0/3563819108 shutdown_connections 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- 172.21.15.16:0/3563819108 wait complete. 2024-09-12T03:10:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- start start 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 Processor -- start 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c194020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1912d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563c5c197250 con 0x563c5c194680 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563c5c0aa050 con 0x563c5c189dc0 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563c5c1973d0 con 0x563c5c1894a0 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb816b7d740 10 client.0 _ll_register_callbacks cb 0x563c5c0ea160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.021+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35758/0 (socket says 172.21.15.16:35758) 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 -- 172.21.15.16:0/3896292496 learned_addr learned my addr 172.21.15.16:0/3896292496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c194020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb812540640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 -- 172.21.15.16:0/3896292496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 msgr2=0x563c5c194020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c194020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 -- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 msgr2=0x563c5c1912d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1912d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 -- 172.21.15.16:0/3896292496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c5be603c0 con 0x563c5c194680 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00f5c0 tx=0x7fb7fc015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0eacdc9cb48766 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb7f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc019070 con 0x563c5c194680 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb7fc00f870 con 0x563c5c194680 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.022+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7fc0110d0 con 0x563c5c194680 2024-09-12T03:10:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb7fc00fa70 con 0x563c5c194680 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 10 client.20652 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563c5c1a2c00 con 0x563c5c194680 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc0110d0 con 0x563c5c194680 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb7fc01eab0 con 0x563c5c194680 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 10 client.20652 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 10 client.20652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 20 client.20652 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c1a4010 0x563c5c1a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 4 client.20652 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c5c0fc850 con 0x563c5c1a4010 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.023+0000 7fb812540640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c1a4010 0x563c5c1a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.024+0000 7fb812540640 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c1a4010 0x563c5c1a6400 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837b9564bf2deef6 server_cookie=2f6074c78da48c10 in_seq=0 out_seq=0 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.024+0000 7fb7f5ffb640 10 client.20652 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.024+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/3896292496 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c5c0fc850 con 0x563c5c1a4010 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.024+0000 7fb7f5ffb640 10 client.20652 handle_command_reply: tid=0 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.024+0000 7fb816b7d740 1 client.20652 shutdown 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 msgr2=0x563c5c191870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00f5c0 tx=0x7fb7fc015040 comp rx=0 tx=0).stop 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 shutdown_connections 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c1a4010 0x563c5c1a6400 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c191870 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1912d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3896292496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c194020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 >> 172.21.15.16:0/3896292496 conn(0x563c5c185a30 msgr2=0x563c5c18b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb7f5ffb640 0 client.20652 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 shutdown_connections 2024-09-12T03:10:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 1 -- 172.21.15.16:0/3896292496 wait complete. 2024-09-12T03:10:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.025+0000 7fb816b7d740 20 client.20652 trim_cache size 0 max 16384 2024-09-12T03:10:37.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.52 MB/sec execute 458 sec latency 14389.972 ms 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 Processor -- start 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 -- start start 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c1a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1abb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1af6a0 con 0x563c5c194680 2024-09-12T03:10:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1b11e0 con 0x563c5c189dc0 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb816b7d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1afd70 con 0x563c5c1894a0 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35760/0 (socket says 172.21.15.16:35760) 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 -- 172.21.15.16:0/3305738071 learned_addr learned my addr 172.21.15.16:0/3305738071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb812540640 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 -- 172.21.15.16:0/3305738071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 msgr2=0x563c5c1a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c1a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 -- 172.21.15.16:0/3305738071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 msgr2=0x563c5c1abb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.042+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1abb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb811d3f640 1 -- 172.21.15.16:0/3305738071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c5c189bf0 con 0x563c5c194680 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb811d3f640 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc001eb0 tx=0x7fb7fc02c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc381c853ed3af69 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb7f7fff640 1 -- 172.21.15.16:0/3305738071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc019070 con 0x563c5c194680 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb7f7fff640 1 -- 172.21.15.16:0/3305738071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc005510 con 0x563c5c194680 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb7f7fff640 1 -- 172.21.15.16:0/3305738071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc011070 con 0x563c5c194680 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 -- 172.21.15.16:0/3305738071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 msgr2=0x563c5c1ac080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc001eb0 tx=0x7fb7fc02c760 comp rx=0 tx=0).stop 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 -- 172.21.15.16:0/3305738071 shutdown_connections 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c1ac080 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c1abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 --2- 172.21.15.16:0/3305738071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.043+0000 7fb816b7d740 1 -- 172.21.15.16:0/3305738071 >> 172.21.15.16:0/3305738071 conn(0x563c5c185a30 msgr2=0x563c5c1903c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- 172.21.15.16:0/3305738071 shutdown_connections 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- 172.21.15.16:0/3305738071 wait complete. 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- start start 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 Processor -- start 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1b0b10 con 0x563c5c194680 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1b0440 con 0x563c5c189dc0 2024-09-12T03:10:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1ad680 con 0x563c5c1894a0 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb816b7d740 10 client.0 _ll_register_callbacks cb 0x7fb7fc013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb811d3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35764/0 (socket says 172.21.15.16:35764) 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb811d3f640 1 -- 172.21.15.16:0/4216641404 learned_addr learned my addr 172.21.15.16:0/4216641404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.044+0000 7fb811d3f640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb812540640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 -- 172.21.15.16:0/4216641404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 msgr2=0x563c5c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 -- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 msgr2=0x563c5c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 -- 172.21.15.16:0/4216641404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c5c194390 con 0x563c5c194680 2024-09-12T03:10:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb811d3f640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc011430 tx=0x7fb7fc02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da47f37fea730b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb7f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc019040 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb7fc00f870 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563c5c0f6d60 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb7fc0110a0 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb816b7d740 10 client.20658 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563c5c20e5f0 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.045+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563c5c0f6d60 con 0x563c5c194680 2024-09-12T03:10:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb7fc0413e0 con 0x563c5c194680 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 10 client.20658 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 10 client.20658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 20 client.20658 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c20eb50 0x563c5c210f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 4 client.20658 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:10:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c5c073210 con 0x563c5c20eb50 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.046+0000 7fb812540640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c20eb50 0x563c5c210f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb812540640 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c20eb50 0x563c5c210f40 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2932e9f8f943d6 server_cookie=86598767a62d6fc5 in_seq=0 out_seq=0 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb7f5ffb640 10 client.20658 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb7f5ffb640 1 -- 172.21.15.16:0/4216641404 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563c5c073210 con 0x563c5c20eb50 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb7f5ffb640 10 client.20658 handle_command_reply: tid=0 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 client.20658 shutdown 2024-09-12T03:10:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 msgr2=0x563c5c19a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc011430 tx=0x7fb7fc02c640 comp rx=0 tx=0).stop 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 shutdown_connections 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x563c5c20eb50 0x563c5c210f40 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb7f5ffb640 0 client.20658 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563c5c194680 0x563c5c19a040 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563c5c189dc0 0x563c5c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 --2- 172.21.15.16:0/4216641404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563c5c1894a0 0x563c5c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 >> 172.21.15.16:0/4216641404 conn(0x563c5c185a30 msgr2=0x563c5c192b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 shutdown_connections 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 1 -- 172.21.15.16:0/4216641404 wait complete. 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.047+0000 7fb816b7d740 20 client.20658 trim_cache size 0 max 16384 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:10:37.060 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:10:37.061 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 msgr2=0x7fb7f80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 0x7fb7f80a5040 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb8040bc7d0 tx=0x7fb8040876a0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 msgr2=0x7fb80c074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fb804031a20 tx=0x7fb804031a50 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 shutdown_connections 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb7f80a2b80 0x7fb7f80a5040 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb80c106f10 0x7fb80c074f70 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb80c102790 0x7fb80c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 --2- 172.21.15.16:0/1486189930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb80c101de0 0x7fb80c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 >> 172.21.15.16:0/1486189930 conn(0x7fb80c0fd8c0 msgr2=0x7fb80c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 shutdown_connections 2024-09-12T03:10:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:37.048+0000 7fb8147a4640 1 -- 172.21.15.16:0/1486189930 wait complete. 2024-09-12T03:10:37.970 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:10:37.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:10:38.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.47 MB/sec execute 459 sec latency 15390.022 ms 2024-09-12T03:10:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:38.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:38 smithi016 ceph-mon[30925]: pgmap v1018: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-12T03:10:38.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:38 smithi033 ceph-mon[35092]: pgmap v1018: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-12T03:10:38.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:38 smithi184 ceph-mon[35321]: pgmap v1018: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-09-12T03:10:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 -- 172.21.15.16:0/2310015471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1008d0 msgr2=0x7f0d3c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/2310015471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c100cb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009900 tx=0x7f0d3802a410 comp rx=0 tx=0).stop 2024-09-12T03:10:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 -- 172.21.15.16:0/2310015471 shutdown_connections 2024-09-12T03:10:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/2310015471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 0x7f0d3c110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/2310015471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c101280 0x7f0d3c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/2310015471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c100cb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.704+0000 7f0d43bf4640 1 -- 172.21.15.16:0/2310015471 >> 172.21.15.16:0/2310015471 conn(0x7f0d3c0fc7a0 msgr2=0x7f0d3c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- 172.21.15.16:0/2310015471 shutdown_connections 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- 172.21.15.16:0/2310015471 wait complete. 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 Processor -- start 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- start start 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:38.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 0x7f0d3c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c114260 con 0x7f0d3c101280 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c1140e0 con 0x7f0d3c1008d0 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d43bf4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c1143e0 con 0x7f0d3c10e480 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.705+0000 7f0d4118f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35774/0 (socket says 172.21.15.16:35774) 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 -- 172.21.15.16:0/4142083410 learned_addr learned my addr 172.21.15.16:0/4142083410 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d41990640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d42191640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 0x7f0d3c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 -- 172.21.15.16:0/4142083410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 msgr2=0x7f0d3c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 0x7f0d3c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 -- 172.21.15.16:0/4142083410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 msgr2=0x7f0d3c1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 -- 172.21.15.16:0/4142083410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c19aee0 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d41990640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d4118f640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000b4a0 tx=0x7f0d3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30014070 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d30010af0 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30017d10 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c19b1d0 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.706+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1a7520 con 0x7f0d3c101280 2024-09-12T03:10:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.707+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c06b070 con 0x7f0d3c101280 2024-09-12T03:10:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.709+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0d30017070 con 0x7f0d3c101280 2024-09-12T03:10:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.709+0000 7f0d2effd640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 0x7f0d280a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.709+0000 7f0d41990640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 0x7f0d280a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.709+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0d300ce840 con 0x7f0d3c101280 2024-09-12T03:10:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.710+0000 7f0d41990640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 0x7f0d280a5040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005940 tx=0x7f0d380058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.712+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0d3009b7b0 con 0x7f0d3c101280 2024-09-12T03:10:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.928+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d3c109610 con 0x7f0d3c101280 2024-09-12T03:10:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.929+0000 7f0d2effd640 1 -- 172.21.15.16:0/4142083410 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f0d3009b0b0 con 0x7f0d3c101280 2024-09-12T03:10:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:10:38.931 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:10:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:10:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 msgr2=0x7f0d280a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 0x7f0d280a5040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005940 tx=0x7f0d380058b0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 msgr2=0x7f0d3c1a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000b4a0 tx=0x7f0d3000b960 comp rx=0 tx=0).stop 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 shutdown_connections 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d3c10e480 0x7f0d3c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f0d280a2b80 0x7f0d280a5040 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c101280 0x7f0d3c1a0ae0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 --2- 172.21.15.16:0/4142083410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d3c1008d0 0x7f0d3c1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 >> 172.21.15.16:0/4142083410 conn(0x7f0d3c0fc7a0 msgr2=0x7f0d3c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 shutdown_connections 2024-09-12T03:10:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:38.931+0000 7f0d43bf4640 1 -- 172.21.15.16:0/4142083410 wait complete. 2024-09-12T03:10:39.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.42 MB/sec execute 460 sec latency 16390.125 ms 2024-09-12T03:10:39.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:39 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4142083410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:39.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:39 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4142083410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:39.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:39 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4142083410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:39.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:10:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:40.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.37 MB/sec execute 461 sec latency 17390.169 ms 2024-09-12T03:10:40.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:40 smithi016 ceph-mon[30925]: pgmap v1019: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 251 KiB/s wr, 25 op/s 2024-09-12T03:10:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 -- 172.21.15.16:0/2392296862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241082a0 msgr2=0x7ff82410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 --2- 172.21.15.16:0/2392296862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241082a0 0x7ff82410ec40 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009cd0 tx=0x7ff81c02a8e0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 -- 172.21.15.16:0/2392296862 shutdown_connections 2024-09-12T03:10:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 --2- 172.21.15.16:0/2392296862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241082a0 0x7ff82410ec40 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 --2- 172.21.15.16:0/2392296862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8241036e0 0x7ff824107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 --2- 172.21.15.16:0/2392296862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff824102d30 0x7ff824103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 -- 172.21.15.16:0/2392296862 >> 172.21.15.16:0/2392296862 conn(0x7ff8240fc7d0 msgr2=0x7ff8240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 -- 172.21.15.16:0/2392296862 shutdown_connections 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.322+0000 7ff82bd92640 1 -- 172.21.15.16:0/2392296862 wait complete. 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 Processor -- start 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 -- start start 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff824102d30 0x7ff824100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8241082a0 0x7ff824101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241120a0 con 0x7ff8241036e0 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff824111f20 con 0x7ff824102d30 2024-09-12T03:10:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82bd92640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff824112220 con 0x7ff8241082a0 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35798/0 (socket says 172.21.15.16:35798) 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82932d640 1 -- 172.21.15.16:0/1953928923 learned_addr learned my addr 172.21.15.16:0/1953928923 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff829b2e640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff824102d30 0x7ff824100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.323+0000 7ff82a32f640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8241082a0 0x7ff824101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 -- 172.21.15.16:0/1953928923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8241082a0 msgr2=0x7ff824101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8241082a0 0x7ff824101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 -- 172.21.15.16:0/1953928923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff824102d30 msgr2=0x7ff824100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff824102d30 0x7ff824100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 -- 172.21.15.16:0/1953928923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8241a6990 con 0x7ff8241036e0 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82932d640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff81400d740 tx=0x7ff81400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff81affd640 1 -- 172.21.15.16:0/1953928923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814017070 con 0x7ff8241036e0 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff81affd640 1 -- 172.21.15.16:0/1953928923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff814010460 con 0x7ff8241036e0 2024-09-12T03:10:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8241a6c80 con 0x7ff8241036e0 2024-09-12T03:10:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff81affd640 1 -- 172.21.15.16:0/1953928923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81400f620 con 0x7ff8241036e0 2024-09-12T03:10:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.324+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8241a71c0 con 0x7ff8241036e0 2024-09-12T03:10:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.326+0000 7ff82e16b740 1 Processor -- start 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.326+0000 7ff81affd640 1 -- 172.21.15.16:0/1953928923 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff81401a070 con 0x7ff8241036e0 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 -- start start 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd15760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563e5cc6e2f0 con 0x563e5cd09b40 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563e5cc15cf0 con 0x563e5cd085d0 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82e16b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563e5cc6a2b0 con 0x563e5cd13370 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35812/0 (socket says 172.21.15.16:35812) 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 -- 172.21.15.16:0/2269403727 learned_addr learned my addr 172.21.15.16:0/2269403727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff829b2e640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 -- 172.21.15.16:0/2269403727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 msgr2=0x563e5cd15760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82932d640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd15760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff81affd640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 0x7ff8100a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd15760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 -- 172.21.15.16:0/2269403727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 msgr2=0x563e5cd08a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd08a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82a32f640 1 -- 172.21.15.16:0/2269403727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5cbfa3a0 con 0x563e5cd09b40 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.327+0000 7ff82932d640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd15760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff829b2e640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff81affd640 1 -- 172.21.15.16:0/1953928923 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8140ce870 con 0x7ff8241036e0 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff82a32f640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 0x7ff8100a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff82a32f640 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff81c035000 tx=0x7ff81c031570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11b3a6c20c6ea2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff80ffff640 1 -- 172.21.15.16:0/2269403727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c031bc0 con 0x563e5cd09b40 2024-09-12T03:10:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff80ffff640 1 -- 172.21.15.16:0/2269403727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81c036070 con 0x563e5cd09b40 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.328+0000 7ff80ffff640 1 -- 172.21.15.16:0/2269403727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c030d80 con 0x563e5cd09b40 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- 172.21.15.16:0/2269403727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 msgr2=0x563e5cd08090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82a32f640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 0x7ff8100a50c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff81c049000 tx=0x7ff81c04a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff81c035000 tx=0x7ff81c031570 comp rx=0 tx=0).stop 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- 172.21.15.16:0/2269403727 shutdown_connections 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd15760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2269403727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd08090 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- 172.21.15.16:0/2269403727 >> 172.21.15.16:0/2269403727 conn(0x563e5ccd1310 msgr2=0x563e5cd04c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- 172.21.15.16:0/2269403727 shutdown_connections 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- 172.21.15.16:0/2269403727 wait complete. 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 -- start start 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.329+0000 7ff82e16b740 1 Processor -- start 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd12cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd104e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563e5cd16070 con 0x563e5cd09b40 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563e5cc29060 con 0x563e5cd085d0 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563e5cd161f0 con 0x563e5cd13370 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82e16b740 10 client.0 _ll_register_callbacks cb 0x563e5c97c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35828/0 (socket says 172.21.15.16:35828) 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 -- 172.21.15.16:0/2235974947 learned_addr learned my addr 172.21.15.16:0/2235974947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff829b2e640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd12cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82a32f640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 -- 172.21.15.16:0/2235974947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 msgr2=0x563e5cd104e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd104e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 -- 172.21.15.16:0/2235974947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 msgr2=0x563e5cd12cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd12cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82932d640 1 -- 172.21.15.16:0/2235974947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5c9df3c0 con 0x563e5cd09b40 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.330+0000 7ff82a32f640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82932d640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff8140cc380 tx=0x7ff814070000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82100c484eeeb5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814021050 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff814010610 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff81c036070 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff8140d1e60 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 10 client.20676 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563e5cd21a20 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81c036070 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff814097750 con 0x563e5cd09b40 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 10 client.20676 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 10 client.20676 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:10:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 20 client.20676 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.331+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd22ee0 0x563e5cd252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.332+0000 7ff82e16b740 4 client.20676 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.332+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e5cc7b860 con 0x563e5cd22ee0 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.332+0000 7ff829b2e640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd22ee0 0x563e5cd252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.332+0000 7ff80d7fa640 10 client.20676 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.332+0000 7ff829b2e640 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd22ee0 0x563e5cd252d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4d9c7c2102c21abf server_cookie=140f743934f15b5a in_seq=0 out_seq=0 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff80d7fa640 1 -- 172.21.15.16:0/2235974947 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e5cc7b860 con 0x563e5cd22ee0 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff80d7fa640 10 client.20676 handle_command_reply: tid=0 2024-09-12T03:10:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 client.20676 shutdown 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 msgr2=0x563e5cd0ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff8140cc380 tx=0x7ff814070000 comp rx=0 tx=0).stop 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 shutdown_connections 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0ff80 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd22ee0 0x563e5cd252d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 --2- 172.21.15.16:0/2235974947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd12cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff80d7fa640 0 client.20676 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:10:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 >> 172.21.15.16:0/2235974947 conn(0x563e5ccd1310 msgr2=0x563e5cd0a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 shutdown_connections 2024-09-12T03:10:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 1 -- 172.21.15.16:0/2235974947 wait complete. 2024-09-12T03:10:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.333+0000 7ff82e16b740 20 client.20676 trim_cache size 0 max 16384 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 Processor -- start 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 -- start start 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd1f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd2ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241af6a0 con 0x563e5cd09b40 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241b11e0 con 0x563e5cd085d0 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82e16b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241afd70 con 0x563e5cd13370 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.346+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35834/0 (socket says 172.21.15.16:35834) 2024-09-12T03:10:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 -- 172.21.15.16:0/1798968063 learned_addr learned my addr 172.21.15.16:0/1798968063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff829b2e640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd1f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82a32f640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd2ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 -- 172.21.15.16:0/1798968063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 msgr2=0x563e5cd2ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd2ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 -- 172.21.15.16:0/1798968063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 msgr2=0x563e5cd1f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd1f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 -- 172.21.15.16:0/1798968063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5cd0b120 con 0x563e5cd09b40 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82a32f640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd2ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff82932d640 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff8140a3da0 tx=0x7ff8140704a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef415fb49ce2cf3a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff80f7fe640 1 -- 172.21.15.16:0/1798968063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814021070 con 0x563e5cd09b40 2024-09-12T03:10:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff80f7fe640 1 -- 172.21.15.16:0/1798968063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81409b7e0 con 0x563e5cd09b40 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.347+0000 7ff80f7fe640 1 -- 172.21.15.16:0/1798968063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81409f0c0 con 0x563e5cd09b40 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- 172.21.15.16:0/1798968063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 msgr2=0x563e5cd0f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff8140a3da0 tx=0x7ff8140704a0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- 172.21.15.16:0/1798968063 shutdown_connections 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd13370 0x563e5cd2ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd09b40 0x563e5cd0f840 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 --2- 172.21.15.16:0/1798968063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd085d0 0x563e5cd1f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- 172.21.15.16:0/1798968063 >> 172.21.15.16:0/1798968063 conn(0x563e5ccd1310 msgr2=0x563e5cd0e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- 172.21.15.16:0/1798968063 shutdown_connections 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- 172.21.15.16:0/1798968063 wait complete. 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 -- start start 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.348+0000 7ff82e16b740 1 Processor -- start 2024-09-12T03:10:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd09b40 0x563e5cd177d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd13370 0x563e5cd17d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241b0b10 con 0x563e5cd085d0 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241b0440 con 0x563e5cd13370 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241ad680 con 0x563e5cd09b40 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82e16b740 10 client.0 _ll_register_callbacks cb 0x7ff814009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35846/0 (socket says 172.21.15.16:35846) 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 -- 172.21.15.16:0/608200274 learned_addr learned my addr 172.21.15.16:0/608200274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82932d640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd09b40 0x563e5cd177d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff82a32f640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd13370 0x563e5cd17d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 -- 172.21.15.16:0/608200274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd09b40 msgr2=0x563e5cd177d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd09b40 0x563e5cd177d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 -- 172.21.15.16:0/608200274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd13370 msgr2=0x563e5cd17d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd13370 0x563e5cd17d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 -- 172.21.15.16:0/608200274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5cd8cbc0 con 0x563e5cd085d0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff829b2e640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x563e5cd25ae0 tx=0x7ff820010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f69269ee35dff server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.349+0000 7ff80d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820005220 con 0x563e5cd085d0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff8200053c0 con 0x563e5cd085d0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff81409b7e0 con 0x563e5cd085d0 2024-09-12T03:10:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff8200055c0 con 0x563e5cd085d0 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 10 client.20682 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563e5cd8de70 con 0x563e5cd085d0 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81409b7e0 con 0x563e5cd085d0 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff82001dba0 con 0x563e5cd085d0 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 10 client.20682 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 10 client.20682 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 20 client.20682 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd8e5e0 0x563e5cd909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 4 client.20682 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e5cc75d70 con 0x563e5cd8e5e0 2024-09-12T03:10:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.350+0000 7ff82932d640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd8e5e0 0x563e5cd909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff80d7fa640 10 client.20682 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82932d640 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd8e5e0 0x563e5cd909d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d7977002d030e20 server_cookie=c7cedd7e1d26dddd in_seq=0 out_seq=0 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff80d7fa640 1 -- 172.21.15.16:0/608200274 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e5cc75d70 con 0x563e5cd8e5e0 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff80d7fa640 10 client.20682 handle_command_reply: tid=0 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 client.20682 shutdown 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 msgr2=0x563e5cd17290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x563e5cd25ae0 tx=0x7ff820010af0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 shutdown_connections 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x563e5cd8e5e0 0x563e5cd909d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563e5cd13370 0x563e5cd17d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff80d7fa640 0 client.20682 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563e5cd09b40 0x563e5cd177d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 --2- 172.21.15.16:0/608200274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563e5cd085d0 0x563e5cd17290 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 >> 172.21.15.16:0/608200274 conn(0x563e5ccd1310 msgr2=0x563e5cd034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 shutdown_connections 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 1 -- 172.21.15.16:0/608200274 wait complete. 2024-09-12T03:10:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.351+0000 7ff82e16b740 20 client.20682 trim_cache size 0 max 16384 2024-09-12T03:10:40.362 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 msgr2=0x7ff8100a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 0x7ff8100a50c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff81c049000 tx=0x7ff81c04a000 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 msgr2=0x7ff824100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff81400d740 tx=0x7ff81400db40 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 shutdown_connections 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8241082a0 0x7ff824101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8100a2c00 0x7ff8100a50c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8241036e0 0x7ff824100ed0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 --2- 172.21.15.16:0/1953928923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff824102d30 0x7ff824100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 >> 172.21.15.16:0/1953928923 conn(0x7ff8240fc7d0 msgr2=0x7ff824106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 shutdown_connections 2024-09-12T03:10:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:40.352+0000 7ff82bd92640 1 -- 172.21.15.16:0/1953928923 wait complete. 2024-09-12T03:10:40.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:40 smithi033 ceph-mon[35092]: pgmap v1019: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 251 KiB/s wr, 25 op/s 2024-09-12T03:10:40.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:40 smithi184 ceph-mon[35321]: pgmap v1019: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 251 KiB/s wr, 25 op/s 2024-09-12T03:10:41.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.32 MB/sec execute 462 sec latency 18390.236 ms 2024-09-12T03:10:41.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:10:41.320 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.781+0000 7f058e846640 1 -- 172.21.15.16:0/3062979527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 msgr2=0x7f05881021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.781+0000 7f058e846640 1 --2- 172.21.15.16:0/3062979527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881021c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f0578009900 tx=0x7f057802a410 comp rx=0 tx=0).stop 2024-09-12T03:10:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 -- 172.21.15.16:0/3062979527 shutdown_connections 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 --2- 172.21.15.16:0/3062979527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588106f10 0x7f058810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 --2- 172.21.15.16:0/3062979527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588102790 0x7f05881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 --2- 172.21.15.16:0/3062979527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881021c0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 -- 172.21.15.16:0/3062979527 >> 172.21.15.16:0/3062979527 conn(0x7f05880fd8c0 msgr2=0x7f05880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 -- 172.21.15.16:0/3062979527 shutdown_connections 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 -- 172.21.15.16:0/3062979527 wait complete. 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 Processor -- start 2024-09-12T03:10:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.782+0000 7f058e846640 1 -- start start 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588102790 0x7f05881989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588106f10 0x7f058819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0588111f70 con 0x7f0588101de0 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0588111df0 con 0x7f0588102790 2024-09-12T03:10:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058e846640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05881120f0 con 0x7f0588106f10 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35866/0 (socket says 172.21.15.16:35866) 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f0587fff640 1 -- 172.21.15.16:0/2236702238 learned_addr learned my addr 172.21.15.16:0/2236702238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f058cde3640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588106f10 0x7f058819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f05877fe640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588102790 0x7f05881989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.783+0000 7f0587fff640 1 -- 172.21.15.16:0/2236702238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588106f10 msgr2=0x7f058819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f0587fff640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588106f10 0x7f058819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f0587fff640 1 -- 172.21.15.16:0/2236702238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588102790 msgr2=0x7f05881989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f0587fff640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588102790 0x7f05881989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f0587fff640 1 -- 172.21.15.16:0/2236702238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058819d460 con 0x7f0588101de0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f0587fff640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0578006010 tx=0x7f05780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f05857fa640 1 -- 172.21.15.16:0/2236702238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057803f030 con 0x7f0588101de0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058819d6f0 con 0x7f0588101de0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f05857fa640 1 -- 172.21.15.16:0/2236702238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0578003ce0 con 0x7f0588101de0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f05857fa640 1 -- 172.21.15.16:0/2236702238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578035070 con 0x7f0588101de0 2024-09-12T03:10:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.784+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05881a5030 con 0x7f0588101de0 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.786+0000 7f05857fa640 1 -- 172.21.15.16:0/2236702238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0578042030 con 0x7f0588101de0 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 Processor -- start 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 -- start start 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f523090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f5235d0 0x55ca8f523a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 0x55ca8f530760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f4892f0 con 0x55ca8f524b40 2024-09-12T03:10:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f430cf0 con 0x55ca8f52e370 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0590c1f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f4852b0 con 0x55ca8f5235d0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f058cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f523090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f058cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f523090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35878/0 (socket says 172.21.15.16:35878) 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f058cde3640 1 -- 172.21.15.16:0/499209655 learned_addr learned my addr 172.21.15.16:0/499209655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f05857fa640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 0x7f05700a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f05877fe640 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 0x55ca8f530760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f0587fff640 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f5235d0 0x55ca8f523a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f058cde3640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 0x7f05700a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.787+0000 7f05857fa640 1 -- 172.21.15.16:0/2236702238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f05780ef610 con 0x7f0588101de0 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 -- 172.21.15.16:0/499209655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f5235d0 msgr2=0x55ca8f523a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f5235d0 0x55ca8f523a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 -- 172.21.15.16:0/499209655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 msgr2=0x55ca8f523090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f523090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 -- 172.21.15.16:0/499209655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8f4153a0 con 0x55ca8f52e370 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f05877fe640 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 0x55ca8f530760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f057c005fd0 tx=0x7f057c00bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e368dcbc655d89e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f058cde3640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 0x7f05700a5040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0580005f40 tx=0x7f0580004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f056e7fc640 1 -- 172.21.15.16:0/499209655 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c003bd0 con 0x55ca8f52e370 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f056e7fc640 1 -- 172.21.15.16:0/499209655 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f057c015070 con 0x55ca8f52e370 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.788+0000 7f056e7fc640 1 -- 172.21.15.16:0/499209655 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c004c60 con 0x55ca8f52e370 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- 172.21.15.16:0/499209655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 msgr2=0x55ca8f530760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 0x55ca8f530760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f057c005fd0 tx=0x7f057c00bb20 comp rx=0 tx=0).stop 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- 172.21.15.16:0/499209655 shutdown_connections 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f52e370 0x55ca8f530760 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f5235d0 0x55ca8f523a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 --2- 172.21.15.16:0/499209655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f523090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- 172.21.15.16:0/499209655 >> 172.21.15.16:0/499209655 conn(0x55ca8f4ec310 msgr2=0x55ca8f51fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- 172.21.15.16:0/499209655 shutdown_connections 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- 172.21.15.16:0/499209655 wait complete. 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.789+0000 7f0590c1f740 1 -- start start 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 Processor -- start 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f531070 con 0x55ca8f5235d0 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f444060 con 0x55ca8f524b40 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ca8f5311f0 con 0x55ca8f52e370 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0590c1f740 10 client.0 _ll_register_callbacks cb 0x55ca8f1978c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35890/0 (socket says 172.21.15.16:35890) 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 -- 172.21.15.16:0/3549990530 learned_addr learned my addr 172.21.15.16:0/3549990530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f05877fe640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f058cde3640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 -- 172.21.15.16:0/3549990530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 msgr2=0x55ca8f52b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 -- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 msgr2=0x55ca8f52afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 -- 172.21.15.16:0/3549990530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8f1fa3c0 con 0x55ca8f5235d0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f0587fff640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f05780bbf20 tx=0x7f0578009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bffc12e57efeb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.790+0000 7f05677fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578049070 con 0x55ca8f5235d0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f05780bc1f0 con 0x55ca8f5235d0 2024-09-12T03:10:41.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f057c015070 con 0x55ca8f5235d0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f05780bb8f0 con 0x55ca8f5235d0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 10 client.20694 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ca8f53cad0 con 0x55ca8f5235d0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f057c015070 con 0x55ca8f5235d0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f05780c46c0 con 0x55ca8f5235d0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 10 client.20694 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 10 client.20694 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 20 client.20694 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f53de70 0x55ca8f5402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 4 client.20694 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca8f496860 con 0x55ca8f53de70 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.791+0000 7f05877fe640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f53de70 0x55ca8f5402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.792+0000 7f05677fe640 10 client.20694 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:10:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.792+0000 7f05877fe640 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f53de70 0x55ca8f5402a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=42560ae2b472579d server_cookie=fb01b8026fac4bd4 in_seq=0 out_seq=0 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f05677fe640 1 -- 172.21.15.16:0/3549990530 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca8f496860 con 0x55ca8f53de70 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f05677fe640 10 client.20694 handle_command_reply: tid=0 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 client.20694 shutdown 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 msgr2=0x55ca8f52dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f05780bbf20 tx=0x7f0578009560 comp rx=0 tx=0).stop 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 shutdown_connections 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f53de70 0x55ca8f5402a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f05677fe640 0 client.20694 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:10:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3549990530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f52dd30 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 >> 172.21.15.16:0/3549990530 conn(0x55ca8f4ec310 msgr2=0x55ca8f525960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 shutdown_connections 2024-09-12T03:10:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 1 -- 172.21.15.16:0/3549990530 wait complete. 2024-09-12T03:10:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.793+0000 7f0590c1f740 20 client.20694 trim_cache size 0 max 16384 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 Processor -- start 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 -- start start 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05881ad480 con 0x55ca8f5235d0 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05881aefc0 con 0x55ca8f524b40 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0590c1f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05881adb50 con 0x55ca8f52e370 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0587fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35904/0 (socket says 172.21.15.16:35904) 2024-09-12T03:10:41.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0587fff640 1 -- 172.21.15.16:0/1452418004 learned_addr learned my addr 172.21.15.16:0/1452418004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f05877fe640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f058cde3640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0587fff640 1 -- 172.21.15.16:0/1452418004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 msgr2=0x55ca8f52fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.806+0000 7f0587fff640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0587fff640 1 -- 172.21.15.16:0/1452418004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 msgr2=0x55ca8f52f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0587fff640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0587fff640 1 -- 172.21.15.16:0/1452418004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8f51fed0 con 0x55ca8f5235d0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f05877fe640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0587fff640 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f05780c4620 tx=0x7f05780ef1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a147c9ea5d8057 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f056dffb640 1 -- 172.21.15.16:0/1452418004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578049070 con 0x55ca8f5235d0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f056dffb640 1 -- 172.21.15.16:0/1452418004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05780ef3f0 con 0x55ca8f5235d0 2024-09-12T03:10:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f056dffb640 1 -- 172.21.15.16:0/1452418004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057803bcb0 con 0x55ca8f5235d0 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0590c1f740 1 -- 172.21.15.16:0/1452418004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 msgr2=0x55ca8f53a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0590c1f740 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f05780c4620 tx=0x7f05780ef1e0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0590c1f740 1 -- 172.21.15.16:0/1452418004 shutdown_connections 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0590c1f740 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f52fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.807+0000 7f0590c1f740 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f524b40 0x55ca8f52f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 --2- 172.21.15.16:0/1452418004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f5235d0 0x55ca8f53a310 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- 172.21.15.16:0/1452418004 >> 172.21.15.16:0/1452418004 conn(0x55ca8f4ec310 msgr2=0x55ca8f5298b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- 172.21.15.16:0/1452418004 shutdown_connections 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- 172.21.15.16:0/1452418004 wait complete. 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- start start 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 Processor -- start 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 0x55ca8f536ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f545900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05881ae8f0 con 0x55ca8f524b40 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05881ae220 con 0x55ca8f5235d0 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05881ab460 con 0x55ca8f52e370 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f0590c1f740 10 client.0 _ll_register_callbacks cb 0x7f057803b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.808+0000 7f05877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35906/0 (socket says 172.21.15.16:35906) 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 -- 172.21.15.16:0/3452017898 learned_addr learned my addr 172.21.15.16:0/3452017898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f0587fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 0x55ca8f536ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f058cde3640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f545900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 -- 172.21.15.16:0/3452017898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 msgr2=0x55ca8f545900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f545900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 -- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 msgr2=0x55ca8f536ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 0x55ca8f536ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 -- 172.21.15.16:0/3452017898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8f546040 con 0x55ca8f524b40 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f058cde3640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f545900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f0587fff640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 0x55ca8f536ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05877fe640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f057c004580 tx=0x7f057c00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a660774b74f0d62d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05677fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c034060 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f057c003af0 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca8f490d70 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.809+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f057c003d30 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 10 client.20700 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ca8f5a8b50 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca8f490d70 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f057c0114f0 con 0x55ca8f524b40 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 10 client.20700 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 10 client.20700 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 20 client.20700 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f5a8f30 0x55ca8f5ab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:41.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 4 client.20700 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca8f40d170 con 0x55ca8f5a8f30 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.810+0000 7f0587fff640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f5a8f30 0x55ca8f5ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f05677fe640 10 client.20700 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0587fff640 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f5a8f30 0x55ca8f5ab320 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae11396b215f19ab server_cookie=f77bd830e9716128 in_seq=0 out_seq=0 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f05677fe640 1 -- 172.21.15.16:0/3452017898 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca8f40d170 con 0x55ca8f5a8f30 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f05677fe640 10 client.20700 handle_command_reply: tid=0 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 client.20700 shutdown 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 msgr2=0x55ca8f5453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f057c004580 tx=0x7f057c00be90 comp rx=0 tx=0).stop 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 shutdown_connections 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ca8f5a8f30 0x55ca8f5ab320 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ca8f52e370 0x55ca8f545900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca8f524b40 0x55ca8f5453c0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 --2- 172.21.15.16:0/3452017898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ca8f5235d0 0x55ca8f536ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 >> 172.21.15.16:0/3452017898 conn(0x55ca8f4ec310 msgr2=0x55ca8f525110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f05677fe640 0 client.20700 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 shutdown_connections 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 1 -- 172.21.15.16:0/3452017898 wait complete. 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.811+0000 7f0590c1f740 20 client.20700 trim_cache size 0 max 16384 2024-09-12T03:10:41.818 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 msgr2=0x7f05700a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 0x7f05700a5040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0580005f40 tx=0x7f0580004ab0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 msgr2=0x7f05881984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0578006010 tx=0x7f05780042f0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 shutdown_connections 2024-09-12T03:10:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0588106f10 0x7f058819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0588102790 0x7f05881989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f05700a2b80 0x7f05700a5040 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 --2- 172.21.15.16:0/2236702238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0588101de0 0x7f05881984b0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 >> 172.21.15.16:0/2236702238 conn(0x7f05880fd8c0 msgr2=0x7f05880fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 shutdown_connections 2024-09-12T03:10:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:41.812+0000 7f058e846640 1 -- 172.21.15.16:0/2236702238 wait complete. 2024-09-12T03:10:42.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.27 MB/sec execute 463 sec latency 19390.321 ms 2024-09-12T03:10:42.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:42 smithi016 ceph-mon[30925]: pgmap v1020: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-09-12T03:10:42.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:42 smithi033 ceph-mon[35092]: pgmap v1020: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-09-12T03:10:42.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:10:42.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:42 smithi184 ceph-mon[35321]: pgmap v1020: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 33 KiB/s wr, 25 op/s 2024-09-12T03:10:42.776 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:43.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.23 MB/sec execute 464 sec latency 20390.366 ms 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 -- 172.21.15.16:0/1177319592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 msgr2=0x7ffadc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 --2- 172.21.15.16:0/1177319592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc10ea90 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7ffad4009ca0 tx=0x7ffad402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 -- 172.21.15.16:0/1177319592 shutdown_connections 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 --2- 172.21.15.16:0/1177319592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc10ea90 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 --2- 172.21.15.16:0/1177319592 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc102790 0x7ffadc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 --2- 172.21.15.16:0/1177319592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc101de0 0x7ffadc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.252+0000 7ffae14b5640 1 -- 172.21.15.16:0/1177319592 >> 172.21.15.16:0/1177319592 conn(0x7ffadc0fd8c0 msgr2=0x7ffadc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.253+0000 7ffae14b5640 1 -- 172.21.15.16:0/1177319592 shutdown_connections 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.253+0000 7ffae14b5640 1 -- 172.21.15.16:0/1177319592 wait complete. 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.253+0000 7ffae14b5640 1 Processor -- start 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.253+0000 7ffae14b5640 1 -- start start 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc101de0 0x7ffadc0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc102790 0x7ffadc074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc111f70 con 0x7ffadc106f10 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc111df0 con 0x7ffadc101de0 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1120f0 con 0x7ffadc102790 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35928/0 (socket says 172.21.15.16:35928) 2024-09-12T03:10:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 -- 172.21.15.16:0/2327081675 learned_addr learned my addr 172.21.15.16:0/2327081675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc102790 0x7ffadc074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadaffd640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc101de0 0x7ffadc0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 -- 172.21.15.16:0/2327081675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc102790 msgr2=0x7ffadc074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc102790 0x7ffadc074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 -- 172.21.15.16:0/2327081675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc101de0 msgr2=0x7ffadc0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc101de0 0x7ffadc0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 -- 172.21.15.16:0/2327081675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffadc1a6870 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffadb7fe640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ffad4031a20 tx=0x7ffad4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffac3fff640 1 -- 172.21.15.16:0/2327081675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4003c30 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffadc1a6b00 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.254+0000 7ffac3fff640 1 -- 172.21.15.16:0/2327081675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad4003dd0 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.255+0000 7ffac3fff640 1 -- 172.21.15.16:0/2327081675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad402fc40 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.255+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffadc1a7040 con 0x7ffadc106f10 2024-09-12T03:10:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffac3fff640 1 -- 172.21.15.16:0/2327081675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ffad4004050 con 0x7ffadc106f10 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffae388e740 1 Processor -- start 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffae388e740 1 -- start start 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffae388e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d64090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffae388e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.257+0000 7ffae388e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d71760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffae388e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f73cca2f0 con 0x561f73d645d0 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffae388e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f73c71cf0 con 0x561f73d65b40 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffae388e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f73cc62b0 con 0x561f73d6f370 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35936/0 (socket says 172.21.15.16:35936) 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 -- 172.21.15.16:0/1756145431 learned_addr learned my addr 172.21.15.16:0/1756145431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffac3fff640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 0x7ffac40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffada7fc640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d64090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffada7fc640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d71760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadb7fe640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 0x7ffac40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 -- 172.21.15.16:0/1756145431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 msgr2=0x561f73d71760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d71760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 -- 172.21.15.16:0/1756145431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 msgr2=0x561f73d64090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d64090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 -- 172.21.15.16:0/1756145431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f73c563a0 con 0x561f73d645d0 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffac3fff640 1 -- 172.21.15.16:0/2327081675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffad40f0220 con 0x7ffadc106f10 2024-09-12T03:10:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffada7fc640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d71760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadaffd640 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00bcd0 tx=0x7ffacc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ae0f76aa3c2068 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffac17fa640 1 -- 172.21.15.16:0/1756145431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc005220 con 0x561f73d645d0 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.258+0000 7ffadb7fe640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 0x7ffac40a5040 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ffad40bc8e0 tx=0x7ffad402f710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffac17fa640 1 -- 172.21.15.16:0/1756145431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffacc004c80 con 0x561f73d645d0 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffac17fa640 1 -- 172.21.15.16:0/1756145431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0117a0 con 0x561f73d645d0 2024-09-12T03:10:43.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 -- 172.21.15.16:0/1756145431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 msgr2=0x561f73d64a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00bcd0 tx=0x7ffacc0107a0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 -- 172.21.15.16:0/1756145431 shutdown_connections 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d71760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d645d0 0x561f73d64a10 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 --2- 172.21.15.16:0/1756145431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 -- 172.21.15.16:0/1756145431 >> 172.21.15.16:0/1756145431 conn(0x561f73d2d310 msgr2=0x561f73d60c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 -- 172.21.15.16:0/1756145431 shutdown_connections 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.259+0000 7ffae388e740 1 -- 172.21.15.16:0/1756145431 wait complete. 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 -- start start 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 Processor -- start 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d6ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d6bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f73d72070 con 0x561f73d6f370 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f73c85060 con 0x561f73d65b40 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f73d721f0 con 0x561f73d645d0 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffae388e740 10 client.0 _ll_register_callbacks cb 0x561f739d88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35950/0 (socket says 172.21.15.16:35950) 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 -- 172.21.15.16:0/4177445237 learned_addr learned my addr 172.21.15.16:0/4177445237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d6bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d6ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 -- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 msgr2=0x561f73d6ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d6ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 -- 172.21.15.16:0/4177445237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 msgr2=0x561f73d6bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d6bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.260+0000 7ffada7fc640 1 -- 172.21.15.16:0/4177445237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f73a3b3c0 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ffad0009ef0 tx=0x7ffad0009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bab4329e3040c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0020430 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffad001f3d0 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffacc004c80 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffad00205d0 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 10 client.20712 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561f73d7db40 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffacc004c80 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ffad001f9f0 con 0x561f73d6f370 2024-09-12T03:10:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 10 client.20712 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 10 client.20712 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 20 client.20712 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73d7ef50 0x561f73d81340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 4 client.20712 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.261+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f73cd7860 con 0x561f73d7ef50 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.262+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73d7ef50 0x561f73d81340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.262+0000 7ffabeff5640 10 client.20712 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.262+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73d7ef50 0x561f73d81340 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa3709ff7e0a7526 server_cookie=17966cf0cb3b2a83 in_seq=0 out_seq=0 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffabeff5640 1 -- 172.21.15.16:0/4177445237 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f73cd7860 con 0x561f73d7ef50 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffabeff5640 10 client.20712 handle_command_reply: tid=0 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 client.20712 shutdown 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 msgr2=0x561f73d6c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ffad0009ef0 tx=0x7ffad0009a40 comp rx=0 tx=0).stop 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 shutdown_connections 2024-09-12T03:10:43.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d6c5e0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d6bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73d7ef50 0x561f73d81340 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 --2- 172.21.15.16:0/4177445237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d6ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 >> 172.21.15.16:0/4177445237 conn(0x561f73d2d310 msgr2=0x561f73d66930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffabeff5640 0 client.20712 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 shutdown_connections 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 1 -- 172.21.15.16:0/4177445237 wait complete. 2024-09-12T03:10:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.263+0000 7ffae388e740 20 client.20712 trim_cache size 0 max 16384 2024-09-12T03:10:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 Processor -- start 2024-09-12T03:10:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 -- start start 2024-09-12T03:10:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d60e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d7b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1af730 con 0x561f73d6f370 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1b1270 con 0x561f73d65b40 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffae388e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1afe00 con 0x561f73d645d0 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35956/0 (socket says 172.21.15.16:35956) 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 -- 172.21.15.16:0/4270649140 learned_addr learned my addr 172.21.15.16:0/4270649140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d60e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d7b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 -- 172.21.15.16:0/4270649140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 msgr2=0x561f73d60e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d60e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 -- 172.21.15.16:0/4270649140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 msgr2=0x561f73d7b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d7b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 -- 172.21.15.16:0/4270649140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f73d6e960 con 0x561f73d6f370 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.277+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ffad00097c0 tx=0x7ffad0005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354fdaf6462228bd server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffac0ff9640 1 -- 172.21.15.16:0/4270649140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0030070 con 0x561f73d6f370 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffac0ff9640 1 -- 172.21.15.16:0/4270649140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad0026030 con 0x561f73d6f370 2024-09-12T03:10:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffac0ff9640 1 -- 172.21.15.16:0/4270649140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0020770 con 0x561f73d6f370 2024-09-12T03:10:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 -- 172.21.15.16:0/4270649140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 msgr2=0x561f73d67120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ffad00097c0 tx=0x7ffad0005970 comp rx=0 tx=0).stop 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 -- 172.21.15.16:0/4270649140 shutdown_connections 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d6f370 0x561f73d67120 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d65b40 0x561f73d7b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 --2- 172.21.15.16:0/4270649140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d645d0 0x561f73d60e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 -- 172.21.15.16:0/4270649140 >> 172.21.15.16:0/4270649140 conn(0x561f73d2d310 msgr2=0x561f73d60700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 -- 172.21.15.16:0/4270649140 shutdown_connections 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.278+0000 7ffae388e740 1 -- 172.21.15.16:0/4270649140 wait complete. 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 -- start start 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 Processor -- start 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d645d0 0x561f73d776f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d689e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1b0ba0 con 0x561f73d65b40 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1b04d0 con 0x561f73d645d0 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc1ad710 con 0x561f73d6f370 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffae388e740 10 client.0 _ll_register_callbacks cb 0x7ffad00119f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35962/0 (socket says 172.21.15.16:35962) 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffadaffd640 1 -- 172.21.15.16:0/2935178476 learned_addr learned my addr 172.21.15.16:0/2935178476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.279+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d645d0 0x561f73d776f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 -- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 msgr2=0x561f73d689e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d689e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 -- 172.21.15.16:0/2935178476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d645d0 msgr2=0x561f73d776f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d645d0 0x561f73d776f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 -- 172.21.15.16:0/2935178476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f73d69210 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffadaffd640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ffacc006010 tx=0x7ffacc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86f8653a1fd3ec8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffabeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc00b810 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ffacc005250 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffad0026030 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffacc038050 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffae388e740 10 client.20718 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561f73de9730 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.280+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad0026030 con 0x561f73d65b40 2024-09-12T03:10:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ffacc028590 con 0x561f73d65b40 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 10 client.20718 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 10 client.20718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 20 client.20718 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73de9c00 0x561f73dec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 4 client.20718 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f73cd1d70 con 0x561f73de9c00 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73de9c00 0x561f73dec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffada7fc640 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73de9c00 0x561f73dec020 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e6532718a10e94f9 server_cookie=222d74112388e645 in_seq=0 out_seq=0 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffabeff5640 10 client.20718 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffabeff5640 1 -- 172.21.15.16:0/2935178476 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f73cd1d70 con 0x561f73de9c00 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffabeff5640 10 client.20718 handle_command_reply: tid=0 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.281+0000 7ffae388e740 1 client.20718 shutdown 2024-09-12T03:10:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 msgr2=0x561f73d77c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ffacc006010 tx=0x7ffacc00b480 comp rx=0 tx=0).stop 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 shutdown_connections 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x561f73de9c00 0x561f73dec020 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f73d6f370 0x561f73d689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f73d65b40 0x561f73d77c30 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffabeff5640 0 client.20718 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 --2- 172.21.15.16:0/2935178476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f73d645d0 0x561f73d776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 >> 172.21.15.16:0/2935178476 conn(0x561f73d2d310 msgr2=0x561f73d6d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 shutdown_connections 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 1 -- 172.21.15.16:0/2935178476 wait complete. 2024-09-12T03:10:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae388e740 20 client.20718 trim_cache size 0 max 16384 2024-09-12T03:10:43.290 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:43.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 msgr2=0x7ffac40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 0x7ffac40a5040 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ffad40bc8e0 tx=0x7ffad402f710 comp rx=0 tx=0).stop 2024-09-12T03:10:43.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 msgr2=0x7ffadc074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.282+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ffad4031a20 tx=0x7ffad4031a50 comp rx=0 tx=0).stop 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 shutdown_connections 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ffac40a2b80 0x7ffac40a5040 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc106f10 0x7ffadc074f70 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffadc102790 0x7ffadc074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 --2- 172.21.15.16:0/2327081675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffadc101de0 0x7ffadc0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 >> 172.21.15.16:0/2327081675 conn(0x7ffadc0fd8c0 msgr2=0x7ffadc0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 shutdown_connections 2024-09-12T03:10:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:43.283+0000 7ffae14b5640 1 -- 172.21.15.16:0/2327081675 wait complete. 2024-09-12T03:10:43.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:10:44.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.18 MB/sec execute 465 sec latency 21390.403 ms 2024-09-12T03:10:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:44.449 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:44 smithi016 ceph-mon[30925]: pgmap v1021: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-09-12T03:10:44.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:44 smithi033 ceph-mon[35092]: pgmap v1021: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-09-12T03:10:44.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:44 smithi184 ceph-mon[35321]: pgmap v1021: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-09-12T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 -- 172.21.15.16:0/4082369763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564101de0 msgr2=0x7f85641021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 --2- 172.21.15.16:0/4082369763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564101de0 0x7f85641021c0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f855402a410 comp rx=0 tx=0).stop 2024-09-12T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 -- 172.21.15.16:0/4082369763 shutdown_connections 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 --2- 172.21.15.16:0/4082369763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 0x7f856410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 --2- 172.21.15.16:0/4082369763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564102790 0x7f85641067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 --2- 172.21.15.16:0/4082369763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564101de0 0x7f85641021c0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 -- 172.21.15.16:0/4082369763 >> 172.21.15.16:0/4082369763 conn(0x7f85640fd8c0 msgr2=0x7f85640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 -- 172.21.15.16:0/4082369763 shutdown_connections 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 -- 172.21.15.16:0/4082369763 wait complete. 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.618+0000 7f856a355640 1 Processor -- start 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 -- start start 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101de0 0x7f8564198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 0x7f856419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111f70 con 0x7f8564102790 2024-09-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111df0 con 0x7f8564101de0 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f856a355640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85641120f0 con 0x7f8564106f10 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35986/0 (socket says 172.21.15.16:35986) 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 -- 172.21.15.16:0/2294489965 learned_addr learned my addr 172.21.15.16:0/2294489965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f8563fff640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101de0 0x7f8564198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85688f2640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 0x7f856419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 -- 172.21.15.16:0/2294489965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 msgr2=0x7f856419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 0x7f856419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 -- 172.21.15.16:0/2294489965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101de0 msgr2=0x7f8564198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101de0 0x7f8564198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.619+0000 7f85637fe640 1 -- 172.21.15.16:0/2294489965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856419d3e0 con 0x7f8564102790 2024-09-12T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f85637fe640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f855800d950 tx=0x7f855800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f85617fa640 1 -- 172.21.15.16:0/2294489965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558017030 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f85617fa640 1 -- 172.21.15.16:0/2294489965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8558004020 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856419d6d0 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f85617fa640 1 -- 172.21.15.16:0/2294489965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85580047a0 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.620+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85641a4fb0 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.622+0000 7f85617fa640 1 -- 172.21.15.16:0/2294489965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f85580041d0 con 0x7f8564102790 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.622+0000 7f856c72e740 1 Processor -- start 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.622+0000 7f856c72e740 1 -- start start 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x559685641090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x559685652b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5596855a72f0 con 0x5596856415d0 2024-09-12T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55968554ecf0 con 0x559685642b40 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f856c72e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596855a32b0 con 0x559685650780 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35998/0 (socket says 172.21.15.16:35998) 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 -- 172.21.15.16:0/1882120988 learned_addr learned my addr 172.21.15.16:0/1882120988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85617fa640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 0x7f854c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85637fe640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x559685652b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85688f2640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x559685641090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 -- 172.21.15.16:0/1882120988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 msgr2=0x559685652b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x559685652b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85688f2640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 0x7f854c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 -- 172.21.15.16:0/1882120988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 msgr2=0x559685641090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x559685641090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 -- 172.21.15.16:0/1882120988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596855333a0 con 0x5596856415d0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85637fe640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x559685652b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f85617fa640 1 -- 172.21.15.16:0/2294489965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f85580cea10 con 0x7f8564102790 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.623+0000 7f8563fff640 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f8554005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b91f9a10b15082 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f85688f2640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 0x7f854c0a5040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f855c009fd0 tx=0x7f855c005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f854a7fc640 1 -- 172.21.15.16:0/1882120988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554040070 con 0x5596856415d0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f854a7fc640 1 -- 172.21.15.16:0/1882120988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85540319f0 con 0x5596856415d0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f854a7fc640 1 -- 172.21.15.16:0/1882120988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554035040 con 0x5596856415d0 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 -- 172.21.15.16:0/1882120988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 msgr2=0x559685641a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f8554005e80 comp rx=0 tx=0).stop 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 -- 172.21.15.16:0/1882120988 shutdown_connections 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x559685652b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x559685641a10 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 --2- 172.21.15.16:0/1882120988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x559685641090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 -- 172.21.15.16:0/1882120988 >> 172.21.15.16:0/1882120988 conn(0x55968560a310 msgr2=0x55968563dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 -- 172.21.15.16:0/1882120988 shutdown_connections 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.624+0000 7f856c72e740 1 -- 172.21.15.16:0/1882120988 wait complete. 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 -- start start 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 Processor -- start 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x55968564d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559685653420 con 0x5596856415d0 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559685562060 con 0x559685642b40 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5596856535a0 con 0x559685650780 2024-09-12T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f856c72e740 10 client.0 _ll_register_callbacks cb 0x5596852b58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36002/0 (socket says 172.21.15.16:36002) 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f8563fff640 1 -- 172.21.15.16:0/30370430 learned_addr learned my addr 172.21.15.16:0/30370430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.625+0000 7f85637fe640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x55968564d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85688f2640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 -- 172.21.15.16:0/30370430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 msgr2=0x55968564d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 -- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 msgr2=0x55968564d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x55968564d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 -- 172.21.15.16:0/30370430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596853183c0 con 0x5596856415d0 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85688f2640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f8563fff640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f855402a450 tx=0x7f8554005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee11010f84bc34 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85437fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554040030 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85540319f0 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8554004260 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8554035040 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f856c72e740 10 client.20730 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55968565ee30 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.626+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85540319f0 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f8554003850 con 0x5596856415d0 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 10 client.20730 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 10 client.20730 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 20 client.20730 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x559685660180 0x559685662570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 4 client.20730 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596855b4860 con 0x559685660180 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f85637fe640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x559685660180 0x559685662570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f85637fe640 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x559685660180 0x559685662570 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e84d32ba2749982f server_cookie=c8cef344f251ff6a in_seq=0 out_seq=0 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.627+0000 7f85437fe640 10 client.20730 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f85437fe640 1 -- 172.21.15.16:0/30370430 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5596855b4860 con 0x559685660180 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f85437fe640 10 client.20730 handle_command_reply: tid=0 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f856c72e740 1 client.20730 shutdown 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 msgr2=0x5596856500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f855402a450 tx=0x7f8554005250 comp rx=0 tx=0).stop 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 shutdown_connections 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.628+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x559685660180 0x559685662570 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685642b40 0x55968564d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 --2- 172.21.15.16:0/30370430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x5596856500e0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f85437fe640 0 client.20730 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 >> 172.21.15.16:0/30370430 conn(0x55968560a310 msgr2=0x559685643930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 shutdown_connections 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 1 -- 172.21.15.16:0/30370430 wait complete. 2024-09-12T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.629+0000 7f856c72e740 20 client.20730 trim_cache size 0 max 16384 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 Processor -- start 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 -- start start 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685642b40 0x55968564c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 0x559685667b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85641ab370 con 0x5596856415d0 2024-09-12T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f85641aceb0 con 0x559685650780 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.642+0000 7f856c72e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85641aba40 con 0x559685642b40 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f8563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36004/0 (socket says 172.21.15.16:36004) 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f8563fff640 1 -- 172.21.15.16:0/1196905237 learned_addr learned my addr 172.21.15.16:0/1196905237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 0x559685667b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85637fe640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685642b40 0x55968564c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 -- 172.21.15.16:0/1196905237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685642b40 msgr2=0x55968564c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685642b40 0x55968564c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 -- 172.21.15.16:0/1196905237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 msgr2=0x55968565c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 -- 172.21.15.16:0/1196905237 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559685644120 con 0x559685650780 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f8563fff640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f85688f2640 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 0x559685667b40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f855c013010 tx=0x7f855c0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=309ef15763723ef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.643+0000 7f8549ffb640 1 -- 172.21.15.16:0/1196905237 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c010d00 con 0x559685650780 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f8549ffb640 1 -- 172.21.15.16:0/1196905237 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f855c024070 con 0x559685650780 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f8549ffb640 1 -- 172.21.15.16:0/1196905237 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c005440 con 0x559685650780 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 -- 172.21.15.16:0/1196905237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 msgr2=0x559685667b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 0x559685667b40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f855c013010 tx=0x7f855c0096c0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 -- 172.21.15.16:0/1196905237 shutdown_connections 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559685650780 0x559685667b40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685642b40 0x55968564c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 --2- 172.21.15.16:0/1196905237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596856415d0 0x55968565c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 -- 172.21.15.16:0/1196905237 >> 172.21.15.16:0/1196905237 conn(0x55968560a310 msgr2=0x55968564bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 -- 172.21.15.16:0/1196905237 shutdown_connections 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.644+0000 7f856c72e740 1 -- 172.21.15.16:0/1196905237 wait complete. 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 -- start start 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 Processor -- start 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 0x559685659060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85641ac7e0 con 0x559685642b40 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f85641ac110 con 0x5596856415d0 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85641a9350 con 0x559685650780 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f856c72e740 10 client.0 _ll_register_callbacks cb 0x7f8554005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36008/0 (socket says 172.21.15.16:36008) 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 -- 172.21.15.16:0/500310642 learned_addr learned my addr 172.21.15.16:0/500310642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f8563fff640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 0x559685659060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85688f2640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 -- 172.21.15.16:0/500310642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 msgr2=0x55968564a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 -- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 msgr2=0x559685659060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 0x559685659060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f85637fe640 1 -- 172.21.15.16:0/500310642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968564afb0 con 0x559685642b40 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.645+0000 7f8563fff640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 0x559685659060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85637fe640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8558097c50 tx=0x7f85580cec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b621fd780a9ceb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855809b280 con 0x559685642b40 2024-09-12T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f8558099580 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f855c024070 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f855809f630 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 10 client.20736 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5596856cac50 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f855c024070 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f85580bb1d0 con 0x559685642b40 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 10 client.20736 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 10 client.20736 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 20 client.20736 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5596856cb420 0x5596856cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 4 client.20736 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.646+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596855aed70 con 0x5596856cb420 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f8563fff640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5596856cb420 0x5596856cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f8563fff640 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5596856cb420 0x5596856cd840 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e756551f8c877909 server_cookie=4834f3c7f812345 in_seq=0 out_seq=0 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f85437fe640 10 client.20736 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f85437fe640 1 -- 172.21.15.16:0/500310642 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596855aed70 con 0x5596856cb420 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f85437fe640 10 client.20736 handle_command_reply: tid=0 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.647+0000 7f856c72e740 1 client.20736 shutdown 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 msgr2=0x55968564a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8558097c50 tx=0x7f85580cec10 comp rx=0 tx=0).stop 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 shutdown_connections 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x5596856cb420 0x5596856cd840 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559685650780 0x55968564a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559685642b40 0x55968564a190 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 --2- 172.21.15.16:0/500310642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5596856415d0 0x559685659060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 >> 172.21.15.16:0/500310642 conn(0x55968560a310 msgr2=0x55968564c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f85437fe640 0 client.20736 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 shutdown_connections 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 1 -- 172.21.15.16:0/500310642 wait complete. 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.648+0000 7f856c72e740 20 client.20736 trim_cache size 0 max 16384 2024-09-12T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 msgr2=0x7f854c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 0x7f854c0a5040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f855c009fd0 tx=0x7f855c005bd0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 msgr2=0x7f8564198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f855800d950 tx=0x7f855800de10 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 shutdown_connections 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8564106f10 0x7f856419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f854c0a2b80 0x7f854c0a5040 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8564102790 0x7f8564198970 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 --2- 172.21.15.16:0/2294489965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101de0 0x7f8564198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 >> 172.21.15.16:0/2294489965 conn(0x7f85640fd8c0 msgr2=0x7f85640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 shutdown_connections 2024-09-12T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:44.649+0000 7f856a355640 1 -- 172.21.15.16:0/2294489965 wait complete. 2024-09-12T03:10:45.037 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.13 MB/sec execute 466 sec latency 22390.518 ms 2024-09-12T03:10:45.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:10:45.602 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:46.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.08 MB/sec execute 467 sec latency 23390.580 ms 2024-09-12T03:10:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 -- 172.21.15.16:0/3821585478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4101de0 msgr2=0x7f48d41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 --2- 172.21.15.16:0/3821585478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4101de0 0x7f48d41021c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009900 tx=0x7f48c402a410 comp rx=0 tx=0).stop 2024-09-12T03:10:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 -- 172.21.15.16:0/3821585478 shutdown_connections 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 --2- 172.21.15.16:0/3821585478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4106f10 0x7f48d410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 --2- 172.21.15.16:0/3821585478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4102790 0x7f48d41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 --2- 172.21.15.16:0/3821585478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4101de0 0x7f48d41021c0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 -- 172.21.15.16:0/3821585478 >> 172.21.15.16:0/3821585478 conn(0x7f48d40fd8c0 msgr2=0x7f48d40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 -- 172.21.15.16:0/3821585478 shutdown_connections 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.067+0000 7f48d9212640 1 -- 172.21.15.16:0/3821585478 wait complete. 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.068+0000 7f48d9212640 1 Processor -- start 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.068+0000 7f48d9212640 1 -- start start 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4101de0 0x7f48d4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 0x7f48d419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4111f70 con 0x7f48d4102790 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4111df0 con 0x7f48d4106f10 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d9212640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41120f0 con 0x7f48d4101de0 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60684/0 (socket says 172.21.15.16:60684) 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 -- 172.21.15.16:0/237968697 learned_addr learned my addr 172.21.15.16:0/237968697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4101de0 0x7f48d4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d359e640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 0x7f48d419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 -- 172.21.15.16:0/237968697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4101de0 msgr2=0x7f48d4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4101de0 0x7f48d4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 -- 172.21.15.16:0/237968697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 msgr2=0x7f48d419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 0x7f48d419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 -- 172.21.15.16:0/237968697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d419d3e0 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d359e640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 0x7f48d419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48d259c640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f48c800d950 tx=0x7f48c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.069+0000 7f48bbfff640 1 -- 172.21.15.16:0/237968697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8017070 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.070+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d419d6d0 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.070+0000 7f48bbfff640 1 -- 172.21.15.16:0/237968697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c8009c30 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.070+0000 7f48bbfff640 1 -- 172.21.15.16:0/237968697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c800f6b0 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.070+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d41a4fb0 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48bbfff640 1 -- 172.21.15.16:0/237968697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f48c80109d0 con 0x7f48d4102790 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 Processor -- start 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 -- start start 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612c834a2f0 con 0x5612c83ef370 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5612c82f1cf0 con 0x5612c83e5b40 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.072+0000 7f48db5eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612c83462b0 con 0x5612c83e45d0 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48bbfff640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 0x7f48bc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59558/0 (socket says 172.21.15.16:59558) 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60692/0 (socket says 172.21.15.16:60692) 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 -- 172.21.15.16:0/436706913 learned_addr learned my addr 172.21.15.16:0/436706913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 0x7f48bc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48bbfff640 1 -- 172.21.15.16:0/237968697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48c80ceca0 con 0x7f48d4102790 2024-09-12T03:10:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 -- 172.21.15.16:0/436706913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 msgr2=0x5612c83e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 -- 172.21.15.16:0/436706913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 msgr2=0x5612c83f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 -- 172.21.15.16:0/436706913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c82d63a0 con 0x5612c83e5b40 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d259c640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013910 tx=0x7f48cc013dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c66d610d57da0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48d359e640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 0x7f48bc0a5040 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00d000 tx=0x7f48cc01d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.073+0000 7f48b97fa640 1 -- 172.21.15.16:0/436706913 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc004d20 con 0x5612c83e5b40 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48b97fa640 1 -- 172.21.15.16:0/436706913 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48cc021030 con 0x5612c83e5b40 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48b97fa640 1 -- 172.21.15.16:0/436706913 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc01d660 con 0x5612c83e5b40 2024-09-12T03:10:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 -- 172.21.15.16:0/436706913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 msgr2=0x5612c83e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013910 tx=0x7f48cc013dd0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 -- 172.21.15.16:0/436706913 shutdown_connections 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 --2- 172.21.15.16:0/436706913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83e4090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 -- 172.21.15.16:0/436706913 >> 172.21.15.16:0/436706913 conn(0x5612c83ad310 msgr2=0x5612c83e0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 -- 172.21.15.16:0/436706913 shutdown_connections 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.074+0000 7f48db5eb740 1 -- 172.21.15.16:0/436706913 wait complete. 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 -- start start 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 Processor -- start 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83ec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612c83f2010 con 0x5612c83ef370 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5612c8305060 con 0x5612c83e5b40 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612c83f2190 con 0x5612c83e45d0 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48db5eb740 10 client.0 _ll_register_callbacks cb 0x5612c80588c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60700/0 (socket says 172.21.15.16:60700) 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 -- 172.21.15.16:0/313769113 learned_addr learned my addr 172.21.15.16:0/313769113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d259c640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 -- 172.21.15.16:0/313769113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 msgr2=0x5612c83eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 -- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 msgr2=0x5612c83ec020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83ec020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d359e640 1 -- 172.21.15.16:0/313769113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c80bb3c0 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.075+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48d359e640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f48cc005930 tx=0x7f48cc0097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d03897312649b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc033060 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48cc004a40 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48cc021030 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f48cc004c40 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 10 client.20748 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5612c83fda10 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48cc021030 con 0x5612c83ef370 2024-09-12T03:10:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f48cc01d6e0 con 0x5612c83ef370 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 10 client.20748 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 10 client.20748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 20 client.20748 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c83fee70 0x5612c8401260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 4 client.20748 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.076+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612c8357860 con 0x5612c83fee70 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.077+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c83fee70 0x5612c8401260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.077+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c83fee70 0x5612c8401260 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b88b3c341810cd84 server_cookie=f210a406fcc2c975 in_seq=0 out_seq=0 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.077+0000 7f48b6ff5640 10 client.20748 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48b6ff5640 1 -- 172.21.15.16:0/313769113 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612c8357860 con 0x5612c83fee70 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48b6ff5640 10 client.20748 handle_command_reply: tid=0 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 client.20748 shutdown 2024-09-12T03:10:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 msgr2=0x5612c83ec630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f48cc005930 tx=0x7f48cc0097c0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 shutdown_connections 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c83fee70 0x5612c8401260 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83ec630 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83ec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 --2- 172.21.15.16:0/313769113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 >> 172.21.15.16:0/313769113 conn(0x5612c83ad310 msgr2=0x5612c83e6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48b6ff5640 0 client.20748 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 shutdown_connections 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 1 -- 172.21.15.16:0/313769113 wait complete. 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.078+0000 7f48db5eb740 20 client.20748 trim_cache size 0 max 16384 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 Processor -- start 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 -- start start 2024-09-12T03:10:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 0x5612c83e7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 0x5612c83e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41ad5e0 con 0x5612c83e45d0 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41af120 con 0x5612c83ef370 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48db5eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41adcb0 con 0x5612c83e5b40 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60702/0 (socket says 172.21.15.16:60702) 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.091+0000 7f48d2d9d640 1 -- 172.21.15.16:0/127851359 learned_addr learned my addr 172.21.15.16:0/127851359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d359e640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 0x5612c83e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d259c640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 0x5612c83e7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 -- 172.21.15.16:0/127851359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 msgr2=0x5612c83e7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 0x5612c83e7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 -- 172.21.15.16:0/127851359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 msgr2=0x5612c83e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 0x5612c83e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 -- 172.21.15.16:0/127851359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c83ef050 con 0x5612c83e45d0 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d259c640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 0x5612c83e7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d359e640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 0x5612c83e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f48c402a930 tx=0x7f48c40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772fe76a774626c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48b8ff9640 1 -- 172.21.15.16:0/127851359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4035040 con 0x5612c83e45d0 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48b8ff9640 1 -- 172.21.15.16:0/127851359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c4005020 con 0x5612c83e45d0 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.092+0000 7f48b8ff9640 1 -- 172.21.15.16:0/127851359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4004a00 con 0x5612c83e45d0 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- 172.21.15.16:0/127851359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 msgr2=0x5612c83fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f48c402a930 tx=0x7f48c40058d0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- 172.21.15.16:0/127851359 shutdown_connections 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83ef370 0x5612c83e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e5b40 0x5612c83e7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 --2- 172.21.15.16:0/127851359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83e45d0 0x5612c83fb2b0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- 172.21.15.16:0/127851359 >> 172.21.15.16:0/127851359 conn(0x5612c83ad310 msgr2=0x5612c83ee0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- 172.21.15.16:0/127851359 shutdown_connections 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- 172.21.15.16:0/127851359 wait complete. 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 -- start start 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.093+0000 7f48db5eb740 1 Processor -- start 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83f7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41aea50 con 0x5612c83ef370 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41ae380 con 0x5612c83e5b40 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41ab5c0 con 0x5612c83e45d0 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 10 client.0 _ll_register_callbacks cb 0x7f48c809b360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60712/0 (socket says 172.21.15.16:60712) 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 -- 172.21.15.16:0/1720250897 learned_addr learned my addr 172.21.15.16:0/1720250897 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d259c640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 -- 172.21.15.16:0/1720250897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 msgr2=0x5612c83f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 -- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 msgr2=0x5612c83f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 -- 172.21.15.16:0/1720250897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c83e90b0 con 0x5612c83ef370 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d259c640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48d359e640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f48cc01dad0 tx=0x7f48cc011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7d2bcdbd9a2902 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc03e030 con 0x5612c83ef370 2024-09-12T03:10:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f48cc02b070 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.094+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48c4005020 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f48cc033040 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 10 client.20754 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5612c8469220 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c4005020 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f48cc005030 con 0x5612c83ef370 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 10 client.20754 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 10 client.20754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 20 client.20754 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c84698a0 0x5612c846bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 4 client.20754 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:10:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612c8351d70 con 0x5612c84698a0 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.095+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c84698a0 0x5612c846bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48b6ff5640 10 client.20754 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48d2d9d640 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c84698a0 0x5612c846bcc0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b267bbe0a725c6c server_cookie=ca2aca4855e4b0ac in_seq=0 out_seq=0 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48b6ff5640 1 -- 172.21.15.16:0/1720250897 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612c8351d70 con 0x5612c84698a0 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48b6ff5640 10 client.20754 handle_command_reply: tid=0 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 client.20754 shutdown 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 msgr2=0x5612c83e88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f48cc01dad0 tx=0x7f48cc011850 comp rx=0 tx=0).stop 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 shutdown_connections 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5612c84698a0 0x5612c846bcc0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612c83ef370 0x5612c83e88d0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48b6ff5640 0 client.20754 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612c83e5b40 0x5612c83f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 --2- 172.21.15.16:0/1720250897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612c83e45d0 0x5612c83f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 >> 172.21.15.16:0/1720250897 conn(0x5612c83ad310 msgr2=0x5612c83df4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 shutdown_connections 2024-09-12T03:10:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 1 -- 172.21.15.16:0/1720250897 wait complete. 2024-09-12T03:10:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.096+0000 7f48db5eb740 20 client.20754 trim_cache size 0 max 16384 2024-09-12T03:10:46.113 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 msgr2=0x7f48bc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 0x7f48bc0a5040 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00d000 tx=0x7f48cc01d000 comp rx=0 tx=0).stop 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 msgr2=0x7f48d4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f48c800d950 tx=0x7f48c800de10 comp rx=0 tx=0).stop 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 shutdown_connections 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48d4106f10 0x7f48d419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f48bc0a2b80 0x7f48bc0a5040 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d4102790 0x7f48d4198970 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 --2- 172.21.15.16:0/237968697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d4101de0 0x7f48d4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 >> 172.21.15.16:0/237968697 conn(0x7f48d40fd8c0 msgr2=0x7f48d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 shutdown_connections 2024-09-12T03:10:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:46.097+0000 7f48d9212640 1 -- 172.21.15.16:0/237968697 wait complete. 2024-09-12T03:10:46.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:46 smithi016 ceph-mon[30925]: pgmap v1022: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-09-12T03:10:46.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:46 smithi033 ceph-mon[35092]: pgmap v1022: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-09-12T03:10:46.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:46 smithi184 ceph-mon[35321]: pgmap v1022: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-09-12T03:10:47.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 22.04 MB/sec execute 468 sec latency 24390.689 ms 2024-09-12T03:10:47.815 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:10:47.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:10:48.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 21.99 MB/sec execute 469 sec latency 25390.729 ms 2024-09-12T03:10:48.085 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:48.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:48 smithi033 ceph-mon[35092]: pgmap v1023: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-09-12T03:10:48.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:48 smithi016 ceph-mon[30925]: pgmap v1023: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-09-12T03:10:48.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:48 smithi184 ceph-mon[35321]: pgmap v1023: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 -- 172.21.15.16:0/3292148889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff920102d40 msgr2=0x7ff920103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 --2- 172.21.15.16:0/3292148889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff920102d40 0x7ff920103120 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7ff910009900 tx=0x7ff91002a480 comp rx=0 tx=0).stop 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 -- 172.21.15.16:0/3292148889 shutdown_connections 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 --2- 172.21.15.16:0/3292148889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff92010c170 0x7ff92010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 --2- 172.21.15.16:0/3292148889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff92010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 --2- 172.21.15.16:0/3292148889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff920102d40 0x7ff920103120 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 -- 172.21.15.16:0/3292148889 >> 172.21.15.16:0/3292148889 conn(0x7ff9200fc7a0 msgr2=0x7ff9200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 -- 172.21.15.16:0/3292148889 shutdown_connections 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.586+0000 7ff92698b640 1 -- 172.21.15.16:0/3292148889 wait complete. 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 Processor -- start 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 -- start start 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff920102d40 0x7ff920198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff9201989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff920111fb0 con 0x7ff92010c170 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff920111e30 con 0x7ff920103660 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff92698b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff920112130 con 0x7ff920102d40 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.587+0000 7ff924f28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60730/0 (socket says 172.21.15.16:60730) 2024-09-12T03:10:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 -- 172.21.15.16:0/3645629296 learned_addr learned my addr 172.21.15.16:0/3645629296 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff920102d40 0x7ff920198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff9201989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 -- 172.21.15.16:0/3645629296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff920102d40 msgr2=0x7ff920198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff920102d40 0x7ff920198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 -- 172.21.15.16:0/3645629296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 msgr2=0x7ff9201989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff9201989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 -- 172.21.15.16:0/3645629296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92019d420 con 0x7ff92010c170 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff9201989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff924f28640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff91800b970 tx=0x7ff91800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91d7fa640 1 -- 172.21.15.16:0/3645629296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918019070 con 0x7ff92010c170 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92019d710 con 0x7ff92010c170 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91d7fa640 1 -- 172.21.15.16:0/3645629296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91800cb40 con 0x7ff92010c170 2024-09-12T03:10:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff91d7fa640 1 -- 172.21.15.16:0/3645629296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918007650 con 0x7ff92010c170 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.588+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9201a4ff0 con 0x7ff92010c170 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.590+0000 7ff91d7fa640 1 -- 172.21.15.16:0/3645629296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff91801c070 con 0x7ff92010c170 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 Processor -- start 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 -- start start 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e694a60 0x562f8e694e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e6a4150 0x562f8e6a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f8e5fa280 con 0x562f8e695380 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f8e5a1b10 con 0x562f8e6a4150 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff928d64740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562f8e5f6240 con 0x562f8e694a60 2024-09-12T03:10:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91d7fa640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 0x7ff9080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60742/0 (socket says 172.21.15.16:60742) 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e694a60 0x562f8e694e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91ffff640 1 -- 172.21.15.16:0/1979342251 learned_addr learned my addr 172.21.15.16:0/1979342251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e6a4150 0x562f8e6a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.591+0000 7ff924f28640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 0x7ff9080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91d7fa640 1 -- 172.21.15.16:0/3645629296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff9180ce6d0 con 0x7ff92010c170 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 -- 172.21.15.16:0/1979342251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e694a60 msgr2=0x562f8e694e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e694a60 0x562f8e694e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 -- 172.21.15.16:0/1979342251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e6a4150 msgr2=0x562f8e6a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e6a4150 0x562f8e6a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 -- 172.21.15.16:0/1979342251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f8e5ae1f0 con 0x562f8e695380 2024-09-12T03:10:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff91ffff640 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff910005ab0 tx=0x7ff910031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b76eadc6a8f691f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff924f28640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 0x7ff9080a5040 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff91809ac40 tx=0x7ff918065bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff9067fc640 1 -- 172.21.15.16:0/1979342251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91003f070 con 0x562f8e695380 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff9067fc640 1 -- 172.21.15.16:0/1979342251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff910031a20 con 0x562f8e695380 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.592+0000 7ff9067fc640 1 -- 172.21.15.16:0/1979342251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910035070 con 0x562f8e695380 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- 172.21.15.16:0/1979342251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 msgr2=0x562f8e6a3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff910005ab0 tx=0x7ff910031630 comp rx=0 tx=0).stop 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- 172.21.15.16:0/1979342251 shutdown_connections 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e6a4150 0x562f8e6a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a3c10 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 --2- 172.21.15.16:0/1979342251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e694a60 0x562f8e694e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- 172.21.15.16:0/1979342251 >> 172.21.15.16:0/1979342251 conn(0x562f8e6910f0 msgr2=0x562f8e6914d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- 172.21.15.16:0/1979342251 shutdown_connections 2024-09-12T03:10:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- 172.21.15.16:0/1979342251 wait complete. 2024-09-12T03:10:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.593+0000 7ff928d64740 1 -- start start 2024-09-12T03:10:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 Processor -- start 2024-09-12T03:10:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6a1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f8e6a6d20 con 0x562f8e695380 2024-09-12T03:10:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f8e5b4f10 con 0x562f8e694a60 2024-09-12T03:10:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562f8e6a6ea0 con 0x562f8e6a4150 2024-09-12T03:10:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff928d64740 10 client.0 _ll_register_callbacks cb 0x562f8e57da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60756/0 (socket says 172.21.15.16:60756) 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 -- 172.21.15.16:0/3699304336 learned_addr learned my addr 172.21.15.16:0/3699304336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 -- 172.21.15.16:0/3699304336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 msgr2=0x562f8e6a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 -- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 msgr2=0x562f8e6a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.594+0000 7ff91ffff640 1 -- 172.21.15.16:0/3699304336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f8e36b6a0 con 0x562f8e695380 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff910009900 tx=0x7ff9100049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece3fd2fa47a435b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91004b030 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff910031820 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff910031a20 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff910035070 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff928d64740 10 client.20766 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562f8e6b29c0 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff910031a20 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.595+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff91003b4c0 con 0x562f8e695380 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 10 client.20766 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 10 client.20766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 20 client.20766 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e6b3d10 0x562f8e6b6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 4 client.20766 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f8e6077f0 con 0x562f8e6b3d10 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e6b3d10 0x562f8e6b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff8ff7fe640 10 client.20766 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.596+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e6b3d10 0x562f8e6b6100 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ad3e235ee01c540 server_cookie=33fe692bb75ca35a in_seq=0 out_seq=0 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3699304336 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562f8e6077f0 con 0x562f8e6b3d10 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff8ff7fe640 10 client.20766 handle_command_reply: tid=0 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 client.20766 shutdown 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 msgr2=0x562f8e6a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff910009900 tx=0x7ff9100049d0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 shutdown_connections 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e6b3d10 0x562f8e6b6100 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e695380 0x562f8e6a15d0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 --2- 172.21.15.16:0/3699304336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff8ff7fe640 0 client.20766 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 >> 172.21.15.16:0/3699304336 conn(0x562f8e6910f0 msgr2=0x562f8e696e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 shutdown_connections 2024-09-12T03:10:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.597+0000 7ff928d64740 1 -- 172.21.15.16:0/3699304336 wait complete. 2024-09-12T03:10:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.598+0000 7ff928d64740 20 client.20766 trim_cache size 0 max 16384 2024-09-12T03:10:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 Processor -- start 2024-09-12T03:10:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 -- start start 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 0x562f8e6bb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6bbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201ab1b0 con 0x562f8e694a60 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201accf0 con 0x562f8e695380 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff928d64740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201ab880 con 0x562f8e6a4150 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60758/0 (socket says 172.21.15.16:60758) 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 -- 172.21.15.16:0/1456728712 learned_addr learned my addr 172.21.15.16:0/1456728712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 -- 172.21.15.16:0/1456728712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 msgr2=0x562f8e6bbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91ffff640 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 0x562f8e6bb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6bbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 -- 172.21.15.16:0/1456728712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 msgr2=0x562f8e6bb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 0x562f8e6bb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91f7fe640 1 -- 172.21.15.16:0/1456728712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f8e692280 con 0x562f8e694a60 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.615+0000 7ff91ffff640 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 0x562f8e6bb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x562f8e6b6670 tx=0x7ff914004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e510135170ad5d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff905ffb640 1 -- 172.21.15.16:0/1456728712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914012df0 con 0x562f8e694a60 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff905ffb640 1 -- 172.21.15.16:0/1456728712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff914011d70 con 0x562f8e694a60 2024-09-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff905ffb640 1 -- 172.21.15.16:0/1456728712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140047a0 con 0x562f8e694a60 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 -- 172.21.15.16:0/1456728712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 msgr2=0x562f8e696cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x562f8e6b6670 tx=0x7ff914004730 comp rx=0 tx=0).stop 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 -- 172.21.15.16:0/1456728712 shutdown_connections 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e6a4150 0x562f8e6bbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e695380 0x562f8e6bb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 --2- 172.21.15.16:0/1456728712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e694a60 0x562f8e696cf0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 -- 172.21.15.16:0/1456728712 >> 172.21.15.16:0/1456728712 conn(0x562f8e6910f0 msgr2=0x562f8e6965e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 -- 172.21.15.16:0/1456728712 shutdown_connections 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.616+0000 7ff928d64740 1 -- 172.21.15.16:0/1456728712 wait complete. 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 -- start start 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 Processor -- start 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6aca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e695380 0x562f8e6a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201ac620 con 0x562f8e6a4150 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201abf50 con 0x562f8e694a60 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9201a9190 con 0x562f8e695380 2024-09-12T03:10:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff928d64740 10 client.0 _ll_register_callbacks cb 0x7ff92006b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60770/0 (socket says 172.21.15.16:60770) 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.617+0000 7ff91f7fe640 1 -- 172.21.15.16:0/3252793055 learned_addr learned my addr 172.21.15.16:0/3252793055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e695380 0x562f8e6a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 -- 172.21.15.16:0/3252793055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e695380 msgr2=0x562f8e6a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e695380 0x562f8e6a7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 -- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 msgr2=0x562f8e6aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6aca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 -- 172.21.15.16:0/3252793055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f8e71d6a0 con 0x562f8e6a4150 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff91f7fe640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff914012590 tx=0x7ff91400dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac16793f3eb7989 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:48.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914011bb0 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff91401b040 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff914011d70 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff9140215d0 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff928d64740 10 client.20772 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.618+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562f8e71e810 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff914011d70 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7ff91401f980 con 0x562f8e6a4150 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 10 client.20772 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 10 client.20772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 20 client.20772 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e71ede0 0x562f8e7211d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 4 client.20772 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:10:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562f8e601d00 con 0x562f8e71ede0 2024-09-12T03:10:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e71ede0 0x562f8e7211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.619+0000 7ff91ffff640 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e71ede0 0x562f8e7211d0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4329ad47be25f6 server_cookie=77c5cdf3d79327a2 in_seq=0 out_seq=0 2024-09-12T03:10:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.620+0000 7ff8ff7fe640 10 client.20772 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff8ff7fe640 1 -- 172.21.15.16:0/3252793055 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562f8e601d00 con 0x562f8e71ede0 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff8ff7fe640 10 client.20772 handle_command_reply: tid=0 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 client.20772 shutdown 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 msgr2=0x562f8e6a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff914012590 tx=0x7ff91400dfa0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 shutdown_connections 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x562f8e71ede0 0x562f8e7211d0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f8e6a4150 0x562f8e6a84f0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562f8e695380 0x562f8e6a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 --2- 172.21.15.16:0/3252793055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f8e694a60 0x562f8e6aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 >> 172.21.15.16:0/3252793055 conn(0x562f8e6910f0 msgr2=0x562f8e690270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff8ff7fe640 0 client.20772 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 shutdown_connections 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 1 -- 172.21.15.16:0/3252793055 wait complete. 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.621+0000 7ff928d64740 20 client.20772 trim_cache size 0 max 16384 2024-09-12T03:10:48.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:48.627 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:10:48.627 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f2dfeb1a-6ec9-421d-a211-daeed43197bd", 2024-09-12T03:10:48.627 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:10:48.627 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 msgr2=0x7ff9080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 0x7ff9080a5040 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff91809ac40 tx=0x7ff918065bc0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 msgr2=0x7ff92019cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff91800b970 tx=0x7ff91800be30 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 shutdown_connections 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff9080a2b80 0x7ff9080a5040 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92010c170 0x7ff92019cd40 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff920103660 0x7ff9201989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 --2- 172.21.15.16:0/3645629296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff920102d40 0x7ff920198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 >> 172.21.15.16:0/3645629296 conn(0x7ff9200fc7a0 msgr2=0x7ff9200fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 shutdown_connections 2024-09-12T03:10:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:48.622+0000 7ff92698b640 1 -- 172.21.15.16:0/3645629296 wait complete. 2024-09-12T03:10:49.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 21.94 MB/sec execute 470 sec latency 26390.795 ms 2024-09-12T03:10:49.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:10:49.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:49.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:49 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:49 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.609 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:10:49.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:49 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:49.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:10:50.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 21.90 MB/sec execute 471 sec latency 27391.097 ms 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.095+0000 7f9900ae8640 1 -- 172.21.15.16:0/3317844844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc103760 msgr2=0x7f98fc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.095+0000 7f9900ae8640 1 --2- 172.21.15.16:0/3317844844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc103760 0x7f98fc107ba0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009930 tx=0x7f98e402a420 comp rx=0 tx=0).stop 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 -- 172.21.15.16:0/3317844844 shutdown_connections 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 --2- 172.21.15.16:0/3317844844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 0x7f98fc10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 --2- 172.21.15.16:0/3317844844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc103760 0x7f98fc107ba0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 --2- 172.21.15.16:0/3317844844 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc102db0 0x7f98fc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 -- 172.21.15.16:0/3317844844 >> 172.21.15.16:0/3317844844 conn(0x7f98fc0fc7d0 msgr2=0x7f98fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 -- 172.21.15.16:0/3317844844 shutdown_connections 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.096+0000 7f9900ae8640 1 -- 172.21.15.16:0/3317844844 wait complete. 2024-09-12T03:10:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 Processor -- start 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 -- start start 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc103760 0x7f98fc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 0x7f98fc19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc112050 con 0x7f98fc102db0 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc111ed0 con 0x7f98fc103760 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f9900ae8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1121d0 con 0x7f98fc1082e0 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60792/0 (socket says 172.21.15.16:60792) 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 -- 172.21.15.16:0/2837531280 learned_addr learned my addr 172.21.15.16:0/2837531280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc103760 0x7f98fc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fad9d640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 0x7f98fc19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 -- 172.21.15.16:0/2837531280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 msgr2=0x7f98fc19cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 0x7f98fc19cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 -- 172.21.15.16:0/2837531280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc103760 msgr2=0x7f98fc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc103760 0x7f98fc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.097+0000 7f98fa59c640 1 -- 172.21.15.16:0/2837531280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc19d580 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f98fa59c640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c890 tx=0x7f98f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f98f37fe640 1 -- 172.21.15.16:0/2837531280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4004220 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98fc19d870 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f98f37fe640 1 -- 172.21.15.16:0/2837531280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f40043c0 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f98f37fe640 1 -- 172.21.15.16:0/2837531280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4004a00 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.098+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98fc1a5150 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.100+0000 7f98f37fe640 1 -- 172.21.15.16:0/2837531280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f98f40079b0 con 0x7f98fc102db0 2024-09-12T03:10:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.100+0000 7f9902ec1740 1 Processor -- start 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 -- start start 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd196550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd198e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624dd0ed2e0 con 0x5624dd187360 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5624dd094ce0 con 0x5624dd196a90 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f9902ec1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624dd0e92a0 con 0x5624dd187c80 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60794/0 (socket says 172.21.15.16:60794) 2024-09-12T03:10:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 -- 172.21.15.16:0/3634038262 learned_addr learned my addr 172.21.15.16:0/3634038262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fa59c640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98f37fe640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 0x7f98e80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 -- 172.21.15.16:0/3634038262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 msgr2=0x5624dd196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 -- 172.21.15.16:0/3634038262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 msgr2=0x5624dd198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fad9d640 1 -- 172.21.15.16:0/3634038262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624dd079440 con 0x5624dd187360 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98fa59c640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.101+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98f37fe640 1 -- 172.21.15.16:0/2837531280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f98f40cece0 con 0x7f98fc102db0 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98fad9d640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 0x7f98e80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98fad9d640 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f98ec007ec0 tx=0x7f98ec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b520e870327f7c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98f0ff9640 1 -- 172.21.15.16:0/3634038262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec010ac0 con 0x5624dd187360 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98f0ff9640 1 -- 172.21.15.16:0/3634038262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ec01ae00 con 0x5624dd187360 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.102+0000 7f98f0ff9640 1 -- 172.21.15.16:0/3634038262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec019c60 con 0x5624dd187360 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f98fad9d640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 0x7f98e80a5040 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f98ec01e000 tx=0x7f98ec01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- 172.21.15.16:0/3634038262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 msgr2=0x5624dd187740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f98ec007ec0 tx=0x7f98ec00f790 comp rx=0 tx=0).stop 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- 172.21.15.16:0/3634038262 shutdown_connections 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- 172.21.15.16:0/3634038262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187360 0x5624dd187740 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- 172.21.15.16:0/3634038262 >> 172.21.15.16:0/3634038262 conn(0x5624dd183a30 msgr2=0x5624dd183e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- 172.21.15.16:0/3634038262 shutdown_connections 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- 172.21.15.16:0/3634038262 wait complete. 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 -- start start 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 Processor -- start 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd196410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.103+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd193c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624dd199660 con 0x5624dd187c80 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5624dd0a8050 con 0x5624dd196a90 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624dd1997e0 con 0x5624dd187360 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 10 client.0 _ll_register_callbacks cb 0x5624dd0e8160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60798/0 (socket says 172.21.15.16:60798) 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 -- 172.21.15.16:0/1204250007 learned_addr learned my addr 172.21.15.16:0/1204250007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98fa59c640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98fad9d640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd193c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 -- 172.21.15.16:0/1204250007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 msgr2=0x5624dd196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 -- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 msgr2=0x5624dd193c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd193c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 -- 172.21.15.16:0/1204250007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624dce5e3c0 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98fad9d640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd193c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000c00 tx=0x7f98e40044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e07a2ebb69553ae server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e4035070 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f98e4003b40 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.104+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ec01ae00 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f98e4003d40 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 10 client.20784 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5624dd1a5090 con 0x5624dd187c80 2024-09-12T03:10:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ec01ae00 con 0x5624dd187c80 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f98e403b690 con 0x5624dd187c80 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 10 client.20784 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 10 client.20784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 20 client.20784 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd1a65a0 0x5624dd1a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 4 client.20784 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624dd0fa850 con 0x5624dd1a65a0 2024-09-12T03:10:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.105+0000 7f98fa59c640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd1a65a0 0x5624dd1a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f98fa59c640 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd1a65a0 0x5624dd1a89c0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a9bed9b5baa9494 server_cookie=9eabe782d52b632b in_seq=0 out_seq=0 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f98e1ffb640 10 client.20784 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f98e1ffb640 1 -- 172.21.15.16:0/1204250007 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5624dd0fa850 con 0x5624dd1a65a0 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f98e1ffb640 10 client.20784 handle_command_reply: tid=0 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 client.20784 shutdown 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 msgr2=0x5624dd1936c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000c00 tx=0x7f98e40044f0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 shutdown_connections 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd1a65a0 0x5624dd1a89c0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd196a90 0x5624dd193c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd187c80 0x5624dd1936c0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f98e1ffb640 0 client.20784 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 --2- 172.21.15.16:0/1204250007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 >> 172.21.15.16:0/1204250007 conn(0x5624dd183a30 msgr2=0x5624dd189790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.106+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 shutdown_connections 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.107+0000 7f9902ec1740 1 -- 172.21.15.16:0/1204250007 wait complete. 2024-09-12T03:10:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.107+0000 7f9902ec1740 20 client.20784 trim_cache size 0 max 16384 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 Processor -- start 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 -- start start 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd1a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187c80 0x5624dd195630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1ab3c0 con 0x5624dd196a90 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1acf00 con 0x5624dd187c80 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f9902ec1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1aba90 con 0x5624dd187360 2024-09-12T03:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.125+0000 7f98fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60806/0 (socket says 172.21.15.16:60806) 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 -- 172.21.15.16:0/592254634 learned_addr learned my addr 172.21.15.16:0/592254634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fa59c640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd1a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187c80 0x5624dd195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 -- 172.21.15.16:0/592254634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 msgr2=0x5624dd1a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd1a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 -- 172.21.15.16:0/592254634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187c80 msgr2=0x5624dd195630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187c80 0x5624dd195630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 -- 172.21.15.16:0/592254634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624dd1834d0 con 0x5624dd196a90 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fa59c640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd1a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98fad9d640 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f98ec007dc0 tx=0x7f98ec027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9276f4a335d1cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98e3fff640 1 -- 172.21.15.16:0/592254634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec034070 con 0x5624dd196a90 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98e3fff640 1 -- 172.21.15.16:0/592254634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ec003e50 con 0x5624dd196a90 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.126+0000 7f98e3fff640 1 -- 172.21.15.16:0/592254634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec01aa70 con 0x5624dd196a90 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 -- 172.21.15.16:0/592254634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 msgr2=0x5624dd1ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f98ec007dc0 tx=0x7f98ec027000 comp rx=0 tx=0).stop 2024-09-12T03:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 -- 172.21.15.16:0/592254634 shutdown_connections 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd1ae270 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187c80 0x5624dd195630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 --2- 172.21.15.16:0/592254634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187360 0x5624dd1a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 -- 172.21.15.16:0/592254634 >> 172.21.15.16:0/592254634 conn(0x5624dd183a30 msgr2=0x5624dd194f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 -- 172.21.15.16:0/592254634 shutdown_connections 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.127+0000 7f9902ec1740 1 -- 172.21.15.16:0/592254634 wait complete. 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 -- start start 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 Processor -- start 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd19b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd20a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1ac830 con 0x5624dd196a90 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1ac160 con 0x5624dd187360 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc1a93a0 con 0x5624dd187c80 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f9902ec1740 10 client.0 _ll_register_callbacks cb 0x7f98e4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f98fa59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f98fa59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59684/0 (socket says 172.21.15.16:59684) 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f98fa59c640 1 -- 172.21.15.16:0/24566251 learned_addr learned my addr 172.21.15.16:0/24566251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.128+0000 7f98fad9d640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd20a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 -- 172.21.15.16:0/24566251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 msgr2=0x5624dd19b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd19b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 -- 172.21.15.16:0/24566251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 msgr2=0x5624dd20a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd20a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 -- 172.21.15.16:0/24566251 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624dd20a8c0 con 0x5624dd187360 2024-09-12T03:10:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd19b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98fa59c640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005fd0 tx=0x7f98f4005300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf9e1de96d068773 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f401d040 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f98f4097c10 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ec003e50 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f98f409f8a0 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f9902ec1740 10 client.29948 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.129+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5624dd211ca0 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ec003e50 con 0x5624dd187360 2024-09-12T03:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f98f4084b30 con 0x5624dd187360 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 10 client.29948 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 10 client.29948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 20 client.29948 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd212370 0x5624dd214760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 4 client.29948 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624dd0f4d60 con 0x5624dd212370 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd212370 0x5624dd214760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f98e1ffb640 10 client.29948 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.130+0000 7f98f9d9b640 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd212370 0x5624dd214760 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ade609d77b8ade5c server_cookie=684fbcadf49b478a in_seq=0 out_seq=0 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f98e1ffb640 1 -- 172.21.15.16:0/24566251 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5624dd0f4d60 con 0x5624dd212370 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f98e1ffb640 10 client.29948 handle_command_reply: tid=0 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 client.29948 shutdown 2024-09-12T03:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 msgr2=0x5624dd19b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005fd0 tx=0x7f98f4005300 comp rx=0 tx=0).stop 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 shutdown_connections 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5624dd212370 0x5624dd214760 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624dd196a90 0x5624dd20a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624dd187c80 0x5624dd19b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 --2- 172.21.15.16:0/24566251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5624dd187360 0x5624dd19b0c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f98e1ffb640 0 client.29948 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 >> 172.21.15.16:0/24566251 conn(0x5624dd183a30 msgr2=0x5624dd195740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 shutdown_connections 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 1 -- 172.21.15.16:0/24566251 wait complete. 2024-09-12T03:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.131+0000 7f9902ec1740 20 client.29948 trim_cache size 0 max 16384 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "f2dfeb1a-6ec9-421d-a211-daeed43197bd": { 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f2dfeb1a-6ec9-421d-a211-daeed43197bd", 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:10:50.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:50.142 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 msgr2=0x7f98e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 0x7f98e80a5040 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f98ec01e000 tx=0x7f98ec01f000 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 msgr2=0x7f98fc1985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c890 tx=0x7f98f400cd50 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 shutdown_connections 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98fc1082e0 0x7f98fc19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98fc103760 0x7f98fc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f98e80a2b80 0x7f98e80a5040 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 --2- 172.21.15.16:0/2837531280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98fc102db0 0x7f98fc1985d0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 >> 172.21.15.16:0/2837531280 conn(0x7f98fc0fc7d0 msgr2=0x7f98fc106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 shutdown_connections 2024-09-12T03:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:50.132+0000 7f9900ae8640 1 -- 172.21.15.16:0/2837531280 wait complete. 2024-09-12T03:10:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:50 smithi016 ceph-mon[30925]: pgmap v1024: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-09-12T03:10:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:50 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:50 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:50 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:10:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:50 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:50 smithi033 ceph-mon[35092]: pgmap v1024: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-09-12T03:10:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:50 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:50 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:50 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:10:50.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:50 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:50 smithi184 ceph-mon[35321]: pgmap v1024: 97 pgs: 97 active+clean; 813 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-09-12T03:10:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:50 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:50 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:50 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:10:50.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:50 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:10:50.871 INFO:tasks.cephfs.filesystem:scrub status for tag:f2dfeb1a-6ec9-421d-a211-daeed43197bd - {'path': '/', 'tag': 'f2dfeb1a-6ec9-421d-a211-daeed43197bd', 'options': 'recursive,force'} 2024-09-12T03:10:51.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 21.85 MB/sec execute 472 sec latency 28391.180 ms 2024-09-12T03:10:51.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 -- 172.21.15.16:0/1124767021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c101410 msgr2=0x7f526c10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 --2- 172.21.15.16:0/1124767021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c101410 0x7f526c10d940 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5264009cd0 tx=0x7f526402a8f0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 -- 172.21.15.16:0/1124767021 shutdown_connections 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 --2- 172.21.15.16:0/1124767021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c101410 0x7f526c10d940 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 --2- 172.21.15.16:0/1124767021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c100a70 0x7f526c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 --2- 172.21.15.16:0/1124767021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c106a70 0x7f526c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 -- 172.21.15.16:0/1124767021 >> 172.21.15.16:0/1124767021 conn(0x7f526c0fc790 msgr2=0x7f526c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 -- 172.21.15.16:0/1124767021 shutdown_connections 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.261+0000 7f5272a48640 1 -- 172.21.15.16:0/1124767021 wait complete. 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 Processor -- start 2024-09-12T03:10:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 -- start start 2024-09-12T03:10:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c100a70 0x7f526c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 0x7f526c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f526c110e70 con 0x7f526c106a70 2024-09-12T03:10:51.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f526c110cf0 con 0x7f526c101410 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5272a48640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f526c110ff0 con 0x7f526c100a70 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5270fe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5270fe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60828/0 (socket says 172.21.15.16:60828) 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f5270fe5640 1 -- 172.21.15.16:0/1639433636 learned_addr learned my addr 172.21.15.16:0/1639433636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.262+0000 7f526b7fe640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 0x7f526c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f526bfff640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c100a70 0x7f526c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 -- 172.21.15.16:0/1639433636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c100a70 msgr2=0x7f526c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c100a70 0x7f526c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 -- 172.21.15.16:0/1639433636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 msgr2=0x7f526c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 0x7f526c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 -- 172.21.15.16:0/1639433636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c19d400 con 0x7f526c106a70 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f526b7fe640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 0x7f526c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:10:51.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5270fe5640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5264005cc0 tx=0x7f5264009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264003d80 con 0x7f526c106a70 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526c19d690 con 0x7f526c106a70 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5264040070 con 0x7f526c106a70 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264045b10 con 0x7f526c106a70 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.263+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526c1a4fd0 con 0x7f526c106a70 2024-09-12T03:10:51.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.264+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f526c100ed0 con 0x7f526c106a70 2024-09-12T03:10:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.266+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5264004700 con 0x7f526c106a70 2024-09-12T03:10:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.266+0000 7f52697fa640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 0x7f52540a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.266+0000 7f526bfff640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 0x7f52540a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.266+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f52640f0250 con 0x7f526c106a70 2024-09-12T03:10:51.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.267+0000 7f526bfff640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 0x7f52540a5040 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f525c005fd0 tx=0x7f525c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:51.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.270+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f52640bd250 con 0x7f526c106a70 2024-09-12T03:10:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.492+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f526c1997b0 con 0x7f526c106a70 2024-09-12T03:10:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.493+0000 7f52697fa640 1 -- 172.21.15.16:0/1639433636 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f526c1997b0 con 0x7f526c106a70 2024-09-12T03:10:51.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:10:51.493 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:10:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 msgr2=0x7f52540a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 0x7f52540a5040 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f525c005fd0 tx=0x7f525c00b390 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 msgr2=0x7f526c19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5264005cc0 tx=0x7f5264009800 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 shutdown_connections 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f52540a2b80 0x7f52540a5040 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f526c106a70 0x7f526c19cd20 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f526c101410 0x7f526c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 --2- 172.21.15.16:0/1639433636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f526c100a70 0x7f526c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 >> 172.21.15.16:0/1639433636 conn(0x7f526c0fc790 msgr2=0x7f526c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 shutdown_connections 2024-09-12T03:10:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:51.495+0000 7f5272a48640 1 -- 172.21.15.16:0/1639433636 wait complete. 2024-09-12T03:10:52.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231935 21.80 MB/sec execute 473 sec latency 29391.264 ms 2024-09-12T03:10:52.210 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id b83f3b93 --release --await' 2024-09-12T03:10:52.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id b83f3b93 --release --await 2024-09-12T03:10:52.471 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:10:52.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:52 smithi033 ceph-mon[35092]: pgmap v1025: 97 pgs: 97 active+clean; 824 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-09-12T03:10:52.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1639433636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:52 smithi184 ceph-mon[35321]: pgmap v1025: 97 pgs: 97 active+clean; 824 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-09-12T03:10:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1639433636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:52.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:52 smithi016 ceph-mon[30925]: pgmap v1025: 97 pgs: 97 active+clean; 824 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-09-12T03:10:52.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1639433636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 -- 172.21.15.16:0/597072572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 msgr2=0x7fd46010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 --2- 172.21.15.16:0/597072572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd46010e610 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd458009ca0 tx=0x7fd45802a950 comp rx=0 tx=0).stop 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 -- 172.21.15.16:0/597072572 shutdown_connections 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 --2- 172.21.15.16:0/597072572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd46010e610 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 --2- 172.21.15.16:0/597072572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460103660 0x7fd46010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 --2- 172.21.15.16:0/597072572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460102d40 0x7fd460103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 -- 172.21.15.16:0/597072572 >> 172.21.15.16:0/597072572 conn(0x7fd4600fc7a0 msgr2=0x7fd4600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 -- 172.21.15.16:0/597072572 shutdown_connections 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 -- 172.21.15.16:0/597072572 wait complete. 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.946+0000 7fd46849f640 1 Processor -- start 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 -- start start 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460102d40 0x7fd4601009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460103660 0x7fd460100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd460111f70 con 0x7fd46010c170 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd460111df0 con 0x7fd460102d40 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46849f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601120f0 con 0x7fd460103660 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60844/0 (socket says 172.21.15.16:60844) 2024-09-12T03:10:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 -- 172.21.15.16:0/2972701177 learned_addr learned my addr 172.21.15.16:0/2972701177 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd46623b640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460102d40 0x7fd4601009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd465a3a640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460103660 0x7fd460100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 -- 172.21.15.16:0/2972701177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460103660 msgr2=0x7fd460100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460103660 0x7fd460100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 -- 172.21.15.16:0/2972701177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460102d40 msgr2=0x7fd4601009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460102d40 0x7fd4601009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.947+0000 7fd466a3c640 1 -- 172.21.15.16:0/2972701177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4601a68b0 con 0x7fd46010c170 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd466a3c640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd45802ae70 tx=0x7fd45802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd4577fe640 1 -- 172.21.15.16:0/2972701177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458034040 con 0x7fd46010c170 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4601a6b40 con 0x7fd46010c170 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd4577fe640 1 -- 172.21.15.16:0/2972701177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd458030920 con 0x7fd46010c170 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd4577fe640 1 -- 172.21.15.16:0/2972701177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458045480 con 0x7fd46010c170 2024-09-12T03:10:52.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.948+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4601a7070 con 0x7fd46010c170 2024-09-12T03:10:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.950+0000 7fd4577fe640 1 -- 172.21.15.16:0/2972701177 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd458037020 con 0x7fd46010c170 2024-09-12T03:10:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.950+0000 7fd46a878740 1 Processor -- start 2024-09-12T03:10:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.950+0000 7fd46a878740 1 -- start start 2024-09-12T03:10:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d305a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3064d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5617d2fbf620 con 0x5617d3065240 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5617d2f66e50 con 0x5617d305a060 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46a878740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5617d2fbb5e0 con 0x5617d305a980 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60846/0 (socket says 172.21.15.16:60846) 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 -- 172.21.15.16:0/2185282118 learned_addr learned my addr 172.21.15.16:0/2185282118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd46623b640 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3064d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d305a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd4577fe640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 0x7fd44c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 -- 172.21.15.16:0/2185282118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 msgr2=0x5617d3064d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3064d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 -- 172.21.15.16:0/2185282118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 msgr2=0x5617d305a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d305a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd465a3a640 1 -- 172.21.15.16:0/2185282118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617d2f57d40 con 0x5617d3065240 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.951+0000 7fd466a3c640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 0x7fd44c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.952+0000 7fd4577fe640 1 -- 172.21.15.16:0/2972701177 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd4580efe50 con 0x7fd46010c170 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.952+0000 7fd465a3a640 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fd45000bfa0 tx=0x7fd45000b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efd4a5c72473274 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.952+0000 7fd454ff9640 1 -- 172.21.15.16:0/2185282118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45001d5f0 con 0x5617d3065240 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.952+0000 7fd454ff9640 1 -- 172.21.15.16:0/2185282118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd450010910 con 0x5617d3065240 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.952+0000 7fd454ff9640 1 -- 172.21.15.16:0/2185282118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45001c820 con 0x5617d3065240 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd466a3c640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 0x7fd44c0a50c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd458082530 tx=0x7fd45808f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- 172.21.15.16:0/2185282118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 msgr2=0x5617d3067630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fd45000bfa0 tx=0x7fd45000b7a0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- 172.21.15.16:0/2185282118 shutdown_connections 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3067630 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3064d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- 172.21.15.16:0/2185282118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d305a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- 172.21.15.16:0/2185282118 >> 172.21.15.16:0/2185282118 conn(0x5617d30565f0 msgr2=0x5617d30569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- 172.21.15.16:0/2185282118 shutdown_connections 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- 172.21.15.16:0/2185282118 wait complete. 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- start start 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 Processor -- start 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d3062210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3062750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5617d3067e10 con 0x5617d3065240 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5617d2f47f10 con 0x5617d305a060 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.953+0000 7fd46a878740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5617d3067f90 con 0x5617d305a980 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd46a878740 10 client.0 _ll_register_callbacks cb 0x5617d2f6bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60854/0 (socket says 172.21.15.16:60854) 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 -- 172.21.15.16:0/1592025355 learned_addr learned my addr 172.21.15.16:0/1592025355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d3062210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd46623b640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3062750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 -- 172.21.15.16:0/1592025355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 msgr2=0x5617d3062750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3062750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 -- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 msgr2=0x5617d3062210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d3062210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 -- 172.21.15.16:0/1592025355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617d2cce580 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b540 tx=0x7fd45000bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a35d783a0db41f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45001c7a0 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd450010910 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.954+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd45001c940 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd45001cb40 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 10 client.20808 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5617d2d98960 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd450010910 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd45001c070 con 0x5617d3065240 2024-09-12T03:10:52.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 10 client.20808 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 10 client.20808 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 20 client.20808 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d3074d10 0x5617d3077100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 4 client.20808 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617d2fccb90 con 0x5617d3074d10 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.955+0000 7fd46623b640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d3074d10 0x5617d3077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd46623b640 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d3074d10 0x5617d3077100 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ed92bb18c2318c0 server_cookie=5f5a83fa61b4be51 in_seq=0 out_seq=0 2024-09-12T03:10:52.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd449ffb640 10 client.20808 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd449ffb640 1 -- 172.21.15.16:0/1592025355 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5617d2fccb90 con 0x5617d3074d10 2024-09-12T03:10:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd449ffb640 10 client.20808 handle_command_reply: tid=0 2024-09-12T03:10:52.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd46a878740 1 client.20808 shutdown 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 msgr2=0x5617d3062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.956+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b540 tx=0x7fd45000bee0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 shutdown_connections 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d3074d10 0x5617d3077100 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d3065240 0x5617d3062c90 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd449ffb640 0 client.20808 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a980 0x5617d3062750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 --2- 172.21.15.16:0/1592025355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a060 0x5617d3062210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 >> 172.21.15.16:0/1592025355 conn(0x5617d30565f0 msgr2=0x5617d305c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 shutdown_connections 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 1 -- 172.21.15.16:0/1592025355 wait complete. 2024-09-12T03:10:52.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.957+0000 7fd46a878740 20 client.20808 trim_cache size 0 max 16384 2024-09-12T03:10:52.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 Processor -- start 2024-09-12T03:10:52.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 -- start start 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a060 0x5617d305c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 0x5617d307cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601af700 con 0x5617d305a980 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601b1240 con 0x5617d3065240 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46a878740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601afdd0 con 0x5617d305a060 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60858/0 (socket says 172.21.15.16:60858) 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 -- 172.21.15.16:0/1449930017 learned_addr learned my addr 172.21.15.16:0/1449930017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a060 0x5617d305c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 -- 172.21.15.16:0/1449930017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a060 msgr2=0x5617d305c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a060 0x5617d305c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 0x5617d307cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 -- 172.21.15.16:0/1449930017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 msgr2=0x5617d307cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 0x5617d307cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 -- 172.21.15.16:0/1449930017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617d3059e90 con 0x5617d305a980 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd465a3a640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 0x5617d307cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.974+0000 7fd46623b640 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd45c004650 tx=0x7fd45c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd04cea21929978 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd44bfff640 1 -- 172.21.15.16:0/1449930017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c003ea0 con 0x5617d305a980 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd44bfff640 1 -- 172.21.15.16:0/1449930017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd45c011920 con 0x5617d305a980 2024-09-12T03:10:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd44bfff640 1 -- 172.21.15.16:0/1449930017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c01dab0 con 0x5617d305a980 2024-09-12T03:10:52.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 -- 172.21.15.16:0/1449930017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 msgr2=0x5617d307c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:52.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd45c004650 tx=0x7fd45c00b390 comp rx=0 tx=0).stop 2024-09-12T03:10:52.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 -- 172.21.15.16:0/1449930017 shutdown_connections 2024-09-12T03:10:52.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d3065240 0x5617d307cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a980 0x5617d307c870 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 --2- 172.21.15.16:0/1449930017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d305a060 0x5617d305c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 -- 172.21.15.16:0/1449930017 >> 172.21.15.16:0/1449930017 conn(0x5617d30565f0 msgr2=0x5617d305bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 -- 172.21.15.16:0/1449930017 shutdown_connections 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.975+0000 7fd46a878740 1 -- 172.21.15.16:0/1449930017 wait complete. 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 -- start start 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 Processor -- start 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a980 0x5617d306a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d3065240 0x5617d306a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601b0b70 con 0x5617d305a060 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601b04a0 con 0x5617d305a980 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601ad6e0 con 0x5617d3065240 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46a878740 10 client.0 _ll_register_callbacks cb 0x7fd450010d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60872/0 (socket says 172.21.15.16:60872) 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 -- 172.21.15.16:0/3935619872 learned_addr learned my addr 172.21.15.16:0/3935619872 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd46623b640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a980 0x5617d306a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d3065240 0x5617d306a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 -- 172.21.15.16:0/3935619872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d3065240 msgr2=0x5617d306a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d3065240 0x5617d306a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 -- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a980 msgr2=0x5617d306a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:10:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a980 0x5617d306a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.976+0000 7fd465a3a640 1 -- 172.21.15.16:0/3935619872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617d3057280 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd465a3a640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b510 tx=0x7fd45000bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305f79cf56ef1966 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45000b6d0 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd450004020 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd45c011920 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd4500108f0 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 10 client.20814 fetch_fsmap learned FSMap version 39 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5617d30e03e0 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd45c011920 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd45001c040 con 0x5617d305a060 2024-09-12T03:10:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 10 client.20814 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 10 client.20814 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 20 client.20814 populate_metadata read hostname 'smithi016' 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d30e0ad0 0x5617d30e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 4 client.20814 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b83f3b93", "release": true, "await": true}] 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.977+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b83f3b93", "release": true, "await": true}) -- 0x5617d2fc70a0 con 0x5617d30e0ad0 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.978+0000 7fd46623b640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d30e0ad0 0x5617d30e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.978+0000 7fd46623b640 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d30e0ad0 0x5617d30e2ef0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e794caba14cf11a server_cookie=f5f3be2ad2c8c7da in_seq=0 out_seq=0 2024-09-12T03:10:52.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:52.978+0000 7fd449ffb640 10 client.20814 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd449ffb640 1 -- 172.21.15.16:0/3935619872 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5617d2fc70a0 con 0x5617d30e0ad0 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd449ffb640 10 client.20814 handle_command_reply: tid=0 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 client.20814 shutdown 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 msgr2=0x5617d306d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b510 tx=0x7fd45000bee0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 shutdown_connections 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5617d30e0ad0 0x5617d30e2ef0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617d3065240 0x5617d306a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5617d305a980 0x5617d306a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 --2- 172.21.15.16:0/3935619872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617d305a060 0x5617d306d6d0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 >> 172.21.15.16:0/3935619872 conn(0x5617d30565f0 msgr2=0x5617d30611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.024+0000 7fd449ffb640 0 client.20814 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 shutdown_connections 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46a878740 1 -- 172.21.15.16:0/3935619872 wait complete. 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46a878740 20 client.20814 trim_cache size 0 max 16384 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 84, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "b83f3b93": { 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 84, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 150.0, 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:10:53.027 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:10:53.028 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 msgr2=0x7fd44c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 0x7fd44c0a50c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd458082530 tx=0x7fd45808f040 comp rx=0 tx=0).stop 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 msgr2=0x7fd460101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd45802ae70 tx=0x7fd45802f5b0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 shutdown_connections 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd44c0a2c00 0x7fd44c0a50c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd46010c170 0x7fd460101420 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd460103660 0x7fd460100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 --2- 172.21.15.16:0/2972701177 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd460102d40 0x7fd4601009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:10:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.025+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 >> 172.21.15.16:0/2972701177 conn(0x7fd4600fc7a0 msgr2=0x7fd4600fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:10:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.026+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 shutdown_connections 2024-09-12T03:10:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:10:53.026+0000 7fd46849f640 1 -- 172.21.15.16:0/2972701177 wait complete. 2024-09-12T03:10:53.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 231950 21.76 MB/sec execute 474 sec latency 30379.233 ms 2024-09-12T03:10:53.934 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b83f3b93', total seconds elapsed: 34.3 2024-09-12T03:10:54.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 232603 21.78 MB/sec execute 475 sec latency 29910.841 ms 2024-09-12T03:10:54.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:54 smithi033 ceph-mon[35092]: pgmap v1026: 97 pgs: 97 active+clean; 826 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.0 MiB/s wr, 76 op/s 2024-09-12T03:10:54.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:54 smithi184 ceph-mon[35321]: pgmap v1026: 97 pgs: 97 active+clean; 826 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.0 MiB/s wr, 76 op/s 2024-09-12T03:10:54.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:54 smithi016 ceph-mon[30925]: pgmap v1026: 97 pgs: 97 active+clean; 826 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.0 MiB/s wr, 76 op/s 2024-09-12T03:10:55.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 233258 21.82 MB/sec execute 476 sec latency 102.606 ms 2024-09-12T03:10:56.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 233944 21.84 MB/sec execute 477 sec latency 60.013 ms 2024-09-12T03:10:56.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:56 smithi033 ceph-mon[35092]: pgmap v1027: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-09-12T03:10:56.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:56 smithi184 ceph-mon[35321]: pgmap v1027: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-09-12T03:10:56.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:56 smithi016 ceph-mon[30925]: pgmap v1027: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-09-12T03:10:57.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 234715 21.89 MB/sec execute 478 sec latency 75.836 ms 2024-09-12T03:10:58.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 235298 21.92 MB/sec execute 479 sec latency 51.692 ms 2024-09-12T03:10:58.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:10:58 smithi033 ceph-mon[35092]: pgmap v1028: 97 pgs: 97 active+clean; 826 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 187 op/s 2024-09-12T03:10:58.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:10:58 smithi184 ceph-mon[35321]: pgmap v1028: 97 pgs: 97 active+clean; 826 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 187 op/s 2024-09-12T03:10:58.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:10:58 smithi016 ceph-mon[30925]: pgmap v1028: 97 pgs: 97 active+clean; 826 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 187 op/s 2024-09-12T03:10:59.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 235883 21.94 MB/sec execute 480 sec latency 61.377 ms 2024-09-12T03:11:00.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 236427 21.95 MB/sec execute 481 sec latency 58.891 ms 2024-09-12T03:11:00.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:00 smithi033 ceph-mon[35092]: pgmap v1029: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 208 op/s 2024-09-12T03:11:00.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:00 smithi184 ceph-mon[35321]: pgmap v1029: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 208 op/s 2024-09-12T03:11:00.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:00 smithi016 ceph-mon[30925]: pgmap v1029: 97 pgs: 97 active+clean; 825 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 208 op/s 2024-09-12T03:11:01.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 237066 21.97 MB/sec execute 482 sec latency 77.842 ms 2024-09-12T03:11:02.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 237757 22.00 MB/sec execute 483 sec latency 63.408 ms 2024-09-12T03:11:02.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:02 smithi033 ceph-mon[35092]: pgmap v1030: 97 pgs: 97 active+clean; 830 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.0 MiB/s wr, 368 op/s 2024-09-12T03:11:02.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:02 smithi184 ceph-mon[35321]: pgmap v1030: 97 pgs: 97 active+clean; 830 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.0 MiB/s wr, 368 op/s 2024-09-12T03:11:02.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:02 smithi016 ceph-mon[30925]: pgmap v1030: 97 pgs: 97 active+clean; 830 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 4.0 MiB/s wr, 368 op/s 2024-09-12T03:11:03.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 238608 22.05 MB/sec execute 484 sec latency 58.598 ms 2024-09-12T03:11:03.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:03.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:03.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:03 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:03.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:03 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:03.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:03 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:04.038 INFO:tasks.workunit.client.0.smithi016.stdout: 10 239318 22.09 MB/sec execute 485 sec latency 101.201 ms 2024-09-12T03:11:04.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:04 smithi033 ceph-mon[35092]: pgmap v1031: 97 pgs: 97 active+clean; 833 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-09-12T03:11:04.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:04 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:04.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:04 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:04.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:04 smithi184 ceph-mon[35321]: pgmap v1031: 97 pgs: 97 active+clean; 833 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-09-12T03:11:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:04 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:04.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:04 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:04.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:04 smithi016 ceph-mon[30925]: pgmap v1031: 97 pgs: 97 active+clean; 833 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-09-12T03:11:04.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:04 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:04.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:04 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:05.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 239802 22.11 MB/sec execute 486 sec latency 202.323 ms 2024-09-12T03:11:06.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 240240 22.11 MB/sec execute 487 sec latency 205.537 ms 2024-09-12T03:11:06.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:06 smithi033 ceph-mon[35092]: pgmap v1032: 97 pgs: 97 active+clean; 839 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.1 MiB/s wr, 431 op/s 2024-09-12T03:11:06.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:06 smithi184 ceph-mon[35321]: pgmap v1032: 97 pgs: 97 active+clean; 839 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.1 MiB/s wr, 431 op/s 2024-09-12T03:11:06.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:06 smithi016 ceph-mon[30925]: pgmap v1032: 97 pgs: 97 active+clean; 839 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.1 MiB/s wr, 431 op/s 2024-09-12T03:11:07.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 240917 22.14 MB/sec execute 488 sec latency 69.994 ms 2024-09-12T03:11:08.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 241520 22.15 MB/sec execute 489 sec latency 73.134 ms 2024-09-12T03:11:08.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:08 smithi033 ceph-mon[35092]: pgmap v1033: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2024-09-12T03:11:08.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:08 smithi184 ceph-mon[35321]: pgmap v1033: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2024-09-12T03:11:08.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:08 smithi016 ceph-mon[30925]: pgmap v1033: 97 pgs: 97 active+clean; 847 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2024-09-12T03:11:09.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 242045 22.15 MB/sec execute 490 sec latency 141.596 ms 2024-09-12T03:11:10.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 242930 22.20 MB/sec execute 491 sec latency 68.944 ms 2024-09-12T03:11:10.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:10 smithi033 ceph-mon[35092]: pgmap v1034: 97 pgs: 97 active+clean; 848 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-09-12T03:11:10.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:10 smithi184 ceph-mon[35321]: pgmap v1034: 97 pgs: 97 active+clean; 848 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-09-12T03:11:10.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:10 smithi016 ceph-mon[30925]: pgmap v1034: 97 pgs: 97 active+clean; 848 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-09-12T03:11:11.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 243743 22.26 MB/sec execute 492 sec latency 53.986 ms 2024-09-12T03:11:12.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 244260 22.28 MB/sec execute 493 sec latency 82.765 ms 2024-09-12T03:11:12.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:12 smithi033 ceph-mon[35092]: pgmap v1035: 97 pgs: 97 active+clean; 857 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 540 op/s 2024-09-12T03:11:12.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:12 smithi184 ceph-mon[35321]: pgmap v1035: 97 pgs: 97 active+clean; 857 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 540 op/s 2024-09-12T03:11:12.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:12 smithi016 ceph-mon[30925]: pgmap v1035: 97 pgs: 97 active+clean; 857 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 540 op/s 2024-09-12T03:11:13.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 244899 22.30 MB/sec execute 494 sec latency 66.635 ms 2024-09-12T03:11:14.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 245568 22.33 MB/sec execute 495 sec latency 43.298 ms 2024-09-12T03:11:14.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:14 smithi033 ceph-mon[35092]: pgmap v1036: 97 pgs: 97 active+clean; 864 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 484 op/s 2024-09-12T03:11:14.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:14 smithi184 ceph-mon[35321]: pgmap v1036: 97 pgs: 97 active+clean; 864 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 484 op/s 2024-09-12T03:11:14.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:14 smithi016 ceph-mon[30925]: pgmap v1036: 97 pgs: 97 active+clean; 864 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.7 MiB/s wr, 484 op/s 2024-09-12T03:11:15.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 246150 22.34 MB/sec execute 496 sec latency 56.316 ms 2024-09-12T03:11:16.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 246899 22.36 MB/sec execute 497 sec latency 57.378 ms 2024-09-12T03:11:16.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:16 smithi033 ceph-mon[35092]: pgmap v1037: 97 pgs: 97 active+clean; 866 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 457 op/s 2024-09-12T03:11:16.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:16 smithi184 ceph-mon[35321]: pgmap v1037: 97 pgs: 97 active+clean; 866 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 457 op/s 2024-09-12T03:11:16.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:16 smithi016 ceph-mon[30925]: pgmap v1037: 97 pgs: 97 active+clean; 866 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 457 op/s 2024-09-12T03:11:17.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 247732 22.42 MB/sec execute 498 sec latency 147.706 ms 2024-09-12T03:11:18.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 248463 22.47 MB/sec execute 499 sec latency 70.783 ms 2024-09-12T03:11:18.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:18 smithi033 ceph-mon[35092]: pgmap v1038: 97 pgs: 97 active+clean; 877 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-09-12T03:11:18.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:18 smithi184 ceph-mon[35321]: pgmap v1038: 97 pgs: 97 active+clean; 877 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-09-12T03:11:18.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:18 smithi016 ceph-mon[30925]: pgmap v1038: 97 pgs: 97 active+clean; 877 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-09-12T03:11:19.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 248993 22.48 MB/sec execute 500 sec latency 55.064 ms 2024-09-12T03:11:20.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 249629 22.50 MB/sec execute 501 sec latency 49.294 ms 2024-09-12T03:11:20.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:20 smithi033 ceph-mon[35092]: pgmap v1039: 97 pgs: 97 active+clean; 876 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-09-12T03:11:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:20 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:11:20.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:20 smithi184 ceph-mon[35321]: pgmap v1039: 97 pgs: 97 active+clean; 876 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-09-12T03:11:20.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:20 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:11:20.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:20 smithi016 ceph-mon[30925]: pgmap v1039: 97 pgs: 97 active+clean; 876 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-09-12T03:11:20.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:20 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:11:20.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:11:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 250290 22.51 MB/sec execute 502 sec latency 48.792 ms 2024-09-12T03:11:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:21.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:11:21.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:11:21.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:11:21.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:11:21.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:11:21.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:11:21.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:11:21.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:11:21.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:11:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 -- 172.21.15.16:0/3132613999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4101df0 msgr2=0x7f01f41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/3132613999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4101df0 0x7f01f41021d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009900 tx=0x7f01f002a410 comp rx=0 tx=0).stop 2024-09-12T03:11:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 -- 172.21.15.16:0/3132613999 shutdown_connections 2024-09-12T03:11:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/3132613999 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f4106f20 0x7f01f410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/3132613999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f41027a0 0x7f01f41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/3132613999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4101df0 0x7f01f41021d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 -- 172.21.15.16:0/3132613999 >> 172.21.15.16:0/3132613999 conn(0x7f01f40fd8f0 msgr2=0x7f01f40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 -- 172.21.15.16:0/3132613999 shutdown_connections 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 -- 172.21.15.16:0/3132613999 wait complete. 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.872+0000 7f01fbcbe640 1 Processor -- start 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 -- start start 2024-09-12T03:11:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 0x7f01f41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4106f20 0x7f01f419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4111f80 con 0x7f01f4106f20 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4111e00 con 0x7f01f4101df0 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fbcbe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4112100 con 0x7f01f41027a0 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 0x7f01f41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45278/0 (socket says 172.21.15.16:45278) 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01fa25b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4106f20 0x7f01f419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 0x7f01f41989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52320/0 (socket says 172.21.15.16:52320) 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 -- 172.21.15.16:0/1536326891 learned_addr learned my addr 172.21.15.16:0/1536326891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 -- 172.21.15.16:0/1536326891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 msgr2=0x7f01f41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 0x7f01f41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 -- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4106f20 msgr2=0x7f01f419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4106f20 0x7f01f419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.873+0000 7f01f9a5a640 1 -- 172.21.15.16:0/1536326891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f419d410 con 0x7f01f4101df0 2024-09-12T03:11:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006010 tx=0x7f01f0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01eaffd640 1 -- 172.21.15.16:0/1536326891 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f003f030 con 0x7f01f4101df0 2024-09-12T03:11:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f419d6a0 con 0x7f01f4101df0 2024-09-12T03:11:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f41a4fe0 con 0x7f01f4101df0 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01eaffd640 1 -- 172.21.15.16:0/1536326891 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f0004440 con 0x7f01f4101df0 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.874+0000 7f01eaffd640 1 -- 172.21.15.16:0/1536326891 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0035070 con 0x7f01f4101df0 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.876+0000 7f01eaffd640 1 -- 172.21.15.16:0/1536326891 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f01f0042030 con 0x7f01f4101df0 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 Processor -- start 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 -- start start 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01eaffd640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 0x7f01e00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afc4a0 0x55afd0afc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b09a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55afd0a622e0 con 0x55afd0b07680 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55afd0a09ce0 con 0x55afd0afc4a0 2024-09-12T03:11:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01fe097740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55afd0a5e2a0 con 0x55afd0afcdc0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01eaffd640 1 -- 172.21.15.16:0/1536326891 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01f00ef2e0 con 0x7f01f4101df0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52330/0 (socket says 172.21.15.16:52330) 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.877+0000 7f01f9259640 1 -- 172.21.15.16:0/2931255930 learned_addr learned my addr 172.21.15.16:0/2931255930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01fa25b640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 0x7f01e00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01fa25b640 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b09a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afc4a0 0x55afd0afc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 -- 172.21.15.16:0/2931255930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afc4a0 msgr2=0x55afd0afc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afc4a0 0x55afd0afc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 -- 172.21.15.16:0/2931255930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 msgr2=0x55afd0b09a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b09a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 -- 172.21.15.16:0/2931255930 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd09ee440 con 0x55afd0afcdc0 2024-09-12T03:11:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01f9259640 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009c90 tx=0x7f01e4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b692323715f7715c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01fa25b640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 0x7f01e00a50c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55afd0af9db0 tx=0x7f01ec009b70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01dffff640 1 -- 172.21.15.16:0/2931255930 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e401e070 con 0x55afd0afcdc0 2024-09-12T03:11:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01dffff640 1 -- 172.21.15.16:0/2931255930 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01e4019b10 con 0x55afd0afcdc0 2024-09-12T03:11:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.878+0000 7f01dffff640 1 -- 172.21.15.16:0/2931255930 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e400eaf0 con 0x55afd0afcdc0 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 -- 172.21.15.16:0/2931255930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 msgr2=0x55afd0b07140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009c90 tx=0x7f01e4011810 comp rx=0 tx=0).stop 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 -- 172.21.15.16:0/2931255930 shutdown_connections 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b09a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b07140 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 --2- 172.21.15.16:0/2931255930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afc4a0 0x55afd0afc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.879+0000 7f01fe097740 1 -- 172.21.15.16:0/2931255930 >> 172.21.15.16:0/2931255930 conn(0x55afd0af8a30 msgr2=0x55afd0af8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.880+0000 7f01fe097740 1 -- 172.21.15.16:0/2931255930 shutdown_connections 2024-09-12T03:11:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.880+0000 7f01fe097740 1 -- 172.21.15.16:0/2931255930 wait complete. 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.880+0000 7f01fe097740 1 -- start start 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.880+0000 7f01fe097740 1 Processor -- start 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 0x55afd0b06f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 0x55afd0b041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55afd0b0a250 con 0x55afd0b07680 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55afd0a1d050 con 0x55afd0afcdc0 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55afd0b0a3d0 con 0x55afd0afc4a0 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 0x55afd0b06f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fa25b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fa25b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45012/0 (socket says 172.21.15.16:45012) 2024-09-12T03:11:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fa25b640 1 -- 172.21.15.16:0/3962702737 learned_addr learned my addr 172.21.15.16:0/3962702737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fe097740 10 client.0 _ll_register_callbacks cb 0x55afd0a5d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 -- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 msgr2=0x55afd0b041b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9259640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 0x55afd0b041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 0x55afd0b041b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 -- 172.21.15.16:0/3962702737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 msgr2=0x55afd0b046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9a5a640 1 -- 172.21.15.16:0/3962702737 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd07d33c0 con 0x55afd0afc4a0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01fa25b640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.881+0000 7f01f9259640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 0x55afd0b041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 0x55afd0b06f00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f01f00ecd90 tx=0x7f01f00bb6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a1b5e858d4a0cb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01e4019b10 con 0x55afd0afc4a0 2024-09-12T03:11:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01dd7fa640 1 -- 172.21.15.16:0/3962702737 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0049040 con 0x55afd0afc4a0 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01dd7fa640 1 -- 172.21.15.16:0/3962702737 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01f0087770 con 0x55afd0afc4a0 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01dd7fa640 1 -- 172.21.15.16:0/3962702737 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f01f00d9050 con 0x55afd0afc4a0 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01fe097740 10 client.29920 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.882+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55afd0b0a310 con 0x55afd0afc4a0 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01dd7fa640 1 -- 172.21.15.16:0/3962702737 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f01f00bbe00 con 0x55afd0afc4a0 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 10 client.29920 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 10 client.29920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 20 client.29920 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b170f0 0x55afd0b195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 4 client.29920 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd0a6f850 con 0x55afd0b170f0 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01f9259640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b170f0 0x55afd0b195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01dd7fa640 10 client.29920 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.883+0000 7f01f9259640 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b170f0 0x55afd0b195b0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dccd93d50619416 server_cookie=dde80a55e26072f8 in_seq=0 out_seq=0 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.884+0000 7f01dd7fa640 1 -- 172.21.15.16:0/3962702737 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55afd0a6f850 con 0x55afd0b170f0 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.884+0000 7f01dd7fa640 10 client.29920 handle_command_reply: tid=0 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.884+0000 7f01fe097740 1 client.29920 shutdown 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 msgr2=0x55afd0b06f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 0x55afd0b06f00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f01f00ecd90 tx=0x7f01f00bb6e0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 shutdown_connections 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b170f0 0x55afd0b195b0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0b07680 0x55afd0b046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0afcdc0 0x55afd0b041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 --2- 172.21.15.16:0/3962702737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afc4a0 0x55afd0b06f00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 >> 172.21.15.16:0/3962702737 conn(0x55afd0af8a30 msgr2=0x55afd0af7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01dd7fa640 0 client.29920 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 shutdown_connections 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.885+0000 7f01fe097740 1 -- 172.21.15.16:0/3962702737 wait complete. 2024-09-12T03:11:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.886+0000 7f01fe097740 20 client.29920 trim_cache size 0 max 16384 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 Processor -- start 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 -- start start 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0afe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b1eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800e280 con 0x55afd0afc4a0 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800fdc0 con 0x55afd0b07680 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fe097740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800e950 con 0x55afd0afcdc0 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45018/0 (socket says 172.21.15.16:45018) 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0afe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01f9a5a640 1 -- 172.21.15.16:0/615814884 learned_addr learned my addr 172.21.15.16:0/615814884 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.904+0000 7f01fa25b640 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b1eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 -- 172.21.15.16:0/615814884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 msgr2=0x55afd0afe600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0afe600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 -- 172.21.15.16:0/615814884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 msgr2=0x55afd0b1eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b1eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 -- 172.21.15.16:0/615814884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd0b04f70 con 0x55afd0afc4a0 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9259640 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0afe600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f01f00c4230 tx=0x7f01f00a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d6c261738e6c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01df7fe640 1 -- 172.21.15.16:0/615814884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0049040 con 0x55afd0afc4a0 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01df7fe640 1 -- 172.21.15.16:0/615814884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01e4019b10 con 0x55afd0afc4a0 2024-09-12T03:11:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.905+0000 7f01df7fe640 1 -- 172.21.15.16:0/615814884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f00da240 con 0x55afd0afc4a0 2024-09-12T03:11:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- 172.21.15.16:0/615814884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 msgr2=0x55afd0b13540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f01f00c4230 tx=0x7f01f00a6000 comp rx=0 tx=0).stop 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- 172.21.15.16:0/615814884 shutdown_connections 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b1eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0afe600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 --2- 172.21.15.16:0/615814884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b13540 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- 172.21.15.16:0/615814884 >> 172.21.15.16:0/615814884 conn(0x55afd0af8a30 msgr2=0x55afd0afdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- 172.21.15.16:0/615814884 shutdown_connections 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- 172.21.15.16:0/615814884 wait complete. 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 -- start start 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.906+0000 7f01fe097740 1 Processor -- start 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b0b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b0bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800f6f0 con 0x55afd0afc4a0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800f020 con 0x55afd0b07680 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01d800c260 con 0x55afd0afcdc0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fe097740 10 client.0 _ll_register_callbacks cb 0x7f01f0005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9259640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b0b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9a5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45028/0 (socket says 172.21.15.16:45028) 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9a5a640 1 -- 172.21.15.16:0/2847791742 learned_addr learned my addr 172.21.15.16:0/2847791742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01fa25b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b0bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9259640 1 -- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 msgr2=0x55afd0b0bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9259640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b0bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9259640 1 -- 172.21.15.16:0/2847791742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 msgr2=0x55afd0b0b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.907+0000 7f01f9259640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01f9259640 1 -- 172.21.15.16:0/2847791742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd0b852a0 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01f9259640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b0b860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x55afd0b03a40 tx=0x7f01e4011e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bb9172a1a5c8469 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e401e040 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01e4019b10 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f01e4019630 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f01e4005c50 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01fe097740 10 client.29926 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55afd0b864b0 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.908+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01e4019b10 con 0x55afd0afcdc0 2024-09-12T03:11:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f01e400e990 con 0x55afd0afcdc0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 10 client.29926 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 10 client.29926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 20 client.29926 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b86aa0 0x55afd0b88ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 4 client.29926 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55afd0a69d60 con 0x55afd0b86aa0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b86aa0 0x55afd0b88ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01f9a5a640 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b86aa0 0x55afd0b88ec0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a62f6084495d55 server_cookie=ef5f6047ffda747c in_seq=0 out_seq=0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.909+0000 7f01dd7fa640 10 client.29926 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01dd7fa640 1 -- 172.21.15.16:0/2847791742 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55afd0a69d60 con 0x55afd0b86aa0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01dd7fa640 10 client.29926 handle_command_reply: tid=0 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 client.29926 shutdown 2024-09-12T03:11:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 msgr2=0x55afd0b0b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b0b860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x55afd0b03a40 tx=0x7f01e4011e40 comp rx=0 tx=0).stop 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 shutdown_connections 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55afd0b86aa0 0x55afd0b88ec0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55afd0b07680 0x55afd0b0bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55afd0afcdc0 0x55afd0b0b860 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 --2- 172.21.15.16:0/2847791742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55afd0afc4a0 0x55afd0b0b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 >> 172.21.15.16:0/2847791742 conn(0x55afd0af8a30 msgr2=0x55afd0afe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01dd7fa640 0 client.29926 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 shutdown_connections 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 1 -- 172.21.15.16:0/2847791742 wait complete. 2024-09-12T03:11:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.910+0000 7f01fe097740 20 client.29926 trim_cache size 0 max 16384 2024-09-12T03:11:21.921 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:11:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:11:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:11:21.921 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 msgr2=0x7f01e00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 0x7f01e00a50c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55afd0af9db0 tx=0x7f01ec009b70 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 msgr2=0x7f01f4198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006010 tx=0x7f01f0004040 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 shutdown_connections 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f01e00a2c00 0x7f01e00a50c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f4106f20 0x7f01f419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01f41027a0 0x7f01f41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 --2- 172.21.15.16:0/1536326891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f01f4101df0 0x7f01f4198460 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.911+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 >> 172.21.15.16:0/1536326891 conn(0x7f01f40fd8f0 msgr2=0x7f01f40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.912+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 shutdown_connections 2024-09-12T03:11:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:21.912+0000 7f01fbcbe640 1 -- 172.21.15.16:0/1536326891 wait complete. 2024-09-12T03:11:22.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 250812 22.53 MB/sec execute 503 sec latency 132.966 ms 2024-09-12T03:11:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:22 smithi184 ceph-mon[35321]: pgmap v1040: 97 pgs: 97 active+clean; 866 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 560 op/s 2024-09-12T03:11:22.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:22 smithi016 ceph-mon[30925]: pgmap v1040: 97 pgs: 97 active+clean; 866 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 560 op/s 2024-09-12T03:11:22.726 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:11:22.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:11:22.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:22 smithi033 ceph-mon[35092]: pgmap v1040: 97 pgs: 97 active+clean; 866 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.4 MiB/s wr, 560 op/s 2024-09-12T03:11:23.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 251576 22.56 MB/sec execute 504 sec latency 67.600 ms 2024-09-12T03:11:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 -- 172.21.15.16:0/4109425440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c102570 msgr2=0x7fab4c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 --2- 172.21.15.16:0/4109425440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c102570 0x7fab4c1029b0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fab40009900 tx=0x7fab4002a480 comp rx=0 tx=0).stop 2024-09-12T03:11:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 -- 172.21.15.16:0/4109425440 shutdown_connections 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 --2- 172.21.15.16:0/4109425440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102ef0 0x7fab4c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 --2- 172.21.15.16:0/4109425440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c102570 0x7fab4c1029b0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 --2- 172.21.15.16:0/4109425440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c100e20 0x7fab4c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 -- 172.21.15.16:0/4109425440 >> 172.21.15.16:0/4109425440 conn(0x7fab4c0fc7d0 msgr2=0x7fab4c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 -- 172.21.15.16:0/4109425440 shutdown_connections 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.619+0000 7fab521f0640 1 -- 172.21.15.16:0/4109425440 wait complete. 2024-09-12T03:11:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.620+0000 7fab521f0640 1 Processor -- start 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.620+0000 7fab521f0640 1 -- start start 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.620+0000 7fab521f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.620+0000 7fab521f0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102570 0x7fab4c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab521f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c102ef0 0x7fab4c1a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab521f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c110f80 con 0x7fab4c100e20 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab521f0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c110e00 con 0x7fab4c102570 2024-09-12T03:11:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab521f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c111100 con 0x7fab4c102ef0 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102570 0x7fab4c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45042/0 (socket says 172.21.15.16:45042) 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 -- 172.21.15.16:0/630471900 learned_addr learned my addr 172.21.15.16:0/630471900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4bfff640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c102ef0 0x7fab4c1a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 -- 172.21.15.16:0/630471900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c102ef0 msgr2=0x7fab4c1a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c102ef0 0x7fab4c1a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 -- 172.21.15.16:0/630471900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102570 msgr2=0x7fab4c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102570 0x7fab4c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 -- 172.21.15.16:0/630471900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c1a48b0 con 0x7fab4c100e20 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.621+0000 7fab4b7fe640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00e9f0 tx=0x7fab3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.622+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00cd80 con 0x7fab4c100e20 2024-09-12T03:11:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.622+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4c1a4a80 con 0x7fab4c100e20 2024-09-12T03:11:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.622+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab3c010910 con 0x7fab4c100e20 2024-09-12T03:11:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.622+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c0188f0 con 0x7fab4c100e20 2024-09-12T03:11:23.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.622+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab4c1a4f40 con 0x7fab4c100e20 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.623+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4c101200 con 0x7fab4c100e20 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.624+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fab3c010ac0 con 0x7fab4c100e20 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.625+0000 7fab48ff9640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 0x7fab340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.625+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fab3c0cf3c0 con 0x7fab4c100e20 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.625+0000 7fab4affd640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 0x7fab340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:23.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.626+0000 7fab4affd640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 0x7fab340a5040 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fab40009230 tx=0x7fab400058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.630+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab3c09bce0 con 0x7fab4c100e20 2024-09-12T03:11:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.932+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab4c1a5170 con 0x7fab4c100e20 2024-09-12T03:11:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.933+0000 7fab48ff9640 1 -- 172.21.15.16:0/630471900 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fab3c09ffe0 con 0x7fab4c100e20 2024-09-12T03:11:23.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:11:23.934 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:11:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:11:23.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 msgr2=0x7fab340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 0x7fab340a5040 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fab40009230 tx=0x7fab400058d0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 msgr2=0x7fab4c1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00e9f0 tx=0x7fab3c00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 shutdown_connections 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab4c102ef0 0x7fab4c1a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab4c102570 0x7fab4c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.935+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fab340a2b80 0x7fab340a5040 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.936+0000 7fab521f0640 1 --2- 172.21.15.16:0/630471900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4c100e20 0x7fab4c1992f0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.936+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 >> 172.21.15.16:0/630471900 conn(0x7fab4c0fc7d0 msgr2=0x7fab4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.936+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 shutdown_connections 2024-09-12T03:11:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:23.936+0000 7fab521f0640 1 -- 172.21.15.16:0/630471900 wait complete. 2024-09-12T03:11:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 252342 22.60 MB/sec execute 505 sec latency 69.425 ms 2024-09-12T03:11:24.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:24 smithi184 ceph-mon[35321]: pgmap v1041: 97 pgs: 97 active+clean; 870 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 499 op/s 2024-09-12T03:11:24.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:24 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/630471900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:11:24.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:24 smithi016 ceph-mon[30925]: pgmap v1041: 97 pgs: 97 active+clean; 870 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 499 op/s 2024-09-12T03:11:24.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:24 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/630471900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:11:24.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:24 smithi033 ceph-mon[35092]: pgmap v1041: 97 pgs: 97 active+clean; 870 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 499 op/s 2024-09-12T03:11:24.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:24 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/630471900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:11:25.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:11:25.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 252917 22.62 MB/sec execute 506 sec latency 98.991 ms 2024-09-12T03:11:25.335 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 -- 172.21.15.16:0/3152110786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 msgr2=0x7f09981029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 --2- 172.21.15.16:0/3152110786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981029b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0988009890 tx=0x7f098802a410 comp rx=0 tx=0).stop 2024-09-12T03:11:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 -- 172.21.15.16:0/3152110786 shutdown_connections 2024-09-12T03:11:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 --2- 172.21.15.16:0/3152110786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f099810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 --2- 172.21.15.16:0/3152110786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981029b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 --2- 172.21.15.16:0/3152110786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 -- 172.21.15.16:0/3152110786 >> 172.21.15.16:0/3152110786 conn(0x7f09980fc7d0 msgr2=0x7f09980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.927+0000 7f09a039a640 1 -- 172.21.15.16:0/3152110786 shutdown_connections 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.928+0000 7f09a039a640 1 -- 172.21.15.16:0/3152110786 wait complete. 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.928+0000 7f09a039a640 1 Processor -- start 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.928+0000 7f09a039a640 1 -- start start 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f09981a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46548/0 (socket says 172.21.15.16:46548) 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e136640 1 -- 172.21.15.16:0/212220724 learned_addr learned my addr 172.21.15.16:0/212220724 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099d935640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110f50 con 0x7f0998100e20 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110dd0 con 0x7f0998102ef0 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09981110d0 con 0x7f0998102570 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f09981a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 -- 172.21.15.16:0/212220724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 msgr2=0x7f09981998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 -- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 msgr2=0x7f0998199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.929+0000 7f099e937640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0998197150 con 0x7f0998102ef0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f099e937640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f09981a43e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0990009fd0 tx=0x7f099000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f098f7fe640 1 -- 172.21.15.16:0/212220724 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099000ce20 con 0x7f0998102ef0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f098f7fe640 1 -- 172.21.15.16:0/212220724 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0990005020 con 0x7f0998102ef0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0998197440 con 0x7f0998102ef0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f098f7fe640 1 -- 172.21.15.16:0/212220724 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990010720 con 0x7f0998102ef0 2024-09-12T03:11:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.930+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0998197930 con 0x7f0998102ef0 2024-09-12T03:11:25.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f098f7fe640 1 -- 172.21.15.16:0/212220724 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0990005540 con 0x7f0998102ef0 2024-09-12T03:11:25.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 Processor -- start 2024-09-12T03:11:25.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 -- start start 2024-09-12T03:11:25.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2a3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 0x564e0b2a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e0b2092f0 con 0x564e0b2a4b40 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e0b1b0cf0 con 0x564e0b2a35d0 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f09a2773740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e0b2052b0 con 0x564e0b2ae370 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f099d935640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f099e136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 0x564e0b2a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f099d935640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2a3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46550/0 (socket says 172.21.15.16:46550) 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.933+0000 7f099d935640 1 -- 172.21.15.16:0/1271250072 learned_addr learned my addr 172.21.15.16:0/1271250072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099d935640 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 -- 172.21.15.16:0/1271250072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 msgr2=0x564e0b2b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 -- 172.21.15.16:0/1271250072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 msgr2=0x564e0b2a3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2a3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 -- 172.21.15.16:0/1271250072 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0b1953a0 con 0x564e0b2a35d0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099d935640 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f098f7fe640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 0x7f09840a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099d935640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 0x7f09840a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f098f7fe640 1 -- 172.21.15.16:0/212220724 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09900cf6b0 con 0x7f0998102ef0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f099e136640 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 0x564e0b2a3a10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0994009cc0 tx=0x7f0994011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d5569d3afd87306 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f098cdea640 1 -- 172.21.15.16:0/1271250072 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994004230 con 0x564e0b2a35d0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.934+0000 7f098cdea640 1 -- 172.21.15.16:0/1271250072 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09940043d0 con 0x564e0b2a35d0 2024-09-12T03:11:25.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f098cdea640 1 -- 172.21.15.16:0/1271250072 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994004e10 con 0x564e0b2a35d0 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 -- 172.21.15.16:0/1271250072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 msgr2=0x564e0b2a3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f099d935640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 0x7f09840a5040 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f098802f4f0 tx=0x7f0988041040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 0x564e0b2a3a10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0994009cc0 tx=0x7f0994011810 comp rx=0 tx=0).stop 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 -- 172.21.15.16:0/1271250072 shutdown_connections 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a35d0 0x564e0b2a3a10 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0994009cc0 tx=0x7f0994011810 comp rx=0 tx=0).stop 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 --2- 172.21.15.16:0/1271250072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2a3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 -- 172.21.15.16:0/1271250072 >> 172.21.15.16:0/1271250072 conn(0x564e0b26c310 msgr2=0x564e0b29fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.935+0000 7f09a2773740 1 -- 172.21.15.16:0/1271250072 shutdown_connections 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 -- 172.21.15.16:0/1271250072 wait complete. 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 -- start start 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 Processor -- start 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 0x564e0b2ab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564e0b2b1110 con 0x564e0b2abcb0 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e0b1c4060 con 0x564e0b2a4b40 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564e0b2b1290 con 0x564e0b2ae370 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f09a2773740 10 client.0 _ll_register_callbacks cb 0x564e0af178c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f099e937640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f099d935640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f099e136640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 0x564e0b2ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f099e937640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46552/0 (socket says 172.21.15.16:46552) 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.936+0000 7f099e937640 1 -- 172.21.15.16:0/4197862241 learned_addr learned my addr 172.21.15.16:0/4197862241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 -- 172.21.15.16:0/4197862241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 msgr2=0x564e0b2ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 -- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 msgr2=0x564e0b2ba5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 -- 172.21.15.16:0/4197862241 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0af7a3c0 con 0x564e0b2a4b40 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f099e136640 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 0x564e0b2ab230 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x564e0b2a40f0 tx=0x7f0994003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd77d98398f7424a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f0981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994028020 con 0x564e0b2a4b40 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0994003e20 con 0x564e0b2a4b40 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09940043d0 con 0x564e0b2a4b40 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0994009e70 con 0x564e0b2a4b40 2024-09-12T03:11:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f09a2773740 10 client.30002 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.937+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564e0b2bd1b0 con 0x564e0b2a4b40 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09940043d0 con 0x564e0b2a4b40 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f0994018070 con 0x564e0b2a4b40 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 10 client.30002 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 10 client.30002 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 20 client.30002 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b2be500 0x564e0b2c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 4 client.30002 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e0b216860 con 0x564e0b2be500 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.938+0000 7f099d935640 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b2be500 0x564e0b2c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.939+0000 7f0981ffb640 10 client.30002 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.939+0000 7f099d935640 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b2be500 0x564e0b2c08f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9c9d5549c821f195 server_cookie=eacca5b897f913d0 in_seq=0 out_seq=0 2024-09-12T03:11:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.939+0000 7f0981ffb640 1 -- 172.21.15.16:0/4197862241 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e0b216860 con 0x564e0b2be500 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.939+0000 7f0981ffb640 10 client.30002 handle_command_reply: tid=0 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.939+0000 7f09a2773740 1 client.30002 shutdown 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 msgr2=0x564e0b2ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 0x564e0b2ab230 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x564e0b2a40f0 tx=0x7f0994003820 comp rx=0 tx=0).stop 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 shutdown_connections 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b2be500 0x564e0b2c08f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 --2- 172.21.15.16:0/4197862241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2a4b40 0x564e0b2ab230 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 >> 172.21.15.16:0/4197862241 conn(0x564e0b26c310 msgr2=0x564e0b2a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f0981ffb640 0 client.30002 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 shutdown_connections 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 1 -- 172.21.15.16:0/4197862241 wait complete. 2024-09-12T03:11:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.940+0000 7f09a2773740 20 client.30002 trim_cache size 0 max 16384 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 Processor -- start 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 -- start start 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ba8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09981af660 con 0x564e0b2a4b40 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09981b11a0 con 0x564e0b2abcb0 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.960+0000 7f09a2773740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09981afd30 con 0x564e0b2ae370 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46554/0 (socket says 172.21.15.16:46554) 2024-09-12T03:11:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 -- 172.21.15.16:0/3103236428 learned_addr learned my addr 172.21.15.16:0/3103236428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e937640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ba8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099d935640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 -- 172.21.15.16:0/3103236428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 msgr2=0x564e0b2ba8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ba8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 -- 172.21.15.16:0/3103236428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 msgr2=0x564e0b2ba3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 -- 172.21.15.16:0/3103236428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0b2a0960 con 0x564e0b2a4b40 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099d935640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.961+0000 7f099e136640 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f0994011850 tx=0x7f0994009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f99d4d97a648e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.962+0000 7f0983fff640 1 -- 172.21.15.16:0/3103236428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994028040 con 0x564e0b2a4b40 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.962+0000 7f0983fff640 1 -- 172.21.15.16:0/3103236428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099401f040 con 0x564e0b2a4b40 2024-09-12T03:11:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.962+0000 7f0983fff640 1 -- 172.21.15.16:0/3103236428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994004520 con 0x564e0b2a4b40 2024-09-12T03:11:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 -- 172.21.15.16:0/3103236428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 msgr2=0x564e0b2aab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f0994011850 tx=0x7f0994009260 comp rx=0 tx=0).stop 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 -- 172.21.15.16:0/3103236428 shutdown_connections 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2ae370 0x564e0b2ba8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2abcb0 0x564e0b2ba3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 --2- 172.21.15.16:0/3103236428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2aab10 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 -- 172.21.15.16:0/3103236428 >> 172.21.15.16:0/3103236428 conn(0x564e0b26c310 msgr2=0x564e0b29ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 -- 172.21.15.16:0/3103236428 shutdown_connections 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.963+0000 7f09a2773740 1 -- 172.21.15.16:0/3103236428 wait complete. 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.964+0000 7f09a2773740 1 -- start start 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.964+0000 7f09a2773740 1 Processor -- start 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.964+0000 7f09a2773740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2abcb0 0x564e0b2b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 0x564e0b2aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09981b0ad0 con 0x564e0b2a4b40 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09981b0400 con 0x564e0b2ae370 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09981ad620 con 0x564e0b2abcb0 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46558/0 (socket says 172.21.15.16:46558) 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e136640 1 -- 172.21.15.16:0/4158857382 learned_addr learned my addr 172.21.15.16:0/4158857382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f09a2773740 10 client.0 _ll_register_callbacks cb 0x7f09900093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 0x564e0b2aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099d935640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2abcb0 0x564e0b2b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 -- 172.21.15.16:0/4158857382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2abcb0 msgr2=0x564e0b2b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2abcb0 0x564e0b2b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 -- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 msgr2=0x564e0b2b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.965+0000 7f099e937640 1 -- 172.21.15.16:0/4158857382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0b2a6120 con 0x564e0b2ae370 2024-09-12T03:11:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f099e937640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 0x564e0b2aed30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09900a4660 tx=0x7f099009c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7faae95b5f5118 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f0981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09900a4b90 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f09900a4700 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099401f040 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f09900a4900 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f09a2773740 10 client.30008 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564e0b2b5f20 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.966+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099401f040 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f099009ade0 con 0x564e0b2ae370 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 10 client.30008 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 10 client.30008 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 20 client.30008 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b329060 0x564e0b32b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 4 client.30008 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e0b2abb00 con 0x564e0b329060 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f099e136640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b329060 0x564e0b32b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f0981ffb640 10 client.30008 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.967+0000 7f099e136640 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b329060 0x564e0b32b480 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=55094463d2ae4ebb server_cookie=34dce77ed251a13c in_seq=0 out_seq=0 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f0981ffb640 1 -- 172.21.15.16:0/4158857382 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e0b2abb00 con 0x564e0b329060 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f0981ffb640 10 client.30008 handle_command_reply: tid=0 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 client.30008 shutdown 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 msgr2=0x564e0b2aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 0x564e0b2aed30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09900a4660 tx=0x7f099009c6e0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 shutdown_connections 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x564e0b329060 0x564e0b32b480 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f0981ffb640 0 client.30008 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:11:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e0b2ae370 0x564e0b2aed30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564e0b2abcb0 0x564e0b2b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 --2- 172.21.15.16:0/4158857382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564e0b2a4b40 0x564e0b2b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 >> 172.21.15.16:0/4158857382 conn(0x564e0b26c310 msgr2=0x564e0b2a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 shutdown_connections 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 1 -- 172.21.15.16:0/4158857382 wait complete. 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.968+0000 7f09a2773740 20 client.30008 trim_cache size 0 max 16384 2024-09-12T03:11:25.974 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 msgr2=0x7f09840a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 0x7f09840a5040 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f098802f4f0 tx=0x7f0988041040 comp rx=0 tx=0).stop 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 msgr2=0x7f09981a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f09981a43e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0990009fd0 tx=0x7f099000ef10 comp rx=0 tx=0).stop 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 shutdown_connections 2024-09-12T03:11:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0998102ef0 0x7f09981a43e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0998102570 0x7f09981998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f09840a2b80 0x7f09840a5040 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 --2- 172.21.15.16:0/212220724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0998100e20 0x7f0998199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 >> 172.21.15.16:0/212220724 conn(0x7f09980fc7d0 msgr2=0x7f09981048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.969+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 shutdown_connections 2024-09-12T03:11:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:25.970+0000 7f09a039a640 1 -- 172.21.15.16:0/212220724 wait complete. 2024-09-12T03:11:26.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 253498 22.65 MB/sec execute 507 sec latency 72.781 ms 2024-09-12T03:11:26.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:26 smithi184 ceph-mon[35321]: pgmap v1042: 97 pgs: 97 active+clean; 873 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 452 op/s 2024-09-12T03:11:26.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:26 smithi016 ceph-mon[30925]: pgmap v1042: 97 pgs: 97 active+clean; 873 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 452 op/s 2024-09-12T03:11:26.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:11:26.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:26 smithi033 ceph-mon[35092]: pgmap v1042: 97 pgs: 97 active+clean; 873 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 452 op/s 2024-09-12T03:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:27.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 254083 22.67 MB/sec execute 508 sec latency 57.070 ms 2024-09-12T03:11:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 -- 172.21.15.16:0/1649469976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480102ef0 msgr2=0x7fc48010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 --2- 172.21.15.16:0/1649469976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480102ef0 0x7fc48010da40 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fc478009ca0 tx=0x7fc47802a950 comp rx=0 tx=0).stop 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 -- 172.21.15.16:0/1649469976 shutdown_connections 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 --2- 172.21.15.16:0/1649469976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480102ef0 0x7fc48010da40 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 --2- 172.21.15.16:0/1649469976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 --2- 172.21.15.16:0/1649469976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480100e20 0x7fc480101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 -- 172.21.15.16:0/1649469976 >> 172.21.15.16:0/1649469976 conn(0x7fc4800fc7d0 msgr2=0x7fc4800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 -- 172.21.15.16:0/1649469976 shutdown_connections 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.552+0000 7fc48807e640 1 -- 172.21.15.16:0/1649469976 wait complete. 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 Processor -- start 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 -- start start 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 0x7fc480198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 0x7fc48019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc480110f80 con 0x7fc480100e20 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc480110e00 con 0x7fc480102ef0 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48807e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc480111100 con 0x7fc480102570 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 0x7fc480198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc485619640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 0x7fc480198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46584/0 (socket says 172.21.15.16:46584) 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc485e1a640 1 -- 172.21.15.16:0/858344413 learned_addr learned my addr 172.21.15.16:0/858344413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.553+0000 7fc48661b640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 0x7fc48019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 -- 172.21.15.16:0/858344413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 msgr2=0x7fc48019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 0x7fc48019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 -- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 msgr2=0x7fc480198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 0x7fc480198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 -- 172.21.15.16:0/858344413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48019d410 con 0x7fc480102570 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc48661b640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 0x7fc48019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc485619640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801989a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc470009bc0 tx=0x7fc47000b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc476ffd640 1 -- 172.21.15.16:0/858344413 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47001e070 con 0x7fc480102570 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48019d700 con 0x7fc480102570 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc476ffd640 1 -- 172.21.15.16:0/858344413 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4700198f0 con 0x7fc480102570 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc476ffd640 1 -- 172.21.15.16:0/858344413 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4700187f0 con 0x7fc480102570 2024-09-12T03:11:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.554+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc480107a10 con 0x7fc480102570 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.557+0000 7fc48a457740 1 Processor -- start 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.557+0000 7fc476ffd640 1 -- 172.21.15.16:0/858344413 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc470021070 con 0x7fc480102570 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 -- start start 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0aa7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ab4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560fc0a0d2f0 con 0x560fc0ab2370 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fc09b4cf0 con 0x560fc0aa75d0 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48a457740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560fc0a092b0 con 0x560fc0aa8b40 2024-09-12T03:11:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48661b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0aa7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48661b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0aa7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47774/0 (socket says 172.21.15.16:47774) 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc48661b640 1 -- 172.21.15.16:0/3764547938 learned_addr learned my addr 172.21.15.16:0/3764547938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33832/0 (socket says 172.21.15.16:33832) 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.558+0000 7fc485619640 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ab4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 -- 172.21.15.16:0/3764547938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 msgr2=0x560fc0aa7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0aa7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 -- 172.21.15.16:0/3764547938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 msgr2=0x560fc0ab4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ab4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 -- 172.21.15.16:0/3764547938 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc09993a0 con 0x560fc0aa75d0 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc476ffd640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 0x7fc46c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc48661b640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 0x7fc46c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc485e1a640 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009c80 tx=0x7fc47c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceb3dba2c9654aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc476ffd640 1 -- 172.21.15.16:0/858344413 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc4700d27d0 con 0x7fc480102570 2024-09-12T03:11:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc46bfff640 1 -- 172.21.15.16:0/3764547938 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c004230 con 0x560fc0aa75d0 2024-09-12T03:11:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.559+0000 7fc46bfff640 1 -- 172.21.15.16:0/3764547938 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c0043d0 con 0x560fc0aa75d0 2024-09-12T03:11:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc46bfff640 1 -- 172.21.15.16:0/3764547938 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c004df0 con 0x560fc0aa75d0 2024-09-12T03:11:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 -- 172.21.15.16:0/3764547938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 msgr2=0x560fc0aa7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009c80 tx=0x7fc47c011810 comp rx=0 tx=0).stop 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48661b640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 0x7fc46c0a5040 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc47802fd80 tx=0x7fc478005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 -- 172.21.15.16:0/3764547938 shutdown_connections 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ab4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aa7a10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 --2- 172.21.15.16:0/3764547938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0aa7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.560+0000 7fc48a457740 1 -- 172.21.15.16:0/3764547938 >> 172.21.15.16:0/3764547938 conn(0x560fc0a70310 msgr2=0x560fc0aa3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.561+0000 7fc48a457740 1 -- 172.21.15.16:0/3764547938 shutdown_connections 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.561+0000 7fc48a457740 1 -- 172.21.15.16:0/3764547938 wait complete. 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.561+0000 7fc48a457740 1 -- start start 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.561+0000 7fc48a457740 1 Processor -- start 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0aa8b40 0x560fc0aaf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0ab2370 0x560fc0aaf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560fc0ab5070 con 0x560fc0aa8b40 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fc09c8060 con 0x560fc0ab2370 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560fc0ab51f0 con 0x560fc0aa75d0 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48a457740 10 client.0 _ll_register_callbacks cb 0x560fc071b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47776/0 (socket says 172.21.15.16:47776) 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 -- 172.21.15.16:0/2886909529 learned_addr learned my addr 172.21.15.16:0/2886909529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485619640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0aa8b40 0x560fc0aaf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc48661b640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0ab2370 0x560fc0aaf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 -- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0ab2370 msgr2=0x560fc0aaf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0ab2370 0x560fc0aaf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 -- 172.21.15.16:0/2886909529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0aa8b40 msgr2=0x560fc0aaf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0aa8b40 0x560fc0aaf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.562+0000 7fc485e1a640 1 -- 172.21.15.16:0/2886909529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc077e3c0 con 0x560fc0aa75d0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc485e1a640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc47c000c00 tx=0x7fc47c019b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c31503fc278765 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc4697fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c028020 con 0x560fc0aa75d0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc47c024620 con 0x560fc0aa75d0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc47c0043d0 con 0x560fc0aa75d0 2024-09-12T03:11:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc47c0046c0 con 0x560fc0aa75d0 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.563+0000 7fc48a457740 10 client.29962 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560fc0ac0b40 con 0x560fc0aa75d0 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c0043d0 con 0x560fc0aa75d0 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fc47c018070 con 0x560fc0aa75d0 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 10 client.29962 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 10 client.29962 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 20 client.29962 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0ac1e90 0x560fc0ac4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 4 client.29962 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.564+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fc0a1a860 con 0x560fc0ac1e90 2024-09-12T03:11:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.565+0000 7fc485619640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0ac1e90 0x560fc0ac4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.565+0000 7fc4697fa640 10 client.29962 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.565+0000 7fc485619640 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0ac1e90 0x560fc0ac4280 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9843e80a4360da44 server_cookie=7734679a462bc4ae in_seq=0 out_seq=0 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.568+0000 7fc4697fa640 1 -- 172.21.15.16:0/2886909529 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560fc0a1a860 con 0x560fc0ac1e90 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.568+0000 7fc4697fa640 10 client.29962 handle_command_reply: tid=0 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.568+0000 7fc48a457740 1 client.29962 shutdown 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 msgr2=0x560fc0ab1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc47c000c00 tx=0x7fc47c019b20 comp rx=0 tx=0).stop 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 shutdown_connections 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0ac1e90 0x560fc0ac4280 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0ab2370 0x560fc0aaf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0aa8b40 0x560fc0aaf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 --2- 172.21.15.16:0/2886909529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0ab1d80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 >> 172.21.15.16:0/2886909529 conn(0x560fc0a70310 msgr2=0x560fc0aa9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc4697fa640 0 client.29962 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 shutdown_connections 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 1 -- 172.21.15.16:0/2886909529 wait complete. 2024-09-12T03:11:27.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.569+0000 7fc48a457740 20 client.29962 trim_cache size 0 max 16384 2024-09-12T03:11:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 Processor -- start 2024-09-12T03:11:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 -- start start 2024-09-12T03:11:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0abe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 0x560fc0aae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801ab290 con 0x560fc0ab2370 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801acdd0 con 0x560fc0aa8b40 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48a457740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801ab960 con 0x560fc0aa75d0 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48661b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc485619640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 0x560fc0aae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0abe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48661b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46618/0 (socket says 172.21.15.16:46618) 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.586+0000 7fc48661b640 1 -- 172.21.15.16:0/1590625579 learned_addr learned my addr 172.21.15.16:0/1590625579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 -- 172.21.15.16:0/1590625579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 msgr2=0x560fc0aae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 0x560fc0aae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 -- 172.21.15.16:0/1590625579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 msgr2=0x560fc0ac9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 -- 172.21.15.16:0/1590625579 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc0ab4f50 con 0x560fc0aa75d0 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485619640 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 0x560fc0aae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc48661b640 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0abe310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009620 tx=0x7fc47c004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f0abb0c76fae06 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc46b7fe640 1 -- 172.21.15.16:0/1590625579 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c028070 con 0x560fc0aa75d0 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc46b7fe640 1 -- 172.21.15.16:0/1590625579 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c01f040 con 0x560fc0aa75d0 2024-09-12T03:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.587+0000 7fc46b7fe640 1 -- 172.21.15.16:0/1590625579 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c02f9a0 con 0x560fc0aa75d0 2024-09-12T03:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.588+0000 7fc48a457740 1 -- 172.21.15.16:0/1590625579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 msgr2=0x560fc0abe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.588+0000 7fc48a457740 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0abe310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009620 tx=0x7fc47c004600 comp rx=0 tx=0).stop 2024-09-12T03:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 -- 172.21.15.16:0/1590625579 shutdown_connections 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0ac9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa8b40 0x560fc0aae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 --2- 172.21.15.16:0/1590625579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa75d0 0x560fc0abe310 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 -- 172.21.15.16:0/1590625579 >> 172.21.15.16:0/1590625579 conn(0x560fc0a70310 msgr2=0x560fc0aad900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 -- 172.21.15.16:0/1590625579 shutdown_connections 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 -- 172.21.15.16:0/1590625579 wait complete. 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 -- start start 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.589+0000 7fc48a457740 1 Processor -- start 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0abaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0aaba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801ac700 con 0x560fc0ab2370 2024-09-12T03:11:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801ac030 con 0x560fc0aa75d0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4801a9270 con 0x560fc0aa8b40 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc48a457740 10 client.0 _ll_register_callbacks cb 0x7fc470005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.590+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33868/0 (socket says 172.21.15.16:33868) 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 -- 172.21.15.16:0/1814925000 learned_addr learned my addr 172.21.15.16:0/1814925000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc48661b640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0aaba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 -- 172.21.15.16:0/1814925000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 msgr2=0x560fc0abaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0abaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 -- 172.21.15.16:0/1814925000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 msgr2=0x560fc0aaba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0aaba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 -- 172.21.15.16:0/1814925000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc0aac240 con 0x560fc0aa75d0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc485e1a640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0198c0 tx=0x7fc47c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fae995280edeb7f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc4697fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c028040 con 0x560fc0aa75d0 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc47c004860 con 0x560fc0aa75d0 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.591+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc47c01f040 con 0x560fc0aa75d0 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc47c047050 con 0x560fc0aa75d0 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 10 client.30026 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x560fc0b2c520 con 0x560fc0aa75d0 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c01f040 con 0x560fc0aa75d0 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fc47c018040 con 0x560fc0aa75d0 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 10 client.30026 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 10 client.30026 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 20 client.30026 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0b2cbd0 0x560fc0b2efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 4 client.30026 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.592+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fc0a14d70 con 0x560fc0b2cbd0 2024-09-12T03:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.593+0000 7fc485619640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0b2cbd0 0x560fc0b2efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.593+0000 7fc485619640 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0b2cbd0 0x560fc0b2efe0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=625439208595d200 server_cookie=4c56abad6ae8bd8e in_seq=0 out_seq=0 2024-09-12T03:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.593+0000 7fc4697fa640 10 client.30026 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc4697fa640 1 -- 172.21.15.16:0/1814925000 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fc0a14d70 con 0x560fc0b2cbd0 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc4697fa640 10 client.30026 handle_command_reply: tid=0 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 client.30026 shutdown 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 msgr2=0x560fc0aba760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0198c0 tx=0x7fc47c009780 comp rx=0 tx=0).stop 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 shutdown_connections 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x560fc0b2cbd0 0x560fc0b2efe0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fc0ab2370 0x560fc0aaba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc0aa8b40 0x560fc0abaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc4697fa640 0 client.30026 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 --2- 172.21.15.16:0/1814925000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fc0aa75d0 0x560fc0aba760 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 >> 172.21.15.16:0/1814925000 conn(0x560fc0a70310 msgr2=0x560fc0aae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 shutdown_connections 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 1 -- 172.21.15.16:0/1814925000 wait complete. 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.596+0000 7fc48a457740 20 client.30026 trim_cache size 0 max 16384 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 msgr2=0x7fc46c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 0x7fc46c0a5040 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc47802fd80 tx=0x7fc478005c50 comp rx=0 tx=0).stop 2024-09-12T03:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 msgr2=0x7fc4801989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801989a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc470009bc0 tx=0x7fc47000b690 comp rx=0 tx=0).stop 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 shutdown_connections 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc480102ef0 0x7fc48019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc480102570 0x7fc4801989a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc46c0a2b80 0x7fc46c0a5040 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 --2- 172.21.15.16:0/858344413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc480100e20 0x7fc480198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 >> 172.21.15.16:0/858344413 conn(0x7fc4800fc7d0 msgr2=0x7fc480103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 shutdown_connections 2024-09-12T03:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:27.597+0000 7fc48807e640 1 -- 172.21.15.16:0/858344413 wait complete. 2024-09-12T03:11:28.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 254678 22.67 MB/sec execute 509 sec latency 46.065 ms 2024-09-12T03:11:28.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:11:28.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:28 smithi184 ceph-mon[35321]: pgmap v1043: 97 pgs: 97 active+clean; 884 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-09-12T03:11:28.548 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:28 smithi033 ceph-mon[35092]: pgmap v1043: 97 pgs: 97 active+clean; 884 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-09-12T03:11:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:28.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:28 smithi016 ceph-mon[30925]: pgmap v1043: 97 pgs: 97 active+clean; 884 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-09-12T03:11:29.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 255393 22.69 MB/sec execute 510 sec latency 74.287 ms 2024-09-12T03:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.247+0000 7f94fdb2d640 1 -- 172.21.15.16:0/732416029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 msgr2=0x7f94f810e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.247+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/732416029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 0x7f94f810e2b0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f94e4013080 tx=0x7f94e4032730 comp rx=0 tx=0).stop 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 -- 172.21.15.16:0/732416029 shutdown_connections 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/732416029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 0x7f94f810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/732416029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 0x7f94f810e2b0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/732416029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 -- 172.21.15.16:0/732416029 >> 172.21.15.16:0/732416029 conn(0x7f94f8073660 msgr2=0x7f94f8073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 -- 172.21.15.16:0/732416029 shutdown_connections 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.250+0000 7f94fdb2d640 1 -- 172.21.15.16:0/732416029 wait complete. 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.251+0000 7f94fdb2d640 1 Processor -- start 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.251+0000 7f94fdb2d640 1 -- start start 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 0x7f94f81a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 0x7f94f819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8116430 con 0x7f94f8109d70 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94f81162b0 con 0x7f94f810e7f0 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fdb2d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8118680 con 0x7f94f8109450 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fcb2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fcb2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47808/0 (socket says 172.21.15.16:47808) 2024-09-12T03:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fcb2b640 1 -- 172.21.15.16:0/2136482712 learned_addr learned my addr 172.21.15.16:0/2136482712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.252+0000 7f94fd32c640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 0x7f94f819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.253+0000 7f94fcb2b640 1 -- 172.21.15.16:0/2136482712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 msgr2=0x7f94f819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.253+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 0x7f94f819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.253+0000 7f94fcb2b640 1 -- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 msgr2=0x7f94f81a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.253+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 0x7f94f81a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.253+0000 7f94fcb2b640 1 -- 172.21.15.16:0/2136482712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f819ed40 con 0x7f94f8109450 2024-09-12T03:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.255+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00d950 tx=0x7f94ec00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.256+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2136482712 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec017070 con 0x7f94f8109450 2024-09-12T03:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.256+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f819f030 con 0x7f94f8109450 2024-09-12T03:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.256+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f807a450 con 0x7f94f8109450 2024-09-12T03:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.256+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2136482712 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ec009c30 con 0x7f94f8109450 2024-09-12T03:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.256+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2136482712 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00f690 con 0x7f94f8109450 2024-09-12T03:11:29.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 Processor -- start 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 -- start start 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d24240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d24bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d505c8a2f0 con 0x55d505d24780 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d505c31cf0 con 0x55d505d2f520 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.263+0000 7f9501168740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d505c862b0 con 0x55d505d25cf0 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.264+0000 7f94f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.264+0000 7f94f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33906/0 (socket says 172.21.15.16:33906) 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.264+0000 7f94f7fff640 1 -- 172.21.15.16:0/3777876392 learned_addr learned my addr 172.21.15.16:0/3777876392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:29.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.264+0000 7f94fd32c640 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d24240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.265+0000 7f94f7fff640 1 -- 172.21.15.16:0/3777876392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 msgr2=0x55d505d24240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.265+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d24240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.265+0000 7f94f7fff640 1 -- 172.21.15.16:0/3777876392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 msgr2=0x55d505d24bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.265+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d24bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.265+0000 7f94f7fff640 1 -- 172.21.15.16:0/3777876392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d505c163a0 con 0x55d505d2f520 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.266+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2136482712 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f94ec010460 con 0x7f94f8109450 2024-09-12T03:11:29.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.266+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f94e403afd0 tx=0x7f94e4039470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151efab6f5407537 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.271+0000 7f94e2ffd640 1 -- 172.21.15.16:0/3777876392 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e403e040 con 0x55d505d2f520 2024-09-12T03:11:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.271+0000 7f94e2ffd640 1 -- 172.21.15.16:0/3777876392 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e400fbe0 con 0x55d505d2f520 2024-09-12T03:11:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.272+0000 7f94f5ffb640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 0x7f94e80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.272+0000 7f9501168740 1 -- 172.21.15.16:0/3777876392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 msgr2=0x55d505d31910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.272+0000 7f9501168740 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f94e403afd0 tx=0x7f94e4039470 comp rx=0 tx=0).stop 2024-09-12T03:11:29.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.272+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2136482712 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94ec0cec20 con 0x7f94f8109450 2024-09-12T03:11:29.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.276+0000 7f94fd32c640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 0x7f94e80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.277+0000 7f9501168740 1 -- 172.21.15.16:0/3777876392 shutdown_connections 2024-09-12T03:11:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.277+0000 7f9501168740 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d31910 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.277+0000 7f9501168740 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d24bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.277+0000 7f9501168740 1 --2- 172.21.15.16:0/3777876392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d24240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.277+0000 7f9501168740 1 -- 172.21.15.16:0/3777876392 >> 172.21.15.16:0/3777876392 conn(0x55d505ced310 msgr2=0x55d505d20e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.281+0000 7f9501168740 1 -- 172.21.15.16:0/3777876392 shutdown_connections 2024-09-12T03:11:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.282+0000 7f94fd32c640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 0x7f94e80a5250 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94f000bd80 tx=0x7f94f000b4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.284+0000 7f9501168740 1 -- 172.21.15.16:0/3777876392 wait complete. 2024-09-12T03:11:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.285+0000 7f9501168740 1 -- start start 2024-09-12T03:11:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.285+0000 7f9501168740 1 Processor -- start 2024-09-12T03:11:29.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d2ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 0x55d505d2c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d505d32260 con 0x55d505d25cf0 2024-09-12T03:11:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d505c45060 con 0x55d505d24780 2024-09-12T03:11:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.286+0000 7f9501168740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d505d323e0 con 0x55d505d2f520 2024-09-12T03:11:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.287+0000 7f9501168740 10 client.0 _ll_register_callbacks cb 0x55d5059988c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.287+0000 7f94fd32c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.287+0000 7f94fd32c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47822/0 (socket says 172.21.15.16:47822) 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.287+0000 7f94f7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 0x55d505d2c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.287+0000 7f94fd32c640 1 -- 172.21.15.16:0/645285538 learned_addr learned my addr 172.21.15.16:0/645285538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.288+0000 7f94fd32c640 1 -- 172.21.15.16:0/645285538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 msgr2=0x55d505d2ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.288+0000 7f94fd32c640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d2ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.289+0000 7f94fd32c640 1 -- 172.21.15.16:0/645285538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 msgr2=0x55d505d2c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.289+0000 7f94fd32c640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 0x55d505d2c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.289+0000 7f94fd32c640 1 -- 172.21.15.16:0/645285538 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5059fb3c0 con 0x55d505d2f520 2024-09-12T03:11:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.289+0000 7f94f7fff640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 0x55d505d2c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:11:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.291+0000 7f94fd32c640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f94f001bc00 tx=0x7f94f0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba0342bdf01b50ea server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.293+0000 7f94d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.297+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f002a060 con 0x55d505d2f520 2024-09-12T03:11:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.298+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94e400fbe0 con 0x55d505d2f520 2024-09-12T03:11:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.298+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94f0019bd0 con 0x55d505d2f520 2024-09-12T03:11:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.305+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f94f0019dd0 con 0x55d505d2f520 2024-09-12T03:11:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.305+0000 7f9501168740 10 client.29977 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.305+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d505d4ecf0 con 0x55d505d2f520 2024-09-12T03:11:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.306+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e400fbe0 con 0x55d505d2f520 2024-09-12T03:11:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f94f002e5a0 con 0x55d505d2f520 2024-09-12T03:11:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 10 client.29977 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 10 client.29977 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:11:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 20 client.29977 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505d505f0 0x55d505d52ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 4 client.29977 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:29.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.307+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d505c97860 con 0x55d505d505f0 2024-09-12T03:11:29.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.309+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505d505f0 0x55d505d52ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.311+0000 7f94d7fff640 10 client.29977 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:11:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.311+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505d505f0 0x55d505d52ab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8dcacfd4c4fc2de9 server_cookie=95349d6e0dc73063 in_seq=0 out_seq=0 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.321+0000 7f94d7fff640 1 -- 172.21.15.16:0/645285538 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d505c97860 con 0x55d505d505f0 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.321+0000 7f94d7fff640 10 client.29977 handle_command_reply: tid=0 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.321+0000 7f9501168740 1 client.29977 shutdown 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 msgr2=0x55d505d2c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f94f001bc00 tx=0x7f94f0004a40 comp rx=0 tx=0).stop 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 shutdown_connections 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505d505f0 0x55d505d52ab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d2f520 0x55d505d2c580 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d25cf0 0x55d505d2c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 --2- 172.21.15.16:0/645285538 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d2ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 >> 172.21.15.16:0/645285538 conn(0x55d505ced310 msgr2=0x55d505d26ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.322+0000 7f94d7fff640 0 client.29977 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:11:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.323+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 shutdown_connections 2024-09-12T03:11:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.323+0000 7f9501168740 1 -- 172.21.15.16:0/645285538 wait complete. 2024-09-12T03:11:29.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.326+0000 7f9501168740 20 client.29977 trim_cache size 0 max 16384 2024-09-12T03:11:29.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.355+0000 7f9501168740 1 Processor -- start 2024-09-12T03:11:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.355+0000 7f9501168740 1 -- start start 2024-09-12T03:11:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d2b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d272d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc00e4f0 con 0x55d505d24780 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc010030 con 0x55d505d2f520 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.356+0000 7f9501168740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc00ebc0 con 0x55d505d25cf0 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46676/0 (socket says 172.21.15.16:46676) 2024-09-12T03:11:29.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 -- 172.21.15.16:0/723862442 learned_addr learned my addr 172.21.15.16:0/723862442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 -- 172.21.15.16:0/723862442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 msgr2=0x55d505d2b900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d2b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 -- 172.21.15.16:0/723862442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 msgr2=0x55d505d272d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d272d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 -- 172.21.15.16:0/723862442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d505d20480 con 0x55d505d24780 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.357+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009610 tx=0x7f94ec00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98c460d2a68980b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.358+0000 7f94e27fc640 1 -- 172.21.15.16:0/723862442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec020040 con 0x55d505d24780 2024-09-12T03:11:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.358+0000 7f94e27fc640 1 -- 172.21.15.16:0/723862442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ec09f7e0 con 0x55d505d24780 2024-09-12T03:11:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.358+0000 7f94e27fc640 1 -- 172.21.15.16:0/723862442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec097aa0 con 0x55d505d24780 2024-09-12T03:11:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 -- 172.21.15.16:0/723862442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 msgr2=0x55d505d2c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009610 tx=0x7f94ec00bf60 comp rx=0 tx=0).stop 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 -- 172.21.15.16:0/723862442 shutdown_connections 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d2f520 0x55d505d272d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d2b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 --2- 172.21.15.16:0/723862442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d24780 0x55d505d2c710 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.359+0000 7f9501168740 1 -- 172.21.15.16:0/723862442 >> 172.21.15.16:0/723862442 conn(0x55d505ced310 msgr2=0x55d505d2c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.360+0000 7f9501168740 1 -- 172.21.15.16:0/723862442 shutdown_connections 2024-09-12T03:11:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.362+0000 7f9501168740 1 -- 172.21.15.16:0/723862442 wait complete. 2024-09-12T03:11:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.362+0000 7f9501168740 1 -- start start 2024-09-12T03:11:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.363+0000 7f9501168740 1 Processor -- start 2024-09-12T03:11:29.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d29010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d29550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc00f960 con 0x55d505d2f520 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc00f290 con 0x55d505d24780 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc00c4d0 con 0x55d505d25cf0 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f9501168740 10 client.0 _ll_register_callbacks cb 0x7f94f000ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f94fd32c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f94fd32c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46684/0 (socket says 172.21.15.16:46684) 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.364+0000 7f94fd32c640 1 -- 172.21.15.16:0/3641834709 learned_addr learned my addr 172.21.15.16:0/3641834709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 -- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 msgr2=0x55d505d29550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d29550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 -- 172.21.15.16:0/3641834709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 msgr2=0x55d505d29010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d29010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 -- 172.21.15.16:0/3641834709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d505dba390 con 0x55d505d2f520 2024-09-12T03:11:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94fd32c640 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f94f0019760 tx=0x7f94f0019c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6617ad67a2fe43 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:11:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f003c060 con 0x55d505d2f520 2024-09-12T03:11:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f94f002e3a0 con 0x55d505d2f520 2024-09-12T03:11:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.365+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94ec09f7e0 con 0x55d505d2f520 2024-09-12T03:11:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.366+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f94f002a020 con 0x55d505d2f520 2024-09-12T03:11:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.366+0000 7f9501168740 10 client.20883 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.366+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d505dbb630 con 0x55d505d2f520 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ec09f7e0 con 0x55d505d2f520 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f94f0005610 con 0x55d505d2f520 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 10 client.20883 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 10 client.20883 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 20 client.20883 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505dbbd20 0x55d505dbe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 4 client.20883 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.367+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d505c91d70 con 0x55d505dbbd20 2024-09-12T03:11:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.368+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505dbbd20 0x55d505dbe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.375+0000 7f94d7fff640 10 client.20883 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:11:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.375+0000 7f94fcb2b640 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505dbbd20 0x55d505dbe180 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d296b4887df57175 server_cookie=e041c7f97c5bf124 in_seq=0 out_seq=0 2024-09-12T03:11:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f94d7fff640 1 -- 172.21.15.16:0/3641834709 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d505c91d70 con 0x55d505dbbd20 2024-09-12T03:11:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f94d7fff640 10 client.20883 handle_command_reply: tid=0 2024-09-12T03:11:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 client.20883 shutdown 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 msgr2=0x55d505d29a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f94f0019760 tx=0x7f94f0019c00 comp rx=0 tx=0).stop 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 shutdown_connections 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d505d2f520 0x55d505d29a90 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55d505dbbd20 0x55d505dbe180 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d505d25cf0 0x55d505d29550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 --2- 172.21.15.16:0/3641834709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d505d24780 0x55d505d29010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 >> 172.21.15.16:0/3641834709 conn(0x55d505ced310 msgr2=0x55d505d2d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.377+0000 7f94d7fff640 0 client.20883 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:11:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.378+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 shutdown_connections 2024-09-12T03:11:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.378+0000 7f9501168740 1 -- 172.21.15.16:0/3641834709 wait complete. 2024-09-12T03:11:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.378+0000 7f9501168740 20 client.20883 trim_cache size 0 max 16384 2024-09-12T03:11:29.380 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.379+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 msgr2=0x7f94e80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.379+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 0x7f94e80a5250 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94f000bd80 tx=0x7f94f000b4d0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.379+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 msgr2=0x7f94f81a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.379+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00d950 tx=0x7f94ec00de10 comp rx=0 tx=0).stop 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 shutdown_connections 2024-09-12T03:11:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94f810e7f0 0x7f94f819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f94e80a2d90 0x7f94e80a5250 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94f8109d70 0x7f94f81a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 --2- 172.21.15.16:0/2136482712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8109450 0x7f94f81a4450 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 >> 172.21.15.16:0/2136482712 conn(0x7f94f8073660 msgr2=0x7f94f8110440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 shutdown_connections 2024-09-12T03:11:29.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:29.380+0000 7f94fdb2d640 1 -- 172.21.15.16:0/2136482712 wait complete. 2024-09-12T03:11:30.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 255805 22.70 MB/sec execute 511 sec latency 230.419 ms 2024-09-12T03:11:30.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:30 smithi184 ceph-mon[35321]: pgmap v1044: 97 pgs: 97 active+clean; 886 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-09-12T03:11:30.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:30 smithi016 ceph-mon[30925]: pgmap v1044: 97 pgs: 97 active+clean; 886 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-09-12T03:11:30.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:11:30.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:30 smithi033 ceph-mon[35092]: pgmap v1044: 97 pgs: 97 active+clean; 886 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 464 op/s 2024-09-12T03:11:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:31.039 INFO:tasks.workunit.client.0.smithi016.stdout: 10 256459 22.72 MB/sec execute 512 sec latency 188.327 ms 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 -- 172.21.15.16:0/3920861329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc106f20 msgr2=0x7f38fc10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 --2- 172.21.15.16:0/3920861329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc106f20 0x7f38fc10eaa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009ca0 tx=0x7f38f402a8e0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 -- 172.21.15.16:0/3920861329 shutdown_connections 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 --2- 172.21.15.16:0/3920861329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc106f20 0x7f38fc10eaa0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 --2- 172.21.15.16:0/3920861329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 0x7f38fc1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 --2- 172.21.15.16:0/3920861329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc101df0 0x7f38fc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.537+0000 7f3903d0d640 1 -- 172.21.15.16:0/3920861329 >> 172.21.15.16:0/3920861329 conn(0x7f38fc0fd8f0 msgr2=0x7f38fc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.538+0000 7f3903d0d640 1 -- 172.21.15.16:0/3920861329 shutdown_connections 2024-09-12T03:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.538+0000 7f3903d0d640 1 -- 172.21.15.16:0/3920861329 wait complete. 2024-09-12T03:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.538+0000 7f3903d0d640 1 Processor -- start 2024-09-12T03:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 -- start start 2024-09-12T03:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 0x7f38fc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33960/0 (socket says 172.21.15.16:33960) 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3901aa9640 1 -- 172.21.15.16:0/1705961862 learned_addr learned my addr 172.21.15.16:0/1705961862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc106f20 0x7f38fc19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc111f80 con 0x7f38fc106f20 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc111e00 con 0x7f38fc101df0 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc112100 con 0x7f38fc1027a0 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.539+0000 7f39012a8640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 0x7f38fc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f39022aa640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc106f20 0x7f38fc19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 -- 172.21.15.16:0/1705961862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 msgr2=0x7f38fc198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 0x7f38fc198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 -- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc106f20 msgr2=0x7f38fc19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc106f20 0x7f38fc19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc19d190 con 0x7f38fc101df0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.540+0000 7f3901aa9640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f38f800ea60 tx=0x7f38f800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.541+0000 7f38f2ffd640 1 -- 172.21.15.16:0/1705961862 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f800cde0 con 0x7f38fc101df0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.541+0000 7f38f2ffd640 1 -- 172.21.15.16:0/1705961862 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38f80035f0 con 0x7f38fc101df0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.541+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc19d480 con 0x7f38fc101df0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.541+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38fc074420 con 0x7f38fc101df0 2024-09-12T03:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.541+0000 7f38f2ffd640 1 -- 172.21.15.16:0/1705961862 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f8010670 con 0x7f38fc101df0 2024-09-12T03:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.545+0000 7f38f2ffd640 1 -- 172.21.15.16:0/1705961862 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f38f80037a0 con 0x7f38fc101df0 2024-09-12T03:11:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.545+0000 7f39060e6740 1 Processor -- start 2024-09-12T03:11:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.545+0000 7f39060e6740 1 -- start start 2024-09-12T03:11:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc467b40 0x563fdc466090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f38f2ffd640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 0x7f38e80a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc471370 0x563fdc473760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fdc3cc2f0 con 0x563fdc471370 2024-09-12T03:11:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fdc373cf0 con 0x563fdc4665d0 2024-09-12T03:11:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39060e6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fdc3c82b0 con 0x563fdc467b40 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33970/0 (socket says 172.21.15.16:33970) 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f3901aa9640 1 -- 172.21.15.16:0/3784839404 learned_addr learned my addr 172.21.15.16:0/3784839404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.546+0000 7f39022aa640 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc467b40 0x563fdc466090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 -- 172.21.15.16:0/3784839404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc467b40 msgr2=0x563fdc466090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc467b40 0x563fdc466090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 -- 172.21.15.16:0/3784839404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc471370 msgr2=0x563fdc473760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc471370 0x563fdc473760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 -- 172.21.15.16:0/3784839404 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fdc3583a0 con 0x563fdc4665d0 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f38f2ffd640 1 -- 172.21.15.16:0/1705961862 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38f80cf260 con 0x7f38fc101df0 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f3901aa9640 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f38f8004020 tx=0x7f38f809b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe61274b3f3f9aa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f39012a8640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 0x7f38e80a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f38e7fff640 1 -- 172.21.15.16:0/3784839404 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b4070 con 0x563fdc4665d0 2024-09-12T03:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f38e7fff640 1 -- 172.21.15.16:0/3784839404 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38f809bad0 con 0x563fdc4665d0 2024-09-12T03:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.547+0000 7f38e7fff640 1 -- 172.21.15.16:0/3784839404 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f8098140 con 0x563fdc4665d0 2024-09-12T03:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39012a8640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 0x7f38e80a50c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f38fc108200 tx=0x7f38ec00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39060e6740 1 -- 172.21.15.16:0/3784839404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 msgr2=0x563fdc466a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39060e6740 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f38f8004020 tx=0x7f38f809b4f0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39060e6740 1 -- 172.21.15.16:0/3784839404 shutdown_connections 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39060e6740 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc471370 0x563fdc473760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.549+0000 7f39060e6740 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc4665d0 0x563fdc466a10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.550+0000 7f39060e6740 1 --2- 172.21.15.16:0/3784839404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc467b40 0x563fdc466090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.550+0000 7f39060e6740 1 -- 172.21.15.16:0/3784839404 >> 172.21.15.16:0/3784839404 conn(0x563fdc42f310 msgr2=0x563fdc462c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.550+0000 7f39060e6740 1 -- 172.21.15.16:0/3784839404 shutdown_connections 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.550+0000 7f39060e6740 1 -- 172.21.15.16:0/3784839404 wait complete. 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.550+0000 7f39060e6740 1 -- start start 2024-09-12T03:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 Processor -- start 2024-09-12T03:11:31.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc470d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc46e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563fdc474070 con 0x563fdc4665d0 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563fdc387060 con 0x563fdc467b40 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fdc4741f0 con 0x563fdc471370 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39060e6740 10 client.0 _ll_register_callbacks cb 0x563fdc0da8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:31.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.551+0000 7f39012a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33972/0 (socket says 172.21.15.16:33972) 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 -- 172.21.15.16:0/630192496 learned_addr learned my addr 172.21.15.16:0/630192496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39022aa640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc46e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f3901aa9640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc470d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 -- 172.21.15.16:0/630192496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 msgr2=0x563fdc46e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc46e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 -- 172.21.15.16:0/630192496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 msgr2=0x563fdc470d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc470d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 -- 172.21.15.16:0/630192496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fdc13d3c0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f39012a8640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0116c0 tx=0x7f38ec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d0787de7eeb82e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.552+0000 7f38e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec01e8f0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38f809bad0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38ec01ea90 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f38ec01ecd0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f39060e6740 10 client.30041 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563fdc47fc30 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.553+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38f809bad0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f38ec01c3f0 con 0x563fdc467b40 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 10 client.30041 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 10 client.30041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:11:31.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 20 client.30041 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 4 client.30041 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fdc3d9860 con 0x563fdc481090 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.554+0000 7f3901aa9640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.555+0000 7f38e57fa640 10 client.30041 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:11:31.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.555+0000 7f3901aa9640 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4834b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a10f2d11d275adf4 server_cookie=548f1bf3acad0c38 in_seq=0 out_seq=0 2024-09-12T03:11:31.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.555+0000 7f38e57fa640 1 -- 172.21.15.16:0/630192496 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563fdc3d9860 con 0x563fdc481090 2024-09-12T03:11:31.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.555+0000 7f38e57fa640 10 client.30041 handle_command_reply: tid=0 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.555+0000 7f39060e6740 1 client.30041 shutdown 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 msgr2=0x563fdc46e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0116c0 tx=0x7f38ec004730 comp rx=0 tx=0).stop 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 shutdown_connections 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4834b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f38e57fa640 0 client.30041 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc46e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc46e090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 --2- 172.21.15.16:0/630192496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc470d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 >> 172.21.15.16:0/630192496 conn(0x563fdc42f310 msgr2=0x563fdc4689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 shutdown_connections 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 1 -- 172.21.15.16:0/630192496 wait complete. 2024-09-12T03:11:31.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.556+0000 7f39060e6740 20 client.30041 trim_cache size 0 max 16384 2024-09-12T03:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 Processor -- start 2024-09-12T03:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 -- start start 2024-09-12T03:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc488d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1af810 con 0x563fdc4665d0 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1b1350 con 0x563fdc467b40 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.574+0000 7f39060e6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1afee0 con 0x563fdc471370 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46732/0 (socket says 172.21.15.16:46732) 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:33974/0 (socket says 172.21.15.16:33974) 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f3901aa9640 1 -- 172.21.15.16:0/872947125 learned_addr learned my addr 172.21.15.16:0/872947125 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39022aa640 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc488d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 -- 172.21.15.16:0/872947125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 msgr2=0x563fdc488d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc488d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 -- 172.21.15.16:0/872947125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 msgr2=0x563fdc462d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 -- 172.21.15.16:0/872947125 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fdc470900 con 0x563fdc467b40 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f3901aa9640 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.575+0000 7f39012a8640 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f38ec000c00 tx=0x7f38ec010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96f0a67fc6c6075f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.576+0000 7f38e77fe640 1 -- 172.21.15.16:0/872947125 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec00b7f0 con 0x563fdc467b40 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.576+0000 7f38e77fe640 1 -- 172.21.15.16:0/872947125 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ec010d70 con 0x563fdc467b40 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.576+0000 7f38e77fe640 1 -- 172.21.15.16:0/872947125 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec031450 con 0x563fdc467b40 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 -- 172.21.15.16:0/872947125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 msgr2=0x563fdc47d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f38ec000c00 tx=0x7f38ec010910 comp rx=0 tx=0).stop 2024-09-12T03:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 -- 172.21.15.16:0/872947125 shutdown_connections 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc471370 0x563fdc488d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc467b40 0x563fdc47d310 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 --2- 172.21.15.16:0/872947125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc4665d0 0x563fdc462d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 -- 172.21.15.16:0/872947125 >> 172.21.15.16:0/872947125 conn(0x563fdc42f310 msgr2=0x563fdc46d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 -- 172.21.15.16:0/872947125 shutdown_connections 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.577+0000 7f39060e6740 1 -- 172.21.15.16:0/872947125 wait complete. 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.578+0000 7f39060e6740 1 -- start start 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.578+0000 7f39060e6740 1 Processor -- start 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.578+0000 7f39060e6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 0x563fdc46b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.578+0000 7f39060e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 0x563fdc46b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 0x563fdc46b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39060e6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc471370 0x563fdc46bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f3901aa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 0x563fdc46b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47890/0 (socket says 172.21.15.16:47890) 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f3901aa9640 1 -- 172.21.15.16:0/2338179606 learned_addr learned my addr 172.21.15.16:0/2338179606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1b0c80 con 0x563fdc467b40 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1b05b0 con 0x563fdc471370 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1ad7f0 con 0x563fdc4665d0 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39060e6740 10 client.0 _ll_register_callbacks cb 0x7f38ec010b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39012a8640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 0x563fdc46b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39022aa640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc471370 0x563fdc46bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39012a8640 1 -- 172.21.15.16:0/2338179606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 msgr2=0x563fdc46b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.579+0000 7f39012a8640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 0x563fdc46b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f39012a8640 1 -- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc471370 msgr2=0x563fdc46bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f39012a8640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc471370 0x563fdc46bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f39012a8640 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fdc4eae20 con 0x563fdc467b40 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f39012a8640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 0x563fdc46b730 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x563fdc46c840 tx=0x7f38ec005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efcb6e586a5d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f38e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec01e6f0 con 0x563fdc467b40 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f38ec01e890 con 0x563fdc467b40 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.580+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38ec010d70 con 0x563fdc467b40 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.581+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f38ec01ea90 con 0x563fdc467b40 2024-09-12T03:11:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.581+0000 7f39060e6740 10 client.20898 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.581+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563fdc4ec100 con 0x563fdc467b40 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.581+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ec031930 con 0x563fdc467b40 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f38ec03a760 con 0x563fdc467b40 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 10 client.20898 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 10 client.20898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 20 client.20898 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 4 client.20898 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f38ec010d70 con 0x563fdc481090 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.582+0000 7f3901aa9640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f38e57fa640 10 client.20898 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f3901aa9640 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4ee680 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4c0479a6438c708e server_cookie=207ad0dabc60d346 in_seq=0 out_seq=0 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f38e57fa640 1 -- 172.21.15.16:0/2338179606 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f38ec010d70 con 0x563fdc481090 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f38e57fa640 10 client.20898 handle_command_reply: tid=0 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 client.20898 shutdown 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 msgr2=0x563fdc46b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 0x563fdc46b730 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x563fdc46c840 tx=0x7f38ec005690 comp rx=0 tx=0).stop 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 shutdown_connections 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x563fdc481090 0x563fdc4ee680 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563fdc471370 0x563fdc46bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563fdc467b40 0x563fdc46b730 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 --2- 172.21.15.16:0/2338179606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fdc4665d0 0x563fdc46b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f38e57fa640 0 client.20898 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 >> 172.21.15.16:0/2338179606 conn(0x563fdc42f310 msgr2=0x563fdc461500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.583+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 shutdown_connections 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f39060e6740 1 -- 172.21.15.16:0/2338179606 wait complete. 2024-09-12T03:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f39060e6740 20 client.20898 trim_cache size 0 max 16384 2024-09-12T03:11:31.590 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 msgr2=0x7f38e80a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 0x7f38e80a50c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f38fc108200 tx=0x7f38ec00b390 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 msgr2=0x7f38fc1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f38f800ea60 tx=0x7f38f800ef20 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 shutdown_connections 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.584+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f38e80a2c00 0x7f38e80a50c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38fc106f20 0x7f38fc19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1027a0 0x7f38fc198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 --2- 172.21.15.16:0/1705961862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38fc101df0 0x7f38fc1981e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 >> 172.21.15.16:0/1705961862 conn(0x7f38fc0fd8f0 msgr2=0x7f38fc0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 shutdown_connections 2024-09-12T03:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:31.585+0000 7f3903d0d640 1 -- 172.21.15.16:0/1705961862 wait complete. 2024-09-12T03:11:32.040 INFO:tasks.workunit.client.0.smithi016.stdout: 10 257097 22.75 MB/sec execute 513 sec latency 69.760 ms 2024-09-12T03:11:32.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:32 smithi184 ceph-mon[35321]: pgmap v1045: 97 pgs: 97 active+clean; 899 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 572 op/s 2024-09-12T03:11:32.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:32 smithi016 ceph-mon[30925]: pgmap v1045: 97 pgs: 97 active+clean; 899 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 572 op/s 2024-09-12T03:11:32.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:11:32.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:32 smithi033 ceph-mon[35092]: pgmap v1045: 97 pgs: 97 active+clean; 899 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 572 op/s 2024-09-12T03:11:33.040 INFO:tasks.workunit.client.0.smithi016.stdout: 10 257716 22.76 MB/sec execute 514 sec latency 83.560 ms 2024-09-12T03:11:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:33.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:33 smithi016 ceph-mon[30925]: pgmap v1046: 97 pgs: 97 active+clean; 903 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 498 op/s 2024-09-12T03:11:33.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:33 smithi184 ceph-mon[35321]: pgmap v1046: 97 pgs: 97 active+clean; 903 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 498 op/s 2024-09-12T03:11:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 -- 172.21.15.16:0/117113203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0114ab0 msgr2=0x7facc0116ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 --2- 172.21.15.16:0/117113203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0114ab0 0x7facc0116ea0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7facb8009ca0 tx=0x7facb802a950 comp rx=0 tx=0).stop 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 -- 172.21.15.16:0/117113203 shutdown_connections 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 --2- 172.21.15.16:0/117113203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0114ab0 0x7facc0116ea0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 --2- 172.21.15.16:0/117113203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0109b10 0x7facc0114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 --2- 172.21.15.16:0/117113203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 0x7facc01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 -- 172.21.15.16:0/117113203 >> 172.21.15.16:0/117113203 conn(0x7facc006e140 msgr2=0x7facc006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 -- 172.21.15.16:0/117113203 shutdown_connections 2024-09-12T03:11:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.613+0000 7facc8122640 1 -- 172.21.15.16:0/117113203 wait complete. 2024-09-12T03:11:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.614+0000 7facc8122640 1 Processor -- start 2024-09-12T03:11:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.614+0000 7facc8122640 1 -- start start 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.614+0000 7facc8122640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 0x7facc01119f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc8122640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc8122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0114ab0 0x7facc0112470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc8122640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facc010c1f0 con 0x7facc0114ab0 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc8122640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facc010c070 con 0x7facc0109b10 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc8122640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facc010c370 con 0x7facc01091f0 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc66bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0114ab0 0x7facc0112470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34000/0 (socket says 172.21.15.16:34000) 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 -- 172.21.15.16:0/2508388133 learned_addr learned my addr 172.21.15.16:0/2508388133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc5ebe640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 0x7facc01119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 -- 172.21.15.16:0/2508388133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 msgr2=0x7facc01119f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 0x7facc01119f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 -- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0114ab0 msgr2=0x7facc0112470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0114ab0 0x7facc0112470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 -- 172.21.15.16:0/2508388133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc01b35e0 con 0x7facc0109b10 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.615+0000 7facc56bd640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.616+0000 7facb6ffd640 1 -- 172.21.15.16:0/2508388133 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc00cd80 con 0x7facc0109b10 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.616+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc01b3870 con 0x7facc0109b10 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.616+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc01b3dd0 con 0x7facc0109b10 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.616+0000 7facb6ffd640 1 -- 172.21.15.16:0/2508388133 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facbc010910 con 0x7facc0109b10 2024-09-12T03:11:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.616+0000 7facb6ffd640 1 -- 172.21.15.16:0/2508388133 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc018990 con 0x7facc0109b10 2024-09-12T03:11:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.619+0000 7facb6ffd640 1 -- 172.21.15.16:0/2508388133 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7facbc018b30 con 0x7facc0109b10 2024-09-12T03:11:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.619+0000 7facca4fb740 1 Processor -- start 2024-09-12T03:11:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 -- start start 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665e6370 0x5562665e8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facc66bf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facc66bf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47916/0 (socket says 172.21.15.16:47916) 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562665412f0 con 0x5562665e6370 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562664e8cf0 con 0x5562665db5d0 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facca4fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55626653d2b0 con 0x5562665dcb40 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:34010/0 (socket says 172.21.15.16:34010) 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.620+0000 7facc5ebe640 1 -- 172.21.15.16:0/2754964813 learned_addr learned my addr 172.21.15.16:0/2754964813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 -- 172.21.15.16:0/2754964813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 msgr2=0x5562665db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 -- 172.21.15.16:0/2754964813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665e6370 msgr2=0x5562665e8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665e6370 0x5562665e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 -- 172.21.15.16:0/2754964813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562664cd3a0 con 0x5562665db5d0 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc66bf640 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facb6ffd640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 0x7facac0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.621+0000 7facc5ebe640 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7facb0009cf0 tx=0x7facb0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3547f14ab745daa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.622+0000 7facc66bf640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 0x7facac0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.622+0000 7facb6ffd640 1 -- 172.21.15.16:0/2508388133 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7facbc0cf1a0 con 0x7facc0109b10 2024-09-12T03:11:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.622+0000 7facabfff640 1 -- 172.21.15.16:0/2754964813 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0004230 con 0x5562665db5d0 2024-09-12T03:11:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.622+0000 7facabfff640 1 -- 172.21.15.16:0/2754964813 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb00043d0 con 0x5562665db5d0 2024-09-12T03:11:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.622+0000 7facabfff640 1 -- 172.21.15.16:0/2754964813 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0019960 con 0x5562665db5d0 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 -- 172.21.15.16:0/2754964813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 msgr2=0x5562665dba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7facb0009cf0 tx=0x7facb0011810 comp rx=0 tx=0).stop 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 -- 172.21.15.16:0/2754964813 shutdown_connections 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665e6370 0x5562665e8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665db5d0 0x5562665dba10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 --2- 172.21.15.16:0/2754964813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facca4fb740 1 -- 172.21.15.16:0/2754964813 >> 172.21.15.16:0/2754964813 conn(0x5562665a4310 msgr2=0x5562665d7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.623+0000 7facc66bf640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 0x7facac0a50c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7facb8006010 tx=0x7facb8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.624+0000 7facca4fb740 1 -- 172.21.15.16:0/2754964813 shutdown_connections 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.624+0000 7facca4fb740 1 -- 172.21.15.16:0/2754964813 wait complete. 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.624+0000 7facca4fb740 1 -- start start 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.624+0000 7facca4fb740 1 Processor -- start 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc56bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc56bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:47926/0 (socket says 172.21.15.16:47926) 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46780/0 (socket says 172.21.15.16:46780) 2024-09-12T03:11:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc56bd640 1 -- 172.21.15.16:0/3927242155 learned_addr learned my addr 172.21.15.16:0/3927242155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facc66bf640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562665e9070 con 0x5562665db5d0 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562664fc060 con 0x5562665e6370 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5562665e91f0 con 0x5562665dcb40 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.625+0000 7facca4fb740 10 client.0 _ll_register_callbacks cb 0x55626624f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 -- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 msgr2=0x5562665e3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665e3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 -- 172.21.15.16:0/3927242155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 msgr2=0x5562665e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562662b23c0 con 0x5562665dcb40 2024-09-12T03:11:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc5ebe640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facc56bd640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7facbc0cccb0 tx=0x7facbc097680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69e320dbd5b7127 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7faca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc0ce030 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb00043d0 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7facbc09b060 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7facbc09ba10 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facca4fb740 10 client.30004 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.626+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5562665f4b20 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb00043d0 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7facbc010ac0 con 0x5562665dcb40 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 10 client.30004 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 10 client.30004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 20 client.30004 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x5562665f8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 4 client.30004 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55626654e860 con 0x5562665f5f10 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.627+0000 7facc5ebe640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x5562665f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.628+0000 7faca97fa640 10 client.30004 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.628+0000 7facc5ebe640 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x5562665f8300 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa04c26094da4cfb server_cookie=d8d2397fc0348de5 in_seq=0 out_seq=0 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7faca97fa640 1 -- 172.21.15.16:0/3927242155 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55626654e860 con 0x5562665f5f10 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7faca97fa640 10 client.30004 handle_command_reply: tid=0 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 client.30004 shutdown 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 msgr2=0x5562665e3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7facbc0cccb0 tx=0x7facbc097680 comp rx=0 tx=0).stop 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 shutdown_connections 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x5562665f8300 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665e3010 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7faca97fa640 0 client.30004 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 --2- 172.21.15.16:0/3927242155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.629+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 >> 172.21.15.16:0/3927242155 conn(0x5562665a4310 msgr2=0x5562665dd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.630+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 shutdown_connections 2024-09-12T03:11:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.630+0000 7facca4fb740 1 -- 172.21.15.16:0/3927242155 wait complete. 2024-09-12T03:11:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.630+0000 7facca4fb740 20 client.30004 trim_cache size 0 max 16384 2024-09-12T03:11:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.646+0000 7facca4fb740 1 Processor -- start 2024-09-12T03:11:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 -- start start 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faca400e310 con 0x5562665db5d0 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faca400fe50 con 0x5562665e6370 2024-09-12T03:11:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facca4fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faca400e9e0 con 0x5562665dcb40 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46788/0 (socket says 172.21.15.16:46788) 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 -- 172.21.15.16:0/1143775111 learned_addr learned my addr 172.21.15.16:0/1143775111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc66bf640 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc56bd640 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 -- 172.21.15.16:0/1143775111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 msgr2=0x5562665d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665d7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 -- 172.21.15.16:0/1143775111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 msgr2=0x5562665d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665d6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.647+0000 7facc5ebe640 1 -- 172.21.15.16:0/1143775111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562665dc590 con 0x5562665db5d0 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.648+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7facb0005fd0 tx=0x7facb00194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e0549b5e24c203 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.648+0000 7facab7fe640 1 -- 172.21.15.16:0/1143775111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0004310 con 0x5562665db5d0 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.648+0000 7facab7fe640 1 -- 172.21.15.16:0/1143775111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb002f020 con 0x5562665db5d0 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.648+0000 7facab7fe640 1 -- 172.21.15.16:0/1143775111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0003a90 con 0x5562665db5d0 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 -- 172.21.15.16:0/1143775111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 msgr2=0x5562665d2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7facb0005fd0 tx=0x7facb00194c0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 -- 172.21.15.16:0/1143775111 shutdown_connections 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665e6370 0x5562665d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665dcb40 0x5562665d7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 --2- 172.21.15.16:0/1143775111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665d2c60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.649+0000 7facca4fb740 1 -- 172.21.15.16:0/1143775111 >> 172.21.15.16:0/1143775111 conn(0x5562665a4310 msgr2=0x5562665d64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.650+0000 7facca4fb740 1 -- 172.21.15.16:0/1143775111 shutdown_connections 2024-09-12T03:11:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.650+0000 7facca4fb740 1 -- 172.21.15.16:0/1143775111 wait complete. 2024-09-12T03:11:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.650+0000 7facca4fb740 1 -- start start 2024-09-12T03:11:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.650+0000 7facca4fb740 1 Processor -- start 2024-09-12T03:11:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 0x5562665eee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665e6370 0x5562665dfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faca400f780 con 0x5562665db5d0 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faca400f0b0 con 0x5562665dcb40 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facca4fb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faca400c2f0 con 0x5562665e6370 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facc5ebe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46796/0 (socket says 172.21.15.16:46796) 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facc5ebe640 1 -- 172.21.15.16:0/1047517369 learned_addr learned my addr 172.21.15.16:0/1047517369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.651+0000 7facc56bd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 0x5562665eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facca4fb740 10 client.0 _ll_register_callbacks cb 0x7facbc018420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc66bf640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665e6370 0x5562665dfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 -- 172.21.15.16:0/1047517369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665e6370 msgr2=0x5562665dfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665e6370 0x5562665dfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 -- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 msgr2=0x5562665eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 0x5562665eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 -- 172.21.15.16:0/1047517369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562665e0450 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc56bd640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 0x5562665eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7facc5ebe640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7facb0005530 tx=0x7facb0005560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c6545e5e81ca50 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.652+0000 7faca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0038070 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7facb002a040 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb002f020 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7facb0003b20 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7facca4fb740 10 client.20913 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556266660840 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.653+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb002f020 con 0x5562665db5d0 2024-09-12T03:11:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7facb003d4d0 con 0x5562665db5d0 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 10 client.20913 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 10 client.20913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 20 client.20913 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x556266662d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 4 client.20913 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556266548d70 con 0x5562665f5f10 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facc56bd640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x556266662d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7facc56bd640 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x556266662d70 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c36a67eb6ff2a3dd server_cookie=2abfea3910558ea0 in_seq=0 out_seq=0 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.654+0000 7faca97fa640 10 client.20913 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7faca97fa640 1 -- 172.21.15.16:0/1047517369 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556266548d70 con 0x5562665f5f10 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7faca97fa640 10 client.20913 handle_command_reply: tid=0 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 client.20913 shutdown 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 msgr2=0x5562665ee900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7facb0005530 tx=0x7facb0005560 comp rx=0 tx=0).stop 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 shutdown_connections 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5562665f5f10 0x556266662d70 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5562665e6370 0x5562665dfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562665dcb40 0x5562665eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 --2- 172.21.15.16:0/1047517369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562665db5d0 0x5562665ee900 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7faca97fa640 0 client.20913 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 >> 172.21.15.16:0/1047517369 conn(0x5562665a4310 msgr2=0x5562665f1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 shutdown_connections 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 1 -- 172.21.15.16:0/1047517369 wait complete. 2024-09-12T03:11:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.655+0000 7facca4fb740 20 client.20913 trim_cache size 0 max 16384 2024-09-12T03:11:33.661 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:11:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 msgr2=0x7facac0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 0x7facac0a50c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7facb8006010 tx=0x7facb8004a40 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 msgr2=0x7facc0111f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00eeb0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 shutdown_connections 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7facac0a2c00 0x7facac0a50c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc0114ab0 0x7facc0112470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facc0109b10 0x7facc0111f30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 --2- 172.21.15.16:0/2508388133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc01091f0 0x7facc01119f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 >> 172.21.15.16:0/2508388133 conn(0x7facc006e140 msgr2=0x7facc0071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 shutdown_connections 2024-09-12T03:11:33.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:33.657+0000 7facc8122640 1 -- 172.21.15.16:0/2508388133 wait complete. 2024-09-12T03:11:33.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:33 smithi033 ceph-mon[35092]: pgmap v1046: 97 pgs: 97 active+clean; 903 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 498 op/s 2024-09-12T03:11:34.040 INFO:tasks.workunit.client.0.smithi016.stdout: 10 258341 22.80 MB/sec execute 515 sec latency 71.044 ms 2024-09-12T03:11:35.040 INFO:tasks.workunit.client.0.smithi016.stdout: 10 258683 22.80 MB/sec execute 516 sec latency 277.338 ms 2024-09-12T03:11:35.410 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:11:35.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:11:35.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:35 smithi033 ceph-mon[35092]: pgmap v1047: 97 pgs: 97 active+clean; 905 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 557 op/s 2024-09-12T03:11:35.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:35 smithi184 ceph-mon[35321]: pgmap v1047: 97 pgs: 97 active+clean; 905 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 557 op/s 2024-09-12T03:11:35.795 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:35.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:35 smithi016 ceph-mon[30925]: pgmap v1047: 97 pgs: 97 active+clean; 905 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 557 op/s 2024-09-12T03:11:36.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 259129 22.79 MB/sec execute 517 sec latency 81.780 ms 2024-09-12T03:11:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.347+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/47595893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101e20 msgr2=0x7fd6e41056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.347+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/47595893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101e20 0x7fd6e41056b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009900 tx=0x7fd6d402a410 comp rx=0 tx=0).stop 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/47595893 shutdown_connections 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/47595893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4105de0 0x7fd6e410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/47595893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101e20 0x7fd6e41056b0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/47595893 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4101470 0x7fd6e4101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/47595893 >> 172.21.15.16:0/47595893 conn(0x7fd6e4077230 msgr2=0x7fd6e40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/47595893 shutdown_connections 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.348+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/47595893 wait complete. 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 Processor -- start 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 -- start start 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4101e20 0x7fd6e419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 0x7fd6e41a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e4110e40 con 0x7fd6e4101470 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e4110cc0 con 0x7fd6e4101e20 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ec2d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e4110fc0 con 0x7fd6e4105de0 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.349+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57088/0 (socket says 172.21.15.16:57088) 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2716518345 learned_addr learned my addr 172.21.15.16:0/2716518345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea870640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 0x7fd6e41a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6e986e640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4101e20 0x7fd6e419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2716518345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 msgr2=0x7fd6e41a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 0x7fd6e41a32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2716518345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4101e20 msgr2=0x7fd6e419f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4101e20 0x7fd6e419f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2716518345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e41a3a00 con 0x7fd6e4101470 2024-09-12T03:11:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea870640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 0x7fd6e41a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.350+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9f0 tx=0x7fd6e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.351+0000 7fd6db7fe640 1 -- 172.21.15.16:0/2716518345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000cd80 con 0x7fd6e4101470 2024-09-12T03:11:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.351+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e41a3cf0 con 0x7fd6e4101470 2024-09-12T03:11:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.351+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e41ab580 con 0x7fd6e4101470 2024-09-12T03:11:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.351+0000 7fd6db7fe640 1 -- 172.21.15.16:0/2716518345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0005500 con 0x7fd6e4101470 2024-09-12T03:11:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.351+0000 7fd6db7fe640 1 -- 172.21.15.16:0/2716518345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0010690 con 0x7fd6e4101470 2024-09-12T03:11:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 Processor -- start 2024-09-12T03:11:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6db7fe640 1 -- 172.21.15.16:0/2716518345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd6e00046c0 con 0x7fd6e4101470 2024-09-12T03:11:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 -- start start 2024-09-12T03:11:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e79f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e84800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564eb3ddf280 con 0x564eb3e79b60 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564eb3d86b10 con 0x564eb3e84d40 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.354+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564eb3ddb240 con 0x564eb3e7a480 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e84800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42036/0 (socket says 172.21.15.16:42036) 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 -- 172.21.15.16:0/1784792129 learned_addr learned my addr 172.21.15.16:0/1784792129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6ea870640 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e79f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 -- 172.21.15.16:0/1784792129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 msgr2=0x564eb3e84800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e84800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 -- 172.21.15.16:0/1784792129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 msgr2=0x564eb3e79f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e79f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6e986e640 1 -- 172.21.15.16:0/1784792129 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb3d931f0 con 0x564eb3e84d40 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.355+0000 7fd6db7fe640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 0x7fd6d00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6e986e640 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009900 tx=0x7fd6d4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e75d1da188035e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6db7fe640 1 -- 172.21.15.16:0/2716518345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6e00cf0e0 con 0x7fd6e4101470 2024-09-12T03:11:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6d8dea640 1 -- 172.21.15.16:0/1784792129 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4040070 con 0x564eb3e84d40 2024-09-12T03:11:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6d8dea640 1 -- 172.21.15.16:0/1784792129 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4031870 con 0x564eb3e84d40 2024-09-12T03:11:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6d8dea640 1 -- 172.21.15.16:0/1784792129 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4035040 con 0x564eb3e84d40 2024-09-12T03:11:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.356+0000 7fd6ea870640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 0x7fd6d00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ea870640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 0x7fd6d00a50c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009fd0 tx=0x7fd6dc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/1784792129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 msgr2=0x564eb3e87130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009900 tx=0x7fd6d4005be0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/1784792129 shutdown_connections 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e87130 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e84800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/1784792129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e79f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.357+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/1784792129 >> 172.21.15.16:0/1784792129 conn(0x564eb3e760f0 msgr2=0x564eb3e764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/1784792129 shutdown_connections 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/1784792129 wait complete. 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- start start 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 Processor -- start 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e84d40 0x564eb3e82610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564eb3e87910 con 0x564eb3e84d40 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564eb3d99f10 con 0x564eb3e79b60 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564eb3e87a90 con 0x564eb3e7a480 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ee6ac740 10 client.0 _ll_register_callbacks cb 0x564eb3d62a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.358+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:42052/0 (socket says 172.21.15.16:42052) 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2422686017 learned_addr learned my addr 172.21.15.16:0/2422686017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6e986e640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea870640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e84d40 0x564eb3e82610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2422686017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 msgr2=0x564eb3e820d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e820d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2422686017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e84d40 msgr2=0x564eb3e82610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e84d40 0x564eb3e82610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 -- 172.21.15.16:0/2422686017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb3b506a0 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9c0 tx=0x7fd6e0097bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47faec9cb59dcb2b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e009b2b0 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6e00a4130 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.359+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6d4031870 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd6e009c050 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 10 client.30077 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564eb3e93590 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4031870 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd6e00860d0 con 0x564eb3e79b60 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 10 client.30077 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 10 client.30077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 20 client.30077 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3e94a50 0x564eb3e96ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 4 client.30077 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eb3dec7f0 con 0x564eb3e94a50 2024-09-12T03:11:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.360+0000 7fd6e986e640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3e94a50 0x564eb3e96ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.361+0000 7fd6cdffb640 10 client.30077 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.361+0000 7fd6e986e640 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3e94a50 0x564eb3e96ea0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f06c944d0f5d49b server_cookie=fdcfcae932e9fa7b in_seq=0 out_seq=0 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6cdffb640 1 -- 172.21.15.16:0/2422686017 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564eb3dec7f0 con 0x564eb3e94a50 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6cdffb640 10 client.30077 handle_command_reply: tid=0 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 client.30077 shutdown 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 msgr2=0x564eb3e81b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9c0 tx=0x7fd6e0097bf0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 shutdown_connections 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3e94a50 0x564eb3e96ea0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e84d40 0x564eb3e82610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e7a480 0x564eb3e820d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6cdffb640 0 client.30077 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/2422686017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e79b60 0x564eb3e81b90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 >> 172.21.15.16:0/2422686017 conn(0x564eb3e760f0 msgr2=0x564eb3e7be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 shutdown_connections 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/2422686017 wait complete. 2024-09-12T03:11:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.363+0000 7fd6ee6ac740 20 client.30077 trim_cache size 0 max 16384 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.383+0000 7fd6ee6ac740 1 Processor -- start 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.383+0000 7fd6ee6ac740 1 -- start start 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e79b60 0x564eb3e7bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e9ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800e3b0 con 0x564eb3e7a480 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800fef0 con 0x564eb3e84d40 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800ea80 con 0x564eb3e79b60 2024-09-12T03:11:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 -- 172.21.15.16:0/302855763 learned_addr learned my addr 172.21.15.16:0/302855763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ea870640 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e9ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e79b60 0x564eb3e7bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 -- 172.21.15.16:0/302855763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e79b60 msgr2=0x564eb3e7bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e79b60 0x564eb3e7bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 -- 172.21.15.16:0/302855763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 msgr2=0x564eb3e9ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e9ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.384+0000 7fd6e986e640 1 -- 172.21.15.16:0/302855763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb3e76de0 con 0x564eb3e7a480 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.385+0000 7fd6e986e640 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x564eb3e77470 tx=0x7fd6d4005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ab6ce35446380a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.385+0000 7fd6cffff640 1 -- 172.21.15.16:0/302855763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4055070 con 0x564eb3e7a480 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.385+0000 7fd6cffff640 1 -- 172.21.15.16:0/302855763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4031870 con 0x564eb3e7a480 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.385+0000 7fd6cffff640 1 -- 172.21.15.16:0/302855763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4040040 con 0x564eb3e7a480 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/302855763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 msgr2=0x564eb3e9c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x564eb3e77470 tx=0x7fd6d4005420 comp rx=0 tx=0).stop 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/302855763 shutdown_connections 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e84d40 0x564eb3e9ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e7a480 0x564eb3e9c770 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/302855763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e79b60 0x564eb3e7bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/302855763 >> 172.21.15.16:0/302855763 conn(0x564eb3e760f0 msgr2=0x564eb3e7b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/302855763 shutdown_connections 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/302855763 wait complete. 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 -- start start 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.386+0000 7fd6ee6ac740 1 Processor -- start 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 0x564eb3e7f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e84d40 0x564eb3e7f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800f820 con 0x564eb3e79b60 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800f150 con 0x564eb3e7a480 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c800c390 con 0x564eb3e84d40 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ee6ac740 10 client.0 _ll_register_callbacks cb 0x7fd6e00d28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6e986e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 0x564eb3e7f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57116/0 (socket says 172.21.15.16:57116) 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 -- 172.21.15.16:0/4087643399 learned_addr learned my addr 172.21.15.16:0/4087643399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea870640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e84d40 0x564eb3e7f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 -- 172.21.15.16:0/4087643399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e84d40 msgr2=0x564eb3e7f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e84d40 0x564eb3e7f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 -- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 msgr2=0x564eb3e7f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 0x564eb3e7f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 -- 172.21.15.16:0/4087643399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb3efe740 con 0x564eb3e79b60 2024-09-12T03:11:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6e986e640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 0x564eb3e7f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6ea06f640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fd6e009b8e0 tx=0x7fd6e0004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41345d9ea50c74cc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.387+0000 7fd6cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e009b2b0 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6e00b4b50 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6d4031870 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd6e009b720 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 10 client.20931 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564eb3eff9a0 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4031870 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd6e009ada0 con 0x564eb3e79b60 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 10 client.20931 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 10 client.20931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 20 client.20931 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3efff80 0x564eb3f02370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 4 client.20931 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.388+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564eb3de6d00 con 0x564eb3efff80 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.389+0000 7fd6e986e640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3efff80 0x564eb3f02370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.389+0000 7fd6cdffb640 10 client.20931 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.389+0000 7fd6e986e640 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3efff80 0x564eb3f02370 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd75408e6403b05 server_cookie=867f8e83a6ced204 in_seq=0 out_seq=0 2024-09-12T03:11:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.391+0000 7fd6cdffb640 1 -- 172.21.15.16:0/4087643399 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564eb3de6d00 con 0x564eb3efff80 2024-09-12T03:11:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.391+0000 7fd6cdffb640 10 client.20931 handle_command_reply: tid=0 2024-09-12T03:11:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.391+0000 7fd6ee6ac740 1 client.20931 shutdown 2024-09-12T03:11:36.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 msgr2=0x564eb3e7ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fd6e009b8e0 tx=0x7fd6e0004e00 comp rx=0 tx=0).stop 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 shutdown_connections 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564eb3efff80 0x564eb3f02370 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564eb3e84d40 0x564eb3e7f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564eb3e7a480 0x564eb3e7f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 --2- 172.21.15.16:0/4087643399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564eb3e79b60 0x564eb3e7ead0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 >> 172.21.15.16:0/4087643399 conn(0x564eb3e760f0 msgr2=0x564eb3e75270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6cdffb640 0 client.20931 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 shutdown_connections 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.392+0000 7fd6ee6ac740 1 -- 172.21.15.16:0/4087643399 wait complete. 2024-09-12T03:11:36.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.393+0000 7fd6ee6ac740 20 client.20931 trim_cache size 0 max 16384 2024-09-12T03:11:36.396 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:11:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:11:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "344ea588-0d7c-444c-9c4d-89705e3bd620", 2024-09-12T03:11:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:11:36.396 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:11:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.393+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 msgr2=0x7fd6d00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.393+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 0x7fd6d00a50c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009fd0 tx=0x7fd6dc009500 comp rx=0 tx=0).stop 2024-09-12T03:11:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.393+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 msgr2=0x7fd6e419ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.393+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9f0 tx=0x7fd6e000edf0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 shutdown_connections 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6e4105de0 0x7fd6e41a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6e4101e20 0x7fd6e419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd6d00a2c00 0x7fd6d00a50c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 --2- 172.21.15.16:0/2716518345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e4101470 0x7fd6e419ecb0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 >> 172.21.15.16:0/2716518345 conn(0x7fd6e4077230 msgr2=0x7fd6e4103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 shutdown_connections 2024-09-12T03:11:36.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:36.394+0000 7fd6ec2d3640 1 -- 172.21.15.16:0/2716518345 wait complete. 2024-09-12T03:11:37.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 259718 22.80 MB/sec execute 518 sec latency 84.030 ms 2024-09-12T03:11:37.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:11:37.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: pgmap v1048: 97 pgs: 97 active+clean; 914 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 652 op/s 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:11:37.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:37 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:11:37.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: pgmap v1048: 97 pgs: 97 active+clean; 914 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 652 op/s 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:37 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: pgmap v1048: 97 pgs: 97 active+clean; 914 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 652 op/s 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:37.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:37 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:11:38.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 260453 22.83 MB/sec execute 519 sec latency 53.576 ms 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 -- 172.21.15.16:0/3630767215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c102ef0 msgr2=0x7f028c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 --2- 172.21.15.16:0/3630767215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c102ef0 0x7f028c10da40 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f0284009ca0 tx=0x7f028402a950 comp rx=0 tx=0).stop 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 -- 172.21.15.16:0/3630767215 shutdown_connections 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 --2- 172.21.15.16:0/3630767215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c102ef0 0x7f028c10da40 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 --2- 172.21.15.16:0/3630767215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f028c102570 0x7f028c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 --2- 172.21.15.16:0/3630767215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c100e20 0x7f028c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 -- 172.21.15.16:0/3630767215 >> 172.21.15.16:0/3630767215 conn(0x7f028c0fc7d0 msgr2=0x7f028c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 -- 172.21.15.16:0/3630767215 shutdown_connections 2024-09-12T03:11:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.383+0000 7f0292b67640 1 -- 172.21.15.16:0/3630767215 wait complete. 2024-09-12T03:11:38.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 Processor -- start 2024-09-12T03:11:38.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 -- start start 2024-09-12T03:11:38.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f028c102570 0x7f028c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c102ef0 0x7f028c1a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f028c110f80 con 0x7f028c100e20 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f028c110e00 con 0x7f028c102570 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0292b67640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f028c111100 con 0x7f028c102ef0 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57134/0 (socket says 172.21.15.16:57134) 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0290903640 1 -- 172.21.15.16:0/921310428 learned_addr learned my addr 172.21.15.16:0/921310428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.384+0000 7f0291104640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c102ef0 0x7f028c1a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 -- 172.21.15.16:0/921310428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c102ef0 msgr2=0x7f028c1a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c102ef0 0x7f028c1a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 -- 172.21.15.16:0/921310428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f028c102570 msgr2=0x7f028c1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f028c102570 0x7f028c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 -- 172.21.15.16:0/921310428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c1a4930 con 0x7f028c100e20 2024-09-12T03:11:38.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0290903640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f028000e9f0 tx=0x7f028000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0289ffb640 1 -- 172.21.15.16:0/921310428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028000cd80 con 0x7f028c100e20 2024-09-12T03:11:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c1a4b00 con 0x7f028c100e20 2024-09-12T03:11:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.385+0000 7f0289ffb640 1 -- 172.21.15.16:0/921310428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0280010910 con 0x7f028c100e20 2024-09-12T03:11:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.386+0000 7f0289ffb640 1 -- 172.21.15.16:0/921310428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280018990 con 0x7f028c100e20 2024-09-12T03:11:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.386+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c1a5030 con 0x7f028c100e20 2024-09-12T03:11:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.389+0000 7f0289ffb640 1 -- 172.21.15.16:0/921310428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0280010ac0 con 0x7f028c100e20 2024-09-12T03:11:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.389+0000 7f0294f40740 1 Processor -- start 2024-09-12T03:11:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.389+0000 7f0294f40740 1 -- start start 2024-09-12T03:11:38.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0289ffb640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 0x7f027c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0289ffb640 1 -- 172.21.15.16:0/921310428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f02800cf390 con 0x7f028c100e20 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602720880 0x558602720cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0291104640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 0x7f027c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602736110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5586026862e0 con 0x558602733d20 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55860262dce0 con 0x558602720880 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0294f40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5586026822a0 con 0x558602721a20 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0290903640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f028bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602720880 0x558602720cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0290903640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57110/0 (socket says 172.21.15.16:57110) 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0290903640 1 -- 172.21.15.16:0/2201105623 learned_addr learned my addr 172.21.15.16:0/2201105623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.390+0000 7f0291104640 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602736110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 -- 172.21.15.16:0/2201105623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602720880 msgr2=0x558602720cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602720880 0x558602720cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 -- 172.21.15.16:0/2201105623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 msgr2=0x558602736110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602736110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 -- 172.21.15.16:0/2201105623 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558602612440 con 0x558602721a20 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0291104640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 0x7f027c0a5040 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f028c06b050 tx=0x7f028402fa10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0290903640 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f028009b900 tx=0x7f028009c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c35e269926facc1d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0276ffd640 1 -- 172.21.15.16:0/2201105623 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b2030 con 0x558602721a20 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0276ffd640 1 -- 172.21.15.16:0/2201105623 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02800a4870 con 0x558602721a20 2024-09-12T03:11:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.391+0000 7f0276ffd640 1 -- 172.21.15.16:0/2201105623 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280018b30 con 0x558602721a20 2024-09-12T03:11:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 -- 172.21.15.16:0/2201105623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 msgr2=0x558602720340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f028009b900 tx=0x7f028009c3c0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 -- 172.21.15.16:0/2201105623 shutdown_connections 2024-09-12T03:11:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602736110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602720880 0x558602720cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 --2- 172.21.15.16:0/2201105623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602720340 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 -- 172.21.15.16:0/2201105623 >> 172.21.15.16:0/2201105623 conn(0x55860271ca30 msgr2=0x55860271ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 -- 172.21.15.16:0/2201105623 shutdown_connections 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.392+0000 7f0294f40740 1 -- 172.21.15.16:0/2201105623 wait complete. 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.393+0000 7f0294f40740 1 -- start start 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.393+0000 7f0294f40740 1 Processor -- start 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.393+0000 7f0294f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.393+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.393+0000 7f0294f40740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 0x558602723d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0294f40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x558602724260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0294f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558602736a50 con 0x558602720880 2024-09-12T03:11:38.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0294f40740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558602641050 con 0x558602721a20 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0294f40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558602736bd0 con 0x558602733d20 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0294f40740 10 client.0 _ll_register_callbacks cb 0x558602681160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57142/0 (socket says 172.21.15.16:57142) 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0290903640 1 -- 172.21.15.16:0/4023825655 learned_addr learned my addr 172.21.15.16:0/4023825655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f028bfff640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 0x558602723d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.394+0000 7f0291104640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x558602724260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 -- 172.21.15.16:0/4023825655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 msgr2=0x558602724260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x558602724260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 -- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 msgr2=0x558602723d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 0x558602723d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 -- 172.21.15.16:0/4023825655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586023f73c0 con 0x558602720880 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f028bfff640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 0x558602723d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:38.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0291104640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x558602724260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0290903640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f02800ccef0 tx=0x7f02800a0070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ebc638fa9724c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f026bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b8070 con 0x558602720880 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f02800b4e00 con 0x558602720880 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02800a4870 con 0x558602720880 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f02800b2030 con 0x558602720880 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0294f40740 10 client.20943 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.395+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5586027423e0 con 0x558602720880 2024-09-12T03:11:38.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02800aa070 con 0x558602720880 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f02800ad020 con 0x558602720880 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 10 client.20943 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 10 client.20943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 20 client.20943 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558602743730 0x558602745b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 4 client.20943 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:11:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f02800a4870 con 0x558602743730 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.396+0000 7f028bfff640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558602743730 0x558602745b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.397+0000 7f026bfff640 10 client.20943 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.397+0000 7f028bfff640 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558602743730 0x558602745b20 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba8dd50a01067f70 server_cookie=de042f4b516cc28c in_seq=0 out_seq=0 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f026bfff640 1 -- 172.21.15.16:0/4023825655 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f02800a4870 con 0x558602743730 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f026bfff640 10 client.20943 handle_command_reply: tid=0 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 client.20943 shutdown 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 msgr2=0x558602726a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f02800ccef0 tx=0x7f02800a0070 comp rx=0 tx=0).stop 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 shutdown_connections 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x558602743730 0x558602745b20 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x558602724260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602721a20 0x558602723d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 --2- 172.21.15.16:0/4023825655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602720880 0x558602726a70 secure :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f02800ccef0 tx=0x7f02800a0070 comp rx=0 tx=0).stop 2024-09-12T03:11:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.398+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 >> 172.21.15.16:0/4023825655 conn(0x55860271ca30 msgr2=0x558602722750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.399+0000 7f026bfff640 0 client.20943 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.399+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 shutdown_connections 2024-09-12T03:11:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.399+0000 7f0294f40740 1 -- 172.21.15.16:0/4023825655 wait complete. 2024-09-12T03:11:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.399+0000 7f0294f40740 20 client.20943 trim_cache size 0 max 16384 2024-09-12T03:11:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.427+0000 7f0294f40740 1 Processor -- start 2024-09-12T03:11:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.427+0000 7f0294f40740 1 -- start start 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x55860271ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602731bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1ab480 con 0x558602721a20 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1acfc0 con 0x558602733920 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0294f40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1abb50 con 0x558602733d20 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f028bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x55860271ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0291104640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602731bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57148/0 (socket says 172.21.15.16:57148) 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 -- 172.21.15.16:0/2257120768 learned_addr learned my addr 172.21.15.16:0/2257120768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 -- 172.21.15.16:0/2257120768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 msgr2=0x55860271ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x55860271ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 -- 172.21.15.16:0/2257120768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 msgr2=0x558602731bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602731bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0290903640 1 -- 172.21.15.16:0/2257120768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558602721290 con 0x558602721a20 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.428+0000 7f0291104640 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602731bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.429+0000 7f0290903640 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0280097a20 tx=0x7f028009bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55205ab029c6b8a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.429+0000 7f02767fc640 1 -- 172.21.15.16:0/2257120768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b8070 con 0x558602721a20 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.429+0000 7f02767fc640 1 -- 172.21.15.16:0/2257120768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02800a4870 con 0x558602721a20 2024-09-12T03:11:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.429+0000 7f02767fc640 1 -- 172.21.15.16:0/2257120768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800ba2c0 con 0x558602721a20 2024-09-12T03:11:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- 172.21.15.16:0/2257120768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 msgr2=0x558602722e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0280097a20 tx=0x7f028009bb00 comp rx=0 tx=0).stop 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- 172.21.15.16:0/2257120768 shutdown_connections 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602731bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602733d20 0x55860271ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 --2- 172.21.15.16:0/2257120768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602721a20 0x558602722e10 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- 172.21.15.16:0/2257120768 >> 172.21.15.16:0/2257120768 conn(0x55860271ca30 msgr2=0x55860271b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- 172.21.15.16:0/2257120768 shutdown_connections 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- 172.21.15.16:0/2257120768 wait complete. 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 -- start start 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.430+0000 7f0294f40740 1 Processor -- start 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602748f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602746250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602746790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1ac8f0 con 0x558602733d20 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1ac220 con 0x558602733920 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a9460 con 0x558602721a20 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0294f40740 10 client.0 _ll_register_callbacks cb 0x7f0280018e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.431+0000 7f0291104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602746790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602748f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0291104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602746790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57154/0 (socket says 172.21.15.16:57154) 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0291104640 1 -- 172.21.15.16:0/1773998243 learned_addr learned my addr 172.21.15.16:0/1773998243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f028bfff640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602746250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 -- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 msgr2=0x558602746250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602746250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 -- 172.21.15.16:0/1773998243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 msgr2=0x558602746790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:11:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602746790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 -- 172.21.15.16:0/1773998243 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558602746f90 con 0x558602721a20 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0290903640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602748f00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0280097a20 tx=0x7f028009b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c9ac89bf9683d46 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f026bfff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b8030 con 0x558602721a20 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f02800b4e00 con 0x558602721a20 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.432+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02800a4870 con 0x558602721a20 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0280097ea0 con 0x558602721a20 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f0294f40740 10 client.30046 fetch_fsmap learned FSMap version 39 2024-09-12T03:11:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558602746f90 con 0x558602721a20 2024-09-12T03:11:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02800a4870 con 0x558602721a20 2024-09-12T03:11:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f02800ad070 con 0x558602721a20 2024-09-12T03:11:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f0294f40740 10 client.30046 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:11:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.433+0000 7f0294f40740 10 client.30046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.434+0000 7f0294f40740 20 client.30046 populate_metadata read hostname 'smithi016' 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.434+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586027ae360 0x5586027b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.434+0000 7f0294f40740 4 client.30046 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.434+0000 7f028bfff640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586027ae360 0x5586027b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.434+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558602693850 con 0x5586027ae360 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f026bfff640 10 client.30046 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f028bfff640 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586027ae360 0x5586027b0750 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c400defd1a832c8 server_cookie=eba36a7cbcbe4154 in_seq=0 out_seq=0 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f026bfff640 1 -- 172.21.15.16:0/1773998243 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558602693850 con 0x5586027ae360 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f026bfff640 10 client.30046 handle_command_reply: tid=0 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 client.30046 shutdown 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 msgr2=0x558602748f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602748f00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0280097a20 tx=0x7f028009b9d0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 shutdown_connections 2024-09-12T03:11:38.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5586027ae360 0x5586027b0750 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558602733d20 0x558602746790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558602733920 0x558602746250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 --2- 172.21.15.16:0/1773998243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558602721a20 0x558602748f00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 >> 172.21.15.16:0/1773998243 conn(0x55860271ca30 msgr2=0x55860271bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.435+0000 7f026bfff640 0 client.30046 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.436+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 shutdown_connections 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.436+0000 7f0294f40740 1 -- 172.21.15.16:0/1773998243 wait complete. 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.436+0000 7f0294f40740 20 client.30046 trim_cache size 0 max 16384 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:11:38.440 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: "344ea588-0d7c-444c-9c4d-89705e3bd620": { 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "344ea588-0d7c-444c-9c4d-89705e3bd620", 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:11:38.441 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 msgr2=0x7f027c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 0x7f027c0a5040 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f028c06b050 tx=0x7f028402fa10 comp rx=0 tx=0).stop 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 msgr2=0x7f028c199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f028000e9f0 tx=0x7f028000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 shutdown_connections 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c102ef0 0x7f028c1a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f028c102570 0x7f028c1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f027c0a2b80 0x7f027c0a5040 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 --2- 172.21.15.16:0/921310428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f028c100e20 0x7f028c199370 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:11:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 >> 172.21.15.16:0/921310428 conn(0x7f028c0fc7d0 msgr2=0x7f028c1068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:11:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 shutdown_connections 2024-09-12T03:11:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:11:38.437+0000 7f0292b67640 1 -- 172.21.15.16:0/921310428 wait complete. 2024-09-12T03:11:39.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 261106 22.86 MB/sec execute 520 sec latency 51.032 ms 2024-09-12T03:11:39.149 INFO:tasks.cephfs.filesystem:scrub status for tag:344ea588-0d7c-444c-9c4d-89705e3bd620 - {'path': '/', 'tag': '344ea588-0d7c-444c-9c4d-89705e3bd620', 'options': 'recursive,force'} 2024-09-12T03:11:39.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:39 smithi033 ceph-mon[35092]: pgmap v1049: 97 pgs: 97 active+clean; 915 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 600 op/s 2024-09-12T03:11:39.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:39 smithi184 ceph-mon[35321]: pgmap v1049: 97 pgs: 97 active+clean; 915 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 600 op/s 2024-09-12T03:11:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:39 smithi016 ceph-mon[30925]: pgmap v1049: 97 pgs: 97 active+clean; 915 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 600 op/s 2024-09-12T03:11:40.049 INFO:tasks.workunit.client.0.smithi016.stdout: 10 261618 22.87 MB/sec execute 521 sec latency 204.138 ms 2024-09-12T03:11:41.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 262319 22.89 MB/sec execute 522 sec latency 62.838 ms 2024-09-12T03:11:41.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:41 smithi033 ceph-mon[35092]: pgmap v1050: 97 pgs: 97 active+clean; 918 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 724 op/s 2024-09-12T03:11:41.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:41 smithi184 ceph-mon[35321]: pgmap v1050: 97 pgs: 97 active+clean; 918 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 724 op/s 2024-09-12T03:11:41.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:41 smithi016 ceph-mon[30925]: pgmap v1050: 97 pgs: 97 active+clean; 918 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 724 op/s 2024-09-12T03:11:42.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 262979 22.93 MB/sec execute 523 sec latency 46.167 ms 2024-09-12T03:11:43.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 263452 22.94 MB/sec execute 524 sec latency 63.857 ms 2024-09-12T03:11:43.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:43 smithi033 ceph-mon[35092]: pgmap v1051: 97 pgs: 97 active+clean; 909 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:11:43.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:43 smithi184 ceph-mon[35321]: pgmap v1051: 97 pgs: 97 active+clean; 909 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:11:43.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:43 smithi016 ceph-mon[30925]: pgmap v1051: 97 pgs: 97 active+clean; 909 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.0 MiB/s wr, 679 op/s 2024-09-12T03:11:44.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 264001 22.94 MB/sec execute 525 sec latency 113.156 ms 2024-09-12T03:11:45.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 264425 22.93 MB/sec execute 526 sec latency 373.327 ms 2024-09-12T03:11:45.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:45 smithi033 ceph-mon[35092]: pgmap v1052: 97 pgs: 97 active+clean; 900 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.6 MiB/s wr, 668 op/s 2024-09-12T03:11:45.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:45 smithi184 ceph-mon[35321]: pgmap v1052: 97 pgs: 97 active+clean; 900 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.6 MiB/s wr, 668 op/s 2024-09-12T03:11:45.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:45 smithi016 ceph-mon[30925]: pgmap v1052: 97 pgs: 97 active+clean; 900 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.6 MiB/s wr, 668 op/s 2024-09-12T03:11:46.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 265055 22.95 MB/sec execute 527 sec latency 98.806 ms 2024-09-12T03:11:47.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 265767 23.00 MB/sec execute 528 sec latency 55.677 ms 2024-09-12T03:11:47.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:47 smithi033 ceph-mon[35092]: pgmap v1053: 97 pgs: 97 active+clean; 912 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.1 MiB/s wr, 692 op/s 2024-09-12T03:11:47.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:47 smithi184 ceph-mon[35321]: pgmap v1053: 97 pgs: 97 active+clean; 912 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.1 MiB/s wr, 692 op/s 2024-09-12T03:11:47.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:47 smithi016 ceph-mon[30925]: pgmap v1053: 97 pgs: 97 active+clean; 912 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.1 MiB/s wr, 692 op/s 2024-09-12T03:11:48.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 266458 23.01 MB/sec execute 529 sec latency 46.718 ms 2024-09-12T03:11:48.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:48.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:48.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:48.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:48 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:48.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:48.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:48.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:48.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:48 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:48.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:48.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:48.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:48.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:48 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:49.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 267121 23.03 MB/sec execute 530 sec latency 54.302 ms 2024-09-12T03:11:49.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:49 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:11:49.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:49 smithi033 ceph-mon[35092]: pgmap v1054: 97 pgs: 97 active+clean; 913 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 617 op/s 2024-09-12T03:11:49.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:49 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:11:49.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:49 smithi184 ceph-mon[35321]: pgmap v1054: 97 pgs: 97 active+clean; 913 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 617 op/s 2024-09-12T03:11:49.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:49 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:11:49.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:49 smithi016 ceph-mon[30925]: pgmap v1054: 97 pgs: 97 active+clean; 913 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 617 op/s 2024-09-12T03:11:50.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 267577 23.05 MB/sec execute 531 sec latency 230.849 ms 2024-09-12T03:11:51.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 268025 23.06 MB/sec execute 532 sec latency 70.826 ms 2024-09-12T03:11:51.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:51 smithi033 ceph-mon[35092]: pgmap v1055: 97 pgs: 97 active+clean; 916 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 683 op/s 2024-09-12T03:11:51.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:51 smithi184 ceph-mon[35321]: pgmap v1055: 97 pgs: 97 active+clean; 916 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 683 op/s 2024-09-12T03:11:51.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:51 smithi016 ceph-mon[30925]: pgmap v1055: 97 pgs: 97 active+clean; 916 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 683 op/s 2024-09-12T03:11:52.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 268556 23.06 MB/sec execute 533 sec latency 85.086 ms 2024-09-12T03:11:53.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 269324 23.08 MB/sec execute 534 sec latency 113.009 ms 2024-09-12T03:11:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:53 smithi184 ceph-mon[35321]: pgmap v1056: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 647 op/s 2024-09-12T03:11:53.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:53 smithi016 ceph-mon[30925]: pgmap v1056: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 647 op/s 2024-09-12T03:11:53.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:53 smithi033 ceph-mon[35092]: pgmap v1056: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.4 MiB/s wr, 647 op/s 2024-09-12T03:11:54.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 270041 23.12 MB/sec execute 535 sec latency 98.939 ms 2024-09-12T03:11:55.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 270517 23.13 MB/sec execute 536 sec latency 263.670 ms 2024-09-12T03:11:55.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:55 smithi184 ceph-mon[35321]: pgmap v1057: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 587 op/s 2024-09-12T03:11:55.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:55 smithi016 ceph-mon[30925]: pgmap v1057: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 587 op/s 2024-09-12T03:11:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:55 smithi033 ceph-mon[35092]: pgmap v1057: 97 pgs: 97 active+clean; 921 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 587 op/s 2024-09-12T03:11:56.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 271036 23.13 MB/sec execute 537 sec latency 252.421 ms 2024-09-12T03:11:57.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 271642 23.15 MB/sec execute 538 sec latency 52.177 ms 2024-09-12T03:11:57.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:57 smithi184 ceph-mon[35321]: pgmap v1058: 97 pgs: 97 active+clean; 924 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 667 op/s 2024-09-12T03:11:57.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:57 smithi016 ceph-mon[30925]: pgmap v1058: 97 pgs: 97 active+clean; 924 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 667 op/s 2024-09-12T03:11:57.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:57 smithi033 ceph-mon[35092]: pgmap v1058: 97 pgs: 97 active+clean; 924 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.3 MiB/s wr, 667 op/s 2024-09-12T03:11:58.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 272267 23.17 MB/sec execute 539 sec latency 52.607 ms 2024-09-12T03:11:59.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 272843 23.19 MB/sec execute 540 sec latency 84.047 ms 2024-09-12T03:11:59.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:11:59 smithi184 ceph-mon[35321]: pgmap v1059: 97 pgs: 97 active+clean; 927 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 635 op/s 2024-09-12T03:11:59.798 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:11:59 smithi033 ceph-mon[35092]: pgmap v1059: 97 pgs: 97 active+clean; 927 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 635 op/s 2024-09-12T03:11:59.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:11:59 smithi016 ceph-mon[30925]: pgmap v1059: 97 pgs: 97 active+clean; 927 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 635 op/s 2024-09-12T03:12:00.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 273507 23.20 MB/sec execute 541 sec latency 146.245 ms 2024-09-12T03:12:01.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 274013 23.21 MB/sec execute 542 sec latency 186.189 ms 2024-09-12T03:12:01.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:01 smithi184 ceph-mon[35321]: pgmap v1060: 97 pgs: 97 active+clean; 933 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.6 MiB/s wr, 707 op/s 2024-09-12T03:12:01.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:01 smithi016 ceph-mon[30925]: pgmap v1060: 97 pgs: 97 active+clean; 933 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.6 MiB/s wr, 707 op/s 2024-09-12T03:12:01.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:01 smithi033 ceph-mon[35092]: pgmap v1060: 97 pgs: 97 active+clean; 933 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.6 MiB/s wr, 707 op/s 2024-09-12T03:12:02.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 274762 23.26 MB/sec execute 543 sec latency 59.390 ms 2024-09-12T03:12:03.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 275303 23.27 MB/sec execute 544 sec latency 52.938 ms 2024-09-12T03:12:03.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:03 smithi184 ceph-mon[35321]: pgmap v1061: 97 pgs: 97 active+clean; 937 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 666 op/s 2024-09-12T03:12:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:03 smithi016 ceph-mon[30925]: pgmap v1061: 97 pgs: 97 active+clean; 937 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 666 op/s 2024-09-12T03:12:03.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:03 smithi033 ceph-mon[35092]: pgmap v1061: 97 pgs: 97 active+clean; 937 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 666 op/s 2024-09-12T03:12:04.041 INFO:tasks.workunit.client.0.smithi016.stdout: 10 275876 23.27 MB/sec execute 545 sec latency 62.687 ms 2024-09-12T03:12:05.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 276537 23.29 MB/sec execute 546 sec latency 80.790 ms 2024-09-12T03:12:05.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:05 smithi184 ceph-mon[35321]: pgmap v1062: 97 pgs: 97 active+clean; 941 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 611 op/s 2024-09-12T03:12:05.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:05 smithi016 ceph-mon[30925]: pgmap v1062: 97 pgs: 97 active+clean; 941 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 611 op/s 2024-09-12T03:12:05.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:05 smithi033 ceph-mon[35092]: pgmap v1062: 97 pgs: 97 active+clean; 941 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 611 op/s 2024-09-12T03:12:06.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 276925 23.29 MB/sec execute 547 sec latency 320.920 ms 2024-09-12T03:12:07.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 277566 23.31 MB/sec execute 548 sec latency 62.765 ms 2024-09-12T03:12:07.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:07 smithi184 ceph-mon[35321]: pgmap v1063: 97 pgs: 97 active+clean; 927 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 644 op/s 2024-09-12T03:12:07.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:07 smithi016 ceph-mon[30925]: pgmap v1063: 97 pgs: 97 active+clean; 927 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 644 op/s 2024-09-12T03:12:07.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:07 smithi033 ceph-mon[35092]: pgmap v1063: 97 pgs: 97 active+clean; 927 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.4 MiB/s wr, 644 op/s 2024-09-12T03:12:08.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 278300 23.33 MB/sec execute 549 sec latency 50.304 ms 2024-09-12T03:12:09.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 279082 23.37 MB/sec execute 550 sec latency 49.089 ms 2024-09-12T03:12:09.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:12:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:09 smithi184 ceph-mon[35321]: pgmap v1064: 97 pgs: 97 active+clean; 935 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-09-12T03:12:09.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:09 smithi016 ceph-mon[30925]: pgmap v1064: 97 pgs: 97 active+clean; 935 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-09-12T03:12:09.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:09 smithi033 ceph-mon[35092]: pgmap v1064: 97 pgs: 97 active+clean; 935 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 611 op/s 2024-09-12T03:12:10.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 279625 23.40 MB/sec execute 551 sec latency 91.523 ms 2024-09-12T03:12:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.204+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/3899835798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc41083f0 msgr2=0x7f7bc41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.204+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/3899835798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc41083f0 0x7f7bc41087d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009be0 tx=0x7f7bbc02a6c0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.208+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/3899835798 shutdown_connections 2024-09-12T03:12:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.208+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/3899835798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc410d520 0x7f7bc41150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.208+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/3899835798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc4108da0 0x7f7bc410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.208+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/3899835798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc41083f0 0x7f7bc41087d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.208+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/3899835798 >> 172.21.15.16:0/3899835798 conn(0x7f7bc4077af0 msgr2=0x7f7bc4079f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.209+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/3899835798 shutdown_connections 2024-09-12T03:12:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.209+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/3899835798 wait complete. 2024-09-12T03:12:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 Processor -- start 2024-09-12T03:12:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 -- start start 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 0x7f7bc407b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc410d520 0x7f7bc407bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc4118540 con 0x7f7bc41083f0 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41183c0 con 0x7f7bc410d520 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.210+0000 7f7bc8f0a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41186c0 con 0x7f7bc4108da0 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.211+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.211+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36824/0 (socket says 172.21.15.16:36824) 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.211+0000 7f7bc259c640 1 -- 172.21.15.16:0/690275734 learned_addr learned my addr 172.21.15.16:0/690275734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.211+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 0x7f7bc407b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/690275734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc410d520 msgr2=0x7f7bc407bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc410d520 0x7f7bc407bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 msgr2=0x7f7bc40827f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/690275734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc407c380 con 0x7f7bc4108da0 2024-09-12T03:12:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.212+0000 7f7bc259c640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.213+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 0x7f7bc407b880 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb400ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.218+0000 7f7bb37fe640 1 -- 172.21.15.16:0/690275734 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb400ccc0 con 0x7f7bc4108da0 2024-09-12T03:12:10.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.218+0000 7f7bb37fe640 1 -- 172.21.15.16:0/690275734 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb400ee70 con 0x7f7bc4108da0 2024-09-12T03:12:10.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.218+0000 7f7bb37fe640 1 -- 172.21.15.16:0/690275734 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40187e0 con 0x7f7bc4108da0 2024-09-12T03:12:10.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.218+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc407c670 con 0x7f7bc4108da0 2024-09-12T03:12:10.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.218+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc4080640 con 0x7f7bc4108da0 2024-09-12T03:12:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.224+0000 7f7bcb2e3740 1 Processor -- start 2024-09-12T03:12:10.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.225+0000 7f7bcb2e3740 1 -- start start 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.225+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.225+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad9c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad9c6a0 0x55df0ad9ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df0acf72e0 con 0x55df0ad914c0 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df0ac9ece0 con 0x55df0ad91de0 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55df0acf32a0 con 0x55df0ad9c6a0 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad9c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36836/0 (socket says 172.21.15.16:36836) 2024-09-12T03:12:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.226+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/351514144 learned_addr learned my addr 172.21.15.16:0/351514144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.229+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/351514144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad9c6a0 msgr2=0x55df0ad9ea90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:12:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.229+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad9c6a0 0x55df0ad9ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.229+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/351514144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 msgr2=0x55df0ad9c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.229+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad9c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.229+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/351514144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df0ac83440 con 0x55df0ad914c0 2024-09-12T03:12:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.231+0000 7f7bb37fe640 1 -- 172.21.15.16:0/690275734 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7bb4018980 con 0x7f7bc4108da0 2024-09-12T03:12:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.232+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800a970 tx=0x7f7bb800ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ffd0e379d72374 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.234+0000 7f7bb37fe640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 0x7f7bac0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.235+0000 7f7bb37fe640 1 -- 172.21.15.16:0/690275734 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bb40cf5b0 con 0x7f7bc4108da0 2024-09-12T03:12:10.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.236+0000 7f7bc259c640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 0x7f7bac0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.235+0000 7f7bb0ff9640 1 -- 172.21.15.16:0/351514144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800cd80 con 0x55df0ad914c0 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.236+0000 7f7bb0ff9640 1 -- 172.21.15.16:0/351514144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb80035f0 con 0x55df0ad914c0 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.236+0000 7f7bb0ff9640 1 -- 172.21.15.16:0/351514144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800f6b0 con 0x55df0ad914c0 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/351514144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 msgr2=0x55df0ad918a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800a970 tx=0x7f7bb800ae30 comp rx=0 tx=0).stop 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/351514144 shutdown_connections 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad9c6a0 0x55df0ad9ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad9c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.237+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/351514144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad914c0 0x55df0ad918a0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.238+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/351514144 >> 172.21.15.16:0/351514144 conn(0x55df0ad8da50 msgr2=0x55df0ad8de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.241+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/351514144 shutdown_connections 2024-09-12T03:12:10.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.241+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/351514144 wait complete. 2024-09-12T03:12:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.242+0000 7f7bcb2e3740 1 -- start start 2024-09-12T03:12:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.242+0000 7f7bcb2e3740 1 Processor -- start 2024-09-12T03:12:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.243+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 0x55df0ad9bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.243+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 0x55df0ad99290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad997d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df0ad9f270 con 0x55df0ad9c6a0 2024-09-12T03:12:10.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 0x55df0ad9bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df0acb2050 con 0x55df0ad914c0 2024-09-12T03:12:10.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 0x55df0ad9bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45168/0 (socket says 172.21.15.16:45168) 2024-09-12T03:12:10.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.244+0000 7f7bc259c640 1 -- 172.21.15.16:0/3220365138 learned_addr learned my addr 172.21.15.16:0/3220365138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:10.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.245+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55df0ad9f3f0 con 0x55df0ad91de0 2024-09-12T03:12:10.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.245+0000 7f7bcb2e3740 10 client.0 _ll_register_callbacks cb 0x55df0acf2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:10.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.246+0000 7f7bc259c640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 0x7f7bac0a5250 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x55df0ad8e8b0 tx=0x7f7bbc005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.246+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 0x55df0ad99290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.247+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 msgr2=0x55df0ad9bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.247+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 0x55df0ad9bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.247+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/3220365138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 msgr2=0x55df0ad997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.247+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.248+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/3220365138 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df0aa683c0 con 0x55df0ad91de0 2024-09-12T03:12:10.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.249+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 0x55df0ad99290 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55df0ad8edd0 tx=0x7f7bb409c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d474c73f2bde267 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.249+0000 7f7ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:12:10.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.249+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40cf2b0 con 0x55df0ad91de0 2024-09-12T03:12:10.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.250+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bb409b210 con 0x55df0ad91de0 2024-09-12T03:12:10.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.250+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bb80035f0 con 0x55df0ad91de0 2024-09-12T03:12:10.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.250+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7bb40a4a30 con 0x55df0ad91de0 2024-09-12T03:12:10.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.250+0000 7f7bcb2e3740 10 client.30052 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:10.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.250+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55df0adaac80 con 0x55df0ad91de0 2024-09-12T03:12:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.252+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb80035f0 con 0x55df0ad91de0 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7bb40a1300 con 0x55df0ad91de0 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 10 client.30052 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 10 client.30052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 20 client.30052 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0adac190 0x55df0adae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 4 client.30052 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.253+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df0ad04850 con 0x55df0adac190 2024-09-12T03:12:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.255+0000 7f7bc259c640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0adac190 0x55df0adae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.256+0000 7f7bc259c640 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0adac190 0x55df0adae5d0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d420f1abad340827 server_cookie=9e0f51e587588a83 in_seq=0 out_seq=0 2024-09-12T03:12:10.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.257+0000 7f7ba9ffb640 10 client.30052 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:10.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.258+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/3220365138 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55df0ad04850 con 0x55df0adac190 2024-09-12T03:12:10.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.258+0000 7f7ba9ffb640 10 client.30052 handle_command_reply: tid=0 2024-09-12T03:12:10.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.258+0000 7f7bcb2e3740 1 client.30052 shutdown 2024-09-12T03:12:10.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 msgr2=0x55df0ad99290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 0x55df0ad99290 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55df0ad8edd0 tx=0x7f7bb409c4c0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 shutdown_connections 2024-09-12T03:12:10.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0adac190 0x55df0adae5d0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad91de0 0x55df0ad99290 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7ba9ffb640 0 client.30052 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3220365138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad914c0 0x55df0ad9bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.259+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 >> 172.21.15.16:0/3220365138 conn(0x55df0ad8da50 msgr2=0x55df0ad8cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.260+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 shutdown_connections 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.260+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3220365138 wait complete. 2024-09-12T03:12:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.261+0000 7f7bcb2e3740 20 client.30052 trim_cache size 0 max 16384 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.290+0000 7f7bcb2e3740 1 Processor -- start 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.290+0000 7f7bcb2e3740 1 -- start start 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0adb3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0adb43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400e290 con 0x55df0ad9c6a0 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400fdd0 con 0x55df0ad91de0 2024-09-12T03:12:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400e960 con 0x55df0ad914c0 2024-09-12T03:12:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0adb43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52854/0 (socket says 172.21.15.16:52854) 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.291+0000 7f7bc259c640 1 -- 172.21.15.16:0/3520370091 learned_addr learned my addr 172.21.15.16:0/3520370091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0adb3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/3520370091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 msgr2=0x55df0ada8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/3520370091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 msgr2=0x55df0adb3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0adb3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 -- 172.21.15.16:0/3520370091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df0ad8e6e0 con 0x55df0ad9c6a0 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0adb43e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x55df0ad9a840 tx=0x7f7bb800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ae6faf2f95806a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7babfff640 1 -- 172.21.15.16:0/3520370091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8004040 con 0x55df0ad9c6a0 2024-09-12T03:12:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7babfff640 1 -- 172.21.15.16:0/3520370091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb80035f0 con 0x55df0ad9c6a0 2024-09-12T03:12:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.292+0000 7f7babfff640 1 -- 172.21.15.16:0/3520370091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800f670 con 0x55df0ad9c6a0 2024-09-12T03:12:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.293+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3520370091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 msgr2=0x55df0adb43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.293+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0adb43e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x55df0ad9a840 tx=0x7f7bb800cb80 comp rx=0 tx=0).stop 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.293+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3520370091 shutdown_connections 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.293+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0adb43e0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.293+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0adb3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/3520370091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3520370091 >> 172.21.15.16:0/3520370091 conn(0x55df0ad8da50 msgr2=0x55df0ad8cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3520370091 shutdown_connections 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/3520370091 wait complete. 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 -- start start 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 Processor -- start 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.294+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad96100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad96890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400f700 con 0x55df0ad9c6a0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400f030 con 0x55df0ad91de0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba400c320 con 0x55df0ad914c0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bcb2e3740 10 client.0 _ll_register_callbacks cb 0x7f7bb400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52860/0 (socket says 172.21.15.16:52860) 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc259c640 1 -- 172.21.15.16:0/1168563754 learned_addr learned my addr 172.21.15.16:0/1168563754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad96100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/1168563754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 msgr2=0x55df0ada4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/1168563754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 msgr2=0x55df0ad96890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:12:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad96890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad96890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.295+0000 7f7bc1d9b640 1 -- 172.21.15.16:0/1168563754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df0ae15e70 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7bc2d9d640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad96890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7bc1d9b640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad96100 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4098220 tx=0x7f7bb40bac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25f82a93f786b507 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb409b900 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f7bb409ae80 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bb80035f0 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f7bb40cf0f0 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7bcb2e3740 10 client.30113 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55df0ae16fd0 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.296+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb80035f0 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f7bb4096930 con 0x55df0ad91de0 2024-09-12T03:12:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 10 client.30113 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 10 client.30113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 20 client.30113 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0ae17750 0x55df0ae19b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:10.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 4 client.30113 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:12:10.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55df0acfed60 con 0x55df0ae17750 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.297+0000 7f7bc259c640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0ae17750 0x55df0ae19b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.298+0000 7f7bc259c640 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0ae17750 0x55df0ae19b70 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a14fec968a8c82 server_cookie=ab02474319229ef8 in_seq=0 out_seq=0 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.298+0000 7f7ba9ffb640 10 client.30113 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7ba9ffb640 1 -- 172.21.15.16:0/1168563754 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55df0acfed60 con 0x55df0ae17750 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7ba9ffb640 10 client.30113 handle_command_reply: tid=0 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 client.30113 shutdown 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 msgr2=0x55df0ad96100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad96100 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4098220 tx=0x7f7bb40bac30 comp rx=0 tx=0).stop 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 shutdown_connections 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55df0ae17750 0x55df0ae19b70 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7ba9ffb640 0 client.30113 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df0ad9c6a0 0x55df0ad96890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df0ad91de0 0x55df0ad96100 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 --2- 172.21.15.16:0/1168563754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df0ad914c0 0x55df0ada4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 >> 172.21.15.16:0/1168563754 conn(0x55df0ad8da50 msgr2=0x55df0ad9aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.299+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 shutdown_connections 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.300+0000 7f7bcb2e3740 1 -- 172.21.15.16:0/1168563754 wait complete. 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.300+0000 7f7bcb2e3740 20 client.30113 trim_cache size 0 max 16384 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:12:10.303 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:12:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 msgr2=0x7f7bac0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 0x7f7bac0a5250 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x55df0ad8e8b0 tx=0x7f7bbc005bd0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 msgr2=0x7f7bc407b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 0x7f7bc407b880 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb400ec20 comp rx=0 tx=0).stop 2024-09-12T03:12:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 shutdown_connections 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bc410d520 0x7f7bc407bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7bc4108da0 0x7f7bc407b880 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f7bac0a2d90 0x7f7bac0a5250 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 --2- 172.21.15.16:0/690275734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc41083f0 0x7f7bc40827f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 >> 172.21.15.16:0/690275734 conn(0x7f7bc4077af0 msgr2=0x7f7bc4078e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 shutdown_connections 2024-09-12T03:12:10.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:10.301+0000 7f7bc8f0a640 1 -- 172.21.15.16:0/690275734 wait complete. 2024-09-12T03:12:11.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 279833 23.38 MB/sec execute 552 sec latency 391.676 ms 2024-09-12T03:12:11.215 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:12:11.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:12:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:11.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:11 smithi016 ceph-mon[30925]: pgmap v1065: 97 pgs: 97 active+clean; 928 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 648 op/s 2024-09-12T03:12:11.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:11 smithi184 ceph-mon[35321]: pgmap v1065: 97 pgs: 97 active+clean; 928 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 648 op/s 2024-09-12T03:12:11.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:11 smithi033 ceph-mon[35092]: pgmap v1065: 97 pgs: 97 active+clean; 928 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 648 op/s 2024-09-12T03:12:12.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 280271 23.39 MB/sec execute 553 sec latency 85.355 ms 2024-09-12T03:12:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 -- 172.21.15.16:0/3643379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c102ef0 msgr2=0x7fdf7c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 --2- 172.21.15.16:0/3643379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c10da40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009ca0 tx=0x7fdf7402a950 comp rx=0 tx=0).stop 2024-09-12T03:12:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 -- 172.21.15.16:0/3643379229 shutdown_connections 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 --2- 172.21.15.16:0/3643379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c10da40 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 --2- 172.21.15.16:0/3643379229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102570 0x7fdf7c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 --2- 172.21.15.16:0/3643379229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c100e20 0x7fdf7c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 -- 172.21.15.16:0/3643379229 >> 172.21.15.16:0/3643379229 conn(0x7fdf7c0fc7d0 msgr2=0x7fdf7c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 -- 172.21.15.16:0/3643379229 shutdown_connections 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.054+0000 7fdf81497640 1 -- 172.21.15.16:0/3643379229 wait complete. 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.055+0000 7fdf81497640 1 Processor -- start 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 -- start start 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 0x7fdf7c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 0x7fdf7c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c110f80 con 0x7fdf7c100e20 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c110e00 con 0x7fdf7c102570 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf81497640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c111100 con 0x7fdf7c102ef0 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 0x7fdf7c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 0x7fdf7c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36876/0 (socket says 172.21.15.16:36876) 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf7affd640 1 -- 172.21.15.16:0/2761668708 learned_addr learned my addr 172.21.15.16:0/2761668708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf7a7fc640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 0x7fdf7c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.056+0000 7fdf7b7fe640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 -- 172.21.15.16:0/2761668708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102ef0 msgr2=0x7fdf7c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 -- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 msgr2=0x7fdf7c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 0x7fdf7c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 -- 172.21.15.16:0/2761668708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c19d440 con 0x7fdf7c102570 2024-09-12T03:12:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf7a7fc640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 0x7fdf7c1989d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009d20 tx=0x7fdf7000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7001f070 con 0x7fdf7c102570 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf70012510 con 0x7fdf7c102570 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.057+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7001a670 con 0x7fdf7c102570 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.058+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c19d730 con 0x7fdf7c102570 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.058+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c107a10 con 0x7fdf7c102570 2024-09-12T03:12:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.059+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c1021b0 con 0x7fdf7c102570 2024-09-12T03:12:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.061+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdf70022070 con 0x7fdf7c102570 2024-09-12T03:12:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.062+0000 7fdf63fff640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 0x7fdf640a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.062+0000 7fdf7affd640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 0x7fdf640a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.062+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdf700d2a40 con 0x7fdf7c102570 2024-09-12T03:12:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.063+0000 7fdf7affd640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 0x7fdf640a5140 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009c90 tx=0x7fdf6c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:12.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.068+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdf700d4e20 con 0x7fdf7c102570 2024-09-12T03:12:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.325+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf7c199820 con 0x7fdf7c102570 2024-09-12T03:12:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.326+0000 7fdf63fff640 1 -- 172.21.15.16:0/2761668708 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fdf7c199820 con 0x7fdf7c102570 2024-09-12T03:12:12.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:12:12.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:12:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:12:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 msgr2=0x7fdf640a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 0x7fdf640a5140 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009c90 tx=0x7fdf6c005f90 comp rx=0 tx=0).stop 2024-09-12T03:12:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 msgr2=0x7fdf7c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 0x7fdf7c1989d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009d20 tx=0x7fdf7000c6a0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 shutdown_connections 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf7c102ef0 0x7fdf7c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdf7c102570 0x7fdf7c1989d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdf640a2c80 0x7fdf640a5140 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 --2- 172.21.15.16:0/2761668708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c100e20 0x7fdf7c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.328+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 >> 172.21.15.16:0/2761668708 conn(0x7fdf7c0fc7d0 msgr2=0x7fdf7c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.329+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 shutdown_connections 2024-09-12T03:12:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:12.329+0000 7fdf81497640 1 -- 172.21.15.16:0/2761668708 wait complete. 2024-09-12T03:12:12.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:12 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2761668708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:12.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:12 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2761668708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:12.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:12 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2761668708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:13.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 280929 23.40 MB/sec execute 554 sec latency 65.067 ms 2024-09-12T03:12:13.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:12:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:13.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:13 smithi016 ceph-mon[30925]: pgmap v1066: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 616 op/s 2024-09-12T03:12:13.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:13 smithi184 ceph-mon[35321]: pgmap v1066: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 616 op/s 2024-09-12T03:12:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 -- 172.21.15.16:0/2599774015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 msgr2=0x7fb5381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/2599774015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb5381067e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fb528009900 tx=0x7fb52802a410 comp rx=0 tx=0).stop 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 -- 172.21.15.16:0/2599774015 shutdown_connections 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/2599774015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 0x7fb53810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/2599774015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb5381067e0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/2599774015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5381021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 -- 172.21.15.16:0/2599774015 >> 172.21.15.16:0/2599774015 conn(0x7fb5380fd8c0 msgr2=0x7fb5380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.900+0000 7fb53f9d0640 1 -- 172.21.15.16:0/2599774015 shutdown_connections 2024-09-12T03:12:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.901+0000 7fb53f9d0640 1 -- 172.21.15.16:0/2599774015 wait complete. 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.901+0000 7fb53f9d0640 1 Processor -- start 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.901+0000 7fb53f9d0640 1 -- start start 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5380744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb538074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 0x7fb538074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb538111f70 con 0x7fb538102790 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb538111df0 con 0x7fb538101de0 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53f9d0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381120f0 con 0x7fb538106f10 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5380744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53df6d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 0x7fb538074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5380744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45206/0 (socket says 172.21.15.16:45206) 2024-09-12T03:12:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53d76c640 1 -- 172.21.15.16:0/4185688066 learned_addr learned my addr 172.21.15.16:0/4185688066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.902+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb538074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 -- 172.21.15.16:0/4185688066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 msgr2=0x7fb538074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 0x7fb538074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 -- 172.21.15.16:0/4185688066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 msgr2=0x7fb5380744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5380744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 -- 172.21.15.16:0/4185688066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5381a6910 con 0x7fb538102790 2024-09-12T03:12:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb538074a30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb528005940 tx=0x7fb528004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.903+0000 7fb52e7fc640 1 -- 172.21.15.16:0/4185688066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52803f030 con 0x7fb538102790 2024-09-12T03:12:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.904+0000 7fb52e7fc640 1 -- 172.21.15.16:0/4185688066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb528003ad0 con 0x7fb538102790 2024-09-12T03:12:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.904+0000 7fb52e7fc640 1 -- 172.21.15.16:0/4185688066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528035070 con 0x7fb538102790 2024-09-12T03:12:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.904+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5381981b0 con 0x7fb538102790 2024-09-12T03:12:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.904+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb538198640 con 0x7fb538102790 2024-09-12T03:12:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.907+0000 7fb52e7fc640 1 -- 172.21.15.16:0/4185688066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb528042030 con 0x7fb538102790 2024-09-12T03:12:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 Processor -- start 2024-09-12T03:12:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 -- start start 2024-09-12T03:12:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414434090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414441760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56541439a2f0 con 0x565414435b40 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565414341cf0 con 0x5654144345d0 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb541da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5654143962b0 con 0x56541443f370 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb52e7fc640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 0x7fb5240a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45210/0 (socket says 172.21.15.16:45210) 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb53d76c640 1 -- 172.21.15.16:0/646874974 learned_addr learned my addr 172.21.15.16:0/646874974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414441760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.908+0000 7fb52e7fc640 1 -- 172.21.15.16:0/4185688066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb5280ef140 con 0x7fb538102790 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53df6d640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 0x7fb5240a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 -- 172.21.15.16:0/646874974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 msgr2=0x565414441760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414441760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 -- 172.21.15.16:0/646874974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 msgr2=0x565414434090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414434090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414441760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 -- 172.21.15.16:0/646874974 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654143263a0 con 0x5654144345d0 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb53d76c640 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f5c819fc74948d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb5237fe640 1 -- 172.21.15.16:0/646874974 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534022020 con 0x5654144345d0 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb5237fe640 1 -- 172.21.15.16:0/646874974 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb53401e400 con 0x5654144345d0 2024-09-12T03:12:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.909+0000 7fb5237fe640 1 -- 172.21.15.16:0/646874974 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53401d5c0 con 0x5654144345d0 2024-09-12T03:12:13.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- 172.21.15.16:0/646874974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 msgr2=0x565414434a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb53df6d640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 0x7fb5240a5190 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb5300098b0 tx=0x7fb530009840 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534005f90 comp rx=0 tx=0).stop 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- 172.21.15.16:0/646874974 shutdown_connections 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414441760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x565414434a10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 --2- 172.21.15.16:0/646874974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414434090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- 172.21.15.16:0/646874974 >> 172.21.15.16:0/646874974 conn(0x5654143fd310 msgr2=0x565414430c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- 172.21.15.16:0/646874974 shutdown_connections 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- 172.21.15.16:0/646874974 wait complete. 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.910+0000 7fb541da9740 1 -- start start 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 Processor -- start 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x56541443ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 0x56541443c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x565414442070 con 0x56541443f370 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565414355060 con 0x565414435b40 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5654144421f0 con 0x5654144345d0 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb53cf6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb53df6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 0x56541443c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb53cf6b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45220/0 (socket says 172.21.15.16:45220) 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3946033841 learned_addr learned my addr 172.21.15.16:0/3946033841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb53d76c640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x56541443ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.911+0000 7fb541da9740 10 client.0 _ll_register_callbacks cb 0x5654140a88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3946033841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 msgr2=0x56541443ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x56541443ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 msgr2=0x56541443c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 0x56541443c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3946033841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56541410b3c0 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53df6d640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 0x56541443c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb5280ecc50 tx=0x7fb5280bb3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e32e17cf41858a00 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528049050 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb53401e400 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb52803bd80 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb5280c4620 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb541da9740 10 client.30131 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.912+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56541444da10 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb53401e400 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb5280bcce0 con 0x565414435b40 2024-09-12T03:12:13.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 10 client.30131 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 10 client.30131 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 20 client.30131 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56541444ee70 0x5654144512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 4 client.30131 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654143a7860 con 0x56541444ee70 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb53d76c640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56541444ee70 0x5654144512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb520ff9640 10 client.30131 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.913+0000 7fb53d76c640 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56541444ee70 0x5654144512b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ce420bc96b28cec server_cookie=9f3bc4509cbea14b in_seq=0 out_seq=0 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb520ff9640 1 -- 172.21.15.16:0/3946033841 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5654143a7860 con 0x56541444ee70 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb520ff9640 10 client.30131 handle_command_reply: tid=0 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 client.30131 shutdown 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 msgr2=0x56541443bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb5280ecc50 tx=0x7fb5280bb3c0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 shutdown_connections 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541443f370 0x56541443c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x56541444ee70 0x5654144512b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565414435b40 0x56541443bfb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb520ff9640 0 client.30131 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 --2- 172.21.15.16:0/3946033841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x56541443ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 >> 172.21.15.16:0/3946033841 conn(0x5654143fd310 msgr2=0x565414436930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 shutdown_connections 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.914+0000 7fb541da9740 1 -- 172.21.15.16:0/3946033841 wait complete. 2024-09-12T03:12:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.915+0000 7fb541da9740 20 client.30131 trim_cache size 0 max 16384 2024-09-12T03:12:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 Processor -- start 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 -- start start 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x56541444b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414430700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381ab1b0 con 0x565414435b40 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381accf0 con 0x5654144345d0 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb541da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381ab880 con 0x56541443f370 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x56541444b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53cf6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53cf6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36926/0 (socket says 172.21.15.16:36926) 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3338249492 learned_addr learned my addr 172.21.15.16:0/3338249492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53df6d640 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414430700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.931+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x56541444b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45232/0 (socket says 172.21.15.16:45232) 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3338249492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 msgr2=0x565414430700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414430700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3338249492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 msgr2=0x56541444b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x56541444b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 -- 172.21.15.16:0/3338249492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565414441f50 con 0x565414435b40 2024-09-12T03:12:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb5280bb9f0 tx=0x7fb5280bc020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b06d2b00dc9b6b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb522ffd640 1 -- 172.21.15.16:0/3338249492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528049070 con 0x565414435b40 2024-09-12T03:12:13.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb522ffd640 1 -- 172.21.15.16:0/3338249492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5280d4070 con 0x565414435b40 2024-09-12T03:12:13.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.932+0000 7fb522ffd640 1 -- 172.21.15.16:0/3338249492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5280b80d0 con 0x565414435b40 2024-09-12T03:12:13.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- 172.21.15.16:0/3338249492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 msgr2=0x565414435300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb5280bb9f0 tx=0x7fb5280bc020 comp rx=0 tx=0).stop 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- 172.21.15.16:0/3338249492 shutdown_connections 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56541443f370 0x565414430700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565414435b40 0x565414435300 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 --2- 172.21.15.16:0/3338249492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5654144345d0 0x56541444b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- 172.21.15.16:0/3338249492 >> 172.21.15.16:0/3338249492 conn(0x5654143fd310 msgr2=0x56541442ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- 172.21.15.16:0/3338249492 shutdown_connections 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- 172.21.15.16:0/3338249492 wait complete. 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.933+0000 7fb541da9740 1 -- start start 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 Processor -- start 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 0x565414443750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 0x565414443c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381ac620 con 0x56541444ee70 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381abf50 con 0x56541443f370 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5381a9190 con 0x5654144345d0 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb541da9740 10 client.0 _ll_register_callbacks cb 0x7fb528009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb53df6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 0x565414443c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb53d76c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:52914/0 (socket says 172.21.15.16:52914) 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb53d76c640 1 -- 172.21.15.16:0/840936315 learned_addr learned my addr 172.21.15.16:0/840936315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.934+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 0x565414443750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 -- 172.21.15.16:0/840936315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 msgr2=0x565414443210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 -- 172.21.15.16:0/840936315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 msgr2=0x565414443750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:13.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 0x565414443750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 -- 172.21.15.16:0/840936315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565414437120 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53d76c640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53cf6b640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 0x565414443750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb53df6d640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 0x565414443c90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb530004a80 tx=0x7fb530009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e462cab692675d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530004d20 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb53001db20 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5280d4070 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb53001dd20 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb541da9740 10 client.20982 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.935+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5654144b9dd0 con 0x56541444ee70 2024-09-12T03:12:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5280d4070 con 0x56541444ee70 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fb530026070 con 0x56541444ee70 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 10 client.20982 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 10 client.20982 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 20 client.20982 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5654144ba570 0x5654144bc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 4 client.20982 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654143a1d70 con 0x5654144ba570 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.936+0000 7fb53d76c640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5654144ba570 0x5654144bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb520ff9640 10 client.20982 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb53d76c640 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5654144ba570 0x5654144bc990 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fbc162abbb685e18 server_cookie=e24b0352cc65e79a in_seq=0 out_seq=0 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb520ff9640 1 -- 172.21.15.16:0/840936315 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654143a1d70 con 0x5654144ba570 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb520ff9640 10 client.20982 handle_command_reply: tid=0 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 client.20982 shutdown 2024-09-12T03:12:13.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 msgr2=0x565414443c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 0x565414443c90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb530004a80 tx=0x7fb530009600 comp rx=0 tx=0).stop 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 shutdown_connections 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56541444ee70 0x565414443c90 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5654144ba570 0x5654144bc990 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56541443f370 0x565414443750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 --2- 172.21.15.16:0/840936315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5654144345d0 0x565414443210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 >> 172.21.15.16:0/840936315 conn(0x5654143fd310 msgr2=0x565414430700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.937+0000 7fb520ff9640 0 client.20982 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 shutdown_connections 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb541da9740 1 -- 172.21.15.16:0/840936315 wait complete. 2024-09-12T03:12:13.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb541da9740 20 client.20982 trim_cache size 0 max 16384 2024-09-12T03:12:13.944 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:12:13.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 msgr2=0x7fb5240a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 0x7fb5240a5190 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb5300098b0 tx=0x7fb530009840 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 msgr2=0x7fb538074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb538074a30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb528005940 tx=0x7fb528004270 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 shutdown_connections 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb538106f10 0x7fb538074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb5240a2cd0 0x7fb5240a5190 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538102790 0x7fb538074a30 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 --2- 172.21.15.16:0/4185688066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb538101de0 0x7fb5380744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.938+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 >> 172.21.15.16:0/4185688066 conn(0x7fb5380fd8c0 msgr2=0x7fb5380fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.939+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 shutdown_connections 2024-09-12T03:12:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:13.939+0000 7fb53f9d0640 1 -- 172.21.15.16:0/4185688066 wait complete. 2024-09-12T03:12:13.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:13 smithi033 ceph-mon[35092]: pgmap v1066: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 616 op/s 2024-09-12T03:12:14.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 281620 23.44 MB/sec execute 555 sec latency 80.081 ms 2024-09-12T03:12:14.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:12:15.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 282337 23.46 MB/sec execute 556 sec latency 55.938 ms 2024-09-12T03:12:15.262 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:15.731 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:15 smithi016 ceph-mon[30925]: pgmap v1067: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 569 op/s 2024-09-12T03:12:15.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:15 smithi184 ceph-mon[35321]: pgmap v1067: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 569 op/s 2024-09-12T03:12:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 -- 172.21.15.16:0/672428396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 msgr2=0x7fe50c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 --2- 172.21.15.16:0/672428396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c1067f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009900 tx=0x7fe4fc02a3f0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 -- 172.21.15.16:0/672428396 shutdown_connections 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 --2- 172.21.15.16:0/672428396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c106f20 0x7fe50c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 --2- 172.21.15.16:0/672428396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c1067f0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 --2- 172.21.15.16:0/672428396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c101df0 0x7fe50c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 -- 172.21.15.16:0/672428396 >> 172.21.15.16:0/672428396 conn(0x7fe50c0fd8f0 msgr2=0x7fe50c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 -- 172.21.15.16:0/672428396 shutdown_connections 2024-09-12T03:12:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.841+0000 7fe514377640 1 -- 172.21.15.16:0/672428396 wait complete. 2024-09-12T03:12:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 Processor -- start 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 -- start start 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 0x7fe50c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c106f20 0x7fe50c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c112070 con 0x7fe50c1027a0 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c111ef0 con 0x7fe50c101df0 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.842+0000 7fe514377640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1121f0 con 0x7fe50c106f20 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe511912640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 0x7fe50c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe511912640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49072/0 (socket says 172.21.15.16:49072) 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe511912640 1 -- 172.21.15.16:0/2642650340 learned_addr learned my addr 172.21.15.16:0/2642650340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512914640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c106f20 0x7fe50c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 -- 172.21.15.16:0/2642650340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c106f20 msgr2=0x7fe50c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c106f20 0x7fe50c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 -- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 msgr2=0x7fe50c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe512113640 1 -- 172.21.15.16:0/2642650340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1a6af0 con 0x7fe50c101df0 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.843+0000 7fe511912640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe512113640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 0x7fe50c0744f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b3f0 tx=0x7fe50800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe5037fe640 1 -- 172.21.15.16:0/2642650340 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508015020 con 0x7fe50c101df0 2024-09-12T03:12:15.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe5037fe640 1 -- 172.21.15.16:0/2642650340 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe508011d10 con 0x7fe50c101df0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c1a6de0 con 0x7fe50c101df0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe5037fe640 1 -- 172.21.15.16:0/2642650340 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50800f2e0 con 0x7fe50c101df0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.844+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c1a72e0 con 0x7fe50c101df0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 Processor -- start 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 -- start start 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe5037fe640 1 -- 172.21.15.16:0/2642650340 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe508018020 con 0x7fe50c101df0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4927fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b49294f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559b491e52f0 con 0x559b49280b40 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559b4918ccf0 con 0x559b49292b80 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.847+0000 7fe516750740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559b491e12b0 con 0x559b4927f5d0 2024-09-12T03:12:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49078/0 (socket says 172.21.15.16:49078) 2024-09-12T03:12:15.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 -- 172.21.15.16:0/1523533571 learned_addr learned my addr 172.21.15.16:0/1523533571 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512113640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4927fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe511912640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b49294f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 -- 172.21.15.16:0/1523533571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 msgr2=0x559b4927fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4927fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 -- 172.21.15.16:0/1523533571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 msgr2=0x559b49294f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b49294f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe512914640 1 -- 172.21.15.16:0/1523533571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b491713a0 con 0x559b49280b40 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.848+0000 7fe5037fe640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 0x7fe4f80a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe512113640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 0x7fe4f80a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe5037fe640 1 -- 172.21.15.16:0/2642650340 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe5080ce6f0 con 0x7fe50c101df0 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe511912640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b49294f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe512914640 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe50400a430 tx=0x7fe504009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56d0187994d7e8b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe500dea640 1 -- 172.21.15.16:0/1523533571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504013790 con 0x559b49280b40 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe512113640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 0x7fe4f80a5100 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe5080cc570 tx=0x7fe508000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe500dea640 1 -- 172.21.15.16:0/1523533571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe504019070 con 0x559b49280b40 2024-09-12T03:12:15.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.849+0000 7fe500dea640 1 -- 172.21.15.16:0/1523533571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5040122e0 con 0x559b49280b40 2024-09-12T03:12:15.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.850+0000 7fe516750740 1 -- 172.21.15.16:0/1523533571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 msgr2=0x559b4927f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.850+0000 7fe516750740 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe50400a430 tx=0x7fe504009650 comp rx=0 tx=0).stop 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.851+0000 7fe516750740 1 -- 172.21.15.16:0/1523533571 shutdown_connections 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.851+0000 7fe516750740 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b49294f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.851+0000 7fe516750740 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4927fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.851+0000 7fe516750740 1 --2- 172.21.15.16:0/1523533571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49280b40 0x559b4927f090 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.851+0000 7fe516750740 1 -- 172.21.15.16:0/1523533571 >> 172.21.15.16:0/1523533571 conn(0x559b49248310 msgr2=0x559b4927bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.852+0000 7fe516750740 1 -- 172.21.15.16:0/1523533571 shutdown_connections 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.852+0000 7fe516750740 1 -- 172.21.15.16:0/1523533571 wait complete. 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.852+0000 7fe516750740 1 -- start start 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.852+0000 7fe516750740 1 Processor -- start 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b4927f5d0 0x559b49292540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4928f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b4928fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559b49295880 con 0x559b49292b80 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559b491a0060 con 0x559b4927f5d0 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559b49295a00 con 0x559b49280b40 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe516750740 10 client.0 _ll_register_callbacks cb 0x559b48ef38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe511912640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4928f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b4928fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe511912640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4928f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36556/0 (socket says 172.21.15.16:36556) 2024-09-12T03:12:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe511912640 1 -- 172.21.15.16:0/4225249981 learned_addr learned my addr 172.21.15.16:0/4225249981 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b4927f5d0 0x559b49292540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 -- 172.21.15.16:0/4225249981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 msgr2=0x559b4928f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4928f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 -- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b4927f5d0 msgr2=0x559b49292540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b4927f5d0 0x559b49292540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.853+0000 7fe512914640 1 -- 172.21.15.16:0/4225249981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b48f563c0 con 0x559b49292b80 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe512914640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b4928fde0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fe50400a400 tx=0x7fe504014a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff7b4293ece2299 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe4f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504013790 con 0x559b49292b80 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe504013930 con 0x559b49292b80 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe504019070 con 0x559b49292b80 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe504013b30 con 0x559b49292b80 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe516750740 10 client.20994 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559b492a12b0 con 0x559b49292b80 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.854+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe504019070 con 0x559b49292b80 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe50401e480 con 0x559b49292b80 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 10 client.20994 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 10 client.20994 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 20 client.20994 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b492a2770 0x559b492a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 4 client.20994 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.855+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b491f2860 con 0x559b492a2770 2024-09-12T03:12:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.856+0000 7fe511912640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b492a2770 0x559b492a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.856+0000 7fe511912640 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b492a2770 0x559b492a4b60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cda688dcaa2fc8ac server_cookie=34f47d856ae852b5 in_seq=0 out_seq=0 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.856+0000 7fe4f5ffb640 10 client.20994 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.859+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/4225249981 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b491f2860 con 0x559b492a2770 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.859+0000 7fe4f5ffb640 10 client.20994 handle_command_reply: tid=0 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.859+0000 7fe516750740 1 client.20994 shutdown 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 msgr2=0x559b4928fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b4928fde0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fe50400a400 tx=0x7fe504014a50 comp rx=0 tx=0).stop 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 shutdown_connections 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b492a2770 0x559b492a4b60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b4928fde0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4928f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 --2- 172.21.15.16:0/4225249981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b4927f5d0 0x559b49292540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 >> 172.21.15.16:0/4225249981 conn(0x559b49248310 msgr2=0x559b49281930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe4f5ffb640 0 client.20994 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:12:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 shutdown_connections 2024-09-12T03:12:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 1 -- 172.21.15.16:0/4225249981 wait complete. 2024-09-12T03:12:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.860+0000 7fe516750740 20 client.20994 trim_cache size 0 max 16384 2024-09-12T03:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.881+0000 7fe516750740 1 Processor -- start 2024-09-12T03:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.881+0000 7fe516750740 1 -- start start 2024-09-12T03:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 0x559b4929eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b4928e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1ab5c0 con 0x559b4927f5d0 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1ad100 con 0x559b49292b80 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe516750740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1abc90 con 0x559b49280b40 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36562/0 (socket says 172.21.15.16:36562) 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 -- 172.21.15.16:0/2583896241 learned_addr learned my addr 172.21.15.16:0/2583896241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe512914640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b4928e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 -- 172.21.15.16:0/2583896241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 msgr2=0x559b4928e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b4928e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe512914640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 0x559b4929eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 -- 172.21.15.16:0/2583896241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 msgr2=0x559b4929eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.882+0000 7fe511912640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 0x559b4929eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe511912640 1 -- 172.21.15.16:0/2583896241 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b4928ffd0 con 0x559b49280b40 2024-09-12T03:12:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe512914640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 0x559b4929eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe511912640 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc02a9e0 tx=0x7fe4fc02a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a45f1d2f2f703261 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe4f7fff640 1 -- 172.21.15.16:0/2583896241 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc049030 con 0x559b49280b40 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe4f7fff640 1 -- 172.21.15.16:0/2583896241 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4fc005020 con 0x559b49280b40 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.883+0000 7fe4f7fff640 1 -- 172.21.15.16:0/2583896241 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc035040 con 0x559b49280b40 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 -- 172.21.15.16:0/2583896241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 msgr2=0x559b4927b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc02a9e0 tx=0x7fe4fc02a8f0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 -- 172.21.15.16:0/2583896241 shutdown_connections 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49292b80 0x559b4928e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b49280b40 0x559b4927b610 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 --2- 172.21.15.16:0/2583896241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b4927f5d0 0x559b4929eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 -- 172.21.15.16:0/2583896241 >> 172.21.15.16:0/2583896241 conn(0x559b49248310 msgr2=0x559b4928e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 -- 172.21.15.16:0/2583896241 shutdown_connections 2024-09-12T03:12:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.885+0000 7fe516750740 1 -- 172.21.15.16:0/2583896241 wait complete. 2024-09-12T03:12:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.886+0000 7fe516750740 1 -- start start 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.886+0000 7fe516750740 1 Processor -- start 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49280b40 0x559b492964f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe512914640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe512914640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36566/0 (socket says 172.21.15.16:36566) 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe512914640 1 -- 172.21.15.16:0/724289813 learned_addr learned my addr 172.21.15.16:0/724289813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b49296a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1aca30 con 0x559b49292b80 2024-09-12T03:12:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1ac360 con 0x559b49280b40 2024-09-12T03:12:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.887+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1a95a0 con 0x559b4927f5d0 2024-09-12T03:12:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe516750740 10 client.0 _ll_register_callbacks cb 0x7fe504009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b49296a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 -- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49280b40 msgr2=0x559b492964f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:12:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49280b40 0x559b492964f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 -- 172.21.15.16:0/724289813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 msgr2=0x559b49296a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b49296a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe512914640 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b49296ff0 con 0x559b4927f5d0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.888+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4fc005020 con 0x559b4927f5d0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe512914640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe504014a20 tx=0x7fe50400c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8026951ccbbfae server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe4f5ffb640 10 client.30097 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/724289813 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504013590 con 0x559b4927f5d0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/724289813 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe504013730 con 0x559b4927f5d0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/724289813 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe50401ea90 con 0x559b4927f5d0 2024-09-12T03:12:15.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe516750740 10 client.30097 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:15.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.889+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=39}) -- 0x559b4930cd70 con 0x559b4927f5d0 2024-09-12T03:12:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/724289813 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe50401ec30 con 0x559b4927f5d0 2024-09-12T03:12:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 10 client.30097 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 10 client.30097 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 20 client.30097 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b4930d370 0x559b4930f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 4 client.30097 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b491ecd70 con 0x559b4930d370 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.890+0000 7fe511912640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b4930d370 0x559b4930f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.891+0000 7fe511912640 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b4930d370 0x559b4930f760 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f081a8725eb1116f server_cookie=babda03003cbaa9d in_seq=0 out_seq=0 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.891+0000 7fe4f5ffb640 10 client.30097 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.891+0000 7fe4f5ffb640 1 -- 172.21.15.16:0/724289813 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b491ecd70 con 0x559b4930d370 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.891+0000 7fe4f5ffb640 10 client.30097 handle_command_reply: tid=0 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.891+0000 7fe516750740 1 client.30097 shutdown 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 msgr2=0x559b4929b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe504014a20 tx=0x7fe50400c040 comp rx=0 tx=0).stop 2024-09-12T03:12:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 shutdown_connections 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x559b4930d370 0x559b4930f760 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe4f5ffb640 0 client.30097 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b49292b80 0x559b49296a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559b49280b40 0x559b492964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 --2- 172.21.15.16:0/724289813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559b4927f5d0 0x559b4929b1c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.892+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 >> 172.21.15.16:0/724289813 conn(0x559b49248310 msgr2=0x559b4928e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.893+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 shutdown_connections 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.893+0000 7fe516750740 1 -- 172.21.15.16:0/724289813 wait complete. 2024-09-12T03:12:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.893+0000 7fe516750740 20 client.30097 trim_cache size 0 max 16384 2024-09-12T03:12:15.895 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.893+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 msgr2=0x7fe4f80a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.893+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 0x7fe4f80a5100 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe5080cc570 tx=0x7fe508000bc0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 msgr2=0x7fe50c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 0x7fe50c0744f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b3f0 tx=0x7fe50800b8b0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 shutdown_connections 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe50c106f20 0x7fe50c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe4f80a2c40 0x7fe4f80a5100 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe50c1027a0 0x7fe50c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 --2- 172.21.15.16:0/2642650340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe50c101df0 0x7fe50c0744f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 >> 172.21.15.16:0/2642650340 conn(0x7fe50c0fd8f0 msgr2=0x7fe50c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 shutdown_connections 2024-09-12T03:12:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:15.894+0000 7fe514377640 1 -- 172.21.15.16:0/2642650340 wait complete. 2024-09-12T03:12:15.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:15 smithi033 ceph-mon[35092]: pgmap v1067: 97 pgs: 97 active+clean; 932 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.0 MiB/s wr, 569 op/s 2024-09-12T03:12:16.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 282690 23.45 MB/sec execute 557 sec latency 215.702 ms 2024-09-12T03:12:16.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:12:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:17.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 283348 23.48 MB/sec execute 558 sec latency 76.507 ms 2024-09-12T03:12:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/707079390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 msgr2=0x7f6a94107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/707079390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a94107b60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009860 tx=0x7f6a8802a340 comp rx=0 tx=0).stop 2024-09-12T03:12:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/707079390 shutdown_connections 2024-09-12T03:12:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/707079390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a941082a0 0x7f6a9410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/707079390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a94107b60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/707079390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a94102d70 0x7f6a94103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/707079390 >> 172.21.15.16:0/707079390 conn(0x7f6a940fc7d0 msgr2=0x7f6a940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/707079390 shutdown_connections 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.387+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/707079390 wait complete. 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 Processor -- start 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 -- start start 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a941a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a941082a0 0x7f6a9419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94112040 con 0x7f6a94102d70 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94111ec0 con 0x7f6a941082a0 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a9a5ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941121c0 con 0x7f6a94103720 2024-09-12T03:12:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49122/0 (socket says 172.21.15.16:49122) 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a93fff640 1 -- 172.21.15.16:0/1594215276 learned_addr learned my addr 172.21.15.16:0/1594215276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a941a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.388+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a941082a0 0x7f6a9419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 -- 172.21.15.16:0/1594215276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a941082a0 msgr2=0x7f6a9419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a941082a0 0x7f6a9419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 -- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 msgr2=0x7f6a941a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 -- 172.21.15.16:0/1594215276 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9419aee0 con 0x7f6a94103720 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a93fff640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a941a0a40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009780 tx=0x7f6a88033ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a917fa640 1 -- 172.21.15.16:0/1594215276 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8803f020 con 0x7f6a94103720 2024-09-12T03:12:17.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9419b170 con 0x7f6a94103720 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a917fa640 1 -- 172.21.15.16:0/1594215276 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a88004040 con 0x7f6a94103720 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a917fa640 1 -- 172.21.15.16:0/1594215276 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88037040 con 0x7f6a94103720 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.389+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a941a72b0 con 0x7f6a94103720 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a917fa640 1 -- 172.21.15.16:0/1594215276 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6a88042020 con 0x7f6a94103720 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 Processor -- start 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 -- start start 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 0x562a18cf7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18d04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a18c5d2f0 con 0x562a18cf8b40 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a18c04cf0 con 0x562a18d02370 2024-09-12T03:12:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562a18c592b0 con 0x562a18cf75d0 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a98b8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a98b8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49130/0 (socket says 172.21.15.16:49130) 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a98b8b640 1 -- 172.21.15.16:0/444177304 learned_addr learned my addr 172.21.15.16:0/444177304 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a93fff640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 0x562a18cf7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.392+0000 7f6a937fe640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18d04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a917fa640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 0x7f6a7c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 -- 172.21.15.16:0/444177304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 msgr2=0x562a18cf7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 0x562a18cf7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 -- 172.21.15.16:0/444177304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 msgr2=0x562a18d04760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18d04760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 -- 172.21.15.16:0/444177304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a18be93a0 con 0x562a18cf8b40 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a93fff640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 0x562a18cf7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:12:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a937fe640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18d04760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 0x7f6a7c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a917fa640 1 -- 172.21.15.16:0/1594215276 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6a880ef080 con 0x7f6a94103720 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e550 tx=0x7f6a8c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf342640f4a9231b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a7a7fc640 1 -- 172.21.15.16:0/444177304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c01bb50 con 0x562a18cf8b40 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 0x7f6a7c0a5040 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c023000 tx=0x7f6a8c024000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a7a7fc640 1 -- 172.21.15.16:0/444177304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c01bcf0 con 0x562a18cf8b40 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.393+0000 7f6a7a7fc640 1 -- 172.21.15.16:0/444177304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c01a3e0 con 0x562a18cf8b40 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/444177304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 msgr2=0x562a18cf7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e550 tx=0x7f6a8c00ea10 comp rx=0 tx=0).stop 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/444177304 shutdown_connections 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18d04760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf75d0 0x562a18cf7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/444177304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf8b40 0x562a18cf7090 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/444177304 >> 172.21.15.16:0/444177304 conn(0x562a18cc0310 msgr2=0x562a18cf3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/444177304 shutdown_connections 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.394+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/444177304 wait complete. 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 -- start start 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 Processor -- start 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cff020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cff610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a18d05070 con 0x562a18cf75d0 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a18c18060 con 0x562a18d02370 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562a18d051f0 con 0x562a18cf8b40 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a9c9c7740 10 client.0 _ll_register_callbacks cb 0x562a1896b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.395+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49138/0 (socket says 172.21.15.16:49138) 2024-09-12T03:12:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 -- 172.21.15.16:0/3681912122 learned_addr learned my addr 172.21.15.16:0/3681912122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a937fe640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 -- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 msgr2=0x562a18cff020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cff020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 -- 172.21.15.16:0/3681912122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 msgr2=0x562a18cff610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cff610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 -- 172.21.15.16:0/3681912122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a189ce3c0 con 0x562a18cf75d0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a93fff640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f6a840129f0 tx=0x7f6a84012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f3ade0986b7e07 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400fac0 con 0x562a18cf75d0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6a8400fc60 con 0x562a18cf75d0 2024-09-12T03:12:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.396+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a8c01bcf0 con 0x562a18cf75d0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6a8400fe60 con 0x562a18cf75d0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 10 client.21009 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562a18d10b70 con 0x562a18cf75d0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c01bcf0 con 0x562a18cf75d0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6a840105a0 con 0x562a18cf75d0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 10 client.21009 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 10 client.21009 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 20 client.21009 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d12030 0x562a18d14420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 4 client.21009 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.397+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a18c6a860 con 0x562a18d12030 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.398+0000 7f6a937fe640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d12030 0x562a18d14420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.398+0000 7f6a937fe640 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d12030 0x562a18d14420 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46c56de37c4fc5c server_cookie=408f09ab5b4c953f in_seq=0 out_seq=0 2024-09-12T03:12:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.398+0000 7f6a737fe640 10 client.21009 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a737fe640 1 -- 172.21.15.16:0/3681912122 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562a18c6a860 con 0x562a18d12030 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a737fe640 10 client.21009 handle_command_reply: tid=0 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 client.21009 shutdown 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 msgr2=0x562a18d01d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f6a840129f0 tx=0x7f6a84012eb0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 shutdown_connections 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d12030 0x562a18d14420 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/3681912122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18d01d70 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a737fe640 0 client.21009 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 >> 172.21.15.16:0/3681912122 conn(0x562a18cc0310 msgr2=0x562a18cf9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 shutdown_connections 2024-09-12T03:12:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/3681912122 wait complete. 2024-09-12T03:12:17.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.399+0000 7f6a9c9c7740 20 client.21009 trim_cache size 0 max 16384 2024-09-12T03:12:17.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 Processor -- start 2024-09-12T03:12:17.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 -- start start 2024-09-12T03:12:17.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cf2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cfa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cfa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941af760 con 0x562a18cf75d0 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941b12a0 con 0x562a18d02370 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.414+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941afe30 con 0x562a18cf8b40 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a98b8b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cfa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cf2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a98b8b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cfa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45274/0 (socket says 172.21.15.16:45274) 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a98b8b640 1 -- 172.21.15.16:0/462110996 learned_addr learned my addr 172.21.15.16:0/462110996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 -- 172.21.15.16:0/462110996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 msgr2=0x562a18cfa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cfa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 -- 172.21.15.16:0/462110996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 msgr2=0x562a18cfa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cfa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 -- 172.21.15.16:0/462110996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a18cf4730 con 0x562a18cf75d0 2024-09-12T03:12:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a93fff640 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cf2bf0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6a84010bb0 tx=0x7f6a840092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc74904e9f8df8ee server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a79ffb640 1 -- 172.21.15.16:0/462110996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400fac0 con 0x562a18cf75d0 2024-09-12T03:12:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a79ffb640 1 -- 172.21.15.16:0/462110996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8402b070 con 0x562a18cf75d0 2024-09-12T03:12:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.415+0000 7f6a79ffb640 1 -- 172.21.15.16:0/462110996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a840048a0 con 0x562a18cf75d0 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/462110996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 msgr2=0x562a18cf2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cf2bf0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6a84010bb0 tx=0x7f6a840092c0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/462110996 shutdown_connections 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18d02370 0x562a18cfa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18cf8b40 0x562a18cfa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/462110996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cf2bf0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/462110996 >> 172.21.15.16:0/462110996 conn(0x562a18cc0310 msgr2=0x562a18cf24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/462110996 shutdown_connections 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.416+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/462110996 wait complete. 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 -- start start 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 Processor -- start 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18cf8b40 0x562a18cfc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 0x562a18cfcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941b0bd0 con 0x562a18cf75d0 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941b0500 con 0x562a18cf8b40 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a941ad740 con 0x562a18d02370 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a9c9c7740 10 client.0 _ll_register_callbacks cb 0x7f6a84009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.417+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49170/0 (socket says 172.21.15.16:49170) 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 -- 172.21.15.16:0/1049723899 learned_addr learned my addr 172.21.15.16:0/1049723899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18cf8b40 0x562a18cfc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 0x562a18cfcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 -- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 msgr2=0x562a18cfcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 0x562a18cfcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 -- 172.21.15.16:0/1049723899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18cf8b40 msgr2=0x562a18cfc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18cf8b40 0x562a18cfc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 -- 172.21.15.16:0/1049723899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a18d7bca0 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a98b8b640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 0x562a18cfcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a93fff640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f6a84010690 tx=0x7f6a84005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953abb89b0f43882 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400f790 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f6a840050d0 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.418+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a8402b070 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6a84005690 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 10 client.21015 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x562a18d7cf60 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8402b070 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f6a840036f0 con 0x562a18cf75d0 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 10 client.21015 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 10 client.21015 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:12:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 20 client.21015 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d7d610 0x562a18d7fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 4 client.21015 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:12:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a18c64d70 con 0x562a18d7d610 2024-09-12T03:12:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.419+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d7d610 0x562a18d7fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.420+0000 7f6a937fe640 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d7d610 0x562a18d7fa30 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e1dfea38df9e7f4 server_cookie=8f325a9028e83c0d in_seq=0 out_seq=0 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.420+0000 7f6a737fe640 10 client.21015 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.422+0000 7f6a737fe640 1 -- 172.21.15.16:0/1049723899 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a18c64d70 con 0x562a18d7d610 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.422+0000 7f6a737fe640 10 client.21015 handle_command_reply: tid=0 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.422+0000 7f6a9c9c7740 1 client.21015 shutdown 2024-09-12T03:12:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 msgr2=0x562a18cfc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f6a84010690 tx=0x7f6a84005970 comp rx=0 tx=0).stop 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 shutdown_connections 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x562a18d7d610 0x562a18d7fa30 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a737fe640 0 client.21015 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a18d02370 0x562a18cfcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a18cf8b40 0x562a18cfc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 --2- 172.21.15.16:0/1049723899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a18cf75d0 0x562a18cfc050 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 >> 172.21.15.16:0/1049723899 conn(0x562a18cc0310 msgr2=0x562a18cf2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 shutdown_connections 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 1 -- 172.21.15.16:0/1049723899 wait complete. 2024-09-12T03:12:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9c9c7740 20 client.21015 trim_cache size 0 max 16384 2024-09-12T03:12:17.426 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:12:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.423+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 msgr2=0x7f6a7c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 0x7f6a7c0a5040 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c023000 tx=0x7f6a8c024000 comp rx=0 tx=0).stop 2024-09-12T03:12:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 msgr2=0x7f6a941a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a941a0a40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009780 tx=0x7f6a88033ae0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 shutdown_connections 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6a941082a0 0x7f6a9419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a94103720 0x7f6a941a0a40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f6a7c0a2b80 0x7f6a7c0a5040 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 --2- 172.21.15.16:0/1594215276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a94102d70 0x7f6a941a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 >> 172.21.15.16:0/1594215276 conn(0x7f6a940fc7d0 msgr2=0x7f6a941009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 shutdown_connections 2024-09-12T03:12:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:17.424+0000 7f6a9a5ee640 1 -- 172.21.15.16:0/1594215276 wait complete. 2024-09-12T03:12:17.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:17 smithi016 ceph-mon[30925]: pgmap v1068: 97 pgs: 97 active+clean; 933 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.4 MiB/s wr, 633 op/s 2024-09-12T03:12:17.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:17 smithi184 ceph-mon[35321]: pgmap v1068: 97 pgs: 97 active+clean; 933 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.4 MiB/s wr, 633 op/s 2024-09-12T03:12:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:17 smithi033 ceph-mon[35092]: pgmap v1068: 97 pgs: 97 active+clean; 933 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.4 MiB/s wr, 633 op/s 2024-09-12T03:12:18.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 283964 23.51 MB/sec execute 559 sec latency 143.751 ms 2024-09-12T03:12:18.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:12:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2487215445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 msgr2=0x7f47e0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2487215445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e0101200 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009900 tx=0x7f47d002a480 comp rx=0 tx=0).stop 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2487215445 shutdown_connections 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2487215445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102ef0 0x7f47e010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2487215445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102570 0x7f47e01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2487215445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e0101200 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2487215445 >> 172.21.15.16:0/2487215445 conn(0x7f47e00fc7d0 msgr2=0x7f47e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2487215445 shutdown_connections 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.899+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2487215445 wait complete. 2024-09-12T03:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 Processor -- start 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 -- start start 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 0x7f47e0198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102ef0 0x7f47e019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47e0110f80 con 0x7f47e0100e20 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47e0110e00 con 0x7f47e0102ef0 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.900+0000 7f47e5d5e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47e0111100 con 0x7f47e0102570 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49196/0 (socket says 172.21.15.16:49196) 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 -- 172.21.15.16:0/2382017721 learned_addr learned my addr 172.21.15.16:0/2382017721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47d7fff640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 0x7f47e0198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47dffff640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102ef0 0x7f47e019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 -- 172.21.15.16:0/2382017721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 msgr2=0x7f47e0198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 0x7f47e0198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 -- 172.21.15.16:0/2382017721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102ef0 msgr2=0x7f47e019cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102ef0 0x7f47e019cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 -- 172.21.15.16:0/2382017721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e019d470 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47d7fff640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 0x7f47e0198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47df7fe640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f47d0006010 tx=0x7f47d0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.901+0000 7f47dd7fa640 1 -- 172.21.15.16:0/2382017721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d003f070 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.902+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e019d700 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.902+0000 7f47dd7fa640 1 -- 172.21.15.16:0/2382017721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d0031c30 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.902+0000 7f47dd7fa640 1 -- 172.21.15.16:0/2382017721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0035030 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.902+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e01a5040 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.904+0000 7f47dd7fa640 1 -- 172.21.15.16:0/2382017721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f47d0042070 con 0x7f47e0100e20 2024-09-12T03:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.904+0000 7f47e8137740 1 Processor -- start 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.904+0000 7f47e8137740 1 -- start start 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de906a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de917b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3de86c2f0 con 0x55a3de907b40 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a3de813cf0 con 0x55a3de9065d0 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47e8137740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a3de8682b0 con 0x55a3de915780 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dd7fa640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 0x7f47cc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49202/0 (socket says 172.21.15.16:49202) 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 -- 172.21.15.16:0/1480235184 learned_addr learned my addr 172.21.15.16:0/1480235184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 0x7f47cc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47df7fe640 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de906a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47d7fff640 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de917b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dd7fa640 1 -- 172.21.15.16:0/2382017721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47d00ef690 con 0x7f47e0100e20 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 -- 172.21.15.16:0/1480235184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 msgr2=0x55a3de917b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de917b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 -- 172.21.15.16:0/1480235184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 msgr2=0x55a3de906a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de906a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 -- 172.21.15.16:0/1480235184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3de7f83a0 con 0x55a3de907b40 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.905+0000 7f47dffff640 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f47d8011e20 tx=0x7f47d800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da626c287dcf6db4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47dffff640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 0x7f47cc0a5040 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f47d800fc90 tx=0x7f47d801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47d5ffb640 1 -- 172.21.15.16:0/1480235184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d801dcb0 con 0x55a3de907b40 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47d5ffb640 1 -- 172.21.15.16:0/1480235184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d801de50 con 0x55a3de907b40 2024-09-12T03:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47d5ffb640 1 -- 172.21.15.16:0/1480235184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8004280 con 0x55a3de907b40 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47e8137740 1 -- 172.21.15.16:0/1480235184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 msgr2=0x55a3de906090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.906+0000 7f47e8137740 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f47d8011e20 tx=0x7f47d800f790 comp rx=0 tx=0).stop 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- 172.21.15.16:0/1480235184 shutdown_connections 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de917b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de906a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- 172.21.15.16:0/1480235184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de907b40 0x55a3de906090 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- 172.21.15.16:0/1480235184 >> 172.21.15.16:0/1480235184 conn(0x55a3de8cf310 msgr2=0x55a3de902c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- 172.21.15.16:0/1480235184 shutdown_connections 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- 172.21.15.16:0/1480235184 wait complete. 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- start start 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 Processor -- start 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de9151b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de912bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3de9183c0 con 0x55a3de915780 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a3de827060 con 0x55a3de9065d0 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.907+0000 7f47e8137740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a3de918540 con 0x55a3de907b40 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47e8137740 10 client.0 _ll_register_callbacks cb 0x55a3de57a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36684/0 (socket says 172.21.15.16:36684) 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 -- 172.21.15.16:0/3906456910 learned_addr learned my addr 172.21.15.16:0/3906456910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47df7fe640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de9151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47dffff640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de912bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 -- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 msgr2=0x55a3de9151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de9151b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 -- 172.21.15.16:0/3906456910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 msgr2=0x55a3de912bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de912bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 -- 172.21.15.16:0/3906456910 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3de5dd3c0 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47d7fff640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f47c8012d00 tx=0x7f47c800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3cf0965cbaa5d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47beffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8012110 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47c8004050 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.908+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47d801de50 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f47c80122b0 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 10 client.30124 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a3de923f00 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d801de50 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f47c8010550 con 0x55a3de907b40 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 10 client.30124 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:18.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 10 client.30124 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 20 client.30124 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de925410 0x55a3de927800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 4 client.30124 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3de879860 con 0x55a3de925410 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.909+0000 7f47df7fe640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de925410 0x55a3de927800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.910+0000 7f47beffd640 10 client.30124 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.910+0000 7f47df7fe640 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de925410 0x55a3de927800 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=597108c232297211 server_cookie=22fc547f3c2e10e5 in_seq=0 out_seq=0 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47beffd640 1 -- 172.21.15.16:0/3906456910 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a3de879860 con 0x55a3de925410 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47beffd640 10 client.30124 handle_command_reply: tid=0 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 client.30124 shutdown 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 msgr2=0x55a3de912460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f47c8012d00 tx=0x7f47c800f790 comp rx=0 tx=0).stop 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 shutdown_connections 2024-09-12T03:12:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de925410 0x55a3de927800 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de912bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de907b40 0x55a3de912460 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47beffd640 0 client.30124 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 --2- 172.21.15.16:0/3906456910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de9065d0 0x55a3de9151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 >> 172.21.15.16:0/3906456910 conn(0x55a3de8cf310 msgr2=0x55a3de908960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 shutdown_connections 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 1 -- 172.21.15.16:0/3906456910 wait complete. 2024-09-12T03:12:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.911+0000 7f47e8137740 20 client.30124 trim_cache size 0 max 16384 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 Processor -- start 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 -- start start 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de9065d0 0x55a3de911d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de92d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01ab330 con 0x55a3de915780 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01ace70 con 0x55a3de907b40 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.926+0000 7f47e8137740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01aba00 con 0x55a3de9065d0 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45314/0 (socket says 172.21.15.16:45314) 2024-09-12T03:12:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 -- 172.21.15.16:0/116608486 learned_addr learned my addr 172.21.15.16:0/116608486 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47dffff640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de92d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47df7fe640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de9065d0 0x55a3de911d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 -- 172.21.15.16:0/116608486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de9065d0 msgr2=0x55a3de911d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de9065d0 0x55a3de911d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 -- 172.21.15.16:0/116608486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 msgr2=0x55a3de92d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de92d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 -- 172.21.15.16:0/116608486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3de927b70 con 0x55a3de907b40 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47dffff640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de92d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.927+0000 7f47d7fff640 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f47c8010b30 tx=0x7f47c80059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1a412942f08a08 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.928+0000 7f47d57fa640 1 -- 172.21.15.16:0/116608486 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8010bb0 con 0x55a3de907b40 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.928+0000 7f47d57fa640 1 -- 172.21.15.16:0/116608486 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47c8010d50 con 0x55a3de907b40 2024-09-12T03:12:18.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.928+0000 7f47d57fa640 1 -- 172.21.15.16:0/116608486 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003990 con 0x55a3de907b40 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.928+0000 7f47e8137740 1 -- 172.21.15.16:0/116608486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 msgr2=0x55a3de912260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f47c8010b30 tx=0x7f47c80059b0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 -- 172.21.15.16:0/116608486 shutdown_connections 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de915780 0x55a3de92d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de912260 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 --2- 172.21.15.16:0/116608486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de9065d0 0x55a3de911d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 -- 172.21.15.16:0/116608486 >> 172.21.15.16:0/116608486 conn(0x55a3de8cf310 msgr2=0x55a3de90d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 -- 172.21.15.16:0/116608486 shutdown_connections 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 -- 172.21.15.16:0/116608486 wait complete. 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 -- start start 2024-09-12T03:12:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.929+0000 7f47e8137740 1 Processor -- start 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de919190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de9196d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01ac7a0 con 0x55a3de9065d0 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01ac0d0 con 0x55a3de907b40 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47e01a9310 con 0x55a3de915780 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47e8137740 10 client.0 _ll_register_callbacks cb 0x7f47c8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49234/0 (socket says 172.21.15.16:49234) 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 -- 172.21.15.16:0/1846067727 learned_addr learned my addr 172.21.15.16:0/1846067727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47dffff640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de9196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47d7fff640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de919190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 -- 172.21.15.16:0/1846067727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 msgr2=0x55a3de9196d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de9196d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 -- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 msgr2=0x55a3de919190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de919190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 -- 172.21.15.16:0/1846067727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3de919dc0 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47dffff640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de9196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47df7fe640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f47d00b8980 tx=0x7f47d008be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2b76fa4e72f068 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.930+0000 7f47beffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0049040 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f47d003b460 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47c8010d50 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f47d008a030 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 10 client.21033 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a3de98ffe0 con 0x55a3de9065d0 2024-09-12T03:12:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47c8010d50 con 0x55a3de9065d0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f47d00c46e0 con 0x55a3de9065d0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 10 client.21033 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 10 client.21033 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 20 client.21033 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de9906b0 0x55a3de992ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 4 client.21033 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.931+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3de873d70 con 0x55a3de9906b0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47d7fff640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de9906b0 0x55a3de992ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47d7fff640 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de9906b0 0x55a3de992ad0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c12298a3081edeb7 server_cookie=329913ddf56b0f5f in_seq=0 out_seq=0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47beffd640 10 client.21033 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47beffd640 1 -- 172.21.15.16:0/1846067727 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3de873d70 con 0x55a3de9906b0 2024-09-12T03:12:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47beffd640 10 client.21033 handle_command_reply: tid=0 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47e8137740 1 client.21033 shutdown 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 msgr2=0x55a3de91de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f47d00b8980 tx=0x7f47d008be60 comp rx=0 tx=0).stop 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 shutdown_connections 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.932+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55a3de9906b0 0x55a3de992ad0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3de915780 0x55a3de9196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3de907b40 0x55a3de919190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 --2- 172.21.15.16:0/1846067727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3de9065d0 0x55a3de91de10 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 >> 172.21.15.16:0/1846067727 conn(0x55a3de8cf310 msgr2=0x55a3de911cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47beffd640 0 client.21033 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 shutdown_connections 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 1 -- 172.21.15.16:0/1846067727 wait complete. 2024-09-12T03:12:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e8137740 20 client.21033 trim_cache size 0 max 16384 2024-09-12T03:12:18.942 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:12:18.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 msgr2=0x7f47cc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 0x7f47cc0a5040 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f47d800fc90 tx=0x7f47d801e000 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 msgr2=0x7f47e01984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f47d0006010 tx=0x7f47d0003c10 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 shutdown_connections 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47e0102ef0 0x7f47e019cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47e0102570 0x7f47e0198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f47cc0a2b80 0x7f47cc0a5040 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 --2- 172.21.15.16:0/2382017721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e0100e20 0x7f47e01984c0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.933+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 >> 172.21.15.16:0/2382017721 conn(0x7f47e00fc7d0 msgr2=0x7f47e006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.934+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 shutdown_connections 2024-09-12T03:12:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:18.934+0000 7f47e5d5e640 1 -- 172.21.15.16:0/2382017721 wait complete. 2024-09-12T03:12:19.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 284357 23.51 MB/sec execute 560 sec latency 95.945 ms 2024-09-12T03:12:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:19 smithi016 ceph-mon[30925]: pgmap v1069: 97 pgs: 97 active+clean; 926 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.4 MiB/s wr, 597 op/s 2024-09-12T03:12:19.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:12:19.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:19 smithi184 ceph-mon[35321]: pgmap v1069: 97 pgs: 97 active+clean; 926 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.4 MiB/s wr, 597 op/s 2024-09-12T03:12:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:19.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:19 smithi033 ceph-mon[35092]: pgmap v1069: 97 pgs: 97 active+clean; 926 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.4 MiB/s wr, 597 op/s 2024-09-12T03:12:20.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 284872 23.50 MB/sec execute 561 sec latency 98.819 ms 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.711+0000 7fc735b11640 1 -- 172.21.15.16:0/4213366720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730103730 msgr2=0x7fc73010bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.711+0000 7fc735b11640 1 --2- 172.21.15.16:0/4213366720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730103730 0x7fc73010bc20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc724009860 tx=0x7fc72402a3d0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 -- 172.21.15.16:0/4213366720 shutdown_connections 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 --2- 172.21.15.16:0/4213366720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 --2- 172.21.15.16:0/4213366720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730103730 0x7fc73010bc20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 --2- 172.21.15.16:0/4213366720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730102e10 0x7fc7301031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 -- 172.21.15.16:0/4213366720 >> 172.21.15.16:0/4213366720 conn(0x7fc7300fc7d0 msgr2=0x7fc7300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.712+0000 7fc735b11640 1 -- 172.21.15.16:0/4213366720 shutdown_connections 2024-09-12T03:12:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.713+0000 7fc735b11640 1 -- 172.21.15.16:0/4213366720 wait complete. 2024-09-12T03:12:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.714+0000 7fc735b11640 1 Processor -- start 2024-09-12T03:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.714+0000 7fc735b11640 1 -- start start 2024-09-12T03:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 0x7fc7301a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc730111fd0 con 0x7fc730103730 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc730111e50 con 0x7fc73010c1b0 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.715+0000 7fc735b11640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc730112150 con 0x7fc730102e10 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.716+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.716+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36728/0 (socket says 172.21.15.16:36728) 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.716+0000 7fc72f7fe640 1 -- 172.21.15.16:0/1984632411 learned_addr learned my addr 172.21.15.16:0/1984632411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.716+0000 7fc72effd640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 0x7fc7301a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72ffff640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72f7fe640 1 -- 172.21.15.16:0/1984632411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 msgr2=0x7fc73010ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72f7fe640 1 -- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 msgr2=0x7fc7301a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 0x7fc7301a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72f7fe640 1 -- 172.21.15.16:0/1984632411 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73010b5a0 con 0x7fc730102e10 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72ffff640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.717+0000 7fc72effd640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 0x7fc7301a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e9f0 tx=0x7fc72000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc72cff9640 1 -- 172.21.15.16:0/1984632411 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72000cde0 con 0x7fc730102e10 2024-09-12T03:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc72cff9640 1 -- 172.21.15.16:0/1984632411 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7200035f0 con 0x7fc730102e10 2024-09-12T03:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73010b890 con 0x7fc730102e10 2024-09-12T03:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc72cff9640 1 -- 172.21.15.16:0/1984632411 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720010690 con 0x7fc730102e10 2024-09-12T03:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.718+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc730197630 con 0x7fc730102e10 2024-09-12T03:12:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 Processor -- start 2024-09-12T03:12:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 -- start start 2024-09-12T03:12:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ac8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ac8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556504a2e2f0 con 0x556504ac85d0 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565049d5cf0 con 0x556504ad3370 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.722+0000 7fc737eea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556504a2a2b0 con 0x556504ac9b40 2024-09-12T03:12:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45338/0 (socket says 172.21.15.16:45338) 2024-09-12T03:12:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 -- 172.21.15.16:0/1029469940 learned_addr learned my addr 172.21.15.16:0/1029469940 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 -- 172.21.15.16:0/1029469940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 msgr2=0x556504ac8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ac8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72cff9640 1 -- 172.21.15.16:0/1984632411 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc7200037a0 con 0x7fc730102e10 2024-09-12T03:12:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 -- 172.21.15.16:0/1029469940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 msgr2=0x556504ac8a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:12:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ac8a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.723+0000 7fc72effd640 1 -- 172.21.15.16:0/1029469940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565049ba3a0 con 0x556504ad3370 2024-09-12T03:12:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc72effd640 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc724009860 tx=0x7fc724030ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd3e357b895d7e19 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc72cff9640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 0x7fc7180a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc715ffb640 1 -- 172.21.15.16:0/1029469940 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7240306a0 con 0x556504ad3370 2024-09-12T03:12:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc715ffb640 1 -- 172.21.15.16:0/1029469940 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc724041030 con 0x556504ad3370 2024-09-12T03:12:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc715ffb640 1 -- 172.21.15.16:0/1029469940 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724004920 con 0x556504ad3370 2024-09-12T03:12:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.724+0000 7fc72ffff640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 0x7fc7180a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc72cff9640 1 -- 172.21.15.16:0/1984632411 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc7200cf2b0 con 0x7fc730102e10 2024-09-12T03:12:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 -- 172.21.15.16:0/1029469940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 msgr2=0x556504ad5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc72ffff640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 0x7fc7180a50c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc728009370 tx=0x7fc7280094b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc724009860 tx=0x7fc724030ee0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 -- 172.21.15.16:0/1029469940 shutdown_connections 2024-09-12T03:12:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad5760 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ac8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 --2- 172.21.15.16:0/1029469940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ac8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 -- 172.21.15.16:0/1029469940 >> 172.21.15.16:0/1029469940 conn(0x556504a91310 msgr2=0x556504ac4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.725+0000 7fc737eea740 1 -- 172.21.15.16:0/1029469940 shutdown_connections 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.726+0000 7fc737eea740 1 -- 172.21.15.16:0/1029469940 wait complete. 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.726+0000 7fc737eea740 1 -- start start 2024-09-12T03:12:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.726+0000 7fc737eea740 1 Processor -- start 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ad0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556504ad6010 con 0x556504ac85d0 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565049e9060 con 0x556504ad3370 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556504ad6190 con 0x556504ac9b40 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc737eea740 10 client.0 _ll_register_callbacks cb 0x55650473c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49262/0 (socket says 172.21.15.16:49262) 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.727+0000 7fc72f7fe640 1 -- 172.21.15.16:0/2464770717 learned_addr learned my addr 172.21.15.16:0/2464770717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 -- 172.21.15.16:0/2464770717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 msgr2=0x556504ad0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ad0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 -- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 msgr2=0x556504ad0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 -- 172.21.15.16:0/2464770717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55650479f3c0 con 0x556504ac85d0 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e9c0 tx=0x7fc72009c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7255cb62f3e23124 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc70effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7200a4790 con 0x556504ac85d0 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc7200a4300 con 0x556504ac85d0 2024-09-12T03:12:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.728+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc724041030 con 0x556504ac85d0 2024-09-12T03:12:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc720009500 con 0x556504ac85d0 2024-09-12T03:12:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 10 client.21042 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556504ae1b20 con 0x556504ac85d0 2024-09-12T03:12:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc724041030 con 0x556504ac85d0 2024-09-12T03:12:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fc72009b0e0 con 0x556504ac85d0 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 10 client.21042 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 10 client.21042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 20 client.21042 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504ae5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 4 client.21042 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.729+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556504a3b860 con 0x556504ae3030 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.730+0000 7fc72effd640 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504ae5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.732+0000 7fc72effd640 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504ae5480 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d43f135e21281d5f server_cookie=91ca38cf9f88fb91 in_seq=0 out_seq=0 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.732+0000 7fc70effd640 10 client.21042 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.733+0000 7fc70effd640 1 -- 172.21.15.16:0/2464770717 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556504a3b860 con 0x556504ae3030 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.733+0000 7fc70effd640 10 client.21042 handle_command_reply: tid=0 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.733+0000 7fc737eea740 1 client.21042 shutdown 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 msgr2=0x556504ad2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e9c0 tx=0x7fc72009c220 comp rx=0 tx=0).stop 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 shutdown_connections 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504ae5480 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc70effd640 0 client.21042 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504ad0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac9b40 0x556504ad0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 --2- 172.21.15.16:0/2464770717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac85d0 0x556504ad2dc0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 >> 172.21.15.16:0/2464770717 conn(0x556504a91310 msgr2=0x556504aca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 shutdown_connections 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 1 -- 172.21.15.16:0/2464770717 wait complete. 2024-09-12T03:12:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.734+0000 7fc737eea740 20 client.21042 trim_cache size 0 max 16384 2024-09-12T03:12:20.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 Processor -- start 2024-09-12T03:12:20.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 -- start start 2024-09-12T03:12:20.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ac9b40 0x556504acb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ad3370 0x556504acb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc73019fb10 con 0x556504ad3370 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7301bdef0 con 0x556504ac9b40 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc737eea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7301bca80 con 0x556504ac85d0 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36768/0 (socket says 172.21.15.16:36768) 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc72f7fe640 1 -- 172.21.15.16:0/476818959 learned_addr learned my addr 172.21.15.16:0/476818959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.752+0000 7fc72ffff640 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ad3370 0x556504acb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72effd640 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ac9b40 0x556504acb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 -- 172.21.15.16:0/476818959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ac9b40 msgr2=0x556504acb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ac9b40 0x556504acb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 -- 172.21.15.16:0/476818959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ad3370 msgr2=0x556504acb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ad3370 0x556504acb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 -- 172.21.15.16:0/476818959 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556504ac5960 con 0x556504ac85d0 2024-09-12T03:12:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.753+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc7200cce10 tx=0x7fc72009a030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19a53a3bf68fe54d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.754+0000 7fc7157fa640 1 -- 172.21.15.16:0/476818959 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720009500 con 0x556504ac85d0 2024-09-12T03:12:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.754+0000 7fc7157fa640 1 -- 172.21.15.16:0/476818959 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc72009ce50 con 0x556504ac85d0 2024-09-12T03:12:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.754+0000 7fc7157fa640 1 -- 172.21.15.16:0/476818959 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7200a4540 con 0x556504ac85d0 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 -- 172.21.15.16:0/476818959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 msgr2=0x556504ac3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc7200cce10 tx=0x7fc72009a030 comp rx=0 tx=0).stop 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 -- 172.21.15.16:0/476818959 shutdown_connections 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ad3370 0x556504acb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ac9b40 0x556504acb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 --2- 172.21.15.16:0/476818959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504ac3bf0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 -- 172.21.15.16:0/476818959 >> 172.21.15.16:0/476818959 conn(0x556504a91310 msgr2=0x556504ac34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 -- 172.21.15.16:0/476818959 shutdown_connections 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.755+0000 7fc737eea740 1 -- 172.21.15.16:0/476818959 wait complete. 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.756+0000 7fc737eea740 1 -- start start 2024-09-12T03:12:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.756+0000 7fc737eea740 1 Processor -- start 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.756+0000 7fc737eea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504acd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc737eea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 0x556504acd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504acd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc72f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504acd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36772/0 (socket says 172.21.15.16:36772) 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc72effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 0x556504acd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc72f7fe640 1 -- 172.21.15.16:0/2933180770 learned_addr learned my addr 172.21.15.16:0/2933180770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504acdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7301bd820 con 0x556504ac9b40 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7301bd150 con 0x556504ad3370 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc730103230 con 0x556504ac85d0 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.757+0000 7fc72ffff640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504acdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc737eea740 10 client.0 _ll_register_callbacks cb 0x7fc720003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 -- 172.21.15.16:0/2933180770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 msgr2=0x556504acd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504acd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 -- 172.21.15.16:0/2933180770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 msgr2=0x556504acd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 0x556504acd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556504b4cec0 con 0x556504ad3370 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72effd640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 0x556504acd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc72ffff640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504acdcb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc728009bc0 tx=0x7fc7280099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6888254962711737 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc70effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.758+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728023070 con 0x556504ad3370 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc728003c30 con 0x556504ad3370 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72009ce50 con 0x556504ad3370 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc72802f050 con 0x556504ad3370 2024-09-12T03:12:20.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc737eea740 10 client.30194 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x556504b4e180 con 0x556504ad3370 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc72009ce50 con 0x556504ad3370 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.759+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fc72800f040 con 0x556504ad3370 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 10 client.30194 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 10 client.30194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 20 client.30194 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504b50940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 4 client.30194 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.760+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556504a35d70 con 0x556504ae3030 2024-09-12T03:12:20.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.761+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504b50940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.762+0000 7fc72f7fe640 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504b50940 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdb5888dbb5df96d server_cookie=5f95cd734970e688 in_seq=0 out_seq=0 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.762+0000 7fc70effd640 10 client.30194 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc70effd640 1 -- 172.21.15.16:0/2933180770 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556504a35d70 con 0x556504ae3030 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc70effd640 10 client.30194 handle_command_reply: tid=0 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 client.30194 shutdown 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 msgr2=0x556504acdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504acdcb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc728009bc0 tx=0x7fc7280099b0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 shutdown_connections 2024-09-12T03:12:20.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x556504ae3030 0x556504b50940 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc70effd640 0 client.30194 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556504ad3370 0x556504acdcb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556504ac9b40 0x556504acd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 --2- 172.21.15.16:0/2933180770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556504ac85d0 0x556504acd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.763+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 >> 172.21.15.16:0/2933180770 conn(0x556504a91310 msgr2=0x556504ad18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.764+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 shutdown_connections 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.764+0000 7fc737eea740 1 -- 172.21.15.16:0/2933180770 wait complete. 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.764+0000 7fc737eea740 20 client.30194 trim_cache size 0 max 16384 2024-09-12T03:12:20.766 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:12:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 msgr2=0x7fc7180a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 0x7fc7180a50c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc728009370 tx=0x7fc7280094b0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 msgr2=0x7fc7301a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:20.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e9f0 tx=0x7fc72000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 shutdown_connections 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc73010c1b0 0x7fc73010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc7180a2c00 0x7fc7180a50c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730103730 0x7fc7301a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 --2- 172.21.15.16:0/1984632411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc730102e10 0x7fc7301a4780 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.765+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 >> 172.21.15.16:0/1984632411 conn(0x7fc7300fc7d0 msgr2=0x7fc730101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.766+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 shutdown_connections 2024-09-12T03:12:20.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:20.766+0000 7fc735b11640 1 -- 172.21.15.16:0/1984632411 wait complete. 2024-09-12T03:12:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 285173 23.48 MB/sec execute 562 sec latency 258.319 ms 2024-09-12T03:12:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:21 smithi184 ceph-mon[35321]: pgmap v1070: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.6 MiB/s wr, 644 op/s 2024-09-12T03:12:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:12:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:12:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:12:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:12:21.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:21 smithi016 ceph-mon[30925]: pgmap v1070: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.6 MiB/s wr, 644 op/s 2024-09-12T03:12:21.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:12:21.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:12:21.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:12:21.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:12:21.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:21 smithi033 ceph-mon[35092]: pgmap v1070: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.6 MiB/s wr, 644 op/s 2024-09-12T03:12:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:12:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:12:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:12:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:12:22.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 285713 23.49 MB/sec execute 563 sec latency 78.932 ms 2024-09-12T03:12:22.820 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:12:22.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:12:23.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 286608 23.52 MB/sec execute 564 sec latency 50.259 ms 2024-09-12T03:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:23.790 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:23 smithi016 ceph-mon[30925]: pgmap v1071: 97 pgs: 97 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 594 op/s 2024-09-12T03:12:23.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:23 smithi184 ceph-mon[35321]: pgmap v1071: 97 pgs: 97 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 594 op/s 2024-09-12T03:12:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 -- 172.21.15.16:0/873596610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 msgr2=0x7f086c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 --2- 172.21.15.16:0/873596610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c103110 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f0860009930 tx=0x7f086002a420 comp rx=0 tx=0).stop 2024-09-12T03:12:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 -- 172.21.15.16:0/873596610 shutdown_connections 2024-09-12T03:12:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 --2- 172.21.15.16:0/873596610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 0x7f086c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 --2- 172.21.15.16:0/873596610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 --2- 172.21.15.16:0/873596610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c103110 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 -- 172.21.15.16:0/873596610 >> 172.21.15.16:0/873596610 conn(0x7f086c0fc7d0 msgr2=0x7f086c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 -- 172.21.15.16:0/873596610 shutdown_connections 2024-09-12T03:12:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.896+0000 7f0872f93640 1 -- 172.21.15.16:0/873596610 wait complete. 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 Processor -- start 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 -- start start 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 0x7f086c19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1120a0 con 0x7f086c102d30 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f086c111f20 con 0x7f086c1036e0 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0872f93640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f086c112220 con 0x7f086c1082a0 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45374/0 (socket says 172.21.15.16:45374) 2024-09-12T03:12:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f086bfff640 1 -- 172.21.15.16:0/2602607306 learned_addr learned my addr 172.21.15.16:0/2602607306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.897+0000 7f0871530640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 0x7f086c19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 -- 172.21.15.16:0/2602607306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 msgr2=0x7f086c19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 0x7f086c19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 -- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 msgr2=0x7f086c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 -- 172.21.15.16:0/2602607306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c19d570 con 0x7f086c1036e0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f0870d2f640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f086bfff640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f085800d740 tx=0x7f085800dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f0869ffb640 1 -- 172.21.15.16:0/2602607306 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858017070 con 0x7f086c1036e0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c19d860 con 0x7f086c1036e0 2024-09-12T03:12:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f0869ffb640 1 -- 172.21.15.16:0/2602607306 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085800de50 con 0x7f086c1036e0 2024-09-12T03:12:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.898+0000 7f0869ffb640 1 -- 172.21.15.16:0/2602607306 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085800f6b0 con 0x7f086c1036e0 2024-09-12T03:12:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.899+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c1008c0 con 0x7f086c1036e0 2024-09-12T03:12:23.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.901+0000 7f0869ffb640 1 -- 172.21.15.16:0/2602607306 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f085801a070 con 0x7f086c1036e0 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.901+0000 7f087536c740 1 Processor -- start 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 -- start start 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179fa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179fd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff17955280 con 0x55ff179efb60 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff178fcb10 con 0x55ff179fad40 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f087536c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff17951240 con 0x55ff179f0480 2024-09-12T03:12:23.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49298/0 (socket says 172.21.15.16:49298) 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 -- 172.21.15.16:0/666515963 learned_addr learned my addr 172.21.15.16:0/666515963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0870d2f640 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179fa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f086bfff640 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179fd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 -- 172.21.15.16:0/666515963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 msgr2=0x55ff179fa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179fa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 -- 172.21.15.16:0/666515963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 msgr2=0x55ff179fd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179fd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0871530640 1 -- 172.21.15.16:0/666515963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff179091f0 con 0x55ff179efb60 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.902+0000 7f0869ffb640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 0x7f085c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0870d2f640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 0x7f085c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0869ffb640 1 -- 172.21.15.16:0/2602607306 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f08580ceca0 con 0x7f086c1036e0 2024-09-12T03:12:23.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0871530640 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f0864009f70 tx=0x7f0864010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e616843408425d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0856ffd640 1 -- 172.21.15.16:0/666515963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864016070 con 0x55ff179efb60 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0856ffd640 1 -- 172.21.15.16:0/666515963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864004d20 con 0x55ff179efb60 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.903+0000 7f0856ffd640 1 -- 172.21.15.16:0/666515963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864012870 con 0x55ff179efb60 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- 172.21.15.16:0/666515963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 msgr2=0x55ff179eff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f0864009f70 tx=0x7f0864010740 comp rx=0 tx=0).stop 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f0870d2f640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 0x7f085c0a50c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0860009900 tx=0x7f086003c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- 172.21.15.16:0/666515963 shutdown_connections 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179fd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179fa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 --2- 172.21.15.16:0/666515963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179eff40 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- 172.21.15.16:0/666515963 >> 172.21.15.16:0/666515963 conn(0x55ff179ec0f0 msgr2=0x55ff179ec4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- 172.21.15.16:0/666515963 shutdown_connections 2024-09-12T03:12:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- 172.21.15.16:0/666515963 wait complete. 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.904+0000 7f087536c740 1 -- start start 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 Processor -- start 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179f8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ff179fd910 con 0x55ff179efb60 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ff1790ff10 con 0x55ff179fad40 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ff179fda90 con 0x55ff179f0480 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f087536c740 10 client.0 _ll_register_callbacks cb 0x55ff178d8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f0871530640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49314/0 (socket says 172.21.15.16:49314) 2024-09-12T03:12:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f0870d2f640 1 -- 172.21.15.16:0/3369940722 learned_addr learned my addr 172.21.15.16:0/3369940722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.905+0000 7f086bfff640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 -- 172.21.15.16:0/3369940722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 msgr2=0x55ff179f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 -- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 msgr2=0x55ff179f8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179f8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 -- 172.21.15.16:0/3369940722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff176c66a0 con 0x55ff179efb60 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f0860005ec0 tx=0x7f086002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f41511075a0f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f084bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860035040 con 0x55ff179efb60 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f0860003cc0 con 0x55ff179efb60 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0864004d20 con 0x55ff179efb60 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f086004f430 con 0x55ff179efb60 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f087536c740 10 client.21060 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ff17a11cd0 con 0x55ff179efb60 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.906+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864004d20 con 0x55ff179efb60 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f0860058070 con 0x55ff179efb60 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 10 client.21060 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 10 client.21060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 20 client.21060 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 4 client.21060 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff179627f0 con 0x55ff17a13190 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.907+0000 7f086bfff640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.908+0000 7f086bfff640 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a155c0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=642b5daf257b5094 server_cookie=633e1dedf9471ed7 in_seq=0 out_seq=0 2024-09-12T03:12:23.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.908+0000 7f084bfff640 10 client.21060 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:23.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f084bfff640 1 -- 172.21.15.16:0/3369940722 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ff179627f0 con 0x55ff17a13190 2024-09-12T03:12:23.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f084bfff640 10 client.21060 handle_command_reply: tid=0 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 client.21060 shutdown 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 msgr2=0x55ff179f7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f0860005ec0 tx=0x7f086002a420 comp rx=0 tx=0).stop 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 shutdown_connections 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a155c0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f084bfff640 0 client.21060 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179fad40 0x55ff179f8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179f0480 0x55ff179f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 --2- 172.21.15.16:0/3369940722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179efb60 0x55ff179f7bd0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 >> 172.21.15.16:0/3369940722 conn(0x55ff179ec0f0 msgr2=0x55ff179eb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 shutdown_connections 2024-09-12T03:12:23.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.909+0000 7f087536c740 1 -- 172.21.15.16:0/3369940722 wait complete. 2024-09-12T03:12:23.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.910+0000 7f087536c740 20 client.21060 trim_cache size 0 max 16384 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.929+0000 7f087536c740 1 Processor -- start 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 -- start start 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff179f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a1b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ab3a0 con 0x55ff179fad40 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1acee0 con 0x55ff179f0480 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f087536c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1aba70 con 0x55ff179efb60 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a1b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff179f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45400/0 (socket says 172.21.15.16:45400) 2024-09-12T03:12:23.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f086bfff640 1 -- 172.21.15.16:0/1628129623 learned_addr learned my addr 172.21.15.16:0/1628129623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:23.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff179f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:36808/0 (socket says 172.21.15.16:36808) 2024-09-12T03:12:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 -- 172.21.15.16:0/1628129623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 msgr2=0x55ff179f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff179f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 -- 172.21.15.16:0/1628129623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 msgr2=0x55ff17a1aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f0871530640 1 -- 172.21.15.16:0/1628129623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff179e7830 con 0x55ff179fad40 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.930+0000 7f086bfff640 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.931+0000 7f0871530640 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a1b420 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f0864005ad0 tx=0x7f0864009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a396b46867e4130d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.931+0000 7f08567fc640 1 -- 172.21.15.16:0/1628129623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864016070 con 0x55ff179fad40 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.931+0000 7f08567fc640 1 -- 172.21.15.16:0/1628129623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864003e50 con 0x55ff179fad40 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.931+0000 7f08567fc640 1 -- 172.21.15.16:0/1628129623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864004830 con 0x55ff179fad40 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- 172.21.15.16:0/1628129623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 msgr2=0x55ff17a1b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a1b420 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f0864005ad0 tx=0x7f0864009840 comp rx=0 tx=0).stop 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- 172.21.15.16:0/1628129623 shutdown_connections 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a1b420 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a1aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 --2- 172.21.15.16:0/1628129623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff179f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- 172.21.15.16:0/1628129623 >> 172.21.15.16:0/1628129623 conn(0x55ff179ec0f0 msgr2=0x55ff179f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- 172.21.15.16:0/1628129623 shutdown_connections 2024-09-12T03:12:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- 172.21.15.16:0/1628129623 wait complete. 2024-09-12T03:12:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 -- start start 2024-09-12T03:12:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.932+0000 7f087536c740 1 Processor -- start 2024-09-12T03:12:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff17a07bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a76ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ac810 con 0x55ff179fad40 2024-09-12T03:12:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ac140 con 0x55ff179f0480 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1a9380 con 0x55ff179efb60 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f087536c740 10 client.0 _ll_register_callbacks cb 0x7f0860005df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f0870d2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff17a07bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f086bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:45408/0 (socket says 172.21.15.16:45408) 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f086bfff640 1 -- 172.21.15.16:0/3878655599 learned_addr learned my addr 172.21.15.16:0/3878655599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.933+0000 7f0871530640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a76ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 -- 172.21.15.16:0/3878655599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 msgr2=0x55ff17a07bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff17a07bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 -- 172.21.15.16:0/3878655599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 msgr2=0x55ff17a76ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a76ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 -- 172.21.15.16:0/3878655599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff179f02b0 con 0x55ff179f0480 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff17a07bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f0871530640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a76ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f086bfff640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f08580ccb20 tx=0x7f0858010d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82150e34605d0801 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f084bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858021040 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0864003e50 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f08580a3cf0 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0858099980 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f087536c740 10 client.30212 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ff17a7e7f0 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.934+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864003e50 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f0858084b50 con 0x55ff179f0480 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 10 client.30212 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 10 client.30212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 20 client.30212 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a80f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 4 client.30212 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:12:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ff1795cd00 con 0x55ff17a13190 2024-09-12T03:12:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a80f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f0870d2f640 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a80f20 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41e8ee8d3f68e95 server_cookie=bdad761e17989012 in_seq=0 out_seq=0 2024-09-12T03:12:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.935+0000 7f084bfff640 10 client.30212 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f084bfff640 1 -- 172.21.15.16:0/3878655599 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ff1795cd00 con 0x55ff17a13190 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f084bfff640 10 client.30212 handle_command_reply: tid=0 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 client.30212 shutdown 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 msgr2=0x55ff17a08130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f08580ccb20 tx=0x7f0858010d40 comp rx=0 tx=0).stop 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 shutdown_connections 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55ff17a13190 0x55ff17a80f20 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ff179fad40 0x55ff17a76ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ff179f0480 0x55ff17a08130 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 --2- 172.21.15.16:0/3878655599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ff179efb60 0x55ff17a07bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 >> 172.21.15.16:0/3878655599 conn(0x55ff179ec0f0 msgr2=0x55ff179f7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f084bfff640 0 client.30212 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 shutdown_connections 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 1 -- 172.21.15.16:0/3878655599 wait complete. 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.937+0000 7f087536c740 20 client.30212 trim_cache size 0 max 16384 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d2ac9f91-4e2b-464f-bca3-8aaddf5ab237", 2024-09-12T03:12:23.942 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:12:23.943 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:12:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 msgr2=0x7f085c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 0x7f085c0a50c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0860009900 tx=0x7f086003c040 comp rx=0 tx=0).stop 2024-09-12T03:12:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 msgr2=0x7f086c198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f085800d740 tx=0x7f085800dc00 comp rx=0 tx=0).stop 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 shutdown_connections 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f086c1082a0 0x7f086c19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f086c1036e0 0x7f086c198b00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f085c0a2c00 0x7f085c0a50c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 --2- 172.21.15.16:0/2602607306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c102d30 0x7f086c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 >> 172.21.15.16:0/2602607306 conn(0x7f086c0fc7d0 msgr2=0x7f086c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 shutdown_connections 2024-09-12T03:12:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:23.938+0000 7f0872f93640 1 -- 172.21.15.16:0/2602607306 wait complete. 2024-09-12T03:12:23.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:23 smithi033 ceph-mon[35092]: pgmap v1071: 97 pgs: 97 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 594 op/s 2024-09-12T03:12:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 287295 23.56 MB/sec execute 565 sec latency 78.501 ms 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:24.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:24 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:24 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.941 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:24.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:24 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 287841 23.59 MB/sec execute 566 sec latency 44.399 ms 2024-09-12T03:12:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 -- 172.21.15.16:0/706022573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac102d30 msgr2=0x7f88ac103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 --2- 172.21.15.16:0/706022573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac102d30 0x7f88ac103110 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009930 tx=0x7f88a002a420 comp rx=0 tx=0).stop 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 -- 172.21.15.16:0/706022573 shutdown_connections 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 --2- 172.21.15.16:0/706022573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 0x7f88ac10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 --2- 172.21.15.16:0/706022573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac1036e0 0x7f88ac107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 --2- 172.21.15.16:0/706022573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac102d30 0x7f88ac103110 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 -- 172.21.15.16:0/706022573 >> 172.21.15.16:0/706022573 conn(0x7f88ac0fc7d0 msgr2=0x7f88ac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 -- 172.21.15.16:0/706022573 shutdown_connections 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.448+0000 7f88b2e51640 1 -- 172.21.15.16:0/706022573 wait complete. 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 Processor -- start 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 -- start start 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 0x7f88ac198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 0x7f88ac19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1120a0 con 0x7f88ac1036e0 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac111f20 con 0x7f88ac1082a0 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b2e51640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac112220 con 0x7f88ac102d30 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 0x7f88ac198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48214/0 (socket says 172.21.15.16:48214) 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 -- 172.21.15.16:0/1914395317 learned_addr learned my addr 172.21.15.16:0/1914395317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 0x7f88ac198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38844/0 (socket says 172.21.15.16:38844) 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b13ee640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 0x7f88ac19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 -- 172.21.15.16:0/1914395317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 msgr2=0x7f88ac19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 0x7f88ac19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 -- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 msgr2=0x7f88ac198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 0x7f88ac198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.449+0000 7f88b0bed640 1 -- 172.21.15.16:0/1914395317 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac19d590 con 0x7f88ac102d30 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88b0bed640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009230 tx=0x7f88a0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88a9ffb640 1 -- 172.21.15.16:0/1914395317 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a003f030 con 0x7f88ac102d30 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac19d820 con 0x7f88ac102d30 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88a9ffb640 1 -- 172.21.15.16:0/1914395317 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a0004440 con 0x7f88ac102d30 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88a9ffb640 1 -- 172.21.15.16:0/1914395317 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0035070 con 0x7f88ac102d30 2024-09-12T03:12:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.450+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac1a5160 con 0x7f88ac102d30 2024-09-12T03:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 Processor -- start 2024-09-12T03:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88a9ffb640 1 -- 172.21.15.16:0/1914395317 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f88a0042030 con 0x7f88ac102d30 2024-09-12T03:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 -- start start 2024-09-12T03:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad5107140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad5109a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ad50622e0 con 0x564ad5107680 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ad5009ce0 con 0x564ad50fcdc0 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b522a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564ad505e2a0 con 0x564ad50fc4a0 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b0bed640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad5107140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:48228/0 (socket says 172.21.15.16:48228) 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b13ee640 1 -- 172.21.15.16:0/412515182 learned_addr learned my addr 172.21.15.16:0/412515182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88abfff640 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad5109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.453+0000 7f88b13ee640 1 -- 172.21.15.16:0/412515182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 msgr2=0x564ad5107140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b13ee640 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad5107140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b13ee640 1 -- 172.21.15.16:0/412515182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 msgr2=0x564ad5109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b13ee640 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad5109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b13ee640 1 -- 172.21.15.16:0/412515182 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad4fee440 con 0x564ad50fc4a0 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88a9ffb640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 0x7f889c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b13ee640 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009f50 tx=0x7f88a40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ee31ef226ac9a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88b0bed640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 0x7f889c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f88a9ffb640 1 -- 172.21.15.16:0/1914395317 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f88a00ef3e0 con 0x7f88ac102d30 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f8896ffd640 1 -- 172.21.15.16:0/412515182 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4004d20 con 0x564ad50fc4a0 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f8896ffd640 1 -- 172.21.15.16:0/412515182 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a4004050 con 0x564ad50fc4a0 2024-09-12T03:12:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.454+0000 7f8896ffd640 1 -- 172.21.15.16:0/412515182 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4005260 con 0x564ad50fc4a0 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 -- 172.21.15.16:0/412515182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 msgr2=0x564ad50fc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b0bed640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 0x7f889c0a5040 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88a00b80b0 tx=0x7f88a00f2eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009f50 tx=0x7f88a40118d0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 -- 172.21.15.16:0/412515182 shutdown_connections 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad5109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad5107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 --2- 172.21.15.16:0/412515182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad50fc880 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 -- 172.21.15.16:0/412515182 >> 172.21.15.16:0/412515182 conn(0x564ad50f8a30 msgr2=0x564ad50f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 -- 172.21.15.16:0/412515182 shutdown_connections 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.455+0000 7f88b522a740 1 -- 172.21.15.16:0/412515182 wait complete. 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 -- start start 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 Processor -- start 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fcdc0 0x564ad51041f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad5107680 0x564ad5104730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ad510a250 con 0x564ad50fc4a0 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ad501d050 con 0x564ad5107680 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564ad510a3d0 con 0x564ad50fcdc0 2024-09-12T03:12:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b522a740 10 client.0 _ll_register_callbacks cb 0x564ad505d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38870/0 (socket says 172.21.15.16:38870) 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b13ee640 1 -- 172.21.15.16:0/396459449 learned_addr learned my addr 172.21.15.16:0/396459449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88b13ee640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad5107680 0x564ad5104730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.456+0000 7f88abfff640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fcdc0 0x564ad51041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 -- 172.21.15.16:0/396459449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fcdc0 msgr2=0x564ad51041f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fcdc0 0x564ad51041f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 -- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad5107680 msgr2=0x564ad5104730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad5107680 0x564ad5104730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 -- 172.21.15.16:0/396459449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad4dd33c0 con 0x564ad50fc4a0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b13ee640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f88a40049a0 tx=0x7f88a40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f28fa19f9ecdb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f888bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a402c030 con 0x564ad50fc4a0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88a4004050 con 0x564ad50fc4a0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.457+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f88a4025670 con 0x564ad50fc4a0 2024-09-12T03:12:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f88a4025870 con 0x564ad50fc4a0 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 10 client.21078 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564ad5115ba0 con 0x564ad50fc4a0 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a4004050 con 0x564ad50fc4a0 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f88a40289c0 con 0x564ad50fc4a0 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 10 client.21078 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 10 client.21078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 20 client.21078 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad5116fb0 0x564ad51193a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 4 client.21078 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.458+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ad506f850 con 0x564ad5116fb0 2024-09-12T03:12:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.459+0000 7f88abfff640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad5116fb0 0x564ad51193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.459+0000 7f88abfff640 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad5116fb0 0x564ad51193a0 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=385711a1d731bbeb server_cookie=856417f6c734651f in_seq=0 out_seq=0 2024-09-12T03:12:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.459+0000 7f888bfff640 10 client.21078 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f888bfff640 1 -- 172.21.15.16:0/396459449 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ad506f850 con 0x564ad5116fb0 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f888bfff640 10 client.21078 handle_command_reply: tid=0 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 client.21078 shutdown 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 msgr2=0x564ad5106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f88a40049a0 tx=0x7f88a40045a0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 shutdown_connections 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad5116fb0 0x564ad51193a0 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad5107680 0x564ad5104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fcdc0 0x564ad51041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 --2- 172.21.15.16:0/396459449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad5106f40 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f888bfff640 0 client.21078 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.460+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 >> 172.21.15.16:0/396459449 conn(0x564ad50f8a30 msgr2=0x564ad50fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.461+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 shutdown_connections 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.461+0000 7f88b522a740 1 -- 172.21.15.16:0/396459449 wait complete. 2024-09-12T03:12:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.461+0000 7f88b522a740 20 client.21078 trim_cache size 0 max 16384 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.486+0000 7f88b522a740 1 Processor -- start 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 -- start start 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad51031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad5107680 0x564ad511f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1ad680 con 0x564ad50fc4a0 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1af1c0 con 0x564ad50fcdc0 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b522a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1add50 con 0x564ad5107680 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad5107680 0x564ad511f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.487+0000 7f88abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.488+0000 7f88abfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40434/0 (socket says 172.21.15.16:40434) 2024-09-12T03:12:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.488+0000 7f88abfff640 1 -- 172.21.15.16:0/3027249907 learned_addr learned my addr 172.21.15.16:0/3027249907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.491+0000 7f88abfff640 1 -- 172.21.15.16:0/3027249907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad5107680 msgr2=0x564ad511f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.491+0000 7f88abfff640 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad5107680 0x564ad511f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.491+0000 7f88abfff640 1 -- 172.21.15.16:0/3027249907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 msgr2=0x564ad51031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.491+0000 7f88abfff640 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad51031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.491+0000 7f88abfff640 1 -- 172.21.15.16:0/3027249907 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad50fcbf0 con 0x564ad50fcdc0 2024-09-12T03:12:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.502+0000 7f88abfff640 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8898000c00 tx=0x7f8898005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25108abe2dc7fe3d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.503+0000 7f88967fc640 1 -- 172.21.15.16:0/3027249907 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898005220 con 0x564ad50fcdc0 2024-09-12T03:12:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.503+0000 7f88967fc640 1 -- 172.21.15.16:0/3027249907 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8898004c80 con 0x564ad50fcdc0 2024-09-12T03:12:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.503+0000 7f88967fc640 1 -- 172.21.15.16:0/3027249907 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88980279a0 con 0x564ad50fcdc0 2024-09-12T03:12:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.504+0000 7f88b522a740 1 -- 172.21.15.16:0/3027249907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 msgr2=0x564ad511eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.504+0000 7f88b522a740 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8898000c00 tx=0x7f8898005f90 comp rx=0 tx=0).stop 2024-09-12T03:12:25.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.505+0000 7f88b522a740 1 -- 172.21.15.16:0/3027249907 shutdown_connections 2024-09-12T03:12:25.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.505+0000 7f88b522a740 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad5107680 0x564ad511f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.505+0000 7f88b522a740 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad50fcdc0 0x564ad511eb10 secure :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8898000c00 tx=0x7f8898005f90 comp rx=0 tx=0).stop 2024-09-12T03:12:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.505+0000 7f88b522a740 1 --2- 172.21.15.16:0/3027249907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad50fc4a0 0x564ad51031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.505+0000 7f88b522a740 1 -- 172.21.15.16:0/3027249907 >> 172.21.15.16:0/3027249907 conn(0x564ad50f8a30 msgr2=0x564ad50f7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.506+0000 7f88b522a740 1 -- 172.21.15.16:0/3027249907 shutdown_connections 2024-09-12T03:12:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.506+0000 7f88b522a740 1 -- 172.21.15.16:0/3027249907 wait complete. 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.507+0000 7f88b522a740 1 -- start start 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.507+0000 7f88b522a740 1 Processor -- start 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.507+0000 7f88b522a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad510d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.507+0000 7f88b522a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad510da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.507+0000 7f88b522a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b522a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1aeaf0 con 0x564ad5107680 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b522a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1ae420 con 0x564ad510df70 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b522a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1ab660 con 0x564ad50fc4a0 2024-09-12T03:12:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b522a740 10 client.0 _ll_register_callbacks cb 0x7f8898009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b13ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:40442/0 (socket says 172.21.15.16:40442) 2024-09-12T03:12:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b13ee640 1 -- 172.21.15.16:0/3935067795 learned_addr learned my addr 172.21.15.16:0/3935067795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.508+0000 7f88b13ee640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad510d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88abfff640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad510da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 -- 172.21.15.16:0/3935067795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 msgr2=0x564ad510d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad510d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 -- 172.21.15.16:0/3935067795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 msgr2=0x564ad510da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad510da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 -- 172.21.15.16:0/3935067795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad517f0d0 con 0x564ad510df70 2024-09-12T03:12:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f88b13ee640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f88a4011ed0 tx=0x7f88a40049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c80e53ab2b43e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f888bfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:12:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.510+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a402c070 con 0x564ad510df70 2024-09-12T03:12:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.511+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f88a4025c10 con 0x564ad510df70 2024-09-12T03:12:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.512+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8898004c80 con 0x564ad510df70 2024-09-12T03:12:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.512+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f88a4025e30 con 0x564ad510df70 2024-09-12T03:12:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.514+0000 7f88b522a740 10 client.30230 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.514+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564ad51933e0 con 0x564ad510df70 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.516+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8898004c80 con 0x564ad510df70 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f88a4040680 con 0x564ad510df70 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 10 client.30230 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 10 client.30230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 20 client.30230 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad51939a0 0x564ad5195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 4 client.30230 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.517+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564ad5069d60 con 0x564ad51939a0 2024-09-12T03:12:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.519+0000 7f88abfff640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad51939a0 0x564ad5195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.523+0000 7f88abfff640 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad51939a0 0x564ad5195dc0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c887a64b63279d server_cookie=cfe2df38afd1d58f in_seq=0 out_seq=0 2024-09-12T03:12:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.524+0000 7f888bfff640 10 client.30230 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.524+0000 7f888bfff640 1 -- 172.21.15.16:0/3935067795 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564ad5069d60 con 0x564ad51939a0 2024-09-12T03:12:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.524+0000 7f888bfff640 10 client.30230 handle_command_reply: tid=0 2024-09-12T03:12:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.524+0000 7f88b522a740 1 client.30230 shutdown 2024-09-12T03:12:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.528+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 msgr2=0x564ad517eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.528+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f88a4011ed0 tx=0x7f88a40049d0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 shutdown_connections 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x564ad51939a0 0x564ad5195dc0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ad510df70 0x564ad517eae0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ad5107680 0x564ad510da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 --2- 172.21.15.16:0/3935067795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564ad50fc4a0 0x564ad510d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 >> 172.21.15.16:0/3935067795 conn(0x564ad50f8a30 msgr2=0x564ad51031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.530+0000 7f888bfff640 0 client.30230 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.534+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 shutdown_connections 2024-09-12T03:12:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.536+0000 7f88b522a740 1 -- 172.21.15.16:0/3935067795 wait complete. 2024-09-12T03:12:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.536+0000 7f88b522a740 20 client.30230 trim_cache size 0 max 16384 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "d2ac9f91-4e2b-464f-bca3-8aaddf5ab237": { 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d2ac9f91-4e2b-464f-bca3-8aaddf5ab237", 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:25.539 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:12:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.537+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 msgr2=0x7f889c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.537+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 0x7f889c0a5040 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88a00b80b0 tx=0x7f88a00f2eb0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.538+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 msgr2=0x7f88ac1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.538+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009230 tx=0x7f88a0004040 comp rx=0 tx=0).stop 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.539+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 shutdown_connections 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.540+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac1082a0 0x7f88ac19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.540+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f889c0a2b80 0x7f889c0a5040 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.540+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ac1036e0 0x7f88ac198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.540+0000 7f88b2e51640 1 --2- 172.21.15.16:0/1914395317 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88ac102d30 0x7f88ac1985e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.540+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 >> 172.21.15.16:0/1914395317 conn(0x7f88ac0fc7d0 msgr2=0x7f88ac105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.542+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 shutdown_connections 2024-09-12T03:12:25.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:25.544+0000 7f88b2e51640 1 -- 172.21.15.16:0/1914395317 wait complete. 2024-09-12T03:12:25.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:25.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:12:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:25 smithi016 ceph-mon[30925]: pgmap v1072: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.1 MiB/s wr, 606 op/s 2024-09-12T03:12:25.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:25.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:25 smithi033 ceph-mon[35092]: pgmap v1072: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.1 MiB/s wr, 606 op/s 2024-09-12T03:12:26.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 288127 23.58 MB/sec execute 567 sec latency 310.917 ms 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:12:26.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:26.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:12:26.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:25 smithi184 ceph-mon[35321]: pgmap v1072: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.1 MiB/s wr, 606 op/s 2024-09-12T03:12:26.524 INFO:tasks.cephfs.filesystem:scrub status for tag:d2ac9f91-4e2b-464f-bca3-8aaddf5ab237 - {'path': '/', 'tag': 'd2ac9f91-4e2b-464f-bca3-8aaddf5ab237', 'options': 'recursive,force'} 2024-09-12T03:12:27.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 288600 23.57 MB/sec execute 568 sec latency 119.671 ms 2024-09-12T03:12:27.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:27 smithi016 ceph-mon[30925]: pgmap v1073: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 624 op/s 2024-09-12T03:12:27.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:27 smithi033 ceph-mon[35092]: pgmap v1073: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 624 op/s 2024-09-12T03:12:28.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 289225 23.58 MB/sec execute 569 sec latency 74.335 ms 2024-09-12T03:12:28.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:27 smithi184 ceph-mon[35321]: pgmap v1073: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 624 op/s 2024-09-12T03:12:29.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 289883 23.60 MB/sec execute 570 sec latency 65.789 ms 2024-09-12T03:12:29.846 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:29 smithi184 ceph-mon[35321]: pgmap v1074: 97 pgs: 97 active+clean; 941 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 564 op/s 2024-09-12T03:12:29.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:29 smithi016 ceph-mon[30925]: pgmap v1074: 97 pgs: 97 active+clean; 941 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 564 op/s 2024-09-12T03:12:29.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:29 smithi033 ceph-mon[35092]: pgmap v1074: 97 pgs: 97 active+clean; 941 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 564 op/s 2024-09-12T03:12:30.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 290507 23.62 MB/sec execute 571 sec latency 78.859 ms 2024-09-12T03:12:31.042 INFO:tasks.workunit.client.0.smithi016.stdout: 10 290996 23.61 MB/sec execute 572 sec latency 386.998 ms 2024-09-12T03:12:31.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:31 smithi016 ceph-mon[30925]: pgmap v1075: 97 pgs: 97 active+clean; 922 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.9 MiB/s wr, 687 op/s 2024-09-12T03:12:31.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:31 smithi033 ceph-mon[35092]: pgmap v1075: 97 pgs: 97 active+clean; 922 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.9 MiB/s wr, 687 op/s 2024-09-12T03:12:32.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 291622 23.65 MB/sec execute 573 sec latency 78.262 ms 2024-09-12T03:12:32.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:31 smithi184 ceph-mon[35321]: pgmap v1075: 97 pgs: 97 active+clean; 922 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.9 MiB/s wr, 687 op/s 2024-09-12T03:12:33.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 292372 23.68 MB/sec execute 574 sec latency 40.622 ms 2024-09-12T03:12:33.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:33 smithi184 ceph-mon[35321]: pgmap v1076: 97 pgs: 97 active+clean; 929 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.6 MiB/s wr, 636 op/s 2024-09-12T03:12:33.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:33 smithi016 ceph-mon[30925]: pgmap v1076: 97 pgs: 97 active+clean; 929 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.6 MiB/s wr, 636 op/s 2024-09-12T03:12:33.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:33 smithi033 ceph-mon[35092]: pgmap v1076: 97 pgs: 97 active+clean; 929 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.6 MiB/s wr, 636 op/s 2024-09-12T03:12:34.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 292833 23.69 MB/sec execute 575 sec latency 127.542 ms 2024-09-12T03:12:35.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 293360 23.69 MB/sec execute 576 sec latency 55.011 ms 2024-09-12T03:12:35.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:35 smithi016 ceph-mon[30925]: pgmap v1077: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.7 MiB/s wr, 627 op/s 2024-09-12T03:12:35.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:35 smithi033 ceph-mon[35092]: pgmap v1077: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.7 MiB/s wr, 627 op/s 2024-09-12T03:12:36.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 293836 23.68 MB/sec execute 577 sec latency 273.893 ms 2024-09-12T03:12:36.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:35 smithi184 ceph-mon[35321]: pgmap v1077: 97 pgs: 97 active+clean; 932 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.7 MiB/s wr, 627 op/s 2024-09-12T03:12:37.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 294542 23.70 MB/sec execute 578 sec latency 84.971 ms 2024-09-12T03:12:37.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:37 smithi016 ceph-mon[30925]: pgmap v1078: 97 pgs: 97 active+clean; 936 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.9 MiB/s wr, 609 op/s 2024-09-12T03:12:37.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:37 smithi033 ceph-mon[35092]: pgmap v1078: 97 pgs: 97 active+clean; 936 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.9 MiB/s wr, 609 op/s 2024-09-12T03:12:38.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 295372 23.73 MB/sec execute 579 sec latency 81.657 ms 2024-09-12T03:12:38.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:37 smithi184 ceph-mon[35321]: pgmap v1078: 97 pgs: 97 active+clean; 936 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.9 MiB/s wr, 609 op/s 2024-09-12T03:12:38.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:12:39.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:12:39.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:12:39.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:38 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:12:39.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:12:39.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:12:39.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:12:39.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:38 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:12:39.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 296028 23.78 MB/sec execute 580 sec latency 59.091 ms 2024-09-12T03:12:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:12:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:12:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:12:39.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:38 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:12:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:39 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:12:39.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:39 smithi016 ceph-mon[30925]: pgmap v1079: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 614 op/s 2024-09-12T03:12:39.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:39 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:12:39.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:39 smithi033 ceph-mon[35092]: pgmap v1079: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 614 op/s 2024-09-12T03:12:40.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 296675 23.79 MB/sec execute 581 sec latency 42.394 ms 2024-09-12T03:12:40.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:39 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:12:40.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:39 smithi184 ceph-mon[35321]: pgmap v1079: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 614 op/s 2024-09-12T03:12:41.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 297213 23.80 MB/sec execute 582 sec latency 215.517 ms 2024-09-12T03:12:41.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:41 smithi016 ceph-mon[30925]: pgmap v1080: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.4 MiB/s wr, 662 op/s 2024-09-12T03:12:41.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:41 smithi033 ceph-mon[35092]: pgmap v1080: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.4 MiB/s wr, 662 op/s 2024-09-12T03:12:42.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 297624 23.80 MB/sec execute 583 sec latency 266.086 ms 2024-09-12T03:12:42.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:41 smithi184 ceph-mon[35321]: pgmap v1080: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.4 MiB/s wr, 662 op/s 2024-09-12T03:12:43.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 298216 23.81 MB/sec execute 584 sec latency 75.378 ms 2024-09-12T03:12:43.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:43 smithi016 ceph-mon[30925]: pgmap v1081: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.2 MiB/s wr, 631 op/s 2024-09-12T03:12:43.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:43 smithi033 ceph-mon[35092]: pgmap v1081: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.2 MiB/s wr, 631 op/s 2024-09-12T03:12:44.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 298950 23.83 MB/sec execute 585 sec latency 63.966 ms 2024-09-12T03:12:44.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:43 smithi184 ceph-mon[35321]: pgmap v1081: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.2 MiB/s wr, 631 op/s 2024-09-12T03:12:45.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 299766 23.86 MB/sec execute 586 sec latency 57.981 ms 2024-09-12T03:12:45.364 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 33.7 2024-09-12T03:12:45.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:12:45.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:45 smithi016 ceph-mon[30925]: pgmap v1082: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 573 op/s 2024-09-12T03:12:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:45.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:45 smithi033 ceph-mon[35092]: pgmap v1082: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 573 op/s 2024-09-12T03:12:46.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 300380 23.89 MB/sec execute 587 sec latency 99.074 ms 2024-09-12T03:12:46.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:45 smithi184 ceph-mon[35321]: pgmap v1082: 97 pgs: 97 active+clean; 950 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 573 op/s 2024-09-12T03:12:46.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 -- 172.21.15.16:0/3832418607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4074590 msgr2=0x7fb1c4074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 --2- 172.21.15.16:0/3832418607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4074590 0x7fb1c4074970 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009860 tx=0x7fb1b802a3d0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 -- 172.21.15.16:0/3832418607 shutdown_connections 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 --2- 172.21.15.16:0/3832418607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4112b90 0x7fb1c4114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 --2- 172.21.15.16:0/3832418607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074eb0 0x7fb1c4112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 --2- 172.21.15.16:0/3832418607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4074590 0x7fb1c4074970 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.299+0000 7fb1caefc640 1 -- 172.21.15.16:0/3832418607 >> 172.21.15.16:0/3832418607 conn(0x7fb1c40fd8c0 msgr2=0x7fb1c40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.300+0000 7fb1caefc640 1 -- 172.21.15.16:0/3832418607 shutdown_connections 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.300+0000 7fb1caefc640 1 -- 172.21.15.16:0/3832418607 wait complete. 2024-09-12T03:12:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.300+0000 7fb1caefc640 1 Processor -- start 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.300+0000 7fb1caefc640 1 -- start start 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4074eb0 0x7fb1c41a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 0x7fb1c419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4118550 con 0x7fb1c4112b90 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41183d0 con 0x7fb1c4074590 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1caefc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41186d0 con 0x7fb1c4074eb0 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c8c98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c8c98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47176/0 (socket says 172.21.15.16:47176) 2024-09-12T03:12:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c9499640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 0x7fb1c419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c8c98640 1 -- 172.21.15.16:0/132412679 learned_addr learned my addr 172.21.15.16:0/132412679 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c3fff640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4074eb0 0x7fb1c41a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c9499640 1 -- 172.21.15.16:0/132412679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4074eb0 msgr2=0x7fb1c41a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.301+0000 7fb1c9499640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4074eb0 0x7fb1c41a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c9499640 1 -- 172.21.15.16:0/132412679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 msgr2=0x7fb1c41a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c9499640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c9499640 1 -- 172.21.15.16:0/132412679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c419f030 con 0x7fb1c4112b90 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c8c98640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c9499640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 0x7fb1c419e900 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005fd0 tx=0x7fb1bc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc00cdf0 con 0x7fb1c4112b90 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c419f320 con 0x7fb1c4112b90 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc010430 con 0x7fb1c4112b90 2024-09-12T03:12:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc00f620 con 0x7fb1c4112b90 2024-09-12T03:12:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.302+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c41ab610 con 0x7fb1c4112b90 2024-09-12T03:12:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.303+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c4074970 con 0x7fb1c4112b90 2024-09-12T03:12:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.305+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb1bc0108e0 con 0x7fb1c4112b90 2024-09-12T03:12:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.306+0000 7fb1c1ffb640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 0x7fb1b40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.306+0000 7fb1c8c98640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 0x7fb1b40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.306+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb1bc0cf3d0 con 0x7fb1c4112b90 2024-09-12T03:12:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.307+0000 7fb1c8c98640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 0x7fb1b40a5040 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80095f0 tx=0x7fb1b80058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.313+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb1bc09c340 con 0x7fb1c4112b90 2024-09-12T03:12:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.590+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1c419fe30 con 0x7fb1c4112b90 2024-09-12T03:12:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.592+0000 7fb1c1ffb640 1 -- 172.21.15.16:0/132412679 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fb1c419fe30 con 0x7fb1c4112b90 2024-09-12T03:12:46.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:12:46.593 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:12:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 msgr2=0x7fb1b40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 0x7fb1b40a5040 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80095f0 tx=0x7fb1b80058b0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 msgr2=0x7fb1c419e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 0x7fb1c419e900 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005fd0 tx=0x7fb1bc00ef10 comp rx=0 tx=0).stop 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 shutdown_connections 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb1b40a2b80 0x7fb1b40a5040 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c4112b90 0x7fb1c419e900 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c4074eb0 0x7fb1c41a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 --2- 172.21.15.16:0/132412679 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb1c4074590 0x7fb1c41a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 >> 172.21.15.16:0/132412679 conn(0x7fb1c40fd8c0 msgr2=0x7fb1c40ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 shutdown_connections 2024-09-12T03:12:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:46.595+0000 7fb1caefc640 1 -- 172.21.15.16:0/132412679 wait complete. 2024-09-12T03:12:47.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 300772 23.89 MB/sec execute 588 sec latency 236.971 ms 2024-09-12T03:12:47.576 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 90 --expiration 153.7 --await-for 45.0' 2024-09-12T03:12:47.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db / --timeout 90 --expiration 153.7 --await-for 45.0 2024-09-12T03:12:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:47.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:47 smithi016 ceph-mon[30925]: pgmap v1083: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:12:47.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:47 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/132412679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:47.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:47 smithi033 ceph-mon[35092]: pgmap v1083: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:12:47.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:47 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/132412679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:48.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301397 23.91 MB/sec execute 589 sec latency 51.339 ms 2024-09-12T03:12:48.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:47 smithi184 ceph-mon[35321]: pgmap v1083: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.4 MiB/s wr, 667 op/s 2024-09-12T03:12:48.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:47 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/132412679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 -- 172.21.15.16:0/3512363463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 msgr2=0x7fcbc4100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3512363463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc4100ca0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009900 tx=0x7fcbc002a410 comp rx=0 tx=0).stop 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 -- 172.21.15.16:0/3512363463 shutdown_connections 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3512363463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc4105ec0 0x7fcbc4106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3512363463 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc41011e0 0x7fcbc4105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3512363463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc4100ca0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 -- 172.21.15.16:0/3512363463 >> 172.21.15.16:0/3512363463 conn(0x7fcbc40fc7d0 msgr2=0x7fcbc40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 -- 172.21.15.16:0/3512363463 shutdown_connections 2024-09-12T03:12:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.416+0000 7fcbcb486640 1 -- 172.21.15.16:0/3512363463 wait complete. 2024-09-12T03:12:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.417+0000 7fcbcb486640 1 Processor -- start 2024-09-12T03:12:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.417+0000 7fcbcb486640 1 -- start start 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc41011e0 0x7fcbc41a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4105ec0 0x7fcbc419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc4118630 con 0x7fcbc41008c0 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41184b0 con 0x7fcbc4105ec0 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbcb486640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41187b0 con 0x7fcbc41011e0 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41198/0 (socket says 172.21.15.16:41198) 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 -- 172.21.15.16:0/3419965937 learned_addr learned my addr 172.21.15.16:0/3419965937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:48.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc41011e0 0x7fcbc41a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4105ec0 0x7fcbc419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 -- 172.21.15.16:0/3419965937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc41011e0 msgr2=0x7fcbc41a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc41011e0 0x7fcbc41a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 -- 172.21.15.16:0/3419965937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4105ec0 msgr2=0x7fcbc419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4105ec0 0x7fcbc419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 -- 172.21.15.16:0/3419965937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc419f070 con 0x7fcbc41008c0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.418+0000 7fcbc9222640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0006010 tx=0x7fcbc0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.419+0000 7fcbba7fc640 1 -- 172.21.15.16:0/3419965937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc003f070 con 0x7fcbc41008c0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.419+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc419f300 con 0x7fcbc41008c0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.419+0000 7fcbba7fc640 1 -- 172.21.15.16:0/3419965937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc0031c30 con 0x7fcbc41008c0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.419+0000 7fcbba7fc640 1 -- 172.21.15.16:0/3419965937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0035030 con 0x7fcbc41008c0 2024-09-12T03:12:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.419+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc41ab650 con 0x7fcbc41008c0 2024-09-12T03:12:48.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.421+0000 7fcbba7fc640 1 -- 172.21.15.16:0/3419965937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcbc0042070 con 0x7fcbc41008c0 2024-09-12T03:12:48.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.421+0000 7fcbcd85f740 1 Processor -- start 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 -- start start 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e85fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 0x56259e86a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56259e7c52a0 con 0x56259e86a940 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56259e76cb10 con 0x56259e860080 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56259e7c1260 con 0x56259e85f760 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbba7fc640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 0x7fcbb00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 0x56259e86a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41200/0 (socket says 172.21.15.16:41200) 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc9a23640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e85fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 0x56259e86a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:47202/0 (socket says 172.21.15.16:47202) 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 -- 172.21.15.16:0/1149343594 learned_addr learned my addr 172.21.15.16:0/1149343594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 0x7fcbb00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 -- 172.21.15.16:0/1149343594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 msgr2=0x56259e85fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e85fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 -- 172.21.15.16:0/1149343594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 msgr2=0x56259e86a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 0x56259e86a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.422+0000 7fcbc8a21640 1 -- 172.21.15.16:0/1149343594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56259e536b90 con 0x56259e86a940 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009c60 tx=0x7fcbb4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ba9b658777895b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbba7fc640 1 -- 172.21.15.16:0/3419965937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbc00ef690 con 0x7fcbc41008c0 2024-09-12T03:12:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 0x7fcbb00a5040 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc005fd0 tx=0x7fcbbc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbaf7fe640 1 -- 172.21.15.16:0/1149343594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4004230 con 0x56259e86a940 2024-09-12T03:12:48.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbaf7fe640 1 -- 172.21.15.16:0/1149343594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb40043d0 con 0x56259e86a940 2024-09-12T03:12:48.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.423+0000 7fcbaf7fe640 1 -- 172.21.15.16:0/1149343594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4004dc0 con 0x56259e86a940 2024-09-12T03:12:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1149343594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 msgr2=0x56259e86cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009c60 tx=0x7fcbb4011810 comp rx=0 tx=0).stop 2024-09-12T03:12:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1149343594 shutdown_connections 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e86a940 0x56259e86cd30 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e860080 0x56259e86a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.424+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1149343594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e85fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.425+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1149343594 >> 172.21.15.16:0/1149343594 conn(0x56259e85bcf0 msgr2=0x56259e85c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.425+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1149343594 shutdown_connections 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.425+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1149343594 wait complete. 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.425+0000 7fcbcd85f740 1 -- start start 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.425+0000 7fcbcd85f740 1 Processor -- start 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56259e86d510 con 0x56259e860080 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41216/0 (socket says 172.21.15.16:41216) 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 -- 172.21.15.16:0/4196209910 learned_addr learned my addr 172.21.15.16:0/4196209910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc9222640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42668/0 (socket says 172.21.15.16:42668) 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56259e74d6e0 con 0x56259e86a940 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56259e86d690 con 0x56259e85f760 2024-09-12T03:12:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbcd85f740 10 client.0 _ll_register_callbacks cb 0x56259e771a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 -- 172.21.15.16:0/4196209910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 msgr2=0x56259e867820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 -- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 msgr2=0x56259e8682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.426+0000 7fcbc8a21640 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56259e876a00 con 0x56259e860080 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbc9222640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40199c0 tx=0x7fcbb40199f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42e14b909ef5e13 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbacff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402a060 con 0x56259e860080 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbb4019d90 con 0x56259e860080 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb40043d0 con 0x56259e860080 2024-09-12T03:12:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcbb4025480 con 0x56259e860080 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbcd85f740 10 client.21099 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.427+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56259e879370 con 0x56259e860080 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb40043d0 con 0x56259e860080 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcbb4018070 con 0x56259e860080 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 10 client.21099 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 10 client.21099 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 20 client.21099 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e87a830 0x56259e87cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 4 client.21099 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56259e7d29d0 con 0x56259e87a830 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.428+0000 7fcbc9222640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e87a830 0x56259e87cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.429+0000 7fcbacff9640 10 client.21099 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.429+0000 7fcbc9222640 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e87a830 0x56259e87cc20 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b570ae1f1f07ff server_cookie=e757839e27a9a51b in_seq=0 out_seq=0 2024-09-12T03:12:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.430+0000 7fcbacff9640 1 -- 172.21.15.16:0/4196209910 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56259e7d29d0 con 0x56259e87a830 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.430+0000 7fcbacff9640 10 client.21099 handle_command_reply: tid=0 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.430+0000 7fcbcd85f740 1 client.21099 shutdown 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.430+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 msgr2=0x56259e867d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.430+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40199c0 tx=0x7fcbb40199f0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 shutdown_connections 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e87a830 0x56259e87cc20 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbacff9640 0 client.21099 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e867d60 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/4196209910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e867820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 >> 172.21.15.16:0/4196209910 conn(0x56259e85bcf0 msgr2=0x56259e8619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:48.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 shutdown_connections 2024-09-12T03:12:48.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 1 -- 172.21.15.16:0/4196209910 wait complete. 2024-09-12T03:12:48.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.431+0000 7fcbcd85f740 20 client.21099 trim_cache size 0 max 16384 2024-09-12T03:12:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.450+0000 7fcbcd85f740 1 Processor -- start 2024-09-12T03:12:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.450+0000 7fcbcd85f740 1 -- start start 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e862b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e882390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b3b60 con 0x56259e860080 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b56a0 con 0x56259e86a940 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b4230 con 0x56259e85f760 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e862b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41224/0 (socket says 172.21.15.16:41224) 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e862b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:42670/0 (socket says 172.21.15.16:42670) 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 -- 172.21.15.16:0/2622503368 learned_addr learned my addr 172.21.15.16:0/2622503368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e882390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 -- 172.21.15.16:0/2622503368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 msgr2=0x56259e862b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e862b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 -- 172.21.15.16:0/2622503368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 msgr2=0x56259e882390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e882390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 -- 172.21.15.16:0/2622503368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56259e8667d0 con 0x56259e860080 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.451+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4011d90 tx=0x7fcbb4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b868afb8e0acdc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbaeffd640 1 -- 172.21.15.16:0/2622503368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4035070 con 0x56259e860080 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbaeffd640 1 -- 172.21.15.16:0/2622503368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb401f040 con 0x56259e860080 2024-09-12T03:12:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbaeffd640 1 -- 172.21.15.16:0/2622503368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4025620 con 0x56259e860080 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 -- 172.21.15.16:0/2622503368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 msgr2=0x56259e8618c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4011d90 tx=0x7fcbb4004600 comp rx=0 tx=0).stop 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 -- 172.21.15.16:0/2622503368 shutdown_connections 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e882390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e860080 0x56259e8618c0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/2622503368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e85f760 0x56259e862b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.452+0000 7fcbcd85f740 1 -- 172.21.15.16:0/2622503368 >> 172.21.15.16:0/2622503368 conn(0x56259e85bcf0 msgr2=0x56259e8611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.453+0000 7fcbcd85f740 1 -- 172.21.15.16:0/2622503368 shutdown_connections 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.453+0000 7fcbcd85f740 1 -- 172.21.15.16:0/2622503368 wait complete. 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.453+0000 7fcbcd85f740 1 -- start start 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.453+0000 7fcbcd85f740 1 Processor -- start 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e860080 0x56259e8731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8643d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b4fd0 con 0x56259e85f760 2024-09-12T03:12:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b4900 con 0x56259e86a940 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc41b1b40 con 0x56259e860080 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbcd85f740 10 client.0 _ll_register_callbacks cb 0x7fcbb4009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41232/0 (socket says 172.21.15.16:41232) 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 -- 172.21.15.16:0/1625055176 learned_addr learned my addr 172.21.15.16:0/1625055176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9a23640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e860080 0x56259e8731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 -- 172.21.15.16:0/1625055176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e860080 msgr2=0x56259e8731b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e860080 0x56259e8731b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 -- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 msgr2=0x56259e8643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8643d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 -- 172.21.15.16:0/1625055176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56259e864c00 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbc9222640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00dc170 tx=0x7fcbc0031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4248e76c7b4cdf server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.454+0000 7fcbacff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0049040 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fcbc003b460 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb401f040 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcbc00c4b70 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 10 client.21105 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56259e8e5000 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc00d7020 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fcbc00c46e0 con 0x56259e85f760 2024-09-12T03:12:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 10 client.21105 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 10 client.21105 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 20 client.21105 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e8e5740 0x56259e8e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 4 client.21105 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 153.7, "await_for": 45.0}] 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.455+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 153.7, "await_for": 45.0}) -- 0x56259e7ccd20 con 0x56259e8e5740 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.456+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e8e5740 0x56259e8e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.456+0000 7fcbc8a21640 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e8e5740 0x56259e8e7b30 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=588cb2f81842051d server_cookie=6516929e458eeb9b in_seq=0 out_seq=0 2024-09-12T03:12:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:48.456+0000 7fcbacff9640 10 client.21105 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:49.043 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301624 23.89 MB/sec execute 590 sec latency 530.261 ms 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.559+0000 7fcbacff9640 1 -- 172.21.15.16:0/1625055176 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56259e7ccd20 con 0x56259e8e5740 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.559+0000 7fcbacff9640 10 client.21105 handle_command_reply: tid=0 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.559+0000 7fcbcd85f740 1 client.21105 shutdown 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.559+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 msgr2=0x56259e872c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.559+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00dc170 tx=0x7fcbc0031ea0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 shutdown_connections 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56259e8e5740 0x56259e8e7b30 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56259e86a940 0x56259e8643d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56259e860080 0x56259e8731b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 --2- 172.21.15.16:0/1625055176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56259e85f760 0x56259e872c70 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 >> 172.21.15.16:0/1625055176 conn(0x56259e85bcf0 msgr2=0x56259e85ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbacff9640 0 client.21105 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 shutdown_connections 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 1 -- 172.21.15.16:0/1625055176 wait complete. 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcd85f740 20 client.21105 trim_cache size 0 max 16384 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:12:49.562 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 86, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "d4038f41": { 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 86, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 153.7, 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:12:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:12:49.564 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:12:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 msgr2=0x7fcbb00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 0x7fcbb00a5040 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc005fd0 tx=0x7fcbbc009500 comp rx=0 tx=0).stop 2024-09-12T03:12:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 msgr2=0x7fcbc41a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.560+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0006010 tx=0x7fcbc0003c10 comp rx=0 tx=0).stop 2024-09-12T03:12:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 shutdown_connections 2024-09-12T03:12:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4105ec0 0x7fcbc419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbc41011e0 0x7fcbc41a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fcbb00a2b80 0x7fcbb00a5040 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 --2- 172.21.15.16:0/3419965937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc41008c0 0x7fcbc41a4780 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 >> 172.21.15.16:0/3419965937 conn(0x7fcbc40fc7d0 msgr2=0x7fcbc4103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 shutdown_connections 2024-09-12T03:12:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:49.561+0000 7fcbcb486640 1 -- 172.21.15.16:0/3419965937 wait complete. 2024-09-12T03:12:49.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:49 smithi016 ceph-mon[30925]: pgmap v1084: 97 pgs: 97 active+clean; 969 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 612 op/s 2024-09-12T03:12:49.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:49 smithi033 ceph-mon[35092]: pgmap v1084: 97 pgs: 97 active+clean; 969 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 612 op/s 2024-09-12T03:12:50.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.85 MB/sec execute 591 sec latency 1521.696 ms 2024-09-12T03:12:50.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:49 smithi184 ceph-mon[35321]: pgmap v1084: 97 pgs: 97 active+clean; 969 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 612 op/s 2024-09-12T03:12:50.465 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd4038f41', quiesce took 5.1 seconds. Will release after: 28.6 2024-09-12T03:12:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.81 MB/sec execute 592 sec latency 2521.749 ms 2024-09-12T03:12:51.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:51 smithi016 ceph-mon[30925]: pgmap v1085: 97 pgs: 97 active+clean; 963 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 10 MiB/s wr, 662 op/s 2024-09-12T03:12:51.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:51 smithi033 ceph-mon[35092]: pgmap v1085: 97 pgs: 97 active+clean; 963 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 10 MiB/s wr, 662 op/s 2024-09-12T03:12:52.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.77 MB/sec execute 593 sec latency 3521.804 ms 2024-09-12T03:12:52.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:51 smithi184 ceph-mon[35321]: pgmap v1085: 97 pgs: 97 active+clean; 963 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 10 MiB/s wr, 662 op/s 2024-09-12T03:12:53.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.73 MB/sec execute 594 sec latency 4521.837 ms 2024-09-12T03:12:53.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:53 smithi016 ceph-mon[30925]: pgmap v1086: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.5 MiB/s wr, 592 op/s 2024-09-12T03:12:53.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:53 smithi033 ceph-mon[35092]: pgmap v1086: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.5 MiB/s wr, 592 op/s 2024-09-12T03:12:54.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.69 MB/sec execute 595 sec latency 5521.945 ms 2024-09-12T03:12:54.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:53 smithi184 ceph-mon[35321]: pgmap v1086: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.5 MiB/s wr, 592 op/s 2024-09-12T03:12:55.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.65 MB/sec execute 596 sec latency 6522.004 ms 2024-09-12T03:12:55.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:55 smithi016 ceph-mon[30925]: pgmap v1087: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.3 MiB/s wr, 493 op/s 2024-09-12T03:12:55.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:55 smithi033 ceph-mon[35092]: pgmap v1087: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.3 MiB/s wr, 493 op/s 2024-09-12T03:12:56.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.61 MB/sec execute 597 sec latency 7522.110 ms 2024-09-12T03:12:56.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:55 smithi184 ceph-mon[35321]: pgmap v1087: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.3 MiB/s wr, 493 op/s 2024-09-12T03:12:56.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:12:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:57.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.57 MB/sec execute 598 sec latency 8522.174 ms 2024-09-12T03:12:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 -- 172.21.15.16:0/1755299006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074f40 msgr2=0x7f4b7010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1755299006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074f40 0x7f4b7010de80 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009900 tx=0x7f4b6402a480 comp rx=0 tx=0).stop 2024-09-12T03:12:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 -- 172.21.15.16:0/1755299006 shutdown_connections 2024-09-12T03:12:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1755299006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b7010e3c0 0x7f4b70110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1755299006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074f40 0x7f4b7010de80 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1755299006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b70074620 0x7f4b70074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.228+0000 7f4b775aa640 1 -- 172.21.15.16:0/1755299006 >> 172.21.15.16:0/1755299006 conn(0x7f4b700fd8c0 msgr2=0x7f4b700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.229+0000 7f4b775aa640 1 -- 172.21.15.16:0/1755299006 shutdown_connections 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.229+0000 7f4b775aa640 1 -- 172.21.15.16:0/1755299006 wait complete. 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.229+0000 7f4b775aa640 1 Processor -- start 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.229+0000 7f4b775aa640 1 -- start start 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b70074f40 0x7f4b7019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b7010e3c0 0x7f4b7019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70114160 con 0x7f4b70074620 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70113fe0 con 0x7f4b7010e3c0 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b775aa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701142e0 con 0x7f4b70074f40 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47770/0 (socket says 172.21.15.16:47770) 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 -- 172.21.15.16:0/1525648833 learned_addr learned my addr 172.21.15.16:0/1525648833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b74b45640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b70074f40 0x7f4b7019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b7010e3c0 0x7f4b7019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 -- 172.21.15.16:0/1525648833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b70074f40 msgr2=0x7f4b7019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b70074f40 0x7f4b7019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 -- 172.21.15.16:0/1525648833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b7010e3c0 msgr2=0x7f4b7019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b7010e3c0 0x7f4b7019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 -- 172.21.15.16:0/1525648833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7019e6b0 con 0x7f4b70074620 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.230+0000 7f4b75346640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c890 tx=0x7f4b6000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.231+0000 7f4b6e7fc640 1 -- 172.21.15.16:0/1525648833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60004220 con 0x7f4b70074620 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.231+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7019e9a0 con 0x7f4b70074620 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.231+0000 7f4b6e7fc640 1 -- 172.21.15.16:0/1525648833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b600043c0 con 0x7f4b70074620 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.231+0000 7f4b6e7fc640 1 -- 172.21.15.16:0/1525648833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60004a00 con 0x7f4b70074620 2024-09-12T03:12:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.231+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b701a7310 con 0x7f4b70074620 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.233+0000 7f4b6e7fc640 1 -- 172.21.15.16:0/1525648833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4b600079b0 con 0x7f4b70074620 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 Processor -- start 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 -- start start 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e414a0 0x55a9c2e41880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e4c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2e4ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2da72e0 con 0x55a9c2e41dc0 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2d4ece0 con 0x55a9c2e414a0 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b79983740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2da32a0 con 0x55a9c2e4c680 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b6e7fc640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 0x7f4b580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e4c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e4c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47772/0 (socket says 172.21.15.16:47772) 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.234+0000 7f4b75346640 1 -- 172.21.15.16:0/2515182113 learned_addr learned my addr 172.21.15.16:0/2515182113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2e4ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b75b47640 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e414a0 0x55a9c2e41880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 0x7f4b580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b6e7fc640 1 -- 172.21.15.16:0/1525648833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b600ced90 con 0x7f4b70074620 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 -- 172.21.15.16:0/2515182113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e414a0 msgr2=0x55a9c2e41880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e414a0 0x55a9c2e41880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 -- 172.21.15.16:0/2515182113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 msgr2=0x55a9c2e4c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e4c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 -- 172.21.15.16:0/2515182113 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9c2d33440 con 0x55a9c2e4c680 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.235+0000 7f4b74b45640 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2e4ea70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4b64005940 tx=0x7f4b64031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39059a32598727e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 0x7f4b580a5040 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005c40 tx=0x7f4b68005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b577fe640 1 -- 172.21.15.16:0/2515182113 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64040070 con 0x55a9c2e4c680 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b577fe640 1 -- 172.21.15.16:0/2515182113 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b64005020 con 0x55a9c2e4c680 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b577fe640 1 -- 172.21.15.16:0/2515182113 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64035040 con 0x55a9c2e4c680 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 -- 172.21.15.16:0/2515182113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 msgr2=0x55a9c2e4ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2e4ea70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4b64005940 tx=0x7f4b64031630 comp rx=0 tx=0).stop 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 -- 172.21.15.16:0/2515182113 shutdown_connections 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2e4ea70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e4c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.236+0000 7f4b79983740 1 --2- 172.21.15.16:0/2515182113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e414a0 0x55a9c2e41880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.237+0000 7f4b79983740 1 -- 172.21.15.16:0/2515182113 >> 172.21.15.16:0/2515182113 conn(0x55a9c2e3da30 msgr2=0x55a9c2e3de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.237+0000 7f4b79983740 1 -- 172.21.15.16:0/2515182113 shutdown_connections 2024-09-12T03:12:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.237+0000 7f4b79983740 1 -- 172.21.15.16:0/2515182113 wait complete. 2024-09-12T03:12:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.237+0000 7f4b79983740 1 -- start start 2024-09-12T03:12:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.237+0000 7f4b79983740 1 Processor -- start 2024-09-12T03:12:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e491e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2e4f250 con 0x55a9c2e4c680 2024-09-12T03:12:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2d62050 con 0x55a9c2e41dc0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9c2e4f3d0 con 0x55a9c2e414a0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b79983740 10 client.0 _ll_register_callbacks cb 0x55a9c2da2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75b47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75b47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47784/0 (socket says 172.21.15.16:47784) 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:57798/0 (socket says 172.21.15.16:57798) 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75b47640 1 -- 172.21.15.16:0/3351987753 learned_addr learned my addr 172.21.15.16:0/3351987753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b74b45640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 -- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 msgr2=0x55a9c2e491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 -- 172.21.15.16:0/3351987753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 msgr2=0x55a9c2e49720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75346640 1 -- 172.21.15.16:0/3351987753 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9c2b183c0 con 0x55a9c2e414a0 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.238+0000 7f4b75b47640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b75346640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b6009b0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=435f8923dccc1a77 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6001d040 con 0x55a9c2e414a0 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b60007ca0 con 0x55a9c2e414a0 2024-09-12T03:12:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b64005020 con 0x55a9c2e414a0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4b6009bca0 con 0x55a9c2e414a0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b79983740 10 client.30214 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a9c2e5abd0 con 0x55a9c2e414a0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b64005020 con 0x55a9c2e414a0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.239+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4b600b42a0 con 0x55a9c2e414a0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 10 client.30214 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 10 client.30214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 20 client.30214 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2e5bfe0 0x55a9c2e5e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 4 client.30214 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9c2db4850 con 0x55a9c2e5bfe0 2024-09-12T03:12:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b74b45640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2e5bfe0 0x55a9c2e5e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b74b45640 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2e5bfe0 0x55a9c2e5e3f0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d619ab0705f684f server_cookie=e7c39cc597c82175 in_seq=0 out_seq=0 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.240+0000 7f4b54ff9640 10 client.30214 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b54ff9640 1 -- 172.21.15.16:0/3351987753 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a9c2db4850 con 0x55a9c2e5bfe0 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b54ff9640 10 client.30214 handle_command_reply: tid=0 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 client.30214 shutdown 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 msgr2=0x55a9c2e4bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b6009b0e0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 shutdown_connections 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2e5bfe0 0x55a9c2e5e3f0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e49720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b54ff9640 0 client.30214 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 --2- 172.21.15.16:0/3351987753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e4bf30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.241+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 >> 172.21.15.16:0/3351987753 conn(0x55a9c2e3da30 msgr2=0x55a9c2e43760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.242+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 shutdown_connections 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.242+0000 7f4b79983740 1 -- 172.21.15.16:0/3351987753 wait complete. 2024-09-12T03:12:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.242+0000 7f4b79983740 20 client.30214 trim_cache size 0 max 16384 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 Processor -- start 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 -- start start 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e63c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701af790 con 0x55a9c2e4c680 2024-09-12T03:12:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b12d0 con 0x55a9c2e41dc0 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b79983740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701afe60 con 0x55a9c2e414a0 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b75b47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b75b47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47798/0 (socket says 172.21.15.16:47798) 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.259+0000 7f4b75b47640 1 -- 172.21.15.16:0/1118615524 learned_addr learned my addr 172.21.15.16:0/1118615524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75346640 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b74b45640 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e63c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 -- 172.21.15.16:0/1118615524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 msgr2=0x55a9c2e584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 -- 172.21.15.16:0/1118615524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 msgr2=0x55a9c2e63c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e63c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 -- 172.21.15.16:0/1118615524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9c2e3d860 con 0x55a9c2e4c680 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4b68012910 tx=0x7f4b680096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60769b4913c0e94b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b56ffd640 1 -- 172.21.15.16:0/1118615524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004b20 con 0x55a9c2e4c680 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b56ffd640 1 -- 172.21.15.16:0/1118615524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68004cc0 con 0x55a9c2e4c680 2024-09-12T03:12:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.260+0000 7f4b56ffd640 1 -- 172.21.15.16:0/1118615524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68005520 con 0x55a9c2e4c680 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- 172.21.15.16:0/1118615524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 msgr2=0x55a9c2e641a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4b68012910 tx=0x7f4b680096c0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- 172.21.15.16:0/1118615524 shutdown_connections 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e4c680 0x55a9c2e641a0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e63c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 --2- 172.21.15.16:0/1118615524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e414a0 0x55a9c2e584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- 172.21.15.16:0/1118615524 >> 172.21.15.16:0/1118615524 conn(0x55a9c2e3da30 msgr2=0x55a9c2e47ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- 172.21.15.16:0/1118615524 shutdown_connections 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- 172.21.15.16:0/1118615524 wait complete. 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 -- start start 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.261+0000 7f4b79983740 1 Processor -- start 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e50cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2ec5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b0c00 con 0x55a9c2e414a0 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b0530 con 0x55a9c2e41dc0 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701ad770 con 0x55a9c2e4c680 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b79983740 10 client.0 _ll_register_callbacks cb 0x7f4b6009b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:12:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47814/0 (socket says 172.21.15.16:47814) 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 -- 172.21.15.16:0/1961028982 learned_addr learned my addr 172.21.15.16:0/1961028982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75b47640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2ec5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b74b45640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e50cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 -- 172.21.15.16:0/1961028982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 msgr2=0x55a9c2ec5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2ec5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 -- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 msgr2=0x55a9c2e50cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e50cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 -- 172.21.15.16:0/1961028982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9c2ec6170 con 0x55a9c2e414a0 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b75346640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005450 tx=0x7f4b600ccaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18b77ad90921a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.262+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6001d070 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4b6009abf0 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b68004cc0 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4b600a4210 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 10 client.21123 fetch_fsmap learned FSMap version 39 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a9c2ec7320 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68004cc0 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4b600afbe0 con 0x55a9c2e414a0 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 10 client.21123 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:12:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 10 client.21123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 20 client.21123 populate_metadata read hostname 'smithi016' 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2ec78b0 0x55a9c2ec9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 4 client.21123 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.263+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a9c2e49ab0 con 0x55a9c2ec78b0 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b74b45640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2ec78b0 0x55a9c2ec9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b54ff9640 10 client.21123 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b74b45640 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2ec78b0 0x55a9c2ec9ca0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=949080e563bd16c6 server_cookie=8664944a50f710c9 in_seq=0 out_seq=0 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b54ff9640 1 -- 172.21.15.16:0/1961028982 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a9c2e49ab0 con 0x55a9c2ec78b0 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b54ff9640 10 client.21123 handle_command_reply: tid=0 2024-09-12T03:12:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 client.21123 shutdown 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 msgr2=0x55a9c2e50770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005450 tx=0x7f4b600ccaf0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 shutdown_connections 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55a9c2ec78b0 0x55a9c2ec9ca0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9c2e4c680 0x55a9c2ec5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a9c2e41dc0 0x55a9c2e50cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 --2- 172.21.15.16:0/1961028982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9c2e414a0 0x55a9c2e50770 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 >> 172.21.15.16:0/1961028982 conn(0x55a9c2e3da30 msgr2=0x55a9c2e3c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.264+0000 7f4b54ff9640 0 client.21123 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 shutdown_connections 2024-09-12T03:12:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b79983740 1 -- 172.21.15.16:0/1961028982 wait complete. 2024-09-12T03:12:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b79983740 20 client.21123 trim_cache size 0 max 16384 2024-09-12T03:12:57.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:12:57.272 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:12:57.272 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:12:57.272 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 msgr2=0x7f4b580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 0x7f4b580a5040 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005c40 tx=0x7f4b68005bd0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 msgr2=0x7f4b7019a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c890 tx=0x7f4b6000cd50 comp rx=0 tx=0).stop 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 shutdown_connections 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b7010e3c0 0x7f4b7019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b70074f40 0x7f4b7019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4b580a2b80 0x7f4b580a5040 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 --2- 172.21.15.16:0/1525648833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074620 0x7f4b7019a760 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.265+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 >> 172.21.15.16:0/1525648833 conn(0x7f4b700fd8c0 msgr2=0x7f4b700fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.266+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 shutdown_connections 2024-09-12T03:12:57.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:57.266+0000 7f4b775aa640 1 -- 172.21.15.16:0/1525648833 wait complete. 2024-09-12T03:12:57.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:57 smithi016 ceph-mon[30925]: pgmap v1088: 97 pgs: 97 active+clean; 952 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.2 MiB/s wr, 479 op/s 2024-09-12T03:12:57.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:57 smithi033 ceph-mon[35092]: pgmap v1088: 97 pgs: 97 active+clean; 952 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.2 MiB/s wr, 479 op/s 2024-09-12T03:12:58.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 301625 23.53 MB/sec execute 599 sec latency 9522.211 ms 2024-09-12T03:12:58.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:57 smithi184 ceph-mon[35321]: pgmap v1088: 97 pgs: 97 active+clean; 952 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.2 MiB/s wr, 479 op/s 2024-09-12T03:12:58.072 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:12:58.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:12:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:12:59.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- 172.21.15.16:0/2190943369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec103760 msgr2=0x7fb0ec107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:59.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 --2- 172.21.15.16:0/2190943369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec103760 0x7fb0ec107ba0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009930 tx=0x7fb0d802a420 comp rx=0 tx=0).stop 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- 172.21.15.16:0/2190943369 shutdown_connections 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 --2- 172.21.15.16:0/2190943369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 --2- 172.21.15.16:0/2190943369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec103760 0x7fb0ec107ba0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 --2- 172.21.15.16:0/2190943369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec102db0 0x7fb0ec103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- 172.21.15.16:0/2190943369 >> 172.21.15.16:0/2190943369 conn(0x7fb0ec0fc7d0 msgr2=0x7fb0ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- 172.21.15.16:0/2190943369 shutdown_connections 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- 172.21.15.16:0/2190943369 wait complete. 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 Processor -- start 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.769+0000 7fb0f3147640 1 -- start start 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec103760 0x7fb0ec198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec112070 con 0x7fb0ec102db0 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec111ef0 con 0x7fb0ec103760 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec1121f0 con 0x7fb0ec1082e0 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47834/0 (socket says 172.21.15.16:47834) 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 -- 172.21.15.16:0/1574333005 learned_addr learned my addr 172.21.15.16:0/1574333005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0ebfff640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec103760 0x7fb0ec198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f16e4640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 -- 172.21.15.16:0/1574333005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 msgr2=0x7fb0ec19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 -- 172.21.15.16:0/1574333005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec103760 msgr2=0x7fb0ec198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec103760 0x7fb0ec198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 -- 172.21.15.16:0/1574333005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec19d590 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f16e4640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f0ee3640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000d770 tx=0x7fb0e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0017070 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0e0005500 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ec19d880 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.770+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0004e00 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.771+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ec1a5160 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.771+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0ec103190 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.773+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb0e0004fa0 con 0x7fb0ec102db0 2024-09-12T03:12:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.774+0000 7fb0e9ffb640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 0x7fb0dc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.774+0000 7fb0ebfff640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 0x7fb0dc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.774+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0e00cea60 con 0x7fb0ec102db0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.774+0000 7fb0ebfff640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 0x7fb0dc0a50c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005940 tx=0x7fb0d80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:58.778+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0e009b970 con 0x7fb0ec102db0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.003+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0ec199940 con 0x7fb0ec102db0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.004+0000 7fb0e9ffb640 1 -- 172.21.15.16:0/1574333005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7fb0e009b270 con 0x7fb0ec102db0 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 msgr2=0x7fb0dc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 0x7fb0dc0a50c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005940 tx=0x7fb0d80058d0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 msgr2=0x7fb0ec1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000d770 tx=0x7fb0e000dc30 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 shutdown_connections 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0ec1082e0 0x7fb0ec19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec103760 0x7fb0ec198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0dc0a2c00 0x7fb0dc0a50c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 --2- 172.21.15.16:0/1574333005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec102db0 0x7fb0ec1985e0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 >> 172.21.15.16:0/1574333005 conn(0x7fb0ec0fc7d0 msgr2=0x7fb0ec105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 shutdown_connections 2024-09-12T03:12:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:12:59.007+0000 7fb0f3147640 1 -- 172.21.15.16:0/1574333005 wait complete. 2024-09-12T03:12:59.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 600 sec 2024-09-12T03:12:59.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:59 smithi016 ceph-mon[30925]: pgmap v1089: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.9 MiB/s wr, 329 op/s 2024-09-12T03:12:59.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:12:59 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1574333005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:59.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:12:59.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:59 smithi033 ceph-mon[35092]: pgmap v1089: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.9 MiB/s wr, 329 op/s 2024-09-12T03:12:59.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:12:59 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1574333005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:59 smithi184 ceph-mon[35321]: pgmap v1089: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.9 MiB/s wr, 329 op/s 2024-09-12T03:12:59.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:12:59 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1574333005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:12:59.941 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:00.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 601 sec 2024-09-12T03:13:00.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.411+0000 7fd35ea88640 1 -- 172.21.15.16:0/3522659871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358101de0 msgr2=0x7fd3581021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.411+0000 7fd35ea88640 1 --2- 172.21.15.16:0/3522659871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358101de0 0x7fd3581021c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd348009900 tx=0x7fd34802a410 comp rx=0 tx=0).stop 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 -- 172.21.15.16:0/3522659871 shutdown_connections 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 --2- 172.21.15.16:0/3522659871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358106f10 0x7fd35810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 --2- 172.21.15.16:0/3522659871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358102790 0x7fd3581067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 --2- 172.21.15.16:0/3522659871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358101de0 0x7fd3581021c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 -- 172.21.15.16:0/3522659871 >> 172.21.15.16:0/3522659871 conn(0x7fd3580fd8c0 msgr2=0x7fd3580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 -- 172.21.15.16:0/3522659871 shutdown_connections 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 -- 172.21.15.16:0/3522659871 wait complete. 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 Processor -- start 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.412+0000 7fd35ea88640 1 -- start start 2024-09-12T03:13:00.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 0x7fd3580744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358102790 0x7fd358074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd358111f30 con 0x7fd358106f10 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd358111db0 con 0x7fd358101de0 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581120b0 con 0x7fd358102790 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47850/0 (socket says 172.21.15.16:47850) 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 -- 172.21.15.16:0/2050715436 learned_addr learned my addr 172.21.15.16:0/2050715436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd357fff640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358102790 0x7fd358074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35c824640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 0x7fd3580744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 -- 172.21.15.16:0/2050715436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358102790 msgr2=0x7fd358074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358102790 0x7fd358074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 -- 172.21.15.16:0/2050715436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 msgr2=0x7fd3580744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 0x7fd3580744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 -- 172.21.15.16:0/2050715436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3581a6850 con 0x7fd358106f10 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35c824640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 0x7fd3580744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35d025640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eaa0 tx=0x7fd35000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd355ffb640 1 -- 172.21.15.16:0/2050715436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350004d20 con 0x7fd358106f10 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3581a6b40 con 0x7fd358106f10 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.413+0000 7fd355ffb640 1 -- 172.21.15.16:0/2050715436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd350005020 con 0x7fd358106f10 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.414+0000 7fd355ffb640 1 -- 172.21.15.16:0/2050715436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500116d0 con 0x7fd358106f10 2024-09-12T03:13:00.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.414+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3581a7080 con 0x7fd358106f10 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.416+0000 7fd355ffb640 1 -- 172.21.15.16:0/2050715436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd3500051d0 con 0x7fd358106f10 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.416+0000 7fd360e61740 1 Processor -- start 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.416+0000 7fd360e61740 1 -- start start 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10849090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f108495d0 0x561f10849a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 0x561f10856760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd355ffb640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 0x7fd3400a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f107af2f0 con 0x561f1084ab40 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f10756cf0 con 0x561f108495d0 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd360e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f107ab2b0 con 0x561f10854370 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10849090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10849090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47854/0 (socket says 172.21.15.16:47854) 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd357fff640 1 -- 172.21.15.16:0/4073537511 learned_addr learned my addr 172.21.15.16:0/4073537511 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 0x561f10856760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd357fff640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 0x7fd3400a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35c824640 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f108495d0 0x561f10849a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd355ffb640 1 -- 172.21.15.16:0/2050715436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd3500cf1f0 con 0x7fd358106f10 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 -- 172.21.15.16:0/4073537511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f108495d0 msgr2=0x561f10849a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f108495d0 0x561f10849a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 -- 172.21.15.16:0/4073537511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 msgr2=0x561f10849090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10849090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.417+0000 7fd35d025640 1 -- 172.21.15.16:0/4073537511 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f1073b3a0 con 0x561f10854370 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.418+0000 7fd357fff640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 0x7fd3400a5040 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0119d0 tx=0x7fd34c00a8a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.418+0000 7fd35d025640 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 0x561f10856760 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3500ccd50 tx=0x7fd35009b6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e79b648379ae79 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.418+0000 7fd33effd640 1 -- 172.21.15.16:0/4073537511 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35001f070 con 0x561f10854370 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.418+0000 7fd33effd640 1 -- 172.21.15.16:0/4073537511 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd35009c160 con 0x561f10854370 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.418+0000 7fd33effd640 1 -- 172.21.15.16:0/4073537511 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500980d0 con 0x561f10854370 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- 172.21.15.16:0/4073537511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 msgr2=0x561f10856760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 0x561f10856760 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3500ccd50 tx=0x7fd35009b6d0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- 172.21.15.16:0/4073537511 shutdown_connections 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f10854370 0x561f10856760 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f108495d0 0x561f10849a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 --2- 172.21.15.16:0/4073537511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10849090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- 172.21.15.16:0/4073537511 >> 172.21.15.16:0/4073537511 conn(0x561f10812310 msgr2=0x561f10845c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- 172.21.15.16:0/4073537511 shutdown_connections 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- 172.21.15.16:0/4073537511 wait complete. 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 -- start start 2024-09-12T03:13:00.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.419+0000 7fd360e61740 1 Processor -- start 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f108510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10851b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f10857010 con 0x561f1084ab40 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f1076a060 con 0x561f10854370 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f10857190 con 0x561f108495d0 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd360e61740 10 client.0 _ll_register_callbacks cb 0x561f104bd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47864/0 (socket says 172.21.15.16:47864) 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 -- 172.21.15.16:0/870686182 learned_addr learned my addr 172.21.15.16:0/870686182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd35d025640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10851b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd35c824640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f108510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 -- 172.21.15.16:0/870686182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 msgr2=0x561f108510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f108510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 -- 172.21.15.16:0/870686182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 msgr2=0x561f10851b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10851b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 -- 172.21.15.16:0/870686182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f105203c0 con 0x561f1084ab40 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd357fff640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00fc20 tx=0x7fd34c00a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5473c82c28e7bdf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.420+0000 7fd337fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c010b40 con 0x561f1084ab40 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd35009c160 con 0x561f1084ab40 2024-09-12T03:13:00.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd34c010ce0 con 0x561f1084ab40 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd34c005020 con 0x561f1084ab40 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 10 client.21141 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561f10862bf0 con 0x561f1084ab40 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd35009c160 con 0x561f1084ab40 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd34c026030 con 0x561f1084ab40 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 10 client.21141 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 10 client.21141 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 20 client.21141 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108640c0 0x561f10866520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 4 client.21141 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f107bc860 con 0x561f108640c0 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.421+0000 7fd35c824640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108640c0 0x561f10866520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.422+0000 7fd337fff640 10 client.21141 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:00.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.422+0000 7fd35c824640 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108640c0 0x561f10866520 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d81a737600be3044 server_cookie=c6b9f0e81809b765 in_seq=0 out_seq=0 2024-09-12T03:13:00.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.422+0000 7fd337fff640 1 -- 172.21.15.16:0/870686182 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f107bc860 con 0x561f108640c0 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.422+0000 7fd337fff640 10 client.21141 handle_command_reply: tid=0 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.422+0000 7fd360e61740 1 client.21141 shutdown 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 msgr2=0x561f10851630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00fc20 tx=0x7fd34c00a770 comp rx=0 tx=0).stop 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 shutdown_connections 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108640c0 0x561f10866520 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10851b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f1084ab40 0x561f10851630 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 --2- 172.21.15.16:0/870686182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f108510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 >> 172.21.15.16:0/870686182 conn(0x561f10812310 msgr2=0x561f1084b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd337fff640 0 client.21141 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 shutdown_connections 2024-09-12T03:13:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 1 -- 172.21.15.16:0/870686182 wait complete. 2024-09-12T03:13:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.423+0000 7fd360e61740 20 client.21141 trim_cache size 0 max 16384 2024-09-12T03:13:00.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 Processor -- start 2024-09-12T03:13:00.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 -- start start 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f1086bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 0x561f1086c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581af6f0 con 0x561f10854370 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581b1230 con 0x561f1084ab40 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd360e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581afdc0 con 0x561f108495d0 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47878/0 (socket says 172.21.15.16:47878) 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 -- 172.21.15.16:0/554059706 learned_addr learned my addr 172.21.15.16:0/554059706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd357fff640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 0x561f1086c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35c824640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f1086bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 -- 172.21.15.16:0/554059706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 msgr2=0x561f1086bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f1086bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 -- 172.21.15.16:0/554059706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 msgr2=0x561f1086c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 0x561f1086c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd35d025640 1 -- 172.21.15.16:0/554059706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f10846730 con 0x561f10854370 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.436+0000 7fd357fff640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 0x561f1086c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd35d025640 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fd3500cd070 tx=0x7fd350004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54767fdd8db0abc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd33e7fc640 1 -- 172.21.15.16:0/554059706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350064070 con 0x561f10854370 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd33e7fc640 1 -- 172.21.15.16:0/554059706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd35009fe10 con 0x561f10854370 2024-09-12T03:13:00.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd33e7fc640 1 -- 172.21.15.16:0/554059706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35001f040 con 0x561f10854370 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 -- 172.21.15.16:0/554059706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 msgr2=0x561f1084b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fd3500cd070 tx=0x7fd350004a40 comp rx=0 tx=0).stop 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 -- 172.21.15.16:0/554059706 shutdown_connections 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10854370 0x561f1084b110 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f1084ab40 0x561f1086c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 --2- 172.21.15.16:0/554059706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f108495d0 0x561f1086bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.437+0000 7fd360e61740 1 -- 172.21.15.16:0/554059706 >> 172.21.15.16:0/554059706 conn(0x561f10812310 msgr2=0x561f10853170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- 172.21.15.16:0/554059706 shutdown_connections 2024-09-12T03:13:00.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- 172.21.15.16:0/554059706 wait complete. 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- start start 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 Processor -- start 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 0x561f10858020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10858610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581b0b60 con 0x561f108495d0 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581b0490 con 0x561f10854370 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd360e61740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3581ad6d0 con 0x561f1084ab40 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd35c824640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd35c824640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47880/0 (socket says 172.21.15.16:47880) 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.438+0000 7fd35c824640 1 -- 172.21.15.16:0/2386632077 learned_addr learned my addr 172.21.15.16:0/2386632077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd360e61740 10 client.0 _ll_register_callbacks cb 0x7fd34c011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:00.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd357fff640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 0x561f10858020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35d025640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10858610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 -- 172.21.15.16:0/2386632077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 msgr2=0x561f10858020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 0x561f10858020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 -- 172.21.15.16:0/2386632077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 msgr2=0x561f10858610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10858610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 -- 172.21.15.16:0/2386632077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f10858e40 con 0x561f108495d0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35d025640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10858610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd357fff640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 0x561f10858020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd35c824640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd348009230 tx=0x7fd3480042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33451f213114af6 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd337fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348035040 con 0x561f108495d0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd348005240 con 0x561f108495d0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd35009fe10 con 0x561f108495d0 2024-09-12T03:13:00.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd348053050 con 0x561f108495d0 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd360e61740 10 client.21147 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.439+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561f108ceee0 con 0x561f108495d0 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd35009fe10 con 0x561f108495d0 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fd3480036f0 con 0x561f108495d0 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 10 client.21147 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 10 client.21147 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 20 client.21147 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108cf600 0x561f108d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 4 client.21147 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f107b6d70 con 0x561f108cf600 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd357fff640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108cf600 0x561f108d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd337fff640 10 client.21147 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd357fff640 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108cf600 0x561f108d1a10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9cb0d915ca376ea1 server_cookie=111013c8a8c5a49c in_seq=0 out_seq=0 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd337fff640 1 -- 172.21.15.16:0/2386632077 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f107b6d70 con 0x561f108cf600 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd337fff640 10 client.21147 handle_command_reply: tid=0 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.440+0000 7fd360e61740 1 client.21147 shutdown 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 msgr2=0x561f1085cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd348009230 tx=0x7fd3480042c0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 shutdown_connections 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561f108cf600 0x561f108d1a10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f10854370 0x561f10858610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd337fff640 0 client.21147 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f1084ab40 0x561f10858020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 --2- 172.21.15.16:0/2386632077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f108495d0 0x561f1085cc50 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 >> 172.21.15.16:0/2386632077 conn(0x561f10812310 msgr2=0x561f1085f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 shutdown_connections 2024-09-12T03:13:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 1 -- 172.21.15.16:0/2386632077 wait complete. 2024-09-12T03:13:00.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd360e61740 20 client.21147 trim_cache size 0 max 16384 2024-09-12T03:13:00.447 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 msgr2=0x7fd3400a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 0x7fd3400a5040 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0119d0 tx=0x7fd34c00a8a0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 msgr2=0x7fd358074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.441+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eaa0 tx=0x7fd35000eea0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 shutdown_connections 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd3400a2b80 0x7fd3400a5040 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd358106f10 0x7fd358074f70 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd358102790 0x7fd358074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 --2- 172.21.15.16:0/2050715436 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd358101de0 0x7fd3580744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 >> 172.21.15.16:0/2050715436 conn(0x7fd3580fd8c0 msgr2=0x7fd3580fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 shutdown_connections 2024-09-12T03:13:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:00.442+0000 7fd35ea88640 1 -- 172.21.15.16:0/2050715436 wait complete. 2024-09-12T03:13:01.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 602 sec 2024-09-12T03:13:01.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:13:01.563 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:01.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:01 smithi033 ceph-mon[35092]: pgmap v1090: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.5 MiB/s wr, 314 op/s 2024-09-12T03:13:01.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:01 smithi184 ceph-mon[35321]: pgmap v1090: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.5 MiB/s wr, 314 op/s 2024-09-12T03:13:01.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:01 smithi016 ceph-mon[30925]: pgmap v1090: 97 pgs: 97 active+clean; 958 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.5 MiB/s wr, 314 op/s 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 -- 172.21.15.16:0/245203668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 msgr2=0x7fa59c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/245203668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c101200 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009900 tx=0x7fa58c02a480 comp rx=0 tx=0).stop 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 -- 172.21.15.16:0/245203668 shutdown_connections 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/245203668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102ef0 0x7fa59c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/245203668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102570 0x7fa59c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/245203668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c101200 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 -- 172.21.15.16:0/245203668 >> 172.21.15.16:0/245203668 conn(0x7fa59c0fc7d0 msgr2=0x7fa59c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 -- 172.21.15.16:0/245203668 shutdown_connections 2024-09-12T03:13:01.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 -- 172.21.15.16:0/245203668 wait complete. 2024-09-12T03:13:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.997+0000 7fa5a29b4640 1 Processor -- start 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 -- start start 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102570 0x7fa59c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 0x7fa59c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c110fc0 con 0x7fa59c100e20 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c110e40 con 0x7fa59c102ef0 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a29b4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c111140 con 0x7fa59c102570 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa59bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa59bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47898/0 (socket says 172.21.15.16:47898) 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa59bfff640 1 -- 172.21.15.16:0/1636588833 learned_addr learned my addr 172.21.15.16:0/1636588833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 0x7fa59c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.998+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102570 0x7fa59c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 -- 172.21.15.16:0/1636588833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102570 msgr2=0x7fa59c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102570 0x7fa59c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 -- 172.21.15.16:0/1636588833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 msgr2=0x7fa59c19bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 0x7fa59c19bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 -- 172.21.15.16:0/1636588833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c19c3d0 con 0x7fa59c100e20 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 0x7fa59c19bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa59bfff640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa58c006010 tx=0x7fa58c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5997fa640 1 -- 172.21.15.16:0/1636588833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c03f070 con 0x7fa59c100e20 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59c19c660 con 0x7fa59c100e20 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5997fa640 1 -- 172.21.15.16:0/1636588833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa58c031e30 con 0x7fa59c100e20 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5997fa640 1 -- 172.21.15.16:0/1636588833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c035030 con 0x7fa59c100e20 2024-09-12T03:13:02.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:01.999+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59c1a3fa0 con 0x7fa59c100e20 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.002+0000 7fa5997fa640 1 -- 172.21.15.16:0/1636588833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa58c042070 con 0x7fa59c100e20 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.002+0000 7fa5a4d8d740 1 Processor -- start 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.002+0000 7fa5a4d8d740 1 -- start start 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a26a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a35970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ded998c2f0 con 0x55ded9a27b40 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ded9933cf0 con 0x55ded9a33580 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ded99882b0 con 0x55ded9a265d0 2024-09-12T03:13:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47902/0 (socket says 172.21.15.16:47902) 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 -- 172.21.15.16:0/3566901994 learned_addr learned my addr 172.21.15.16:0/3566901994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa59bfff640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a26a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a35970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 -- 172.21.15.16:0/3566901994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 msgr2=0x55ded9a26a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a26a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 -- 172.21.15.16:0/3566901994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 msgr2=0x55ded9a35970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a35970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 -- 172.21.15.16:0/3566901994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ded99183a0 con 0x55ded9a27b40 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a35970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa59bfff640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a26a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5997fa640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 0x7fa5840a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.003+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa59400f760 tx=0x7fa59400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5ea444556d495a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.004+0000 7fa59bfff640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 0x7fa5840a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.004+0000 7fa5997fa640 1 -- 172.21.15.16:0/1636588833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa58c0ef810 con 0x7fa59c100e20 2024-09-12T03:13:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.004+0000 7fa5827fc640 1 -- 172.21.15.16:0/3566901994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594003b70 con 0x55ded9a27b40 2024-09-12T03:13:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.004+0000 7fa5827fc640 1 -- 172.21.15.16:0/3566901994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59400de00 con 0x55ded9a27b40 2024-09-12T03:13:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.004+0000 7fa5827fc640 1 -- 172.21.15.16:0/3566901994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594004800 con 0x55ded9a27b40 2024-09-12T03:13:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3566901994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 msgr2=0x55ded9a26090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa59400f760 tx=0x7fa59400fc20 comp rx=0 tx=0).stop 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa59bfff640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 0x7fa5840a50c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0ed690 tx=0x7fa58c02a480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3566901994 shutdown_connections 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a35970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a26a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3566901994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a26090 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3566901994 >> 172.21.15.16:0/3566901994 conn(0x55ded99ef310 msgr2=0x55ded9a22c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3566901994 shutdown_connections 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3566901994 wait complete. 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 -- start start 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.005+0000 7fa5a4d8d740 1 Processor -- start 2024-09-12T03:13:02.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a265d0 0x55ded9a2e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a27b40 0x55ded9a2ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ded9a29550 con 0x55ded9a33580 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ded9947060 con 0x55ded9a265d0 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ded9a296d0 con 0x55ded9a27b40 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a4d8d740 10 client.0 _ll_register_callbacks cb 0x55ded969a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47904/0 (socket says 172.21.15.16:47904) 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 -- 172.21.15.16:0/1795300558 learned_addr learned my addr 172.21.15.16:0/1795300558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a27b40 0x55ded9a2ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a265d0 0x55ded9a2e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 -- 172.21.15.16:0/1795300558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a27b40 msgr2=0x55ded9a2ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a27b40 0x55ded9a2ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 -- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a265d0 msgr2=0x55ded9a2e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a265d0 0x55ded9a2e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 -- 172.21.15.16:0/1795300558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ded96fd3c0 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa59400a740 tx=0x7fa5940043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb7200ba0562c20 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.006+0000 7fa57b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594004e50 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa594003970 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59400de00 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa594003b70 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 10 client.21159 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ded9a43ef0 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59400de00 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa594018440 con 0x55ded9a33580 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 10 client.21159 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 10 client.21159 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:13:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 20 client.21159 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9a45350 0x55ded9a47740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 4 client.21159 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.007+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ded9999860 con 0x55ded9a45350 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.008+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9a45350 0x55ded9a47740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.008+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9a45350 0x55ded9a47740 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1658f1fb7989fdc6 server_cookie=4a5eb823f79dfff in_seq=0 out_seq=0 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.008+0000 7fa57b7fe640 10 client.21159 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa57b7fe640 1 -- 172.21.15.16:0/1795300558 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ded9999860 con 0x55ded9a45350 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa57b7fe640 10 client.21159 handle_command_reply: tid=0 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 client.21159 shutdown 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 msgr2=0x55ded9a2f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa59400a740 tx=0x7fa5940043e0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 shutdown_connections 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9a45350 0x55ded9a47740 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a33580 0x55ded9a2f0f0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a27b40 0x55ded9a2ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa57b7fe640 0 client.21159 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/1795300558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a265d0 0x55ded9a2e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 >> 172.21.15.16:0/1795300558 conn(0x55ded99ef310 msgr2=0x55ded9a28960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 shutdown_connections 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/1795300558 wait complete. 2024-09-12T03:13:02.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.009+0000 7fa5a4d8d740 20 client.21159 trim_cache size 0 max 16384 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 Processor -- start 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 -- start start 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a4cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a2b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1ab310 con 0x55ded9a27b40 2024-09-12T03:13:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1ace50 con 0x55ded9a33580 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1ab9e0 con 0x55ded9a265d0 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47912/0 (socket says 172.21.15.16:47912) 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3757551500 learned_addr learned my addr 172.21.15.16:0/3757551500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a4cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.022+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a2b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3757551500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 msgr2=0x55ded9a4cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a4cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3757551500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 msgr2=0x55ded9a2b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a2b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3757551500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ded9a22e30 con 0x55ded9a27b40 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fa590009a80 tx=0x7fa590008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a59b37944c6d35e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa581ffb640 1 -- 172.21.15.16:0/3757551500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59001d250 con 0x55ded9a27b40 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa581ffb640 1 -- 172.21.15.16:0/3757551500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59001d8d0 con 0x55ded9a27b40 2024-09-12T03:13:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.023+0000 7fa581ffb640 1 -- 172.21.15.16:0/3757551500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590015880 con 0x55ded9a27b40 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3757551500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 msgr2=0x55ded9a4d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fa590009a80 tx=0x7fa590008040 comp rx=0 tx=0).stop 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3757551500 shutdown_connections 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9a2b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a4d220 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3757551500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a4cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3757551500 >> 172.21.15.16:0/3757551500 conn(0x55ded99ef310 msgr2=0x55ded9a21d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3757551500 shutdown_connections 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3757551500 wait complete. 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 -- start start 2024-09-12T03:13:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.024+0000 7fa5a4d8d740 1 Processor -- start 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a3cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9aaecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1ac780 con 0x55ded9a27b40 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1ac0b0 con 0x55ded9a33580 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c1a9310 con 0x55ded9a265d0 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a4d8d740 10 client.0 _ll_register_callbacks cb 0x7fa59400f5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47920/0 (socket says 172.21.15.16:47920) 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3291433464 learned_addr learned my addr 172.21.15.16:0/3291433464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a3cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9aaecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3291433464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 msgr2=0x55ded9a3cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a3cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 msgr2=0x55ded9aaecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9aaecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 -- 172.21.15.16:0/3291433464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ded9aaf2f0 con 0x55ded9a27b40 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9aaecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa59b7fe640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa590004810 tx=0x7fa5900094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189dfd76f65f844d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa57b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.025+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59002d070 con 0x55ded9a27b40 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59001d8d0 con 0x55ded9a27b40 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa59001da80 con 0x55ded9a27b40 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fa59001dc80 con 0x55ded9a27b40 2024-09-12T03:13:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 10 client.21165 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55ded9ab05b0 con 0x55ded9a27b40 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59001d8d0 con 0x55ded9a27b40 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fa5900264b0 con 0x55ded9a27b40 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 10 client.21165 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 10 client.21165 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 20 client.21165 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9ab0ce0 0x55ded9ab3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 4 client.21165 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ded9993d70 con 0x55ded9ab0ce0 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.026+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9ab0ce0 0x55ded9ab3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa57b7fe640 10 client.21165 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a0f51640 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9ab0ce0 0x55ded9ab3100 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66501968902982e6 server_cookie=b810550b1040002c in_seq=0 out_seq=0 2024-09-12T03:13:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa57b7fe640 1 -- 172.21.15.16:0/3291433464 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ded9993d70 con 0x55ded9ab0ce0 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa57b7fe640 10 client.21165 handle_command_reply: tid=0 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 client.21165 shutdown 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 msgr2=0x55ded9a3d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa590004810 tx=0x7fa5900094d0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 shutdown_connections 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x55ded9ab0ce0 0x55ded9ab3100 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ded9a33580 0x55ded9aaecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ded9a27b40 0x55ded9a3d100 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 --2- 172.21.15.16:0/3291433464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ded9a265d0 0x55ded9a3cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa57b7fe640 0 client.21165 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 >> 172.21.15.16:0/3291433464 conn(0x55ded99ef310 msgr2=0x55ded9a2e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 shutdown_connections 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 1 -- 172.21.15.16:0/3291433464 wait complete. 2024-09-12T03:13:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.027+0000 7fa5a4d8d740 20 client.21165 trim_cache size 0 max 16384 2024-09-12T03:13:02.034 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 msgr2=0x7fa5840a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 0x7fa5840a50c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0ed690 tx=0x7fa58c02a480 comp rx=0 tx=0).stop 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 msgr2=0x7fa59c197420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa58c006010 tx=0x7fa58c003c80 comp rx=0 tx=0).stop 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 shutdown_connections 2024-09-12T03:13:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa59c102ef0 0x7fa59c19bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa59c102570 0x7fa59c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa5840a2c00 0x7fa5840a50c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 --2- 172.21.15.16:0/1636588833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59c100e20 0x7fa59c197420 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 >> 172.21.15.16:0/1636588833 conn(0x7fa59c0fc7d0 msgr2=0x7fa59c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 shutdown_connections 2024-09-12T03:13:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:02.028+0000 7fa5a29b4640 1 -- 172.21.15.16:0/1636588833 wait complete. 2024-09-12T03:13:02.044 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 603 sec 2024-09-12T03:13:02.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:13:03.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 604 sec 2024-09-12T03:13:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 -- 172.21.15.16:0/470788876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8103660 msgr2=0x7f31a810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 --2- 172.21.15.16:0/470788876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8103660 0x7f31a810bb50 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f3198009900 tx=0x7f319802a480 comp rx=0 tx=0).stop 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 -- 172.21.15.16:0/470788876 shutdown_connections 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 --2- 172.21.15.16:0/470788876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 --2- 172.21.15.16:0/470788876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8103660 0x7f31a810bb50 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 --2- 172.21.15.16:0/470788876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8102d40 0x7f31a8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 -- 172.21.15.16:0/470788876 >> 172.21.15.16:0/470788876 conn(0x7f31a80fc7a0 msgr2=0x7f31a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 -- 172.21.15.16:0/470788876 shutdown_connections 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.574+0000 7f31b061e640 1 -- 172.21.15.16:0/470788876 wait complete. 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 Processor -- start 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 -- start start 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8103660 0x7f31a8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31a8112030 con 0x7f31a8102d40 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a8111eb0 con 0x7f31a8103660 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31b061e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81121b0 con 0x7f31a810c170 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47940/0 (socket says 172.21.15.16:47940) 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31ae3ba640 1 -- 172.21.15.16:0/1674531143 learned_addr learned my addr 172.21.15.16:0/1674531143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31adbb9640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8103660 0x7f31a8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.575+0000 7f31aebbb640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 -- 172.21.15.16:0/1674531143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 msgr2=0x7f31a819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 -- 172.21.15.16:0/1674531143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8103660 msgr2=0x7f31a8198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8103660 0x7f31a8198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 -- 172.21.15.16:0/1674531143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a819d530 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31aebbb640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f31a400b4a0 tx=0x7f31a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f319f7fe640 1 -- 172.21.15.16:0/1674531143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4014070 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f319f7fe640 1 -- 172.21.15.16:0/1674531143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a4010af0 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a819d820 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f319f7fe640 1 -- 172.21.15.16:0/1674531143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4017d10 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.576+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a81a5100 con 0x7f31a8102d40 2024-09-12T03:13:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.579+0000 7f319f7fe640 1 -- 172.21.15.16:0/1674531143 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f31a4017070 con 0x7f31a8102d40 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.579+0000 7f31b29f7740 1 Processor -- start 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.579+0000 7f31b29f7740 1 -- start start 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a02950d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a02951a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0294732f0 con 0x55a02950d5d0 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a02941acf0 con 0x55a029518370 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31b29f7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a02946f2b0 con 0x55a02950eb40 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47954/0 (socket says 172.21.15.16:47954) 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 -- 172.21.15.16:0/4177748103 learned_addr learned my addr 172.21.15.16:0/4177748103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31aebbb640 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a02950d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31adbb9640 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a02951a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f319f7fe640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 0x7f31940a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 -- 172.21.15.16:0/4177748103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 msgr2=0x55a02950d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a02950d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 -- 172.21.15.16:0/4177748103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 msgr2=0x55a02951a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a02951a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31ae3ba640 1 -- 172.21.15.16:0/4177748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0293ff3a0 con 0x55a02950d5d0 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.580+0000 7f31aebbb640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 0x7f31940a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f319f7fe640 1 -- 172.21.15.16:0/1674531143 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31a40ce400 con 0x7f31a8102d40 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f31a4088e40 tx=0x7f31a40658f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460c4f558c28fdaa server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f31aebbb640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 0x7f31940a5040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009fd0 tx=0x7f31a000e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f319cff9640 1 -- 172.21.15.16:0/4177748103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a409a7b0 con 0x55a02950d5d0 2024-09-12T03:13:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f319cff9640 1 -- 172.21.15.16:0/4177748103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a4088230 con 0x55a02950d5d0 2024-09-12T03:13:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.581+0000 7f319cff9640 1 -- 172.21.15.16:0/4177748103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4088030 con 0x55a02950d5d0 2024-09-12T03:13:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- 172.21.15.16:0/4177748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 msgr2=0x55a02950da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f31a4088e40 tx=0x7f31a40658f0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- 172.21.15.16:0/4177748103 shutdown_connections 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a02951a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a02950da10 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4177748103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a02950d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- 172.21.15.16:0/4177748103 >> 172.21.15.16:0/4177748103 conn(0x55a0294d6310 msgr2=0x55a029509c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- 172.21.15.16:0/4177748103 shutdown_connections 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- 172.21.15.16:0/4177748103 wait complete. 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 -- start start 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.582+0000 7f31b29f7740 1 Processor -- start 2024-09-12T03:13:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a029514ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a0295155e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a02951b010 con 0x55a02950d5d0 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a02942e060 con 0x55a029518370 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a02951b190 con 0x55a02950eb40 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 10 client.0 _ll_register_callbacks cb 0x55a0291818c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47968/0 (socket says 172.21.15.16:47968) 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 -- 172.21.15.16:0/2325561648 learned_addr learned my addr 172.21.15.16:0/2325561648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a029514ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31aebbb640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a0295155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 -- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 msgr2=0x55a029514ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a029514ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 -- 172.21.15.16:0/2325561648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 msgr2=0x55a0295155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a0295155e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 -- 172.21.15.16:0/2325561648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0291e43c0 con 0x55a02950d5d0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a029514ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31aebbb640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a0295155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f31a400b470 tx=0x7f31a409ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f291f0fdcc856e60 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f3191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a401e070 con 0x55a02950d5d0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31a40d1b70 con 0x55a02950d5d0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.583+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a029480860 con 0x55a02950d5d0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f31a406cf30 con 0x55a02950d5d0 2024-09-12T03:13:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 10 client.21177 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a029526a80 con 0x55a02950d5d0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a029480860 con 0x55a02950d5d0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f31a4064070 con 0x55a02950d5d0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 10 client.21177 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 10 client.21177 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 20 client.21177 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029527dd0 0x55a02952a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 4 client.21177 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.584+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a02947ad70 con 0x55a029527dd0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.585+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029527dd0 0x55a02952a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.585+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029527dd0 0x55a02952a1c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65f04c4fee84afef server_cookie=46d74e15a732e1fa in_seq=0 out_seq=0 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.585+0000 7f3191ffb640 10 client.21177 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f3191ffb640 1 -- 172.21.15.16:0/2325561648 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a02947ad70 con 0x55a029527dd0 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f3191ffb640 10 client.21177 handle_command_reply: tid=0 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 client.21177 shutdown 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 msgr2=0x55a029517d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f31a400b470 tx=0x7f31a409ab40 comp rx=0 tx=0).stop 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 shutdown_connections 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a0295155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029527dd0 0x55a02952a1c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950eb40 0x55a029514ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f3191ffb640 0 client.21177 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2325561648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950d5d0 0x55a029517d40 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 >> 172.21.15.16:0/2325561648 conn(0x55a0294d6310 msgr2=0x55a02950f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 shutdown_connections 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 1 -- 172.21.15.16:0/2325561648 wait complete. 2024-09-12T03:13:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.586+0000 7f31b29f7740 20 client.21177 trim_cache size 0 max 16384 2024-09-12T03:13:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 Processor -- start 2024-09-12T03:13:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 -- start start 2024-09-12T03:13:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029513fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029510660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81ad660 con 0x55a02950eb40 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81af1a0 con 0x55a029518370 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.599+0000 7f31b29f7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81add30 con 0x55a02950d5d0 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47972/0 (socket says 172.21.15.16:47972) 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 -- 172.21.15.16:0/2216665255 learned_addr learned my addr 172.21.15.16:0/2216665255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029513fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 -- 172.21.15.16:0/2216665255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 msgr2=0x55a029513fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029513fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 -- 172.21.15.16:0/2216665255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 msgr2=0x55a029510660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029510660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 -- 172.21.15.16:0/2216665255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a029515950 con 0x55a02950eb40 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f31adbb9640 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f319802a9a0 tx=0x7f31980058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afae7a551111414 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f3193fff640 1 -- 172.21.15.16:0/2216665255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198049070 con 0x55a02950eb40 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f3193fff640 1 -- 172.21.15.16:0/2216665255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3198005500 con 0x55a02950eb40 2024-09-12T03:13:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.600+0000 7f3193fff640 1 -- 172.21.15.16:0/2216665255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198035040 con 0x55a02950eb40 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- 172.21.15.16:0/2216665255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 msgr2=0x55a029510120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f319802a9a0 tx=0x7f31980058d0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- 172.21.15.16:0/2216665255 shutdown_connections 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029510660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a029510120 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 --2- 172.21.15.16:0/2216665255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029513fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- 172.21.15.16:0/2216665255 >> 172.21.15.16:0/2216665255 conn(0x55a0294d6310 msgr2=0x55a0295138c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- 172.21.15.16:0/2216665255 shutdown_connections 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- 172.21.15.16:0/2216665255 wait complete. 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 -- start start 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.601+0000 7f31b29f7740 1 Processor -- start 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029529f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029590700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81aead0 con 0x55a02950eb40 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81ae400 con 0x55a029518370 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31a81ab620 con 0x55a02950d5d0 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31b29f7740 10 client.0 _ll_register_callbacks cb 0x7f3198009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47976/0 (socket says 172.21.15.16:47976) 2024-09-12T03:13:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 -- 172.21.15.16:0/4246194080 learned_addr learned my addr 172.21.15.16:0/4246194080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029529f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31aebbb640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029590700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 -- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 msgr2=0x55a029529f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029529f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 -- 172.21.15.16:0/4246194080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 msgr2=0x55a029590700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029590700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 -- 172.21.15.16:0/4246194080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02952ad20 con 0x55a02950eb40 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31aebbb640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029590700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f31adbb9640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f3198031bc0 tx=0x7f3198005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc2289068e40a46 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f3191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198035050 con 0x55a02950eb40 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.602+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f31980049f0 con 0x55a02950eb40 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3198005500 con 0x55a02950eb40 2024-09-12T03:13:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f319805f050 con 0x55a02950eb40 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 10 client.21183 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a029596160 con 0x55a02950eb40 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3198005500 con 0x55a02950eb40 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f319803c440 con 0x55a02950eb40 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 10 client.21183 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 10 client.21183 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 20 client.21183 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029596830 0x55a029598c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 4 client.21183 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.603+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0293f7170 con 0x55a029596830 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029596830 0x55a029598c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f3191ffb640 10 client.21183 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31ae3ba640 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029596830 0x55a029598c50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a7220d3ebca18831 server_cookie=d4c9df80cfbe0fb in_seq=0 out_seq=0 2024-09-12T03:13:03.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f3191ffb640 1 -- 172.21.15.16:0/4246194080 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0293f7170 con 0x55a029596830 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f3191ffb640 10 client.21183 handle_command_reply: tid=0 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 client.21183 shutdown 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 msgr2=0x55a02952a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f3198031bc0 tx=0x7f3198005bd0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 shutdown_connections 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a029518370 0x55a029590700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a02950eb40 0x55a02952a440 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x55a029596830 0x55a029598c50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 --2- 172.21.15.16:0/4246194080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a02950d5d0 0x55a029529f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f3191ffb640 0 client.21183 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 >> 172.21.15.16:0/4246194080 conn(0x55a0294d6310 msgr2=0x55a029516850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 shutdown_connections 2024-09-12T03:13:03.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.604+0000 7f31b29f7740 1 -- 172.21.15.16:0/4246194080 wait complete. 2024-09-12T03:13:03.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b29f7740 20 client.21183 trim_cache size 0 max 16384 2024-09-12T03:13:03.612 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:03.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 msgr2=0x7f31940a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 0x7f31940a5040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009fd0 tx=0x7f31a000e040 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 msgr2=0x7f31a8198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f31a400b4a0 tx=0x7f31a400b960 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 shutdown_connections 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31a810c170 0x7f31a819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a8103660 0x7f31a8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f31940a2b80 0x7f31940a5040 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 --2- 172.21.15.16:0/1674531143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8102d40 0x7f31a8198580 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 >> 172.21.15.16:0/1674531143 conn(0x7f31a80fc7a0 msgr2=0x7f31a80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 shutdown_connections 2024-09-12T03:13:03.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:03.605+0000 7f31b061e640 1 -- 172.21.15.16:0/1674531143 wait complete. 2024-09-12T03:13:03.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:03 smithi033 ceph-mon[35092]: pgmap v1091: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 212 op/s 2024-09-12T03:13:03.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:03 smithi184 ceph-mon[35321]: pgmap v1091: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 212 op/s 2024-09-12T03:13:03.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:03 smithi016 ceph-mon[30925]: pgmap v1091: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 212 op/s 2024-09-12T03:13:04.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 605 sec 2024-09-12T03:13:04.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:13:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:05.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 606 sec 2024-09-12T03:13:05.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 -- 172.21.15.16:0/3436705385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 msgr2=0x7f1f540748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 --2- 172.21.15.16:0/3436705385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f540748f0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009900 tx=0x7f1f4402a480 comp rx=0 tx=0).stop 2024-09-12T03:13:05.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 -- 172.21.15.16:0/3436705385 shutdown_connections 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 --2- 172.21.15.16:0/3436705385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 0x7f1f54110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 --2- 172.21.15.16:0/3436705385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 0x7f1f5410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 --2- 172.21.15.16:0/3436705385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f540748f0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.050+0000 7f1f59e21640 1 -- 172.21.15.16:0/3436705385 >> 172.21.15.16:0/3436705385 conn(0x7f1f540fd8c0 msgr2=0x7f1f540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.051+0000 7f1f59e21640 1 -- 172.21.15.16:0/3436705385 shutdown_connections 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.051+0000 7f1f59e21640 1 -- 172.21.15.16:0/3436705385 wait complete. 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.051+0000 7f1f59e21640 1 Processor -- start 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.051+0000 7f1f59e21640 1 -- start start 2024-09-12T03:13:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 0x7f1f5419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 0x7f1f5419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54114170 con 0x7f1f54074510 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54113ff0 con 0x7f1f54074e30 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f59e21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541142f0 con 0x7f1f5410e440 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47988/0 (socket says 172.21.15.16:47988) 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 -- 172.21.15.16:0/2552711203 learned_addr learned my addr 172.21.15.16:0/2552711203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f53fff640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 0x7f1f5419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f4affd640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 0x7f1f5419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 -- 172.21.15.16:0/2552711203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 msgr2=0x7f1f5419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 0x7f1f5419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 -- 172.21.15.16:0/2552711203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 msgr2=0x7f1f5419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 0x7f1f5419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 -- 172.21.15.16:0/2552711203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5419f6c0 con 0x7f1f54074510 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.052+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f1f44006010 tx=0x7f1f440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.053+0000 7f1f517fa640 1 -- 172.21.15.16:0/2552711203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4403f030 con 0x7f1f54074510 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.053+0000 7f1f517fa640 1 -- 172.21.15.16:0/2552711203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f44003ce0 con 0x7f1f54074510 2024-09-12T03:13:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.053+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5419f950 con 0x7f1f54074510 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.053+0000 7f1f517fa640 1 -- 172.21.15.16:0/2552711203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44035070 con 0x7f1f54074510 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.053+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f541a7290 con 0x7f1f54074510 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f517fa640 1 -- 172.21.15.16:0/2552711203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1f44042030 con 0x7f1f54074510 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f5c1fa740 1 Processor -- start 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f5c1fa740 1 -- start start 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc02a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.055+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdb682f0 con 0x55d2cdc03b40 2024-09-12T03:13:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdb0fcf0 con 0x55d2cdc025d0 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdb642b0 con 0x55d2cdc0d370 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48004/0 (socket says 172.21.15.16:48004) 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 -- 172.21.15.16:0/980860218 learned_addr learned my addr 172.21.15.16:0/980860218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f517fa640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 0x7f1f400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f4affd640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f537fe640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc02a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 -- 172.21.15.16:0/980860218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 msgr2=0x55d2cdc0f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 -- 172.21.15.16:0/980860218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 msgr2=0x55d2cdc02a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc02a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 -- 172.21.15.16:0/980860218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2cdaf43a0 con 0x55d2cdc03b40 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f4affd640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f537fe640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc02a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f517fa640 1 -- 172.21.15.16:0/2552711203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f440ef4c0 con 0x7f1f54074510 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 0x7f1f400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f53fff640 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00af80 tx=0x7f1f4c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fc452dca55f12e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.056+0000 7f1f49ffb640 1 -- 172.21.15.16:0/980860218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c012c80 con 0x55d2cdc03b40 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f49ffb640 1 -- 172.21.15.16:0/980860218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f4c024430 con 0x55d2cdc03b40 2024-09-12T03:13:05.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f49ffb640 1 -- 172.21.15.16:0/980860218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c019600 con 0x55d2cdc03b40 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/980860218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 msgr2=0x55d2cdc02090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f53fff640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 0x7f1f400a50c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c01f000 tx=0x7f1f4c01c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00af80 tx=0x7f1f4c00f850 comp rx=0 tx=0).stop 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/980860218 shutdown_connections 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc02a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/980860218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc02090 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/980860218 >> 172.21.15.16:0/980860218 conn(0x55d2cdbcb310 msgr2=0x55d2cdbfec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/980860218 shutdown_connections 2024-09-12T03:13:05.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.057+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/980860218 wait complete. 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 -- start start 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 Processor -- start 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc0cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdc10070 con 0x55d2cdc03b40 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdb23060 con 0x55d2cdc025d0 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d2cdc101f0 con 0x55d2cdc0d370 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f5c1fa740 10 client.0 _ll_register_callbacks cb 0x55d2cd8768c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.058+0000 7f1f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48020/0 (socket says 172.21.15.16:48020) 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 -- 172.21.15.16:0/1796268984 learned_addr learned my addr 172.21.15.16:0/1796268984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:05.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f537fe640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc0cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f53fff640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 -- 172.21.15.16:0/1796268984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 msgr2=0x55d2cdc0a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 -- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 msgr2=0x55d2cdc0cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc0cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 -- 172.21.15.16:0/1796268984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2cd8d93c0 con 0x55d2cdc03b40 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f4affd640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c012700 tx=0x7f1f3c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c5ba2a856e44f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c00f940 con 0x55d2cdc03b40 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f3c00fae0 con 0x55d2cdc03b40 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f4c024430 con 0x55d2cdc03b40 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1f3c00fce0 con 0x55d2cdc03b40 2024-09-12T03:13:05.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f5c1fa740 10 client.21195 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d2cdc1b9f0 con 0x55d2cdc03b40 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.059+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f4c024430 con 0x55d2cdc03b40 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1f3c01c420 con 0x55d2cdc03b40 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 10 client.21195 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 10 client.21195 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 20 client.21195 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc1ce50 0x55d2cdc1f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 4 client.21195 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2cdb75860 con 0x55d2cdc1ce50 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f537fe640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc1ce50 0x55d2cdc1f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f537fe640 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc1ce50 0x55d2cdc1f2b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27ff281521450165 server_cookie=b7c01616e8d6f152 in_seq=0 out_seq=0 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.060+0000 7f1f32ffd640 10 client.21195 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f32ffd640 1 -- 172.21.15.16:0/1796268984 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d2cdb75860 con 0x55d2cdc1ce50 2024-09-12T03:13:05.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f32ffd640 10 client.21195 handle_command_reply: tid=0 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 client.21195 shutdown 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 msgr2=0x55d2cdc09f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c012700 tx=0x7f1f3c012bc0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 shutdown_connections 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc1ce50 0x55d2cdc1f2b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc0d370 0x55d2cdc0a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc03b40 0x55d2cdc09f10 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f32ffd640 0 client.21195 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/1796268984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc025d0 0x55d2cdc0cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 >> 172.21.15.16:0/1796268984 conn(0x55d2cdbcb310 msgr2=0x55d2cdc04930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 shutdown_connections 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/1796268984 wait complete. 2024-09-12T03:13:05.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.061+0000 7f1f5c1fa740 20 client.21195 trim_cache size 0 max 16384 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.074+0000 7f1f5c1fa740 1 Processor -- start 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.074+0000 7f1f5c1fa740 1 -- start start 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc025d0 0x55d2cdc05120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc03b40 0x55d2cdc05660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541af810 con 0x55d2cdbcb310 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b1350 con 0x55d2cdc03b40 2024-09-12T03:13:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541afee0 con 0x55d2cdc025d0 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48026/0 (socket says 172.21.15.16:48026) 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 -- 172.21.15.16:0/2298205115 learned_addr learned my addr 172.21.15.16:0/2298205115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f53fff640 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc03b40 0x55d2cdc05660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f4affd640 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc025d0 0x55d2cdc05120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 -- 172.21.15.16:0/2298205115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc025d0 msgr2=0x55d2cdc05120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc025d0 0x55d2cdc05120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 -- 172.21.15.16:0/2298205115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc03b40 msgr2=0x55d2cdc05660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc03b40 0x55d2cdc05660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 -- 172.21.15.16:0/2298205115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2cdc1f8e0 con 0x55d2cdbcb310 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f1f440b82a0 tx=0x7f1f4408efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c0024488ab68fe server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f497fa640 1 -- 172.21.15.16:0/2298205115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44049040 con 0x55d2cdbcb310 2024-09-12T03:13:05.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f497fa640 1 -- 172.21.15.16:0/2298205115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f44004460 con 0x55d2cdbcb310 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.075+0000 7f1f497fa640 1 -- 172.21.15.16:0/2298205115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440b84b0 con 0x55d2cdbcb310 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2298205115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 msgr2=0x55d2cdc0be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f1f440b82a0 tx=0x7f1f4408efc0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2298205115 shutdown_connections 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdc03b40 0x55d2cdc05660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc025d0 0x55d2cdc05120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2298205115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdbcb310 0x55d2cdc0be80 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2298205115 >> 172.21.15.16:0/2298205115 conn(0x55d2cdc0a5b0 msgr2=0x55d2cdc0b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2298205115 shutdown_connections 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2298205115 wait complete. 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.076+0000 7f1f5c1fa740 1 -- start start 2024-09-12T03:13:05.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 Processor -- start 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc03b40 0x55d2cdc11e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b0c80 con 0x55d2cdc025d0 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b05b0 con 0x55d2cdbcb310 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541ad7f0 con 0x55d2cdc03b40 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f5c1fa740 10 client.0 _ll_register_callbacks cb 0x7f1f3c009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48034/0 (socket says 172.21.15.16:48034) 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:39474/0 (socket says 172.21.15.16:39474) 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 -- 172.21.15.16:0/2275328069 learned_addr learned my addr 172.21.15.16:0/2275328069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f53fff640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc03b40 0x55d2cdc11e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 -- 172.21.15.16:0/2275328069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc03b40 msgr2=0x55d2cdc11e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc03b40 0x55d2cdc11e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 -- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 msgr2=0x55d2cdc11400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f4affd640 1 -- 172.21.15.16:0/2275328069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2cdc86d50 con 0x55d2cdc025d0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.077+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f4affd640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0126d0 tx=0x7f1f3c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3251c299e237c56e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c01ccc0 con 0x55d2cdc025d0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f1f3c00f790 con 0x55d2cdc025d0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f44004460 con 0x55d2cdc025d0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1f3c01ce60 con 0x55d2cdc025d0 2024-09-12T03:13:05.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 10 client.21201 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d2cdc88010 con 0x55d2cdc025d0 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f44004460 con 0x55d2cdc025d0 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f1f3c003950 con 0x55d2cdc025d0 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 10 client.21201 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 10 client.21201 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 20 client.21201 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc0d370 0x55d2cdc8a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 4 client.21201 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.078+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d2cdb6fd70 con 0x55d2cdc0d370 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc0d370 0x55d2cdc8a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f537fe640 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc0d370 0x55d2cdc8a7d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=993632a7059881f9 server_cookie=8d19e39508f322c in_seq=0 out_seq=0 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f32ffd640 10 client.21201 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f32ffd640 1 -- 172.21.15.16:0/2275328069 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d2cdb6fd70 con 0x55d2cdc0d370 2024-09-12T03:13:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f32ffd640 10 client.21201 handle_command_reply: tid=0 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 client.21201 shutdown 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 msgr2=0x55d2cdc11940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0126d0 tx=0x7f1f3c009f60 comp rx=0 tx=0).stop 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 shutdown_connections 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x55d2cdc0d370 0x55d2cdc8a7d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d2cdc03b40 0x55d2cdc11e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2cdc025d0 0x55d2cdc11940 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 --2- 172.21.15.16:0/2275328069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d2cdbcb310 0x55d2cdc11400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f32ffd640 0 client.21201 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.079+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 >> 172.21.15.16:0/2275328069 conn(0x55d2cdc0a5b0 msgr2=0x55d2cdc083b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 shutdown_connections 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f5c1fa740 1 -- 172.21.15.16:0/2275328069 wait complete. 2024-09-12T03:13:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f5c1fa740 20 client.21201 trim_cache size 0 max 16384 2024-09-12T03:13:05.096 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:05.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 msgr2=0x7f1f400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 0x7f1f400a50c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c01f000 tx=0x7f1f4c01c000 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 msgr2=0x7f1f5419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f1f44006010 tx=0x7f1f440042f0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 shutdown_connections 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f5410e440 0x7f1f5419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1f54074e30 0x7f1f5419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f1f400a2c00 0x7f1f400a50c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 --2- 172.21.15.16:0/2552711203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f54074510 0x7f1f5419a710 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 >> 172.21.15.16:0/2552711203 conn(0x7f1f540fd8c0 msgr2=0x7f1f540fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 shutdown_connections 2024-09-12T03:13:05.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:05.080+0000 7f1f59e21640 1 -- 172.21.15.16:0/2552711203 wait complete. 2024-09-12T03:13:05.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:05 smithi033 ceph-mon[35092]: pgmap v1092: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 4.5 MiB/s wr, 176 op/s 2024-09-12T03:13:05.778 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:05 smithi016 ceph-mon[30925]: pgmap v1092: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 4.5 MiB/s wr, 176 op/s 2024-09-12T03:13:05.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:05 smithi184 ceph-mon[35321]: pgmap v1092: 97 pgs: 97 active+clean; 954 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 4.5 MiB/s wr, 176 op/s 2024-09-12T03:13:05.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:13:06.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 607 sec 2024-09-12T03:13:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.612+0000 7f41d9d47640 1 -- 172.21.15.16:0/3562922651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d4109830 msgr2=0x7f41d4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.612+0000 7f41d9d47640 1 --2- 172.21.15.16:0/3562922651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d4109830 0x7f41d4109c90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009900 tx=0x7f41c802a440 comp rx=0 tx=0).stop 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.613+0000 7f41d9d47640 1 -- 172.21.15.16:0/3562922651 shutdown_connections 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.613+0000 7f41d9d47640 1 --2- 172.21.15.16:0/3562922651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41d411b140 0x7f41d411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.613+0000 7f41d9d47640 1 --2- 172.21.15.16:0/3562922651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d4109830 0x7f41d4109c90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.613+0000 7f41d9d47640 1 --2- 172.21.15.16:0/3562922651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d4109450 0x7f41d4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.613+0000 7f41d9d47640 1 -- 172.21.15.16:0/3562922651 >> 172.21.15.16:0/3562922651 conn(0x7f41d406e140 msgr2=0x7f41d406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.615+0000 7f41d37fe640 1 --2- 172.21.15.16:0/3562922651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d4109450 0x7f41d4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.617+0000 7f41d9d47640 1 -- 172.21.15.16:0/3562922651 shutdown_connections 2024-09-12T03:13:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.618+0000 7f41d9d47640 1 -- 172.21.15.16:0/3562922651 wait complete. 2024-09-12T03:13:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 Processor -- start 2024-09-12T03:13:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 -- start start 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41d4109830 0x7f41d41a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d41abd60 0x7f41d41ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41d4120de0 con 0x7f41d411b140 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41d4120c60 con 0x7f41d4109830 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.619+0000 7f41d9d47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41d4120f60 con 0x7f41d41abd60 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.620+0000 7f41d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.620+0000 7f41d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39856/0 (socket says 172.21.15.16:39856) 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.620+0000 7f41d2ffd640 1 -- 172.21.15.16:0/2381094885 learned_addr learned my addr 172.21.15.16:0/2381094885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.620+0000 7f41d3fff640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d41abd60 0x7f41d41ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 -- 172.21.15.16:0/2381094885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d41abd60 msgr2=0x7f41d41ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d41abd60 0x7f41d41ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 -- 172.21.15.16:0/2381094885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41d4109830 msgr2=0x7f41d41a7490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41d4109830 0x7f41d41a7490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 -- 172.21.15.16:0/2381094885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d41ac820 con 0x7f41d411b140 2024-09-12T03:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000c00 tx=0x7f41c8005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.621+0000 7f41d0ff9640 1 -- 172.21.15.16:0/2381094885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8040030 con 0x7f41d411b140 2024-09-12T03:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.622+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d41acab0 con 0x7f41d411b140 2024-09-12T03:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.622+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d41b4230 con 0x7f41d411b140 2024-09-12T03:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.622+0000 7f41d0ff9640 1 -- 172.21.15.16:0/2381094885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c8005480 con 0x7f41d411b140 2024-09-12T03:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.622+0000 7f41d0ff9640 1 -- 172.21.15.16:0/2381094885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c803c8d0 con 0x7f41d411b140 2024-09-12T03:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.627+0000 7f41dc120740 1 Processor -- start 2024-09-12T03:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.627+0000 7f41d0ff9640 1 -- 172.21.15.16:0/2381094885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f41c8043030 con 0x7f41d411b140 2024-09-12T03:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.628+0000 7f41dc120740 1 -- start start 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.628+0000 7f41dc120740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 0x559a7d310090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41dc120740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d310a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41dc120740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41dc120740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559a7d2762f0 con 0x559a7d311b40 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41dc120740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559a7d21dcf0 con 0x559a7d31b370 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41dc120740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559a7d2722b0 con 0x559a7d3105d0 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d0ff9640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 0x7f41bc0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d0ff9640 1 -- 172.21.15.16:0/2381094885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41c80ef590 con 0x7f41d411b140 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d3fff640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 0x7f41bc0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:59996/0 (socket says 172.21.15.16:59996) 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d2ffd640 1 -- 172.21.15.16:0/3437681939 learned_addr learned my addr 172.21.15.16:0/3437681939 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.629+0000 7f41d3fff640 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 0x559a7d310090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d2ffd640 1 -- 172.21.15.16:0/3437681939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 msgr2=0x559a7d310a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d310a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d2ffd640 1 -- 172.21.15.16:0/3437681939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 msgr2=0x559a7d310090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 0x559a7d310090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d2ffd640 1 -- 172.21.15.16:0/3437681939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a7d2023a0 con 0x559a7d31b370 2024-09-12T03:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.630+0000 7f41d3fff640 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 0x559a7d310090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.631+0000 7f41d3fff640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 0x7f41bc0a51d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f41cc005fd0 tx=0x7f41cc009630 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.631+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f41c80ed0a0 tx=0x7f41c8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f39cdea8314c2095 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.631+0000 7f41b9ffb640 1 -- 172.21.15.16:0/3437681939 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c804a040 con 0x559a7d31b370 2024-09-12T03:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.631+0000 7f41b9ffb640 1 -- 172.21.15.16:0/3437681939 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c80bbe00 con 0x559a7d31b370 2024-09-12T03:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.631+0000 7f41b9ffb640 1 -- 172.21.15.16:0/3437681939 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c80bc060 con 0x559a7d31b370 2024-09-12T03:13:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.633+0000 7f41dc120740 1 -- 172.21.15.16:0/3437681939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 msgr2=0x559a7d31d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.633+0000 7f41dc120740 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f41c80ed0a0 tx=0x7f41c8004610 comp rx=0 tx=0).stop 2024-09-12T03:13:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.634+0000 7f41dc120740 1 -- 172.21.15.16:0/3437681939 shutdown_connections 2024-09-12T03:13:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.634+0000 7f41dc120740 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d31b370 0x559a7d31d760 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.634+0000 7f41dc120740 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d310a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.634+0000 7f41dc120740 1 --2- 172.21.15.16:0/3437681939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d311b40 0x559a7d310090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.634+0000 7f41dc120740 1 -- 172.21.15.16:0/3437681939 >> 172.21.15.16:0/3437681939 conn(0x559a7d2d9310 msgr2=0x559a7d30cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.635+0000 7f41dc120740 1 -- 172.21.15.16:0/3437681939 shutdown_connections 2024-09-12T03:13:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.635+0000 7f41dc120740 1 -- 172.21.15.16:0/3437681939 wait complete. 2024-09-12T03:13:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.635+0000 7f41dc120740 1 -- start start 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 Processor -- start 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d31ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d318020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d318610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559a7d31de30 con 0x559a7d31b370 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559a7d231060 con 0x559a7d311b40 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559a7d31dfb0 con 0x559a7d3105d0 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.636+0000 7f41dc120740 10 client.0 _ll_register_callbacks cb 0x559a7cf848c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.637+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d318610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.637+0000 7f41d37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d31ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.637+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d318610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39868/0 (socket says 172.21.15.16:39868) 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.637+0000 7f41d3fff640 1 -- 172.21.15.16:0/3936132038 learned_addr learned my addr 172.21.15.16:0/3936132038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.637+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d318020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.638+0000 7f41d37fe640 1 -- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 msgr2=0x559a7d318020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.638+0000 7f41d37fe640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d318020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.638+0000 7f41d37fe640 1 -- 172.21.15.16:0/3936132038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 msgr2=0x559a7d318610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.638+0000 7f41d37fe640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d318610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.638+0000 7f41d37fe640 1 -- 172.21.15.16:0/3936132038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a7cfe73c0 con 0x559a7d3105d0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.639+0000 7f41d37fe640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d31ad70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x559a7d30dac0 tx=0x7f41c400a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab6d0d245b67e97 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.639+0000 7f41b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.639+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4004220 con 0x559a7d3105d0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.639+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41c40043c0 con 0x559a7d3105d0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.639+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c80bbe00 con 0x559a7d3105d0 2024-09-12T03:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.640+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f41c400ae20 con 0x559a7d3105d0 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 10 client.30292 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559a7d329870 con 0x559a7d3105d0 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c80bbe00 con 0x559a7d3105d0 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f41c4003ad0 con 0x559a7d3105d0 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 10 client.30292 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 10 client.30292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 20 client.30292 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d32b1a0 0x559a7d32d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 4 client.30292 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.641+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a7d283860 con 0x559a7d32b1a0 2024-09-12T03:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.642+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d32b1a0 0x559a7d32d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.646+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d32b1a0 0x559a7d32d640 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=190dd8ecd40256dc server_cookie=f7d8902bf6640115 in_seq=0 out_seq=0 2024-09-12T03:13:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.646+0000 7f41b2ffd640 10 client.30292 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.651+0000 7f41b2ffd640 1 -- 172.21.15.16:0/3936132038 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559a7d283860 con 0x559a7d32b1a0 2024-09-12T03:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.651+0000 7f41b2ffd640 10 client.30292 handle_command_reply: tid=0 2024-09-12T03:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.651+0000 7f41dc120740 1 client.30292 shutdown 2024-09-12T03:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 msgr2=0x559a7d31ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d31ad70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x559a7d30dac0 tx=0x7f41c400a680 comp rx=0 tx=0).stop 2024-09-12T03:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 shutdown_connections 2024-09-12T03:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d32b1a0 0x559a7d32d640 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d318610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d318020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 --2- 172.21.15.16:0/3936132038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d31ad70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 >> 172.21.15.16:0/3936132038 conn(0x559a7d2d9310 msgr2=0x559a7d3129c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41b2ffd640 0 client.30292 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 shutdown_connections 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 1 -- 172.21.15.16:0/3936132038 wait complete. 2024-09-12T03:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.652+0000 7f41dc120740 20 client.30292 trim_cache size 0 max 16384 2024-09-12T03:13:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.670+0000 7f41dc120740 1 Processor -- start 2024-09-12T03:13:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 -- start start 2024-09-12T03:13:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d3131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d3136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d332f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41d37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d3131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41d37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d3131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34656/0 (socket says 172.21.15.16:34656) 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41d37fe640 1 -- 172.21.15.16:0/967475066 learned_addr learned my addr 172.21.15.16:0/967475066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000e400 con 0x559a7d31b370 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000ff40 con 0x559a7d311b40 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.671+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000ead0 con 0x559a7d3105d0 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d332f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d3136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 -- 172.21.15.16:0/967475066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 msgr2=0x559a7d3131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d3131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 -- 172.21.15.16:0/967475066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 msgr2=0x559a7d3136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d3136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.672+0000 7f41d3fff640 1 -- 172.21.15.16:0/967475066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a7d313c30 con 0x559a7d31b370 2024-09-12T03:13:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.673+0000 7f41d3fff640 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d332f50 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009370 tx=0x7f41cc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d200321ad0de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.673+0000 7f41b97fa640 1 -- 172.21.15.16:0/967475066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0141f0 con 0x559a7d31b370 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.673+0000 7f41b97fa640 1 -- 172.21.15.16:0/967475066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41cc004520 con 0x559a7d31b370 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.673+0000 7f41b97fa640 1 -- 172.21.15.16:0/967475066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0219b0 con 0x559a7d31b370 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.674+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 msgr2=0x559a7d332f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.674+0000 7f41dc120740 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d332f50 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009370 tx=0x7f41cc0095c0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 shutdown_connections 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d332f50 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d311b40 0x559a7d3136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 --2- 172.21.15.16:0/967475066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d3105d0 0x559a7d3131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 >> 172.21.15.16:0/967475066 conn(0x559a7d2d9310 msgr2=0x559a7d30bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 shutdown_connections 2024-09-12T03:13:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 -- 172.21.15.16:0/967475066 wait complete. 2024-09-12T03:13:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.675+0000 7f41dc120740 1 -- start start 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 Processor -- start 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 0x559a7d323740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d3201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000f870 con 0x559a7d31b370 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000f1a0 con 0x559a7d3105d0 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c000c3e0 con 0x559a7d311b40 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41dc120740 10 client.0 _ll_register_callbacks cb 0x7f41c4005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60022/0 (socket says 172.21.15.16:60022) 2024-09-12T03:13:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.676+0000 7f41d37fe640 1 -- 172.21.15.16:0/1441721268 learned_addr learned my addr 172.21.15.16:0/1441721268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 0x559a7d323740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d3fff640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d3201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 -- 172.21.15.16:0/1441721268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 msgr2=0x559a7d323740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 0x559a7d323740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 -- 172.21.15.16:0/1441721268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 msgr2=0x559a7d3201d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d3201d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 -- 172.21.15.16:0/1441721268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a7d3207c0 con 0x559a7d3105d0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d3fff640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d3201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 0x559a7d323740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41d37fe640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f41c4003f40 tx=0x7f41c4004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=166ab682963de1de server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.677+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4033060 con 0x559a7d3105d0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41cc004520 con 0x559a7d3105d0 2024-09-12T03:13:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f41c4005450 con 0x559a7d3105d0 2024-09-12T03:13:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f41c4005650 con 0x559a7d3105d0 2024-09-12T03:13:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 10 client.30338 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559a7d3967f0 con 0x559a7d3105d0 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41cc004520 con 0x559a7d3105d0 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f41c4004810 con 0x559a7d3105d0 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 10 client.30338 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 10 client.30338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 20 client.30338 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d396f40 0x559a7d399360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 4 client.30338 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.678+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a7d27dd70 con 0x559a7d396f40 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.679+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d396f40 0x559a7d399360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.680+0000 7f41b2ffd640 10 client.30338 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.680+0000 7f41d2ffd640 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d396f40 0x559a7d399360 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=329917777743a63e server_cookie=b61c0e63ff7c1ce5 in_seq=0 out_seq=0 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1441721268 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a7d27dd70 con 0x559a7d396f40 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41b2ffd640 10 client.30338 handle_command_reply: tid=0 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 client.30338 shutdown 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 msgr2=0x559a7d323200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f41c4003f40 tx=0x7f41c4004500 comp rx=0 tx=0).stop 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 shutdown_connections 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x559a7d396f40 0x559a7d399360 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559a7d31b370 0x559a7d3201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559a7d311b40 0x559a7d323740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 --2- 172.21.15.16:0/1441721268 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559a7d3105d0 0x559a7d323200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 >> 172.21.15.16:0/1441721268 conn(0x559a7d2d9310 msgr2=0x559a7d319880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41b2ffd640 0 client.30338 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 shutdown_connections 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 1 -- 172.21.15.16:0/1441721268 wait complete. 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.681+0000 7f41dc120740 20 client.30338 trim_cache size 0 max 16384 2024-09-12T03:13:06.684 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.682+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 msgr2=0x7f41bc0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.682+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 0x7f41bc0a51d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f41cc005fd0 tx=0x7f41cc009630 comp rx=0 tx=0).stop 2024-09-12T03:13:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.682+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 msgr2=0x7f41d41a79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.682+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000c00 tx=0x7f41c8005080 comp rx=0 tx=0).stop 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 shutdown_connections 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41d41abd60 0x7f41d41ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f41bc0a2d10 0x7f41bc0a51d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d411b140 0x7f41d41a79d0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 --2- 172.21.15.16:0/2381094885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41d4109830 0x7f41d41a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 >> 172.21.15.16:0/2381094885 conn(0x7f41d406e140 msgr2=0x7f41d411be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 shutdown_connections 2024-09-12T03:13:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:06.683+0000 7f41d9d47640 1 -- 172.21.15.16:0/2381094885 wait complete. 2024-09-12T03:13:07.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 608 sec 2024-09-12T03:13:07.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:07 smithi016 ceph-mon[30925]: pgmap v1093: 97 pgs: 97 active+clean; 953 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-09-12T03:13:07.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:07 smithi033 ceph-mon[35092]: pgmap v1093: 97 pgs: 97 active+clean; 953 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-09-12T03:13:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:07 smithi184 ceph-mon[35321]: pgmap v1093: 97 pgs: 97 active+clean; 953 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-09-12T03:13:08.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 609 sec 2024-09-12T03:13:08.310 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-12T03:13:08.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-12T03:13:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:09.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 610 sec 2024-09-12T03:13:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 -- 172.21.15.16:0/915084357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 msgr2=0x7f4d40100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 --2- 172.21.15.16:0/915084357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40100ee0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009930 tx=0x7f4d2802a420 comp rx=0 tx=0).stop 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 -- 172.21.15.16:0/915084357 shutdown_connections 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 --2- 172.21.15.16:0/915084357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 0x7f4d4010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 --2- 172.21.15.16:0/915084357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40100ee0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 --2- 172.21.15.16:0/915084357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 0x7f4d40106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 -- 172.21.15.16:0/915084357 >> 172.21.15.16:0/915084357 conn(0x7f4d400fc7a0 msgr2=0x7f4d400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 -- 172.21.15.16:0/915084357 shutdown_connections 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.086+0000 7f4d448ec640 1 -- 172.21.15.16:0/915084357 wait complete. 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 Processor -- start 2024-09-12T03:13:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 -- start start 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 0x7f4d40199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 0x7f4d401a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d40110e80 con 0x7f4d40100a80 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d40110d00 con 0x7f4d40106a80 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d448ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d40111000 con 0x7f4d40101420 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39910/0 (socket says 172.21.15.16:39910) 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.087+0000 7f4d3e59c640 1 -- 172.21.15.16:0/1181779239 learned_addr learned my addr 172.21.15.16:0/1181779239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d37fff640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 0x7f4d40199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 0x7f4d401a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 -- 172.21.15.16:0/1181779239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 msgr2=0x7f4d40199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 0x7f4d40199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 -- 172.21.15.16:0/1181779239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 msgr2=0x7f4d401a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 0x7f4d401a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 -- 172.21.15.16:0/1181779239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d401a48d0 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e9f0 tx=0x7f4d3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d377fe640 1 -- 172.21.15.16:0/1181779239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3000cd80 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d377fe640 1 -- 172.21.15.16:0/1181779239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d30010910 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d401a4aa0 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d377fe640 1 -- 172.21.15.16:0/1181779239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30018990 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.088+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d401a4fb0 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.090+0000 7f4d377fe640 1 -- 172.21.15.16:0/1181779239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4d30010ac0 con 0x7f4d40100a80 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 Processor -- start 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 -- start start 2024-09-12T03:13:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d377fe640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 0x7f4d2c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee30c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee33540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56384ed87280 con 0x56384ee31150 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56384ed2eb10 con 0x56384ee25f70 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56384ed83240 con 0x56384ee26890 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.091+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 0x7f4d2c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d377fe640 1 -- 172.21.15.16:0/1181779239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d300cf260 con 0x7f4d40100a80 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60060/0 (socket says 172.21.15.16:60060) 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 -- 172.21.15.16:0/656347511 learned_addr learned my addr 172.21.15.16:0/656347511 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d37fff640 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee30c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee33540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 -- 172.21.15.16:0/656347511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 msgr2=0x56384ee30c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee30c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 -- 172.21.15.16:0/656347511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 msgr2=0x56384ee33540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee33540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 -- 172.21.15.16:0/656347511 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384ed3b1f0 con 0x56384ee25f70 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 0x7f4d2c0a50c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800a780 tx=0x7f4d3800ae00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4d3009bab0 tx=0x7f4d3009b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4fb8b2b88f043e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.092+0000 7f4d34ff9640 1 -- 172.21.15.16:0/656347511 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3009bb20 con 0x56384ee25f70 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d34ff9640 1 -- 172.21.15.16:0/656347511 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d3009fe20 con 0x56384ee25f70 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d34ff9640 1 -- 172.21.15.16:0/656347511 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300ab8e0 con 0x56384ee25f70 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 -- 172.21.15.16:0/656347511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 msgr2=0x56384ee26350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4d3009bab0 tx=0x7f4d3009b540 comp rx=0 tx=0).stop 2024-09-12T03:13:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 -- 172.21.15.16:0/656347511 shutdown_connections 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee33540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee30c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/656347511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee26350 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 -- 172.21.15.16:0/656347511 >> 172.21.15.16:0/656347511 conn(0x56384ee22500 msgr2=0x56384ee228e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 -- 172.21.15.16:0/656347511 shutdown_connections 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.093+0000 7f4d46cc5740 1 -- 172.21.15.16:0/656347511 wait complete. 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 -- start start 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 Processor -- start 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee2e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56384ee33d20 con 0x56384ee31150 2024-09-12T03:13:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56384ed41f10 con 0x56384ee25f70 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56384ee33ea0 con 0x56384ee26890 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d46cc5740 10 client.0 _ll_register_callbacks cb 0x56384ed0aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60062/0 (socket says 172.21.15.16:60062) 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39928/0 (socket says 172.21.15.16:39928) 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 -- 172.21.15.16:0/401543547 learned_addr learned my addr 172.21.15.16:0/401543547 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d37fff640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee2e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 -- 172.21.15.16:0/401543547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 msgr2=0x56384ee2e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee2e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 -- 172.21.15.16:0/401543547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 msgr2=0x56384ee2eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3e59c640 1 -- 172.21.15.16:0/401543547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384eaf86a0 con 0x56384ee25f70 2024-09-12T03:13:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.094+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e9c0 tx=0x7f4d300cf0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5c8e7aadaa24016 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300aed50 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d300ab7f0 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d3009fe20 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d3009bb20 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 10 client.30350 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56384ee3f960 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d3009fe20 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4d3009af70 con 0x56384ee25f70 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 10 client.30350 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 10 client.30350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 20 client.30350 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384ee40cb0 0x56384ee430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 4 client.30350 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.095+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56384ed947f0 con 0x56384ee40cb0 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.096+0000 7f4d37fff640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384ee40cb0 0x56384ee430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.096+0000 7f4d37fff640 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384ee40cb0 0x56384ee430a0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de26edb922ea187c server_cookie=cd6872f75309453b in_seq=0 out_seq=0 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.096+0000 7f4d25ffb640 10 client.30350 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d25ffb640 1 -- 172.21.15.16:0/401543547 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56384ed947f0 con 0x56384ee40cb0 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d25ffb640 10 client.30350 handle_command_reply: tid=0 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 client.30350 shutdown 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 msgr2=0x56384ee2e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e9c0 tx=0x7f4d300cf0a0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 shutdown_connections 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384ee40cb0 0x56384ee430a0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee2eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee2e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d25ffb640 0 client.30350 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/401543547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee2e080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 >> 172.21.15.16:0/401543547 conn(0x56384ee22500 msgr2=0x56384ee1d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 shutdown_connections 2024-09-12T03:13:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 1 -- 172.21.15.16:0/401543547 wait complete. 2024-09-12T03:13:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.097+0000 7f4d46cc5740 20 client.30350 trim_cache size 0 max 16384 2024-09-12T03:13:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 Processor -- start 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 -- start start 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 0x56384ee2b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee26890 0x56384ee2b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401ab450 con 0x56384ee31150 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401acf90 con 0x56384ee26890 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401abb20 con 0x56384ee25f70 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39940/0 (socket says 172.21.15.16:39940) 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3ed9d640 1 -- 172.21.15.16:0/84331919 learned_addr learned my addr 172.21.15.16:0/84331919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 0x56384ee2b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d37fff640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee26890 0x56384ee2b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 -- 172.21.15.16:0/84331919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee26890 msgr2=0x56384ee2b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee26890 0x56384ee2b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 -- 172.21.15.16:0/84331919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 msgr2=0x56384ee1ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3e59c640 1 -- 172.21.15.16:0/84331919 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384ee266c0 con 0x56384ee25f70 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.115+0000 7f4d3ed9d640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 0x56384ee2b2e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c6a0 tx=0x7f4d30009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9427b583255ee461 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d27fff640 1 -- 172.21.15.16:0/84331919 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30097a00 con 0x56384ee25f70 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d27fff640 1 -- 172.21.15.16:0/84331919 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d30097ba0 con 0x56384ee25f70 2024-09-12T03:13:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d27fff640 1 -- 172.21.15.16:0/84331919 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30018e20 con 0x56384ee25f70 2024-09-12T03:13:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 -- 172.21.15.16:0/84331919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 msgr2=0x56384ee2b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 0x56384ee2b2e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c6a0 tx=0x7f4d30009700 comp rx=0 tx=0).stop 2024-09-12T03:13:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 -- 172.21.15.16:0/84331919 shutdown_connections 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee1ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee26890 0x56384ee2b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/84331919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee25f70 0x56384ee2b2e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.116+0000 7f4d46cc5740 1 -- 172.21.15.16:0/84331919 >> 172.21.15.16:0/84331919 conn(0x56384ee22500 msgr2=0x56384ee2d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- 172.21.15.16:0/84331919 shutdown_connections 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- 172.21.15.16:0/84331919 wait complete. 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- start start 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 Processor -- start 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee43a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee43fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401ac8c0 con 0x56384ee31150 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401ac1f0 con 0x56384ee25f70 2024-09-12T03:13:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401a9450 con 0x56384ee26890 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d46cc5740 10 client.0 _ll_register_callbacks cb 0x7f4d38014750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d3ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee43fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d37fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee43a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d3e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:60082/0 (socket says 172.21.15.16:60082) 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.117+0000 7f4d3e59c640 1 -- 172.21.15.16:0/582541937 learned_addr learned my addr 172.21.15.16:0/582541937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 -- 172.21.15.16:0/582541937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 msgr2=0x56384ee43a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee43a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 -- 172.21.15.16:0/582541937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 msgr2=0x56384ee43fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee43fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 -- 172.21.15.16:0/582541937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384ee447f0 con 0x56384ee25f70 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d3e59c640 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4d3009b700 tx=0x7f4d30000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e731f8d9e6cd3c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300af040 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4d30010430 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d30097ba0 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d3009af70 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 10 client.30356 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56384ee447f0 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d30097ba0 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7f4d300a1020 con 0x56384ee25f70 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 10 client.30356 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 10 client.30356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 20 client.30356 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.118+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384eeab7a0 0x56384eeadb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.119+0000 7f4d46cc5740 4 client.30356 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.119+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56384ed8ed00 con 0x56384eeab7a0 2024-09-12T03:13:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.119+0000 7f4d37fff640 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384eeab7a0 0x56384eeadb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.119+0000 7f4d25ffb640 10 client.30356 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.119+0000 7f4d37fff640 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384eeab7a0 0x56384eeadb90 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca33c0366e54d327 server_cookie=87fb4fbce874d91 in_seq=0 out_seq=0 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d25ffb640 1 -- 172.21.15.16:0/582541937 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56384ed8ed00 con 0x56384eeab7a0 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d25ffb640 10 client.30356 handle_command_reply: tid=0 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 client.30356 shutdown 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 msgr2=0x56384ee46730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4d3009b700 tx=0x7f4d30000bc0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 shutdown_connections 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x56384eeab7a0 0x56384eeadb90 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ee31150 0x56384ee43fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56384ee26890 0x56384ee43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 --2- 172.21.15.16:0/582541937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56384ee25f70 0x56384ee46730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d25ffb640 0 client.30356 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 >> 172.21.15.16:0/582541937 conn(0x56384ee22500 msgr2=0x56384ee3c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 shutdown_connections 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 1 -- 172.21.15.16:0/582541937 wait complete. 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.120+0000 7f4d46cc5740 20 client.30356 trim_cache size 0 max 16384 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d82a2ea5-7858-46e9-993d-d7922b0b6308", 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-09-12T03:13:09.125 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 msgr2=0x7f4d2c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 0x7f4d2c0a50c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800a780 tx=0x7f4d3800ae00 comp rx=0 tx=0).stop 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 msgr2=0x7f4d40199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e9f0 tx=0x7f4d3000eeb0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 shutdown_connections 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d40106a80 0x7f4d401a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d40101420 0x7f4d40199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f4d2c0a2c00 0x7f4d2c0a50c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 --2- 172.21.15.16:0/1181779239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d40100a80 0x7f4d40199310 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 >> 172.21.15.16:0/1181779239 conn(0x7f4d400fc7a0 msgr2=0x7f4d400fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 shutdown_connections 2024-09-12T03:13:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:09.121+0000 7f4d448ec640 1 -- 172.21.15.16:0/1181779239 wait complete. 2024-09-12T03:13:09.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:09 smithi033 ceph-mon[35092]: pgmap v1094: 97 pgs: 97 active+clean; 951 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 4.4 MiB/s wr, 165 op/s 2024-09-12T03:13:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:09 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:09 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:09 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:09.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:09 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:09 smithi016 ceph-mon[30925]: pgmap v1094: 97 pgs: 97 active+clean; 951 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 4.4 MiB/s wr, 165 op/s 2024-09-12T03:13:09.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:09 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:09.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:09 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:09 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:09.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:09 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:09 smithi184 ceph-mon[35321]: pgmap v1094: 97 pgs: 97 active+clean; 951 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 4.4 MiB/s wr, 165 op/s 2024-09-12T03:13:09.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:09 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:09 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:09 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:09.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:09 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:09.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:13:10.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 611 sec 2024-09-12T03:13:10.074 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 -- 172.21.15.16:0/1042283803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 msgr2=0x7fe8e810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/1042283803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e810df40 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009900 tx=0x7fe8d802a410 comp rx=0 tx=0).stop 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 -- 172.21.15.16:0/1042283803 shutdown_connections 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/1042283803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e810e480 0x7fe8e8110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/1042283803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e810df40 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/1042283803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e81008d0 0x7fe8e8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.527+0000 7fe8efaf2640 1 -- 172.21.15.16:0/1042283803 >> 172.21.15.16:0/1042283803 conn(0x7fe8e80fc7a0 msgr2=0x7fe8e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- 172.21.15.16:0/1042283803 shutdown_connections 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- 172.21.15.16:0/1042283803 wait complete. 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 Processor -- start 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- start start 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e81008d0 0x7fe8e819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e810e480 0x7fe8e819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81141a0 con 0x7fe8e8101280 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e8114020 con 0x7fe8e810e480 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.528+0000 7fe8efaf2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e8114320 con 0x7fe8e81008d0 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39970/0 (socket says 172.21.15.16:39970) 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 -- 172.21.15.16:0/2139081068 learned_addr learned my addr 172.21.15.16:0/2139081068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e810e480 0x7fe8e819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e81008d0 0x7fe8e819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 -- 172.21.15.16:0/2139081068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e81008d0 msgr2=0x7fe8e819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e81008d0 0x7fe8e819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 -- 172.21.15.16:0/2139081068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e810e480 msgr2=0x7fe8e819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e810e480 0x7fe8e819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 -- 172.21.15.16:0/2139081068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e819f6d0 con 0x7fe8e8101280 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8006010 tx=0x7fe8d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8deffd640 1 -- 172.21.15.16:0/2139081068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d803f030 con 0x7fe8e8101280 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e819f960 con 0x7fe8e8101280 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8deffd640 1 -- 172.21.15.16:0/2139081068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8d8003ce0 con 0x7fe8e8101280 2024-09-12T03:13:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8deffd640 1 -- 172.21.15.16:0/2139081068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8035070 con 0x7fe8e8101280 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.529+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e81a72a0 con 0x7fe8e8101280 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.531+0000 7fe8deffd640 1 -- 172.21.15.16:0/2139081068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe8d8042030 con 0x7fe8e8101280 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 Processor -- start 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 -- start start 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6327880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6332140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 0x5578a6334a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5578a628d2e0 con 0x5578a6332680 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5578a6234ce0 con 0x5578a63274a0 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5578a62892a0 con 0x5578a6327dc0 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 0x5578a6334a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8ed88e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6332140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 0x5578a6334a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39980/0 (socket says 172.21.15.16:39980) 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8ed08d640 1 -- 172.21.15.16:0/1918158870 learned_addr learned my addr 172.21.15.16:0/1918158870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.532+0000 7fe8deffd640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 0x7fe8d40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6327880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 0x7fe8d40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 -- 172.21.15.16:0/1918158870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 msgr2=0x5578a6327880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6327880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 -- 172.21.15.16:0/1918158870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 msgr2=0x5578a6334a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 0x5578a6334a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 -- 172.21.15.16:0/1918158870 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a6219440 con 0x5578a6327dc0 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6327880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8deffd640 1 -- 172.21.15.16:0/2139081068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe8d80ef610 con 0x7fe8e8101280 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6332140 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400dbe0 tx=0x7fe8e400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7455913d6a372e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 0x7fe8d40a5040 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806a0f0 tx=0x7fe8e0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8d3fff640 1 -- 172.21.15.16:0/1918158870 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4005020 con 0x5578a6327dc0 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8d3fff640 1 -- 172.21.15.16:0/1918158870 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e40047a0 con 0x5578a6327dc0 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.533+0000 7fe8d3fff640 1 -- 172.21.15.16:0/1918158870 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e40116a0 con 0x5578a6327dc0 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/1918158870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 msgr2=0x5578a6332140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6332140 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400dbe0 tx=0x7fe8e400f730 comp rx=0 tx=0).stop 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/1918158870 shutdown_connections 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6332680 0x5578a6334a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6332140 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/1918158870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6327880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/1918158870 >> 172.21.15.16:0/1918158870 conn(0x5578a6323a30 msgr2=0x5578a6323e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/1918158870 shutdown_connections 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/1918158870 wait complete. 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.534+0000 7fe8f1ecb740 1 -- start start 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 Processor -- start 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6331fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a632f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5578a6335250 con 0x5578a6327dc0 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5578a6248050 con 0x5578a63274a0 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5578a63353d0 con 0x5578a6332680 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8f1ecb740 10 client.0 _ll_register_callbacks cb 0x5578a6288160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39990/0 (socket says 172.21.15.16:39990) 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 -- 172.21.15.16:0/401929386 learned_addr learned my addr 172.21.15.16:0/401929386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a632f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6331fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 -- 172.21.15.16:0/401929386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 msgr2=0x5578a632f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a632f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 -- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 msgr2=0x5578a6331fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6331fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed08d640 1 -- 172.21.15.16:0/401929386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a5ffe3c0 con 0x5578a6327dc0 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.535+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6331fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8005ab0 tx=0x7fe8d8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5197fe11dcf4017f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8049070 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe8d80bc3a0 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8e40047a0 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe8d80bbd10 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 10 client.21243 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5578a6340bd0 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e40047a0 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe8d80c4720 con 0x5578a6327dc0 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 10 client.21243 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 10 client.21243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 20 client.21243 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a6341f70 0x5578a63443a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 4 client.21243 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.536+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578a629a850 con 0x5578a6341f70 2024-09-12T03:13:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.537+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a6341f70 0x5578a63443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.537+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a6341f70 0x5578a63443a0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68845a1bf060138c server_cookie=ec3378f61dca9d87 in_seq=0 out_seq=0 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.537+0000 7fe8d17fa640 10 client.21243 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8d17fa640 1 -- 172.21.15.16:0/401929386 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5578a629a850 con 0x5578a6341f70 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8d17fa640 10 client.21243 handle_command_reply: tid=0 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 client.21243 shutdown 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 msgr2=0x5578a632f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8005ab0 tx=0x7fe8d8005c90 comp rx=0 tx=0).stop 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 shutdown_connections 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a6341f70 0x5578a63443a0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a632f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a6327dc0 0x5578a632f250 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/401929386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a63274a0 0x5578a6331fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 >> 172.21.15.16:0/401929386 conn(0x5578a6323a30 msgr2=0x5578a6329710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8d17fa640 0 client.21243 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 shutdown_connections 2024-09-12T03:13:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/401929386 wait complete. 2024-09-12T03:13:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.538+0000 7fe8f1ecb740 20 client.21243 trim_cache size 0 max 16384 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 Processor -- start 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 -- start start 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6327dc0 0x5578a632db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a6329600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81af7d0 con 0x5578a63274a0 2024-09-12T03:13:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81b1310 con 0x5578a6327dc0 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81afea0 con 0x5578a6332680 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39994/0 (socket says 172.21.15.16:39994) 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 -- 172.21.15.16:0/3119384405 learned_addr learned my addr 172.21.15.16:0/3119384405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a6329600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6327dc0 0x5578a632db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 -- 172.21.15.16:0/3119384405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 msgr2=0x5578a6329600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a6329600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 -- 172.21.15.16:0/3119384405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6327dc0 msgr2=0x5578a632db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6327dc0 0x5578a632db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 -- 172.21.15.16:0/3119384405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a6334dd0 con 0x5578a63274a0 2024-09-12T03:13:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.556+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400dbe0 tx=0x7fe8e4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bded629960affc server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8d37fe640 1 -- 172.21.15.16:0/3119384405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4004030 con 0x5578a63274a0 2024-09-12T03:13:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8d37fe640 1 -- 172.21.15.16:0/3119384405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e40041d0 con 0x5578a63274a0 2024-09-12T03:13:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8d37fe640 1 -- 172.21.15.16:0/3119384405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4028470 con 0x5578a63274a0 2024-09-12T03:13:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/3119384405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 msgr2=0x5578a633e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400dbe0 tx=0x7fe8e4005850 comp rx=0 tx=0).stop 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/3119384405 shutdown_connections 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6332680 0x5578a6329600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6327dc0 0x5578a632db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/3119384405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a633e4b0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/3119384405 >> 172.21.15.16:0/3119384405 conn(0x5578a6323a30 msgr2=0x5578a6328ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.557+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/3119384405 shutdown_connections 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/3119384405 wait complete. 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 -- start start 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 Processor -- start 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6336a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6332680 0x5578a6336fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81b0c40 con 0x5578a63274a0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81b0570 con 0x5578a6332680 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e81ad790 con 0x5578a6327dc0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.558+0000 7fe8f1ecb740 10 client.0 _ll_register_callbacks cb 0x7fe8d80b87f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40000/0 (socket says 172.21.15.16:40000) 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 -- 172.21.15.16:0/216787969 learned_addr learned my addr 172.21.15.16:0/216787969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ee08f640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6332680 0x5578a6336fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6336a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 -- 172.21.15.16:0/216787969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 msgr2=0x5578a6336a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6336a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 -- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6332680 msgr2=0x5578a6336fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6332680 0x5578a6336fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 -- 172.21.15.16:0/216787969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a63abe40 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8ed88e640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400d9a0 tx=0x7fe8e400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06aac6eb515b94d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4003d50 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe8e4003ef0 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8e40041d0 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.559+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe8e40035f0 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 10 client.21249 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5578a63ad0d0 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e40041d0 con 0x5578a63274a0 2024-09-12T03:13:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fe8e403bc30 con 0x5578a63274a0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 10 client.21249 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 10 client.21249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 20 client.21249 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a63ad5f0 0x5578a63af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 4 client.21249 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5578a632fb20 con 0x5578a63ad5f0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.560+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a63ad5f0 0x5578a63af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8ed08d640 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a63ad5f0 0x5578a63af9e0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e67ee2f80ed64241 server_cookie=b199cece0b4bf4e9 in_seq=0 out_seq=0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8d17fa640 10 client.21249 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8d17fa640 1 -- 172.21.15.16:0/216787969 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5578a632fb20 con 0x5578a63ad5f0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8d17fa640 10 client.21249 handle_command_reply: tid=0 2024-09-12T03:13:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 client.21249 shutdown 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 msgr2=0x5578a6336530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400d9a0 tx=0x7fe8e400df00 comp rx=0 tx=0).stop 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 shutdown_connections 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x5578a63ad5f0 0x5578a63af9e0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5578a6332680 0x5578a6336fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5578a6327dc0 0x5578a6336a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8d17fa640 0 client.21249 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 --2- 172.21.15.16:0/216787969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578a63274a0 0x5578a6336530 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 >> 172.21.15.16:0/216787969 conn(0x5578a6323a30 msgr2=0x5578a6322350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 shutdown_connections 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 1 -- 172.21.15.16:0/216787969 wait complete. 2024-09-12T03:13:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.561+0000 7fe8f1ecb740 20 client.21249 trim_cache size 0 max 16384 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "d82a2ea5-7858-46e9-993d-d7922b0b6308": { 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d82a2ea5-7858-46e9-993d-d7922b0b6308", 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:10.567 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 msgr2=0x7fe8d40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 0x7fe8d40a5040 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806a0f0 tx=0x7fe8e0009500 comp rx=0 tx=0).stop 2024-09-12T03:13:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 msgr2=0x7fe8e819ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8006010 tx=0x7fe8d80042f0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 shutdown_connections 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8e810e480 0x7fe8e819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe8d40a2b80 0x7fe8d40a5040 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e8101280 0x7fe8e819ac60 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 --2- 172.21.15.16:0/2139081068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8e81008d0 0x7fe8e819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 >> 172.21.15.16:0/2139081068 conn(0x7fe8e80fc7a0 msgr2=0x7fe8e80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 shutdown_connections 2024-09-12T03:13:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:10.562+0000 7fe8efaf2640 1 -- 172.21.15.16:0/2139081068 wait complete. 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub queued for path: / 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: active paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:10 smithi184 ceph-mon[35321]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub queued for path: / 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: active paths [/] 2024-09-12T03:13:10.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:10 smithi016 ceph-mon[30925]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub queued for path: / 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: active paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:10.998 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:10 smithi033 ceph-mon[35092]: scrub summary: idle+waiting paths [/] 2024-09-12T03:13:11.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 612 sec 2024-09-12T03:13:11.336 INFO:tasks.cephfs.filesystem:scrub status for tag:d82a2ea5-7858-46e9-993d-d7922b0b6308 - {'path': '/', 'tag': 'd82a2ea5-7858-46e9-993d-d7922b0b6308', 'options': 'recursive,force'} 2024-09-12T03:13:11.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:11 smithi016 ceph-mon[30925]: pgmap v1095: 97 pgs: 97 active+clean; 944 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 5.0 MiB/s wr, 217 op/s 2024-09-12T03:13:11.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:11 smithi184 ceph-mon[35321]: pgmap v1095: 97 pgs: 97 active+clean; 944 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 5.0 MiB/s wr, 217 op/s 2024-09-12T03:13:11.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:11 smithi033 ceph-mon[35092]: pgmap v1095: 97 pgs: 97 active+clean; 944 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 5.0 MiB/s wr, 217 op/s 2024-09-12T03:13:12.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 613 sec 2024-09-12T03:13:13.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 614 sec 2024-09-12T03:13:13.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:13 smithi184 ceph-mon[35321]: pgmap v1096: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-12T03:13:13.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:13 smithi016 ceph-mon[30925]: pgmap v1096: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-12T03:13:13.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:13 smithi033 ceph-mon[35092]: pgmap v1096: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-09-12T03:13:14.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 615 sec 2024-09-12T03:13:15.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 616 sec 2024-09-12T03:13:15.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:15 smithi184 ceph-mon[35321]: pgmap v1097: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.4 MiB/s wr, 225 op/s 2024-09-12T03:13:15.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:15 smithi016 ceph-mon[30925]: pgmap v1097: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.4 MiB/s wr, 225 op/s 2024-09-12T03:13:15.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:15 smithi033 ceph-mon[35092]: pgmap v1097: 97 pgs: 97 active+clean; 945 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.4 MiB/s wr, 225 op/s 2024-09-12T03:13:16.045 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 617 sec 2024-09-12T03:13:17.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 618 sec 2024-09-12T03:13:17.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:17 smithi184 ceph-mon[35321]: pgmap v1098: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 5.1 MiB/s wr, 240 op/s 2024-09-12T03:13:17.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:17 smithi016 ceph-mon[30925]: pgmap v1098: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 5.1 MiB/s wr, 240 op/s 2024-09-12T03:13:17.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:17 smithi033 ceph-mon[35092]: pgmap v1098: 97 pgs: 97 active+clean; 946 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 5.1 MiB/s wr, 240 op/s 2024-09-12T03:13:18.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 619 sec 2024-09-12T03:13:19.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 620 sec 2024-09-12T03:13:19.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:13:19.334 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:19.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:19 smithi016 ceph-mon[30925]: pgmap v1099: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-09-12T03:13:19.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:19 smithi184 ceph-mon[35321]: pgmap v1099: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 -- 172.21.15.16:0/1017244509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd4106f20 msgr2=0x7f9bd410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 --2- 172.21.15.16:0/1017244509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd4106f20 0x7f9bd410eaa0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009ca0 tx=0x7f9bcc02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 -- 172.21.15.16:0/1017244509 shutdown_connections 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 --2- 172.21.15.16:0/1017244509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd4106f20 0x7f9bd410eaa0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 --2- 172.21.15.16:0/1017244509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd41027a0 0x7f9bd41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 --2- 172.21.15.16:0/1017244509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4101df0 0x7f9bd41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 -- 172.21.15.16:0/1017244509 >> 172.21.15.16:0/1017244509 conn(0x7f9bd40fd8f0 msgr2=0x7f9bd40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 -- 172.21.15.16:0/1017244509 shutdown_connections 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.840+0000 7f9bda777640 1 -- 172.21.15.16:0/1017244509 wait complete. 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 Processor -- start 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 -- start start 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd4101df0 0x7f9bd41983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4106f20 0x7f9bd419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd4111f80 con 0x7f9bd41027a0 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd4111e00 con 0x7f9bd4106f20 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bda777640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd4112100 con 0x7f9bd4101df0 2024-09-12T03:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42344/0 (socket says 172.21.15.16:42344) 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bd37fe640 1 -- 172.21.15.16:0/3794805545 learned_addr learned my addr 172.21.15.16:0/3794805545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bd3fff640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd4101df0 0x7f9bd41983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.841+0000 7f9bd8d14640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4106f20 0x7f9bd419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 -- 172.21.15.16:0/3794805545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd4101df0 msgr2=0x7f9bd41983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd4101df0 0x7f9bd41983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 -- 172.21.15.16:0/3794805545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4106f20 msgr2=0x7f9bd419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4106f20 0x7f9bd419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 -- 172.21.15.16:0/3794805545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd419d370 con 0x7f9bd41027a0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd37fe640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800d770 tx=0x7f9bc800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8017070 con 0x7f9bd41027a0 2024-09-12T03:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd419d660 con 0x7f9bd41027a0 2024-09-12T03:13:19.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd41a4f40 con 0x7f9bd41027a0 2024-09-12T03:13:19.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc8005500 con 0x7f9bd41027a0 2024-09-12T03:13:19.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.842+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004e00 con 0x7f9bd41027a0 2024-09-12T03:13:19.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.843+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc0005290 con 0x7f9bd41027a0 2024-09-12T03:13:19.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.845+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9bc80046c0 con 0x7f9bd41027a0 2024-09-12T03:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.846+0000 7f9bd17fa640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 0x7f9bbc0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.846+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f9bc80ceb30 con 0x7f9bd41027a0 2024-09-12T03:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.846+0000 7f9bd3fff640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 0x7f9bbc0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.847+0000 7f9bd3fff640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 0x7f9bbc0a50c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009bd0 tx=0x7f9bc4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:19.849+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9bc809b4d0 con 0x7f9bd41027a0 2024-09-12T03:13:19.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:19 smithi033 ceph-mon[35092]: pgmap v1099: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-09-12T03:13:20.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 621 sec 2024-09-12T03:13:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.081+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bc0005580 con 0x7f9bd41027a0 2024-09-12T03:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.082+0000 7f9bd17fa640 1 -- 172.21.15.16:0/3794805545 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11969 (secure 0 0 0) 0x7f9bc809f7d0 con 0x7f9bd41027a0 2024-09-12T03:13:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:13:20.082 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"btime":"2024-09-12T03:03:48:922811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:03:48.783725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 39 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.084+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 msgr2=0x7f9bbc0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.084+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 0x7f9bbc0a50c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009bd0 tx=0x7f9bc4005f90 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.084+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 msgr2=0x7f9bd4198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.084+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800d770 tx=0x7f9bc800dc30 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 shutdown_connections 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9bd4106f20 0x7f9bd419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f9bbc0a2c00 0x7f9bbc0a50c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd41027a0 0x7f9bd4198900 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 --2- 172.21.15.16:0/3794805545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bd4101df0 0x7f9bd41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 >> 172.21.15.16:0/3794805545 conn(0x7f9bd40fd8f0 msgr2=0x7f9bd40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:20.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 shutdown_connections 2024-09-12T03:13:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:20.085+0000 7f9bda777640 1 -- 172.21.15.16:0/3794805545 wait complete. 2024-09-12T03:13:20.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3794805545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:20.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3794805545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:20.834 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id d4038f41 --release --await' 2024-09-12T03:13:20.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --set-id d4038f41 --release --await 2024-09-12T03:13:20.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3794805545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 622 sec 2024-09-12T03:13:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.622+0000 7fac12ff6640 1 -- 172.21.15.16:0/2418915164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 msgr2=0x7fac0c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2418915164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c10ec40 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fac04009cd0 tx=0x7fac0402a8e0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- 172.21.15.16:0/2418915164 shutdown_connections 2024-09-12T03:13:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2418915164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c10ec40 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2418915164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c1036e0 0x7fac0c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2418915164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c102d30 0x7fac0c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- 172.21.15.16:0/2418915164 >> 172.21.15.16:0/2418915164 conn(0x7fac0c0fc7d0 msgr2=0x7fac0c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- 172.21.15.16:0/2418915164 shutdown_connections 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- 172.21.15.16:0/2418915164 wait complete. 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 Processor -- start 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- start start 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 0x7fac0c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 0x7fac0c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1120a0 con 0x7fac0c1082a0 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.623+0000 7fac12ff6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c111f20 con 0x7fac0c1036e0 2024-09-12T03:13:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac12ff6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c112220 con 0x7fac0c102d30 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 0x7fac0c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42352/0 (socket says 172.21.15.16:42352) 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 -- 172.21.15.16:0/2991766366 learned_addr learned my addr 172.21.15.16:0/2991766366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac10d92640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 0x7fac0c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 -- 172.21.15.16:0/2991766366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 msgr2=0x7fac0c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 0x7fac0c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 -- 172.21.15.16:0/2991766366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 msgr2=0x7fac0c19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 0x7fac0c19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 -- 172.21.15.16:0/2991766366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c19f6b0 con 0x7fac0c1082a0 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac10d92640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 0x7fac0c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac0bfff640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 0x7fac0c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac11593640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fac04005cc0 tx=0x7fac040093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac09ffb640 1 -- 172.21.15.16:0/2991766366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04003e00 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0c19f940 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac09ffb640 1 -- 172.21.15.16:0/2991766366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac04004520 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.624+0000 7fac09ffb640 1 -- 172.21.15.16:0/2991766366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04030db0 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.625+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac0c1a7280 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.627+0000 7fac09ffb640 1 -- 172.21.15.16:0/2991766366 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fac040046d0 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.628+0000 7fac09ffb640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 0x7fabfc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.628+0000 7fac09ffb640 1 -- 172.21.15.16:0/2991766366 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac0402e030 con 0x7fac0c1082a0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.628+0000 7fac10d92640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 0x7fabfc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 0x7fabfc0a5040 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fac00009fc0 tx=0x7fac00005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 Processor -- start 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 -- start start 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e460ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4611630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8e4569620 con 0x55c8e460f240 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c8e4510e50 con 0x55c8e4604060 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac153cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c8e45655e0 con 0x55c8e4604980 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:41908/0 (socket says 172.21.15.16:41908) 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 -- 172.21.15.16:0/2229312455 learned_addr learned my addr 172.21.15.16:0/2229312455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac11593640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4611630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac0bfff640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e460ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 -- 172.21.15.16:0/2229312455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 msgr2=0x55c8e460ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e460ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 -- 172.21.15.16:0/2229312455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 msgr2=0x55c8e4611630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.629+0000 7fac10d92640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4611630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fac10d92640 1 -- 172.21.15.16:0/2229312455 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e4501d40 con 0x55c8e4604060 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fac11593640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4611630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fac10d92640 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fac000042c0 tx=0x7fac00014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef1f1e0e0b63a176 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fabf6ffd640 1 -- 172.21.15.16:0/2229312455 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0000f480 con 0x55c8e4604060 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fabf6ffd640 1 -- 172.21.15.16:0/2229312455 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac0000fb00 con 0x55c8e4604060 2024-09-12T03:13:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.630+0000 7fabf6ffd640 1 -- 172.21.15.16:0/2229312455 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0001cbb0 con 0x55c8e4604060 2024-09-12T03:13:21.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 -- 172.21.15.16:0/2229312455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 msgr2=0x55c8e4604440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fac000042c0 tx=0x7fac00014000 comp rx=0 tx=0).stop 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 -- 172.21.15.16:0/2229312455 shutdown_connections 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4611630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e460ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 --2- 172.21.15.16:0/2229312455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e4604440 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 -- 172.21.15.16:0/2229312455 >> 172.21.15.16:0/2229312455 conn(0x55c8e46005f0 msgr2=0x55c8e46009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 -- 172.21.15.16:0/2229312455 shutdown_connections 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.631+0000 7fac153cf740 1 -- 172.21.15.16:0/2229312455 wait complete. 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.632+0000 7fac153cf740 1 -- start start 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.632+0000 7fac153cf740 1 Processor -- start 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.632+0000 7fac153cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e460c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.632+0000 7fac153cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e460c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e460f240 0x55c8e460cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e460c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e460c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e460c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42378/0 (socket says 172.21.15.16:42378) 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac0bfff640 1 -- 172.21.15.16:0/3364805255 learned_addr learned my addr 172.21.15.16:0/3364805255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac11593640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e460f240 0x55c8e460cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8e4611e10 con 0x55c8e4604980 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c8e44f1f10 con 0x55c8e4604060 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c8e4611f90 con 0x55c8e460f240 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 10 client.0 _ll_register_callbacks cb 0x55c8e4515db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 -- 172.21.15.16:0/3364805255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e460f240 msgr2=0x55c8e460cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e460f240 0x55c8e460cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 -- 172.21.15.16:0/3364805255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 msgr2=0x55c8e460c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e460c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e4278580 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac0000fb00 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fac10d92640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e460c160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fac0001c730 tx=0x7fac0001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abcb6f7cc3035f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.633+0000 7fabebfff640 10 client.30392 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fabebfff640 1 -- 172.21.15.16:0/3364805255 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0000fcb0 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fabebfff640 1 -- 172.21.15.16:0/3364805255 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac0000f480 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fabebfff640 1 -- 172.21.15.16:0/3364805255 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fac0000f6c0 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 10 client.30392 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55c8e4342960 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fabebfff640 1 -- 172.21.15.16:0/3364805255 <== mon.2 v2:172.21.15.33:3300/0 4 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fac0001a070 con 0x55c8e4604060 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 10 client.30392 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 10 client.30392 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:13:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 20 client.30392 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.634+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e4620f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.635+0000 7fac153cf740 4 client.30392 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.635+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8e4576b90 con 0x55c8e461eb40 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.635+0000 7fac0bfff640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e4620f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.635+0000 7fac0bfff640 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e4620f30 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574277d9bc8c931f server_cookie=80a59cd5ef22e5b7 in_seq=0 out_seq=0 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.635+0000 7fabebfff640 10 client.30392 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fabebfff640 1 -- 172.21.15.16:0/3364805255 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c8e4576b90 con 0x55c8e461eb40 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fabebfff640 10 client.30392 handle_command_reply: tid=0 2024-09-12T03:13:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 client.30392 shutdown 2024-09-12T03:13:21.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:21 smithi016 ceph-mon[30925]: pgmap v1100: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 6.0 MiB/s wr, 258 op/s 2024-09-12T03:13:21.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:21 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:13:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 msgr2=0x55c8e460c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e460c160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fac0001c730 tx=0x7fac0001c4c0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 shutdown_connections 2024-09-12T03:13:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e4620f30 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fabebfff640 0 client.30392 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.636+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e460f240 0x55c8e460cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e460c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 1 --2- 172.21.15.16:0/3364805255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e460c160 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 >> 172.21.15.16:0/3364805255 conn(0x55c8e46005f0 msgr2=0x55c8e46062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 shutdown_connections 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 1 -- 172.21.15.16:0/3364805255 wait complete. 2024-09-12T03:13:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.637+0000 7fac153cf740 20 client.30392 trim_cache size 0 max 16384 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.658+0000 7fac153cf740 1 Processor -- start 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.658+0000 7fac153cf740 1 -- start start 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 0x55c8e460b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 0x55c8e4626ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1af6b0 con 0x55c8e4604980 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1b11f0 con 0x55c8e460f240 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac153cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1afd80 con 0x55c8e4604060 2024-09-12T03:13:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42386/0 (socket says 172.21.15.16:42386) 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 -- 172.21.15.16:0/3302825315 learned_addr learned my addr 172.21.15.16:0/3302825315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac10d92640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 0x55c8e460b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac11593640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 0x55c8e4626ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 -- 172.21.15.16:0/3302825315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 msgr2=0x55c8e460b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 0x55c8e460b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 -- 172.21.15.16:0/3302825315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 msgr2=0x55c8e4626ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 0x55c8e4626ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac0bfff640 1 -- 172.21.15.16:0/3302825315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e4611f40 con 0x55c8e4604980 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac10d92640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 0x55c8e460b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.659+0000 7fac11593640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 0x55c8e4626ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.660+0000 7fac0bfff640 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x55c8e460b9f0 tx=0x7fabf8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8423255a2e7590e server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.660+0000 7fabf67fc640 1 -- 172.21.15.16:0/3302825315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8022020 con 0x55c8e4604980 2024-09-12T03:13:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.660+0000 7fabf67fc640 1 -- 172.21.15.16:0/3302825315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabf80042f0 con 0x55c8e4604980 2024-09-12T03:13:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.660+0000 7fabf67fc640 1 -- 172.21.15.16:0/3302825315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf80108d0 con 0x55c8e4604980 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- 172.21.15.16:0/3302825315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 msgr2=0x55c8e4626590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x55c8e460b9f0 tx=0x7fabf8005f90 comp rx=0 tx=0).stop 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- 172.21.15.16:0/3302825315 shutdown_connections 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e460f240 0x55c8e4626ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e4604980 0x55c8e4626590 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 --2- 172.21.15.16:0/3302825315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604060 0x55c8e460b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- 172.21.15.16:0/3302825315 >> 172.21.15.16:0/3302825315 conn(0x55c8e46005f0 msgr2=0x55c8e460e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- 172.21.15.16:0/3302825315 shutdown_connections 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- 172.21.15.16:0/3302825315 wait complete. 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 -- start start 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.661+0000 7fac153cf740 1 Processor -- start 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e46147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e4614ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1b0b20 con 0x55c8e460f240 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1b0450 con 0x55c8e4604060 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ad690 con 0x55c8e4604980 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac153cf740 10 client.0 _ll_register_callbacks cb 0x7fac0001c570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42388/0 (socket says 172.21.15.16:42388) 2024-09-12T03:13:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 -- 172.21.15.16:0/1537625351 learned_addr learned my addr 172.21.15.16:0/1537625351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac0bfff640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e4614ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac10d92640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e46147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 -- 172.21.15.16:0/1537625351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 msgr2=0x55c8e4614ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e4614ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 -- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 msgr2=0x55c8e46147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e46147a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 -- 172.21.15.16:0/1537625351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e4689ae0 con 0x55c8e460f240 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac0bfff640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e4614ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac10d92640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e46147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fac11593640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac040bd740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61114c87e92e355 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.662+0000 7fabebfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac040c5b70 con 0x55c8e460f240 2024-09-12T03:13:21.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fac040b9570 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabf80042f0 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fac040dbb40 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 10 client.21273 fetch_fsmap learned FSMap version 39 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c8e468ae60 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac040d8070 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 39) ==== 8362+0+0 (secure 0 0 0) 0x7fac0404e070 con 0x55c8e460f240 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 10 client.21273 fetch_fsmap finished waiting for FSMap version 39 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 10 client.21273 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 20 client.21273 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e468d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 4 client.21273 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d4038f41", "release": true, "await": true}] 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.663+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d4038f41", "release": true, "await": true}) -- 0x7fabf80042f0 con 0x55c8e461eb40 2024-09-12T03:13:21.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.664+0000 7fac10d92640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e468d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:21.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.664+0000 7fac10d92640 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e468d590 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8507b0d2e3b72858 server_cookie=b217900e098991ea in_seq=0 out_seq=0 2024-09-12T03:13:21.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.664+0000 7fabebfff640 10 client.21273 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fabebfff640 1 -- 172.21.15.16:0/1537625351 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7fabf80042f0 con 0x55c8e461eb40 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fabebfff640 10 client.21273 handle_command_reply: tid=0 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 client.21273 shutdown 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 msgr2=0x55c8e4689570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac040bd740 comp rx=0 tx=0).stop 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 shutdown_connections 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55c8e461eb40 0x55c8e468d590 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8e460f240 0x55c8e4689570 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c8e4604980 0x55c8e4614ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 --2- 172.21.15.16:0/1537625351 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c8e4604060 0x55c8e46147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 >> 172.21.15.16:0/1537625351 conn(0x55c8e46005f0 msgr2=0x55c8e45fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fabebfff640 0 client.21273 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 shutdown_connections 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.718+0000 7fac153cf740 1 -- 172.21.15.16:0/1537625351 wait complete. 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac153cf740 20 client.21273 trim_cache size 0 max 16384 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:21.721 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 39, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 88, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "d4038f41": { 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 88, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:21.722 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 153.7, 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 msgr2=0x7fabfc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 0x7fabfc0a5040 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fac00009fc0 tx=0x7fac00005f90 comp rx=0 tx=0).stop 2024-09-12T03:13:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 msgr2=0x7fac0c19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fac04005cc0 tx=0x7fac040093d0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 shutdown_connections 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fabfc0a2b80 0x7fabfc0a5040 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac0c1082a0 0x7fac0c19efd0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac0c1036e0 0x7fac0c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 --2- 172.21.15.16:0/2991766366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0c102d30 0x7fac0c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.719+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 >> 172.21.15.16:0/2991766366 conn(0x7fac0c0fc7d0 msgr2=0x7fac0c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.720+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 shutdown_connections 2024-09-12T03:13:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:21.720+0000 7fac12ff6640 1 -- 172.21.15.16:0/2991766366 wait complete. 2024-09-12T03:13:21.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:21 smithi184 ceph-mon[35321]: pgmap v1100: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 6.0 MiB/s wr, 258 op/s 2024-09-12T03:13:21.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:21 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:13:21.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:21 smithi033 ceph-mon[35092]: pgmap v1100: 97 pgs: 97 active+clean; 942 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 6.0 MiB/s wr, 258 op/s 2024-09-12T03:13:21.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:21 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:13:22.046 INFO:tasks.workunit.client.0.smithi016.stdout: 10 cleanup 623 sec 2024-09-12T03:13:22.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:13:22.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:13:22.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:22 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:13:22.736 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd4038f41', total seconds elapsed: 37.4 2024-09-12T03:13:22.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:22 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:13:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:22 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:13:22.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:22 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:13:22.996 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:22 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:13:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:22 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:13:22.997 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:22 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:13:23.031 INFO:tasks.workunit.client.0.smithi016.stdout: 0 cleanup 624 sec 2024-09-12T03:13:23.031 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T03:13:23.031 INFO:tasks.workunit.client.0.smithi016.stdout: Operation Count AvgLat MaxLat 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: ---------------------------------------- 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: NTCreateX 451480 6.359 33208.600 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Close 331441 0.003 0.243 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Rename 19131 28.663 32904.532 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Unlink 91270 10.627 33139.343 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Deltree 2 127.337 189.950 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Mkdir 1 0.007 0.007 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Qpathinfo 409382 2.776 799.704 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Qfileinfo 71427 0.002 23.889 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Qfsinfo 75054 0.249 38.711 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Sfileinfo 36756 2.921 30007.886 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: Find 158190 0.326 223.640 2024-09-12T03:13:23.032 INFO:tasks.workunit.client.0.smithi016.stdout: WriteX 223355 0.984 32947.006 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout: ReadX 708348 0.031 188.699 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout: LockX 1474 4.522 64.077 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout: UnlockX 1474 3.807 16.968 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout: Flush 31615 8.318 421.669 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-09-12T03:13:23.033 INFO:tasks.workunit.client.0.smithi016.stdout:Throughput 23.5288 MB/sec 10 clients 10 procs max_latency=30379.233 ms 2024-09-12T03:13:23.033 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-12T03:13:23.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: pgmap v1101: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 5.8 MiB/s wr, 217 op/s 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:13:24.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:23 smithi016 ceph-mon[30925]: scrub summary: idle 2024-09-12T03:13:24.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: pgmap v1101: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 5.8 MiB/s wr, 217 op/s 2024-09-12T03:13:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:13:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:13:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:13:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:13:24.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:23 smithi033 ceph-mon[35092]: scrub summary: idle 2024-09-12T03:13:24.265 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-12T03:13:24.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-12T03:13:24.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: pgmap v1101: 97 pgs: 97 active+clean; 943 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 5.8 MiB/s wr, 217 op/s 2024-09-12T03:13:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:13:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:13:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:13:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:13:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:23 smithi184 ceph-mon[35321]: scrub summary: idle 2024-09-12T03:13:24.586 DEBUG:teuthology.parallel:result is None 2024-09-12T03:13:24.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-12T03:13:24.618 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-12T03:13:24.618 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-12T03:13:24.628 INFO:tasks.quiescer:joining Quiescers 2024-09-12T03:13:24.628 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-12T03:13:24.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:13:24.921 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:25.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:24 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:25.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:24 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:24 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 -- 172.21.15.16:0/526752570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f41036e0 msgr2=0x7f21f4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/526752570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f41036e0 0x7f21f4107b90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009930 tx=0x7f21dc02a420 comp rx=0 tx=0).stop 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 -- 172.21.15.16:0/526752570 shutdown_connections 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/526752570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 0x7f21f410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/526752570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f41036e0 0x7f21f4107b90 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/526752570 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f4102d30 0x7f21f4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 -- 172.21.15.16:0/526752570 >> 172.21.15.16:0/526752570 conn(0x7f21f40fc7d0 msgr2=0x7f21f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 -- 172.21.15.16:0/526752570 shutdown_connections 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.435+0000 7f21f8bc8640 1 -- 172.21.15.16:0/526752570 wait complete. 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 Processor -- start 2024-09-12T03:13:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 -- start start 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 0x7f21f4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 0x7f21f419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21f41120a0 con 0x7f21f4102d30 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21f4111f20 con 0x7f21f41036e0 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f8bc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21f4112220 con 0x7f21f41082a0 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41920/0 (socket says 172.21.15.16:41920) 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 -- 172.21.15.16:0/129392878 learned_addr learned my addr 172.21.15.16:0/129392878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f2d9d640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 0x7f21f419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f1d9b640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 0x7f21f4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 -- 172.21.15.16:0/129392878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 msgr2=0x7f21f419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 0x7f21f419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 -- 172.21.15.16:0/129392878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 msgr2=0x7f21f4198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 0x7f21f4198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f259c640 1 -- 172.21.15.16:0/129392878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f419d5f0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.436+0000 7f21f1d9b640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 0x7f21f4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21f259c640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f21e400ebc0 tx=0x7f21e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4010630 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f419d8e0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e4010cb0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4018bd0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.437+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f41a51c0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.438+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f4103110 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.440+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f21e40107d0 con 0x7f21f4102d30 2024-09-12T03:13:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.441+0000 7f21eb7fe640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 0x7f21e00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.441+0000 7f21f1d9b640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 0x7f21e00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.441+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f21e40cf8e0 con 0x7f21f4102d30 2024-09-12T03:13:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.441+0000 7f21f1d9b640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 0x7f21e00a5040 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.444+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f21e409c850 con 0x7f21f4102d30 2024-09-12T03:13:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.659+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f410a2b0 con 0x7f21f4102d30 2024-09-12T03:13:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.660+0000 7f21eb7fe640 1 -- 172.21.15.16:0/129392878 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11973 (secure 0 0 0) 0x7f21e409c150 con 0x7f21f4102d30 2024-09-12T03:13:25.660 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:13:25.661 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"btime":"2024-09-12T03:13:24:902625+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:13:24.004742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:13:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 41 2024-09-12T03:13:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.662+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 msgr2=0x7f21e00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.662+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 0x7f21e00a5040 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc005a20 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.662+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 msgr2=0x7f21f4198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.662+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f21e400ebc0 tx=0x7f21e400c6a0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 shutdown_connections 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f41082a0 0x7f21f419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21f41036e0 0x7f21f4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f21e00a2b80 0x7f21e00a5040 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 --2- 172.21.15.16:0/129392878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f4102d30 0x7f21f4198640 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 >> 172.21.15.16:0/129392878 conn(0x7f21f40fc7d0 msgr2=0x7f21f4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 shutdown_connections 2024-09-12T03:13:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:25.663+0000 7f21f8bc8640 1 -- 172.21.15.16:0/129392878 wait complete. 2024-09-12T03:13:26.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:25 smithi033 ceph-mon[35092]: pgmap v1102: 97 pgs: 97 active+clean; 911 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.0 MiB/s wr, 193 op/s 2024-09-12T03:13:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:25 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:26.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:25 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/129392878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:26.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:25 smithi184 ceph-mon[35321]: pgmap v1102: 97 pgs: 97 active+clean; 911 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.0 MiB/s wr, 193 op/s 2024-09-12T03:13:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:25 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:26.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:25 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/129392878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:25 smithi016 ceph-mon[30925]: pgmap v1102: 97 pgs: 97 active+clean; 911 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.0 MiB/s wr, 193 op/s 2024-09-12T03:13:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:25 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:25 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/129392878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:26.455 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --cancel --all' 2024-09-12T03:13:26.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- timeout 105 ceph tell mds.24500 quiesce db --cancel --all 2024-09-12T03:13:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:27.136 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:26 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 -- 172.21.15.16:0/261055673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c1036e0 msgr2=0x7f504c10bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 --2- 172.21.15.16:0/261055673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c1036e0 0x7f504c10bbd0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f5040009900 tx=0x7f504002a480 comp rx=0 tx=0).stop 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 -- 172.21.15.16:0/261055673 shutdown_connections 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 --2- 172.21.15.16:0/261055673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 0x7f504c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 --2- 172.21.15.16:0/261055673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c1036e0 0x7f504c10bbd0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.222+0000 7f505133f640 1 --2- 172.21.15.16:0/261055673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c102dc0 0x7f504c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.223+0000 7f505133f640 1 -- 172.21.15.16:0/261055673 >> 172.21.15.16:0/261055673 conn(0x7f504c0fc7a0 msgr2=0x7f504c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.223+0000 7f505133f640 1 -- 172.21.15.16:0/261055673 shutdown_connections 2024-09-12T03:13:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.223+0000 7f505133f640 1 -- 172.21.15.16:0/261055673 wait complete. 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.223+0000 7f505133f640 1 Processor -- start 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.223+0000 7f505133f640 1 -- start start 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c1036e0 0x7f504c100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 0x7f504c1014a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f504c111f70 con 0x7f504c102dc0 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f504c111df0 con 0x7f504c1036e0 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f505133f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1120f0 con 0x7f504c10c1a0 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41936/0 (socket says 172.21.15.16:41936) 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 -- 172.21.15.16:0/679419779 learned_addr learned my addr 172.21.15.16:0/679419779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504a7fc640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c1036e0 0x7f504c100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504b7fe640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 0x7f504c1014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 -- 172.21.15.16:0/679419779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 msgr2=0x7f504c1014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 0x7f504c1014a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 -- 172.21.15.16:0/679419779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c1036e0 msgr2=0x7f504c100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c1036e0 0x7f504c100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 -- 172.21.15.16:0/679419779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504c1a6900 con 0x7f504c102dc0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f504affd640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f503c00d950 tx=0x7f503c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.224+0000 7f5033fff640 1 -- 172.21.15.16:0/679419779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c017070 con 0x7f504c102dc0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.225+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504c1a6bf0 con 0x7f504c102dc0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.225+0000 7f5033fff640 1 -- 172.21.15.16:0/679419779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f503c009c30 con 0x7f504c102dc0 2024-09-12T03:13:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.225+0000 7f5033fff640 1 -- 172.21.15.16:0/679419779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c00f6b0 con 0x7f504c102dc0 2024-09-12T03:13:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.225+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f504c1a7100 con 0x7f504c102dc0 2024-09-12T03:13:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5033fff640 1 -- 172.21.15.16:0/679419779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f503c0109d0 con 0x7f504c102dc0 2024-09-12T03:13:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5053718740 1 Processor -- start 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5053718740 1 -- start start 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5053718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5053718740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.227+0000 7f5053718740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94eb0be0 0x55cf94eb2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f5053718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf94e0a200 con 0x55cf94ea5a00 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f5053718740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf94db0dc0 con 0x55cf94ea6320 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f5053718740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cf94e061c0 con 0x55cf94eb0be0 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41938/0 (socket says 172.21.15.16:41938) 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 -- 172.21.15.16:0/517794312 learned_addr learned my addr 172.21.15.16:0/517794312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f5033fff640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 0x7f50340a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504affd640 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504a7fc640 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94eb0be0 0x55cf94eb2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 0x7f50340a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 -- 172.21.15.16:0/517794312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94eb0be0 msgr2=0x55cf94eb2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94eb0be0 0x55cf94eb2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 -- 172.21.15.16:0/517794312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 msgr2=0x55cf94eb06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 -- 172.21.15.16:0/517794312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf94b7b620 con 0x55cf94ea5a00 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f5033fff640 1 -- 172.21.15.16:0/679419779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f503c0cec80 con 0x7f504c102dc0 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.228+0000 7f504b7fe640 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5044014940 tx=0x7f5044014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9c42b5a74ec78f server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f50317fa640 1 -- 172.21.15.16:0/517794312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5044014050 con 0x55cf94ea5a00 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f50317fa640 1 -- 172.21.15.16:0/517794312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50440141f0 con 0x55cf94ea5a00 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f50317fa640 1 -- 172.21.15.16:0/517794312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5044013640 con 0x55cf94ea5a00 2024-09-12T03:13:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 -- 172.21.15.16:0/517794312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 msgr2=0x55cf94ea5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f504b7fe640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 0x7f50340a5040 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5044021000 tx=0x7f5044020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5044014940 tx=0x7f5044014e00 comp rx=0 tx=0).stop 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 -- 172.21.15.16:0/517794312 shutdown_connections 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94eb0be0 0x55cf94eb2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 --2- 172.21.15.16:0/517794312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94ea5a00 0x55cf94ea5de0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 -- 172.21.15.16:0/517794312 >> 172.21.15.16:0/517794312 conn(0x55cf94ea1f90 msgr2=0x55cf94ea2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 -- 172.21.15.16:0/517794312 shutdown_connections 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.229+0000 7f5053718740 1 -- 172.21.15.16:0/517794312 wait complete. 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 -- start start 2024-09-12T03:13:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 Processor -- start 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94eb0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ead6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf94eb37b0 con 0x55cf94eb0be0 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cf94da4c50 con 0x55cf94ea5a00 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cf94eb3930 con 0x55cf94ea6320 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f5053718740 10 client.0 _ll_register_callbacks cb 0x55cf94db6ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41950/0 (socket says 172.21.15.16:41950) 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f504a7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ead6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f504b7fe640 1 -- 172.21.15.16:0/3088736300 learned_addr learned my addr 172.21.15.16:0/3088736300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.230+0000 7f504affd640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94eb0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 -- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 msgr2=0x55cf94eb0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94eb0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 -- 172.21.15.16:0/3088736300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 msgr2=0x55cf94eadc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 -- 172.21.15.16:0/3088736300 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf94eae460 con 0x55cf94ea6320 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ead6f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5040005940 tx=0x7f5040031920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=813661a3adb133d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f502eff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040035040 con 0x55cf94ea6320 2024-09-12T03:13:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5040004240 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50440141f0 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5040004440 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f5053718740 10 client.30373 fetch_fsmap learned FSMap version 43 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.231+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cf94ebf2b0 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50440141f0 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 43) ==== 8382+0+0 (secure 0 0 0) 0x7f5040004cb0 con 0x55cf94ea6320 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 10 client.30373 fetch_fsmap finished waiting for FSMap version 43 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 10 client.30373 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 20 client.30373 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94ec0770 0x55cf94ec2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 4 client.30373 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf94e17770 con 0x55cf94ec0770 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.232+0000 7f504affd640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94ec0770 0x55cf94ec2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.233+0000 7f502eff5640 10 client.30373 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.233+0000 7f504affd640 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94ec0770 0x55cf94ec2bb0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84bd55984189d9a3 server_cookie=8e347179713c5708 in_seq=0 out_seq=0 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.233+0000 7f502eff5640 1 -- 172.21.15.16:0/3088736300 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cf94e17770 con 0x55cf94ec0770 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.233+0000 7f502eff5640 10 client.30373 handle_command_reply: tid=0 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.233+0000 7f5053718740 1 client.30373 shutdown 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 msgr2=0x55cf94ead6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ead6f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5040005940 tx=0x7f5040031920 comp rx=0 tx=0).stop 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 shutdown_connections 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94ec0770 0x55cf94ec2bb0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eadc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ead6f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f502eff5640 0 client.30373 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 --2- 172.21.15.16:0/3088736300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94eb0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 >> 172.21.15.16:0/3088736300 conn(0x55cf94ea1f90 msgr2=0x55cf94ea11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 shutdown_connections 2024-09-12T03:13:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 1 -- 172.21.15.16:0/3088736300 wait complete. 2024-09-12T03:13:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.234+0000 7f5053718740 20 client.30373 trim_cache size 0 max 16384 2024-09-12T03:13:27.248 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:26 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 Processor -- start 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 -- start start 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94ebca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ec8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1af660 con 0x55cf94eb0be0 2024-09-12T03:13:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1b1200 con 0x55cf94ea5a00 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f5053718740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1afd30 con 0x55cf94ea6320 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41964/0 (socket says 172.21.15.16:41964) 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 -- 172.21.15.16:0/3871730435 learned_addr learned my addr 172.21.15.16:0/3871730435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504affd640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94ebca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ec8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 -- 172.21.15.16:0/3871730435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 msgr2=0x55cf94ec8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ec8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 -- 172.21.15.16:0/3871730435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 msgr2=0x55cf94ebca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94ebca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504b7fe640 1 -- 172.21.15.16:0/3871730435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf94ea1dc0 con 0x55cf94eb0be0 2024-09-12T03:13:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.251+0000 7f504affd640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94ebca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f5044009290 tx=0x7f5044010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c77c761e5cb3f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5030ff9640 1 -- 172.21.15.16:0/3871730435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50440120c0 con 0x55cf94eb0be0 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5030ff9640 1 -- 172.21.15.16:0/3871730435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5044013400 con 0x55cf94eb0be0 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5030ff9640 1 -- 172.21.15.16:0/3871730435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5044005740 con 0x55cf94eb0be0 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5053718740 1 -- 172.21.15.16:0/3871730435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 msgr2=0x55cf94ec89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5053718740 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f5044009290 tx=0x7f5044010000 comp rx=0 tx=0).stop 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5053718740 1 -- 172.21.15.16:0/3871730435 shutdown_connections 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.252+0000 7f5053718740 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94ec89c0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea6320 0x55cf94ec8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 --2- 172.21.15.16:0/3871730435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea5a00 0x55cf94ebca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- 172.21.15.16:0/3871730435 >> 172.21.15.16:0/3871730435 conn(0x55cf94ea1f90 msgr2=0x55cf94ea11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- 172.21.15.16:0/3871730435 shutdown_connections 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- 172.21.15.16:0/3871730435 wait complete. 2024-09-12T03:13:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- start start 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 Processor -- start 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea5a00 0x55cf94eb8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1b0b00 con 0x55cf94eb0be0 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1b0400 con 0x55cf94ea6320 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1ad640 con 0x55cf94ea5a00 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.253+0000 7f5053718740 10 client.0 _ll_register_callbacks cb 0x7f50400315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea5a00 0x55cf94eb8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41978/0 (socket says 172.21.15.16:41978) 2024-09-12T03:13:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 -- 172.21.15.16:0/3748751781 learned_addr learned my addr 172.21.15.16:0/3748751781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 -- 172.21.15.16:0/3748751781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea5a00 msgr2=0x55cf94eb8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea5a00 0x55cf94eb8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 -- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 msgr2=0x55cf94eb90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 -- 172.21.15.16:0/3748751781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf94eb9730 con 0x55cf94eb0be0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504a7fc640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f504b7fe640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5044015010 tx=0x7f5044009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea4ac437fc3ba77 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f502eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5044041060 con 0x55cf94eb0be0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f5044023040 con 0x55cf94eb0be0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5044013400 con 0x55cf94eb0be0 2024-09-12T03:13:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5044005e10 con 0x55cf94eb0be0 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f5053718740 10 client.21297 fetch_fsmap learned FSMap version 43 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.254+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cf94f2b9b0 con 0x55cf94eb0be0 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5044013400 con 0x55cf94eb0be0 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) ==== 8382+0+0 (secure 0 0 0) 0x7f504400f040 con 0x55cf94eb0be0 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 10 client.21297 fetch_fsmap finished waiting for FSMap version 43 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 10 client.21297 resolve_mds: validated gid 24500 aka daemon mds.b 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 20 client.21297 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94f2bf40 0x55cf94f2e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 4 client.21297 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55cf94e11c80 con 0x55cf94f2bf40 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.255+0000 7f504affd640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94f2bf40 0x55cf94f2e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f504affd640 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94f2bf40 0x55cf94f2e360 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=125a8fbb25ba9188 server_cookie=32829313eb0276e4 in_seq=0 out_seq=0 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f502eff5640 10 client.21297 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f502eff5640 1 -- 172.21.15.16:0/3748751781 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55cf94e11c80 con 0x55cf94f2bf40 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f502eff5640 10 client.21297 handle_command_reply: tid=0 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 client.21297 shutdown 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 msgr2=0x55cf94eb5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5044015010 tx=0x7f5044009d90 comp rx=0 tx=0).stop 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 shutdown_connections 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55cf94f2bf40 0x55cf94f2e360 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf94eb0be0 0x55cf94eb5b50 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cf94ea6320 0x55cf94eb90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f502eff5640 0 client.21297 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 --2- 172.21.15.16:0/3748751781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cf94ea5a00 0x55cf94eb8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 >> 172.21.15.16:0/3748751781 conn(0x55cf94ea1f90 msgr2=0x55cf94eaef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 shutdown_connections 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 1 -- 172.21.15.16:0/3748751781 wait complete. 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.256+0000 7f5053718740 20 client.21297 trim_cache size 0 max 16384 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 43, 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24500, 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 88, 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": {} 2024-09-12T03:13:27.263 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 msgr2=0x7f50340a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 0x7f50340a5040 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5044021000 tx=0x7f5044020000 comp rx=0 tx=0).stop 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 msgr2=0x7f504c100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f503c00d950 tx=0x7f503c00de10 comp rx=0 tx=0).stop 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 shutdown_connections 2024-09-12T03:13:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f504c10c1a0 0x7f504c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f504c1036e0 0x7f504c100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50340a2b80 0x7f50340a5040 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 --2- 172.21.15.16:0/679419779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f504c102dc0 0x7f504c100a20 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 >> 172.21.15.16:0/679419779 conn(0x7f504c0fc7a0 msgr2=0x7f504c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 shutdown_connections 2024-09-12T03:13:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:27.257+0000 7f505133f640 1 -- 172.21.15.16:0/679419779 wait complete. 2024-09-12T03:13:27.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:26 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:28.037 INFO:tasks.quiescer:done joining Quiescers 2024-09-12T03:13:28.037 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-12T03:13:28.049 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-12T03:13:28.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:27 smithi033 ceph-mon[35092]: pgmap v1103: 97 pgs: 97 active+clean; 818 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 5.1 MiB/s wr, 226 op/s 2024-09-12T03:13:28.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:27 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:28.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:27 smithi184 ceph-mon[35321]: pgmap v1103: 97 pgs: 97 active+clean; 818 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 5.1 MiB/s wr, 226 op/s 2024-09-12T03:13:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:27 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:27 smithi016 ceph-mon[30925]: pgmap v1103: 97 pgs: 97 active+clean; 818 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 5.1 MiB/s wr, 226 op/s 2024-09-12T03:13:28.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:27 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:29.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:28 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:29.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:28 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:29.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:28 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:29 smithi033 ceph-mon[35092]: pgmap v1104: 97 pgs: 97 active+clean; 740 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-09-12T03:13:30.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:29 smithi033 ceph-mon[35092]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:30.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:29 smithi184 ceph-mon[35321]: pgmap v1104: 97 pgs: 97 active+clean; 740 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-09-12T03:13:30.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:29 smithi184 ceph-mon[35321]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:29 smithi016 ceph-mon[30925]: pgmap v1104: 97 pgs: 97 active+clean; 740 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-09-12T03:13:30.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:29 smithi016 ceph-mon[30925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-12T03:13:32.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:31 smithi033 ceph-mon[35092]: pgmap v1105: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-09-12T03:13:32.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:31 smithi184 ceph-mon[35321]: pgmap v1105: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-09-12T03:13:32.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:31 smithi016 ceph-mon[30925]: pgmap v1105: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-09-12T03:13:34.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:33 smithi033 ceph-mon[35092]: pgmap v1106: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-09-12T03:13:34.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:33 smithi184 ceph-mon[35321]: pgmap v1106: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-09-12T03:13:34.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:33 smithi016 ceph-mon[30925]: pgmap v1106: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-09-12T03:13:36.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:35 smithi033 ceph-mon[35092]: pgmap v1107: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-09-12T03:13:36.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:35 smithi184 ceph-mon[35321]: pgmap v1107: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-09-12T03:13:36.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:35 smithi016 ceph-mon[30925]: pgmap v1107: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-09-12T03:13:38.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:37 smithi033 ceph-mon[35092]: pgmap v1108: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 850 KiB/s wr, 116 op/s 2024-09-12T03:13:38.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:37 smithi184 ceph-mon[35321]: pgmap v1108: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 850 KiB/s wr, 116 op/s 2024-09-12T03:13:38.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:37 smithi016 ceph-mon[30925]: pgmap v1108: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 850 KiB/s wr, 116 op/s 2024-09-12T03:13:40.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:39 smithi033 ceph-mon[35092]: pgmap v1109: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 233 KiB/s wr, 56 op/s 2024-09-12T03:13:40.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:39 smithi184 ceph-mon[35321]: pgmap v1109: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 233 KiB/s wr, 56 op/s 2024-09-12T03:13:40.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:39 smithi016 ceph-mon[30925]: pgmap v1109: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 233 KiB/s wr, 56 op/s 2024-09-12T03:13:41.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-12T03:13:41.614 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:42.010 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:41 smithi016 ceph-mon[30925]: pgmap v1110: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 -- 172.21.15.16:0/3318139880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924102570 msgr2=0x7fe9241029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 --2- 172.21.15.16:0/3318139880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924102570 0x7fe9241029b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fe918009900 tx=0x7fe91802a480 comp rx=0 tx=0).stop 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 -- 172.21.15.16:0/3318139880 shutdown_connections 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 --2- 172.21.15.16:0/3318139880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102ef0 0x7fe92410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 --2- 172.21.15.16:0/3318139880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924102570 0x7fe9241029b0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 --2- 172.21.15.16:0/3318139880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924100e20 0x7fe924101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 -- 172.21.15.16:0/3318139880 >> 172.21.15.16:0/3318139880 conn(0x7fe9240fc7d0 msgr2=0x7fe9240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 -- 172.21.15.16:0/3318139880 shutdown_connections 2024-09-12T03:13:42.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.036+0000 7fe929f90640 1 -- 172.21.15.16:0/3318139880 wait complete. 2024-09-12T03:13:42.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 Processor -- start 2024-09-12T03:13:42.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 -- start start 2024-09-12T03:13:42.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102570 0x7fe924199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 0x7fe9241a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe924110f80 con 0x7fe924100e20 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe924110e00 con 0x7fe924102ef0 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.037+0000 7fe929f90640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe924111100 con 0x7fe924102570 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60262/0 (socket says 172.21.15.16:60262) 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 -- 172.21.15.16:0/1764243385 learned_addr learned my addr 172.21.15.16:0/1764243385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe923fff640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 0x7fe9241a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 -- 172.21.15.16:0/1764243385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102570 msgr2=0x7fe924199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe922ffd640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102570 0x7fe924199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102570 0x7fe924199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 -- 172.21.15.16:0/1764243385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 msgr2=0x7fe9241a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 0x7fe9241a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 -- 172.21.15.16:0/1764243385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9241a48d0 con 0x7fe924100e20 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe923fff640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 0x7fe9241a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe9237fe640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e9f0 tx=0x7fe91400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe920ff9640 1 -- 172.21.15.16:0/1764243385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91400cd80 con 0x7fe924100e20 2024-09-12T03:13:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9241a4aa0 con 0x7fe924100e20 2024-09-12T03:13:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe920ff9640 1 -- 172.21.15.16:0/1764243385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe914010910 con 0x7fe924100e20 2024-09-12T03:13:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9241a4fe0 con 0x7fe924100e20 2024-09-12T03:13:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.038+0000 7fe920ff9640 1 -- 172.21.15.16:0/1764243385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9140188f0 con 0x7fe924100e20 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe920ff9640 1 -- 172.21.15.16:0/1764243385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe914010ac0 con 0x7fe924100e20 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 Processor -- start 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 -- start start 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e63140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe920ff9640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 0x7fe90c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e65a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3dbe2e0 con 0x55f2a3e584a0 2024-09-12T03:13:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3d65ce0 con 0x55f2a3e58dc0 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe92c369740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3dba2a0 con 0x55f2a3e63680 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60272/0 (socket says 172.21.15.16:60272) 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe923fff640 1 -- 172.21.15.16:0/3113346619 learned_addr learned my addr 172.21.15.16:0/3113346619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.041+0000 7fe9237fe640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 0x7fe90c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe920ff9640 1 -- 172.21.15.16:0/1764243385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9140cf500 con 0x7fe924100e20 2024-09-12T03:13:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e65a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe922ffd640 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e63140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 -- 172.21.15.16:0/3113346619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 msgr2=0x55f2a3e65a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e65a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 -- 172.21.15.16:0/3113346619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 msgr2=0x55f2a3e63140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e63140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 -- 172.21.15.16:0/3113346619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a3d4a440 con 0x55f2a3e584a0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe9237fe640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 0x7fe90c0a50c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe914098170 tx=0x7fe9140911f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe923fff640 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe91c014ad0 tx=0x7fe91c009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71163f7c12ec8a16 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe909ffb640 1 -- 172.21.15.16:0/3113346619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0142f0 con 0x55f2a3e584a0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe909ffb640 1 -- 172.21.15.16:0/3113346619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91c00d860 con 0x55f2a3e584a0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.042+0000 7fe909ffb640 1 -- 172.21.15.16:0/3113346619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c005310 con 0x55f2a3e584a0 2024-09-12T03:13:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 -- 172.21.15.16:0/3113346619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 msgr2=0x55f2a3e58880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe91c014ad0 tx=0x7fe91c009670 comp rx=0 tx=0).stop 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 -- 172.21.15.16:0/3113346619 shutdown_connections 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e65a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e63140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 --2- 172.21.15.16:0/3113346619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e58880 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 -- 172.21.15.16:0/3113346619 >> 172.21.15.16:0/3113346619 conn(0x55f2a3e54a30 msgr2=0x55f2a3e54e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 -- 172.21.15.16:0/3113346619 shutdown_connections 2024-09-12T03:13:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.043+0000 7fe92c369740 1 -- 172.21.15.16:0/3113346619 wait complete. 2024-09-12T03:13:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 -- start start 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 Processor -- start 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e584a0 0x55f2a3e62f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e601e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3e66250 con 0x55f2a3e63680 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3d79050 con 0x55f2a3e58dc0 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3e663d0 con 0x55f2a3e584a0 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe92c369740 10 client.0 _ll_register_callbacks cb 0x55f2a3db9160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60280/0 (socket says 172.21.15.16:60280) 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe923fff640 1 -- 172.21.15.16:0/4053367647 learned_addr learned my addr 172.21.15.16:0/4053367647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe922ffd640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.044+0000 7fe923fff640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e584a0 0x55f2a3e62f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 -- 172.21.15.16:0/4053367647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e584a0 msgr2=0x55f2a3e62f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e584a0 0x55f2a3e62f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 -- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 msgr2=0x55f2a3e601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 -- 172.21.15.16:0/4053367647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a3b2f3c0 con 0x55f2a3e63680 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe923fff640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00a010 tx=0x7fe91c0146d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f60243f0458c3c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c004c40 con 0x55f2a3e63680 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe91c00d860 con 0x55f2a3e63680 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe91c004de0 con 0x55f2a3e63680 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe91c005090 con 0x55f2a3e63680 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe92c369740 10 client.21309 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f2a3e71ba0 con 0x55f2a3e63680 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.045+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91c00d860 con 0x55f2a3e63680 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fe91c02f020 con 0x55f2a3e63680 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 10 client.21309 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 10 client.21309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 20 client.21309 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3e72ef0 0x55f2a3e752e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 4 client.21309 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a3dcb850 con 0x55f2a3e72ef0 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe922ffd640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3e72ef0 0x55f2a3e752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe922ffd640 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3e72ef0 0x55f2a3e752e0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7dd81611f3f668a server_cookie=f5ae31f927f0ab1e in_seq=0 out_seq=0 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.046+0000 7fe902ffd640 10 client.21309 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe902ffd640 1 -- 172.21.15.16:0/4053367647 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f2a3dcb850 con 0x55f2a3e72ef0 2024-09-12T03:13:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe902ffd640 10 client.21309 handle_command_reply: tid=0 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 client.21309 shutdown 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 msgr2=0x55f2a3e60720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00a010 tx=0x7fe91c0146d0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 shutdown_connections 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3e72ef0 0x55f2a3e752e0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3e60720 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe902ffd640 0 client.21309 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 --2- 172.21.15.16:0/4053367647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e584a0 0x55f2a3e62f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 >> 172.21.15.16:0/4053367647 conn(0x55f2a3e54a30 msgr2=0x55f2a3e5a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 shutdown_connections 2024-09-12T03:13:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.047+0000 7fe92c369740 1 -- 172.21.15.16:0/4053367647 wait complete. 2024-09-12T03:13:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.048+0000 7fe92c369740 20 client.21309 trim_cache size 0 max 16384 2024-09-12T03:13:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 Processor -- start 2024-09-12T03:13:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 -- start start 2024-09-12T03:13:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e7a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e7ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241ad600 con 0x55f2a3e584a0 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241af200 con 0x55f2a3e58dc0 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe92c369740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241adcd0 con 0x55f2a3e63680 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60286/0 (socket says 172.21.15.16:60286) 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 -- 172.21.15.16:0/1710843438 learned_addr learned my addr 172.21.15.16:0/1710843438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe922ffd640 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e7a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 -- 172.21.15.16:0/1710843438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 msgr2=0x55f2a3e7ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e7ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 -- 172.21.15.16:0/1710843438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 msgr2=0x55f2a3e7a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e7a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 -- 172.21.15.16:0/1710843438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a3e54860 con 0x55f2a3e584a0 2024-09-12T03:13:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe922ffd640 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e7a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:42.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.065+0000 7fe923fff640 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe91c031af0 tx=0x7fe91c009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6c61b26b54f3c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe9097fa640 1 -- 172.21.15.16:0/1710843438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c00d860 con 0x55f2a3e584a0 2024-09-12T03:13:42.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe9097fa640 1 -- 172.21.15.16:0/1710843438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91c027040 con 0x55f2a3e584a0 2024-09-12T03:13:42.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe9097fa640 1 -- 172.21.15.16:0/1710843438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c005490 con 0x55f2a3e584a0 2024-09-12T03:13:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 -- 172.21.15.16:0/1710843438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 msgr2=0x55f2a3e5f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe91c031af0 tx=0x7fe91c009d70 comp rx=0 tx=0).stop 2024-09-12T03:13:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 -- 172.21.15.16:0/1710843438 shutdown_connections 2024-09-12T03:13:42.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e63680 0x55f2a3e7ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e7a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 --2- 172.21.15.16:0/1710843438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e584a0 0x55f2a3e5f990 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 -- 172.21.15.16:0/1710843438 >> 172.21.15.16:0/1710843438 conn(0x55f2a3e54a30 msgr2=0x55f2a3e53b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 -- 172.21.15.16:0/1710843438 shutdown_connections 2024-09-12T03:13:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.066+0000 7fe92c369740 1 -- 172.21.15.16:0/1710843438 wait complete. 2024-09-12T03:13:42.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 -- start start 2024-09-12T03:13:42.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 Processor -- start 2024-09-12T03:13:42.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e584a0 0x55f2a3e68a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e68f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241aead0 con 0x55f2a3e63680 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241ae3d0 con 0x55f2a3e584a0 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241ab5e0 con 0x55f2a3e58dc0 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe92c369740 10 client.0 _ll_register_callbacks cb 0x7fe914005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe923fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60288/0 (socket says 172.21.15.16:60288) 2024-09-12T03:13:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe923fff640 1 -- 172.21.15.16:0/2947161820 learned_addr learned my addr 172.21.15.16:0/2947161820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.067+0000 7fe923fff640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e584a0 0x55f2a3e68a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe922ffd640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e68f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 -- 172.21.15.16:0/2947161820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e58dc0 msgr2=0x55f2a3e68f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e68f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 -- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e584a0 msgr2=0x55f2a3e68a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e584a0 0x55f2a3e68a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 -- 172.21.15.16:0/2947161820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a3edbdf0 con 0x55f2a3e63680 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe923fff640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fe91c031510 tx=0x7fe91c031a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7502e3104d08d48a server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:42.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c031760 con 0x55f2a3e63680 2024-09-12T03:13:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe91c005670 con 0x55f2a3e63680 2024-09-12T03:13:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe91c027040 con 0x55f2a3e63680 2024-09-12T03:13:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe91c00d860 con 0x55f2a3e63680 2024-09-12T03:13:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe92c369740 10 client.21315 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f2a3edcf80 con 0x55f2a3e63680 2024-09-12T03:13:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.068+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91c027040 con 0x55f2a3e63680 2024-09-12T03:13:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fe91c0226e0 con 0x55f2a3e63680 2024-09-12T03:13:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 10 client.21315 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 10 client.21315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 20 client.21315 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3edd6f0 0x55f2a3edfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 4 client.21315 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2a3dc5d60 con 0x55f2a3edd6f0 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe922ffd640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3edd6f0 0x55f2a3edfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe922ffd640 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3edd6f0 0x55f2a3edfb10 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cfceddb4ed68086 server_cookie=70d00f0593d5e40d in_seq=0 out_seq=0 2024-09-12T03:13:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.069+0000 7fe902ffd640 10 client.21315 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe902ffd640 1 -- 172.21.15.16:0/2947161820 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f2a3dc5d60 con 0x55f2a3edd6f0 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe902ffd640 10 client.21315 handle_command_reply: tid=0 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 client.21315 shutdown 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 msgr2=0x55f2a3edb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fe91c031510 tx=0x7fe91c031a00 comp rx=0 tx=0).stop 2024-09-12T03:13:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 shutdown_connections 2024-09-12T03:13:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f2a3edd6f0 0x55f2a3edfb10 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a3e63680 0x55f2a3edb8b0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f2a3e58dc0 0x55f2a3e68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 --2- 172.21.15.16:0/2947161820 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f2a3e584a0 0x55f2a3e68a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 >> 172.21.15.16:0/2947161820 conn(0x55f2a3e54a30 msgr2=0x55f2a3e61a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe902ffd640 0 client.21315 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 shutdown_connections 2024-09-12T03:13:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 1 -- 172.21.15.16:0/2947161820 wait complete. 2024-09-12T03:13:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe92c369740 20 client.21315 trim_cache size 0 max 16384 2024-09-12T03:13:42.086 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:42.086 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-09-12T03:13:42.086 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-09-12T03:13:42.086 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 msgr2=0x7fe90c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 0x7fe90c0a50c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe914098170 tx=0x7fe9140911f0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 msgr2=0x7fe924199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.070+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e9f0 tx=0x7fe91400eeb0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 shutdown_connections 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe924102ef0 0x7fe9241a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe924102570 0x7fe924199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe90c0a2c00 0x7fe90c0a50c0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 --2- 172.21.15.16:0/1764243385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe924100e20 0x7fe924199310 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 >> 172.21.15.16:0/1764243385 conn(0x7fe9240fc7d0 msgr2=0x7fe9240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 shutdown_connections 2024-09-12T03:13:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:42.071+0000 7fe929f90640 1 -- 172.21.15.16:0/1764243385 wait complete. 2024-09-12T03:13:42.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:41 smithi033 ceph-mon[35092]: pgmap v1110: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-09-12T03:13:42.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:41 smithi184 ceph-mon[35321]: pgmap v1110: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 31 op/s 2024-09-12T03:13:42.955 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-12T03:13:42.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:13:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- 172.21.15.16:0/1504794454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102ef0 msgr2=0x7fdac010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 --2- 172.21.15.16:0/1504794454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102ef0 0x7fdac010da40 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009900 tx=0x7fdab802a480 comp rx=0 tx=0).stop 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- 172.21.15.16:0/1504794454 shutdown_connections 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 --2- 172.21.15.16:0/1504794454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102ef0 0x7fdac010da40 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 --2- 172.21.15.16:0/1504794454 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0102570 0x7fdac01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 --2- 172.21.15.16:0/1504794454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0100e20 0x7fdac0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- 172.21.15.16:0/1504794454 >> 172.21.15.16:0/1504794454 conn(0x7fdac00fc7d0 msgr2=0x7fdac00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- 172.21.15.16:0/1504794454 shutdown_connections 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- 172.21.15.16:0/1504794454 wait complete. 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 Processor -- start 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.700+0000 7fdac7e98640 1 -- start start 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0100e20 0x7fdac0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0102ef0 0x7fdac0197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdac0111020 con 0x7fdac0102570 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdac0110ea0 con 0x7fdac0100e20 2024-09-12T03:13:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac7e98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdac01111a0 con 0x7fdac0102ef0 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60308/0 (socket says 172.21.15.16:60308) 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 -- 172.21.15.16:0/3593802286 learned_addr learned my addr 172.21.15.16:0/3593802286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5c34640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0100e20 0x7fdac0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac6435640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0102ef0 0x7fdac0197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 -- 172.21.15.16:0/3593802286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0102ef0 msgr2=0x7fdac0197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0102ef0 0x7fdac0197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 -- 172.21.15.16:0/3593802286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0100e20 msgr2=0x7fdac0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0100e20 0x7fdac0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 -- 172.21.15.16:0/3593802286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac01a6ab0 con 0x7fdac0102570 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.701+0000 7fdac5433640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdab000b4a0 tx=0x7fdab000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.702+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0014070 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.702+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac01a6da0 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.702+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdab0010af0 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.702+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0017d10 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.702+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac01a72e0 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.703+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac0101280 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.704+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdab0017070 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.704+0000 7fdab6ffd640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 0x7fdaac0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.705+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fdab00ce740 con 0x7fdac0102570 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.705+0000 7fdac5c34640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 0x7fdaac0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.705+0000 7fdac5c34640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 0x7fdaac0a5040 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00a400 tx=0x7fdabc00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.708+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdab009b6b0 con 0x7fdac0102570 2024-09-12T03:13:43.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.919+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdac006a560 con 0x7fdac0102570 2024-09-12T03:13:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.920+0000 7fdab6ffd640 1 -- 172.21.15.16:0/3593802286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11978 (secure 0 0 0) 0x7fdac006a560 con 0x7fdac0102570 2024-09-12T03:13:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:13:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"btime":"2024-09-12T03:13:28:924710+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:13:28.852245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:13:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-09-12T03:13:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 msgr2=0x7fdaac0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 0x7fdaac0a5040 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00a400 tx=0x7fdabc00d040 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 msgr2=0x7fdac0197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdab000b4a0 tx=0x7fdab000b960 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 shutdown_connections 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdac0102ef0 0x7fdac0197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fdaac0a2b80 0x7fdaac0a5040 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0102570 0x7fdac0197780 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 --2- 172.21.15.16:0/3593802286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdac0100e20 0x7fdac0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 >> 172.21.15.16:0/3593802286 conn(0x7fdac00fc7d0 msgr2=0x7fdac00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 shutdown_connections 2024-09-12T03:13:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:43.923+0000 7fdac7e98640 1 -- 172.21.15.16:0/3593802286 wait complete. 2024-09-12T03:13:44.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:43 smithi016 ceph-mon[30925]: pgmap v1111: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:13:44.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:43 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3593802286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:43 smithi033 ceph-mon[35092]: pgmap v1111: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:13:44.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:43 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3593802286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:43 smithi184 ceph-mon[35321]: pgmap v1111: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:13:44.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:43 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3593802286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:13:44.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-12T03:13:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:45.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 -- 172.21.15.16:0/1927572508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 msgr2=0x7fb0d810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 --2- 172.21.15.16:0/1927572508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d810e9d0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009cd0 tx=0x7fb0d002a8e0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 -- 172.21.15.16:0/1927572508 shutdown_connections 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 --2- 172.21.15.16:0/1927572508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d810e9d0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 --2- 172.21.15.16:0/1927572508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 0x7fb0d8101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 --2- 172.21.15.16:0/1927572508 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 0x7fb0d8107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 -- 172.21.15.16:0/1927572508 >> 172.21.15.16:0/1927572508 conn(0x7fb0d80fd710 msgr2=0x7fb0d80ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 -- 172.21.15.16:0/1927572508 shutdown_connections 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.428+0000 7fb0dd803640 1 -- 172.21.15.16:0/1927572508 wait complete. 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 Processor -- start 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 -- start start 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 0x7fb0d81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 0x7fb0d819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8111f00 con 0x7fb0d8102390 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8111d80 con 0x7fb0d8107b00 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0dd803640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8112080 con 0x7fb0d81019f0 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.429+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49714/0 (socket says 172.21.15.16:49714) 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2970300079 learned_addr learned my addr 172.21.15.16:0/2970300079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 0x7fb0d81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 0x7fb0d819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2970300079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 msgr2=0x7fb0d81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 0x7fb0d81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2970300079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 msgr2=0x7fb0d819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 0x7fb0d819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2970300079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d819d370 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00c890 tx=0x7fb0cc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0dc801640 1 -- 172.21.15.16:0/2970300079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc004220 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0dc801640 1 -- 172.21.15.16:0/2970300079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc0043c0 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d819d660 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0dc801640 1 -- 172.21.15.16:0/2970300079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0049e0 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.430+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d81a4f40 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dc801640 1 -- 172.21.15.16:0/2970300079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb0cc0079b0 con 0x7fb0d8102390 2024-09-12T03:13:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 Processor -- start 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 -- start start 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x561826092270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x56182609f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561825ff82f0 con 0x5618260927b0 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561825f9fcf0 con 0x56182609d550 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.433+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561825ff42b0 con 0x561826093d20 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x561826092270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49720/0 (socket says 172.21.15.16:49720) 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/261919251 learned_addr learned my addr 172.21.15.16:0/261919251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x561826092270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.16:34676/0 (socket says 172.21.15.16:34676) 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0dc801640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 0x7fb0c00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 0x7fb0c00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0dc801640 1 -- 172.21.15.16:0/2970300079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0cc0cefa0 con 0x7fb0d8102390 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x56182609f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/261919251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 msgr2=0x561826092270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x561826092270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/261919251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 msgr2=0x56182609f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x56182609f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/261919251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561825f843a0 con 0x5618260927b0 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x56182609f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800bc50 tx=0x7fb0c80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec24f2c52b116c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0bdffb640 1 -- 172.21.15.16:0/261919251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c801c480 con 0x5618260927b0 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.434+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 0x7fb0c00a50c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0004ab0 tx=0x7fb0d002a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0bdffb640 1 -- 172.21.15.16:0/261919251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c801cb00 con 0x5618260927b0 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0bdffb640 1 -- 172.21.15.16:0/261919251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8011b90 con 0x5618260927b0 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/261919251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 msgr2=0x561826092bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800bc50 tx=0x7fb0c80107a0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/261919251 shutdown_connections 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x56182609f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x561826092bf0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/261919251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x561826092270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/261919251 >> 172.21.15.16:0/261919251 conn(0x56182605b310 msgr2=0x56182608ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/261919251 shutdown_connections 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.435+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/261919251 wait complete. 2024-09-12T03:13:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 -- start start 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 Processor -- start 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618260927b0 0x56182609cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56182609d550 0x56182609a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5618260a0250 con 0x561826093d20 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561825fb3060 con 0x5618260927b0 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5618260a03d0 con 0x56182609d550 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.436+0000 7fb0dfbdc740 10 client.0 _ll_register_callbacks cb 0x561825d068c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49724/0 (socket says 172.21.15.16:49724) 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 -- 172.21.15.16:0/3140613813 learned_addr learned my addr 172.21.15.16:0/3140613813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618260927b0 0x56182609cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56182609d550 0x56182609a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 -- 172.21.15.16:0/3140613813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56182609d550 msgr2=0x56182609a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56182609d550 0x56182609a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 -- 172.21.15.16:0/3140613813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618260927b0 msgr2=0x56182609cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618260927b0 0x56182609cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 -- 172.21.15.16:0/3140613813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561825d693c0 con 0x561826093d20 2024-09-12T03:13:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc09b850 tx=0x7fb0cc09b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddaf1a472f13a30d server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc01d030 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0cc09bb20 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c801cb00 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb0cc0a4480 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.437+0000 7fb0dfbdc740 10 client.21333 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5618260abd30 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c801cb00 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fb0cc086070 con 0x561826093d20 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 10 client.21333 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 10 client.21333 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:13:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 20 client.21333 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5618260ad140 0x5618260af570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 4 client.21333 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561826005860 con 0x5618260ad140 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5618260ad140 0x5618260af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0b6ffd640 10 client.21333 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.438+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5618260ad140 0x5618260af570 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=332eeb42cdf4022b server_cookie=557f5c3be304bba4 in_seq=0 out_seq=0 2024-09-12T03:13:45.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/3140613813 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561826005860 con 0x5618260ad140 2024-09-12T03:13:45.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0b6ffd640 10 client.21333 handle_command_reply: tid=0 2024-09-12T03:13:45.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 client.21333 shutdown 2024-09-12T03:13:45.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 msgr2=0x56182609a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc09b850 tx=0x7fb0cc09b4c0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 shutdown_connections 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56182609d550 0x56182609a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x56182609a250 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x5618260ad140 0x5618260af570 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/3140613813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618260927b0 0x56182609cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0b6ffd640 0 client.21333 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.439+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 >> 172.21.15.16:0/3140613813 conn(0x56182605b310 msgr2=0x561826094b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.440+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 shutdown_connections 2024-09-12T03:13:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.440+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/3140613813 wait complete. 2024-09-12T03:13:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.440+0000 7fb0dfbdc740 20 client.21333 trim_cache size 0 max 16384 2024-09-12T03:13:45.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.452+0000 7fb0dfbdc740 1 Processor -- start 2024-09-12T03:13:45.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 -- start start 2024-09-12T03:13:45.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5618260927b0 0x56182609c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ad2e0 con 0x561826093d20 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81aeee0 con 0x56182609d550 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ad9b0 con 0x5618260927b0 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49734/0 (socket says 172.21.15.16:49734) 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2184408045 learned_addr learned my addr 172.21.15.16:0/2184408045 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5618260927b0 0x56182609c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2184408045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5618260927b0 msgr2=0x56182609c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5618260927b0 0x56182609c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2184408045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 msgr2=0x5618260b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 -- 172.21.15.16:0/2184408045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56182609ad50 con 0x561826093d20 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc098240 tx=0x7fb0cc099dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b959aa5a7621458 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.453+0000 7fb0bd7fa640 1 -- 172.21.15.16:0/2184408045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc01d070 con 0x561826093d20 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0bd7fa640 1 -- 172.21.15.16:0/2184408045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc084e50 con 0x561826093d20 2024-09-12T03:13:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0bd7fa640 1 -- 172.21.15.16:0/2184408045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0a4250 con 0x561826093d20 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/2184408045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 msgr2=0x5618260b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc098240 tx=0x7fb0cc099dd0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/2184408045 shutdown_connections 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561826093d20 0x5618260b4e40 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/2184408045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5618260927b0 0x56182609c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/2184408045 >> 172.21.15.16:0/2184408045 conn(0x56182605b310 msgr2=0x56182609bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/2184408045 shutdown_connections 2024-09-12T03:13:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.454+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/2184408045 wait complete. 2024-09-12T03:13:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 -- start start 2024-09-12T03:13:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 Processor -- start 2024-09-12T03:13:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x5618260a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260a2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ae7b0 con 0x5618260927b0 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ae0b0 con 0x56182609d550 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d81ab2c0 con 0x561826093d20 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0dfbdc740 10 client.0 _ll_register_callbacks cb 0x7fb0cc004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49750/0 (socket says 172.21.15.16:49750) 2024-09-12T03:13:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/673381740 learned_addr learned my addr 172.21.15.16:0/673381740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x5618260a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.455+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/673381740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 msgr2=0x5618260a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x5618260a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/673381740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 msgr2=0x5618260a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 -- 172.21.15.16:0/673381740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618260a2e60 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d77fe640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0d6ffd640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fb0c801e010 tx=0x7fb0c800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd65ed72a7fbefd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8005450 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fb0c80055f0 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0cc084e50 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb0c800b850 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0dfbdc740 10 client.21339 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x561826118940 con 0x5618260927b0 2024-09-12T03:13:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.456+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc084e50 con 0x5618260927b0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fb0c802f070 con 0x5618260927b0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 10 client.21339 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 10 client.21339 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 20 client.21339 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561826118ed0 0x56182611b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 4 client.21339 mds_command: new command op to 14595 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 --> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561825fffd70 con 0x561826118ed0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561826118ed0 0x56182611b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0b6ffd640 10 client.21339 ms_handle_connect on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0d67fc640 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561826118ed0 0x56182611b2c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e4664ca1decc8799 server_cookie=bbdb0d41651d2751 in_seq=0 out_seq=0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0b6ffd640 1 -- 172.21.15.16:0/673381740 <== mds.4 v2:172.21.15.16:6840/1593859053 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561825fffd70 con 0x561826118ed0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0b6ffd640 10 client.21339 handle_command_reply: tid=0 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 1 client.21339 shutdown 2024-09-12T03:13:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 msgr2=0x5618260a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.457+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fb0c801e010 tx=0x7fb0c800b7e0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 shutdown_connections 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:6840/1593859053,v1:172.21.15.16:6841/1593859053] conn(0x561826118ed0 0x56182611b2c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56182609d550 0x5618260a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561826093d20 0x5618260a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 --2- 172.21.15.16:0/673381740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5618260927b0 0x5618260a5840 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 >> 172.21.15.16:0/673381740 conn(0x56182605b310 msgr2=0x56182609c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0b6ffd640 0 client.21339 ms_handle_reset on v2:172.21.15.16:6840/1593859053 2024-09-12T03:13:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 shutdown_connections 2024-09-12T03:13:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 1 -- 172.21.15.16:0/673381740 wait complete. 2024-09-12T03:13:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dfbdc740 20 client.21339 trim_cache size 0 max 16384 2024-09-12T03:13:45.468 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 msgr2=0x7fb0c00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 0x7fb0c00a50c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0004ab0 tx=0x7fb0d002a8e0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 msgr2=0x7fb0d8198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00c890 tx=0x7fb0cc00cd50 comp rx=0 tx=0).stop 2024-09-12T03:13:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.458+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 shutdown_connections 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0d8107b00 0x7fb0d819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fb0c00a2c00 0x7fb0c00a50c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d8102390 0x7fb0d8198900 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 --2- 172.21.15.16:0/2970300079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0d81019f0 0x7fb0d81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 >> 172.21.15.16:0/2970300079 conn(0x7fb0d80fd710 msgr2=0x7fb0d80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 shutdown_connections 2024-09-12T03:13:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:45.459+0000 7fb0dd803640 1 -- 172.21.15.16:0/2970300079 wait complete. 2024-09-12T03:13:46.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-12T03:13:46.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:45 smithi016 ceph-mon[30925]: pgmap v1112: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-12T03:13:46.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:45 smithi033 ceph-mon[35092]: pgmap v1112: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-12T03:13:46.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:45 smithi184 ceph-mon[35321]: pgmap v1112: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-12T03:13:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 -- 172.21.15.16:0/2865373192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102ef0 msgr2=0x7fe9cc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/2865373192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc10da40 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009ca0 tx=0x7fe9c402a950 comp rx=0 tx=0).stop 2024-09-12T03:13:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 -- 172.21.15.16:0/2865373192 shutdown_connections 2024-09-12T03:13:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/2865373192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc10da40 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/2865373192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc102570 0x7fe9cc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/2865373192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc100e20 0x7fe9cc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 -- 172.21.15.16:0/2865373192 >> 172.21.15.16:0/2865373192 conn(0x7fe9cc0fc7d0 msgr2=0x7fe9cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 -- 172.21.15.16:0/2865373192 shutdown_connections 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.785+0000 7fe9d3c36640 1 -- 172.21.15.16:0/2865373192 wait complete. 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 Processor -- start 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 -- start start 2024-09-12T03:13:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 0x7fe9cc199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc1a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc111070 con 0x7fe9cc102570 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc110ef0 con 0x7fe9cc100e20 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.786+0000 7fe9d3c36640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1111f0 con 0x7fe9cc102ef0 2024-09-12T03:13:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49768/0 (socket says 172.21.15.16:49768) 2024-09-12T03:13:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 -- 172.21.15.16:0/3576033966 learned_addr learned my addr 172.21.15.16:0/3576033966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 0x7fe9cc199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc1a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 -- 172.21.15.16:0/3576033966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 msgr2=0x7fe9cc1a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc1a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 -- 172.21.15.16:0/3576033966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 msgr2=0x7fe9cc199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 0x7fe9cc199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 -- 172.21.15.16:0/3576033966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc1a4ac0 con 0x7fe9cc102570 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 0x7fe9cc199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc1a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b4a0 tx=0x7fe9bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9c2ffd640 1 -- 172.21.15.16:0/3576033966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc014070 con 0x7fe9cc102570 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1a4c90 con 0x7fe9cc102570 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9c2ffd640 1 -- 172.21.15.16:0/3576033966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9bc010af0 con 0x7fe9cc102570 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9c2ffd640 1 -- 172.21.15.16:0/3576033966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc017d10 con 0x7fe9cc102570 2024-09-12T03:13:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.787+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc1a5180 con 0x7fe9cc102570 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9c2ffd640 1 -- 172.21.15.16:0/3576033966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe9bc017070 con 0x7fe9cc102570 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 Processor -- start 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 -- start start 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b4d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b5eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631d1ab32f0 con 0x5631d1b4d5d0 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5631d1a5acf0 con 0x5631d1b4eb40 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d600f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5631d1aaf2b0 con 0x5631d1b5c780 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d19d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.790+0000 7fe9d19d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49772/0 (socket says 172.21.15.16:49772) 2024-09-12T03:13:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 -- 172.21.15.16:0/4068242572 learned_addr learned my addr 172.21.15.16:0/4068242572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9c2ffd640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 0x7fe9b80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b5eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b4d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 0x7fe9b80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 -- 172.21.15.16:0/4068242572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 msgr2=0x5631d1b5eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b5eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 -- 172.21.15.16:0/4068242572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 msgr2=0x5631d1b4d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b4d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9c2ffd640 1 -- 172.21.15.16:0/3576033966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9bc0ce450 con 0x7fe9cc102570 2024-09-12T03:13:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 -- 172.21.15.16:0/4068242572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d1a3f3a0 con 0x5631d1b4d5d0 2024-09-12T03:13:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b4d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800aa10 tx=0x7fe9c800aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d9e35f18591878 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9b7fff640 1 -- 172.21.15.16:0/4068242572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8015070 con 0x5631d1b4d5d0 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 0x7fe9b80a5040 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009a60 tx=0x7fe9c40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9b7fff640 1 -- 172.21.15.16:0/4068242572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c800cc60 con 0x5631d1b4d5d0 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.791+0000 7fe9b7fff640 1 -- 172.21.15.16:0/4068242572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80106c0 con 0x5631d1b4d5d0 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 -- 172.21.15.16:0/4068242572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 msgr2=0x5631d1b4da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800aa10 tx=0x7fe9c800aed0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 -- 172.21.15.16:0/4068242572 shutdown_connections 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b5eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b4da10 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 --2- 172.21.15.16:0/4068242572 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b4d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 -- 172.21.15.16:0/4068242572 >> 172.21.15.16:0/4068242572 conn(0x5631d1b16310 msgr2=0x5631d1b49c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 -- 172.21.15.16:0/4068242572 shutdown_connections 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.792+0000 7fe9d600f740 1 -- 172.21.15.16:0/4068242572 wait complete. 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 -- start start 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 Processor -- start 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 0x5631d1b5c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631d1b5f480 con 0x5631d1b5c780 2024-09-12T03:13:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5631d1a6e060 con 0x5631d1b4eb40 2024-09-12T03:13:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5631d1b5f600 con 0x5631d1b4d5d0 2024-09-12T03:13:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.793+0000 7fe9d600f740 10 client.0 _ll_register_callbacks cb 0x5631d17c18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d21d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d21d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49780/0 (socket says 172.21.15.16:49780) 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d21d3640 1 -- 172.21.15.16:0/727498936 learned_addr learned my addr 172.21.15.16:0/727498936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54258/0 (socket says 172.21.15.16:54258) 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 0x5631d1b5c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 -- 172.21.15.16:0/727498936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 msgr2=0x5631d1b5c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 0x5631d1b5c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 -- 172.21.15.16:0/727498936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 msgr2=0x5631d1b598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 -- 172.21.15.16:0/727498936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d18243c0 con 0x5631d1b4eb40 2024-09-12T03:13:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 0x5631d1b5c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x5631d1b4e070 tx=0x7fe9bc069c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf382060fecf5a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc065da0 con 0x5631d1b4eb40 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9c800cc60 con 0x5631d1b4eb40 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.794+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9bc017360 con 0x5631d1b4eb40 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe9bc09b3c0 con 0x5631d1b4eb40 2024-09-12T03:13:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 10 client.30455 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5631d1b6aef0 con 0x5631d1b4eb40 2024-09-12T03:13:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c800cc60 con 0x5631d1b4eb40 2024-09-12T03:13:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fe9bc097330 con 0x5631d1b4eb40 2024-09-12T03:13:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 10 client.30455 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 10 client.30455 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 20 client.30455 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1b6c300 0x5631d1b6e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 4 client.30455 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631d1ac0860 con 0x5631d1b6c300 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.795+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1b6c300 0x5631d1b6e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.796+0000 7fe9b57fa640 10 client.30455 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.796+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1b6c300 0x5631d1b6e780 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39d57d1076b656e4 server_cookie=b7cb8d654a34998f in_seq=0 out_seq=0 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9b57fa640 1 -- 172.21.15.16:0/727498936 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5631d1ac0860 con 0x5631d1b6c300 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9b57fa640 10 client.30455 handle_command_reply: tid=0 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 client.30455 shutdown 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 msgr2=0x5631d1b59360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x5631d1b4e070 tx=0x7fe9bc069c40 comp rx=0 tx=0).stop 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 shutdown_connections 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1b6c300 0x5631d1b6e780 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b5c780 0x5631d1b598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4eb40 0x5631d1b59360 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9b57fa640 0 client.30455 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 --2- 172.21.15.16:0/727498936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4d5d0 0x5631d1b5c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 >> 172.21.15.16:0/727498936 conn(0x5631d1b16310 msgr2=0x5631d1b4f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 shutdown_connections 2024-09-12T03:13:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 1 -- 172.21.15.16:0/727498936 wait complete. 2024-09-12T03:13:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.797+0000 7fe9d600f740 20 client.30455 trim_cache size 0 max 16384 2024-09-12T03:13:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 Processor -- start 2024-09-12T03:13:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 -- start start 2024-09-12T03:13:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4eb40 0x5631d1b49700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b5c780 0x5631d1b49c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1ab470 con 0x5631d1b4d5d0 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1ad070 con 0x5631d1b5c780 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d600f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1abb40 con 0x5631d1b4eb40 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d19d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d19d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49784/0 (socket says 172.21.15.16:49784) 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3433276663 learned_addr learned my addr 172.21.15.16:0/3433276663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b5c780 0x5631d1b49c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4eb40 0x5631d1b49700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3433276663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4eb40 msgr2=0x5631d1b49700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.811+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4eb40 0x5631d1b49700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3433276663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b5c780 msgr2=0x5631d1b49c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b5c780 0x5631d1b49c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3433276663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d1b50120 con 0x5631d1b4d5d0 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800a930 tx=0x7fe9c800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ef7a0044a943c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9b77fe640 1 -- 172.21.15.16:0/3433276663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8015040 con 0x5631d1b4d5d0 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9b77fe640 1 -- 172.21.15.16:0/3433276663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c800cc60 con 0x5631d1b4d5d0 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.812+0000 7fe9b77fe640 1 -- 172.21.15.16:0/3433276663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8027a20 con 0x5631d1b4d5d0 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- 172.21.15.16:0/3433276663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 msgr2=0x5631d1b68750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800a930 tx=0x7fe9c800a2d0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- 172.21.15.16:0/3433276663 shutdown_connections 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b5c780 0x5631d1b49c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b4eb40 0x5631d1b49700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3433276663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4d5d0 0x5631d1b68750 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- 172.21.15.16:0/3433276663 >> 172.21.15.16:0/3433276663 conn(0x5631d1b16310 msgr2=0x5631d1b57c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- 172.21.15.16:0/3433276663 shutdown_connections 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- 172.21.15.16:0/3433276663 wait complete. 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 -- start start 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.813+0000 7fe9d600f740 1 Processor -- start 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 0x5631d1b648c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b55750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1ac940 con 0x5631d1b4eb40 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1ac240 con 0x5631d1b4d5d0 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1a9450 con 0x5631d1b5c780 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d600f740 10 client.0 _ll_register_callbacks cb 0x7fe9bc00b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d11d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 0x5631d1b648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d11d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49790/0 (socket says 172.21.15.16:49790) 2024-09-12T03:13:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d11d1640 1 -- 172.21.15.16:0/3858434730 learned_addr learned my addr 172.21.15.16:0/3858434730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d21d3640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b55750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3858434730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 msgr2=0x5631d1b55750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b55750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3858434730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 msgr2=0x5631d1b64e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 -- 172.21.15.16:0/3858434730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d1b55c90 con 0x5631d1b4d5d0 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9d19d2640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 0x5631d1b648c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8000c00 tx=0x7fe9c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23a6aaf897dfcf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.814+0000 7fe9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8015050 con 0x5631d1b4d5d0 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9c800cc60 con 0x5631d1b4d5d0 2024-09-12T03:13:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fe9c8027930 con 0x5631d1b4d5d0 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe9c8027b70 con 0x5631d1b4d5d0 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 10 client.30461 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5631d1bd6710 con 0x5631d1b4d5d0 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c800cc60 con 0x5631d1b4d5d0 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7fe9c801a820 con 0x5631d1b4d5d0 2024-09-12T03:13:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 10 client.30461 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 10 client.30461 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 20 client.30461 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1bd6a90 0x5631d1bd8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 4 client.30461 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.815+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 --> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5631d1abad70 con 0x5631d1bd6a90 2024-09-12T03:13:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1bd6a90 0x5631d1bd8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d11d1640 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1bd6a90 0x5631d1bd8e80 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=37bc065b539b656b server_cookie=3fd8a7b152711299 in_seq=0 out_seq=0 2024-09-12T03:13:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9b57fa640 10 client.30461 ms_handle_connect on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9b57fa640 1 -- 172.21.15.16:0/3858434730 <== mds.3 v2:172.21.15.33:6834/2990713107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5631d1abad70 con 0x5631d1bd6a90 2024-09-12T03:13:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9b57fa640 10 client.30461 handle_command_reply: tid=0 2024-09-12T03:13:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 client.30461 shutdown 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 msgr2=0x5631d1b648c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 0x5631d1b648c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8000c00 tx=0x7fe9c80046c0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 shutdown_connections 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:6834/2990713107,v1:172.21.15.33:6835/2990713107] conn(0x5631d1bd6a90 0x5631d1bd8e80 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9b57fa640 0 client.30461 ms_handle_reset on v2:172.21.15.33:6834/2990713107 2024-09-12T03:13:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5631d1b5c780 0x5631d1b55750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631d1b4eb40 0x5631d1b64e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 --2- 172.21.15.16:0/3858434730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5631d1b4d5d0 0x5631d1b648c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 >> 172.21.15.16:0/3858434730 conn(0x5631d1b16310 msgr2=0x5631d1b4f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 shutdown_connections 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 1 -- 172.21.15.16:0/3858434730 wait complete. 2024-09-12T03:13:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.816+0000 7fe9d600f740 20 client.30461 trim_cache size 0 max 16384 2024-09-12T03:13:46.852 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 msgr2=0x7fe9b80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 0x7fe9b80a5040 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009a60 tx=0x7fe9c40095c0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 msgr2=0x7fe9cc199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b4a0 tx=0x7fe9bc00b960 comp rx=0 tx=0).stop 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 shutdown_connections 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9cc102ef0 0x7fe9cc1a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fe9b80a2b80 0x7fe9b80a5040 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9cc102570 0x7fe9cc199a40 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 --2- 172.21.15.16:0/3576033966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe9cc100e20 0x7fe9cc199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 >> 172.21.15.16:0/3576033966 conn(0x7fe9cc0fc7d0 msgr2=0x7fe9cc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 shutdown_connections 2024-09-12T03:13:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:46.817+0000 7fe9d3c36640 1 -- 172.21.15.16:0/3576033966 wait complete. 2024-09-12T03:13:47.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-12T03:13:47.878 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:48.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:47 smithi033 ceph-mon[35092]: pgmap v1113: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-12T03:13:48.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:47 smithi016 ceph-mon[30925]: pgmap v1113: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-12T03:13:48.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:47 smithi184 ceph-mon[35321]: pgmap v1113: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-12T03:13:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.350+0000 7f4624bd4640 1 -- 172.21.15.16:0/2764292791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201036e0 msgr2=0x7f4620107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.350+0000 7f4624bd4640 1 --2- 172.21.15.16:0/2764292791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201036e0 0x7f4620107b90 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f4608009930 tx=0x7f460802a420 comp rx=0 tx=0).stop 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 -- 172.21.15.16:0/2764292791 shutdown_connections 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 --2- 172.21.15.16:0/2764292791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201082a0 0x7f462010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 --2- 172.21.15.16:0/2764292791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201036e0 0x7f4620107b90 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 --2- 172.21.15.16:0/2764292791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f4620103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 -- 172.21.15.16:0/2764292791 >> 172.21.15.16:0/2764292791 conn(0x7f46200fc7d0 msgr2=0x7f46200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 -- 172.21.15.16:0/2764292791 shutdown_connections 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 -- 172.21.15.16:0/2764292791 wait complete. 2024-09-12T03:13:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 Processor -- start 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.351+0000 7f4624bd4640 1 -- start start 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f46201985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201036e0 0x7f4620198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46201120a0 con 0x7f46201082a0 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4620111f20 con 0x7f46201036e0 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4620112220 con 0x7f4620102d30 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49812/0 (socket says 172.21.15.16:49812) 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 -- 172.21.15.16:0/1922593786 learned_addr learned my addr 172.21.15.16:0/1922593786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461e59c640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f46201985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461dd9b640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201036e0 0x7f4620198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 -- 172.21.15.16:0/1922593786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 msgr2=0x7f46201985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f46201985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 -- 172.21.15.16:0/1922593786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201036e0 msgr2=0x7f4620198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201036e0 0x7f4620198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 -- 172.21.15.16:0/1922593786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462019d550 con 0x7f46201082a0 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461e59c640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f46201985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f461ed9d640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f461000c970 tx=0x7f461000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f46177fe640 1 -- 172.21.15.16:0/1922593786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610007bb0 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462019d840 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f46177fe640 1 -- 172.21.15.16:0/1922593786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4610007d50 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.352+0000 7f46177fe640 1 -- 172.21.15.16:0/1922593786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461000f450 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.353+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46201a5120 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f46177fe640 1 -- 172.21.15.16:0/1922593786 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4610004f60 con 0x7f46201082a0 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 Processor -- start 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 -- start start 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d08850a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d0885d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560d087b62f0 con 0x560d08851b40 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d0875dcf0 con 0x560d088505d0 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f4626fad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d087b22b0 con 0x560d0885b370 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.355+0000 7f461dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49822/0 (socket says 172.21.15.16:49822) 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 -- 172.21.15.16:0/3514869395 learned_addr learned my addr 172.21.15.16:0/3514869395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f46177fe640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 0x7f460c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461e59c640 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d08850a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d0885d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461ed9d640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 0x7f460c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f46177fe640 1 -- 172.21.15.16:0/1922593786 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f46100ced10 con 0x7f46201082a0 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 -- 172.21.15.16:0/3514869395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 msgr2=0x560d0885d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d0885d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 -- 172.21.15.16:0/3514869395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 msgr2=0x560d08850a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d08850a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 -- 172.21.15.16:0/3514869395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d087423a0 con 0x560d08851b40 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461dd9b640 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4608005ab0 tx=0x7f46080058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbab53182b1048b server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f461ed9d640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 0x7f460c0a50c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4610007590 tx=0x7f4610066150 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f4614ff9640 1 -- 172.21.15.16:0/3514869395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608035050 con 0x560d08851b40 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f4614ff9640 1 -- 172.21.15.16:0/3514869395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608005040 con 0x560d08851b40 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.356+0000 7f4614ff9640 1 -- 172.21.15.16:0/3514869395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608004920 con 0x560d08851b40 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 -- 172.21.15.16:0/3514869395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 msgr2=0x560d08850090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4608005ab0 tx=0x7f46080058b0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 -- 172.21.15.16:0/3514869395 shutdown_connections 2024-09-12T03:13:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d0885d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d08850a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 --2- 172.21.15.16:0/3514869395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08850090 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 -- 172.21.15.16:0/3514869395 >> 172.21.15.16:0/3514869395 conn(0x560d08819310 msgr2=0x560d0884cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 -- 172.21.15.16:0/3514869395 shutdown_connections 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.357+0000 7f4626fad740 1 -- 172.21.15.16:0/3514869395 wait complete. 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 -- start start 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 Processor -- start 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d0885ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088587e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560d0885dfb0 con 0x560d08851b40 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d08771060 con 0x560d088505d0 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d0885e130 con 0x560d0885b370 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f4626fad740 10 client.0 _ll_register_callbacks cb 0x560d084c48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49828/0 (socket says 172.21.15.16:49828) 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 -- 172.21.15.16:0/1417371906 learned_addr learned my addr 172.21.15.16:0/1417371906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461dd9b640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d0885ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461dd9b640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 -- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 msgr2=0x560d088587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 -- 172.21.15.16:0/1417371906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 msgr2=0x560d0885ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d0885ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.358+0000 7f461e59c640 1 -- 172.21.15.16:0/1417371906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d085273c0 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f461e59c640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4618013a20 tx=0x7f4618013ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8a1a5be96a6d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618003970 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f4618003b10 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4608005040 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f4618003d10 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 10 client.21369 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x560d08869af0 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608005040 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7f461801a070 con 0x560d08851b40 2024-09-12T03:13:48.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 10 client.21369 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 10 client.21369 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 20 client.21369 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d0886af00 0x560d0886d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.359+0000 7f4626fad740 4 client.21369 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.360+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d087c3860 con 0x560d0886af00 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.360+0000 7f461dd9b640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d0886af00 0x560d0886d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.360+0000 7f461dd9b640 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d0886af00 0x560d0886d360 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f257b6e36ff512fb server_cookie=b7b3a01e04e1802 in_seq=0 out_seq=0 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.360+0000 7f4605ffb640 10 client.21369 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4605ffb640 1 -- 172.21.15.16:0/1417371906 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560d087c3860 con 0x560d0886af00 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4605ffb640 10 client.21369 handle_command_reply: tid=0 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 client.21369 shutdown 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 msgr2=0x560d08858050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4618013a20 tx=0x7f4618013ee0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 shutdown_connections 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d0886af00 0x560d0886d360 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08858050 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4605ffb640 0 client.21369 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 --2- 172.21.15.16:0/1417371906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d088505d0 0x560d0885ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 >> 172.21.15.16:0/1417371906 conn(0x560d08819310 msgr2=0x560d08852990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 shutdown_connections 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 1 -- 172.21.15.16:0/1417371906 wait complete. 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.361+0000 7f4626fad740 20 client.21369 trim_cache size 0 max 16384 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 Processor -- start 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 -- start start 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d088505d0 0x560d08859fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 0x560d088536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46201ab220 con 0x560d08851b40 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46201ace20 con 0x560d0885b370 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.374+0000 7f4626fad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46201ab8f0 con 0x560d088505d0 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49844/0 (socket says 172.21.15.16:49844) 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 -- 172.21.15.16:0/4086383188 learned_addr learned my addr 172.21.15.16:0/4086383188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461dd9b640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 0x560d088536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461dd9b640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d088505d0 0x560d08859fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 -- 172.21.15.16:0/4086383188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d088505d0 msgr2=0x560d08859fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d088505d0 0x560d08859fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 -- 172.21.15.16:0/4086383188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 msgr2=0x560d088536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 0x560d088536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 -- 172.21.15.16:0/4086383188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d08858b50 con 0x560d08851b40 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461dd9b640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 0x560d088536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.375+0000 7f461e59c640 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f4618025740 tx=0x7f4618005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3eb99d411fbf54 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.376+0000 7f4607fff640 1 -- 172.21.15.16:0/4086383188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618011e90 con 0x560d08851b40 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.376+0000 7f4607fff640 1 -- 172.21.15.16:0/4086383188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461802a070 con 0x560d08851b40 2024-09-12T03:13:48.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.376+0000 7f4607fff640 1 -- 172.21.15.16:0/4086383188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618005840 con 0x560d08851b40 2024-09-12T03:13:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- 172.21.15.16:0/4086383188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 msgr2=0x560d08853180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f4618025740 tx=0x7f4618005bd0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- 172.21.15.16:0/4086383188 shutdown_connections 2024-09-12T03:13:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0885b370 0x560d088536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d08851b40 0x560d08853180 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- 172.21.15.16:0/4086383188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d088505d0 0x560d08859fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- 172.21.15.16:0/4086383188 >> 172.21.15.16:0/4086383188 conn(0x560d08819310 msgr2=0x560d088598b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- 172.21.15.16:0/4086383188 shutdown_connections 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- 172.21.15.16:0/4086383188 wait complete. 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- start start 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 Processor -- start 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d08851b40 0x560d0885fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088d4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46201ac6f0 con 0x560d088505d0 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46201abff0 con 0x560d08851b40 2024-09-12T03:13:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46201a9240 con 0x560d0885b370 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.377+0000 7f4626fad740 10 client.0 _ll_register_callbacks cb 0x7f4618009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49860/0 (socket says 172.21.15.16:49860) 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 -- 172.21.15.16:0/625823738 learned_addr learned my addr 172.21.15.16:0/625823738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461e59c640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d08851b40 0x560d0885fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 -- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 msgr2=0x560d088d4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088d4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 -- 172.21.15.16:0/625823738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d08851b40 msgr2=0x560d0885fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d08851b40 0x560d0885fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 -- 172.21.15.16:0/625823738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d088d5180 con 0x560d088505d0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f461dd9b640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4608031550 tx=0x7f4608005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebac8da16c96407 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f4605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608035040 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f460803a690 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.378+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f461802a070 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f460803a890 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 10 client.21375 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x560d088d6450 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461802a070 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7f4608052460 con 0x560d088505d0 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 10 client.21375 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 10 client.21375 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 20 client.21375 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d088d6730 0x560d088d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 4 client.21375 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 --> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d087bdd70 con 0x560d088d6730 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.379+0000 7f461e59c640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d088d6730 0x560d088d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f461e59c640 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d088d6730 0x560d088d8b20 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ebf8f5a09bc37dc server_cookie=b6e43f48aa5f6ee5 in_seq=0 out_seq=0 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4605ffb640 10 client.21375 ms_handle_connect on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4605ffb640 1 -- 172.21.15.16:0/625823738 <== mds.2 v2:172.21.15.184:6838/598714982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d087bdd70 con 0x560d088d6730 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4605ffb640 10 client.21375 handle_command_reply: tid=0 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 client.21375 shutdown 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 msgr2=0x560d0885f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4608031550 tx=0x7f4608005fd0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 shutdown_connections 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:6838/598714982,v1:172.21.15.184:6839/598714982] conn(0x560d088d6730 0x560d088d8b20 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d0885b370 0x560d088d4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d08851b40 0x560d0885fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4605ffb640 0 client.21375 ms_handle_reset on v2:172.21.15.184:6838/598714982 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 --2- 172.21.15.16:0/625823738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560d088505d0 0x560d0885f840 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 >> 172.21.15.16:0/625823738 conn(0x560d08819310 msgr2=0x560d0885a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 shutdown_connections 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 1 -- 172.21.15.16:0/625823738 wait complete. 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.380+0000 7f4626fad740 20 client.21375 trim_cache size 0 max 16384 2024-09-12T03:13:48.397 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 msgr2=0x7f460c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 0x7f460c0a50c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4610007590 tx=0x7f4610066150 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 msgr2=0x7f462019ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f461000c970 tx=0x7f461000ce30 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 shutdown_connections 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f460c0a2c00 0x7f460c0a50c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46201082a0 0x7f462019ce70 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46201036e0 0x7f4620198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 --2- 172.21.15.16:0/1922593786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4620102d30 0x7f46201985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 >> 172.21.15.16:0/1922593786 conn(0x7f46200fc7d0 msgr2=0x7f4620105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:48.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 shutdown_connections 2024-09-12T03:13:48.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:48.381+0000 7f4624bd4640 1 -- 172.21.15.16:0/1922593786 wait complete. 2024-09-12T03:13:49.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-12T03:13:49.409 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 -- 172.21.15.16:0/4137739930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0101420 msgr2=0x7ff1e010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/4137739930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0101420 0x7ff1e010d950 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009cd0 tx=0x7ff1d802a8e0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 -- 172.21.15.16:0/4137739930 shutdown_connections 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/4137739930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0101420 0x7ff1e010d950 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/4137739930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0100a80 0x7ff1e0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/4137739930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0106a80 0x7ff1e0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 -- 172.21.15.16:0/4137739930 >> 172.21.15.16:0/4137739930 conn(0x7ff1e00fc7a0 msgr2=0x7ff1e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 -- 172.21.15.16:0/4137739930 shutdown_connections 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.868+0000 7ff1e82c7640 1 -- 172.21.15.16:0/4137739930 wait complete. 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 Processor -- start 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 -- start start 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0101420 0x7ff1e01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 0x7ff1e01a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0110e80 con 0x7ff1e0100a80 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0110d00 con 0x7ff1e0106a80 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e82c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0111000 con 0x7ff1e0101420 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49878/0 (socket says 172.21.15.16:49878) 2024-09-12T03:13:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 -- 172.21.15.16:0/2028261678 learned_addr learned my addr 172.21.15.16:0/2028261678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6864640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 0x7ff1e01a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e5862640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0101420 0x7ff1e01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 -- 172.21.15.16:0/2028261678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0101420 msgr2=0x7ff1e01997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0101420 0x7ff1e01997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 -- 172.21.15.16:0/2028261678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 msgr2=0x7ff1e01a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 0x7ff1e01a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6063640 1 -- 172.21.15.16:0/2028261678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e01a4850 con 0x7ff1e0100a80 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.869+0000 7ff1e6864640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 0x7ff1e01a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00e9f0 tx=0x7ff1dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1d77fe640 1 -- 172.21.15.16:0/2028261678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc00cd80 con 0x7ff1e0100a80 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e01a4a20 con 0x7ff1e0100a80 2024-09-12T03:13:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1d77fe640 1 -- 172.21.15.16:0/2028261678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc005500 con 0x7ff1e0100a80 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1d77fe640 1 -- 172.21.15.16:0/2028261678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc010690 con 0x7ff1e0100a80 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.870+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01a4ee0 con 0x7ff1e0100a80 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.872+0000 7ff1d77fe640 1 -- 172.21.15.16:0/2028261678 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff1dc0046c0 con 0x7ff1e0100a80 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.872+0000 7ff1ea6a0740 1 Processor -- start 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.872+0000 7ff1ea6a0740 1 -- start start 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b7f1172f0 con 0x558b7f1bc370 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b7f0becf0 con 0x558b7f1b2b40 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558b7f1132b0 con 0x558b7f1b15d0 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49880/0 (socket says 172.21.15.16:49880) 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 -- 172.21.15.16:0/4156883590 learned_addr learned my addr 172.21.15.16:0/4156883590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e6063640 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e6864640 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1d77fe640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 0x7ff1cc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 -- 172.21.15.16:0/4156883590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 msgr2=0x558b7f1b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 -- 172.21.15.16:0/4156883590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 msgr2=0x558b7f1b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 -- 172.21.15.16:0/4156883590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b7f0a33a0 con 0x558b7f1bc370 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e6864640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 0x7ff1cc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1d77fe640 1 -- 172.21.15.16:0/2028261678 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1dc0cf570 con 0x7ff1e0100a80 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.873+0000 7ff1e5862640 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000bc80 tx=0x7ff1d000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b4bbdf4a64cad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1d4dea640 1 -- 172.21.15.16:0/4156883590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0015070 con 0x558b7f1bc370 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1d4dea640 1 -- 172.21.15.16:0/4156883590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d0011920 con 0x558b7f1bc370 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1d4dea640 1 -- 172.21.15.16:0/4156883590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00108b0 con 0x558b7f1bc370 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4156883590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 msgr2=0x558b7f1be760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000bc80 tx=0x7ff1d000f730 comp rx=0 tx=0).stop 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1e6864640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 0x7ff1cc0a5040 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009a60 tx=0x7ff1d803e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4156883590 shutdown_connections 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1be760 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4156883590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4156883590 >> 172.21.15.16:0/4156883590 conn(0x558b7f17a310 msgr2=0x558b7f1adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.874+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4156883590 shutdown_connections 2024-09-12T03:13:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4156883590 wait complete. 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 -- start start 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 Processor -- start 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1bbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1bc370 0x558b7f1b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b7f1bf070 con 0x558b7f1b2b40 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b7f0d2060 con 0x558b7f1b15d0 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558b7f1bf1f0 con 0x558b7f1bc370 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1ea6a0740 10 client.0 _ll_register_callbacks cb 0x558b7ee258c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1e5862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1e6063640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1bbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1e5862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49888/0 (socket says 172.21.15.16:49888) 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1e5862640 1 -- 172.21.15.16:0/2801078509 learned_addr learned my addr 172.21.15.16:0/2801078509 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.875+0000 7ff1e6864640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1bc370 0x558b7f1b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 -- 172.21.15.16:0/2801078509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1bc370 msgr2=0x558b7f1b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1bc370 0x558b7f1b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 -- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 msgr2=0x558b7f1bbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1bbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 -- 172.21.15.16:0/2801078509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b7ee883c0 con 0x558b7f1b2b40 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1bbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1e5862640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000bfa0 tx=0x7ff1d000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b54873f87b54db server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0015070 con 0x558b7f1b2b40 2024-09-12T03:13:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1d0011ad0 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d0011920 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff1d0011cd0 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1ea6a0740 10 client.21387 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558b7f1cab10 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.876+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d0011920 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7ff1d0004a60 con 0x558b7f1b2b40 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 10 client.21387 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 10 client.21387 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 20 client.21387 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f1cbf10 0x558b7f1ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 4 client.21387 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b7f124860 con 0x558b7f1cbf10 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.877+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f1cbf10 0x558b7f1ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1e6063640 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f1cbf10 0x558b7f1ce300 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=191c9f90e7f103d server_cookie=c1c21efb57fabcca in_seq=0 out_seq=0 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1c9ffb640 10 client.21387 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/2801078509 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558b7f124860 con 0x558b7f1cbf10 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1c9ffb640 10 client.21387 handle_command_reply: tid=0 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 client.21387 shutdown 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 msgr2=0x558b7f1b9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000bfa0 tx=0x7ff1d000bfd0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 shutdown_connections 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f1cbf10 0x558b7f1ce300 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1bc370 0x558b7f1b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1b2b40 0x558b7f1b9010 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1c9ffb640 0 client.21387 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/2801078509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1bbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 >> 172.21.15.16:0/2801078509 conn(0x558b7f17a310 msgr2=0x558b7f1b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.878+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 shutdown_connections 2024-09-12T03:13:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.879+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/2801078509 wait complete. 2024-09-12T03:13:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.879+0000 7ff1ea6a0740 20 client.21387 trim_cache size 0 max 16384 2024-09-12T03:13:49.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 Processor -- start 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 -- start start 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ad590 con 0x558b7f1bc370 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01af190 con 0x558b7f1b2b40 2024-09-12T03:13:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01adc60 con 0x558b7f1b15d0 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49900/0 (socket says 172.21.15.16:49900) 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 -- 172.21.15.16:0/4185431682 learned_addr learned my addr 172.21.15.16:0/4185431682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e5862640 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6063640 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 -- 172.21.15.16:0/4185431682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 msgr2=0x558b7f1b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 -- 172.21.15.16:0/4185431682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 msgr2=0x558b7f1c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.892+0000 7ff1e6864640 1 -- 172.21.15.16:0/4185431682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b7f1ae960 con 0x558b7f1bc370 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1e6864640 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8035000 tx=0x7ff1d8040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93085c96c7ce6b03 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1cbfff640 1 -- 172.21.15.16:0/4185431682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8003970 con 0x558b7f1bc370 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1cbfff640 1 -- 172.21.15.16:0/4185431682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8036040 con 0x558b7f1bc370 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1cbfff640 1 -- 172.21.15.16:0/4185431682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8030800 con 0x558b7f1bc370 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4185431682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 msgr2=0x558b7f1b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8035000 tx=0x7ff1d8040000 comp rx=0 tx=0).stop 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4185431682 shutdown_connections 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b4120 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b2b40 0x558b7f1c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/4185431682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b15d0 0x558b7f1b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4185431682 >> 172.21.15.16:0/4185431682 conn(0x558b7f17a310 msgr2=0x558b7f1a8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4185431682 shutdown_connections 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.893+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/4185431682 wait complete. 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 -- start start 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 Processor -- start 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 0x558b7f1c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01aea60 con 0x558b7f1bc370 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ae360 con 0x558b7f1b15d0 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ab570 con 0x558b7f1b2b40 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1ea6a0740 10 client.0 _ll_register_callbacks cb 0x7ff1dc0980a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1e6864640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1e6864640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49910/0 (socket says 172.21.15.16:49910) 2024-09-12T03:13:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1e6864640 1 -- 172.21.15.16:0/3147008006 learned_addr learned my addr 172.21.15.16:0/3147008006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1e6063640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.894+0000 7ff1e5862640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 0x558b7f1c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 -- 172.21.15.16:0/3147008006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 msgr2=0x558b7f1c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 0x558b7f1c4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 -- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 msgr2=0x558b7f1c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 -- 172.21.15.16:0/3147008006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b7f1c8850 con 0x558b7f1bc370 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e5862640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 0x558b7f1c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6063640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1e6864640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80048a0 tx=0x7ff1d8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce92866d312806e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8058070 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1d8003d80 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d8036040 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff1d8063050 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1ea6a0740 10 client.21393 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558b7f2363b0 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.895+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8036040 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7ff1d8030070 con 0x558b7f1bc370 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 10 client.21393 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 10 client.21393 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 20 client.21393 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f236ad0 0x558b7f238ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 4 client.21393 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 --> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b7f11ed70 con 0x558b7f236ad0 2024-09-12T03:13:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1e6063640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f236ad0 0x558b7f238ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1e6063640 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f236ad0 0x558b7f238ef0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fad1ee7e6dbd7f7b server_cookie=a5c7e769b736983c in_seq=0 out_seq=0 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1c9ffb640 10 client.21393 ms_handle_connect on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1c9ffb640 1 -- 172.21.15.16:0/3147008006 <== mds.1 v2:172.21.15.33:6836/4242490884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b7f11ed70 con 0x558b7f236ad0 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1c9ffb640 10 client.21393 handle_command_reply: tid=0 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.896+0000 7ff1ea6a0740 1 client.21393 shutdown 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 msgr2=0x558b7f1b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80048a0 tx=0x7ff1d8009560 comp rx=0 tx=0).stop 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 shutdown_connections 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:6836/4242490884,v1:172.21.15.33:6837/4242490884] conn(0x558b7f236ad0 0x558b7f238ef0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b7f1bc370 0x558b7f1b5bf0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558b7f1b2b40 0x558b7f1c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 --2- 172.21.15.16:0/3147008006 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b7f1b15d0 0x558b7f1c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 >> 172.21.15.16:0/3147008006 conn(0x558b7f17a310 msgr2=0x558b7f1ba870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1c9ffb640 0 client.21393 ms_handle_reset on v2:172.21.15.33:6836/4242490884 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 shutdown_connections 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 1 -- 172.21.15.16:0/3147008006 wait complete. 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1ea6a0740 20 client.21393 trim_cache size 0 max 16384 2024-09-12T03:13:49.909 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 msgr2=0x7ff1cc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 0x7ff1cc0a5040 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009a60 tx=0x7ff1d803e040 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 msgr2=0x7ff1e0199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00e9f0 tx=0x7ff1dc00edf0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 shutdown_connections 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1e0106a80 0x7ff1e01a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e0101420 0x7ff1e01997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff1cc0a2b80 0x7ff1cc0a5040 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 --2- 172.21.15.16:0/2028261678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0100a80 0x7ff1e0199290 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.897+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 >> 172.21.15.16:0/2028261678 conn(0x7ff1e00fc7a0 msgr2=0x7ff1e00fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.898+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 shutdown_connections 2024-09-12T03:13:49.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:49.898+0000 7ff1e82c7640 1 -- 172.21.15.16:0/2028261678 wait complete. 2024-09-12T03:13:50.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:49 smithi016 ceph-mon[30925]: pgmap v1114: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:13:50.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:49 smithi033 ceph-mon[35092]: pgmap v1114: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:13:50.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:49 smithi184 ceph-mon[35321]: pgmap v1114: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:13:50.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-12T03:13:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:51.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 -- 172.21.15.16:0/2027217951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190106f00 msgr2=0x7ff19010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 --2- 172.21.15.16:0/2027217951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190106f00 0x7ff19010ea80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7ff188009ca0 tx=0x7ff18802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 -- 172.21.15.16:0/2027217951 shutdown_connections 2024-09-12T03:13:51.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 --2- 172.21.15.16:0/2027217951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190106f00 0x7ff19010ea80 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 --2- 172.21.15.16:0/2027217951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190102780 0x7ff1901067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 --2- 172.21.15.16:0/2027217951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190101dd0 0x7ff1901021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 -- 172.21.15.16:0/2027217951 >> 172.21.15.16:0/2027217951 conn(0x7ff1900fd8b0 msgr2=0x7ff1900ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 -- 172.21.15.16:0/2027217951 shutdown_connections 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 -- 172.21.15.16:0/2027217951 wait complete. 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.412+0000 7ff195b03640 1 Processor -- start 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 -- start start 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190102780 0x7ff190074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190106f00 0x7ff190074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff190111f60 con 0x7ff190102780 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff190111de0 con 0x7ff190101dd0 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff195b03640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901120e0 con 0x7ff190106f00 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.16:54382/0 (socket says 172.21.15.16:54382) 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190102780 0x7ff190074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3106196742 learned_addr learned my addr 172.21.15.16:0/3106196742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18ffff640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190106f00 0x7ff190074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.413+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3106196742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190106f00 msgr2=0x7ff190074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190106f00 0x7ff190074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190102780 msgr2=0x7ff190074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190102780 0x7ff190074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3106196742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1901a6840 con 0x7ff190101dd0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c920 tx=0x7ff17800cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18cff9640 1 -- 172.21.15.16:0/3106196742 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178007c70 con 0x7ff190101dd0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1901a6b30 con 0x7ff190101dd0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18cff9640 1 -- 172.21.15.16:0/3106196742 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff178007e10 con 0x7ff190101dd0 2024-09-12T03:13:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff18cff9640 1 -- 172.21.15.16:0/3106196742 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178018890 con 0x7ff190101dd0 2024-09-12T03:13:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.414+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1901a70c0 con 0x7ff190101dd0 2024-09-12T03:13:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 Processor -- start 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 -- start start 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f1d39272f0 con 0x55f1d39cc370 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f1d38cecf0 con 0x55f1d39c15d0 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff197edc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f1d39232b0 con 0x55f1d39c2b40 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49920/0 (socket says 172.21.15.16:49920) 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 -- 172.21.15.16:0/2277122483 learned_addr learned my addr 172.21.15.16:0/2277122483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18ffff640 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 -- 172.21.15.16:0/2277122483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 msgr2=0x55f1d39c1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 -- 172.21.15.16:0/2277122483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 msgr2=0x55f1d39c1a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39c1a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.417+0000 7ff18effd640 1 -- 172.21.15.16:0/2277122483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d38b33a0 con 0x55f1d39cc370 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.418+0000 7ff18cff9640 1 -- 172.21.15.16:0/3106196742 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff178018a70 con 0x7ff190101dd0 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.418+0000 7ff18effd640 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff180009c10 tx=0x7ff180011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30dea89c10de348 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.418+0000 7ff185ffb640 1 -- 172.21.15.16:0/2277122483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180004230 con 0x55f1d39cc370 2024-09-12T03:13:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.418+0000 7ff185ffb640 1 -- 172.21.15.16:0/2277122483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1800043d0 con 0x55f1d39cc370 2024-09-12T03:13:51.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.418+0000 7ff185ffb640 1 -- 172.21.15.16:0/2277122483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180004e10 con 0x55f1d39cc370 2024-09-12T03:13:51.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff18cff9640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 0x7ff17c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff18ffff640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 0x7ff17c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff18cff9640 1 -- 172.21.15.16:0/3106196742 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1780cf0d0 con 0x7ff190101dd0 2024-09-12T03:13:51.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 -- 172.21.15.16:0/2277122483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 msgr2=0x55f1d39ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff180009c10 tx=0x7ff180011810 comp rx=0 tx=0).stop 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 -- 172.21.15.16:0/2277122483 shutdown_connections 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39ce760 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 --2- 172.21.15.16:0/2277122483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 -- 172.21.15.16:0/2277122483 >> 172.21.15.16:0/2277122483 conn(0x55f1d398a310 msgr2=0x55f1d39bdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 -- 172.21.15.16:0/2277122483 shutdown_connections 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff197edc740 1 -- 172.21.15.16:0/2277122483 wait complete. 2024-09-12T03:13:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.419+0000 7ff18ffff640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 0x7ff17c0a5040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff188004af0 tx=0x7ff188004a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 -- start start 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 Processor -- start 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f1d39cf070 con 0x55f1d39cc370 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f1d38e2060 con 0x55f1d39c15d0 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f1d39cf1f0 con 0x55f1d39c2b40 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff197edc740 10 client.0 _ll_register_callbacks cb 0x55f1d36358c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff18ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff18ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49928/0 (socket says 172.21.15.16:49928) 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff18ffff640 1 -- 172.21.15.16:0/2433336340 learned_addr learned my addr 172.21.15.16:0/2433336340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.420+0000 7ff18effd640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2433336340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 msgr2=0x55f1d39c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2433336340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 msgr2=0x55f1d39c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2433336340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d36983c0 con 0x55f1d39c15d0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18ffff640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39cbd90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55f1d39beac0 tx=0x7ff17809c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f62394d5db765679 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff16effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1780b3020 con 0x55f1d39c15d0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1800043d0 con 0x55f1d39c15d0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.421+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff17800fb30 con 0x55f1d39c15d0 2024-09-12T03:13:51.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff1780afe60 con 0x55f1d39c15d0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 10 client.30503 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f1d39dac20 con 0x55f1d39c15d0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1800043d0 con 0x55f1d39c15d0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7ff1780860b0 con 0x55f1d39c15d0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 10 client.30503 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 10 client.30503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 20 client.30503 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d39dbf70 0x55f1d39de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 4 client.30503 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d3934860 con 0x55f1d39dbf70 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.422+0000 7ff18effd640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d39dbf70 0x55f1d39de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.423+0000 7ff18effd640 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d39dbf70 0x55f1d39de390 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91ae39c043ec941f server_cookie=c108b36840556a8e in_seq=0 out_seq=0 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.423+0000 7ff16effd640 10 client.30503 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff16effd640 1 -- 172.21.15.16:0/2433336340 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1d3934860 con 0x55f1d39dbf70 2024-09-12T03:13:51.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff16effd640 10 client.30503 handle_command_reply: tid=0 2024-09-12T03:13:51.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 client.30503 shutdown 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 msgr2=0x55f1d39cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39cbd90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55f1d39beac0 tx=0x7ff17809c100 comp rx=0 tx=0).stop 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 shutdown_connections 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d39dbf70 0x55f1d39de390 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39cc370 0x55f1d39c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 --2- 172.21.15.16:0/2433336340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39c15d0 0x55f1d39cbd90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 >> 172.21.15.16:0/2433336340 conn(0x55f1d398a310 msgr2=0x55f1d39c3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.424+0000 7ff16effd640 0 client.30503 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:51.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.425+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 shutdown_connections 2024-09-12T03:13:51.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.425+0000 7ff197edc740 1 -- 172.21.15.16:0/2433336340 wait complete. 2024-09-12T03:13:51.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.425+0000 7ff197edc740 20 client.30503 trim_cache size 0 max 16384 2024-09-12T03:13:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 Processor -- start 2024-09-12T03:13:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 -- start start 2024-09-12T03:13:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 0x55f1d39bddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901af600 con 0x55f1d39c2b40 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b1200 con 0x55f1d39cc370 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.450+0000 7ff197edc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901afcd0 con 0x55f1d39c15d0 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49932/0 (socket says 172.21.15.16:49932) 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18effd640 1 -- 172.21.15.16:0/2638756083 learned_addr learned my addr 172.21.15.16:0/2638756083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 0x55f1d39bddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18ffff640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2638756083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 msgr2=0x55f1d39e3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39e3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2638756083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 msgr2=0x55f1d39d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 -- 172.21.15.16:0/2638756083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d39c7590 con 0x55f1d39c15d0 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18effd640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 0x55f1d39bddf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff1780982c0 tx=0x7ff178099e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb402cb8779819 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff1857fa640 1 -- 172.21.15.16:0/2638756083 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1780b3070 con 0x55f1d39c15d0 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff1857fa640 1 -- 172.21.15.16:0/2638756083 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff178018d60 con 0x55f1d39c15d0 2024-09-12T03:13:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.451+0000 7ff1857fa640 1 -- 172.21.15.16:0/2638756083 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1780860f0 con 0x55f1d39c15d0 2024-09-12T03:13:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 -- 172.21.15.16:0/2638756083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 msgr2=0x55f1d39bddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 0x55f1d39bddf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff1780982c0 tx=0x7ff178099e50 comp rx=0 tx=0).stop 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 -- 172.21.15.16:0/2638756083 shutdown_connections 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c2b40 0x55f1d39d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 --2- 172.21.15.16:0/2638756083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c15d0 0x55f1d39bddf0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 -- 172.21.15.16:0/2638756083 >> 172.21.15.16:0/2638756083 conn(0x55f1d398a310 msgr2=0x55f1d39bd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 -- 172.21.15.16:0/2638756083 shutdown_connections 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.452+0000 7ff197edc740 1 -- 172.21.15.16:0/2638756083 wait complete. 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 -- start start 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 Processor -- start 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b0ad0 con 0x55f1d39c15d0 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901b03d0 con 0x55f1d39cc370 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901ad5e0 con 0x55f1d39c2b40 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff197edc740 10 client.0 _ll_register_callbacks cb 0x7ff17800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff18f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.453+0000 7ff18f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49942/0 (socket says 172.21.15.16:49942) 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3674010236 learned_addr learned my addr 172.21.15.16:0/3674010236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18effd640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18ffff640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3674010236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 msgr2=0x55f1d39d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39d0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 msgr2=0x55f1d39d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-12T03:13:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 -- 172.21.15.16:0/3674010236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d3a45ce0 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff18f7fe640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ff17809b8d0 tx=0x7ff178000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedda1c115d99768 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff16effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17807e070 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7ff1780b4b40 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.454+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff178018d60 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff1780b3070 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 10 client.21411 fetch_fsmap learned FSMap version 45 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f1d3a46fc0 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff178018d60 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 8390+0+0 (secure 0 0 0) 0x7ff17809a800 con 0x55f1d39c15d0 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 10 client.21411 fetch_fsmap finished waiting for FSMap version 45 2024-09-12T03:13:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 10 client.21411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 20 client.21411 populate_metadata read hostname 'smithi016' 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d3a473f0 0x55f1d3a497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 4 client.21411 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 --> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1d392ed70 con 0x55f1d3a473f0 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.455+0000 7ff18effd640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d3a473f0 0x55f1d3a497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff18effd640 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d3a473f0 0x55f1d3a497e0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4751613f4285a42 server_cookie=6db3d9103e11a765 in_seq=0 out_seq=0 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff16effd640 10 client.21411 ms_handle_connect on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff16effd640 1 -- 172.21.15.16:0/3674010236 <== mds.0 v2:172.21.15.33:6838/3603617308 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1d392ed70 con 0x55f1d3a473f0 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff16effd640 10 client.21411 handle_command_reply: tid=0 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 client.21411 shutdown 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 msgr2=0x55f1d39d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ff17809b8d0 tx=0x7ff178000bc0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 shutdown_connections 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:6838/3603617308,v1:172.21.15.33:6839/3603617308] conn(0x55f1d3a473f0 0x55f1d3a497e0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f1d39cc370 0x55f1d39d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1d39c2b40 0x55f1d39d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff16effd640 0 client.21411 ms_handle_reset on v2:172.21.15.33:6838/3603617308 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 --2- 172.21.15.16:0/3674010236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1d39c15d0 0x55f1d39d02c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 >> 172.21.15.16:0/3674010236 conn(0x55f1d398a310 msgr2=0x55f1d39bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 shutdown_connections 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 1 -- 172.21.15.16:0/3674010236 wait complete. 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.456+0000 7ff197edc740 20 client.21411 trim_cache size 0 max 16384 2024-09-12T03:13:51.463 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 msgr2=0x7ff17c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 0x7ff17c0a5040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff188004af0 tx=0x7ff188004a80 comp rx=0 tx=0).stop 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 msgr2=0x7ff1900744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c920 tx=0x7ff17800cde0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 shutdown_connections 2024-09-12T03:13:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff190106f00 0x7ff190074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff17c0a2b80 0x7ff17c0a5040 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff190102780 0x7ff190074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 --2- 172.21.15.16:0/3106196742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff190101dd0 0x7ff1900744f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 >> 172.21.15.16:0/3106196742 conn(0x7ff1900fd8b0 msgr2=0x7ff1900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 shutdown_connections 2024-09-12T03:13:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:51.457+0000 7ff195b03640 1 -- 172.21.15.16:0/3106196742 wait complete. 2024-09-12T03:13:52.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:51 smithi016 ceph-mon[30925]: pgmap v1115: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-09-12T03:13:52.083 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_086c737fa75c22d521a3932c59b31c5e1d1c9ef4/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-12T03:13:52.084 INFO:tasks.fwd_scrub:done joining 2024-09-12T03:13:52.084 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-12T03:13:52.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 -- 172.21.15.16:0/1037724112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 msgr2=0x7f50a4100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 --2- 172.21.15.16:0/1037724112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a4100e10 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5094009ca0 tx=0x7f509402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 -- 172.21.15.16:0/1037724112 shutdown_connections 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 --2- 172.21.15.16:0/1037724112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 --2- 172.21.15.16:0/1037724112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 0x7f50a4102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 --2- 172.21.15.16:0/1037724112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a4100e10 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 -- 172.21.15.16:0/1037724112 >> 172.21.15.16:0/1037724112 conn(0x7f50a40fc170 msgr2=0x7f50a40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 -- 172.21.15.16:0/1037724112 shutdown_connections 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.200+0000 7f50a9827640 1 -- 172.21.15.16:0/1037724112 wait complete. 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 Processor -- start 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 -- start start 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 0x7f50a4102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50a410c690 con 0x7f50a4100a10 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50a410c510 con 0x7f50a41025c0 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a9827640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50a410c810 con 0x7f50a4101c00 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49964/0 (socket says 172.21.15.16:49964) 2024-09-12T03:13:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a2ffd640 1 -- 172.21.15.16:0/1043900616 learned_addr learned my addr 172.21.15.16:0/1043900616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a27fc640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 0x7f50a4102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a37fe640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.201+0000 7f50a2ffd640 1 -- 172.21.15.16:0/1043900616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 msgr2=0x7f50a4102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a2ffd640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 0x7f50a4102fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a2ffd640 1 -- 172.21.15.16:0/1043900616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 msgr2=0x7f50a4103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a2ffd640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a2ffd640 1 -- 172.21.15.16:0/1043900616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a4103db0 con 0x7f50a4100a10 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a37fe640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a2ffd640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f5094004af0 tx=0x7f5094031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509402fb60 con 0x7f50a4100a10 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a41a1fb0 con 0x7f50a4100a10 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f509402fd00 con 0x7f50a4100a10 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5094046920 con 0x7f50a4100a10 2024-09-12T03:13:52.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.202+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a41a2410 con 0x7f50a4100a10 2024-09-12T03:13:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.203+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a41a2700 con 0x7f50a4100a10 2024-09-12T03:13:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.204+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f5094030c30 con 0x7f50a4100a10 2024-09-12T03:13:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.205+0000 7f50a8825640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 0x7f50900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.205+0000 7f50a27fc640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 0x7f50900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.205+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f50940f0880 con 0x7f50a4100a10 2024-09-12T03:13:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.206+0000 7f50a27fc640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 0x7f50900a5050 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f50a4104710 tx=0x7f508c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.209+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f50940821b0 con 0x7f50a4100a10 2024-09-12T03:13:52.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:51 smithi033 ceph-mon[35092]: pgmap v1115: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-09-12T03:13:52.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:51 smithi184 ceph-mon[35321]: pgmap v1115: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-09-12T03:13:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.391+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50a41a29f0 con 0x7f50a4100a10 2024-09-12T03:13:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.392+0000 7f50a8825640 1 -- 172.21.15.16:0/1043900616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f50940bd7f0 con 0x7f50a4100a10 2024-09-12T03:13:52.394 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:13:52.394 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 msgr2=0x7f50900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 0x7f50900a5050 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f50a4104710 tx=0x7f508c00b390 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 msgr2=0x7f50a41049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f5094004af0 tx=0x7f5094031ee0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 shutdown_connections 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50a41025c0 0x7f50a4103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a4101c00 0x7f50a4102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f50900a2b90 0x7f50900a5050 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 --2- 172.21.15.16:0/1043900616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4100a10 0x7f50a41049a0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.394+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 >> 172.21.15.16:0/1043900616 conn(0x7f50a40fc170 msgr2=0x7f50a40693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.395+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 shutdown_connections 2024-09-12T03:13:52.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.395+0000 7f50a9827640 1 -- 172.21.15.16:0/1043900616 wait complete. 2024-09-12T03:13:52.409 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-12T02:41:16.432997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-12T02:45:43.869635+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-12T02:45:46.156619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-12T03:13:52.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- 172.21.15.16:0/2015074519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650104e20 msgr2=0x7fd650107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 --2- 172.21.15.16:0/2015074519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650104e20 0x7fd650107210 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fd638009900 tx=0x7fd63802a410 comp rx=0 tx=0).stop 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- 172.21.15.16:0/2015074519 shutdown_connections 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 --2- 172.21.15.16:0/2015074519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650107750 0x7fd650109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 --2- 172.21.15.16:0/2015074519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650104e20 0x7fd650107210 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 --2- 172.21.15.16:0/2015074519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 0x7fd6501048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- 172.21.15.16:0/2015074519 >> 172.21.15.16:0/2015074519 conn(0x7fd6500fc070 msgr2=0x7fd6500fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- 172.21.15.16:0/2015074519 shutdown_connections 2024-09-12T03:13:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- 172.21.15.16:0/2015074519 wait complete. 2024-09-12T03:13:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 Processor -- start 2024-09-12T03:13:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.516+0000 7fd65579f640 1 -- start start 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 0x7fd650101e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650104e20 0x7fd6501004a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd65010d6b0 con 0x7fd650107750 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd65010d530 con 0x7fd650104e20 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd65010d830 con 0x7fd6501024f0 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49980/0 (socket says 172.21.15.16:49980) 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 -- 172.21.15.16:0/1331318287 learned_addr learned my addr 172.21.15.16:0/1331318287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd64ffff640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 0x7fd650101e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd647fff640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650104e20 0x7fd6501004a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 -- 172.21.15.16:0/1331318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 msgr2=0x7fd650101e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 0x7fd650101e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 -- 172.21.15.16:0/1331318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650104e20 msgr2=0x7fd6501004a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650104e20 0x7fd6501004a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 -- 172.21.15.16:0/1331318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd650101270 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd654f9e640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fd64800ccb0 tx=0x7fd64800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64800f450 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64800fad0 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6501a1e40 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.517+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648018ac0 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.518+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6501a22a0 con 0x7fd650107750 2024-09-12T03:13:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.518+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6500fdb60 con 0x7fd650107750 2024-09-12T03:13:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.520+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd64800fc80 con 0x7fd650107750 2024-09-12T03:13:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.520+0000 7fd64dffb640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 0x7fd63c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.520+0000 7fd64ffff640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 0x7fd63c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.520+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fd6480cf770 con 0x7fd650107750 2024-09-12T03:13:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.521+0000 7fd64ffff640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 0x7fd63c0a50d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd64000abe0 tx=0x7fd64000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.523+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd648014030 con 0x7fd650107750 2024-09-12T03:13:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.698+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd650062df0 con 0x7fd650107750 2024-09-12T03:13:52.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.698+0000 7fd64dffb640 1 -- 172.21.15.16:0/1331318287 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fd64809c760 con 0x7fd650107750 2024-09-12T03:13:52.699 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 msgr2=0x7fd63c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 0x7fd63c0a50d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd64000abe0 tx=0x7fd64000a2d0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 msgr2=0x7fd6501009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fd64800ccb0 tx=0x7fd64800a620 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 shutdown_connections 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fd63c0a2c10 0x7fd63c0a50d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650107750 0x7fd6501009e0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd650104e20 0x7fd6501004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 --2- 172.21.15.16:0/1331318287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6501024f0 0x7fd650101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 >> 172.21.15.16:0/1331318287 conn(0x7fd6500fc070 msgr2=0x7fd650102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 shutdown_connections 2024-09-12T03:13:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.701+0000 7fd65579f640 1 -- 172.21.15.16:0/1331318287 wait complete. 2024-09-12T03:13:52.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 -- 172.21.15.16:0/3449299080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4107810 msgr2=0x7efce4109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3449299080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4107810 0x7efce4109c80 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009ca0 tx=0x7efcdc02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 -- 172.21.15.16:0/3449299080 shutdown_connections 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3449299080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4107810 0x7efce4109c80 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3449299080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4104ee0 0x7efce41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3449299080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce41049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 -- 172.21.15.16:0/3449299080 >> 172.21.15.16:0/3449299080 conn(0x7efce40fc1b0 msgr2=0x7efce40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 -- 172.21.15.16:0/3449299080 shutdown_connections 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.818+0000 7efceb8ac640 1 -- 172.21.15.16:0/3449299080 wait complete. 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 Processor -- start 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 -- start start 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 0x7efce419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efce410d820 con 0x7efce4104ee0 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efce410d6a0 con 0x7efce4107810 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efceb8ac640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efce410d9a0 con 0x7efce41025b0 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efce8e47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efce8e47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50004/0 (socket says 172.21.15.16:50004) 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efce8e47640 1 -- 172.21.15.16:0/3057191490 learned_addr learned my addr 172.21.15.16:0/3057191490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efce9648640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.819+0000 7efce9e49640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 0x7efce419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 -- 172.21.15.16:0/3057191490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 msgr2=0x7efce4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 -- 172.21.15.16:0/3057191490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 msgr2=0x7efce419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 0x7efce419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 -- 172.21.15.16:0/3057191490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce419fb20 con 0x7efce4104ee0 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce9e49640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 0x7efce419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce9648640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efce8e47640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7efcd400e9c0 tx=0x7efcd400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd400cd80 con 0x7efce4104ee0 2024-09-12T03:13:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcd4005500 con 0x7efce4104ee0 2024-09-12T03:13:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce419fe10 con 0x7efce4104ee0 2024-09-12T03:13:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4010690 con 0x7efce4104ee0 2024-09-12T03:13:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.820+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce41a02d0 con 0x7efce4104ee0 2024-09-12T03:13:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.821+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce40fdca0 con 0x7efce4104ee0 2024-09-12T03:13:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.822+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efcd40046c0 con 0x7efce4104ee0 2024-09-12T03:13:52.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.823+0000 7efcda7fc640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 0x7efcd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:52.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.823+0000 7efce9648640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 0x7efcd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:52.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.823+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7efcd40cf6b0 con 0x7efce4104ee0 2024-09-12T03:13:52.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.824+0000 7efce9648640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 0x7efcd00a5050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7efce00058c0 tx=0x7efce0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:52.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:52.826+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efcd40541b0 con 0x7efce4104ee0 2024-09-12T03:13:53.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.003+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efce402da10 con 0x7efce4104ee0 2024-09-12T03:13:53.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.003+0000 7efcda7fc640 1 -- 172.21.15.16:0/3057191490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7efcd409c620 con 0x7efce4104ee0 2024-09-12T03:13:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-09-12T03:13:53.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 msgr2=0x7efcd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 0x7efcd00a5050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7efce00058c0 tx=0x7efce0005850 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 msgr2=0x7efce4198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7efcd400e9c0 tx=0x7efcd400ee80 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 shutdown_connections 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce4107810 0x7efce419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7efcd00a2b90 0x7efcd00a5050 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4104ee0 0x7efce4198440 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 --2- 172.21.15.16:0/3057191490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efce41025b0 0x7efce4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.005+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 >> 172.21.15.16:0/3057191490 conn(0x7efce40fc1b0 msgr2=0x7efce4102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.006+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 shutdown_connections 2024-09-12T03:13:53.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.006+0000 7efceb8ac640 1 -- 172.21.15.16:0/3057191490 wait complete. 2024-09-12T03:13:53.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-12T03:13:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- 172.21.15.16:0/1946417638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d8101c00 msgr2=0x7f35d8102080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1946417638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d8101c00 0x7f35d8102080 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009900 tx=0x7f35c802a410 comp rx=0 tx=0).stop 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- 172.21.15.16:0/1946417638 shutdown_connections 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1946417638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d81025c0 0x7f35d8108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1946417638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d8101c00 0x7f35d8102080 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1946417638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 0x7f35d8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- 172.21.15.16:0/1946417638 >> 172.21.15.16:0/1946417638 conn(0x7f35d80fc190 msgr2=0x7f35d80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- 172.21.15.16:0/1946417638 shutdown_connections 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- 172.21.15.16:0/1946417638 wait complete. 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 Processor -- start 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.123+0000 7f35df9a8640 1 -- start start 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 0x7f35d8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d8101c00 0x7f35d81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810c6c0 con 0x7f35d81025c0 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810c540 con 0x7f35d8100a10 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35d810c840 con 0x7f35d8101c00 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35dd744640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 0x7f35d8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50032/0 (socket says 172.21.15.16:50032) 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 -- 172.21.15.16:0/1570841832 learned_addr learned my addr 172.21.15.16:0/1570841832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35dcf43640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d8101c00 0x7f35d81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 -- 172.21.15.16:0/1570841832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d8101c00 msgr2=0x7f35d81983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d8101c00 0x7f35d81983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 -- 172.21.15.16:0/1570841832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 msgr2=0x7f35d8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 0x7f35d8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 -- 172.21.15.16:0/1570841832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d819fa80 con 0x7f35d81025c0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ddf45640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009fd0 tx=0x7f35d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0017070 con 0x7f35d81025c0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d0003e50 con 0x7f35d81025c0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.124+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d819fd70 con 0x7f35d81025c0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.125+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00106c0 con 0x7f35d81025c0 2024-09-12T03:13:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.125+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d81a0230 con 0x7f35d81025c0 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.126+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d81a0460 con 0x7f35d81025c0 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.127+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f35d0003970 con 0x7f35d81025c0 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.127+0000 7f35ce7fc640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 0x7f35c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.127+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f35d00cec90 con 0x7f35d81025c0 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.127+0000 7f35dd744640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 0x7f35c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.128+0000 7f35dd744640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 0x7f35c40a50d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f35d8101a60 tx=0x7f35d400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.131+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f35d009b5b0 con 0x7f35d81025c0 2024-09-12T03:13:53.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1043900616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:13:53.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:52 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1331318287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T03:13:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1043900616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:13:53.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:52 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1331318287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T03:13:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.310+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f35d81a0630 con 0x7f35d81025c0 2024-09-12T03:13:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.310+0000 7f35ce7fc640 1 -- 172.21.15.16:0/1570841832 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f35d009f8b0 con 0x7f35d81025c0 2024-09-12T03:13:53.311 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 msgr2=0x7f35c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 0x7f35c40a50d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f35d8101a60 tx=0x7f35d400b390 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 msgr2=0x7f35d819f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009fd0 tx=0x7f35d000b260 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 shutdown_connections 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f35c40a2c10 0x7f35c40a50d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d81025c0 0x7f35d819f410 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35d8101c00 0x7f35d81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.312+0000 7f35df9a8640 1 --2- 172.21.15.16:0/1570841832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35d8100a10 0x7f35d8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.313+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 >> 172.21.15.16:0/1570841832 conn(0x7f35d80fc190 msgr2=0x7f35d806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.313+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 shutdown_connections 2024-09-12T03:13:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.313+0000 7f35df9a8640 1 -- 172.21.15.16:0/1570841832 wait complete. 2024-09-12T03:13:53.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1043900616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:13:53.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:52 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1331318287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-12T03:13:53.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-12T03:13:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 -- 172.21.15.16:0/1794752241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 msgr2=0x7fc96c109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 --2- 172.21.15.16:0/1794752241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c109c80 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fc964009ca0 tx=0x7fc96402a8c0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 -- 172.21.15.16:0/1794752241 shutdown_connections 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 --2- 172.21.15.16:0/1794752241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c109c80 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 --2- 172.21.15.16:0/1794752241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c104ee0 0x7fc96c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 --2- 172.21.15.16:0/1794752241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c1025b0 0x7fc96c1049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 -- 172.21.15.16:0/1794752241 >> 172.21.15.16:0/1794752241 conn(0x7fc96c0fc1b0 msgr2=0x7fc96c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 -- 172.21.15.16:0/1794752241 shutdown_connections 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.426+0000 7fc972928640 1 -- 172.21.15.16:0/1794752241 wait complete. 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 Processor -- start 2024-09-12T03:13:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 -- start start 2024-09-12T03:13:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 0x7fc96c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c104ee0 0x7fc96c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10d7d0 con 0x7fc96c107810 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10d650 con 0x7fc96c1025b0 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc972928640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10d950 con 0x7fc96c104ee0 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc970ec5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc970ec5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50050/0 (socket says 172.21.15.16:50050) 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.427+0000 7fc970ec5640 1 -- 172.21.15.16:0/3790329483 learned_addr learned my addr 172.21.15.16:0/3790329483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc96bfff640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 0x7fc96c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc96b7fe640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c104ee0 0x7fc96c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 -- 172.21.15.16:0/3790329483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c104ee0 msgr2=0x7fc96c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c104ee0 0x7fc96c198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 -- 172.21.15.16:0/3790329483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 msgr2=0x7fc96c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 0x7fc96c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 -- 172.21.15.16:0/3790329483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c19fa20 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc96bfff640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 0x7fc96c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc970ec5640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc964031a40 tx=0x7fc964031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964003c30 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96c19fcb0 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc964003dd0 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96402fdb0 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.428+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96c1a01a0 con 0x7fc96c107810 2024-09-12T03:13:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.429+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96c0fdca0 con 0x7fc96c107810 2024-09-12T03:13:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.430+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc964004050 con 0x7fc96c107810 2024-09-12T03:13:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.431+0000 7fc9697fa640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 0x7fc9540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:13:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.431+0000 7fc96bfff640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 0x7fc9540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:13:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.431+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fc9640f0370 con 0x7fc96c107810 2024-09-12T03:13:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.431+0000 7fc96bfff640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 0x7fc9540a5050 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00baa0 tx=0x7fc95c005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:13:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.435+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc9640bd2e0 con 0x7fc96c107810 2024-09-12T03:13:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.658+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fc96c062e90 con 0x7fc96c107810 2024-09-12T03:13:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.667+0000 7fc9697fa640 1 -- 172.21.15.16:0/3790329483 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+740369 (secure 0 0 0) 0x7fc9640bcbe0 con 0x7fc96c107810 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 20, 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 14150, 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-12T03:13:53.678 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2564464416 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2564464416 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/2564464416", 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-09-12T02:38:59.915343+0000", 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540701547738038271, 2024-09-12T03:13:53.679 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24109, 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:53.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-12T03:13:53.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.685 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.687 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-09-12T03:13:53.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-12T03:13:53.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-12T03:13:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-12T03:13:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-12T03:13:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:53.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.711 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-12T03:13:53.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-12T03:13:53.715 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-12T03:13:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-12T03:13:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-12T03:13:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-12T03:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-12T03:13:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-12T03:13:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-12T03:13:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-12T03:13:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-12T03:13:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-12T03:13:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-12T03:13:53.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-12T03:13:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-12T03:13:53.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-12T03:13:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.764 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-12T03:13:53.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-12T03:13:53.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-12T03:13:53.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-12T03:13:53.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-12T03:13:53.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-12T03:13:53.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.796 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.799 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-12T03:13:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-12T03:13:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-12T03:13:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-12T03:13:53.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.820 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.823 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-12T03:13:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.829 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-12T03:13:53.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-12T03:13:53.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.832 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-12T03:13:53.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-12T03:13:53.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.842 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.851 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-12T03:13:53.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-12T03:13:53.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-12T03:13:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.857 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-12T03:13:53.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.867 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-12T03:13:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.885 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:53.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.887 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-12T03:13:53.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.894 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-12T03:13:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-12T03:13:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:53.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-12T03:13:53.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-12T03:13:53.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:53.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:53.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.929 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.932 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:53.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:53.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.947 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.954 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:53.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.960 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.963 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-12T03:13:53.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-12T03:13:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-12T03:13:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-12T03:13:53.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-12T03:13:53.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.982 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.985 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:53.994 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.995 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:53.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:53.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:54.000 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.002 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.020 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-12T03:13:54.025 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.025 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.025 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.026 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.027 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.029 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.030 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.031 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-12T03:13:54.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-12T03:13:54.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.039 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.041 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.042 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.051 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.055 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.060 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-12T03:13:54.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-12T03:13:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-12T03:13:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-12T03:13:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-12T03:13:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-12T03:13:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-12T03:13:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-12T03:13:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-12T03:13:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-12T03:13:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-12T03:13:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-12T03:13:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-12T03:13:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-12T03:13:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-12T03:13:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-12T03:13:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-12T03:13:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24116, 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-12T03:13:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-12T03:13:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-12T03:13:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-12T03:13:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-12T03:13:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-12T03:13:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-12T03:13:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-12T03:13:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-12T03:13:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-12T03:13:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-12T03:13:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-12T03:13:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-12T03:13:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-12T03:13:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-12T03:13:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-12T03:13:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-12T03:13:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-12T03:13:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-12T03:13:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-12T03:13:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-12T03:13:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-12T03:13:54.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-12T03:13:54.333 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-12T03:13:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-12T03:13:54.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-12T03:13:54.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-12T03:13:54.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.340 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-12T03:13:54.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-12T03:13:54.343 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-12T03:13:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-12T03:13:54.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-12T03:13:54.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-12T03:13:54.351 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.352 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.353 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.356 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.357 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.359 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.359 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.359 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.359 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.359 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.360 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.360 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.364 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-12T03:13:54.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-12T03:13:54.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-12T03:13:54.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.367 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-12T03:13:54.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-12T03:13:54.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.367 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-12T03:13:54.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-12T03:13:54.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-12T03:13:54.370 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.370 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.371 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.371 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-12T03:13:54.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-12T03:13:54.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.372 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.376 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-12T03:13:54.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-12T03:13:54.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-12T03:13:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-12T03:13:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.380 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-12T03:13:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-12T03:13:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-12T03:13:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-12T03:13:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-12T03:13:54.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.386 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-12T03:13:54.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-12T03:13:54.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-12T03:13:54.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-12T03:13:54.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-12T03:13:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.409 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.411 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-12T03:13:54.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-12T03:13:54.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.419 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-12T03:13:54.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.434 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.439 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.440 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.440 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.440 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.440 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.440 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.441 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.443 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.444 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.452 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-12T03:13:54.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-12T03:13:54.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-12T03:13:54.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.456 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:54.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:54.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-12T03:13:54.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.463 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.464 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.470 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-12T03:13:54.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-12T03:13:54.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.480 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-12T03:13:54.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.489 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.490 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:54.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-12T03:13:54.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.496 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-12T03:13:54.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-12T03:13:54.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-12T03:13:54.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.522 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.526 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.529 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-12T03:13:54.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.544 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-12T03:13:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-12T03:13:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.589 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.592 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.599 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.603 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.606 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.608 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.610 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-12T03:13:54.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.612 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.615 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.616 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.620 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-12T03:13:54.624 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.628 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.630 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.631 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.632 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:54.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:54.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:54.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.649 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-12T03:13:54.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.652 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.654 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:54.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.658 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.659 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-12T03:13:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-12T03:13:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-12T03:13:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-12T03:13:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-12T03:13:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-12T03:13:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-12T03:13:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.778 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-12T03:13:54.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-12T03:13:54.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "restful" 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-12T03:13:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.806 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.807 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.810 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.811 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-12T03:13:54.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-12T03:13:54.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-12T03:13:54.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:54.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-12T03:13:54.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-12T03:13:54.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-12T03:13:54.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.834 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-12T03:13:54.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.845 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-12T03:13:54.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-12T03:13:54.848 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-12T03:13:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-12T03:13:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.858 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-12T03:13:54.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.865 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-12T03:13:54.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.869 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-12T03:13:54.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-12T03:13:54.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.878 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-12T03:13:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-12T03:13:54.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.882 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.886 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-12T03:13:54.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-12T03:13:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-12T03:13:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-12T03:13:54.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.895 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-12T03:13:54.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.900 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.902 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-12T03:13:54.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-12T03:13:54.914 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-12T03:13:54.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-12T03:13:54.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-12T03:13:54.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-12T03:13:54.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-12T03:13:54.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.934 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:54.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:54.937 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.938 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-12T03:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-12T03:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-12T03:13:54.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-12T03:13:54.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-12T03:13:54.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-12T03:13:54.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.967 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-12T03:13:54.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-12T03:13:54.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-12T03:13:54.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.982 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-12T03:13:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-12T03:13:54.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-12T03:13:54.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:54.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:54.999 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-12T03:13:55.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.007 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.010 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-12T03:13:55.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.020 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.022 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.026 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.029 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-12T03:13:55.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-12T03:13:55.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-12T03:13:55.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-12T03:13:55.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.054 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-12T03:13:55.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-12T03:13:55.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.063 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-12T03:13:55.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-12T03:13:55.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-12T03:13:55.069 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.070 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.071 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.077 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.078 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.079 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-12T03:13:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-12T03:13:55.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.091 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-12T03:13:55.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.096 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.099 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-12T03:13:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-12T03:13:55.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.108 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.109 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-12T03:13:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.132 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.135 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-12T03:13:55.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.139 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.140 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.146 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.147 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.150 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.152 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.154 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.157 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-12T03:13:55.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.166 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.169 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-12T03:13:55.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-12T03:13:55.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-12T03:13:55.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.194 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.195 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.196 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-12T03:13:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.206 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.209 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.212 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.218 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-12T03:13:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-12T03:13:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-12T03:13:55.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.262 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-12T03:13:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-12T03:13:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-12T03:13:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-12T03:13:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-12T03:13:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-12T03:13:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-12T03:13:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-12T03:13:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-12T03:13:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-12T03:13:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-09-12T03:13:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-12T03:13:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-12T03:13:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-12T03:13:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-12T03:13:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 3, 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4118962901 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3465030604 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 56395524 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1304938134 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-12T03:13:55.341 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:53 smithi184 ceph-mon[35321]: pgmap v1116: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3057191490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1570841832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:53 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/3790329483' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:53 smithi033 ceph-mon[35092]: pgmap v1116: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3057191490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1570841832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:53 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/3790329483' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 msgr2=0x7fc9540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 0x7fc9540a5050 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00baa0 tx=0x7fc95c005d90 comp rx=0 tx=0).stop 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 msgr2=0x7fc96c19f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc964031a40 tx=0x7fc964031ee0 comp rx=0 tx=0).stop 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 shutdown_connections 2024-09-12T03:13:55.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fc9540a2b90 0x7fc9540a5050 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96c107810 0x7fc96c19f3b0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc96c104ee0 0x7fc96c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 --2- 172.21.15.16:0/3790329483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc96c1025b0 0x7fc96c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 >> 172.21.15.16:0/3790329483 conn(0x7fc96c0fc1b0 msgr2=0x7fc96c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 shutdown_connections 2024-09-12T03:13:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:13:53.672+0000 7fc972928640 1 -- 172.21.15.16:0/3790329483 wait complete. 2024-09-12T03:13:55.415 DEBUG:tasks.check_counter:Getting stats from a 2024-09-12T03:13:55.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-12T03:13:55.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:53 smithi016 ceph-mon[30925]: pgmap v1116: 97 pgs: 97 active+clean; 729 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-09-12T03:13:55.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3057191490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1570841832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-12T03:13:55.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:53 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/3790329483' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-12T03:13:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:56.076 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:55 smithi016 ceph-mon[30925]: pgmap v1117: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 420271, 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 215587, 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 113721958, 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 41046403, 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 50, 2024-09-12T03:13:56.182 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 18.788363779, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 7.084487803, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 15.998767087, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.079256774, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 215583, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 3.742486189, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000017359 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 630519, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.090037149, 2024-09-12T03:13:56.183 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000142 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 1472 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 125896, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 64472, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 34905208, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 11725308, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 12, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-09-12T03:13:56.184 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 5.917832774, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 2.222506521, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 5.016508684, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.017099859, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 64474, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 1.182127665, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000018334 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 188451, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.030442985, 2024-09-12T03:13:56.185 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000161 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 984128, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 298432 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 10421, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 15089, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3593839, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 14223214, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 47, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 10, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 1.710235990, 2024-09-12T03:13:56.186 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 1.146050446, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.666184824, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.044921290, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 15087, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.577823511, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000038299 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 21477, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.003572290, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000166 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3199568, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 9751408 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3722, 2024-09-12T03:13:56.188 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2.156038024, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000579268 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2473, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.103180819, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000041722 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.189 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 6062, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 5636, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5636, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 42.884734581, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.007609072 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 426, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-09-12T03:13:56.190 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 11, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 20, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 3, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 17, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 13, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 226, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 2, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 2, 2024-09-12T03:13:56.191 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 6, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 15462, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 7191, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 426, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 123, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 342, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 1, 2024-09-12T03:13:56.192 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 1, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 14, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 83, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 753, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 226845250, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 83, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:13:56.193 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 83, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 77, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 3755, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 1754, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 845, 2024-09-12T03:13:56.194 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 560, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 5121 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 280, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 1043, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 573, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 980, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 291, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 920, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 920, 2024-09-12T03:13:56.195 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 22, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 1464, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 1, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-09-12T03:13:56.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 6025, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 6025, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-09-12T03:13:56.197 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 8, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 8, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 2, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 11793428, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 0, 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-09-12T03:13:56.198 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1679, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 8.618247757, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.005132964 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 0 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 27, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 1060, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 1033, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 33, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 534, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 501, 2024-09-12T03:13:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 20, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 1891, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 1871, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 2, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 928, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 926, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 61152, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 223520 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 11483, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 16, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 6062, 2024-09-12T03:13:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 257, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 2218, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 912, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 31.945349279, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.035027795 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-09-12T03:13:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2117, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2.413314387, 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001139969 2024-09-12T03:13:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000175042, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000087521 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-09-12T03:13:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1334, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.646770032, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000484835 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 342, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 3.825106380, 2024-09-12T03:13:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.011184521 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-09-12T03:13:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 9, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.052119617, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.005791068 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-09-12T03:13:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 920, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.001899844, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004349891 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 1, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 3, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 2, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 1, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 559, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 559, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 1558, 2024-09-12T03:13:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:13:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:13:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 280039, 2024-09-12T03:13:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 107, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 7496, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 154, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:13:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 119269, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 478, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 3703, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 7836188, 2024-09-12T03:13:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 28, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 3703, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3703, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 18.687713724, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.005046641 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3703, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4067 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 3703, 2024-09-12T03:13:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 344, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 3359, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 10, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 261, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 1940, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 196, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-09-12T03:13:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 933, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 83, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 20, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-09-12T03:13:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-09-12T03:13:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 80, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 2, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 12, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 17, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 5, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 483, 2024-09-12T03:13:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 0, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 141 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 1, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 1487 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-09-12T03:13:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 6, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 6, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 920, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 920, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 556584, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 101697896, 2024-09-12T03:13:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 556584, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 556584, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 101697896, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3703, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 7929108, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3703, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 7929108, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3703, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 3703, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3703, 2024-09-12T03:13:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 3703, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 920, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 92920, 2024-09-12T03:13:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 920, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 261, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 92920, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x55f68e390960": { 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 6025, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 7599124, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 6025, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 1679, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 7599124, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:56.232 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:56.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:55 smithi033 ceph-mon[35092]: pgmap v1117: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-09-12T03:13:56.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:55 smithi184 ceph-mon[35321]: pgmap v1117: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-09-12T03:13:56.936 INFO:tasks.check_counter:Daemon mds.a mds.exported=1 2024-09-12T03:13:56.936 INFO:tasks.check_counter:Daemon mds.a mds.imported=1 2024-09-12T03:13:56.936 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=280 2024-09-12T03:13:56.936 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=1043 2024-09-12T03:13:56.936 DEBUG:tasks.check_counter:Getting stats from d 2024-09-12T03:13:56.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-12T03:13:57.200 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:57.684 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 465, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 458, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 826160, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 282960, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.058075229, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.026813521, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.036072828, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 458, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.022637692, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000049427 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 826160, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 282960 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-09-12T03:13:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 349, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 218960, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.024091580, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.019288059, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001442276, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 347, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.015302392, 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000044099 2024-09-12T03:13:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 218960 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-09-12T03:13:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.000327465, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000227788, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-12T03:13:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:13:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:13:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148761, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 31, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-12T03:13:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-12T03:13:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 463, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 757905, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 463, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 463, 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 757905, 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:57.696 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:58.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:57 smithi033 ceph-mon[35092]: pgmap v1118: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-09-12T03:13:58.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:57 smithi184 ceph-mon[35321]: pgmap v1118: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-09-12T03:13:58.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:57 smithi016 ceph-mon[30925]: pgmap v1118: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-09-12T03:13:58.346 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-12T03:13:58.346 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-12T03:13:58.346 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-09-12T03:13:58.346 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-09-12T03:13:58.346 DEBUG:tasks.check_counter:Getting stats from g 2024-09-12T03:13:58.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-12T03:13:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:13:59.081 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:13:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:13:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1678, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1678, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 565481, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 532781, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 7, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.163862580, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.065302151, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.037765216, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.045565309, 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1677, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.072258108, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000043087 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-12T03:13:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 3504, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 3848, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 8488046, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1805579, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.230869497, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.071765980, 2024-09-12T03:13:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.062793921, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.078842917, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3846, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.103469333, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000026903 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-12T03:13:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 691568 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 485, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 462, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 961268, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 282101, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.056285523, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.027338352, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.034920582, 2024-09-12T03:13:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000054688, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 461, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.020233533, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000043890 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 959920, 2024-09-12T03:13:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 281136 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8714, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.201612368, 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000023136 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-09-12T03:13:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-09-12T03:13:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 927, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 927, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 927, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 1, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-09-12T03:13:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-09-12T03:13:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-09-12T03:13:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-09-12T03:13:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-09-12T03:13:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-09-12T03:13:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 6025, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 8, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 2, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 11793428, 2024-09-12T03:13:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 11791136, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 6025 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 10, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 942, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 932, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 8, 2024-09-12T03:13:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 21, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 13, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 927, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 953, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 26, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 51940, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 215328 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-09-12T03:13:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-09-12T03:13:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.106 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.107 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.108 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.109 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.110 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 1, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 3, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 2, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 3587, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-12T03:13:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:13:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:13:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 199278, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 57, 2024-09-12T03:13:59.116 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 7496, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 154, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-12T03:13:59.117 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 685688, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 3803, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 5180, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-09-12T03:13:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 5180, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5180, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 8.081875991, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001560207 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5180, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 5181 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 5180, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 5180, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-09-12T03:13:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 3255, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 1923, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-09-12T03:13:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-09-12T03:13:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-09-12T03:13:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 80, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 2, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 6, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 9, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 3, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-09-12T03:13:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-09-12T03:13:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 5663, 2024-09-12T03:13:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 9477295, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 5663, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5663, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 9477295, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5180, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 7599129, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5180, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 7599129, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 5180, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 5180, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 5180, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 5180, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-12T03:13:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:13:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x55b2c4fd8b40": { 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:13:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:13:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:13:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:13:59.132 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:13:59.943 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-12T03:13:59.943 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-12T03:13:59.943 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-09-12T03:13:59.943 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-09-12T03:13:59.943 DEBUG:tasks.check_counter:Getting stats from j 2024-09-12T03:13:59.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-12T03:14:00.201 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:00.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:13:59 smithi033 ceph-mon[35092]: pgmap v1119: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.8 KiB/s wr, 9 op/s 2024-09-12T03:14:00.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:13:59 smithi184 ceph-mon[35321]: pgmap v1119: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.8 KiB/s wr, 9 op/s 2024-09-12T03:14:00.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:13:59 smithi016 ceph-mon[30925]: pgmap v1119: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.8 KiB/s wr, 9 op/s 2024-09-12T03:14:00.698 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T03:14:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-09-12T03:14:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-09-12T03:14:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-09-12T03:14:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-09-12T03:14:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001960974, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000520203, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:14:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:14:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 462, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 457, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 823360, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 283152, 2024-09-12T03:14:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.058642152, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.027981110, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.036278810, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:14:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 456, 2024-09-12T03:14:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.019013003, 2024-09-12T03:14:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000041695 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 823360, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 283152 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 352, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3392, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 219584, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.025970101, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.019080460, 2024-09-12T03:14:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.003588888, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 351, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.018590079, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000052963 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 219584 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 149735, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 464, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 758035, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 464, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-12T03:14:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 464, 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 758035, 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-12T03:14:00.714 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T03:14:01.426 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-12T03:14:01.426 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-12T03:14:01.426 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-09-12T03:14:01.426 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-09-12T03:14:01.427 DEBUG:tasks.check_counter:Getting stats from b 2024-09-12T03:14:01.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-12T03:14:01.748 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T03:14:02.204 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:01 smithi033 ceph-mon[35092]: pgmap v1120: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-12T03:14:02.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:01 smithi184 ceph-mon[35321]: pgmap v1120: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-12T03:14:02.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:01 smithi016 ceph-mon[30925]: pgmap v1120: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-09-12T03:14:02.932 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-12T03:14:02.932 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:02.932 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 147146, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 180767, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 33064557, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 122967442, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 34, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 12.713946729, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 6.117416863, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 6.596927432, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.801237397, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 180768, 2024-09-12T03:14:02.933 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 8.659876412, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000047906 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 284868, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.057922479, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000203 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 300594, 2024-09-12T03:14:02.934 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 413520, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 107142193, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 368583810, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 17, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 6, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 24.747120673, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 10.883854998, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 15.416006328, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 1.470769262, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 413516, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 20.264349730, 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000049004 2024-09-12T03:14:02.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 601986, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.106162390, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000176 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 98587936, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 281912848 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 540953, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 393042, 2024-09-12T03:14:02.936 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 146946798, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 139497109, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 347, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 10, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 30.710352769, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 12.895644627, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 22.596012273, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.581146333, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 393041, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 14.232563274, 2024-09-12T03:14:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000036211 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 898883, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.157714187, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000175 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1053040, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 334160 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-12T03:14:02.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 63301, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 73.870425888, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001166970 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 40098, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 3.959965905, 2024-09-12T03:14:02.939 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000098757 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 195721, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 135701, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 135701, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 761.692190108, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005613018 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 60020, 2024-09-12T03:14:02.940 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 67, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 4, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 3, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 463, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 231, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 220, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 12, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 132, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 407, 2024-09-12T03:14:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 1, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 1, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 5, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 302671, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 128684, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 60018, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 147, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 2699, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 4, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 168, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 13, 2024-09-12T03:14:02.942 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 129, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 1864, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 4, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 234, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 1864, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 462, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 34, 2024-09-12T03:14:02.943 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 428, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 1402, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 12, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 29535, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 95433, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 63698, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 15178, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 7098, 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 96623 2024-09-12T03:14:02.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 597, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 838, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 466, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 825, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 694, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 1, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 21771, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 21770, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:02.945 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 22, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 2337, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 34, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 4, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 7, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:02.946 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 104033, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 10005, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 1, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 104032, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 10004, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 140, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 12, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 1, 2024-09-12T03:14:02.947 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 139, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 9, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 11, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194352, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 206405323, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 0, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 32404, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 200.450674945, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006185985 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.948 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 0 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 32, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 22026, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 21994, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 49, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 658, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 609, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 463, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 42709, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 42246, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 1, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 24229, 2024-09-12T03:14:02.949 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 24228, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 76328, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 231712 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 264802, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 67, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 195721, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 3171, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 8950, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 21748, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 294.344575465, 2024-09-12T03:14:02.950 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.013534328 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 4426, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 27.750571468, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006269898 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-09-12T03:14:02.951 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 47230, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 47.153197984, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000998373 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:02.953 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.954 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.954 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.954 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:02.954 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 20, 2024-09-12T03:14:02.955 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.023018734, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001150936 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000293945, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000293945 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 35086, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 133.347231590, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.003800582 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 4322, 2024-09-12T03:14:02.957 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 118.201600819, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.027348820 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.958 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 12, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.085001033, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.007083419 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.959 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 27, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.249410362, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.009237420 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:02.960 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1062, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 2.786709329, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.002624020 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8, 2024-09-12T03:14:02.961 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.021445895, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.002680736 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 21759, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 137.729133484, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006329754 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.962 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 3, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 2, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 1, 2024-09-12T03:14:02.963 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 12969, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 12969, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 1588, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 120, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 120, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:02.964 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:02.965 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:02.966 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 303404, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 110, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:02.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 154, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 493597, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 5846, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:02.968 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 59259, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 207020751, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 37, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 59259, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 59259, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 357.141285359, 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006026785 2024-09-12T03:14:02.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 59259, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 60385 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 59259, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 2643, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 56616, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 0, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 51, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 745, 2024-09-12T03:14:02.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 33115, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 449, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 4, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 22354, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 1898, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 102, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:02.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-09-12T03:14:02.972 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 80, 2024-09-12T03:14:02.973 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 2, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 12, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 12, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 1213, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 454 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 1, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-09-12T03:14:02.974 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 2372 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 6, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 6, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 21770, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 21770, 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 988689, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 185727748, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 988689, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 988689, 2024-09-12T03:14:02.976 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 185727748, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:02.977 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 59259, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 209219521, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 59259, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 209219521, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.978 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 59259, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 59259, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 59259, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 59259, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.979 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 21770, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 2198770, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 21770, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:02.980 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 711, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 2198770, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x55dee252a960": { 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 104033, 2024-09-12T03:14:02.981 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 202211019, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 104033, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 32404, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 202211019, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:02.983 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-12T03:14:04.172 INFO:tasks.check_counter:Daemon mds.b mds.exported=4 2024-09-12T03:14:04.172 INFO:tasks.check_counter:Daemon mds.b mds.imported=13 2024-09-12T03:14:04.172 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=597 2024-09-12T03:14:04.172 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=838 2024-09-12T03:14:04.172 DEBUG:tasks.check_counter:Getting stats from e 2024-09-12T03:14:04.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-12T03:14:04.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:03 smithi033 ceph-mon[35092]: pgmap v1121: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T03:14:04.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:03 smithi184 ceph-mon[35321]: pgmap v1121: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T03:14:04.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:03 smithi016 ceph-mon[30925]: pgmap v1121: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-12T03:14:04.434 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 435713, 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 312080, 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 113564469, 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 128480776, 2024-09-12T03:14:05.122 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 16, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 22.513890371, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 8.501127907, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 17.334390549, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.762977930, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 312079, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 12.679851756, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000040630 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 706350, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.119335404, 2024-09-12T03:14:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000168 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 6448, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 4976 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 258348, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 330676, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 89722439, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 307520858, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 16, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 7, 2024-09-12T03:14:05.124 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 20.592876463, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 8.948900866, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 13.336005623, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 1.120581007, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 330675, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 15.729827622, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000047568 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 511144, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.091424302, 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000178 2024-09-12T03:14:05.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 84832912, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 231990432 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 214965, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 217082, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 51902199, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 96351377, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 75, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 10, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 16.697516547, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 8.282375514, 2024-09-12T03:14:05.126 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 9.449286064, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.697170347, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 217080, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 7.609722797, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000035054 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 400573, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.075325384, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000188 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 785056 2024-09-12T03:14:05.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 56755, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 58.346958930, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001028049 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 34446, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 2.970439293, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000086234 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 183013, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 119186, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 119186, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 608.082344739, 2024-09-12T03:14:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005101961 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 63827, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 11, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 22, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 5, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 3, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 14, 2024-09-12T03:14:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 14, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 227, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 5, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 278248, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 116397, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 63826, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 118, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 1686, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 4, 2024-09-12T03:14:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 11, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 4, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 122, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 1309, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 753, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 226845250, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 1309, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:05.132 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 160, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 160, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 1149, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 10, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 8601, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 60328, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 40022, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 11299, 2024-09-12T03:14:05.133 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 5764, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 88151 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 339, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 816, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 461, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 730, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 718, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 1, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 18419, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 18418, 2024-09-12T03:14:05.134 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 22, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 2107, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 4, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:05.135 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 89053, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 89053, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 123, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 123, 2024-09-12T03:14:05.136 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 9, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 174044436, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 0, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 29323, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 167.332692404, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005706533 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 0 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.137 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 24, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 18556, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 18532, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 30, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 408, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 378, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 22, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 33686, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 33664, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 0, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 20812, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 20812, 2024-09-12T03:14:05.138 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 69820, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 223520 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 249081, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 4377, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 183013, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 1605, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 18398, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 18335, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 203.924215992, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.011122127 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.139 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8711, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 70.939203421, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.008143634 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.140 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 36225, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 25.738125700, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000710507 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.141 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:05.142 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.143 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.144 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 34846, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 90.251956896, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.002590023 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-09-12T03:14:05.145 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2617, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 36.940056171, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.014115420 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.146 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 10, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.096527979, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.009652797 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.147 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 25, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.230339355, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.009213574 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:05.148 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.005407113, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000675889 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.149 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 18409, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 179.956512112, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.009775463 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:05.150 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 2, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 1, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 1, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 10955, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 10955, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 1505, 2024-09-12T03:14:05.151 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:05.152 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:05.153 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 304284, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 116, 2024-09-12T03:14:05.154 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 154, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:05.155 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 111200, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 670, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 51582, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 173488655, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 28, 2024-09-12T03:14:05.156 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 51582, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 51582, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 281.524506842, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005457805 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 51582, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 52542 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 51582, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 2054, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 49528, 2024-09-12T03:14:05.157 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 10, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 745, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 30068, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 429, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 18470, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-09-12T03:14:05.158 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 1309, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 20, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-09-12T03:14:05.159 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-09-12T03:14:05.160 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 80, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 2, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 12, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 16, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 4, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 1045, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 0, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 446 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.161 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 1, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 2138 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 6, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 6, 2024-09-12T03:14:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 18418, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 18418, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 909022, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 162753036, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 909022, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:05.163 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 909022, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 162753036, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:05.164 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 51582, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 175348873, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 51582, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 175348873, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.165 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 51582, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 51582, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 51582, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 51582, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.166 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 18418, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1860218, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 18418, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 745, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1860218, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x556c3860a960": { 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 89053, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 169850132, 2024-09-12T03:14:05.168 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 89053, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 29323, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 169850132, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:05.169 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-12T03:14:06.042 INFO:tasks.check_counter:Daemon mds.e mds.exported=4 2024-09-12T03:14:06.042 INFO:tasks.check_counter:Daemon mds.e mds.imported=4 2024-09-12T03:14:06.042 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=339 2024-09-12T03:14:06.042 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=816 2024-09-12T03:14:06.042 DEBUG:tasks.check_counter:Getting stats from h 2024-09-12T03:14:06.042 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-12T03:14:06.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:05 smithi033 ceph-mon[35092]: pgmap v1122: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-12T03:14:06.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:05 smithi184 ceph-mon[35321]: pgmap v1122: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-12T03:14:06.297 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T03:14:06.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:05 smithi016 ceph-mon[30925]: pgmap v1122: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-12T03:14:06.799 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-12T03:14:06.799 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 215730, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 201125, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 52894661, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 78559504, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 40, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 5, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 14.860324365, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 7.136570219, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 9.040675842, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.329218998, 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:06.800 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 201125, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 6.166171900, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000030658 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 391211, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.068472759, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000175 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1003392, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 306752 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.801 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 247305, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 319832, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 79512562, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 251710724, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 16, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 6, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 19.252811130, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 8.275315487, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 12.916708608, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.833174857, 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:06.802 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 319830, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 17.108337123, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000053491 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 510001, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.089558154, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000175 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 70435744, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 176198224 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.803 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 410441, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 263420, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 109064526, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 116939019, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 51, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 10, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 21.249085304, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 7.968799701, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 16.158581819, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.849708474, 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:06.804 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 263418, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 8.842405169, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000033567 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 633724, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.104174502, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000164 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.805 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 55516, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 49.610026016, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000893616 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.806 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 22589, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 1.867721009, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000082682 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 151660, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 86083, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 86083, 2024-09-12T03:14:06.807 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 516.920740411, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006004910 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 65577, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 11, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 28, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 9, 2024-09-12T03:14:06.808 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 8, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 11, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 14, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 208, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 0, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 5, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 220866, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 92896, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 65577, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 116, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:06.809 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 2159, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 2, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 1, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 11, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 125, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 1441, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 753, 2024-09-12T03:14:06.810 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 226845250, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 1441, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 160, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 160, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 1281, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 8, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 8014, 2024-09-12T03:14:06.811 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 41084, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 39993, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 14375, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 5981, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 57143 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 229, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 876, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 489, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 801, 2024-09-12T03:14:06.812 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 586, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 1, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 11987, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 11986, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-09-12T03:14:06.813 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 22, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 2245, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 2, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-09-12T03:14:06.814 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 71181, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 71181, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 89, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 89, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 7, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-09-12T03:14:06.815 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 149238218, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 0, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 28329, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 175.403775576, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.006191668 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 0 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-09-12T03:14:06.816 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 25, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 12115, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 12090, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 31, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 443, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 412, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 28, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 23496, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 23468, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 0, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 13754, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 13754, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 68792, 2024-09-12T03:14:06.817 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 223520 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 193447, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 9632, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 151660, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 2666, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 21126, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 11900, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 204.329210857, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.017170521 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.818 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8941, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 91.584135392, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.010243164 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.819 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 29402, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 25.074820550, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000852827 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:06.821 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-09-12T03:14:06.823 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 19711, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 70.921068594, 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.003598045 2024-09-12T03:14:06.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3584, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 25.921097979, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.007232449 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:06.825 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 10, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.057773606, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005777360 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 46, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 30.644095811, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.666175995 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 512, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.362228404, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000707477 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-09-12T03:14:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 11977, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 68.026309218, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005679745 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-09-12T03:14:06.830 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 2, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 1, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 1, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 8241, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 8241, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 1592, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:06.831 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:06.832 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:06.833 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 296364, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 115, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:06.834 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 154, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-12T03:14:06.835 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 119080, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 724, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 43724, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 147645169, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 28, 2024-09-12T03:14:06.836 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 43724, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 43724, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 250.765997101, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.005735202 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 43724, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 44567 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 43724, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 1968, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 41756, 2024-09-12T03:14:06.837 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 10, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 527, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 28856, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 313, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 12032, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:06.838 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 1441, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 20, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-09-12T03:14:06.839 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-09-12T03:14:06.840 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 80, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 2, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 12, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 17, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 5, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 1019, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 0, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 349 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.841 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 1, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 2274 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 6, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-09-12T03:14:06.842 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 6, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 11986, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 11986, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 873472, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 153822955, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 873472, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:06.843 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 873472, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 153822955, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:06.844 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 43724, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 148855755, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 43724, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 148855755, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.845 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 43724, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 43724, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 43724, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 43724, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.846 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 11986, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1210586, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 11986, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 527, 2024-09-12T03:14:06.847 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1210586, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x559d2fa90960": { 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 71181, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 145043914, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:06.848 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 71181, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 28329, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 145043914, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:06.849 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-12T03:14:07.687 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-09-12T03:14:07.687 INFO:tasks.check_counter:Daemon mds.h mds.imported=11 2024-09-12T03:14:07.687 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=229 2024-09-12T03:14:07.687 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=876 2024-09-12T03:14:07.687 DEBUG:tasks.check_counter:Getting stats from k 2024-09-12T03:14:07.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-12T03:14:07.940 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.b/config 2024-09-12T03:14:08.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:07 smithi033 ceph-mon[35092]: pgmap v1123: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-12T03:14:08.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:07 smithi184 ceph-mon[35321]: pgmap v1123: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-12T03:14:08.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:07 smithi016 ceph-mon[30925]: pgmap v1123: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2231, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2231, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 411620906, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 708698, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 8, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.380833638, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.049923536, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.265570144, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.050171744, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2230, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.076035981, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000034096 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 448 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 1953, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2303, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 378594979, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 1316597, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 7, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 6, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.380337420, 2024-09-12T03:14:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.050148638, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.266788153, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.046905679, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2301, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.068892326, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000029940 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 695856 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 3186, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 3160, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 224102458, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 1142630, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 5, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.404896128, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.094713932, 2024-09-12T03:14:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.222941815, 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.075490266, 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3159, 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.120779685, 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000038233 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 979920, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 286256 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 11129, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.450477307, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000040477 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 0, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 0, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 0, 2024-09-12T03:14:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 99332, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 53258, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 46074, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 23248, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 1, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 0, 2024-09-12T03:14:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:14:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 0, 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 0 2024-09-12T03:14:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 479051, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-09-12T03:14:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 134, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 40, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 784667281, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 1015627262, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 1015626124, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 479051 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 25, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 99515, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 99490, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 71, 2024-09-12T03:14:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 588, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 517, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 99332, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 100632, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 1300, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 0, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 0, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 0, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 137940, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 215328 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-09-12T03:14:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-09-12T03:14:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 3, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 2, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 2, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 0, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 0, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 4169, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 240699, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 93, 2024-09-12T03:14:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 154, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 70086874, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 399086, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 6872, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 21, 2024-09-12T03:14:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 3, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 6872, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 6872, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 9.558845151, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001390984 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 6872, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 6873 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 6872, 2024-09-12T03:14:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 6872, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 3866, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 3004, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 0, 2024-09-12T03:14:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 1, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-09-12T03:14:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 80, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 2, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 11, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 11, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 0, 2024-09-12T03:14:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 2, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 0 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 0, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 0 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 7366, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1013627030, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 7366, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 7366, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1013627030, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 6872, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 1011445165, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 6872, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1011445165, 2024-09-12T03:14:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 6872, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 6872, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 6872, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 6872, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x563788b6ab40": { 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-09-12T03:14:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-09-12T03:14:08.490 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-09-12T03:14:09.226 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-12T03:14:09.226 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-12T03:14:09.226 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-09-12T03:14:09.226 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-09-12T03:14:09.226 DEBUG:tasks.check_counter:Getting stats from c 2024-09-12T03:14:09.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-12T03:14:09.643 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T03:14:10.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:09 smithi184 ceph-mon[35321]: pgmap v1124: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T03:14:10.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:09 smithi033 ceph-mon[35092]: pgmap v1124: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T03:14:10.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-12T03:14:10.386 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:10.386 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 3274, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 3274, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 93254455, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 1040318, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 8, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.307920203, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.075518968, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.125133236, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.084392549, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 3273, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.105445055, 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000032216 2024-09-12T03:14:10.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 448 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 1044, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1392, 2024-09-12T03:14:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 64062887, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 1116942, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 9, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 6, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.129008877, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.036991560, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.056352469, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.026419718, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1390, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.052557745, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000037811 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 785520 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 1934, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1908, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 47485577, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 743371, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 5, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.195257364, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.066036729, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.090690572, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.036289746, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1907, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.067110769, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000035191 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 979264, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 284496 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 9596, 2024-09-12T03:14:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.218326275, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000022751 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-09-12T03:14:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 0, 2024-09-12T03:14:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 21803, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 21803, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 21803, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 2, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 0, 2024-09-12T03:14:10.396 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 0, 2024-09-12T03:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 4, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 234, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 0, 2024-09-12T03:14:10.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 0 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 0, 2024-09-12T03:14:10.399 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:10.400 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:10.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:10.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-09-12T03:14:10.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 0, 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 0, 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 0, 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 0, 2024-09-12T03:14:10.403 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 104033, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 0, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 0, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 0, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 0, 2024-09-12T03:14:10.404 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 139, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 10, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 0, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 4194352, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 206405323, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 206401022, 2024-09-12T03:14:10.405 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 104033 2024-09-12T03:14:10.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 24, 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 21880, 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 21856, 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 26, 2024-09-12T03:14:10.407 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 90, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 64, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 21803, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 22134, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 331, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-09-12T03:14:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 0, 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 0, 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 72536, 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 215328 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 0, 2024-09-12T03:14:10.409 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 0, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 0, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 0, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 0, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.411 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-09-12T03:14:10.412 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.413 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.413 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.413 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-09-12T03:14:10.413 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.414 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.416 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-09-12T03:14:10.417 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.418 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.418 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.419 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:09 smithi016 ceph-mon[30925]: pgmap v1124: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-12T03:14:10.419 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-09-12T03:14:10.420 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.420 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.420 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.420 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-09-12T03:14:10.421 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-09-12T03:14:10.422 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:10.423 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.424 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.426 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.427 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:10.428 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.428 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.428 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.428 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.429 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-09-12T03:14:10.430 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.431 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 3, 2024-09-12T03:14:10.432 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 2, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 2, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 0, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 0, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 4132, 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:10.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:10.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 240252, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 81, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 154, 2024-09-12T03:14:10.438 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 15437421, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 87656, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-09-12T03:14:10.439 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 5755, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 26, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 4, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 5755, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5755, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 4.995899535, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000868097 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5755, 2024-09-12T03:14:10.440 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 5756 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 5755, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 5755, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 0, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 3372, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 0, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 2380, 2024-09-12T03:14:10.441 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 2, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:10.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-09-12T03:14:10.443 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 80, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-09-12T03:14:10.444 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 2, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 11, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 12, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 1, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 0, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 2, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 0 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-09-12T03:14:10.445 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 0, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 0 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:10.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 6248, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 204212293, 2024-09-12T03:14:10.447 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 6248, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 6248, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 204212293, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:10.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5755, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 202219289, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5755, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 202219289, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.449 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5755, 2024-09-12T03:14:10.450 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 5755, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5755, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 5755, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:10.451 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:10.452 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x5653e356eb40": { 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:10.453 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:10.454 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:10.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:10.454 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-12T03:14:11.446 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-12T03:14:11.446 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-12T03:14:11.446 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-09-12T03:14:11.446 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-09-12T03:14:11.447 DEBUG:tasks.check_counter:Getting stats from f 2024-09-12T03:14:11.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-12T03:14:11.704 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T03:14:12.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:11 smithi184 ceph-mon[35321]: pgmap v1125: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-12T03:14:12.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:11 smithi033 ceph-mon[35092]: pgmap v1125: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-12T03:14:12.291 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-12T03:14:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 2598, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 2570, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 29258455, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 954178, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 7, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 3, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.250296201, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.076313961, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.106325566, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.060311619, 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2570, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.100890714, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000039257 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 982448, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 285424 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 748, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1097, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 50521276, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 1022986, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 7, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.107095025, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.025724001, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.055150309, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.018893116, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1095, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.049872532, 2024-09-12T03:14:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000045545 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 785696 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 2688, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 2690, 2024-09-12T03:14:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 67793182, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 854791, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 7, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.206850533, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.051200365, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.084692782, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.059411662, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2688, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.080684789, 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000030016 2024-09-12T03:14:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 9391, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.206926664, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000022034 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 12006, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 12006, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 12006, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-09-12T03:14:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 1, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 0, 2024-09-12T03:14:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:14:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 0, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 0 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-09-12T03:14:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 0, 2024-09-12T03:14:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 71181, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 0, 2024-09-12T03:14:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 0, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 89, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 7, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 0, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 4194304, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 149238218, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 149237619, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 71181 2024-09-12T03:14:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 14, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 12013, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 11999, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 12, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 36, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 24, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 12006, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 12148, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 142, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 0, 2024-09-12T03:14:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 61456, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 215328 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 2, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 1, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 1, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 0, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 0, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 2865, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 232229, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 79, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-09-12T03:14:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 154, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 8499619, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 48272, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-09-12T03:14:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 5537, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 21, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 3, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 5537, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5537, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 6.039878703, 2024-09-12T03:14:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.001090821 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5537, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 5538 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 5537, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 5537, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 0, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 3272, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 0, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 2263, 2024-09-12T03:14:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 1, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-09-12T03:14:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 80, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-09-12T03:14:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 2, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 10, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 11, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 1, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 0, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 2, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 0 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 0, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-09-12T03:14:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 0 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 0, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 6030, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 147001901, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 6030, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 6030, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 147001901, 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5537, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 145045319, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5537, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 145045319, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-09-12T03:14:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5537, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 5537, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5537, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 5537, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x55ae77054b40": { 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:12.340 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-12T03:14:12.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:11 smithi016 ceph-mon[30925]: pgmap v1125: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-09-12T03:14:13.144 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-12T03:14:13.144 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-12T03:14:13.144 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-09-12T03:14:13.144 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-09-12T03:14:13.145 DEBUG:tasks.check_counter:Getting stats from i 2024-09-12T03:14:13.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-12T03:14:13.395 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 532161, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 867709, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 106646119, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 536767156, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 10, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 29.766543488, 2024-09-12T03:14:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 12.355265148, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 16.508827695, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 2.008976119, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 867709, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 23.990138429, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000027647 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1237693, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.184611097, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000149 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 0 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 1042924, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1493077, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 366928867, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 1689206208, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 18, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 7, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 65.487727209, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 30.218427266, 2024-09-12T03:14:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 41.881395941, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 3.810264485, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1493077, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 47.460802725, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000031787 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2131680, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.297430649, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000139 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 339576592, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 1236895856 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 558642, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 857449, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 117508422, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 554401767, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 76, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 11, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 30.186196697, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 12.750823575, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 16.650092452, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 2.301009857, 2024-09-12T03:14:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 857445, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 27.444898458, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000032007 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1245441, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.177827867, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000142 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-09-12T03:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 788608 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 239810, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 309.542185685, 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.001290780 2024-09-12T03:14:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 184575, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 29.490582456, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000159775 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 653910, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 630215, 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-09-12T03:14:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 630215, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 5085.558425802, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.008069561 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 23695, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 629, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 49, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 65, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 6584, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 3868, 2024-09-12T03:14:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 2704, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 12, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 1056, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 597, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 7, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 1036573, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 579738, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 23694, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 70, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 11361, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 26, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 380, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 8, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 170, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 10256, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 753, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 226845250, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 0, 2024-09-12T03:14:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 10256, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 1476, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 1476, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 8780, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 68, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 50124, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 375544, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 222718, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 78784, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 37900, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 409855 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 357, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 461, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 243, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 512, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 1559, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 12, 2024-09-12T03:14:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 99228, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 99216, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 22, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 7095, 2024-09-12T03:14:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 27, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 114, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 0, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 479051, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 373559, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 368701, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 110350, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-09-12T03:14:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 4858, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 568, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 440, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 434, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 134, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 9, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 6, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 784667281, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 1015627262, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 0, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 134285, 2024-09-12T03:14:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1116.485989815, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.008314301 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 0 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 41, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 99558, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 99517, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 88, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 838, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 750, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 6584, 2024-09-12T03:14:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 203586, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 197002, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 111390, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 111390, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 102248, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 223520 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 886888, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 8527, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 653910, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 14063, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 17182, 2024-09-12T03:14:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 99321, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1533.144973362, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.015436261 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 21979, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 274.232048400, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.012477002 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 231538, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 775.933571984, 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003351214 2024-09-12T03:14:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-09-12T03:14:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 99, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.913778328, 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.009230084 2024-09-12T03:14:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-09-12T03:14:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 9, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.005644271, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000627141 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 153581, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1145.952946639, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.007461554 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 20932, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 291.614257386, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.013931504 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-09-12T03:14:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 92, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.801839290, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.008715644 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 207, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 95.616129335, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.461913668 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 3310, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 10.769437831, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003253606 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 99147, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 956.573798976, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.009648035 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 3, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 2, 2024-09-12T03:14:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 1, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 33714, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 33714, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 1515, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 306915, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 135, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 440, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 5, 2024-09-12T03:14:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 154, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 5216471, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 73866, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 252590, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 1037511661, 2024-09-12T03:14:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 30, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 252590, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 252590, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1402.072887143, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.005550785 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 252590, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 256478 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 252590, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 12582, 2024-09-12T03:14:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 240008, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 86, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 481, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 2326, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 136611, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 1098, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 48, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 100020, 2024-09-12T03:14:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 10256, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 962, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-09-12T03:14:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-09-12T03:14:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 80, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 2, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 12, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 12, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 3363, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 0, 2024-09-12T03:14:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 1227 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 1, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 7190 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 6, 2024-09-12T03:14:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 6, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 99216, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 99216, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 2133723, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 356526246, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 2133723, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 2133723, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 356526246, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 252590, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 1047532477, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 252590, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 1047532477, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 252590, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 252590, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 252590, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 252590, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 99216, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 10020816, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 99216, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 2326, 2024-09-12T03:14:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 10020816, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x5561e54a0960": { 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 479051, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 1011432958, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 479051, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 134285, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 1011432958, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:13.941 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-12T03:14:14.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:13 smithi184 ceph-mon[35321]: pgmap v1126: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:13 smithi016 ceph-mon[30925]: pgmap v1126: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:14.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:13 smithi033 ceph-mon[35092]: pgmap v1126: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:14.534 INFO:tasks.check_counter:Daemon mds.i mds.exported=26 2024-09-12T03:14:14.534 INFO:tasks.check_counter:Daemon mds.i mds.imported=8 2024-09-12T03:14:14.534 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=357 2024-09-12T03:14:14.534 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=461 2024-09-12T03:14:14.535 DEBUG:tasks.check_counter:Getting stats from l 2024-09-12T03:14:14.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-12T03:14:14.782 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.c/config 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 3107, 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 3107, 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 47073775, 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 987203, 2024-09-12T03:14:15.345 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 8, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.281299174, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.083678924, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.094191302, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.081322225, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 3106, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.103618451, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000033360 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:15.346 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 448 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 2140, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 2114, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 76889908, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 811036, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 7, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-09-12T03:14:15.347 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.216905338, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.067990768, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.105443490, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.043465551, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2113, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.069508491, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000032895 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.348 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 980736, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 288176 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 806, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1158, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 48421973, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 1041896, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 8, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-09-12T03:14:15.349 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.104864250, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.025862114, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.052064674, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.019640607, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1156, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.042878101, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000037091 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 786224 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 9278, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.214789198, 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000023150 2024-09-12T03:14:15.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 0, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 0, 2024-09-12T03:14:15.352 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-09-12T03:14:15.353 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 18438, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 18438, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 18438, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 1, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 0, 2024-09-12T03:14:15.354 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 0, 2024-09-12T03:14:15.355 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 0, 2024-09-12T03:14:15.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 0 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 0, 2024-09-12T03:14:15.357 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-09-12T03:14:15.358 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-09-12T03:14:15.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 89053, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 0, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 123, 2024-09-12T03:14:15.360 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 9, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 0, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 4194304, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 174044436, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 174042144, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 89053 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.361 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 14, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 18453, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 18439, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 12, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 41, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 29, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 18438, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 18618, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 180, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 0, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 0, 2024-09-12T03:14:15.362 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 67220, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 215328 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-09-12T03:14:15.364 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.366 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.368 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.370 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-09-12T03:14:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.372 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-09-12T03:14:15.373 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 2, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 1, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 2, 2024-09-12T03:14:15.375 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 4135, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-09-12T03:14:15.376 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-12T03:14:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-09-12T03:14:15.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 241804, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 84, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 7496, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 154, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-09-12T03:14:15.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 13027747, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 74000, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 5551, 2024-09-12T03:14:15.380 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 21, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 3, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 5551, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5551, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 4.293913506, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000773538 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5551, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 5552 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 5551, 2024-09-12T03:14:15.381 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 5551, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 3278, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 2271, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-09-12T03:14:15.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 1, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-09-12T03:14:15.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-09-12T03:14:15.384 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 80, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 2, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 11, 2024-09-12T03:14:15.385 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 13, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 2, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 0, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 2, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 0 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 0, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-09-12T03:14:15.386 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 0 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 0, 2024-09-12T03:14:15.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 0, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 6049, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 171812674, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 6049, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 6049, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 171812674, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5551, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 169852444, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5551, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 169852444, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 5551, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 5551, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5551, 2024-09-12T03:14:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 5551, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x55ecfe6c8b40": { 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-09-12T03:14:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-12T03:14:15.395 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-12T03:14:16.135 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-12T03:14:16.135 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-12T03:14:16.135 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-09-12T03:14:16.135 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-09-12T03:14:16.135 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-12T03:14:16.149 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-12T03:14:16.150 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T03:14:16.150 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-12T03:14:16.174 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T03:14:16.174 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-12T03:14:16.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd blocklist ls 2024-09-12T03:14:16.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:16 smithi184 ceph-mon[35321]: pgmap v1127: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-12T03:14:16.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:16 smithi016 ceph-mon[30925]: pgmap v1127: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-12T03:14:16.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:16 smithi033 ceph-mon[35092]: pgmap v1127: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-12T03:14:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1627912650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 msgr2=0x7fde14106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1627912650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14106e60 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fde10009930 tx=0x7fde1002a420 comp rx=0 tx=0).stop 2024-09-12T03:14:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1627912650 shutdown_connections 2024-09-12T03:14:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1627912650 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde1410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1627912650 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 0x7fde14100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1627912650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14106e60 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1627912650 >> 172.21.15.16:0/1627912650 conn(0x7fde140fc7a0 msgr2=0x7fde140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1627912650 shutdown_connections 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.044+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1627912650 wait complete. 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 Processor -- start 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 -- start start 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 0x7fde14197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde14197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde14110e80 con 0x7fde14106a80 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde14110d00 con 0x7fde14101420 2024-09-12T03:14:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1c3e9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde14111000 con 0x7fde14100a80 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1a986640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1a986640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38854/0 (socket says 172.21.15.16:38854) 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1a986640 1 -- 172.21.15.16:0/1830873984 learned_addr learned my addr 172.21.15.16:0/1830873984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.045+0000 7fde1a185640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 0x7fde14197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde19984640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde14197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 -- 172.21.15.16:0/1830873984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 msgr2=0x7fde14197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 0x7fde14197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 -- 172.21.15.16:0/1830873984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 msgr2=0x7fde14197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde14197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 -- 172.21.15.16:0/1830873984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde141a6990 con 0x7fde14106a80 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde19984640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde14197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:14:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1a986640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00eb10 tx=0x7fde0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c004b20 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0c004cc0 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde141a6c80 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c010760 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.046+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde141a71c0 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.047+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde14100ee0 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.048+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fde0c005020 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.049+0000 7fde0b7fe640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 0x7fde000a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.049+0000 7fde1a185640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 0x7fde000a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.049+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fde0c0cf5d0 con 0x7fde14106a80 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.050+0000 7fde1a185640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 0x7fde000a50c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fde10009900 tx=0x7fde100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.054+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fde0c09c4e0 con 0x7fde14106a80 2024-09-12T03:14:17.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.243+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fde1402d7c0 con 0x7fde14106a80 2024-09-12T03:14:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.243+0000 7fde0b7fe640 1 -- 172.21.15.16:0/1830873984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v80) ==== 81+0+586 (secure 0 0 0) 0x7fde0c09bde0 con 0x7fde14106a80 2024-09-12T03:14:17.243 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3425705974 2024-09-13T02:38:59.915197+0000 2024-09-12T03:14:17.243 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/744312028 2024-09-13T02:38:59.915197+0000 2024-09-12T03:14:17.243 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/744312028 2024-09-13T02:38:59.915197+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/540700595 2024-09-13T02:38:59.915197+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2390815323 2024-09-13T02:38:59.915197+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2593008088 2024-09-13T02:38:34.945193+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2594473615 2024-09-13T02:38:34.945193+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2594473615 2024-09-13T02:38:34.945193+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1418983 2024-09-13T02:38:34.945193+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3307112498 2024-09-13T02:38:34.945193+0000 2024-09-12T03:14:17.244 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2024-09-12T03:14:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 msgr2=0x7fde000a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 0x7fde000a50c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fde10009900 tx=0x7fde100058d0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 msgr2=0x7fde14197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00eb10 tx=0x7fde0c00c2d0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 shutdown_connections 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.245+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fde000a2c00 0x7fde000a50c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde14106a80 0x7fde14197ba0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde14101420 0x7fde14197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 --2- 172.21.15.16:0/1830873984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde14100a80 0x7fde14197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 >> 172.21.15.16:0/1830873984 conn(0x7fde140fc7a0 msgr2=0x7fde140fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 shutdown_connections 2024-09-12T03:14:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:17.246+0000 7fde1c3e9640 1 -- 172.21.15.16:0/1830873984 wait complete. 2024-09-12T03:14:17.965 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-12T03:14:17.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T03:14:17.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-12T03:14:18.163 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T03:14:18.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T03:14:18.163 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-12T03:14:18.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:18 smithi184 ceph-mon[35321]: pgmap v1128: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T03:14:18.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:18 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1830873984' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-12T03:14:18.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T03:14:18.299 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-12T03:14:18.343 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-12T03:14:18.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T03:14:18.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:18 smithi016 ceph-mon[30925]: pgmap v1128: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T03:14:18.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:18 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1830873984' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-12T03:14:18.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-12T03:14:18.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-09-12T03:14:18.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:18 smithi033 ceph-mon[35092]: pgmap v1128: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-12T03:14:18.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:18 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1830873984' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-12T03:14:18.516 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-12T03:14:18.528 INFO:tasks.cephadm:Teardown begin 2024-09-12T03:14:18.528 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-12T03:14:18.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:14:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 -- 172.21.15.16:0/3554286556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102570 msgr2=0x7fa0a81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 --2- 172.21.15.16:0/3554286556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102570 0x7fa0a81029b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fa09c009900 tx=0x7fa09c02a480 comp rx=0 tx=0).stop 2024-09-12T03:14:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 -- 172.21.15.16:0/3554286556 shutdown_connections 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 --2- 172.21.15.16:0/3554286556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102ef0 0x7fa0a810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 --2- 172.21.15.16:0/3554286556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102570 0x7fa0a81029b0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 --2- 172.21.15.16:0/3554286556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 0x7fa0a8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 -- 172.21.15.16:0/3554286556 >> 172.21.15.16:0/3554286556 conn(0x7fa0a80fc7d0 msgr2=0x7fa0a80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 -- 172.21.15.16:0/3554286556 shutdown_connections 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.496+0000 7fa0af583640 1 -- 172.21.15.16:0/3554286556 wait complete. 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 Processor -- start 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 -- start start 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 0x7fa0a8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102570 0x7fa0a8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a81120a0 con 0x7fa0a8102ef0 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a8111f20 con 0x7fa0a8100e20 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0af583640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a8112220 con 0x7fa0a8102570 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38862/0 (socket says 172.21.15.16:38862) 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0ad31f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 0x7fa0a8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 -- 172.21.15.16:0/1688401209 learned_addr learned my addr 172.21.15.16:0/1688401209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0acb1e640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102570 0x7fa0a8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 -- 172.21.15.16:0/1688401209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102570 msgr2=0x7fa0a8198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102570 0x7fa0a8198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 -- 172.21.15.16:0/1688401209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 msgr2=0x7fa0a8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 0x7fa0a8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 -- 172.21.15.16:0/1688401209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a819d510 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.497+0000 7fa0adb20640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fa098009fd0 tx=0x7fa09800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.498+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098017070 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.498+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa098003e50 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.498+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a819d800 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.498+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0980106c0 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.498+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a81a50e0 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.499+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a8101200 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.500+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa098003970 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.500+0000 7fa0a67fc640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 0x7fa0900a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.501+0000 7fa0ad31f640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 0x7fa0900a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.501+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7fa0980cec20 con 0x7fa0a8102ef0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.501+0000 7fa0ad31f640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 0x7fa0900a50c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bbe0 tx=0x7fa0a000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.505+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa09809b540 con 0x7fa0a8102ef0 2024-09-12T03:14:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.716+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0a8109090 con 0x7fa0a8102ef0 2024-09-12T03:14:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.717+0000 7fa0a67fc640 1 -- 172.21.15.16:0/1688401209 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11978 (secure 0 0 0) 0x7fa09809f840 con 0x7fa0a8102ef0 2024-09-12T03:14:19.717 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:14:19.717 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"btime":"2024-09-12T03:13:28:924710+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:13:28.852245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:14:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-09-12T03:14:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 msgr2=0x7fa0900a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 0x7fa0900a50c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bbe0 tx=0x7fa0a000b390 comp rx=0 tx=0).stop 2024-09-12T03:14:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 msgr2=0x7fa0a819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fa098009fd0 tx=0x7fa09800b260 comp rx=0 tx=0).stop 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 shutdown_connections 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa0900a2c00 0x7fa0900a50c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0a8102ef0 0x7fa0a819ce30 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0a8102570 0x7fa0a8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 --2- 172.21.15.16:0/1688401209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0a8100e20 0x7fa0a8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.719+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 >> 172.21.15.16:0/1688401209 conn(0x7fa0a80fc7d0 msgr2=0x7fa0a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.720+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 shutdown_connections 2024-09-12T03:14:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:19.720+0000 7fa0af583640 1 -- 172.21.15.16:0/1688401209 wait complete. 2024-09-12T03:14:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:20 smithi184 ceph-mon[35321]: pgmap v1129: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:20.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:20 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1688401209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:20 smithi016 ceph-mon[30925]: pgmap v1129: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:20.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:20 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1688401209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:20 smithi033 ceph-mon[35092]: pgmap v1129: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-12T03:14:20.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:20 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1688401209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:20.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-12T03:14:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 -- 172.21.15.16:0/2950929334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac8102d30 msgr2=0x7f3ac8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/2950929334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac8102d30 0x7f3ac8103110 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009930 tx=0x7f3ac402a440 comp rx=0 tx=0).stop 2024-09-12T03:14:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 -- 172.21.15.16:0/2950929334 shutdown_connections 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/2950929334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 0x7f3ac810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/2950929334 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac81036e0 0x7f3ac8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/2950929334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac8102d30 0x7f3ac8103110 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 -- 172.21.15.16:0/2950929334 >> 172.21.15.16:0/2950929334 conn(0x7f3ac80fc7d0 msgr2=0x7f3ac80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 -- 172.21.15.16:0/2950929334 shutdown_connections 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.356+0000 7f3ad01e6640 1 -- 172.21.15.16:0/2950929334 wait complete. 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 Processor -- start 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 -- start start 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac8102d30 0x7f3ac8100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 0x7f3ac8101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac81120a0 con 0x7f3ac81036e0 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac8111f20 con 0x7f3ac8102d30 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.357+0000 7f3ad01e6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac8112220 con 0x7f3ac81082a0 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38882/0 (socket says 172.21.15.16:38882) 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 -- 172.21.15.16:0/4085818645 learned_addr learned my addr 172.21.15.16:0/4085818645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3ace783640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 0x7f3ac8101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acdf82640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac8102d30 0x7f3ac8100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 -- 172.21.15.16:0/4085818645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 msgr2=0x7f3ac8101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 0x7f3ac8101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 -- 172.21.15.16:0/4085818645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac8102d30 msgr2=0x7f3ac8100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac8102d30 0x7f3ac8100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 -- 172.21.15.16:0/4085818645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac81a69f0 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3acd781640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c8c0 tx=0x7f3ab800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8007bb0 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac81a6ce0 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab8007d50 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8018830 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.358+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac81a7210 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.360+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac806a850 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.361+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3ab8043ae0 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.361+0000 7f3abeffd640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 0x7f3ab40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.362+0000 7f3acdf82640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 0x7f3ab40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.362+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f3ab80cf240 con 0x7f3ac81036e0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.362+0000 7f3acdf82640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 0x7f3ab40a5040 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009900 tx=0x7f3ac40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.368+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ab8014030 con 0x7f3ac81036e0 2024-09-12T03:14:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.582+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ac8103110 con 0x7f3ac81036e0 2024-09-12T03:14:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.583+0000 7f3abeffd640 1 -- 172.21.15.16:0/4085818645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11978 (secure 0 0 0) 0x7f3ab809c150 con 0x7f3ac81036e0 2024-09-12T03:14:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:14:21.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"btime":"2024-09-12T03:13:28:924710+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1858508080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1858508080},{"type":"v1","addr":"172.21.15.16:6835","nonce":1858508080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3544362280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3544362280},{"type":"v1","addr":"172.21.15.16:6837","nonce":3544362280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-12T02:45:47.690653+0000","modified":"2024-09-12T03:13:28.852245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24467,"mds_2":24445,"mds_3":24443,"mds_4":14595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14583":{"gid":14583,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/2216233994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":2216233994},{"type":"v1","addr":"172.21.15.16:6839","nonce":2216233994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14595":{"gid":14595,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.16:6841/1593859053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":1593859053},{"type":"v1","addr":"172.21.15.16:6841","nonce":1593859053}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/897342399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":897342399},{"type":"v1","addr":"172.21.15.184:6833","nonce":897342399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/1900353998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1900353998},{"type":"v1","addr":"172.21.15.33:6833","nonce":1900353998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6835/1041505607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1041505607},{"type":"v1","addr":"172.21.15.184:6835","nonce":1041505607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6837/2803727914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":2803727914},{"type":"v1","addr":"172.21.15.184:6837","nonce":2803727914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.33:6835/2990713107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":2990713107},{"type":"v1","addr":"172.21.15.33:6835","nonce":2990713107}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.184:6839/598714982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":598714982},{"type":"v1","addr":"172.21.15.184:6839","nonce":598714982}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.33:6837/4242490884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":4242490884},{"type":"v1","addr":"172.21.15.33:6837","nonce":4242490884}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.33:6839/3603617308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":3603617308},{"type":"v1","addr":"172.21.15.33:6839","nonce":3603617308}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24467,24445,24443,14595]},"id":1}]} 2024-09-12T03:14:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 msgr2=0x7f3ab40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 0x7f3ab40a5040 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009900 tx=0x7f3ac40058d0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 msgr2=0x7f3ac8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c8c0 tx=0x7f3ab800cd80 comp rx=0 tx=0).stop 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 shutdown_connections 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac81082a0 0x7f3ac8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f3ab40a2b80 0x7f3ab40a5040 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac81036e0 0x7f3ac8100ed0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 --2- 172.21.15.16:0/4085818645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ac8102d30 0x7f3ac8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:21.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 >> 172.21.15.16:0/4085818645 conn(0x7f3ac80fc7d0 msgr2=0x7f3ac8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:21.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 shutdown_connections 2024-09-12T03:14:21.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:21.586+0000 7f3ad01e6640 1 -- 172.21.15.16:0/4085818645 wait complete. 2024-09-12T03:14:22.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:22 smithi016 ceph-mon[30925]: pgmap v1130: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 13 op/s 2024-09-12T03:14:22.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:22 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/4085818645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:22.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-12T03:14:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:22 smithi184 ceph-mon[35321]: pgmap v1130: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 13 op/s 2024-09-12T03:14:22.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:22 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/4085818645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:22 smithi033 ceph-mon[35092]: pgmap v1130: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 13 op/s 2024-09-12T03:14:22.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:22 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/4085818645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-12T03:14:22.498 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:23.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 -- 172.21.15.16:0/3020638122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034102570 msgr2=0x7f70341029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 --2- 172.21.15.16:0/3020638122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034102570 0x7f70341029b0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f7028009900 tx=0x7f702802a480 comp rx=0 tx=0).stop 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 -- 172.21.15.16:0/3020638122 shutdown_connections 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 --2- 172.21.15.16:0/3020638122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 0x7f703410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 --2- 172.21.15.16:0/3020638122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034102570 0x7f70341029b0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 --2- 172.21.15.16:0/3020638122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034100e20 0x7f7034101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 -- 172.21.15.16:0/3020638122 >> 172.21.15.16:0/3020638122 conn(0x7f70340fc7d0 msgr2=0x7f70340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 -- 172.21.15.16:0/3020638122 shutdown_connections 2024-09-12T03:14:23.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.021+0000 7f70392df640 1 -- 172.21.15.16:0/3020638122 wait complete. 2024-09-12T03:14:23.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.022+0000 7f70392df640 1 Processor -- start 2024-09-12T03:14:23.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 -- start start 2024-09-12T03:14:23.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:23.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034102570 0x7f7034198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:23.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 0x7f703419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7034110f80 con 0x7f7034100e20 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7034110e00 con 0x7f7034102ef0 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70392df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7034111100 con 0x7f7034102570 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f7032ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f7032ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38904/0 (socket says 172.21.15.16:38904) 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f7032ffd640 1 -- 172.21.15.16:0/1830422434 learned_addr learned my addr 172.21.15.16:0/1830422434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70337fe640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 0x7f703419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.023+0000 7f70327fc640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034102570 0x7f7034198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 -- 172.21.15.16:0/1830422434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034102570 msgr2=0x7f7034198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034102570 0x7f7034198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 -- 172.21.15.16:0/1830422434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 msgr2=0x7f703419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 0x7f703419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 -- 172.21.15.16:0/1830422434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703419d540 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f7032ffd640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f702400db80 tx=0x7f702400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024017070 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703419d830 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7024003b70 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70340fcbb0 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.024+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70240106c0 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.025+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7020005290 con 0x7f7034100e20 2024-09-12T03:14:23.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.026+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7024004020 con 0x7f7034100e20 2024-09-12T03:14:23.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.027+0000 7f701bfff640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 0x7f701c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:23.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.028+0000 7f70327fc640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 0x7f701c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:23.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.028+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f70240cecf0 con 0x7f7034100e20 2024-09-12T03:14:23.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.028+0000 7f70327fc640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 0x7f701c0a5040 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7028009230 tx=0x7f70280058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:23.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.032+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f70240d3070 con 0x7f7034100e20 2024-09-12T03:14:23.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:23 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:14:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.214+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7020005580 con 0x7f7034100e20 2024-09-12T03:14:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.216+0000 7f701bfff640 1 -- 172.21.15.16:0/1830422434 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22672 (secure 0 0 0) 0x7f7024020070 con 0x7f7034100e20 2024-09-12T03:14:23.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T03:14:23.216 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"ef86fd2a-70af-11ef-bcea-c7b262605968","created":"2024-09-12T02:38:12.620009+0000","modified":"2024-09-12T02:46:01.241215+0000","last_up_change":"2024-09-12T02:44:58.672777+0000","last_in_change":"2024-09-12T02:44:34.763009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T02:41:16.432997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-12T02:45:43.869635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-12T02:45:46.156619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bfd4081-15d2-46ed-922c-c65451eff4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6803","nonce":245735353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6805","nonce":245735353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6809","nonce":245735353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":245735353},{"type":"v1","addr":"172.21.15.16:6807","nonce":245735353}]},"public_addr":"172.21.15.16:6803/245735353","cluster_addr":"172.21.15.16:6805/245735353","heartbeat_back_addr":"172.21.15.16:6809/245735353","heartbeat_front_addr":"172.21.15.16:6807/245735353","state":["exists","up"]},{"osd":1,"uuid":"4ea14c97-f3f6-4c9e-9e00-b2faf37c27a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6801","nonce":2781270412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6803","nonce":2781270412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6807","nonce":2781270412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2781270412},{"type":"v1","addr":"172.21.15.33:6805","nonce":2781270412}]},"public_addr":"172.21.15.33:6801/2781270412","cluster_addr":"172.21.15.33:6803/2781270412","heartbeat_back_addr":"172.21.15.33:6807/2781270412","heartbeat_front_addr":"172.21.15.33:6805/2781270412","state":["exists","up"]},{"osd":2,"uuid":"83c715c5-107d-47dc-9c18-bee07a4bc40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6801","nonce":867017914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6803","nonce":867017914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6807","nonce":867017914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":867017914},{"type":"v1","addr":"172.21.15.184:6805","nonce":867017914}]},"public_addr":"172.21.15.184:6801/867017914","cluster_addr":"172.21.15.184:6803/867017914","heartbeat_back_addr":"172.21.15.184:6807/867017914","heartbeat_front_addr":"172.21.15.184:6805/867017914","state":["exists","up"]},{"osd":3,"uuid":"9e4edbd6-3cd0-43eb-8a99-7dc3909d9e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6811","nonce":3579605426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6813","nonce":3579605426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6817","nonce":3579605426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3579605426},{"type":"v1","addr":"172.21.15.16:6815","nonce":3579605426}]},"public_addr":"172.21.15.16:6811/3579605426","cluster_addr":"172.21.15.16:6813/3579605426","heartbeat_back_addr":"172.21.15.16:6817/3579605426","heartbeat_front_addr":"172.21.15.16:6815/3579605426","state":["exists","up"]},{"osd":4,"uuid":"dd26143a-5c8f-47b6-8f10-7bc4480b6c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6809","nonce":3318678506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6811","nonce":3318678506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6815","nonce":3318678506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3318678506},{"type":"v1","addr":"172.21.15.33:6813","nonce":3318678506}]},"public_addr":"172.21.15.33:6809/3318678506","cluster_addr":"172.21.15.33:6811/3318678506","heartbeat_back_addr":"172.21.15.33:6815/3318678506","heartbeat_front_addr":"172.21.15.33:6813/3318678506","state":["exists","up"]},{"osd":5,"uuid":"269f5e13-d530-4741-8b1f-7e6313cd1ccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6809","nonce":746947299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6811","nonce":746947299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6815","nonce":746947299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":746947299},{"type":"v1","addr":"172.21.15.184:6813","nonce":746947299}]},"public_addr":"172.21.15.184:6809/746947299","cluster_addr":"172.21.15.184:6811/746947299","heartbeat_back_addr":"172.21.15.184:6815/746947299","heartbeat_front_addr":"172.21.15.184:6813/746947299","state":["exists","up"]},{"osd":6,"uuid":"0744cd3c-8a0c-42f5-9e43-bfe91d615dc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6819","nonce":2942447777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6821","nonce":2942447777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6825","nonce":2942447777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2942447777},{"type":"v1","addr":"172.21.15.16:6823","nonce":2942447777}]},"public_addr":"172.21.15.16:6819/2942447777","cluster_addr":"172.21.15.16:6821/2942447777","heartbeat_back_addr":"172.21.15.16:6825/2942447777","heartbeat_front_addr":"172.21.15.16:6823/2942447777","state":["exists","up"]},{"osd":7,"uuid":"165df37e-3ffb-4d76-91b3-a15d9eaaa233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6817","nonce":1053213399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6819","nonce":1053213399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6823","nonce":1053213399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1053213399},{"type":"v1","addr":"172.21.15.33:6821","nonce":1053213399}]},"public_addr":"172.21.15.33:6817/1053213399","cluster_addr":"172.21.15.33:6819/1053213399","heartbeat_back_addr":"172.21.15.33:6823/1053213399","heartbeat_front_addr":"172.21.15.33:6821/1053213399","state":["exists","up"]},{"osd":8,"uuid":"f1980104-b353-42e4-9168-5c1c326da47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6817","nonce":1742401135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6819","nonce":1742401135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6823","nonce":1742401135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1742401135},{"type":"v1","addr":"172.21.15.184:6821","nonce":1742401135}]},"public_addr":"172.21.15.184:6817/1742401135","cluster_addr":"172.21.15.184:6819/1742401135","heartbeat_back_addr":"172.21.15.184:6823/1742401135","heartbeat_front_addr":"172.21.15.184:6821/1742401135","state":["exists","up"]},{"osd":9,"uuid":"1da86df9-e623-41f7-8089-73aef567c50a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6827","nonce":3409413039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6829","nonce":3409413039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6833","nonce":3409413039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3409413039},{"type":"v1","addr":"172.21.15.16:6831","nonce":3409413039}]},"public_addr":"172.21.15.16:6827/3409413039","cluster_addr":"172.21.15.16:6829/3409413039","heartbeat_back_addr":"172.21.15.16:6833/3409413039","heartbeat_front_addr":"172.21.15.16:6831/3409413039","state":["exists","up"]},{"osd":10,"uuid":"5d052784-3463-4b31-8894-92d1ef1d1ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6825","nonce":3476378344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6827","nonce":3476378344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6831","nonce":3476378344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3476378344},{"type":"v1","addr":"172.21.15.33:6829","nonce":3476378344}]},"public_addr":"172.21.15.33:6825/3476378344","cluster_addr":"172.21.15.33:6827/3476378344","heartbeat_back_addr":"172.21.15.33:6831/3476378344","heartbeat_front_addr":"172.21.15.33:6829/3476378344","state":["exists","up"]},{"osd":11,"uuid":"ed5b71a4-67ba-479b-83e0-80e5ce8cfec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6825","nonce":3211758447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6827","nonce":3211758447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6831","nonce":3211758447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3211758447},{"type":"v1","addr":"172.21.15.184:6829","nonce":3211758447}]},"public_addr":"172.21.15.184:6825/3211758447","cluster_addr":"172.21.15.184:6827/3211758447","heartbeat_back_addr":"172.21.15.184:6831/3211758447","heartbeat_front_addr":"172.21.15.184:6829/3211758447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:23.868316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:40:49.242167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:13.789321+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:41:39.157147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:03.515195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:27.619780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:42:52.591195+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:17.225424+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:43:41.488099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:07.055354+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:31.967982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T02:44:56.444729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3425705974":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6801/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:6800/744312028":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/540700595":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2390815323":"2024-09-13T02:38:59.915197+0000","172.21.15.16:0/2593008088":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6801/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:6800/2594473615":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/1418983":"2024-09-13T02:38:34.945193+0000","172.21.15.16:0/3307112498":"2024-09-13T02:38:34.945193+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T03:14:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 msgr2=0x7f701c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 0x7f701c0a5040 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7028009230 tx=0x7f70280058d0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 msgr2=0x7f7034198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f702400db80 tx=0x7f702400b630 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 shutdown_connections 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7034102ef0 0x7f703419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7034102570 0x7f7034198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f701c0a2b80 0x7f701c0a5040 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 --2- 172.21.15.16:0/1830422434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7034100e20 0x7f7034198590 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 >> 172.21.15.16:0/1830422434 conn(0x7f70340fc7d0 msgr2=0x7f703406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.218+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 shutdown_connections 2024-09-12T03:14:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:23.219+0000 7f70392df640 1 -- 172.21.15.16:0/1830422434 wait complete. 2024-09-12T03:14:23.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:23 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:14:23.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:23 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-12T03:14:24.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:24 smithi016 ceph-mon[30925]: pgmap v1131: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:14:24.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:24 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/1830422434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:14:24.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs fail cephfs 2024-09-12T03:14:24.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:24 smithi184 ceph-mon[35321]: pgmap v1131: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:14:24.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:24 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/1830422434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:14:24.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:24 smithi033 ceph-mon[35092]: pgmap v1131: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-12T03:14:24.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:24 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/1830422434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-12T03:14:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:25.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 -- 172.21.15.16:0/3392676003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 msgr2=0x7f74801021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:25.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 --2- 172.21.15.16:0/3392676003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74801021b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f7470009900 tx=0x7f747002a410 comp rx=0 tx=0).stop 2024-09-12T03:14:25.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 -- 172.21.15.16:0/3392676003 shutdown_connections 2024-09-12T03:14:25.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 --2- 172.21.15.16:0/3392676003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480106f00 0x7f748010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 --2- 172.21.15.16:0/3392676003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480102780 0x7f74801067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 --2- 172.21.15.16:0/3392676003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74801021b0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 -- 172.21.15.16:0/3392676003 >> 172.21.15.16:0/3392676003 conn(0x7f74800fd8b0 msgr2=0x7f74800ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 -- 172.21.15.16:0/3392676003 shutdown_connections 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.925+0000 7f74868bc640 1 -- 172.21.15.16:0/3392676003 wait complete. 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.926+0000 7f74868bc640 1 Processor -- start 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.926+0000 7f74868bc640 1 -- start start 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.926+0000 7f74868bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.926+0000 7f74868bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480102780 0x7f7480074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.926+0000 7f74868bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480106f00 0x7f7480074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f74868bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7480111f20 con 0x7f7480101dd0 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f74868bc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7480111da0 con 0x7f7480102780 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f74868bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74801120a0 con 0x7f7480106f00 2024-09-12T03:14:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56248/0 (socket says 172.21.15.16:56248) 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 -- 172.21.15.16:0/655691063 learned_addr learned my addr 172.21.15.16:0/655691063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747f7fe640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480102780 0x7f7480074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f7484e59640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480106f00 0x7f7480074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 -- 172.21.15.16:0/655691063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480106f00 msgr2=0x7f7480074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480106f00 0x7f7480074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 -- 172.21.15.16:0/655691063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480102780 msgr2=0x7f7480074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480102780 0x7f7480074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 -- 172.21.15.16:0/655691063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74801a6940 con 0x7f7480101dd0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747ffff640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f7470006010 tx=0x7f74700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747003f030 con 0x7f7480101dd0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74801a6bd0 con 0x7f7480101dd0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7470003ce0 con 0x7f7480101dd0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.927+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470035070 con 0x7f7480101dd0 2024-09-12T03:14:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.928+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74801a7100 con 0x7f7480101dd0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.928+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74801021b0 con 0x7f7480101dd0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.930+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7470042030 con 0x7f7480101dd0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.931+0000 7f747d7fa640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 0x7f74680a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.931+0000 7f747f7fe640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 0x7f74680a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.931+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8211+0+0 (secure 0 0 0) 0x7f74700ef880 con 0x7f7480101dd0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.931+0000 7f747f7fe640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 0x7f74680a5040 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7474005fd0 tx=0x7f7474005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:25.934+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f74700bc8b0 con 0x7f7480101dd0 2024-09-12T03:14:26.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.159+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f748010a0e0 con 0x7f7480101dd0 2024-09-12T03:14:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.161+0000 7f747d7fa640 1 -- 172.21.15.16:0/655691063 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7f74700bc1b0 con 0x7f7480101dd0 2024-09-12T03:14:26.161 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.164+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 msgr2=0x7f74680a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.164+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 0x7f74680a5040 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7474005fd0 tx=0x7f7474005850 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.164+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 msgr2=0x7f74800744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.164+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f7470006010 tx=0x7f74700042f0 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 shutdown_connections 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480106f00 0x7f7480074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7480102780 0x7f7480074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f74680a2b80 0x7f74680a5040 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 --2- 172.21.15.16:0/655691063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7480101dd0 0x7f74800744f0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.165+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 >> 172.21.15.16:0/655691063 conn(0x7f74800fd8b0 msgr2=0x7f74800fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:26.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.166+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 shutdown_connections 2024-09-12T03:14:26.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:26.166+0000 7f74868bc640 1 -- 172.21.15.16:0/655691063 wait complete. 2024-09-12T03:14:26.283 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-mon[35321]: pgmap v1132: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-12T03:14:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-mon[30925]: pgmap v1132: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-12T03:14:26.407 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a[30921]: 2024-09-12T03:14:26.159+0000 7f2b301f2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: starting mds.a at 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-mds[71459]: -- 172.21.15.16:0/3439498841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eefbb61860 con 0x55eefbb3e800 2024-09-12T03:14:26.408 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:14:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: starting mds.g at 2024-09-12T03:14:26.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-mon[35092]: pgmap v1132: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-12T03:14:26.497 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.497 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.497 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: starting mds.b at 2024-09-12T03:14:26.497 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-mds[64550]: -- 172.21.15.33:0/4201901266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a95bb7bc20 con 0x55a95bb58800 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: starting mds.e at 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: 2024-09-12T03:14:26.249+0000 7f5920b7d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-mds[64851]: -- 172.21.15.33:0/1457310326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8919a3860 con 0x55f891981400 2024-09-12T03:14:26.498 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: starting mds.h at 2024-09-12T03:14:26.499 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.499 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.499 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-mds[64237]: -- 172.21.15.33:0/4255152192 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55badd351860 con 0x55badd32f400 2024-09-12T03:14:26.499 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:14:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: starting mds.k at 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: starting mds.i at 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-mds[65550]: -- 172.21.15.184:0/2509616654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56085d965860 con 0x56085d942800 2024-09-12T03:14:26.545 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: starting mds.l at 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-mds[65861]: -- 172.21.15.184:0/2908740355 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564387063860 con 0x564387040800 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: starting mds.f at 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c[66157]: ignoring --setuser ceph since I am not root 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c[66157]: ignoring --setgroup ceph since I am not root 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-mds[66161]: -- 172.21.15.184:0/3306578635 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559249367860 con 0x559249345000 2024-09-12T03:14:26.546 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:14:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c[66157]: starting mds.c at 2024-09-12T03:14:26.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-12T03:14:27.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-12T03:14:27.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-12T03:14:27.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-12T03:14:27.275 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-12T03:14:27.275 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: osdmap e81: 12 total, 12 up, 12 in 2024-09-12T03:14:27.275 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:27 smithi016 ceph-mon[30925]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-12T03:14:27.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-12T03:14:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-12T03:14:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-12T03:14:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-12T03:14:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: osdmap e81: 12 total, 12 up, 12 in 2024-09-12T03:14:27.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:27 smithi184 ceph-mon[35321]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-12T03:14:27.385 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:27.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-12T03:14:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-12T03:14:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-12T03:14:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/655691063' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-12T03:14:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: osdmap e81: 12 total, 12 up, 12 in 2024-09-12T03:14:27.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:27 smithi033 ceph-mon[35092]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-12T03:14:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- 172.21.15.16:0/207063470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620106f10 msgr2=0x7fa62010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 --2- 172.21.15.16:0/207063470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620106f10 0x7fa62010ea90 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa618009ca0 tx=0x7fa61802a8c0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- 172.21.15.16:0/207063470 shutdown_connections 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 --2- 172.21.15.16:0/207063470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620106f10 0x7fa62010ea90 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 --2- 172.21.15.16:0/207063470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620102790 0x7fa6201067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 --2- 172.21.15.16:0/207063470 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620101de0 0x7fa6201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- 172.21.15.16:0/207063470 >> 172.21.15.16:0/207063470 conn(0x7fa6200fd8c0 msgr2=0x7fa6200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- 172.21.15.16:0/207063470 shutdown_connections 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- 172.21.15.16:0/207063470 wait complete. 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 Processor -- start 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.870+0000 7fa626ab1640 1 -- start start 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620102790 0x7fa620074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620106f10 0x7fa620074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa620111f70 con 0x7fa620101de0 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa620111df0 con 0x7fa620102790 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa626ab1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201120f0 con 0x7fa620106f10 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56306/0 (socket says 172.21.15.16:56306) 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 -- 172.21.15.16:0/420739439 learned_addr learned my addr 172.21.15.16:0/420739439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62504e640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620106f10 0x7fa620074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:27.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 -- 172.21.15.16:0/420739439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620106f10 msgr2=0x7fa620074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:27.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620106f10 0x7fa620074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa61ffff640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620102790 0x7fa620074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:27.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 -- 172.21.15.16:0/420739439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620102790 msgr2=0x7fa620074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620102790 0x7fa620074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 -- 172.21.15.16:0/420739439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6201a67f0 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa62484d640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c920 tx=0x7fa61000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.871+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610007c70 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.872+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa610007e10 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.872+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6201a6ae0 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.872+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610018890 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.872+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201a7020 con 0x7fa620101de0 2024-09-12T03:14:27.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.873+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6201021c0 con 0x7fa620101de0 2024-09-12T03:14:27.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.875+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa610018a70 con 0x7fa620101de0 2024-09-12T03:14:27.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.875+0000 7fa61dffb640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 0x7fa6080a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:27.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.875+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8641+0+0 (secure 0 0 0) 0x7fa6100cf440 con 0x7fa620101de0 2024-09-12T03:14:27.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.875+0000 7fa61ffff640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 0x7fa6080a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:27.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.876+0000 7fa61ffff640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 0x7fa6080a5040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa614005d80 tx=0x7fa614005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:27.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:27.879+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa610014030 con 0x7fa620101de0 2024-09-12T03:14:28.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.092+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa62010a0f0 con 0x7fa620101de0 2024-09-12T03:14:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.316+0000 7fa61dffb640 1 -- 172.21.15.16:0/420739439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) ==== 104+0+0 (secure 0 0 0) 0x7fa61000de20 con 0x7fa620101de0 2024-09-12T03:14:28.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 msgr2=0x7fa6080a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:28.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 0x7fa6080a5040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa614005d80 tx=0x7fa614005c90 comp rx=0 tx=0).stop 2024-09-12T03:14:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 msgr2=0x7fa6200744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c920 tx=0x7fa61000cde0 comp rx=0 tx=0).stop 2024-09-12T03:14:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 shutdown_connections 2024-09-12T03:14:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa620106f10 0x7fa620074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa620102790 0x7fa620074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7fa6080a2b80 0x7fa6080a5040 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 --2- 172.21.15.16:0/420739439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101de0 0x7fa6200744f0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 >> 172.21.15.16:0/420739439 conn(0x7fa6200fd8c0 msgr2=0x7fa6200febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 shutdown_connections 2024-09-12T03:14:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:28.319+0000 7fa626ab1640 1 -- 172.21.15.16:0/420739439 wait complete. 2024-09-12T03:14:28.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: pgmap v1134: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:28.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6838/3980496677,v1:172.21.15.16:6839/3980496677] up:boot 2024-09-12T03:14:28.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.16:6840/375809008,v1:172.21.15.16:6841/375809008] up:boot 2024-09-12T03:14:28.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6833/1647581036,v1:172.21.15.33:6835/1647581036] up:boot 2024-09-12T03:14:28.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6832/1580171372,v1:172.21.15.33:6834/1580171372] up:boot 2024-09-12T03:14:28.498 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6836/1151248232,v1:172.21.15.33:6839/1151248232] up:boot 2024-09-12T03:14:28.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.33:6837/222737787,v1:172.21.15.33:6838/222737787] up:boot 2024-09-12T03:14:28.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6832/257768229,v1:172.21.15.184:6834/257768229] up:boot 2024-09-12T03:14:28.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6833/1684055872,v1:172.21.15.184:6835/1684055872] up:boot 2024-09-12T03:14:28.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6836/1707516454,v1:172.21.15.184:6838/1707516454] up:boot 2024-09-12T03:14:28.499 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: mds.? [v2:172.21.15.184:6837/1835356268,v1:172.21.15.184:6839/1835356268] up:boot 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T03:14:28.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:14:28.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:14:28.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:28 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T03:14:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: pgmap v1134: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6838/3980496677,v1:172.21.15.16:6839/3980496677] up:boot 2024-09-12T03:14:28.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.16:6840/375809008,v1:172.21.15.16:6841/375809008] up:boot 2024-09-12T03:14:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6833/1647581036,v1:172.21.15.33:6835/1647581036] up:boot 2024-09-12T03:14:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6832/1580171372,v1:172.21.15.33:6834/1580171372] up:boot 2024-09-12T03:14:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6836/1151248232,v1:172.21.15.33:6839/1151248232] up:boot 2024-09-12T03:14:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.33:6837/222737787,v1:172.21.15.33:6838/222737787] up:boot 2024-09-12T03:14:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6832/257768229,v1:172.21.15.184:6834/257768229] up:boot 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6833/1684055872,v1:172.21.15.184:6835/1684055872] up:boot 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6836/1707516454,v1:172.21.15.184:6838/1707516454] up:boot 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: mds.? [v2:172.21.15.184:6837/1835356268,v1:172.21.15.184:6839/1835356268] up:boot 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T03:14:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T03:14:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T03:14:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T03:14:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T03:14:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:14:28.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:14:28.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:28 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T03:14:28.562 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: pgmap v1134: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:28.562 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6838/3980496677,v1:172.21.15.16:6839/3980496677] up:boot 2024-09-12T03:14:28.562 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.16:6840/375809008,v1:172.21.15.16:6841/375809008] up:boot 2024-09-12T03:14:28.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6833/1647581036,v1:172.21.15.33:6835/1647581036] up:boot 2024-09-12T03:14:28.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6832/1580171372,v1:172.21.15.33:6834/1580171372] up:boot 2024-09-12T03:14:28.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6836/1151248232,v1:172.21.15.33:6839/1151248232] up:boot 2024-09-12T03:14:28.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.33:6837/222737787,v1:172.21.15.33:6838/222737787] up:boot 2024-09-12T03:14:28.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6832/257768229,v1:172.21.15.184:6834/257768229] up:boot 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6833/1684055872,v1:172.21.15.184:6835/1684055872] up:boot 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6836/1707516454,v1:172.21.15.184:6838/1707516454] up:boot 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: mds.? [v2:172.21.15.184:6837/1835356268,v1:172.21.15.184:6839/1835356268] up:boot 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-12T03:14:28.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-12T03:14:28.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-12T03:14:28.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-12T03:14:28.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-12T03:14:28.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-12T03:14:28.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-12T03:14:28.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-12T03:14:28.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='mgr.14150 172.21.15.16:0/105192241' entity='mgr.x' 2024-09-12T03:14:28.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:28 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-12T03:14:28.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-12T03:14:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:29.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:29 smithi016 ceph-mon[30925]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-12T03:14:29.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:29 smithi016 ceph-mon[30925]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-12T03:14:29.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:29 smithi016 ceph-mon[30925]: Cluster is now healthy 2024-09-12T03:14:29.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:29 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-12T03:14:29.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:29 smithi016 ceph-mon[30925]: fsmap 12 up:standby 2024-09-12T03:14:29.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:29 smithi033 ceph-mon[35092]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-12T03:14:29.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:29 smithi033 ceph-mon[35092]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-12T03:14:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:29 smithi033 ceph-mon[35092]: Cluster is now healthy 2024-09-12T03:14:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:29 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-12T03:14:29.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:29 smithi033 ceph-mon[35092]: fsmap 12 up:standby 2024-09-12T03:14:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:29 smithi184 ceph-mon[35321]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-12T03:14:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:29 smithi184 ceph-mon[35321]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-12T03:14:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:29 smithi184 ceph-mon[35321]: Cluster is now healthy 2024-09-12T03:14:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:29 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/420739439' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-12T03:14:29.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:29 smithi184 ceph-mon[35321]: fsmap 12 up:standby 2024-09-12T03:14:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 -- 172.21.15.16:0/429347482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 msgr2=0x7f2870106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 --2- 172.21.15.16:0/429347482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f2870106e60 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2860009930 tx=0x7f286002a420 comp rx=0 tx=0).stop 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 -- 172.21.15.16:0/429347482 shutdown_connections 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 --2- 172.21.15.16:0/429347482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 0x7f287010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 --2- 172.21.15.16:0/429347482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 0x7f2870100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 --2- 172.21.15.16:0/429347482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f2870106e60 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 -- 172.21.15.16:0/429347482 >> 172.21.15.16:0/429347482 conn(0x7f28700fc7a0 msgr2=0x7f28700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 -- 172.21.15.16:0/429347482 shutdown_connections 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.680+0000 7f2874eba640 1 -- 172.21.15.16:0/429347482 wait complete. 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 Processor -- start 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 -- start start 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 0x7f2870102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 0x7f2870102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2870110e80 con 0x7f2870106a80 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2870110d00 con 0x7f2870101420 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2874eba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2870111000 con 0x7f2870100a80 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f286ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f286ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56316/0 (socket says 172.21.15.16:56316) 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f286ed9d640 1 -- 172.21.15.16:0/2038041372 learned_addr learned my addr 172.21.15.16:0/2038041372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f286e59c640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 0x7f2870102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.681+0000 7f2865d9b640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 0x7f2870102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 -- 172.21.15.16:0/2038041372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 msgr2=0x7f2870102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 0x7f2870102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 -- 172.21.15.16:0/2038041372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 msgr2=0x7f2870102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 0x7f2870102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 -- 172.21.15.16:0/2038041372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28701a6950 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f286ed9d640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f286800cad0 tx=0x7f286800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286800ae20 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2868005020 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28701a6c40 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286800f6f0 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.682+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28701a7180 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.683+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2870100ee0 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.684+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f28680051d0 con 0x7f2870106a80 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.685+0000 7f2867fff640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 0x7f285c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.685+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8641+0+0 (secure 0 0 0) 0x7f28680cf0d0 con 0x7f2870106a80 2024-09-12T03:14:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.685+0000 7f286e59c640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 0x7f285c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.686+0000 7f286e59c640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 0x7f285c0a50c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2860009900 tx=0x7f28600058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.689+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2868014030 con 0x7f2870106a80 2024-09-12T03:14:29.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:29.863+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f28701a7410 con 0x7f2870106a80 2024-09-12T03:14:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.320+0000 7f2867fff640 1 -- 172.21.15.16:0/2038041372 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) ==== 181+0+0 (secure 0 0 0) 0x7f286809b840 con 0x7f2870106a80 2024-09-12T03:14:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 msgr2=0x7f285c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 0x7f285c0a50c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2860009900 tx=0x7f28600058d0 comp rx=0 tx=0).stop 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 msgr2=0x7f28701030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f286800cad0 tx=0x7f286800cf90 comp rx=0 tx=0).stop 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 shutdown_connections 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7f285c0a2c00 0x7f285c0a50c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870106a80 0x7f28701030d0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2870101420 0x7f2870102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 --2- 172.21.15.16:0/2038041372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2870100a80 0x7f2870102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.323+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 >> 172.21.15.16:0/2038041372 conn(0x7f28700fc7a0 msgr2=0x7f28701048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.324+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 shutdown_connections 2024-09-12T03:14:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:30.324+0000 7f2874eba640 1 -- 172.21.15.16:0/2038041372 wait complete. 2024-09-12T03:14:30.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:30 smithi184 ceph-mon[35321]: pgmap v1135: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:30.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:30 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:30 smithi033 ceph-mon[35092]: pgmap v1135: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:30.496 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:30 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:30.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:30 smithi016 ceph-mon[30925]: pgmap v1135: 97 pgs: 97 active+clean; 728 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-12T03:14:30.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:30 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:31.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc shell --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-12T03:14:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/mon.a/config 2024-09-12T03:14:31.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:31 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:31.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:31 smithi016 ceph-mon[30925]: osdmap e82: 12 total, 12 up, 12 in 2024-09-12T03:14:31.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:31 smithi016 ceph-mon[30925]: pgmap v1137: 65 pgs: 65 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:31.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:31 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:31.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:31 smithi184 ceph-mon[35321]: osdmap e82: 12 total, 12 up, 12 in 2024-09-12T03:14:31.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:31 smithi184 ceph-mon[35321]: pgmap v1137: 65 pgs: 65 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:31.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:31 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2038041372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:31.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:31 smithi033 ceph-mon[35092]: osdmap e82: 12 total, 12 up, 12 in 2024-09-12T03:14:31.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:31 smithi033 ceph-mon[35092]: pgmap v1137: 65 pgs: 65 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:31.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 -- 172.21.15.16:0/1398257146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894106f10 msgr2=0x7ff89410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:31.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 --2- 172.21.15.16:0/1398257146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894106f10 0x7ff89410ea90 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009ca0 tx=0x7ff88c02a8c0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 -- 172.21.15.16:0/1398257146 shutdown_connections 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 --2- 172.21.15.16:0/1398257146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894106f10 0x7ff89410ea90 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 --2- 172.21.15.16:0/1398257146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894102790 0x7ff8941067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 --2- 172.21.15.16:0/1398257146 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894101de0 0x7ff8941021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 -- 172.21.15.16:0/1398257146 >> 172.21.15.16:0/1398257146 conn(0x7ff8940fd8c0 msgr2=0x7ff8940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 -- 172.21.15.16:0/1398257146 shutdown_connections 2024-09-12T03:14:31.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.819+0000 7ff89c07a640 1 -- 172.21.15.16:0/1398257146 wait complete. 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 Processor -- start 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 -- start start 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 0x7ff8940744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 0x7ff894074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff894111fc0 con 0x7ff894102790 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff894111e40 con 0x7ff894101de0 2024-09-12T03:14:31.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89c07a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff894112140 con 0x7ff894106f10 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff899615640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff89a617640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 0x7ff894074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff899615640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56342/0 (socket says 172.21.15.16:56342) 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.820+0000 7ff899615640 1 -- 172.21.15.16:0/2760036008 learned_addr learned my addr 172.21.15.16:0/2760036008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff899e16640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 0x7ff8940744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 -- 172.21.15.16:0/2760036008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 msgr2=0x7ff8940744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 0x7ff8940744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 -- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 msgr2=0x7ff894074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 -- 172.21.15.16:0/2760036008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8941a68d0 con 0x7ff894106f10 2024-09-12T03:14:31.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff899615640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff899e16640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 0x7ff8940744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89a617640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 0x7ff894074f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff88c005e30 tx=0x7ff88c003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c031650 con 0x7ff894106f10 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8941a6b60 con 0x7ff894106f10 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff88c0317f0 con 0x7ff894106f10 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c02fb70 con 0x7ff894106f10 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.821+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8941a70f0 con 0x7ff894106f10 2024-09-12T03:14:31.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.822+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8941021c0 con 0x7ff894106f10 2024-09-12T03:14:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.824+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff88c02fd10 con 0x7ff894106f10 2024-09-12T03:14:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.824+0000 7ff88affd640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 0x7ff8800a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-12T03:14:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.824+0000 7ff899e16640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 0x7ff8800a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-12T03:14:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.824+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8087+0+0 (secure 0 0 0) 0x7ff88c0f0550 con 0x7ff894106f10 2024-09-12T03:14:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.825+0000 7ff899e16640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 0x7ff8800a5040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff890005fd0 tx=0x7ff89000b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-12T03:14:31.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:31.828+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff88c0bd4f0 con 0x7ff894106f10 2024-09-12T03:14:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.008+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff89410a150 con 0x7ff894106f10 2024-09-12T03:14:32.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.353+0000 7ff88affd640 1 -- 172.21.15.16:0/2760036008 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) ==== 169+0+0 (secure 0 0 0) 0x7ff88c0bcdf0 con 0x7ff894106f10 2024-09-12T03:14:32.354 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-09-12T03:14:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 msgr2=0x7ff8800a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 0x7ff8800a5040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff890005fd0 tx=0x7ff89000b5f0 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 msgr2=0x7ff894074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 0x7ff894074f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff88c005e30 tx=0x7ff88c003f10 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 shutdown_connections 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff894106f10 0x7ff894074f70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:6800/2564464416,v1:172.21.15.16:6801/2564464416] conn(0x7ff8800a2b80 0x7ff8800a5040 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff894102790 0x7ff894074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 --2- 172.21.15.16:0/2760036008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff894101de0 0x7ff8940744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.356+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 >> 172.21.15.16:0/2760036008 conn(0x7ff8940fd8c0 msgr2=0x7ff8940febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.357+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 shutdown_connections 2024-09-12T03:14:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T03:14:32.357+0000 7ff89c07a640 1 -- 172.21.15.16:0/2760036008 wait complete. 2024-09-12T03:14:32.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:32 smithi016 ceph-mon[30925]: from='client.? 172.21.15.16:0/2760036008' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:32.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:32 smithi016 ceph-mon[30925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:32.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:32 smithi033 ceph-mon[35092]: from='client.? 172.21.15.16:0/2760036008' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:32.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:32 smithi033 ceph-mon[35092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:32.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:32 smithi184 ceph-mon[35321]: from='client.? 172.21.15.16:0/2760036008' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:32.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:32 smithi184 ceph-mon[35321]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-12T03:14:33.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:14:33.056 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:14:33.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:14:33.141 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-12T03:14:33.141 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-12T03:14:33.157 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-12T03:14:33.174 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-12T03:14:33.196 INFO:tasks.cephadm:Stopping all daemons... 2024-09-12T03:14:33.196 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-12T03:14:33.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a 2024-09-12T03:14:33.348 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 systemd[1]: Stopping Ceph mon.a for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:33 smithi184 ceph-mon[35321]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:33 smithi184 ceph-mon[35321]: osdmap e83: 12 total, 12 up, 12 in 2024-09-12T03:14:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:33 smithi184 ceph-mon[35321]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 ceph-mon[30925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 ceph-mon[30925]: osdmap e83: 12 total, 12 up, 12 in 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 ceph-mon[30925]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a[30921]: 2024-09-12T03:14:33.517+0000 7f2b359fd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a[30921]: 2024-09-12T03:14:33.517+0000 7f2b359fd640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-12T03:14:33.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:33 smithi016 podman[147653]: 2024-09-12 03:14:33.589713257 +0000 UTC m=+0.201353470 container died 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3) 2024-09-12T03:14:33.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:33 smithi033 ceph-mon[35092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-12T03:14:33.746 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:33 smithi033 ceph-mon[35092]: osdmap e83: 12 total, 12 up, 12 in 2024-09-12T03:14:33.747 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:33 smithi033 ceph-mon[35092]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-12T03:14:34.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:34 smithi016 podman[147653]: 2024-09-12 03:14:34.0137273 +0000 UTC m=+0.625367508 container cleanup 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-12T03:14:34.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:34 smithi016 bash[147653]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a 2024-09-12T03:14:34.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 12 03:14:34 smithi016 podman[147664]: 2024-09-12 03:14:34.2365687 +0000 UTC m=+0.639066759 container remove 6c85c08c3f0e9e6ef32fc27605061172e7a3214d697cc2fcb4b1e062c36a74c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:14:34.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.a.service' 2024-09-12T03:14:34.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:34.442 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-12T03:14:34.443 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-12T03:14:34.443 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.b 2024-09-12T03:14:34.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:34 smithi033 systemd[1]: Stopping Ceph mon.b for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:35.930 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b[35088]: 2024-09-12T03:14:35.786+0000 7f1ce8442640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:35.930 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b[35088]: 2024-09-12T03:14:35.787+0000 7f1ce8442640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-12T03:14:36.246 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:35 smithi033 podman[73209]: 2024-09-12 03:14:35.93018349 +0000 UTC m=+0.287569102 container died 067da0701270c27a30f054da5347cac89d0702d3309ac28c9d79de72beb2524f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-12T03:14:36.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:36 smithi033 podman[73209]: 2024-09-12 03:14:36.37111966 +0000 UTC m=+0.728505273 container cleanup 067da0701270c27a30f054da5347cac89d0702d3309ac28c9d79de72beb2524f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:14:36.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:36 smithi033 bash[73209]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b 2024-09-12T03:14:36.958 INFO:journalctl@ceph.mon.b.smithi033.stdout:Sep 12 03:14:36 smithi033 podman[73224]: 2024-09-12 03:14:36.597752481 +0000 UTC m=+0.662712423 container remove 067da0701270c27a30f054da5347cac89d0702d3309ac28c9d79de72beb2524f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-12T03:14:37.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.b.service' 2024-09-12T03:14:37.493 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:37.493 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-12T03:14:37.493 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-12T03:14:37.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.c 2024-09-12T03:14:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:37 smithi184 systemd[1]: Stopping Ceph mon.c for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:38.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c[35317]: 2024-09-12T03:14:37.862+0000 7fbad5347640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:38.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c[35317]: 2024-09-12T03:14:37.862+0000 7fbad5347640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-12T03:14:38.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:38 smithi184 podman[75066]: 2024-09-12 03:14:38.097724055 +0000 UTC m=+0.359968690 container stop d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-12T03:14:38.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:38 smithi184 podman[75066]: 2024-09-12 03:14:38.123505048 +0000 UTC m=+0.385749686 container died d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-12T03:14:38.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:38 smithi184 podman[75066]: 2024-09-12 03:14:38.487160673 +0000 UTC m=+0.749405309 container cleanup d0a3a7b751273124e06809f24dfceec23c185a16ed27899bc5a1badcf3af98aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-12T03:14:38.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 12 03:14:38 smithi184 bash[75066]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mon-c 2024-09-12T03:14:38.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mon.c.service' 2024-09-12T03:14:38.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:38.946 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-12T03:14:38.946 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-12T03:14:38.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x 2024-09-12T03:14:39.309 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 03:14:38 smithi016 systemd[1]: Stopping Ceph mgr.x for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:39.598 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 03:14:39 smithi016 podman[147774]: 2024-09-12 03:14:39.312658121 +0000 UTC m=+0.163103843 container died 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-12T03:14:39.907 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 03:14:39 smithi016 podman[147774]: 2024-09-12 03:14:39.697957662 +0000 UTC m=+0.548403383 container cleanup 0b9f0e7e01b58b307313501029ecd6322c897587b3169348c7e4c5122702f704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-12T03:14:39.907 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 12 03:14:39 smithi016 bash[147774]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-x 2024-09-12T03:14:40.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-12T03:14:40.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:40.173 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-12T03:14:40.173 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-12T03:14:40.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.y 2024-09-12T03:14:40.496 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 03:14:40 smithi033 systemd[1]: Stopping Ceph mgr.y for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:40.813 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 03:14:40 smithi033 podman[75858]: 2024-09-12 03:14:40.516156429 +0000 UTC m=+0.147328419 container died 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731) 2024-09-12T03:14:41.164 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 03:14:40 smithi033 podman[75858]: 2024-09-12 03:14:40.888763738 +0000 UTC m=+0.519935792 container cleanup 5c319f05c8579c89203a361b3f85cef67fa191a06e5b4d55cd2598b57c3a0a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-12T03:14:41.165 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Sep 12 03:14:40 smithi033 bash[75858]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-y 2024-09-12T03:14:41.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-12T03:14:41.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:41.333 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-12T03:14:41.333 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-12T03:14:41.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.z 2024-09-12T03:14:41.678 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 03:14:41 smithi184 systemd[1]: Stopping Ceph mgr.z for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:41.965 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 03:14:41 smithi184 podman[75193]: 2024-09-12 03:14:41.679033456 +0000 UTC m=+0.149193026 container died de85bfae89f50a2ea470ec6f1d3b9b014785d7fbf0159d26fcca934b090255f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:14:42.225 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 03:14:42 smithi184 podman[75193]: 2024-09-12 03:14:42.033107449 +0000 UTC m=+0.503267019 container cleanup de85bfae89f50a2ea470ec6f1d3b9b014785d7fbf0159d26fcca934b090255f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-12T03:14:42.225 INFO:journalctl@ceph.mgr.z.smithi184.stdout:Sep 12 03:14:42 smithi184 bash[75193]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mgr-z 2024-09-12T03:14:42.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mgr.z.service' 2024-09-12T03:14:42.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:42.474 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-12T03:14:42.475 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-12T03:14:42.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.0 2024-09-12T03:14:42.903 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:42 smithi016 systemd[1]: Stopping Ceph osd.0 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:43.157 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T03:14:42.902+0000 7f16afec4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:43.157 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T03:14:42.902+0000 7f16afec4640 -1 osd.0 83 *** Got signal Terminated *** 2024-09-12T03:14:43.157 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0[45225]: 2024-09-12T03:14:42.902+0000 7f16afec4640 -1 osd.0 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:14:48.657 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:48 smithi016 podman[147896]: 2024-09-12 03:14:48.321137996 +0000 UTC m=+5.648297771 container died b10a65f8776927851ea055ec7ef6222520dee5582c6a6c43a447fefc79e94503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-12T03:14:49.145 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:48 smithi016 podman[147896]: 2024-09-12 03:14:48.861062367 +0000 UTC m=+6.188222141 container cleanup b10a65f8776927851ea055ec7ef6222520dee5582c6a6c43a447fefc79e94503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-b53b6d3, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:14:49.145 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:48 smithi016 bash[147896]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0 2024-09-12T03:14:49.407 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[147913]: 2024-09-12 03:14:49.145746442 +0000 UTC m=+0.827148704 container remove b10a65f8776927851ea055ec7ef6222520dee5582c6a6c43a447fefc79e94503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-12T03:14:49.749 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.392893801 +0000 UTC m=+0.016601513 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:14:49.749 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.521975952 +0000 UTC m=+0.145683646 container create fb3b3706ece3fa8989a72eaec127cf31610464bdfc8787ac51c6084cd1320d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-12T03:14:50.157 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.74940473 +0000 UTC m=+0.373112424 container init fb3b3706ece3fa8989a72eaec127cf31610464bdfc8787ac51c6084cd1320d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD) 2024-09-12T03:14:50.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.753276895 +0000 UTC m=+0.376984592 container start fb3b3706ece3fa8989a72eaec127cf31610464bdfc8787ac51c6084cd1320d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:14:50.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.869982907 +0000 UTC m=+0.493690613 container attach fb3b3706ece3fa8989a72eaec127cf31610464bdfc8787ac51c6084cd1320d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=squid-b53b6d3, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-12T03:14:50.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 12 03:14:49 smithi016 podman[148004]: 2024-09-12 03:14:49.948343464 +0000 UTC m=+0.572051158 container died fb3b3706ece3fa8989a72eaec127cf31610464bdfc8787ac51c6084cd1320d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3) 2024-09-12T03:14:50.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.0.service' 2024-09-12T03:14:50.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:50.768 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-12T03:14:50.768 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-12T03:14:50.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.1 2024-09-12T03:14:51.093 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:50 smithi033 systemd[1]: Stopping Ceph osd.1 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:51.496 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T03:14:51.091+0000 7f7e50614640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:51.497 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T03:14:51.091+0000 7f7e50614640 -1 osd.1 83 *** Got signal Terminated *** 2024-09-12T03:14:51.497 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1[39589]: 2024-09-12T03:14:51.091+0000 7f7e50614640 -1 osd.1 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:14:56.935 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:56 smithi033 podman[75982]: 2024-09-12 03:14:56.668441813 +0000 UTC m=+5.699958732 container died 86dee804a9e93c54645e6984cdcbe94c876df6ccd504f7ff7381971983de42fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-12T03:14:57.246 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[75982]: 2024-09-12 03:14:57.009685474 +0000 UTC m=+6.041202391 container cleanup 86dee804a9e93c54645e6984cdcbe94c876df6ccd504f7ff7381971983de42fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2) 2024-09-12T03:14:57.246 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 bash[75982]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1 2024-09-12T03:14:57.661 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76000]: 2024-09-12 03:14:57.25287494 +0000 UTC m=+0.582516542 container remove 86dee804a9e93c54645e6984cdcbe94c876df6ccd504f7ff7381971983de42fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-12T03:14:57.973 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76088]: 2024-09-12 03:14:57.662253315 +0000 UTC m=+0.098585455 container create 62c32679b6da33280496476c3141b59e27ca9b4325cdac48c69a33b7e19f12b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-12T03:14:57.973 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76088]: 2024-09-12 03:14:57.579820339 +0000 UTC m=+0.016152458 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:14:57.973 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76088]: 2024-09-12 03:14:57.906399688 +0000 UTC m=+0.342731786 container init 62c32679b6da33280496476c3141b59e27ca9b4325cdac48c69a33b7e19f12b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:14:57.973 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76088]: 2024-09-12 03:14:57.909693059 +0000 UTC m=+0.346025167 container start 62c32679b6da33280496476c3141b59e27ca9b4325cdac48c69a33b7e19f12b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:14:57.973 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:57 smithi033 podman[76088]: 2024-09-12 03:14:57.972903813 +0000 UTC m=+0.409235913 container attach 62c32679b6da33280496476c3141b59e27ca9b4325cdac48c69a33b7e19f12b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-12T03:14:58.247 INFO:journalctl@ceph.osd.1.smithi033.stdout:Sep 12 03:14:58 smithi033 podman[76088]: 2024-09-12 03:14:58.113323221 +0000 UTC m=+0.549655329 container died 62c32679b6da33280496476c3141b59e27ca9b4325cdac48c69a33b7e19f12b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-12T03:14:58.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.1.service' 2024-09-12T03:14:58.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:14:58.786 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-12T03:14:58.786 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-12T03:14:58.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.2 2024-09-12T03:14:59.091 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:14:58 smithi184 systemd[1]: Stopping Ceph osd.2 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:14:59.544 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:14:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T03:14:59.090+0000 7f94b1433640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:14:59.545 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:14:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T03:14:59.090+0000 7f94b1433640 -1 osd.2 83 *** Got signal Terminated *** 2024-09-12T03:14:59.545 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:14:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2[40297]: 2024-09-12T03:14:59.090+0000 7f94b1433640 -1 osd.2 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:04.950 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:04 smithi184 podman[75316]: 2024-09-12 03:15:04.692301498 +0000 UTC m=+5.713316942 container died dfe3b104dfed4d9ce3937437b61843c6197cc57eee3182c6423c0485db4fb190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, ceph=True, org.label-schema.build-date=20240731) 2024-09-12T03:15:05.294 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75316]: 2024-09-12 03:15:05.06313198 +0000 UTC m=+6.084147422 container cleanup dfe3b104dfed4d9ce3937437b61843c6197cc57eee3182c6423c0485db4fb190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:05.295 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 bash[75316]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2 2024-09-12T03:15:05.662 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75331]: 2024-09-12 03:15:05.356095015 +0000 UTC m=+0.651276524 container remove dfe3b104dfed4d9ce3937437b61843c6197cc57eee3182c6423c0485db4fb190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-12T03:15:05.942 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75420]: 2024-09-12 03:15:05.565587601 +0000 UTC m=+0.015611325 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:05.942 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75420]: 2024-09-12 03:15:05.70578059 +0000 UTC m=+0.155804300 container create fc732aab2a9f2a7acec4d9edcc5ec94abbb13300d2281e9cf0bb66b3a18483ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:05.942 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75420]: 2024-09-12 03:15:05.942307657 +0000 UTC m=+0.392331371 container init fc732aab2a9f2a7acec4d9edcc5ec94abbb13300d2281e9cf0bb66b3a18483ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-12T03:15:06.295 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:05 smithi184 podman[75420]: 2024-09-12 03:15:05.945773327 +0000 UTC m=+0.395797037 container start fc732aab2a9f2a7acec4d9edcc5ec94abbb13300d2281e9cf0bb66b3a18483ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:06.295 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:06 smithi184 podman[75420]: 2024-09-12 03:15:06.034023961 +0000 UTC m=+0.484047684 container attach fc732aab2a9f2a7acec4d9edcc5ec94abbb13300d2281e9cf0bb66b3a18483ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:06.590 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 12 03:15:06 smithi184 podman[75420]: 2024-09-12 03:15:06.321706718 +0000 UTC m=+0.771730440 container died fc732aab2a9f2a7acec4d9edcc5ec94abbb13300d2281e9cf0bb66b3a18483ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-12T03:15:06.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.2.service' 2024-09-12T03:15:07.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:07.012 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-12T03:15:07.013 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-12T03:15:07.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.3 2024-09-12T03:15:07.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:07 smithi016 systemd[1]: Stopping Ceph osd.3 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:07.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:07 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T03:15:07.406+0000 7f0ba91ed640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:07.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:07 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T03:15:07.406+0000 7f0ba91ed640 -1 osd.3 83 *** Got signal Terminated *** 2024-09-12T03:15:07.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:07 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T03:15:07.406+0000 7f0ba91ed640 -1 osd.3 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:09.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:09.451+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:10.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:09 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:09.755+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:10.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:10.437+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:10.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:10 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:10.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:10.907 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:10 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T03:15:10.562+0000 7f0ba57f2640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.005023+0000 front 2024-09-12T03:14:44.004929+0000 (oldest deadline 2024-09-12T03:15:09.905084+0000) 2024-09-12T03:15:11.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:11.403+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:11.907 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:11 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3[51880]: 2024-09-12T03:15:11.561+0000 7f0ba57f2640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.005023+0000 front 2024-09-12T03:14:44.004929+0000 (oldest deadline 2024-09-12T03:15:09.905084+0000) 2024-09-12T03:15:11.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:11 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:11.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:12.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:12 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:12.308+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:12.794 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:12.360+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:12.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:12.300+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:13.157 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:12 smithi016 podman[148126]: 2024-09-12 03:15:12.979283845 +0000 UTC m=+5.762749064 container died 60dac1a7e9930d25e03ee5203f98707338de61eca6df520ffc95330053fc2c3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-12T03:15:13.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:12 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:12.745+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:13.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:12 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:12.771+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:13.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:13 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:13.277+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:13.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:13 smithi016 podman[148126]: 2024-09-12 03:15:13.498176421 +0000 UTC m=+6.281641638 container cleanup 60dac1a7e9930d25e03ee5203f98707338de61eca6df520ffc95330053fc2c3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-12T03:15:13.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:13 smithi016 bash[148126]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3 2024-09-12T03:15:13.794 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:13.365+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:13.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:13.340+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:14.038 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:13 smithi016 podman[148144]: 2024-09-12 03:15:13.724642152 +0000 UTC m=+0.738702884 container remove 60dac1a7e9930d25e03ee5203f98707338de61eca6df520ffc95330053fc2c3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-12T03:15:14.039 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:13 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:13.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:14.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:13 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:13.812+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:14.368 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:14 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:14.231+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:14.369 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:13.942025947 +0000 UTC m=+0.016635095 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:14.369 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:14.041986272 +0000 UTC m=+0.116595418 container create 5de7c56a9a416efd463d7446a7e1828dad0728ea15efd9b9a6d4d68bf2fe50ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-12T03:15:14.369 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:14.286166587 +0000 UTC m=+0.360775741 container init 5de7c56a9a416efd463d7446a7e1828dad0728ea15efd9b9a6d4d68bf2fe50ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:14.369 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:14.289505906 +0000 UTC m=+0.364115058 container start 5de7c56a9a416efd463d7446a7e1828dad0728ea15efd9b9a6d4d68bf2fe50ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-12T03:15:14.369 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:14.369360636 +0000 UTC m=+0.443969780 container attach 5de7c56a9a416efd463d7446a7e1828dad0728ea15efd9b9a6d4d68bf2fe50ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-12T03:15:14.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 12 03:15:14 smithi016 podman[148230]: 2024-09-12 03:15:14.490401995 +0000 UTC m=+0.565011146 container died 5de7c56a9a416efd463d7446a7e1828dad0728ea15efd9b9a6d4d68bf2fe50ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-12T03:15:14.794 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:14.329+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:14.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:14.326+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:15.072 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:14 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:14.732+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:15.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.3.service' 2024-09-12T03:15:15.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:15.136 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-12T03:15:15.136 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-12T03:15:15.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.4 2024-09-12T03:15:15.165 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:14 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:14.825+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:15.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:15 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:15.200+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:15.471 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:15 smithi033 systemd[1]: Stopping Ceph osd.4 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:15.746 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:15 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:15.470+0000 7f17fa95b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:15.746 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:15 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:15.470+0000 7f17fa95b640 -1 osd.4 83 *** Got signal Terminated *** 2024-09-12T03:15:15.746 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:15 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:15.470+0000 7f17fa95b640 -1 osd.4 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:15.794 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:15.342+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:15.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:15.363+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:16.156 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:15 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:15.765+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:16.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:15 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:15.866+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:16.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:16 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:16.156+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:16.794 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:16 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:16.369+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:16.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:16 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:16.351+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:17.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:16 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:16.740+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:17.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:16 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:16.894+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:17.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:16 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:16.894+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:17.360 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:17 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:17.321+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:17.360 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:17 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:17.274+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:17.635 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:17 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:17.360+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:17.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:17 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:17.201+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:18.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:17 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:17.766+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:18.246 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:17 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:17.962+0000 7f17f6f60640 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.873409+0000 front 2024-09-12T03:14:54.873440+0000 (oldest deadline 2024-09-12T03:15:17.173601+0000) 2024-09-12T03:15:18.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:17 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:17.878+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:18.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:17 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:17.878+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:18.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:18 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:18.171+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:18.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:18 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:18.171+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:18.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:18 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:18.401+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:18.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:18 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:18.306+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:18.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:18 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:18.366+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:18.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:18 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:18.366+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:19.156 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:18 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:18.792+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:19.246 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:19 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:19.001+0000 7f17f6f60640 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.873409+0000 front 2024-09-12T03:14:54.873440+0000 (oldest deadline 2024-09-12T03:15:17.173601+0000) 2024-09-12T03:15:19.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:18 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:18.847+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:19.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:18 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:18.847+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:19.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:19 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:19.156+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:19.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:19 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:19.156+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:19.795 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:19 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:19.370+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:19.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:19 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:19.328+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:19.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:19 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:19.347+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:19.795 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:19 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:19.347+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:20.109 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:19 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:19.777+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:20.246 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:19 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4[46092]: 2024-09-12T03:15:19.998+0000 7f17f6f60640 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.873409+0000 front 2024-09-12T03:14:54.873440+0000 (oldest deadline 2024-09-12T03:15:17.173601+0000) 2024-09-12T03:15:20.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:19 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:19.896+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:20.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:19 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:19.896+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:20.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:20.109+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:20.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:20.109+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:20.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:20 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:20.400+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:20.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:20 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:20.287+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:20.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:20 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:20.302+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:20.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:20 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:20.302+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:20.746 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:20 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:20.334+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:21.090 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:20 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:20.755+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:21.247 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:20 smithi033 podman[76204]: 2024-09-12 03:15:20.843201938 +0000 UTC m=+5.502644803 container died 2f407a9cb3683ec63ff6d72f25f60efe9518e7acd87712ee56bceeb7c814e6fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-12T03:15:21.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:20 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:20.933+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:21.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:20 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:20.933+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:21.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:21 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:21.091+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:21.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:21 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:21.091+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:21.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:21.396+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:21.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:21.396+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:21.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:21.274+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:21.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:21.308+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:21.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:21 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:21.308+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:21.638 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:21 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:21.326+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:21.638 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:21 smithi033 podman[76204]: 2024-09-12 03:15:21.454434991 +0000 UTC m=+6.113877895 container cleanup 2f407a9cb3683ec63ff6d72f25f60efe9518e7acd87712ee56bceeb7c814e6fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-b53b6d3) 2024-09-12T03:15:21.638 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:21 smithi033 bash[76204]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4 2024-09-12T03:15:21.923 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:21 smithi033 podman[76224]: 2024-09-12 03:15:21.638913622 +0000 UTC m=+0.791959927 container remove 2f407a9cb3683ec63ff6d72f25f60efe9518e7acd87712ee56bceeb7c814e6fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-12T03:15:22.130 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:21 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:21.718+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:22.130 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:21 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:21.718+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:22.242 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:21 smithi033 podman[76312]: 2024-09-12 03:15:21.875353334 +0000 UTC m=+0.016203456 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:22.242 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:22 smithi033 podman[76312]: 2024-09-12 03:15:22.006499812 +0000 UTC m=+0.147349932 container create bf0bec054f81c97f0c83a806589da019ca2412ebb23139ea0c9d148e370d8714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:22.242 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:21 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:21.923+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:22.242 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:21 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:21.923+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:22.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:22.130+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:22.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:22.130+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:22.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:22 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:22.286+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:22.497 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:22 smithi033 podman[76312]: 2024-09-12 03:15:22.242320342 +0000 UTC m=+0.383170457 container init bf0bec054f81c97f0c83a806589da019ca2412ebb23139ea0c9d148e370d8714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-12T03:15:22.497 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:22 smithi033 podman[76312]: 2024-09-12 03:15:22.245954522 +0000 UTC m=+0.386804643 container start bf0bec054f81c97f0c83a806589da019ca2412ebb23139ea0c9d148e370d8714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-12T03:15:22.497 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:22 smithi033 podman[76312]: 2024-09-12 03:15:22.342471375 +0000 UTC m=+0.483321525 container attach bf0bec054f81c97f0c83a806589da019ca2412ebb23139ea0c9d148e370d8714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-12T03:15:22.497 INFO:journalctl@ceph.osd.4.smithi033.stdout:Sep 12 03:15:22 smithi033 podman[76312]: 2024-09-12 03:15:22.443323094 +0000 UTC m=+0.584173224 container died bf0bec054f81c97f0c83a806589da019ca2412ebb23139ea0c9d148e370d8714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-12T03:15:22.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:22 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:22.278+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:22.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:22 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:22.353+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:22.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:22 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:22.353+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:22.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:22 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:22.293+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:22.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:22 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:22.293+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:22.997 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:22 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:22.970+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:22.997 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:22 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:22.970+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:23.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.4.service' 2024-09-12T03:15:23.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:22.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:23.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:22 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:22.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:23.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:23.121 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-12T03:15:23.121 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-12T03:15:23.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.5 2024-09-12T03:15:23.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:23.105+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:23.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:23.105+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:23.439 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:23.277+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:23.440 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:23.257+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:23.440 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:23.257+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:23.440 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 systemd[1]: Stopping Ceph osd.5 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:23.440 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:23.379+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:23.440 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:23.379+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:23.441 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:23.439+0000 7f76e31ca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:23.441 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:23.439+0000 7f76e31ca640 -1 osd.5 83 *** Got signal Terminated *** 2024-09-12T03:15:23.441 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:23 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:23.439+0000 7f76e31ca640 -1 osd.5 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:23.746 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:23 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:23.290+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:24.153 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:23.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:24.154 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:23.746+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:24.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:23 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:23.968+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:24.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:23 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:23.968+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:24.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:24.154+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:24.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:24.154+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:24.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:24 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:24.261+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:24.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:24 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:24.333+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:24.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:24 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:24.333+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:24.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:24 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:24.285+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:24.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:24 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:24.285+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:24.746 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:24 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:24.279+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:25.147 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:24.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:25.147 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:24 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:24.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:25.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:24 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:24.994+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:25.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:24 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:24.994+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:25.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:24 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:24.994+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:25.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:25.147+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:25.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:25.147+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:25.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:25.262+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:25.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:25.309+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:25.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:25.309+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:25.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:25.295+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:25.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:25.295+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:25.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:25 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:25.295+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:25.746 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:25 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:25.249+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:26.126 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:25.812+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:26.126 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:25.812+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:26.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:26.126+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:26.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:26.126+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:26.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:26.201+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:26.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:26.201+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:26.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:26.019+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:26.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:26.019+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:26.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:26 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:26.019+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:26.225+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:26.277+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:26.277+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:26.283+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:26.283+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:26.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:26 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:26.283+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:27.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:26.778+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:27.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:26.778+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:27.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:26 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:26.778+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:27.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:27 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:27.212+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:27.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:27 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:27.212+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:27.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:27 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:27.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:27.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:27 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:27.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:27.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:27 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:27.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:27.215+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:27.215+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:27.266+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:27.266+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:27.266+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:02.624222+0000 front 2024-09-12T03:15:02.624253+0000 (oldest deadline 2024-09-12T03:15:26.724379+0000) 2024-09-12T03:15:27.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:27.252+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:27.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:27.252+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:27.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:27 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:27.252+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:27.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:27 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:27.168+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:27.657 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:27 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:27.168+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:28.133 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:27 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:27.764+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:28.133 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:27 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:27.764+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:28.133 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:27 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:27.764+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:28.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:28.133+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:28.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:28.133+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:28.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:28.133+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:28.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:28 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:28.209+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:28.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:28 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:28.209+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:28.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:28 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:28.030+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:28.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:28 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:28.030+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:28.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:28 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:28.030+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:28.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:28.261+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:28.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:28.261+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:28.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:28.219+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.721350+0000 front 2024-09-12T03:14:45.721372+0000 (oldest deadline 2024-09-12T03:15:11.621456+0000) 2024-09-12T03:15:28.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:28.219+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.122600+0000 front 2024-09-12T03:14:55.122751+0000 (oldest deadline 2024-09-12T03:15:20.923159+0000) 2024-09-12T03:15:28.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5[47118]: 2024-09-12T03:15:28.219+0000 7f76db7c7640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:02.624222+0000 front 2024-09-12T03:15:02.624253+0000 (oldest deadline 2024-09-12T03:15:26.724379+0000) 2024-09-12T03:15:28.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:28.235+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:28.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:28.235+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:28.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:28 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:28.235+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:29.145 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:28.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:29.145 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:28.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:29.145 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:28 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:28.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:29.198 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:28 smithi184 podman[75534]: 2024-09-12 03:15:28.9448493 +0000 UTC m=+5.630402811 container died c80d02f9edaafe1432b4676c83a2d31f2f4806133cfe53748753d0854ad9bb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-12T03:15:29.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:29.145+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:29.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:29.145+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:29.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:29.145+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:29.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:29 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:29.171+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:29.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:29 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:29.171+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:29.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:29 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:29.068+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:29.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:29 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:29.068+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:29.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:29 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:29.068+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:29.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:29 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:29.254+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:29.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:29 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:29.254+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:29.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:29 smithi184 podman[75534]: 2024-09-12 03:15:29.306236527 +0000 UTC m=+5.991790023 container cleanup c80d02f9edaafe1432b4676c83a2d31f2f4806133cfe53748753d0854ad9bb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:29.545 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:29 smithi184 bash[75534]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5 2024-09-12T03:15:29.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:29 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:29.198+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:29.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:29 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:29.198+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:29.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:29 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:29.198+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:29.940 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:29 smithi184 podman[75548]: 2024-09-12 03:15:29.632522367 +0000 UTC m=+0.690454028 container remove c80d02f9edaafe1432b4676c83a2d31f2f4806133cfe53748753d0854ad9bb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3) 2024-09-12T03:15:30.130 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:29.742+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:30.130 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:29.742+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:30.130 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:29 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:29.742+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:30.190 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:30 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:30.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:30.190 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:29 smithi184 podman[75638]: 2024-09-12 03:15:29.843262208 +0000 UTC m=+0.015782175 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:30.190 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:29 smithi184 podman[75638]: 2024-09-12 03:15:29.949809998 +0000 UTC m=+0.122329953 container create 2205e21cc6cd44ad10b268a8a33bb55aa11e7b4c8ded8ac950bb87276179aa7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:30.190 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:30 smithi184 podman[75638]: 2024-09-12 03:15:30.135517558 +0000 UTC m=+0.308037512 container init 2205e21cc6cd44ad10b268a8a33bb55aa11e7b4c8ded8ac950bb87276179aa7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-12T03:15:30.190 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:30 smithi184 podman[75638]: 2024-09-12 03:15:30.138748956 +0000 UTC m=+0.311268917 container start 2205e21cc6cd44ad10b268a8a33bb55aa11e7b4c8ded8ac950bb87276179aa7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5-deactivate, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-12T03:15:30.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:30.130+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:30.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:30.130+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:30.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:30.130+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:30.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:30.172+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:30.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:30.172+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:30.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:30.026+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:30.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:30.026+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:30.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:30 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:30.026+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:30.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:30 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:30.303+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:30.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:30 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:30.303+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:30.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:30 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:30.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:30.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:30 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:30.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:30.546 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:30 smithi184 podman[75638]: 2024-09-12 03:15:30.227048837 +0000 UTC m=+0.399568807 container attach 2205e21cc6cd44ad10b268a8a33bb55aa11e7b4c8ded8ac950bb87276179aa7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS) 2024-09-12T03:15:30.546 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 12 03:15:30 smithi184 podman[75638]: 2024-09-12 03:15:30.325471345 +0000 UTC m=+0.497991316 container died 2205e21cc6cd44ad10b268a8a33bb55aa11e7b4c8ded8ac950bb87276179aa7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:15:30.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.5.service' 2024-09-12T03:15:30.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:30.939 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-12T03:15:30.940 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-12T03:15:30.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.6 2024-09-12T03:15:31.090 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:30 smithi016 systemd[1]: Stopping Ceph osd.6 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:31.090 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:30.744+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:31.090 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:30.744+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:31.090 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:30 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:30.744+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:31.376 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.089+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:31.377 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.089+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:31.377 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.089+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:31.377 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.250+0000 7f634bddd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:31.377 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.250+0000 7f634bddd640 -1 osd.6 83 *** Got signal Terminated *** 2024-09-12T03:15:31.377 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:31.250+0000 7f634bddd640 -1 osd.6 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:31.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:31 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:31.201+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:31.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:31 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:31.201+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:31.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:31 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:31.063+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:31.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:31 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:31.063+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:31.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:31 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:31.063+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:31.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:31 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:31.311+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:31.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:31 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:31.311+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:31.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:31 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:31.205+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:31.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:31 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:31.205+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:31.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:31 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:31.205+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:32.055 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:31.769+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:32.056 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:31.769+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:32.056 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:31.769+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:32.378 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:32.055+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:32.379 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:32.055+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:32.379 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:32.055+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:32.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:32.241+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:32.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:32.241+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:32.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:32.111+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:32.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:32.111+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:32.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:32 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:32.111+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:32.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:32 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:32.345+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:32.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:32 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:32.345+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:32.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:32 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:32.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:32.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:32 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:32.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:32.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:32 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:32.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:33.024 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:32.737+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:33.025 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:32.737+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:33.025 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:32 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:32.737+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:33.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:33.024+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:33.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:33.024+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:33.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:33.024+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:33.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:33 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:33.222+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:33.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:33 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:33.222+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:33.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:33 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:33.104+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:33.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:33 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:33.104+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:33.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:33 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:33.104+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:33.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:33 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:33.361+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:33.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:33 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:33.361+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:33.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:33 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:33.124+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:33.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:33 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:33.124+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:33.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:33 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:33.124+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:34.006 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:33.747+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:34.006 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:33.747+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:34.006 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:33 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:33.747+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:34.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:34.006+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:34.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:34.006+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:34.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:34.006+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:34.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:34.006+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:08.339047+0000 front 2024-09-12T03:15:08.339131+0000 (oldest deadline 2024-09-12T03:15:33.639174+0000) 2024-09-12T03:15:34.446 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:34.118+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:34.446 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:34.118+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:34.446 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:34.118+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:34.447 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:34.341+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:34.447 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:34.341+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:34.447 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:34 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:34.341+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:34.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:34.206+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:34.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:34.206+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:34.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:34.086+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:34.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:34.086+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:34.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:34.086+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:35.030 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:34.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:35.031 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:34.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:35.031 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:34.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:35.031 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:34 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:34.780+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:35.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:35.031+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:35.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:35.031+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:35.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:35.031+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:35.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:35.031+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:08.339047+0000 front 2024-09-12T03:15:08.339131+0000 (oldest deadline 2024-09-12T03:15:33.639174+0000) 2024-09-12T03:15:35.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:35.218+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:35.218+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:35.218+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:35.099+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:35.099+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:35.099+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:35.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:35 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:35.099+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:35.308+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:35.308+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:35.308+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:35.105+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:35.105+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:35.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:35 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:35.105+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:36.046 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:35.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:36.047 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:35.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:36.047 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:35.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:36.047 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:35 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:35.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:36.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:36.046+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.734706+0000 front 2024-09-12T03:14:45.734659+0000 (oldest deadline 2024-09-12T03:15:11.634768+0000) 2024-09-12T03:15:36.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:36.046+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.036929+0000 front 2024-09-12T03:14:55.037062+0000 (oldest deadline 2024-09-12T03:15:17.336797+0000) 2024-09-12T03:15:36.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:36.046+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.738343+0000 front 2024-09-12T03:15:03.738323+0000 (oldest deadline 2024-09-12T03:15:27.838275+0000) 2024-09-12T03:15:36.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6[58396]: 2024-09-12T03:15:36.046+0000 7f63483e2640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:08.339047+0000 front 2024-09-12T03:15:08.339131+0000 (oldest deadline 2024-09-12T03:15:33.639174+0000) 2024-09-12T03:15:36.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:36.199+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:36.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:36.199+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:36.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:36.199+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:36.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:36.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:36.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:36.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:36.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:36.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:36.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:36 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:36.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:36.325+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:36.325+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:36.325+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:36.117+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:36.117+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:36.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:36 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:36.117+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:37.125 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:36 smithi016 podman[148332]: 2024-09-12 03:15:36.821901623 +0000 UTC m=+5.678064776 container died 8976210906fc413ebc46958d166d5adecfb73b8669bfb46bdefbb090119e1074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-12T03:15:37.125 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:36.762+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:37.125 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:36.762+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:37.125 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:36.762+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:37.125 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:36 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:36.762+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:37.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:37 smithi016 podman[148332]: 2024-09-12 03:15:37.300060884 +0000 UTC m=+6.156224091 container cleanup 8976210906fc413ebc46958d166d5adecfb73b8669bfb46bdefbb090119e1074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:15:37.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:37 smithi016 bash[148332]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6 2024-09-12T03:15:37.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:37.217+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:37.217+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:37.217+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:37.062+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:37.062+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:37.062+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:37.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:37 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:37.062+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:37.544 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:37.371+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:37.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:37.371+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:37.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:37.371+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:37.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:37.134+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:37.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:37.134+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:37.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:37 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:37.134+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:37.884 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:37 smithi016 podman[148357]: 2024-09-12 03:15:37.568188509 +0000 UTC m=+0.744959408 container remove 8976210906fc413ebc46958d166d5adecfb73b8669bfb46bdefbb090119e1074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:37.884 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:37.731+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:37.884 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:37.731+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:37.885 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:37.731+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:37.885 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:37 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:37.731+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:38.146 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:37 smithi016 podman[148444]: 2024-09-12 03:15:37.787654348 +0000 UTC m=+0.016433784 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:38.146 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:37 smithi016 podman[148444]: 2024-09-12 03:15:37.902482413 +0000 UTC m=+0.131261897 container create f173d100024082efba9e0dd44c0f49fbc4d95fa99e278cf6e2d7d91586bcc2aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-12T03:15:38.146 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:38 smithi016 podman[148444]: 2024-09-12 03:15:38.146577916 +0000 UTC m=+0.375357352 container init f173d100024082efba9e0dd44c0f49fbc4d95fa99e278cf6e2d7d91586bcc2aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-12T03:15:38.389 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:38.133+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:38.389 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:38.133+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:38.389 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:38.133+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:38.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:38 smithi016 podman[148444]: 2024-09-12 03:15:38.151201127 +0000 UTC m=+0.379980565 container start f173d100024082efba9e0dd44c0f49fbc4d95fa99e278cf6e2d7d91586bcc2aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-12T03:15:38.407 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:38 smithi016 podman[148444]: 2024-09-12 03:15:38.238076699 +0000 UTC m=+0.466856144 container attach f173d100024082efba9e0dd44c0f49fbc4d95fa99e278cf6e2d7d91586bcc2aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-12T03:15:38.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 12 03:15:38 smithi016 podman[148444]: 2024-09-12 03:15:38.358638029 +0000 UTC m=+0.587417465 container died f173d100024082efba9e0dd44c0f49fbc4d95fa99e278cf6e2d7d91586bcc2aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-12T03:15:38.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:38.186+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:38.186+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:38.186+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:38.081+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:38.081+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:38.081+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:38.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:38 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:38.081+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:38.794 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:38.388+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:38.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:38.388+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:38.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:38 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:38.388+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:39.029 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:38 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:38.698+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:39.030 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:38 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:38.698+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:39.030 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:38 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:38.698+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:39.030 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:38 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:38.698+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:39.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.6.service' 2024-09-12T03:15:39.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:39.101 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-12T03:15:39.101 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-12T03:15:39.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.7 2024-09-12T03:15:39.424 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:39.112+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:39.424 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:39.112+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:39.424 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:39.112+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:39.424 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:39.112+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 systemd[1]: Stopping Ceph osd.7 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.148+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.148+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.148+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.423+0000 7f21dd122640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.423+0000 7f21dd122640 -1 osd.7 83 *** Got signal Terminated *** 2024-09-12T03:15:39.425 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:39.423+0000 7f21dd122640 -1 osd.7 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:39.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:39.366+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:39.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:39.366+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:39.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:39.366+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:39.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:39.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:39.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:39.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:39.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:39 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:39.162+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:39.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:39.653+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:39.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:39.653+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:39.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:39.653+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:39.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:39 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:39.653+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:40.139+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:40.139+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:40.139+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:40.139+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:40.154+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:40.154+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:40.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:40 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:40.154+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:40.360+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:40.360+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:40.360+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:40.136+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:40.136+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:40.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:40 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:40.136+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:40.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:40 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:40.628+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:40.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:40 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:40.628+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:40.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:40 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:40.628+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:40.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:40 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:40.628+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:41.403 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:41.135+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:41.403 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:41.135+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:41.403 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:41.135+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:41.496 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:41.152+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:41.152+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:41.152+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:41.109+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:41.109+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:41.109+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:41.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:41 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:41.109+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:41.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:41.402+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:41.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:41.402+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:41.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:41 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:41.402+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:41.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:41 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:41.594+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:41.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:41 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:41.594+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:41.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:41 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:41.594+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:41.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:41 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:41.594+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:42.105+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:42.105+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:42.105+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:42.105+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:18.965747+0000 front 2024-09-12T03:15:18.965799+0000 (oldest deadline 2024-09-12T03:15:41.265823+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:42.064+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:42.064+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:42.064+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:42.064+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:42.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:42 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:42.064+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:42.416+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:42.416+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:42.416+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:42.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:42.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:42.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:42 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:42.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:42.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:42.597+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:42.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:42.597+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:42.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:42.597+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:42.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:42.597+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:42.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:42 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:42.597+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:43.398 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:43.108+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:43.398 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:43.108+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:43.398 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:43.108+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:43.398 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:43.108+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:18.965747+0000 front 2024-09-12T03:15:18.965799+0000 (oldest deadline 2024-09-12T03:15:41.265823+0000) 2024-09-12T03:15:43.398 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:43.094+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:43.399 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:43.094+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:43.399 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:43.094+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:43.399 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:43.094+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:43.399 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:43.094+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:43.418 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:43.139+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:43.419 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:43.139+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:43.419 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:43.139+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:43.794 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:43.418+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:43.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:43.418+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:43.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:43 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:43.418+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:43.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:43 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:43.570+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:43.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:43 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:43.570+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:43.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:43 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:43.570+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:43.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:43 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:43.570+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:43.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:43 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:43.570+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:44.418 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:44.098+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:56.063129+0000 front 2024-09-12T03:14:56.063164+0000 (oldest deadline 2024-09-12T03:15:20.163297+0000) 2024-09-12T03:15:44.418 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:44.098+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.163849+0000 front 2024-09-12T03:15:00.163908+0000 (oldest deadline 2024-09-12T03:15:26.063778+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:44.098+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.764752+0000 front 2024-09-12T03:15:10.764783+0000 (oldest deadline 2024-09-12T03:15:34.264859+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7[52927]: 2024-09-12T03:15:44.098+0000 7f21d571f640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:18.965747+0000 front 2024-09-12T03:15:18.965799+0000 (oldest deadline 2024-09-12T03:15:41.265823+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:44.070+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:44.070+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:44.070+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:44.070+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:44.419 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:44 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:44.070+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:44.426 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:44.154+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:44.426 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:44.154+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:44.427 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:44.154+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:44.427 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:44.154+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:44.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:44.426+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:44.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:44.426+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:44.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:44.426+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:44.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:44 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:44.426+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:44.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:44 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:44.604+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:44.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:44 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:44.604+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:44.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:44 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:44.604+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:44.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:44 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:44.604+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:44.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:44 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:44.604+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:45.299 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:45 smithi033 podman[76416]: 2024-09-12 03:15:45.042124656 +0000 UTC m=+5.746488736 container died 71d64fcf79b4127dfa56f211de5334fede7092951b74944bec745d8d3fe2052d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-12T03:15:45.299 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:45 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:45.038+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:45.300 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:45 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:45.038+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:45.300 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:45 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:45.038+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:45.300 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:45 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:45.038+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:45.300 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:45 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:45.038+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:45.451 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:45.138+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:45.452 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:45.138+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:45.452 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:45.138+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:45.452 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:45.138+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:45.682 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:45 smithi033 podman[76416]: 2024-09-12 03:15:45.406688981 +0000 UTC m=+6.111053060 container cleanup 71d64fcf79b4127dfa56f211de5334fede7092951b74944bec745d8d3fe2052d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:45.682 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:45 smithi033 bash[76416]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7 2024-09-12T03:15:45.755 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:45.451+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:45.755 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:45.451+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:45.755 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:45.451+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:45.755 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:45 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:45.451+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:45.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:45 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:45.644+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:45.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:45 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:45.644+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:45.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:45 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:45.644+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:45.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:45 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:45.644+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:45.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:45 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:45.644+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:45.995 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:45 smithi033 podman[76442]: 2024-09-12 03:15:45.68302682 +0000 UTC m=+0.641571397 container remove 71d64fcf79b4127dfa56f211de5334fede7092951b74944bec745d8d3fe2052d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3) 2024-09-12T03:15:46.246 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:45 smithi033 podman[76535]: 2024-09-12 03:15:45.899059423 +0000 UTC m=+0.015924117 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:46.246 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:46 smithi033 podman[76535]: 2024-09-12 03:15:46.04270465 +0000 UTC m=+0.159569324 container create c552eef9fb660ce0188c33fe7154a126abd5cea587b9ade777afe01df584f01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:46.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:46 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:46.001+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:46.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:46 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:46.001+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:46.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:46 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:46.001+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:46.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:46 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:46.001+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:46.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:46 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:46.001+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:46.418 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:46.150+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:46.418 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:46.150+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:46.418 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:46.150+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:46.418 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:46.150+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:46.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:46 smithi033 podman[76535]: 2024-09-12 03:15:46.278581011 +0000 UTC m=+0.395445685 container init c552eef9fb660ce0188c33fe7154a126abd5cea587b9ade777afe01df584f01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-12T03:15:46.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:46 smithi033 podman[76535]: 2024-09-12 03:15:46.28176579 +0000 UTC m=+0.398630465 container start c552eef9fb660ce0188c33fe7154a126abd5cea587b9ade777afe01df584f01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7-deactivate, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-12T03:15:46.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:46 smithi033 podman[76535]: 2024-09-12 03:15:46.353629958 +0000 UTC m=+0.470494655 container attach c552eef9fb660ce0188c33fe7154a126abd5cea587b9ade777afe01df584f01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:15:46.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Sep 12 03:15:46 smithi033 podman[76535]: 2024-09-12 03:15:46.47667682 +0000 UTC m=+0.593541516 container died c552eef9fb660ce0188c33fe7154a126abd5cea587b9ade777afe01df584f01c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-7-deactivate, RELEASE=squid-b53b6d3, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-12T03:15:46.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:46.417+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:46.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:46.417+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:46.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:46.417+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:46.796 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:46 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:46.417+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:47.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.7.service' 2024-09-12T03:15:47.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:47.058 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-12T03:15:47.058 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-12T03:15:47.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.8 2024-09-12T03:15:47.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:46 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:46.670+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:47.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:46 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:46.670+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:47.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:46 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:46.670+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:47.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:46 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:46.670+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:47.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:46 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:46.670+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:47.357 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 systemd[1]: Stopping Ceph osd.8 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:47.358 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:47.143+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:47.358 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:47.143+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:47.358 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:47.143+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:47.358 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:47.143+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:47.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:47.002+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:47.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:47.002+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:47.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:47.002+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:47.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:47.002+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:47.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:47.002+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.356+0000 7f05a89df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.356+0000 7f05a89df640 -1 osd.8 83 *** Got signal Terminated *** 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.356+0000 7f05a89df640 -1 osd.8 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.407+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.407+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.407+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:47.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:47 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:47.407+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:47.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:47 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:47.624+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:47.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:47 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:47.624+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:47.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:47 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:47.624+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:47.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:47 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:47.624+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:47.907 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:47 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:47.624+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:48.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:48.052+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:48.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:48.052+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:48.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:48.052+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:48.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:48.052+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:48.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:48 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:48.052+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:48.420+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:48.420+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:48.420+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:48.420+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:48.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:48.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:48.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:48.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:48 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:48.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:48 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:48.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:48 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:48.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:48 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:48.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:48 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:48.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:48 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:48.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:49.471 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:49.129+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:49.471 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:49.129+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:49.471 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:49.130+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:49.471 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:49.130+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:49.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:49 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:49.072+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:49.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:49 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:49.072+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:49.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:49 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:49.072+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:49.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:49 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:49.072+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:49.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:49 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:49.072+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:49.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:49.471+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:49.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:49.471+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:49.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:49.471+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:49.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:49.471+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:49.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:49 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:49.471+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:23.509731+0000 front 2024-09-12T03:15:23.509769+0000 (oldest deadline 2024-09-12T03:15:49.409899+0000) 2024-09-12T03:15:50.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:50.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:50.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:50.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:50.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:50.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:49 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:49.673+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:50.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:50.105+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:50.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:50.105+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:50.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:50.105+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:50.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:50.105+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:50.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:50 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:50.105+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:50.520 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:50.111+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:50.520 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:50.111+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:50.520 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:50.111+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:50.520 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:50.111+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:50.520 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:50.111+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:50.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:50.519+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:50.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:50.519+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:50.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:50.519+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:50.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:50.519+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:50.795 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:50 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:50.519+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:23.509731+0000 front 2024-09-12T03:15:23.509769+0000 (oldest deadline 2024-09-12T03:15:49.409899+0000) 2024-09-12T03:15:51.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:51.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:51.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:51.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:51.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:51.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:50 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:50.709+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:51.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:51.131+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:51.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:51.131+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:51.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:51.131+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:51.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:51.131+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:51.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:51 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:51.131+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:51.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:51.125+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:51.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:51.125+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:51.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:51.125+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:51.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:51.125+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:51.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:51.125+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:52.045 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:51.564+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.003557+0000 front 2024-09-12T03:14:54.003518+0000 (oldest deadline 2024-09-12T03:15:16.903601+0000) 2024-09-12T03:15:52.045 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:51.564+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:03.706143+0000 front 2024-09-12T03:15:03.706180+0000 (oldest deadline 2024-09-12T03:15:26.606381+0000) 2024-09-12T03:15:52.045 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:51.564+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:06.606902+0000 front 2024-09-12T03:15:06.607031+0000 (oldest deadline 2024-09-12T03:15:34.207533+0000) 2024-09-12T03:15:52.045 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:51.564+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.708831+0000 front 2024-09-12T03:15:17.708842+0000 (oldest deadline 2024-09-12T03:15:43.509409+0000) 2024-09-12T03:15:52.045 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8[54104]: 2024-09-12T03:15:51.564+0000 7f05a4fe4640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:23.509731+0000 front 2024-09-12T03:15:23.509769+0000 (oldest deadline 2024-09-12T03:15:49.409899+0000) 2024-09-12T03:15:52.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:52.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:52.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:52.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:52.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:52.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:51 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:51.715+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:52.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:52.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:52.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:52.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:52.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:52.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:52.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:52.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:52.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:52 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:52.097+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:52.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:52 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:52.076+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:52.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:52 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:52.076+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:52.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:52 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:52.076+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:52.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:52 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:52.076+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:52.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:52 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:52.076+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:53.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:52 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:52.749+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:53.223 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:52 smithi184 podman[75752]: 2024-09-12 03:15:52.937401985 +0000 UTC m=+5.683232898 container died 84ba08f1808d07ec6ec162bf5a69d9758b944aa2aa317be52469693b60118d24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-12T03:15:53.224 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:53 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:53.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:53.224 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:53 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:53.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:53.224 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:53 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:53.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:53.224 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:53 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:53.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:53.224 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:53 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:53.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:53.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:53 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:53.076+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:53.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:53 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:53.076+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:53.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:53 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:53.076+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:53.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:53 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:53.076+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:53.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:53 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:53.076+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:53.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:53 smithi184 podman[75752]: 2024-09-12 03:15:53.34017353 +0000 UTC m=+6.086004435 container cleanup 84ba08f1808d07ec6ec162bf5a69d9758b944aa2aa317be52469693b60118d24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:15:53.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:53 smithi184 bash[75752]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8 2024-09-12T03:15:53.894 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:53 smithi184 podman[75770]: 2024-09-12 03:15:53.583039727 +0000 UTC m=+0.643232982 container remove 84ba08f1808d07ec6ec162bf5a69d9758b944aa2aa317be52469693b60118d24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-12T03:15:54.152 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:54 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:54.048+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:54.152 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:54 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:54.048+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:54.152 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:54 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:54.048+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:54.152 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:54 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:54.048+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:54.152 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:54 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:54.048+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:54.153 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:53 smithi184 podman[75859]: 2024-09-12 03:15:53.797320987 +0000 UTC m=+0.015599360 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:15:54.153 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:53 smithi184 podman[75859]: 2024-09-12 03:15:53.900353846 +0000 UTC m=+0.118632271 container create ffdb776fb1fd2ec6cada6fcdbc37e87849116179f86b58547a491017ce6f7364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8-deactivate, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-12T03:15:54.153 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:54 smithi184 podman[75859]: 2024-09-12 03:15:54.152910916 +0000 UTC m=+0.371189279 container init ffdb776fb1fd2ec6cada6fcdbc37e87849116179f86b58547a491017ce6f7364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:15:54.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:54.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:54.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:54.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:54.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:54.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:53 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:53.720+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:54.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:54.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:54.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:54.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:54.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:54.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:54 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:54.043+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:15:54.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:54 smithi184 podman[75859]: 2024-09-12 03:15:54.156580795 +0000 UTC m=+0.374859157 container start ffdb776fb1fd2ec6cada6fcdbc37e87849116179f86b58547a491017ce6f7364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-12T03:15:54.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:54 smithi184 podman[75859]: 2024-09-12 03:15:54.252847805 +0000 UTC m=+0.471126176 container attach ffdb776fb1fd2ec6cada6fcdbc37e87849116179f86b58547a491017ce6f7364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8-deactivate, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-12T03:15:54.545 INFO:journalctl@ceph.osd.8.smithi184.stdout:Sep 12 03:15:54 smithi184 podman[75859]: 2024-09-12 03:15:54.341361029 +0000 UTC m=+0.559639394 container died ffdb776fb1fd2ec6cada6fcdbc37e87849116179f86b58547a491017ce6f7364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-12T03:15:54.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.8.service' 2024-09-12T03:15:54.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:15:54.944 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-12T03:15:54.944 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-12T03:15:54.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.9 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:55.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:54.757+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:55.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:54 smithi016 systemd[1]: Stopping Ceph osd.9 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:15:55.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:55.017+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:55.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:55.017+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:55.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:55.017+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:55.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:55.017+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:55.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:55.017+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:55.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:55.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:55.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:55.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:55.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:55.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:55 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:55.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:15:55.657 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.252+0000 7fdfd26c0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:15:55.657 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.252+0000 7fdfd26c0640 -1 osd.9 83 *** Got signal Terminated *** 2024-09-12T03:15:55.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.252+0000 7fdfd26c0640 -1 osd.9 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:15:56.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:56.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:56.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:56.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:56.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:56.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:55 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:55.712+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:56.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:56.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:56.246 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:56.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:56.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:56.247 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:56 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:56.029+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:15:56.249 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:56.032+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:56.249 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:56.032+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:56.249 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:56.032+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:56.249 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:56.032+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:56.249 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:56 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:56.032+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:57.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:57.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:57.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:57.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:57.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:57.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:56 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:56.666+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:57.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:57 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:57.014+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:15:57.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:57.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:57.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:57.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:57.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:57.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:57 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:57.049+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:15:58.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:58.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:58.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:58.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:58.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:58.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:57 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:57.675+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:58.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:58.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:58.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:58.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:58.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:58.498 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:58 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:58.011+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:58.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:58 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:58.052+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:15:59.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:58 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:58.700+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:15:59.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:15:59.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:15:59.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:15:59.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:15:59.296 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:15:59.296 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:15:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:15:59.029+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:15:59.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:15:59.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:15:59.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:15:59.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:15:59.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:15:59.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:15:59 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:15:59.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:00.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:47.116365+0000 front 2024-09-12T03:14:47.116314+0000 (oldest deadline 2024-09-12T03:15:09.416457+0000) 2024-09-12T03:16:00.157 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:55.218416+0000 front 2024-09-12T03:14:55.218627+0000 (oldest deadline 2024-09-12T03:15:21.019188+0000) 2024-09-12T03:16:00.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:01.019836+0000 front 2024-09-12T03:15:01.019786+0000 (oldest deadline 2024-09-12T03:15:26.319778+0000) 2024-09-12T03:16:00.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:10.921268+0000 front 2024-09-12T03:15:10.921295+0000 (oldest deadline 2024-09-12T03:15:34.421393+0000) 2024-09-12T03:16:00.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:16.122262+0000 front 2024-09-12T03:15:16.122214+0000 (oldest deadline 2024-09-12T03:15:42.022315+0000) 2024-09-12T03:16:00.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:25.523248+0000 front 2024-09-12T03:15:25.523233+0000 (oldest deadline 2024-09-12T03:15:49.023164+0000) 2024-09-12T03:16:00.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:15:59 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9[65087]: 2024-09-12T03:15:59.733+0000 7fdfcacbd640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.024407+0000 front 2024-09-12T03:15:33.024283+0000 (oldest deadline 2024-09-12T03:15:58.924440+0000) 2024-09-12T03:16:00.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:00.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:00 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:00.015+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:00.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:00 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:00.074+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:00.961 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:00 smithi016 podman[148545]: 2024-09-12 03:16:00.686874997 +0000 UTC m=+5.537960905 container died f0cca31c98ccade66f223ee4799eccbfa30a155df01fee3dd8c6b37f0563db06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-12T03:16:01.407 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 podman[148545]: 2024-09-12 03:16:01.186136069 +0000 UTC m=+6.037221975 container cleanup f0cca31c98ccade66f223ee4799eccbfa30a155df01fee3dd8c6b37f0563db06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-12T03:16:01.408 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 bash[148545]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9 2024-09-12T03:16:01.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:01.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:01.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:01.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:01.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:01.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:01.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:01 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:01.014+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:01.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:01 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:01.064+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:01.736 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 podman[148564]: 2024-09-12 03:16:01.412412972 +0000 UTC m=+0.720071460 container remove f0cca31c98ccade66f223ee4799eccbfa30a155df01fee3dd8c6b37f0563db06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-12T03:16:01.998 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 podman[148650]: 2024-09-12 03:16:01.639880649 +0000 UTC m=+0.016156817 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:16:01.998 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 podman[148650]: 2024-09-12 03:16:01.763134865 +0000 UTC m=+0.139411021 container create b38a8dd24031d051b3b05dc38ca0a32256b1f1e73ce71e51fd1d49addf84a620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-12T03:16:01.998 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:01 smithi016 podman[148650]: 2024-09-12 03:16:01.998849435 +0000 UTC m=+0.375125592 container init b38a8dd24031d051b3b05dc38ca0a32256b1f1e73ce71e51fd1d49addf84a620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux ) 2024-09-12T03:16:02.407 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:02 smithi016 podman[148650]: 2024-09-12 03:16:02.002237878 +0000 UTC m=+0.378514035 container start b38a8dd24031d051b3b05dc38ca0a32256b1f1e73ce71e51fd1d49addf84a620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3) 2024-09-12T03:16:02.407 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:02 smithi016 podman[148650]: 2024-09-12 03:16:02.099100367 +0000 UTC m=+0.475376527 container attach b38a8dd24031d051b3b05dc38ca0a32256b1f1e73ce71e51fd1d49addf84a620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-12T03:16:02.407 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 12 03:16:02 smithi016 podman[148650]: 2024-09-12 03:16:02.197248004 +0000 UTC m=+0.573524164 container died b38a8dd24031d051b3b05dc38ca0a32256b1f1e73ce71e51fd1d49addf84a620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-12T03:16:02.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:02.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:02.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:02.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:02.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:02.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:02.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:02.018+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:02.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:02 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:02.088+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:02.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.9.service' 2024-09-12T03:16:02.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:02.847 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-12T03:16:02.848 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-12T03:16:02.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.10 2024-09-12T03:16:03.159 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:02 smithi033 systemd[1]: Stopping Ceph osd.10 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.050+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.158+0000 7fea486d8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.158+0000 7fea486d8640 -1 osd.10 83 *** Got signal Terminated *** 2024-09-12T03:16:03.160 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:03 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:03.158+0000 7fea486d8640 -1 osd.10 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:03.295 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:03.043+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:04.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:04 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:04.053+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:04.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:04 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:04.090+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:05.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:05.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:05 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:05.013+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:05.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:05 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:05.140+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:06.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:06.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:06 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:06.055+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:06.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:06 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:06.137+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:07.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:07.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:07 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:07.041+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:07.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:07 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:07.185+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:08.496 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:44.540944+0000 front 2024-09-12T03:14:44.540990+0000 (oldest deadline 2024-09-12T03:15:12.141553+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.442828+0000 front 2024-09-12T03:14:54.442792+0000 (oldest deadline 2024-09-12T03:15:16.743070+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:14:59.044577+0000 front 2024-09-12T03:14:59.044565+0000 (oldest deadline 2024-09-12T03:15:24.944582+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-12T03:15:11.946483+0000 front 2024-09-12T03:15:11.946406+0000 (oldest deadline 2024-09-12T03:15:34.846578+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.147588+0000 front 2024-09-12T03:15:17.147616+0000 (oldest deadline 2024-09-12T03:15:41.847601+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:28.249153+0000 front 2024-09-12T03:15:28.249175+0000 (oldest deadline 2024-09-12T03:15:53.449338+0000) 2024-09-12T03:16:08.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10[59655]: 2024-09-12T03:16:08.080+0000 7fea44cdd640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.449831+0000 front 2024-09-12T03:15:33.449856+0000 (oldest deadline 2024-09-12T03:15:59.349742+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:08.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:08 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:08.152+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:08.996 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:08 smithi033 podman[76636]: 2024-09-12 03:16:08.68358848 +0000 UTC m=+5.635980758 container died fad0881271496ec6e9db4b7ec2da9df0bcce005e6dc58d1504ff42d3421abd3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-12T03:16:09.418 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:09 smithi033 podman[76636]: 2024-09-12 03:16:09.200400285 +0000 UTC m=+6.152792488 container cleanup fad0881271496ec6e9db4b7ec2da9df0bcce005e6dc58d1504ff42d3421abd3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-12T03:16:09.418 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:09 smithi033 bash[76636]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10 2024-09-12T03:16:09.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:09.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:09.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:09.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:09.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:09.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:09.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:09 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:09.187+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:09.746 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:09 smithi033 podman[76651]: 2024-09-12 03:16:09.418294315 +0000 UTC m=+0.727010471 container remove fad0881271496ec6e9db4b7ec2da9df0bcce005e6dc58d1504ff42d3421abd3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3) 2024-09-12T03:16:10.087 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:09 smithi033 podman[76739]: 2024-09-12 03:16:09.738922757 +0000 UTC m=+0.015538111 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:16:10.088 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:09 smithi033 podman[76739]: 2024-09-12 03:16:09.85247712 +0000 UTC m=+0.129092464 container create 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, ceph=True) 2024-09-12T03:16:10.088 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 podman[76739]: 2024-09-12 03:16:10.088256552 +0000 UTC m=+0.364871898 container init 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, GIT_CLEAN=True) 2024-09-12T03:16:10.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 podman[76739]: 2024-09-12 03:16:10.091601365 +0000 UTC m=+0.368216722 container start 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, ceph=True) 2024-09-12T03:16:10.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 podman[76739]: 2024-09-12 03:16:10.163049978 +0000 UTC m=+0.439665323 container attach 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-12T03:16:10.497 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 podman[76739]: 2024-09-12 03:16:10.281311374 +0000 UTC m=+0.557926736 container died 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:10.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:10.166+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:10.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.10.service' 2024-09-12T03:16:10.866 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 podman[76762]: 2024-09-12 03:16:10.825073462 +0000 UTC m=+0.533450380 container remove 77e683a3165cd221d4fc2ba12396671c46b60a7f24a9cd70695784ddc32307fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:16:10.866 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.10.service: Deactivated successfully. 2024-09-12T03:16:10.866 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 systemd[1]: Stopped Ceph osd.10 for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T03:16:10.866 INFO:journalctl@ceph.osd.10.smithi033.stdout:Sep 12 03:16:10 smithi033 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.10.service: Consumed 5min 11.631s CPU time. 2024-09-12T03:16:10.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:10.882 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-12T03:16:10.882 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-12T03:16:10.882 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.11 2024-09-12T03:16:11.189 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:10 smithi184 systemd[1]: Stopping Ceph osd.11 for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.189+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.199+0000 7fbcb2412640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:11.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.199+0000 7fbcb2412640 -1 osd.11 83 *** Got signal Terminated *** 2024-09-12T03:16:11.546 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:11 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:11.199+0000 7fbcb2412640 -1 osd.11 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:12.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:12 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:12.173+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:13.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:13 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:13.195+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:14.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:14 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:14.244+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-12T03:14:45.857350+0000 front 2024-09-12T03:14:45.857332+0000 (oldest deadline 2024-09-12T03:15:09.357447+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-09-12T03:14:54.460181+0000 front 2024-09-12T03:14:54.460018+0000 (oldest deadline 2024-09-12T03:15:17.360121+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-09-12T03:15:00.861358+0000 front 2024-09-12T03:15:00.861152+0000 (oldest deadline 2024-09-12T03:15:24.961445+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-09-12T03:15:17.763940+0000 front 2024-09-12T03:15:17.763994+0000 (oldest deadline 2024-09-12T03:15:43.664021+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-09-12T03:15:27.165035+0000 front 2024-09-12T03:15:27.165075+0000 (oldest deadline 2024-09-12T03:15:50.065163+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-12T03:15:33.466595+0000 front 2024-09-12T03:15:33.466766+0000 (oldest deadline 2024-09-12T03:15:56.966556+0000) 2024-09-12T03:16:15.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:15 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11[61152]: 2024-09-12T03:16:15.211+0000 7fbcaaa0f640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-09-12T03:15:43.769014+0000 front 2024-09-12T03:15:43.769000+0000 (oldest deadline 2024-09-12T03:16:08.468808+0000) 2024-09-12T03:16:16.894 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:16 smithi184 podman[75973]: 2024-09-12 03:16:16.636146273 +0000 UTC m=+5.545515928 container died 2fd6c260a3bd316d728cb16bcea85585be779c455b1a13fd2aaa52ba34d49e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-12T03:16:17.282 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:16 smithi184 podman[75973]: 2024-09-12 03:16:16.998141928 +0000 UTC m=+5.907511579 container cleanup 2fd6c260a3bd316d728cb16bcea85585be779c455b1a13fd2aaa52ba34d49e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2) 2024-09-12T03:16:17.282 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:16 smithi184 bash[75973]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11 2024-09-12T03:16:17.545 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[75989]: 2024-09-12 03:16:17.282828512 +0000 UTC m=+0.643258892 container remove 2fd6c260a3bd316d728cb16bcea85585be779c455b1a13fd2aaa52ba34d49e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:16:17.851 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.494517721 +0000 UTC m=+0.015610184 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-12T03:16:17.852 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.591819868 +0000 UTC m=+0.112912320 container create eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:16:17.852 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.785740703 +0000 UTC m=+0.306833157 container init eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, ceph=True, CEPH_POINT_RELEASE=) 2024-09-12T03:16:17.852 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.788885525 +0000 UTC m=+0.309977978 container start eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-12T03:16:17.852 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.852098817 +0000 UTC m=+0.373191278 container attach eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-12T03:16:18.262 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:17 smithi184 podman[76080]: 2024-09-12 03:16:17.973235143 +0000 UTC m=+0.494327598 container died eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux ) 2024-09-12T03:16:18.523 INFO:journalctl@ceph.osd.11.smithi184.stdout:Sep 12 03:16:18 smithi184 podman[76104]: 2024-09-12 03:16:18.505157771 +0000 UTC m=+0.521638463 container remove eb435e5730ae42ff01050f6a35f15f5439f1c73ebefd0cf854a10b514bc8865d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-osd-11-deactivate, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-12T03:16:18.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@osd.11.service' 2024-09-12T03:16:18.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:18.568 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-12T03:16:18.568 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-12T03:16:18.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.a 2024-09-12T03:16:18.871 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:18 smithi016 systemd[1]: Stopping Ceph mds.a for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:19.157 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:18 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: 2024-09-12T03:16:18.871+0000 7f344b00f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:19.157 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:18 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a[71851]: 2024-09-12T03:16:18.871+0000 7f344b00f640 -1 mds.a *** got signal Terminated *** 2024-09-12T03:16:22.504 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:22 smithi016 podman[148764]: 2024-09-12 03:16:22.23462597 +0000 UTC m=+3.466291844 container died 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-12T03:16:22.907 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:22 smithi016 podman[148764]: 2024-09-12 03:16:22.69529669 +0000 UTC m=+3.926962557 container cleanup 4e06ccc6ffd686dfebac3e68c9df7f2f2c85db167e3ed871097c58372e8970e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2) 2024-09-12T03:16:22.907 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 12 03:16:22 smithi016 bash[148764]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-a 2024-09-12T03:16:23.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.a.service' 2024-09-12T03:16:23.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:23.231 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-12T03:16:23.232 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-12T03:16:23.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.d 2024-09-12T03:16:23.566 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:23 smithi016 systemd[1]: Stopping Ceph mds.d for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:23.907 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d[70666]: 2024-09-12T03:16:23.566+0000 7fdd36ea5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:23.907 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:23 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d[70666]: 2024-09-12T03:16:23.566+0000 7fdd36ea5640 -1 mds.d *** got signal Terminated *** 2024-09-12T03:16:24.849 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:24 smithi016 podman[148884]: 2024-09-12 03:16:24.579249569 +0000 UTC m=+1.114040537 container died 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-12T03:16:25.157 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:25 smithi016 podman[148884]: 2024-09-12 03:16:25.03307848 +0000 UTC m=+1.567869439 container cleanup 28b3f8ce1d552bd1bd8b174b77d3e1b427cab3cbc816a4d129cdcac597c747b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2) 2024-09-12T03:16:25.157 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 12 03:16:25 smithi016 bash[148884]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-d 2024-09-12T03:16:25.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.d.service' 2024-09-12T03:16:25.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:25.654 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-12T03:16:25.654 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-12T03:16:25.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.g 2024-09-12T03:16:25.987 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:25 smithi016 systemd[1]: Stopping Ceph mds.g for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:26.407 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: 2024-09-12T03:16:25.987+0000 7fc4d23dc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:26.407 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:25 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g[71455]: 2024-09-12T03:16:25.987+0000 7fc4d23dc640 -1 mds.g *** got signal Terminated *** 2024-09-12T03:16:30.337 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:29 smithi016 podman[149005]: 2024-09-12 03:16:29.996683878 +0000 UTC m=+4.117191938 container died 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-12T03:16:30.657 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 podman[149005]: 2024-09-12 03:16:30.511964271 +0000 UTC m=+4.632472337 container cleanup 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, ceph=True) 2024-09-12T03:16:30.657 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 bash[149005]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g 2024-09-12T03:16:30.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.g.service' 2024-09-12T03:16:30.965 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 podman[149018]: 2024-09-12 03:16:30.713599995 +0000 UTC m=+0.713482276 container remove 84add6388aebb2c75c8e912a1c20e7fc5c8fd5fa6832491e992e9bf9edd3eb27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD) 2024-09-12T03:16:30.965 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.g.service: Deactivated successfully. 2024-09-12T03:16:30.965 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 systemd[1]: Stopped Ceph mds.g for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T03:16:30.965 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 12 03:16:30 smithi016 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.g.service: Consumed 3.983s CPU time. 2024-09-12T03:16:30.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:30.976 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-12T03:16:30.976 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-12T03:16:30.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.j 2024-09-12T03:16:31.291 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:31 smithi016 systemd[1]: Stopping Ceph mds.j for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:31.657 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j[70250]: 2024-09-12T03:16:31.291+0000 7f771713b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:31.657 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:31 smithi016 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j[70250]: 2024-09-12T03:16:31.291+0000 7f771713b640 -1 mds.j *** got signal Terminated *** 2024-09-12T03:16:33.157 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:32 smithi016 podman[149127]: 2024-09-12 03:16:32.875922677 +0000 UTC m=+1.677496171 container died 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-12T03:16:33.585 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:33 smithi016 podman[149127]: 2024-09-12 03:16:33.432747934 +0000 UTC m=+2.234321427 container cleanup 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:16:33.585 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:33 smithi016 bash[149127]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j 2024-09-12T03:16:33.847 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 12 03:16:33 smithi016 podman[149148]: 2024-09-12 03:16:33.717416376 +0000 UTC m=+0.838724029 container remove 761ab1bbce2068940424d92e32bfa35c415062e778b2530c8d0078b43e83d146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-j, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-12T03:16:33.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.j.service' 2024-09-12T03:16:33.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:33.886 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-12T03:16:33.887 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-12T03:16:33.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.b 2024-09-12T03:16:34.184 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:33 smithi033 systemd[1]: Stopping Ceph mds.b for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:34.496 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: 2024-09-12T03:16:34.184+0000 7f9a6a394640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:34.496 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:34 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b[65145]: 2024-09-12T03:16:34.184+0000 7f9a6a394640 -1 mds.b *** got signal Terminated *** 2024-09-12T03:16:38.459 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:38 smithi033 podman[76850]: 2024-09-12 03:16:38.19700191 +0000 UTC m=+4.105799923 container died e7aac3a46c26a0b052eeaed84506993de9c868a8ec8af75629290bb84ae918c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-12T03:16:38.746 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:38 smithi033 podman[76850]: 2024-09-12 03:16:38.558707956 +0000 UTC m=+4.467505968 container cleanup e7aac3a46c26a0b052eeaed84506993de9c868a8ec8af75629290bb84ae918c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-12T03:16:38.746 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:38 smithi033 bash[76850]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b 2024-09-12T03:16:39.143 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.b.service' 2024-09-12T03:16:39.165 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:38 smithi033 podman[76862]: 2024-09-12 03:16:38.910057551 +0000 UTC m=+0.709609594 container remove e7aac3a46c26a0b052eeaed84506993de9c868a8ec8af75629290bb84ae918c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-12T03:16:39.165 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:39 smithi033 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.b.service: Deactivated successfully. 2024-09-12T03:16:39.165 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:39 smithi033 systemd[1]: Stopped Ceph mds.b for ef86fd2a-70af-11ef-bcea-c7b262605968. 2024-09-12T03:16:39.165 INFO:journalctl@ceph.mds.b.smithi033.stdout:Sep 12 03:16:39 smithi033 systemd[1]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.b.service: Consumed 8min 17.065s CPU time. 2024-09-12T03:16:39.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:39.179 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-12T03:16:39.180 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-12T03:16:39.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.e 2024-09-12T03:16:39.496 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:39 smithi033 systemd[1]: Stopping Ceph mds.e for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:39.996 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: 2024-09-12T03:16:39.496+0000 7f591eb79640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:39.996 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:39 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e[64546]: 2024-09-12T03:16:39.496+0000 7f591eb79640 -1 mds.e *** got signal Terminated *** 2024-09-12T03:16:42.538 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:42 smithi033 podman[76969]: 2024-09-12 03:16:42.274191335 +0000 UTC m=+2.871555897 container died 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731) 2024-09-12T03:16:42.855 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:42 smithi033 podman[76969]: 2024-09-12 03:16:42.637569497 +0000 UTC m=+3.234934063 container cleanup 2620755496c0aa22085d52ab6a3e582bb781da6a127df67674d52daef4760755 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2) 2024-09-12T03:16:42.855 INFO:journalctl@ceph.mds.e.smithi033.stdout:Sep 12 03:16:42 smithi033 bash[76969]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-e 2024-09-12T03:16:42.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.e.service' 2024-09-12T03:16:43.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:43.013 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-12T03:16:43.013 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-12T03:16:43.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.h 2024-09-12T03:16:43.246 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:43 smithi033 systemd[1]: Stopping Ceph mds.h for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:43.746 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: 2024-09-12T03:16:43.332+0000 7f0751578640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:43.746 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:43 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h[64847]: 2024-09-12T03:16:43.332+0000 7f0751578640 -1 mds.h *** got signal Terminated *** 2024-09-12T03:16:46.559 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:46 smithi033 podman[77092]: 2024-09-12 03:16:46.277647678 +0000 UTC m=+3.045565561 container died f10876fe397a1a812e8bef25c943c808b5a84b042239710194de9e9932b9e33e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731) 2024-09-12T03:16:46.936 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:46 smithi033 podman[77092]: 2024-09-12 03:16:46.649724513 +0000 UTC m=+3.417642392 container cleanup f10876fe397a1a812e8bef25c943c808b5a84b042239710194de9e9932b9e33e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-12T03:16:46.936 INFO:journalctl@ceph.mds.h.smithi033.stdout:Sep 12 03:16:46 smithi033 bash[77092]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-h 2024-09-12T03:16:47.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.h.service' 2024-09-12T03:16:47.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:47.122 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-12T03:16:47.123 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-12T03:16:47.123 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.k 2024-09-12T03:16:47.496 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:47 smithi033 systemd[1]: Stopping Ceph mds.k for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:47.996 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: 2024-09-12T03:16:47.689+0000 7f105ff03640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:47.997 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:47 smithi033 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k[64233]: 2024-09-12T03:16:47.689+0000 7f105ff03640 -1 mds.k *** got signal Terminated *** 2024-09-12T03:16:50.556 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:50 smithi033 podman[77226]: 2024-09-12 03:16:50.273621859 +0000 UTC m=+2.931788209 container died 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:16:50.922 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:50 smithi033 podman[77226]: 2024-09-12 03:16:50.647082958 +0000 UTC m=+3.305249307 container cleanup 4492eb4f0e6f5cfc1e1147cb6b4f217cebcfa9fcadb896b71eae9cf06e708efc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-09-12T03:16:50.922 INFO:journalctl@ceph.mds.k.smithi033.stdout:Sep 12 03:16:50 smithi033 bash[77226]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-k 2024-09-12T03:16:51.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.k.service' 2024-09-12T03:16:51.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:51.073 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-12T03:16:51.073 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-12T03:16:51.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.c 2024-09-12T03:16:51.354 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:51 smithi184 systemd[1]: Stopping Ceph mds.c for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:51.703 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c[66157]: 2024-09-12T03:16:51.354+0000 7fe0e1293640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:51.703 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:51 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c[66157]: 2024-09-12T03:16:51.354+0000 7fe0e1293640 -1 mds.c *** got signal Terminated *** 2024-09-12T03:16:54.795 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:54 smithi184 podman[76184]: 2024-09-12 03:16:54.311022322 +0000 UTC m=+3.048554178 container died be66189974ff1233d890bceb4be4e9073f75db2874c149dc6bf567b5ba045b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-12T03:16:55.154 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:54 smithi184 podman[76184]: 2024-09-12 03:16:54.920337049 +0000 UTC m=+3.657868895 container cleanup be66189974ff1233d890bceb4be4e9073f75db2874c149dc6bf567b5ba045b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-12T03:16:55.154 INFO:journalctl@ceph.mds.c.smithi184.stdout:Sep 12 03:16:54 smithi184 bash[76184]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-c 2024-09-12T03:16:55.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.c.service' 2024-09-12T03:16:55.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:55.318 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-12T03:16:55.318 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-12T03:16:55.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.f 2024-09-12T03:16:55.544 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:55 smithi184 systemd[1]: Stopping Ceph mds.f for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:56.044 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: 2024-09-12T03:16:55.641+0000 7fe96d2dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:56.044 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:55 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f[65857]: 2024-09-12T03:16:55.641+0000 7fe96d2dd640 -1 mds.f *** got signal Terminated *** 2024-09-12T03:16:58.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:58 smithi184 podman[76314]: 2024-09-12 03:16:58.311361576 +0000 UTC m=+2.774871601 container died 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-12T03:16:58.974 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:58 smithi184 podman[76314]: 2024-09-12 03:16:58.707221646 +0000 UTC m=+3.170731668 container cleanup 4d7ae73787439ffbd21684463eff4c26f8f1f495beb8662919e04386823b1c52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-12T03:16:58.975 INFO:journalctl@ceph.mds.f.smithi184.stdout:Sep 12 03:16:58 smithi184 bash[76314]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-f 2024-09-12T03:16:59.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.f.service' 2024-09-12T03:16:59.138 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:16:59.138 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-12T03:16:59.138 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-12T03:16:59.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.i 2024-09-12T03:16:59.294 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:16:59 smithi184 systemd[1]: Stopping Ceph mds.i for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:16:59.794 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:16:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: 2024-09-12T03:16:59.452+0000 7f37fac4c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:16:59.795 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:16:59 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i[66454]: 2024-09-12T03:16:59.452+0000 7f37fac4c640 -1 mds.i *** got signal Terminated *** 2024-09-12T03:17:02.661 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:17:02 smithi184 podman[76434]: 2024-09-12 03:17:02.306976419 +0000 UTC m=+2.953457719 container died fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3) 2024-09-12T03:17:02.953 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:17:02 smithi184 podman[76434]: 2024-09-12 03:17:02.744175054 +0000 UTC m=+3.390656347 container cleanup fa46e3df1bc2d028909ed65fa0c58d034f3bb1ee2337fabd05bf65eff758fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-12T03:17:02.953 INFO:journalctl@ceph.mds.i.smithi184.stdout:Sep 12 03:17:02 smithi184 bash[76434]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-i 2024-09-12T03:17:03.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.i.service' 2024-09-12T03:17:03.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:17:03.204 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-12T03:17:03.204 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-12T03:17:03.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.l 2024-09-12T03:17:03.522 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:03 smithi184 systemd[1]: Stopping Ceph mds.l for ef86fd2a-70af-11ef-bcea-c7b262605968... 2024-09-12T03:17:03.795 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: 2024-09-12T03:17:03.522+0000 7f6cb2c76640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-12T03:17:03.795 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:03 smithi184 ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l[65546]: 2024-09-12T03:17:03.522+0000 7f6cb2c76640 -1 mds.l *** got signal Terminated *** 2024-09-12T03:17:06.573 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:06 smithi184 podman[76554]: 2024-09-12 03:17:06.305833055 +0000 UTC m=+2.886550644 container died 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-12T03:17:06.873 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:06 smithi184 podman[76554]: 2024-09-12 03:17:06.630936219 +0000 UTC m=+3.211653809 container cleanup 6f1e9195617fe5968f61d8a24bf241b83a50d1b2248b8f8d856a2fc917b92398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3) 2024-09-12T03:17:06.873 INFO:journalctl@ceph.mds.l.smithi184.stdout:Sep 12 03:17:06 smithi184 bash[76554]: ceph-ef86fd2a-70af-11ef-bcea-c7b262605968-mds-l 2024-09-12T03:17:07.085 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ef86fd2a-70af-11ef-bcea-c7b262605968@mds.l.service' 2024-09-12T03:17:07.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-12T03:17:07.118 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-12T03:17:07.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-12T03:17:07.330 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:17:10.061 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-12T03:17:10.269 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:17:12.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-12T03:17:12.849 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:17:15.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:17:15.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:17:15.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-12T03:17:15.475 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-12T03:17:15.477 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi016/crash 2024-09-12T03:17:15.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash -- . 2024-09-12T03:17:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-12T03:17:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-09-12T03:17:15.528 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi033/crash 2024-09-12T03:17:15.529 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash -- . 2024-09-12T03:17:15.567 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-12T03:17:15.567 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-09-12T03:17:15.569 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi184/crash 2024-09-12T03:17:15.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash -- . 2024-09-12T03:17:15.605 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-12T03:17:15.606 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-09-12T03:17:15.607 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-12T03:17:15.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-12T03:17:15.652 INFO:tasks.cephadm:Compressing logs... 2024-09-12T03:17:15.652 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T03:17:15.655 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T03:17:15.657 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T03:17:15.679 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-12T03:17:15.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-12T03:17:15.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.9.log 2024-09-12T03:17:15.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-12T03:17:15.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-12T03:17:15.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.a.log 2024-09-12T03:17:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-12T03:17:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-12T03:17:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.g.log 2024-09-12T03:17:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.a.log: 82.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-12T03:17:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-12T03:17:15.684 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log 2024-09-12T03:17:15.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.6.log: /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.g.log: 90.4% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-12T03:17:15.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.d.log 2024-09-12T03:17:15.693 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-12T03:17:15.693 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.10.log 2024-09-12T03:17:15.693 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-12T03:17:15.693 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.b.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.k.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log: /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.h.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log: /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.e.log 2024-09-12T03:17:15.694 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.i.log 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-12T03:17:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log: /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.11.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.l.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.8.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.z.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.c.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.f.log 2024-09-12T03:17:15.696 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-12T03:17:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log: 87.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-12T03:17:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-12T03:17:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.d.log: 90.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-12T03:17:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.j.log 2024-09-12T03:17:15.700 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.x.log: 94.7% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.d.log.gz 2024-09-12T03:17:15.700 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-12T03:17:15.708 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.j.log: 94.7% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.j.log.gz 2024-09-12T03:17:15.709 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-12T03:17:15.721 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 83.2% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-12T03:17:15.749 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 95.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-12T03:17:15.780 INFO:teuthology.orchestra.run.smithi033.stderr: 90.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-12T03:17:15.808 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 83.2% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-12T03:17:15.920 INFO:teuthology.orchestra.run.smithi184.stderr: 93.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.z.log.gz 2024-09-12T03:17:15.959 INFO:teuthology.orchestra.run.smithi033.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-12T03:17:15.970 INFO:teuthology.orchestra.run.smithi033.stderr: 93.4% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-12T03:17:15.982 INFO:teuthology.orchestra.run.smithi184.stderr: 90.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-12T03:17:15.984 INFO:teuthology.orchestra.run.smithi033.stderr: 90.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-12T03:17:15.996 INFO:teuthology.orchestra.run.smithi184.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-12T03:17:15.998 INFO:teuthology.orchestra.run.smithi033.stderr: 87.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-12T03:17:16.140 INFO:teuthology.orchestra.run.smithi016.stderr: 90.4% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-12T03:17:16.169 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.g.log.gz 2024-09-12T03:17:16.269 INFO:teuthology.orchestra.run.smithi184.stderr: 90.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-12T03:17:16.333 INFO:teuthology.orchestra.run.smithi184.stderr: 87.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-12T03:17:17.814 INFO:teuthology.orchestra.run.smithi016.stderr: 90.1% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-12T03:17:21.681 INFO:teuthology.orchestra.run.smithi184.stderr: 95.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.f.log.gz 2024-09-12T03:17:22.720 INFO:teuthology.orchestra.run.smithi184.stderr: 95.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.l.log.gz 2024-09-12T03:17:28.203 INFO:teuthology.orchestra.run.smithi033.stderr: 91.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-12T03:17:28.761 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.9.log.gz 2024-09-12T03:17:30.067 INFO:teuthology.orchestra.run.smithi016.stderr: 95.2% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.a.log.gz 2024-09-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi016.stderr: 93.8% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-12T03:17:37.080 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-12T03:17:37.473 INFO:teuthology.orchestra.run.smithi016.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-12T03:17:37.474 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-12T03:17:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m21.806s 2024-09-12T03:17:37.475 INFO:teuthology.orchestra.run.smithi016.stderr:user 1m30.664s 2024-09-12T03:17:37.475 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m3.092s 2024-09-12T03:18:16.030 INFO:teuthology.orchestra.run.smithi184.stderr: 93.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-12T03:18:34.961 INFO:teuthology.orchestra.run.smithi184.stderr: 95.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.c.log.gz 2024-09-12T03:20:40.979 INFO:teuthology.orchestra.run.smithi184.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.8.log.gz 2024-09-12T03:20:51.929 INFO:teuthology.orchestra.run.smithi033.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.10.log.gz 2024-09-12T03:20:57.339 INFO:teuthology.orchestra.run.smithi184.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.11.log.gz 2024-09-12T03:21:00.802 INFO:teuthology.orchestra.run.smithi033.stderr: 94.1% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-12T03:21:22.609 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-12T03:21:39.838 INFO:teuthology.orchestra.run.smithi184.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-12T03:21:53.262 INFO:teuthology.orchestra.run.smithi184.stderr: 94.0% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-12T03:22:05.265 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-12T03:25:23.989 INFO:teuthology.orchestra.run.smithi033.stderr: 94.8% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.k.log.gz 2024-09-12T03:30:33.842 INFO:teuthology.orchestra.run.smithi184.stderr: 93.2% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.i.log.gz 2024-09-12T03:30:33.866 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-09-12T03:30:33.866 INFO:teuthology.orchestra.run.smithi184.stderr:real 13m18.172s 2024-09-12T03:30:33.866 INFO:teuthology.orchestra.run.smithi184.stderr:user 9m25.276s 2024-09-12T03:30:33.866 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m23.662s 2024-09-12T03:34:22.887 INFO:teuthology.orchestra.run.smithi033.stderr: 93.6% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.h.log.gz 2024-09-12T03:36:09.261 INFO:teuthology.orchestra.run.smithi033.stderr: 93.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.e.log.gz 2024-09-12T03:36:47.768 INFO:teuthology.orchestra.run.smithi033.stderr: 93.5% -- replaced with /var/log/ceph/ef86fd2a-70af-11ef-bcea-c7b262605968/ceph-mds.b.log.gz 2024-09-12T03:36:47.770 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-09-12T03:36:47.770 INFO:teuthology.orchestra.run.smithi033.stderr:real 19m32.098s 2024-09-12T03:36:47.770 INFO:teuthology.orchestra.run.smithi033.stderr:user 7m11.214s 2024-09-12T03:36:47.770 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m21.094s 2024-09-12T03:36:47.771 INFO:tasks.cephadm:Archiving logs... 2024-09-12T03:36:47.771 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi016/log 2024-09-12T03:36:47.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T03:36:59.121 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi033/log 2024-09-12T03:36:59.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T03:37:57.663 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi184/log 2024-09-12T03:37:57.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T03:39:25.771 INFO:tasks.cephadm:Removing cluster... 2024-09-12T03:39:25.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force 2024-09-12T03:39:25.986 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:39:26.354 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force 2024-09-12T03:39:26.588 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:39:27.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ef86fd2a-70af-11ef-bcea-c7b262605968 --force 2024-09-12T03:39:27.798 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: ef86fd2a-70af-11ef-bcea-c7b262605968 2024-09-12T03:39:28.510 INFO:tasks.cephadm:Removing cephadm ... 2024-09-12T03:39:28.510 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-12T03:39:28.530 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-12T03:39:28.560 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-12T03:39:28.576 INFO:tasks.cephadm:Teardown complete 2024-09-12T03:39:28.577 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-12T03:39:28.587 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-12T03:39:28.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T03:39:28.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T03:39:28.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T03:39:28.761 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-12T03:39:28.773 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-12T03:39:28.773 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T03:39:28.776 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T03:39:28.779 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T03:39:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-12T03:39:28.793 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-09-12T03:39:28.793 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 -2889us[-2896us] +/- 59ms 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 +1696us[+1688us] +/- 59ms 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +2308us[+2301us] +/- 84ms 2024-09-12T03:39:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 -3078us[-3087us] +/- 59ms 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 +1518us[+1509us] +/- 59ms 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +2118us[+2109us] +/- 84ms 2024-09-12T03:39:28.832 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T03:39:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T03:39:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-09-12T03:39:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -3020us[-3000us] +/- 59ms 2024-09-12T03:39:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 +1555us[+1555us] +/- 59ms 2024-09-12T03:39:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +2169us[+2189us] +/- 84ms 2024-09-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T03:39:28.834 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-12T03:39:28.846 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-12T03:39:28.847 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-12T03:39:28.887 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T03:39:28.989 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T03:39:29.103 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T03:39:29.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T03:39:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:29.248 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T03:39:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:29.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-12T03:39:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:29.308 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-12T03:39:29.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-12T03:39:29.370 INFO:teuthology.task.internal:Duration was 4082.263566 seconds 2024-09-12T03:39:29.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-12T03:39:29.404 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-12T03:39:29.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T03:39:29.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T03:39:29.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T03:39:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T03:39:29.473 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T03:39:29.478 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T03:39:29.848 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-12T03:39:29.848 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T03:39:29.848 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T03:39:29.874 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-09-12T03:39:29.874 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T03:39:29.905 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-09-12T03:39:29.906 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T03:39:29.930 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-12T03:39:29.931 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T03:39:29.933 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T03:39:29.949 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T03:39:29.998 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-12T03:39:29.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T03:39:30.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T03:39:30.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T03:39:32.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-12T03:39:32.443 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-12T03:39:32.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T03:39:32.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T03:39:32.500 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T03:39:32.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-12T03:39:32.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.568 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-12T03:39:32.584 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-09-12T03:39:32.614 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-09-12T03:39:32.632 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:32.647 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:32.664 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T03:39:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T03:39:32.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-12T03:39:32.700 INFO:teuthology.task.internal:Transferring archived files... 2024-09-12T03:39:32.701 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi016 2024-09-12T03:39:32.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T03:39:32.792 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi033 2024-09-12T03:39:32.792 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T03:39:32.834 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-11_21:24:17-fs-squid-distro-default-smithi/7901630/remote/smithi184 2024-09-12T03:39:32.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T03:39:32.874 INFO:teuthology.task.internal:Removing archive directory... 2024-09-12T03:39:32.874 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T03:39:32.877 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T03:39:32.879 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T03:39:32.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-12T03:39:32.936 INFO:teuthology.task.internal:Not uploading archives. 2024-09-12T03:39:32.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-12T03:39:32.971 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-12T03:39:32.972 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T03:39:32.974 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T03:39:32.977 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T03:39:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 03:39 /home/ubuntu/cephtest 2024-09-12T03:39:32.991 INFO:teuthology.orchestra.run.smithi033.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 03:39 /home/ubuntu/cephtest 2024-09-12T03:39:32.993 INFO:teuthology.orchestra.run.smithi184.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 03:39 /home/ubuntu/cephtest 2024-09-12T03:39:33.000 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-12T03:39:33.011 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-12T03:39:33.074 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 4082.263566493988 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-12T03:39:33.074 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T03:39:33.167 INFO:teuthology.run:pass